Cleanup the SSL related structures properly