Make the ssl stream a bit less prone to throwing exceptions on leave