Don't call curl_multi_remove_handle twice on the same handler