Don't call curl_multi_remove_handle twice on the same handler

Sorry for the double issue. Forgot pull's open one as well.