When you call delayed on `_fit_binary`, you save a reference to all input argument.
To solve the issue, we should pass to `_fit_binary` a reference to the sparse `Y and the class index of the job.
So the signature of `_fit_binary` would become `_fit_binary(estimator, X, Y, class_index, class_name)`
where class is the column index. 

I think that you should treat the sparse and dense case in `_fit_binary`.