[MRG] FIX for exception thrown when y does not have all classes

This is a fix for #3581
I think we reached a consensus on this, IRL? :)