Handle the multiclass separately for inverse_label_binarize