Question: Suppose we have a binary classification problem, we have two classes of 1s and 0s as our target. I aim to use a tree classifier to predict 1s and 0s given the features. Further, I can use SHAP values to rank the feature importance that are predictive of 1s and 0s. Until now everything is good! Now suppose that I want to know importance of features that are predictive of 1s only, what is the recommended approach there? I can split my data into two parts (nominally: df_tot = df_zeros + df_ones) and use df_ones in my classifier and then extract the SHAP values for that, however doing so the target would only have 1s and so the model does not really learn to classify anything. So I am wondering how does one approach such problem?

Answer: Let's prepare some binary classification data: from seaborn import load_dataset from sklearn.model_selection import train_test_split from lightgbm import LGBMClassifier import shap titanic = load_dataset("titanic") X = titanic.drop(["survived","alive","adult_male","who",'deck'],1) y = titanic["survived"] features = X.columns cat_features = [] for cat in X.select_dtypes(exclude="number"): cat_features.append(cat) # think about meaningful ordering instead X[cat] = X[cat].astype("category").cat.codes.astype("category") X_train, X_val, y_train, y_val = train_test_split(X,y,train_size=.8, random_state=42) clf = LGBMClassifier(max_depth=3, n_estimators=1000, objective="binary") clf.fit(X_train,y_train, eval_set=(X_val,y_val), early_stopping_rounds=100, verbose=100)  To answer your question, to extract shap values on a per class basis one may subset them by class label: explainer = shap.TreeExplainer(clf) shap_values = explainer.shap_values(X_train) sv = np.array(shap_values) y = clf.predict(X_train).astype("bool") # shap values for survival sv_survive = sv[:,y,:] # shap values for dying sv_die = sv[:,~y,:]  However a more interesting question what you can do with these values. In general, one can gain valuable insights by looking at summary_plot (for the whole dataset): shap.summary_plot(shap_values[1], X_train.astype("float"))    Interpretation (globally):  sex, pclass and age were most influential features in determining outcome being a male, less affluent, and older decreased chances of survival   Top 3 global most influential features can be extracted as follows: idx = np.abs(sv[1,:,:]).mean(0).argsort() features[idx[:-4:-1]] # Index(['sex', 'pclass', 'age'], dtype='object')  If you want to analyze on a per class basis, you may do this separately for survivors (sv[1,y,:]): # top3 features for probability of survival idx = sv[1,y,:].mean(0).argsort() features[idx[:-4:-1]] # Index(['sex', 'pclass', 'age'], dtype='object')  The same for those who did not survive (sv[0,~y,:]): # top3 features for probability of dieing idx = sv[0,~y,:].mean(0).argsort() features[idx[:3]] # Index(['alone', 'embark_town', 'parch'], dtype='object')  Note, we are using mean shap values here and saying we are interested in biggest values for survivors and lowest values for those who are not (lowest values, close to 0, may also mean having no constant, one-directional influence at all). Using mean on abs may also make sense, but the interpretation will be most influential, regardless of direction. To make an educated choice either one prefers means or means of abs' one has to be aware of the following facts:   shap values could be both positive and negative shap values are symmetrical, and increasing/decreasing probability of one class decreases/increases probability of the other by the same amount (due to p₁ = 1 - p₀)   Proof: #shap values sv = np.array(shap_values) #base values ev = np.array(explainer.expected_value) sv_died, sv_survived = sv[:,0,:] # + constant print(sv_died, sv_survived, sep="\n") # [-0.73585563 1.24520748 0.70440429 -0.15443337 -0.01855845 -0.08430467 0.02916375 -0.04846619 0. -0.01035171] # [ 0.73585563 -1.24520748 -0.70440429 0.15443337 0.01855845 0.08430467 -0.02916375 0.04846619 0. 0.01035171]  Most probably you'll find out sex and age played the most influential role both for survivors and not; hence, rather than analyzing most influential features per class, it would be more interesting to see what made two passengers of the same sex and age one survive and the other not (hint: find such cases in the dataset, feed one as background, and analyze shap values for the other, or, try analyzing one class vs the other as background). You may do further analysis with dependence_plot (on a global or per class basis): shap.dependence_plot("sex", shap_values[1], X_train)    Interpretation (globally):  males had lower probability of survival (lower shap values) pclass (affluence) was the next most influential factor: higher pclass (less affluence) decreased chance of survival for female and vice versa for males