Package: wittgenstein
wittgensteinAnd is there not also the case where we play and--make up the rules as we go along?-Ludwig WittgensteinSummaryThis package implements two interpretable coverage-based ruleset algorithms: IREP and RIPPERk, as well as additional features for model interpretation.Performance is similar to sklearn's DecisionTree CART implementation (seePerformance Tests).For explanation of the algorithms, see my article inTowards Data Science, or the papers below, underUseful References.InstallationTo install, use$pipinstallwittgensteinTo uninstall, use$pipuninstallwittgensteinRequirementspandasnumpypython version>=3.6UsageUsage syntax is similar to sklearn's.TrainingOnce you have loaded and split your data...>>>importpandasaspd>>>df=pd.read_csv(dataset_filename)>>>fromsklearn.model_selectionimporttrain_test_split# Or any other mechanism you want to use for data partitioning>>>train,test=train_test_split(df,test_size=.33)Use thefitmethod to train aRIPPERorIREPclassifier:>>>importwittgensteinaslw>>>ripper_clf=lw.RIPPER()# Or irep_clf = lw.IREP() to build a model using IREP>>>ripper_clf.fit(df,class_feat='Poisonous/Edible',pos_class='p')# Or pass X and y data to .fit>>>ripper_clf<RIPPER(max_rules=None,random_state=2,max_rule_conds=None,verbosity=0,max_total_conds=None,k=2,prune_size=0.33,dl_allowance=64,n_discretize_bins=10)withfitruleset># Hyperparameter details available in the docstrings and TDS article belowAccess the underlying trained model with theruleset_attribute, or output it without_model(). A ruleset is a disjunction of conjunctions -- 'V' represents 'or'; '^' represents 'and'.In other words, the model predicts positive class if any of the inner-nested condition-combinations are all true:>>>ripper_clf.out_model()# or ripper_clf.ruleset_[[Odor=f]V[Gill-size=n^Gill-color=b]V[Gill-size=n^Odor=p]V[Odor=c]V[Spore-print-color=r]V[Stalk-surface-below-ring=y^Stalk-surface-above-ring=k]V[Habitat=l^Cap-color=w]V[Stalk-color-above-ring=y]]IREPmodels tend be higher bias,RIPPER's higher variance.ScoringTo score a trained model, use thescorefunction:>>>X_test=test.drop('Poisonous/Edible',axis=1)>>>y_test=test['Poisonous/Edible']>>>ripper_clf.score(test_X,test_y)0.9985686906328078Default scoring metric is accuracy. You can pass in alternate scoring functions, including those available through sklearn:>>>fromsklearn.metricsimportprecision_score,recall_score>>>precision=clf.score(X_test,y_test,precision_score)>>>recall=clf.score(X_test,y_test,recall_score)>>>print(f'precision:{precision}recall:{recall}')precision:0.9914...,recall:0.9953...PredictionTo perform predictions, usepredict:>>>ripper_clf.predict(new_data)[:5][True,True,False,True,False]Predict class probabilities withpredict_proba:>>>ripper_clf.predict_proba(test)# Pairs of negative and positive class probabilitiesarray([[0.01212121,0.98787879],[0.01212121,0.98787879],[0.77777778,0.22222222],[0.2,0.8],...We can also ask our model to tell us why it made each positive prediction usinggive_reasons:>>>ripper_clf.predict(new_data[:5],give_reasons=True)([True,True,False,True,True][<Rule[physician-fee-freeze=n]>],[<Rule[physician-fee-freeze=n]>,<Rule[synfuels-corporation-cutback=y^adoption-of-the-budget-resolution=y^anti-satellite-test-ban=n]>],# This example met multiple sufficient conditions for a positive prediction[],[<Ruleobject:[physician-fee-freeze=n]>],[])Model selectionwittgenstein is compatible with sklearn model_selection tools such ascross_val_scoreandGridSearchCV, as well
as ensemblers likeStackingClassifier.Cross validation:>>># First dummify your categorical features and booleanize your class values to make sklearn happy>>>X_train=pd.get_dummies(X_train,columns=X_train.select_dtypes('object').columns)>>>y_train=y_train.map(lambdax:1ifx=='p'else0)>>>cross_val_score(ripper_clf,X_train,y_train)Grid search:>>>fromsklearn.model_selectionimportGridSearchCV>>>param_grid={"prune_size":[0.33,0.5],"k":[1,2]}>>>grid=GridSearchCV(estimator=ripper,param_grid=param_grid)>>>grid.fit(X_train,y_train)Ensemble:>>>fromsklearn.ensembleimportStackingClassifier>>>fromsklearn.treeimportDecisionTreeClassifier>>>fromsklearn.naive_bayesimportGaussianNB>>>fromsklearn.linear_modelimportLogisticRegression>>>tree=DecisionTreeClassifier(random_state=42)>>>nb=GaussianNB(random_state=42)>>>estimators=[("rip",ripper_clf),("tree",tree),("nb",nb)]>>>ensemble_clf=StackingClassifier(estimators=estimators,final_estimator=LogisticRegression())>>>ensemble_clf.fit(X_train,y_train)Defining and altering modelsYou can directly specify a new model, modify a preexisting model, or train from a preexisting model -- whether to take into account subject matter expertise, to create a baseline for scoring, or for insight into what the model is doing.To specify a new model, useinit_ruleset:>>>ripper_clf=RIPPER(random_state=42)>>>ripper_clf.init_ruleset("[[Cap-shape=x^Cap-color=n] V [Odor=c] V ...]",class_feat=...,pos_class=...)>>>ripper_clf.predict(df)...To modify a preexisting model, useadd_rule,replace_rule,remove_rule, orinsert_rule. To alter a model by index, usereplace_rule_at,remove_rule_at, orinsert_rule_at:>>>ripper_clf.replace_rule_at(1,'[Habitat=l]')>>>ripper_clf.insert_rule(insert_before_rule='[Habitat=l]',new_rule='[Gill-size=n ^ Gill-color=b]')>>>ripper_clf.out_model()[[delicious=y^spooky-looking=y]V[Gill-size=n^Gill-color=b]V[Habitat=l]]To specify a starting point for training, useinitial_modelwhen callingfit:>>>ripper_clf.fit(>>>X_train,>>>y_train,>>>initial_model="[[delicious=y^spooky-looking=y] V [Odor=c]]")Expected string syntax for a Ruleset is[<Rule1> V <Rule2> V ...], for a Rule[<Cond1>^<Cond2>^...], and for a Condfeature=value`. '^' represents 'and'; 'V' represents 'or'. (See theTrainingsection above).Interpreter modelsUse the interpret module to interpret non-wittgenstein models.interpret_modelgenerates a ruleset that approximates some black-box model. It does to by fitting a wittgenstein classifier to the predictions of the other model.# Train the model we want to interpret>>>fromtensorflow.kerasimportSequential>>>fromtensorflow.keras.layersimportDense>>>mlp=Sequential()>>>mlp.add(Dense(60,input_dim=13,activation='relu'))>>>mlp.add(Dense(30,activation='relu'))>>>mlp.add(Dense(1,activation='sigmoid'))>>>mlp.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])>>>mlp.fit(>>>X_train,>>>y_train,>>>batch_size=1,>>>epochs=10)# Create and fit wittgenstein classifier to use as a model interpreter.>>>fromwittgenstein.interpretimportinterpret_model,score_fidelity>>>interpreter=RIPPER(random_state=42)>>>interpret_model(model=mlp,X=X_train,interpreter=interpreter).out_pretty()[[Proline=>1227.0]V[Proline=880.0-1048.0]V[Proline=1048.0-1227.0]V[Proline=736.0-880.0]V[Alcalinityofash=16.8-17.72]]We can also use the now-fitted interpreter to approximate the reasons behind the underlying model's positive predictions. (SeePrediction).>>>preds=(mlp.predict(X_test.tail())>.5).flatten()>>>_,interpretation=interpreter.predict(X_test.tail(),give_reasons=True)>>>print(f'tf preds:{preds}\n')>>>interpretationtfpreds:[TrueFalseFalseTrueFalse][[<Rule[Proline=880.0-1048.0]>],[],[],[<Rule[Proline=736.0-880.0]>,<Rule[Alcalinityofash=16.8-17.72]>],[]]Score how faithfully the interpreter fits the underlying model withscore_fidelity.>>>score_fidelity(>>>X_test,>>>interpreter,>>>model=mlp,>>>score_function=[precision_score,recall_score,f1_score])[1.0,0.7916666666666666,0.8837209302325582]IssuesIf you encounter any issues, or if you have feedback or improvement requests for how wittgenstein could be more helpful for you, please post them toissues, and I'll respond.ContributingContributions are welcome! If you are interested in contributing, let me know atilan.moscovitz@gmail.comor onlinkedin.Useful referencesMy article inTowards Data Scienceexplaining IREP, RIPPER, and wittgensteinFurnkrantz-Widmer IREP paperCohen's RIPPER paperPartial decision treesBayesian RulesetsC4.5 paper including all the gory details on MDLPhilosophical InvestigationsChangelogv0.3.4: 4/3/2022Improvements to predict_proba calculation, including smoothingv0.3.2: 8/8/2021Speedup for binning continuous features (~several orders of magnitude)Add support for expert feedback: Ability to explicitly specify and alter models.Add surrogate interpreterAdd support for non-pandas datasets (ex. numpy arrays)v0.2.3: 5/21/2020Minor bugfixes and optimizationsv0.2.0: 5/4/2020Algorithmic optimizations to improve training speed (~10x - ~100x)Support for training on iterable datatypes besides DataFrames, such as numpy arrays and python listsCompatibility with sklearn ensembling metalearners and sklearn model_selection.predict_probareturns probas in neg, pos orderCertain parameters (hyperparameters, random_state, etc.) should now be passed into IREP/RIPPER constructors rather than the .fit method.Sundry bugfixes