Package: imodelsx
Scikit-learn friendly library to interpret, predict, and prompt-engineer text datasets.📖 demo notebooksModelReferenceOutputDescriptioniPrompt📖,🗂️,🔗,📄ExplanationGenerates a prompt thatexplains patterns in data (Official)D3📖,🗂️,🔗,📄ExplanationExplain the difference between two distributionsAutoPromptㅤㅤ🗂️,🔗,📄ExplanationFind a natural-language promptusing input-gradients (⌛ In progress)Aug-GAM📖,🗂️,🔗,📄Linear modelFit better linear model using an LLMto extract embeddings (Official)Aug-Tree📖,🗂️,🔗,📄Decision treeFit better decision tree using an LLMto expand features (Official)SASCㅤㅤ🗂️,🔗,📄ExplanationExplain a black-box text moduleusing an LLM (Official)Bag of Ngrams🗂️Linear modelLearn a linear model of ngramsLinear Finetune📖,🗂️Black-box modelFinetune a single linear layeron top of LLM embeddings📖Demo notebooks🗂️Doc   🔗 Reference code   📄 Research paper⌛ We plan to support other interpretable algorithms likeRLPrompt,CBMs, andNBDT. If you want to contribute an algorithm, feel free to open a PR 😄QuickstartInstallation:pip install imodelsx(or, for more control, clone and install from source)Demos: see thedemo notebooksiPromptfromimodelsximportexplain_dataset_iprompt,get_add_two_numbers_dataset# get a simple dataset of adding two numbersinput_strings,output_strings=get_add_two_numbers_dataset(num_examples=100)foriinrange(5):print(repr(input_strings[i]),repr(output_strings[i]))# explain the relationship between the inputs and outputs# with a natural-language prompt stringprompts,metadata=explain_dataset_iprompt(input_strings=input_strings,output_strings=output_strings,checkpoint='EleutherAI/gpt-j-6B',# which language model to usenum_learned_tokens=3,# how long of a prompt to learnn_shots=3,# shots per examplen_epochs=15,# how many epochs to searchverbose=0,# how much to printllm_float16=True,# whether to load the model in float_16)--------promptsisalistoffoundnatural-languagepromptstringsD3 (DescribeDistributionalDifferences)fromimodelsximportexplain_dataset_d3hypotheses,hypothesis_scores=explain_dataset_d3(pos=positive_samples,# List[str] of positive examplesneg=negative_samples,# another List[str]num_steps=100,num_folds=2,batch_size=64,)Aug-imodelsUse these just a like a scikit-learn model. During training, they fit better features via LLMs, but at test-time they are extremely fast and completely transparent.fromimodelsximportAugGAMClassifier,AugTreeClassifier,AugGAMRegressor,AugTreeRegressorimportdatasetsimportnumpyasnp# set up datadset=datasets.load_dataset('rotten_tomatoes')['train']dset=dset.select(np.random.choice(len(dset),size=300,replace=False))dset_val=datasets.load_dataset('rotten_tomatoes')['validation']dset_val=dset_val.select(np.random.choice(len(dset_val),size=300,replace=False))# fit modelm=AugGAMClassifier(checkpoint='textattack/distilbert-base-uncased-rotten-tomatoes',ngrams=2,# use bigrams)m.fit(dset['text'],dset['label'])# predictpreds=m.predict(dset_val['text'])print('acc_val',np.mean(preds==dset_val['label']))# interpretprint('Total ngram coefficients: ',len(m.coefs_dict_))print('Most positive ngrams')fork,vinsorted(m.coefs_dict_.items(),key=lambdaitem:item[1],reverse=True)[:8]:print('\t',k,round(v,2))print('Most negative ngrams')fork,vinsorted(m.coefs_dict_.items(),key=lambdaitem:item[1])[:8]:print('\t',k,round(v,2))Easy baselinesEasy-to-fit baselines that follows the same API.fromimodelsximportLinearFinetuneClassifier,LinearNgramClassifier# fit a simple one-layer finetune on top of LLM embeddingsm=LinearFinetuneClassifier(checkpoint='distilbert-base-uncased',)m.fit(dset['text'],dset['label'])preds=m.predict(dset_val['text'])acc=(preds==dset_val['label']).mean()print('validation acc',acc)SASCHere, we explain amodulerather than a datasetfromimodelsximportexplain_module_sasc# a toy module that responds to the length of a stringmod=lambdastr_list:np.array([len(s)forsinstr_list])# a toy dataset where the longest strings are animalstext_str_list=["red","blue","x","1","2","hippopotamus","elephant","rhinoceros"]explanation_dict=explain_module_sasc(text_str_list,mod,ngrams=1,)Related workimodels package (JOSS 2021github) - interpretable ML package for concise, transparent, and accurate predictive modeling (sklearn-compatible).Adaptive wavelet distillation (NeurIPS 2021pdf,github) - distilling a neural network into a concise wavelet modelTransformation importance (ICLR 2020 workshoppdf,github) - using simple reparameterizations, allows for calculating disentangled importances to transformations of the input (e.g. assigning importances to different frequencies)Hierarchical interpretations (ICLR 2019pdf,github) - extends CD to CNNs / arbitrary DNNs, and aggregates explanations into a hierarchyInterpretation regularization (ICML 2020pdf,github) - penalizes CD / ACD scores during training to make models generalize betterPDR interpretability framework (PNAS 2019pdf) - an overarching framewwork for guiding and framing interpretable machine learning