MaNtLE: Model-agnostic Natural Language Explainer
Authors: Rakesh R Menon
Kerem Zaman
Shashank Srivastava
Venue: EMNLP 2023 Main (2023)
Keywords: explainable AI
interpretability
TL;DR: We propose MaNtLE a model-agnostic natural language explainer that analyzes multiple classifier predictions and generates faithful natural language explanations of classifier rationale for structured classification tasks.

Abstract: Understanding the internal reasoning behind the predictions of machine learning systems is increasingly vital, given their rising adoption and acceptance. While previous approaches, such as LIME generate algorithmic explanations by attributing importance to input features for individual examples, recent research indicates that practitioners prefer examining language explanations that explain sub-groups of examples (Lakkaraju et al., 2022). In this paper, we introduce MaNtLE, a model-agnostic natural language explainer that analyzes a set of classifier predictions and generates faithful natural language explanations of classifier rationale for structured classification tasks. MaNtLE uses multi-task training on thousands of synthetic classification tasks to generate faithful explanations. Our experiments indicate that, on average, MaNtLE-generated explanations are at least 11% more faithful compared to LIME and Anchors explanations across three tasks. Human evaluations demonstrate that users can better predict model behavior using explanations from MaNtLE compared to other techniques.

MaNtLE: A Model-agnostic Natural Language Explainer
Rakesh R Menon
Kerem Zaman
Shashank Srivastava
UNC Chapel Hill
(rrmenon, kzaman, ssrivastava)@cs. unc. edu
Abstract
If Himmran lesse ther ther ra crediction ia
%
Understanding the internal reasoning behind the predictions of machine learning systems is increasingly vital, given their rising adoption and acceptance. While previous approaches, such as LIME generate algorithmic explana- tions by attributing importance to imput fea- tures for individual examples, recent research indicates that practitioners prefer examining language explanations that explain sub- sub-groups of examples (Lakkaraju et al., 2022). In this paper, we introduce MaNtLE, a model-agnostic natural language explainer that analyzes of classifier predictions and generates faithful nat- ural language explanations classifier ratio- nale for structured classification tasks. MaNtLE uses multi-task training on thousands of syn- thetic classification tasks generate faithful explanations. Our experiments indicate that, average, MaNtLE- generated explanations are at least 1 % faithful compared to LIME and Anchors explanations across three tasks. Hu- man evaluations demonstrate that users predict model behavior better using explanations from MaNtLE compared to other techniques.'
LOY
peskal
MaNtLE
Fuser age grentee equalis then onddcct retevance cantaiityy
37
11e
doal Zno shot generattaaion
jngas
coeguee not equad to ItS, <.-0O
+153
VS -506
-50
Figure MaNtLE model-agnostic natural lan- guage explainer. Following massive multi-task learning. MaNtLE can generate explanations of decision iiaiionnmaiinggg rationale for classifiers and datasets.
lists (Anchors, Ribeiro et al. (2018)), and model- generated explanations (Rajani al., 2019; Narang et al., 2020). However, these explanations focus on model behavior the level of individual exam- ples rather than subgroups, which makes it hard for designers to identify systemic problems and refine models. Recent studies indicate that this limita- tion contributes to practitioners reluctance to use machine learning systems for critical applications (Lakkaraju et al., 2019, 2022).
1 Introduction
The increasing adoption of black-box machine learning models across various applications (Shi al., 2022; Dada al., 2019) has led a press- ing need for ludestaaddet explanations of their decision- decision-making process. While such models may yield high predictive accuracies, their underly- ing reasoning often remains opaque to users. This lack of transparency is critical barrier to their adoption in critical domains, such healthcare, law, and medicine.
In this paper, we introduce MaNtLE, a model- agnostic natural language explainer that analyzes multiple classifier predictions and generates faith- ful natural language explanations of the classi- fier's reasoning for structured classification tasks, as depicted in Figure 1. The goal of MaNtLE is to explain the rationale of classifiers on real- world tasks. To develop MaNtLE, we fine-tune a T5-Large model on thousands of synthetic classi- fication tasks, each paired with natural language explanations, in a multi-task learning setup follow- ing recent research (Wei et al., 2022: Sanh et al., 2022; Mishra al., 2022; Menon al., 2022). $3.5, we discuss inference procedures to improve explanation quality and adapt the model trained on
To interpret machine learning model decisions, previous research proposed techniques like feature importances (LIME, Ribeiro et al. (2016)), rule
'Code and pre-trained models are available at: https: //github. cwm/ www.ern//aaattttteeeeeeeee

---

synthetic data for real-world tasks. We test MaNtLE explanations on real-world tasks by assessing their ability to aid explanation-guided zero-shot classifiers in the CLUES-Real benchmark (Menon et al., 2022). Our results show that MaNtLE explanations are helpful as human-written expla- nations in guiding classifiers ($5.2). We also pare the faithfulness (Jacovi and Goldberg, 2020) and simulatability (Hase and Bansal, 2020) of ex- planations generated by MaNtLE, LIME, and An- chors for four classification techniques on three real-world tasks ($5.3). When the number of avail- able examples is comparable, MaNtLE explanations are, on average, 37 % more faithful than LIME and 11 % more faithful than Anchors.
We analyze factors contributing to MaNtLE's per- formance and outline opportunities for improving explanations.
Related Work
Explainability methods. Previous research has proposed methods for understanding model ratio- nale, which can be broadly categorized into feature attribution and language explanation methods.
Feature attribution techniques (Kim et al., 2018; Lundberg and Lee, 2017: Sundararajan et al.. 2017) provide information about how models use certain features to derive classification decisions, utilizing methods such as sparse linear regression models (Ribeiro et al., 2016, LIME) and rule lists (Ribeiro et al., 2018, Anchors). These methods have two shortcomings: (a) they explain model behavior in a local region around an example, and (b) they gener- explanations by solving search problem over of instances from the original training data by generating perturbations around an example (typi- cally, ~ 1000 examples). However, gaining access to model predictions on thousands of examples can be resource-intensive or financially unviable for large models, such GPT-3 (Brown et al., 2020).
In user studies ($5.4), we observe that users without machine learning expertise prefer expla- nations from MaNtLE over attribution--core-base explanations by LIME (overall preference of 3.44 vs 2.16 on a five-point Likert scale; < 0.001 t- test). Further, users can predict model behavior better using MaNtLE explanations in at least 25% more instances than LIME and Anchors for the adult dataset (Dua and Graff, 2017). Our results corroborate the findings of Lakkaraju et al. (2022) on the need for providing natural language expla- nations that explain subgroups of examples to help practitioners interpret ML models. We find that increasing the number of examples accessible by MaNtLE can enhance explanation quality, highlight- ing possibilities for using the model in resource- rich settings. Finally, we analyze the contributions of the number of multitask-training tasks used and model size on the quality of MaNtLE's generated explanations.
Recent works like CAGE (Rajani et al., 2019) and WT5 (Narang al., 2020) explore natural language explanations for language understanding tasks. However, these explanations are specific individual examples and focus on improving classi- fication performance rather than interpreting model behavior. Moreover, training these models often demands significant number of human-written explanations, which can be impractical. Our work diverges from this line of research as seek explain model behavior rather than improve classi- fication models, using only a few examples.
In summary, our contributions are:
We develop MaNtLE, a model-agnostic natural lan- guage explainer that generates faithful language explanations of classifier rationale.
We demonstrate the efficacy of MaNtLE by (1) comparing the classification-utiityy of explana- tions with human-written explanations on the CLUES-Real benchmark, and (2) evaluating the faithfulness and simulatability of explanations compared popular approaches.
Multi-task Training of Language Models. Large language models (LLMs) pretrained on large text corpora have shown impressive performances on various tasks (Brown et al., 2020; Scao et al., 2022). Some recent works (Wei et al., 2022; Sanh et al., 2022) have explored multitask training of language models on labeled datasets paired with natural language instructions to enhance zero-shot generalization, a procedure called instruction fine- tuning. In contrast, MaNtLE generates explanations when prompted with reature-predictoe pairs.
We show that human users predict model behav- ior better with MaNtLE explanations compared LIME and Anchors. Human users also rate MaNtLE explanations as better in understandabil- ity, informativeness, and overall preference.

---

3 MaNtLE
number and diversity of tasks used during multi- task training. However, obtaining language ex- planations for thousands of classifiers for training purposes is infeasible. To address this, we adopt a strategy proposed by Menon et al. (2022) and generate synthetic classification tasks programmat- ically, where the tasks have known natural language explanations and the examples are used to predict these explanations. To diversify the set of tasks, we generate tasks with explanations that include a vari- ety of logical operations in addition explanations conditioned on a single feature, e.g., 'If fruit color is red, then apple'. We mirror Menon et al. (2022) in varying synthetic tasks and explanations in terms of the logical operations present, including conjunc- lions, disjunctions, negations, and quantifiers. We refer the reader to ŞA.3 and Table for information on the task variations/compllxitii tions are based on prior research, which explores the linguistic elements used by humans to commu- nicate task intents through language (Chopra et al., 2019).
3.1 Problem Setup
MaNtLE takes as input a set of input-prediction pairs, {(X;, s), .N) ), from a black-box clas- sifier, O,, trained for structured classification task I. The output from MaNtLE is explanation, e E E. where & is the set of all possible natural language explanations. An explanation a natural language statement that describes the behavior of the classi- fier in predicting labels 10M. / for the inputs XI:M- Figure 1 shows illustrative examples of explana- tions generated for three classifiers and tasks.
3.2 Model
We frame the task of explanation generation from input-prediction pairs as a eequence-to-squeene problem. Framing the task sequence problem enables fine-tuning of pre- trained language models to generate language ex- planations. We initialize MaNtLE using T5-Large (Raffel et al., 2020) language model. We lin- earize input-prediction pairs as text and predict text explanations using our model. Our input- linearization strategy converts a set of k examples into text with an additional prompt, explanation: <extra_ise i___d_op id_0>. Figure illustrates this lineariza- tion and encoding process with two examples. On the decoder side, we begin prediction from the <extra_id id__ddd_>>>__p Ø> sentinel token, matching the span- prediction task that T5 is optimized for.
sequence-to-
These varia-
Additionally, we assume that users are interested in understanding the rationales for specific la- bel of a classifier at a time. Thus, we re- re-frame all examples as binary classification examples for MaNtLE, where the label of interest is maintained, as {label} say, and the remaining labels are re- labeled as "not (label)".
3.4 Training Details
lbl
Per-example Text Encoding:
We perform multi-task training for MaNtLE on 200K synthetic datasets that have five features in all tasks covering wide range of explanation types ($3.2). We cap the maximum number of to- kens to 1024 and 64 tokens, respectively, for the input and the output. This corresponds to packing between 10-12 examples in the input when gen- erating explanations for classifier rationale. We optimize the model to maximize the likelihood of ground-truth explanations given the set of exam- ples. Additionally, since MaNtLE derives explana- tions by extracting patterns in examples from both classes ([label} and not {label})) we ensure that at least 10% of the input examples belong to each of the two classes. The model is trained for 2 million steps with a batch size of 4 using Adam W (Loshchilov and Hutter, 2019), learning rate of le-5, and a linear decay schedule with a warmup of 50K steps.
yes
True
equal to yes. b equal to 2. 161 equal to True.
False
equal to no. equal to 2. 1bl equal to False.
MaNILE Input:
example 1: equal to yes. cqual to 2, coual to True.
exaeple 2: equal to no. b equel to 2. 161 equel to False. explanation: soxtra_ld_0>
Figure 2: MaNtLE's strategy for linearizing structured data to text.
3.3 Multi-task Training
To train MaNtLE, we use massive multi-task training following recent advancements in this area (Wei et al., 2022: Sanh et al., 2022; Mishra et al., 2022). This approach equips MaNtLE with the ability to generate explanations for any classifier without the need for additional fine-tuning: The degree of gen- eralization achieved by the model depends on the
'Due to computational constraints, we do not experiment with larger models (T5-3B or 15-11B).
After training, we select the best model check-

---

point using the highest score (sum of all metrics in $3.6) on the validation sets of 20 randomly selected synthetic tasks from those used during training.
the different task complexities described in $A.3 (presence/absence of conjunctionss, negations, and quantifiers; Table 4). These datasets were not used during training and are therefore considered un- seen tasks. We use greedy decoding to generate the explanations from MaNtLE.
3.5 Inference Techniques
For inference, we experiment with three decoding approaches. The first is greedy decoding, where we generate the most likely explanation from MaNtLE conditioned on the input. The second approach aims to improve explanations by sampling multiple candidates and returning the explanation that most faithfully explains model behavior on the inputs (see $3.6 for the definition of faithfulness). For this, we use beam-search, with beam size 20, and generate 20 explanations from MaNtLE, following which we return the most faithful explanation. As- sessing beam-search generations, sequences often repeated the same feature. To di- versify explanations, we develop PerFeat decod- ing, our third decoding approach. Here, we prompt the MaNtLE decoder with 'If (feature_name) to generate sequences for each feature and return the most faithful explanation.
Example generations in Table indicate that while we train MaNtLE equally across all complexi- ties, the generations are biased towards using quan- tifiers in the explanations. Consequentl,, surface- level metrics, such as BLEU, are highest for the generated explanations in the quantifier tasks cate- gory (see Figure 3a). The generated explanations seldom contain conjunctions leading to lower faith- fulness and simulatability measures datasets that contain conjunctive explanations in Figure Fur- ther, generated explanations never have negations in the label, i.e., no 'not (labe1}' explanations. Hence, the faithfulness and simulatability are lower than the no-negation datasets (Figure 3; second and fourth bars in each of the four sets).
found that
4.2 Real-world Datasets: CLUES-Real
We investigate the efficacy of MaNtLE explana- tions compared to human-written explanations for aiding in classifying examples from datasets in the CLUES-Real Real benchmark (Menon et al., 2022). Specifically, we evaluate the accuracy of LaSQuE (Ghosh al., 2022), recent explanation ssplaaation-uuided guided classifier for the benchmark," using both explana- tions across three binary classification tasks from the benchmark's zero-shot evaluation set. Explana- tions justifying the labeling rationale for datasets in CLUES-Real were developed by individuals that were shown a few examples. Hence, this benchmark provides an ideal setting for evaluat- ing MaNtLE explanations, which are also generated based patterns observed in few examples."
3.6 Evaluation Metrics
To evaluate the generated language explanations, we use BERTScore (Zhang* al., 2020), ROUGE (Lin, 2004), and BLEU (Papineni et al., 2002). However, these metrics capture surface-level sim- ilarities between the generated explanation and a reference. In our scenario, we want to ensure that the explanations are faithful to the input and that users can simulate model behavior new exam- ples based on the explanation. We measure faithful- ness by using the explanations to make predictions on the inputs, XI.N. and evaluating how often the labels suggested by the explanations match with the original predictions from the classifier in ques- tion, V1.N-3 Similarly, to measure simulatability, we use the explanations to make predictions on unseen examples from the task, XI.M- and assess their alignment with the ground-truth labels, y'..r. We use semantic parser parse explanations gen- erated for unseen synthetic and real-world tasks.
Results in Table 2 show the the performance of LaSQuE with the best MaNtLE explanations is able to get within 7% of the performance with crowd- sourced explanations across tasks. Additionally, when optimizing for faithfulness, diversifying the pool of candidate explanations is helpful indi- cated by the consistent performance improvement of PerFeat decoding over beam search.
4
Explaining Datasets with MaNtLE
4.1 Synthetic Datasets
"F-AN-TS-XL. (Chung et al., 2022), an effective model at learning from instructions, underperformed LaSQuE here. 'In the implementation, we test the performance of the top- 10 generated MaNtLE explanations (secing 10 input exam- ples) with 10 crowd-sources explanations for each task from CLUES-Real.
Following training, we evaluate MaNtLE by gener- ating explanations for 20 synthetic datasets from
"This approach is also referred to as fidelity in the Explain- able Al literature (Jacovi and Goldberg, 2020).

---

Task Complexity
Ground-truth Explanations
MaNtLE Explanations
If pdsu lesser than or equal 1014,
If pdsu not greater than 1020, then it is certainly no
simple
If vpgu equal to unturtica, then blicket
If vpgu equal to antartica, then definitely blicket
If twgk equal to no, then it is seldom fem
If twak equal to no, then it seldom fem
quantifier
bss equal 4, then certainly
If bgbs equal then seldom
If achw equal to no AND hxva equal to africas, then tupa.
If hxva equal to africus, then it is definitely tupa
conjunction
If kjww greater than equal to 18
If kjwx not lesser than 19, then likely I
OR bzjf greater than 1601. then it is definitely
Table 1: Explanations generated by MaNtLE for unseen synthetic tasks for different task complexities.
BLEU
Falthfulness
Simulatabllity
100
80
40
20
Conj
Simple
Corj
Conj-QQurtt
Both, Label Negations
Cisuse, Negetions
Clause, Negations
(a) BLEU
Faithfulness
(c) Simulatability
Figure 3: Results unseen tasks of different complexities (negations, quantifiers, conjunctions). These are numbers averaged over 20 datasets per task category. Error bars indicate standard deviation.
MaNtLE
Human Explanation Accuracy
we briefly describe the baselines ($5.1) 1) and datasets ($5.2)
Task
Decoding
Accuracy
that we use in experimentation.
Strategy
Greedy
50.6
5.1 Baseline Explanation Methods
banknote authenticate cation
Beam
52.0
56.4
LIME. Ribeiro et (2016) approximates model behavior locally by fitting a linear model to the outputs of the black box model on perturbations sampled from distribution around an input. The linear model output is the probability of an example belonging the same class the input used for constructing the explanation. The quality of LIME explanations depends on the number of perturba- tions sampled around an input. Typically, LIME uses 5000 perturbations to fit a good linear model and generate explanations for tabular datasets.
PerFeat
52.4
Greedy
41.6
indian
Beam
40.5
48.6
liver patient
PerFeat
51.4
Greedy
63.0
tic-tac-toe
Beam
56.8
66.1
endgame
PerFeat
57.8
Table 2: Simulatability of MaNtLE explanations mea- sured by LasQuE (Ghosh et al., 2022). Accuracy mea- sures LaSQuE's ability predict ground-truth labels.
We provide qualitative examples of generated explanations for different tasks in Table 5. While MaNtLE can recover some crowd-sourced explana- tions, we also observed hallucinations and lack of numerical understanding leading to errors in planations. We leave these errors for future studies investigate and address.
Anchors. Ribeiro al. (2018) proposed a tech- nique for building rule lists that predict model be- havior with high precision, i... the label mentioned in the explanation matches the model prediction with a high likelihood when the rules apply. To achieve high precision, Anchors often sacrifices explanation coverage, i.e., the number of instances where the rules apply may be very limited. Anchors employs PAC (probably approximately correct) learning approach to generate rule list based on samples from distribution around an input. Un- like LIME which uses input perturbations, Anchors uses training set samples of the black-box model
5 Interpreting Classifiers with MaNtLE
In this section, we compare the quality of MaNtLE explanations to two popular explanation methods in LIME and Anchors using simulated ($5.3) and human user studies ($5.4). Before the experiments,

---

that is close to the input for its construction.
four classifiers logistic regression, decision trees, neural networks, and gradient-boosted trees.
On the omission of WT5. To evaluate the effi- cacy of natural language explanation generation methods, it is common compare with estab- lished works such as WT5 (Narang et al., 2020) and CAGE (Rajani al., 2019). Nonetheless, we have chosen to abstain from comparing our approach with these works in this section for the following specific reasons:
We report the faithfulness and simulatability of explanations generated by different methods. Here, simulatability is measured as the proportion of test set examples for which the model prediction matches the label in the explanation.
5.3 Automated Evaluation
In this section, we conduct simulated user studies to compare the effectiveness of LIME, Anchors, and MaNtLE in generating explanations for classifiers.
Natural Language Explanations (NLE) base- lines, such as CAGE WT5, joinlly predict labels and supporting explanations enhance classification through knowledge induction.
Setup. For each mussifier-datasee pair, we sub- sample 100 random subsets from the validation set. each with 10 examples and the corresponding pre- dictions from the classifier. Next, for each subset, we generate explanations using LIME, Anchors, and the different MaNtLE variants. For LIME and Anchors, we compute the submodular pick (SP) variant of the explanations (Ribeiro et al., 2016).
In contrast, MaNtLE and methods like LIME or Anchors operate independently from label- predicting classifiers, providing post-hoc ex- planations. MaNtLE can applied model- agnostic manner for any classifier, focusing on explaining classifiers rather than enhanc- ing their performance. This distinction ex- plains the absence of LIME or Anchors as baselines in prior NLE works (Rajani et al.. 2019; Narang et al., 2020).
As mentioned in $5.1, LIME and Anchors need to sample examples perturbations around the input generate high-quality explanations. How- ever, MaNtLE generates explanations without any additional information beyond the examples from the subset. To perform a fair evaluation, we report results for a budget-constraineed setting, wherein methods can make a maximum of 15 classifier calls. This corresponds to performing 1 perturbation per example for LIME and using 5 training examples for Anchors. Budget-constrained scenarios form realistic setting in the current landscape where black-box classifiers, like GPT-3, are expensive to query both monetarily and computationally.
CAGE and WT5 explain single-example pre- dictions, while MaNtLE elucidates patterns within sets of examples. fair comparison with MaNtLE necessitates a method capable of aggregating individual predictions.
5.2 Datasets and Models
Following previous work (Ribeiro et al., 2018), we perform experiments for three structured classifica- tion tasks. The first is the adult t dataset from the UCI-ML repository (Dua and Graff, 2017), where the objective is to predict the income level of person (more less than $50, 000). The second dataset used is the recidivism dataset (Schmidt and Witte, 1988), where the task is to predict if a convict is likely to re-commit crimes. The third dataset is the travel-insurance dataset (Tejashvi, 2019) obtained from Kaggle, where the task is to predict if traveler will purchase travel insurance. In all experiments, we use five features from the available set of features for each dataset. To ensure consistency, we follow the data-procees processing scheme and model architectures used in Ribeiro et al. (2018). For each dataset, we train and explain
Results. In Table 3, we see that LIME falls short of Anchors and MaNtLE variants on all classifier- dataset combinations, likely caused by the small number of perturbations available to LIME. Among different MaNtLE decoding strategies, the results suggest that faithfulness improves with better de- coding strategies, with MaNtLE-PF having the best performance overall. Overall, we observe that across the three datasets, MaNtLE-PFF is more faith- ful than LIME and Anchors by 37% and 11 11%, re- spectively, highlighting the utility of our approach in this budget-constrained scenario.
To address scenarios where many examples are accessible cheaply, in $6.2, we explore ways incorporate more examples to improve the quality of MaNtLE explanations.
*For each dataset, we use the top-5 features that maximize information between labels and examples in the training set.

---

dt
Agb
Score
Score
Score 4
Score
Score
Score
explanation
Faith
sim
Faith
sim
sim
Faith
sim
LIME
LIME
31.3
50.2
53.1
19.8
33.9
50.8
52.8
50.0
Anchor
80.2
70.9
57.8
52.9
57.3
50.7
55.3
52.1
Anchor
Anchor
MANeLE
36.3
19.2
55.8
19.2
56.2
29.6
57.1
19.3
MaNtLE-BS
67.8
52.9
67.1
51.9
67.5
52.6
68.1
51.4
MaNILE
MsNILE
MONtLE PT
74.3
57.4
74.4
54.6
75.1
56.5
75.0
55.9
LIME
53,0
30.8
$4.9
54.0
52.6
51.3
53.0
50.4
0
25
50
75
100
25
100
Anchur
69.6
73.5
T1.77
61.3
69.2 2
45.5
51.1
(a) Perturbation
(b) Simulation
MANELE
58.7
61.1
56.4
63.0
57.5
60.0
55.1
52.8
MANILE BS
73.3
63.5
71.7
63.0
72.7
62.5
68.8
54.1
MANtLE-PP PF
72.6
00.2
71.7
60.5
72.1
60.3
71.8
50.4
Figure 4: Percentage of instances where workers under-
LIME
30.0
51.4
50.0
50.0
50.7
standing of model behavior improved (green), declined (red), and did not change (gray) on reviewing different explanations for the adult t dataset.
Anchor
58.5
60.6
76.1
68.8
77.4
68.5
76.1
68.9
MANELE
50.5
54.3
52.4
51.4
MANCLE-BS B-B
69.7
51.3
69.1
52.8
68.8
52.7
69.1
62.6
MANCLE-PF
51.6
69.3
51.3
69.4
Table 3: Faithfulness and simulatability when executing different explanations for three datasets. Results are averaged over 100 runs (or subsets). Bold numbers indicate the best explamation for a particular classifier- metric pair. SP = submodular pick. BS = beam search, PerFeet decoding.
LIME
Anchor
MaNtLE
4.08
4.08
3.92 4.00
3.16
3.28
3.44
3
2.60
2.16
Understandability Informativeness
Preference
5.4 Human Evaluation
Figure 5: Preference for different explanation tech- niques by workers with at least undergraduate level education, as indicated by Likert ratings on a 1-5 scale.
user studies, we measure the ability of users to interpret explanations and predict the behavior of models trained on the adult dataset. We use the full budget LIME and Anchors explanations and the MaNtLE-PerFeat explanations from the previ- ous section. In a pilot study, we found workers had difficulty interpreting the meaning of different quantifiers. Hence, we post-process MaNtLE expla- nations by converting quantifiers to confidence val- ues based on previous work (Menon et al., 2022).?
Which explanations would general practitioners prefer? For this study, we recruited 25 partic- ipants on Prolific who are currently pursuing at least an undergraduate degree to rate explanations on a 1-5 Likert scale for understandability. infor- mativeness, and overall preference. 8 We chose this demographic to reflect the expected diversity of industry experts' educational backgrounds likely to use explanations to better understand their systems.
Which explanations help users predict model be- havior? In Figure 4, we present the results of our study. We report the percentage of instances where the user predictions of model behavior improve, worsen, or remain the same on perturbed examples (perturbation) and test examples (simulation) af- ter reviewing explanations, following the setup in Hase and Bansal (2020). 23 workers took part in this study conducted on Amazon Mechanical Turk and were compensated $12/hr.
Results in Fig. indicate that workers struggled to comprehend attribution scores from LIME com- pared to MaNtLE. paired-sample t-test for over- all preference revealed significance (2.16 16 vs. 3.4;; p-value < 0.001). In contrast, workers found An- chors informative, but their low coverage hindered their overall preference compared to MaNtLE.
In our results (Figure 4), we found that users can predict model behavior more accurately in 46% of cases after reviewing MaNtLE explanations, com- pared to 15% for LIME and 19% for Anchors. Ad- ditionally, users were less likely to make more in- correct predictions of model behavior after view- ing MaNtLE explanations, with only 19% of cases, compared to 38% for LIME and 31% % for Anchors. Hence, our explanations are more reliable in help- ing users predict model behavior.
Notably, workers prefer MaNtLE explanations for their clarity and simplicity, citing phrases such as 'clearly defines information' and 'uses more layman terms'. Nevertheless, some expressed concerns that the explanations were 'not descriptive enough', semi-specific when compared to LIME'. This suggests that explanations clarifying how each fea- ture affects the classifier's decision-making process could improve user understanding and satisfaction.
FFig. 15 shows examples of post-processed explanations.
FFig. 14 shows how we define the scale for each property.

---

6 Analysis
Faithfulness
Simulatability
100
100
In this section, we analyze MaNtLE based two key aspects: (a) factors affecting the strong gen- eralization of MaNtLE, and (b) stability of MaNtLE when the number of input examples varied.
75
75
50
50
25
25
30
100
300
1000
10
30
100
300
1000
# Input Examples
Input Examples
6.1 How does scale affect the performance of MaNtLE?
Faithfulness
Simulatability
100
100
75
50
50
Model Size
Training Tasks
100
100
80
15-1ar ge
80
0
30
100
300
1000
10
30
100
300 1000
# Input Examples
# Input Examples
15-base
MaNtLE
LIME
60
t3-small
60
MaNtLE PerFeat
Anchor
100
300
1000
10 30 100
Figure 7: Increasing number of input examples im- proves simulatability and faithfulness of MaNtLE-PP explanations. Here, compute metries over 100 rums for two classifiers (logistic regression top row, deci- sion free - bottom row) trained on the adult dataset. LIME and Anchors shown here full-budget submod- ular pick variants. Error bars show standard deviation.
Millions of Parameters
Thousands of Tasks
Faithfulness
Simulatability
Figure 6: Paithfulness and simulatability performance of MaNtLE on held-out tasks with increase in the scale model size (left) and dataset size (right). Error bars indicate standard deviation.
6.2
Can MaNtLE take advantage of more examples?
First, we evaluate the influence of model size and the number of training tasks on the quality of expla- nations generated by MaNtLE. For this, we create a synthetic benchmark consisting of 50K datasets, each with ground-truth explanations using conjunc- tions, which are challenging for MaNtLE to learn. For evaluation, 20 datasets from the benchmark are held-out, and we measure the faithfulness and sim- ulatability of explanations generated by fine-tuned models on these datasets. We fine-tune T5 models using nearly all 50K datasets for model scale exper- iments, and vary tasks between 1K to 50K for task scale experiments (using T5-large model here).
The maximum number of tokens allowed in the T5 encoder limits the input capacity of MaNtLE. This restricts MaNtLE when a large number examples are available for the classifier being explained.
To overcome this challenge, we propose tech- nique that enables MaNtLE-Beam and MaNtLE-PF to handle a greater number of input examples. Our method involves dividing set of N available ex- amples into eight subsets, each containing 10 ex- amples. We utilize these subsets to generate expla- nations and select the explanation with the highest "simulatability" score among the N examples as the best explanation. We evaluate the faithfulness and simulatability of our explanations using this ap- proach to explain logistic regression and decision tree classifiers for the adult dataset.
To study the effect of model scale. we fine-tune different variants of T5, ranging from T5-Small to T5-Large. As seen in Figure 6 (left), increasing the scale of models from 15-Small to T5-Large con- sistently improves both faithfulness and simulata- bility. Moreover, increasing the number of training tasks also enhances both metrics, as seen in Figure 6 (right). Notably, fine-tuning a T5-Large model on smaller number of tasks (1K) leads to poorer performance than T5-Small model fine-tuned on larger number of tasks (50K), even when trained with the same hyperparameters for the same dura- tion. Taken together, expanding multi-task training and increasing model sizes further could improve MaNtLE's performance.
In Figure 7, we see that our procedure enhances the faithfulness and simulatability of MaNtLE-PF, Additionally, the performance of MaNtLE-PF is comparable to the full-budget submodular pick vari- ant of LIME, indicating that our approach enables MaNtLE to achieve comparable explanation qual- ity when provided access large number of ex- amples. However, the procedure does not yield improvements in the metrics for MINtLE-Bcaam im- plying that the diversity of generated explanations

---

is crucial. Expanding the number of subsets could further improve the performance of MaNtLE-PF. We leave this for future work to explore.
ing such reasoning without training MaNtLE from scratch is an intriguing avenue for future research.
Ethies Statement
7 Conclusion
We perform experiments using mixture of syn- thetically generated datasets as well as publicly available datasets from the UCI and Kaggle repos- itories. In human evaluations, workers were pro- vided fair compensation over the federal minimum wage. Our work is a research prototype for an ex- planation method. If the inputs are aligned with the techniques mentioned in this work, we do not foresce risks for models (aside from hallucinations which have been discussed in the limitations sec- tion). However, MaNtLE is fine-tuned from the T5 checkpoint, a pre-trained transformer model. Hence, it is possible that our model may exhibit unwanted biases that are inherent pre-trained language models. This aspect is beyond the scope of our work.
this work, we introduce MaNtLE, a model- agnostic natural language explainer, that generates faithful explanations for structured classifiers. We use recent insights in massive multi-task training to train a model that can generate faithful classifier rationale. As MaNtLE can explain classifiers simply by inspecting predictions made by classifiers, it can be used in model-agnostic fashion similar to popular explanation methods like LIME and An- chors. In simulation studies and human evaluations, we show that MaNtLE explanations are more faith- ful than LIME and comparable in faithfulness to Anchors on multiple datasets. Our work suggests the potential for natural language interfaces to hance end-user trust and promote the safe usage of ML models in critical tasks by providing expla- nations of the classifier decision-making process. Future work can look extend our work to develop "patches" (Murty et al., 2022) for improving classi- fiers, refining decoding techniques for more faithful and simulatable explanations, and integrating more complex reasoning patterns in generated explana- tions. Further, developing models that can gener- ate language explanations for classifiers trained on other modalities, such as raw text and vision, is an interesting avenue for future work.
A Acknowledgments
The authors would like to thank the anonymous reviewers for their suggestions and feedback on the work. This work was supported in part by NSF grant DRL.21 this article are those of the authors and not of the funding agency.
The views contained in
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Limitations
Our method exclusively designed to explain clas- sifiers operating on structured datasets. Utilizing MaNtLE for other input types, such as raw text and images, is out of the scope of this work.
Further, the number of examples that can be packed into the encoder of the MaNtLE is limited to 1024 tokens (limit of T5 encoder). While we exam- ine additional strategies to circumvent this issue, future work could look into additional methods for packing more examples into the input to improve explanation quality.
Sahil Chopra, Michael Henry Tessler, and Noah D. Goodman. 2019. The first crank of the cultural ratchet: Learning and transmitting concepts through language. In CogSei.
Additionally, the logic that can be represented by the outputs of MaNtLE is limited to the kind seen during training. Here, this implies that we may never observe explanations with nested conjunc- tions (three or more feature constraints combined). Future work can identify solutions to incorporate more complex reasoning in explanations. Integrat-
Hyung Won Chung, Le Hou. Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan

---

Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling cniruction-finetnne language mod- els.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Emmanuel Ghenga Dada, Joseph Stephen Bassi, Haruna Chiroma, Shafi'i Muhammad Abdulhamid, Ade- bayo Olusola Adetunmbi, and Opeyemi Emmanuel Ajibuwa. 2019. Machine learning for email spam filtering: review, approaches and open research prob- lems. Heliyon, 5(6))e01803 802.
Rakesh R. Menon, Sayan Ghosh, and Shashank Sri- vastava. 2022. CLUES: benchmark for learning classifiers using natural language explanations. In Proceedings of the 60th Anmual Meeting of the As- sociation for Computational Lingnistics (Volume I: Long Papers), pages 6523-6546, Dublin, Ireland. As- sociation for Computational Linguistics.
Dheeru Dua and Casey Graff. 2017. UCI machine learn- ing repository.
Sayan Ghosh, Rakesh R Menon, and Shashank Srivas- tava. 2022. Lasque: Improved zero-shot classifica- tion from explanations through quantifier modeling and curriculum learning. ArXiv, abs/22 12.09104.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross- task generaliza- tion via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3470-3487, Dublin, Ireland. Association for Computational Linguisties.
Peter Hase and Mohit Bansal. 2020. Evaluating explain- able AI: Which algorithmic explanations help users predict model behavior? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5540-5552, Online. Association for Computational Linguistics.
Shikhar Murty, Christopher Manning, Scott Lundberg, and Marco Tulio Ribeiro. 2022. Fixing model bugs with natural language patches. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 11600-11613, Abu Dhabi, United Arah Emirates. Association for Com- putational Linguistics.
Or Honovich, Uri Shaham, Samuel R Bowman, and Omer Levy. 2022. Instruction induction: From few examples to natural language task descriptions. arXiv preprint 22X.::2205.1077
Alon Jacovi and Yoav Goldberg. 2020. Towards faith- fully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198 4205, On- line. Association for Computational I. Linguistics.
Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020. Wt5!! training text-to-text models to explain their predictions. arXiv preprint 2xxx::2004.14544
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: A method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting Association for Computa- tional Linguisties, ACL '02, page 311-318, USA. Association for Computational Linguistics.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory sayres. 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Pro- ceedings of Machine Learning Research, pages 2668- 2677. PMLR.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin. Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d'Alché -ucc E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc.
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. 2019. Faithful and customizable ex- planations of black box models. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 131-138.
Himabindu Lakkaraju, Dylan Slack, Yuxin Chen, Chen- hao Tan, and Sameer Singh. 2022. Rethinking Ex- plainability a Dialogue: A Practitioner's Perspec- tive. arXiv preprint arXiv:2202.0185.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res,, (1(
Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- fion Brunches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.

---

Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain your- self! leveraging language models for commonsense reasoning. In Proceedings of the 57th Anmual Meet- ing of the Association for Compulational Linguistics, pages 4932-4944 Florence, Italy. Association for Computational Linguistics.
Mukund Sundararajan. Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Pro- ceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Ma- chine Learning Research, pages 3319-3328. PMLR.
Tejashvi. 2019. Travel insurance prediction data.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I 'Trust You?" Ex- plaining the Predictions of Any Classifier. In Pro- ceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. 2022. Finetuned language mod- els are zero-shot learners. In International Confer- ence on Learning Representations.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of fonte-of-thh-ars the-eerrt natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45 45, Online, Association for Computational Linguistics.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Anchors: High- bigh-precision model- agnostic explanations. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey. M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Frics, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In International Conference on Learning Representations.
Tianyi Zhang*, Varsha Kishore*, Felix Wu", Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
A Experiment Details
Here we provide implementation details such as hyperparameters, hardware and software used for developing MaNtLE and running our experiments.
Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.0511
A.1 Libraries
We use the HuggingFace library (Wolf et al., 2020) for all the transformer- based models. For T5 mod- els, we experiment with t5-small, t5-base, and t5-large' across the main experiments and analy- ses. Our main experiment results are based on the t5-large model. Pre-trained checkpoints for these models are publicly available on the HuggingFace library. All models are coded in PyTorch 1.13.1 (Paszke et al., 2019).
Peter Schmidt and Ann D Witte. 1988. Predicting re- cidivism in north carolina, 1978 and 1980. Inter- university Consortium for Political and Social Re- search.
Si Shi, Rita Tse, Wuman Luo, Stefano D' Addona, and Giovanni Pau. 2022. Machine learning-driven credit risk: systemic review. Neural Computing and Ap- plications, 34(17) 14327-14339.
Chandan Singh, John X Morris, Jyoti Aneja, Alexan- der M Rush, and Jianfeng Gao. 2022. Explaining pat- terns in data with language models via interpretable autoprompting. Xixiv preprint ar arXiv:2210.018-8
A.2 Pre-training Hyperparameters
We used the t5-large model and trained it using the AdamW (Loshchilov and Hutter, 2019) opti- mizer with learning rate of le 5 and weight decay of le- 2 for 2, 000, 000 steps using the stan- dard language modeling objective. Each gradient step is computed over a batch of 4 samples with no gradient accumulation steps. The maximum
Shashank Srivastava, Igor Labutov, and Tom Mitchell. 2018. Zero-shor learning of classifiers from natural language quantification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 306-311 16, Melbourne, Australia. Association for Computational Linguistics.
"https://huggingfacee cotps/a-acccotttttereeeeeeeeee

---

length of the input is clipped to 1024 tokens, which roughly corresponds 10-12 input-prediction pairs being encoded in each sample, while we limit the decoder to generating 64 tokens since that was suffi- cient to generate the longest explanations from our training set. We chose the best checkpoint based on the performance over generation metrics well faithfulness and simulatability on 20 held-out validation tasks.
hwyw
nwfh
zosp
ghoq
label
5
yes
6668
5
51
5
1201
3
32
5ot
Explanation: If hwyw greater than
1528, then 5
The model was fine-tuned using full precision on single NVIDIA A A100-PCIE-40GB GPU, 400GB RAM, and 40 CPU cores for ~ 25 days.
Figure 8: Example synthetic dataset used in our multi- task mixture train MaNtLE.
A.3 Pre-training Datasets
acters. Early experiments indicated that reusing feature names across tasks with different classifica- tion logic can hurt model's ability learn gen- crate explanations, hence prompting the random strings for feature names. We show an example of a sampled task for multi-task pre-training in Fig. 8.
Pre-training was performed using programmati- cally generated datasets whose explanations fol- lowed the if-then structure following Menon al. (2022). We also utilize the different complexities described this prior work, which enabled our model to perform diverse types of reasoning. Over- all, there were 24 different complexities that varied by: (a) the presence of quantifiers in explanations, (b) the presence of conjunctions in explanations, and (c) the presence of negations in explanations. The quantifiers adopt in this work, along with their values, follow from prior work in Srivastava al. (2018). For conjunctions, can have tasks with explanations that have nested conjunctions (AND-OR / OR-AND explanations) or simple con- junctions (AND / OR). For negations, there are more subdivisions based on the positioning of the negation. For example, if we have an explanation of the form, 'If a equal to I then yes', then clause negation corresponds to explanation of the form 'If a not equal 1, then yes', and a label negation corresponds 'Iaa equal 1, then not yes'. Hence, the presence of negations can vary by no nega- tions, only clause negations, only label negations, and negations in clause+label. Overall, we have 2 (quantifier) x 3 (conjunctions) x 4 (negations) 21 different complexities. We use ~ 8, 000 tasks per complexity for training, which leads to the mas- sive pre-training dataset of ~ 200, 000 tasks. Table 4 concisely lists the template used for different task complexities.
A.4 Real-world Task Explanation Generation
Owing the pre-training procedure of MaNtLE, which involved feature names single-word low- ercase strings, it essential that input to MaNtLE for real-world tasks formatted in a similar fashion. Therefore, we transform all feature names from real-world tasks into format that can be processed by MaNtLE, accomplished by climinating spaces and converting all characters in the feature names to lowercase. To ensure that the generated expla- nations understandable to humans, perform prot-processing step convert these lowercase feature names back into their original format. This process is crucial for achieving accurate explana- tions using MaNtLE.
A.S Model and Dataset Scale Analysis
For the model-scale analysis, we fine-tune from t5-large checkpoint using the conjunction datasets. These models were trained using similar hyperparameters as that from pre-training. How- ever, since the datasets only come from a single complexity, namely conjunctions, these models were trained for 200, 000 steps.
B Extended Related Work
Each synthetic task consists of five features and sampled based on the synthetic templates in Menon et al. (2022). The range of values in our synthetic datasets thus follow the available range in CLUES. On the sampled datasets, we choose to con- vert feature names to random strings four char-
Instruction Generation by Large Language Models. Some recent works (Honovich al.. 2022: Singh et al., 2022) explore techniques prompt LLMs to generate instructions based on a few examples from synthetic and real-world

---

Task Complexity
Template
Example Explanations
stmple
[cond), then (labal)
pdsa lesser than or equal to 1014, then no nchw equal to
conjunction
[cond1) ANDIOR (cand1), then (label)
equal to africas, then rupa
clause negation Lion
(feet_naaee not equal to (feat_value), then (label)
bygs not equal 4, then
label negalion
If [cond(s)), then not (label)
If acho equal to yes, then not lupa.
clauserlabel negation
[feee not equal to value), then not (label) [cond], then (quantifier) (label}
sooj not equal to then not 5
quantifier
tugk equal then is seldom fem
Table 4: Templates and example explanations for different task complexities in the synthetic training benchmark. For brevity, we omit mentions of combinations of complexities, e.g., conjunction quantifier.
Explanations CLUES-Real
NaNtLE-PF Explanations
Below 3.80 skewness leads to the original.
If skewness lesser than or equal to 3.049, then it occasionally Fake.
banknote-authenticom cation
kurtnnis lesser than or equal to 995, then often Rake kurtosis lesser 9600, then frequently Fake
Kurtosis high value is original.
The SGPT Low percentage so the liver putient was
If SCPT lesser than equal 39, then palient is generally No
indian-liven Sivaaa--tv----aiiii ent
Age group above 40 ensures liver patieni
If age lesser than or equal to 39, then patsent generally No
Without b cotegories middlee midkle square under the Positive group.
Mic-tac-toerendgae
middll--middl--qqaae equal
Game sometimes positive
Table 5: Explanations generated by MaNtLE for CLUES-Real datasets.
datasets. While our training procedure learns to generate explanations for datasets akin to these prior works, our primary objective is to explain classifiers to understand their classification ratio- nale rather than datasets.
Method
Simulatability Score
WT5
55.50
MaNtLE
93.72
Table 6: Comparing the simulatability of explanations generated by WT5 and MaNtLE on a set of 20 unseen synthetic classification tasks.
C OOD Synthetic Task Results
In $4.1, we evaluated MaNtLE explanations on a set of 20 unseen tasks from the synthetic benchmark. The results of this evaluation, presented in Figure 9. include MaNtLE's performance on the full range of generation metrics, as well as measures of faith- fulness and simulatability. As noted previously in $4.1, the performance metrics for generation, such BERT-Score, ROUGE-* and BLEU, have re- vealed that MaNtLE explanations exhibit closer alignment to the ground-truth explanations when the latter includes quantifiers. This phenomenon can be attributed to a bias that MaNtLE acquires towards generating quantifiers towards the end of the training process, as observed in Table 1. How- ever, our analysis of faithfulness and simulatabil- ity scores revealed that MaNtLE explanations were most effective on the simplest datasets that lacked complexities such as negations, quantifiers, or con- junctions, in line with our expectations.
To circumvent the lengthy pre-training process, which took approximately 25 days as detailed in Appendix $A.2. we opted to train a WT5--tyle model on the conjunctions based dataset analyzed in Section $6.1. We fine-tuned both the WT5-style model and MaNtLE by training them on 48,000 tasks for 200K batches. After the fine-tuning stage. we evaluated the simulatabilityy of explanations gen- erated by each method on 20 held-out tasks. For MaNtLE, we directly evaluate the generated expla- nations on the subset of tasks. On the other hand, for WT5, we evaluated the most faithful explana- tion from the set of explanations generated for each example in a batch.
It is important to acknowledge that this repre- sents just one approach to adapting the NLE base- line for explaining sets of examples. Nevertheless, we believe that the exploration of deriving over- arching patterns from multiple single-example ex- planations poses intriguing research questions that warrant separate investigations.
C.1 Comparison with WT5
In this section, we compare the effectiveness of WT5-style model (Narang al., 2020) over our proposed objective. In WT5, the explainer makes predictions for each example, and then we pick the best explanation that's s most faithful to all the input examples as the one that explains the whole set.
From Table 6, we observe that MaNtLE achieves higher simulatability score on the held-out tasks in comparison to the WT5 model, underscoring the generalization utility of our procedure to explain subsets of examples.

---

BERT-Score
ROUGE-T
ROUGE-L
100
100
100
40
40
40
50m pM
Conl
Conj
Samole
Cory
Dong-+Quant
Smmpp
Cuant.
Cory.
Coni Quant.
Negations
Negaiiors
(a) BERT-Score
(b) ROUGE-I
(c) ROUGE-L
BLEU
Falthruiness
Sinulatability
100
100
90
60
40
40
40
Stmple
Ouant
Conj
Conj.-Cuunt.
Simple
Quanc.
Corj.
Oong.
Stmple
Ouant.
Corg.
Quann.
INo, Botn, Clouse, Negations
(No, Both, Cluss, Nogations
(No, 8oth, Clause, Nogations
(d) BLEU
(e) Faithfulness
Simulatability
Figure 9: Results on OOD tasks different complexities (negations, quantifiers, conjunctions). These are numbers averaged over 20 datasets per task category. BERT- BERT-Score, ROUGE, and BLEU scores are the highest for Quantifier datasets since model is more adept at generating content that contains single attribute explanations with quantifiers and negations.
D Extended Analysis
classifiers may operate over more than five features, which is why this evaluation is essential.
D.1 Can MaNtLE take advantage of more examples?
The results in Figure 11 suggest that the faithful- ness of generated explanations is invariant to the number of input features used by classifier that we seek to explain. However, while MaNtLE-PF generates more faithful explanations than MaNtLE, this advantage does not translate to improved simu- latability the number of features increases.
Extending on the results from $6.2, which pre- sented the performance of logistic regression and decision tree models trained on the adult t dataset, we further investigate the performance of neural network and xgboost classifiers in this section. In addition, coverage of the explanations the simulation set (i.e., the set). Consistent with our findings in $6.2, increasing the number of examples used by MaNtLE consistently improves the precision of the explanations, leading to improved overall perfor- mance (Figure 10).
evaluate the precision and
D.3 Do predicted quantifiers reflect the precision of explanations?
We conducted an empirical study to investigate the relationship between quantifiers used in explana- tions generated by MaNtLE and the precision of the explanations upon execution. Specificallyy, we mapped the quantifiers used in the explanations. such as 'certainly', to their corresponding numeric values and analyzed the frequency of the predicted label values when the conditions mentioned in the explanations were met. For instance, if we have generated explanation for the adult dataset of the form, 'if education equal dropout, then income certainly <=50K', where 'certainly' maps to a value 95%, we check if the label is '<=50K' 95% of the time when the education level was dropout on some held-out set of examples. As MaNtLE was
D.2 Can MaNtLE handle tasks with more features?
In this experiment, we evaluate the ability of MaNtLE, and the PerFeat decoding variant of MaNtLE, to generate explanations for logistic re- gression and decision tree classifiers we increase the number of features from 5 to 11 for the adult dataset. As in the prior sections, we measure the faithfulness and simulatability metries.
In $5, we experimented with exactly five features for datasets. However, in world situations,

---

Faithfulness
Simulatability
Precision
Coverage
100
75
50
25
25
25
30
100
300
1000
10
100
300
1000
30
100
300
1000
30
100
300
1000
# Input Examples
# Input Examples
Input Examples
# Input Examples
MaNtLE Beam
MaNtLE PerFeat
LIME
Anchor
(a) Logistic Regression
Faithfulness
Simulatability
Precision
Coverage
100
100
75
75
50
50
50
25
25
25
25
0
0
30
100
10
300
1000
100
Input Examples
# Input Examples
Input Examples
# Input Exemples
MaNtLE Beam
MaNtLE PerFeat
LIME
Anchor
(b) Decision Tree
Faithfulness
Simulatability
Precision
Coverage
100
100
100
100
75
75
75
50
50
50
25
25
25
25
0
0
30
100
10
300 1000
100 # Input Examples
Input Examples
# Input Examples
Input Examples
MaNtLE Beam
MaNtLE PerFeat
LIME
Anchor
(c) Neural Network
Faithfulness
Simulatability
Precision
Coverage
100
100
100
100
75
75
75
50
50
50
50
25
25
25
25
0
0
0
30
100
10
100
1000
30
100
300
1000
100
Input Examples
Input Examples
Input Examples
Input Examples
MaNtLE Beam
MaNtLE PerFeat
LIME
Anchor
(d) XGBoost Classifier
Figure 10: Increasing number input examples improves simulatability and faithfulness of MaNtLE-PF explanations. Here, we show the results by increasing the number of input examples between 10 and 1000 for different classifiers trained on the Adult dataset. While the simultability of LIME, Anchor, and MaNtLE remain constant with increase in the number of input examples, explanations from MaNtLE-PF improve simulatability and faithfulness.

---

Faithfulness
Simulatability
Precision
Coverage
100
100
100
100
75
50
50
50
50
0
10
10
6
10
Input Features
# Input Features
# Input Features
# Input Features
MaNtLE
MaNtLE PerFeat
LIME
Anchor
(a) Logistic Regression
Faithfulness
Simulatability
Precision
Coverage
100
100
100
100
75
75
75
75
25
25
25
25
10
12
10
12
10
6
10
Input Features
# Input Features
# Input Features
Input Features
MaNtLE
MaNtLE PerFeat
LIME
Anchor
(b) Decision Tree
Faithfulness
Simulatability
Precision
Coverage
100
100
100
100
50
50
50
50
25
25
0
10
12
6
10
12
6
10
10
# Input Features
#l Input Features
# Input Features
# Input features
MaNtLE
MaNtLE PerFeat
LIME
Anchor
(c) Neural Network
Faithfulness
Simulatability
Precision
Coverage
100
75
75
75
75
0.
10
10
10
Input Features
Input
Input Features
Input
MaNtLE
MaNtLE PerFeat
LIME
Anchor
XGBoost
Figure 11: Faithfulness of MaNtLE-PF explanations largely independent of the number of input features across models. Here, we compute metrics over 100 runs for the Adult dataset. The simulatability of explanations decreases with an increase in the number of features, as would be naturally expected.

---

1.0
planations (where confidence is measured by the quantifier used in the generated explanation). Fi- nally, in experiments, we ask workers to simulate the behavior of the different classifiers used in our simulation experiments.
Quantifier Confidence
0.8
0.6
CO0
0.4
p 0.78
The templates used in our experiments can be found in Figure 13 (for perturbation and simula- tion experiments) and Figure 14 (for explanation preference experiments). Examples of explanations presented workers can be found in Figure 15.
0.2 0
0.2
0.4 0.6
0.8
1.0
Observed Confidence
Figure 12: Quantifier predictions in MaNtLE explana- tions strongly correlate with the conditional frequency of outcomes in the underlying input.
Worker Accuracy. Here, we present the classifi- cation accuracy (averaged over workers) during the pre-explanation and post-explanation phase of the perturbation and simulation experiments described in $5.4. It is worth noting that individual work- ers may have varying degrees of pre-explanation accuracy, thereby making a direct comparison of raw accuracies between explanation methods mis- leading. Our primary objective is to investigate whether explanations improve the workers' classifi- cation ability. Therefore, we depict the percentage of instances where the explanations led to improved classification performance in Figure 4.
trained on synthetic dataset, whose quantifier datasets were constructed by mapping numeric val- ues words (using the map from (Srivastava et al., 2018)), we can convert the generated quantifiers back numeric values and perform the analysis.
In Figure 12, we plot the correlation between the (predicted) quantifiers and the explanation precision for explanations generated from classi- fiers trained on the adult and travel-insurance datasets. We observe strong correlation between the predicted quantifiers and precision of expla- nations (Spearman p 0.78), indicating that the MaNtLE able to generate accurate and reliable explanations of black-box classificr's s rcasoning process.
Exp. Method
Pre-Exp. Accuracy
Post-Exp.
Experiment
Accuracy
LIME
71.5
67.7 (L)
Perturbation
Anchors
63.1
55.4 ())
MaNtLE
53.8
60.8 (1)
LIME
60.0
56.9 ())
E Human Evaluation
Simulation
Anchors
67.7
63.8 (L)
In this section, we elaborate on our human evalua- tion setup and provide templates as well workers' accuracies in solving the adult task using different explanations.
MaNtLE 66.1
67.7 (1)
Table 7: Average classification accuracies of workers on perturbation and simulation experiments for the adul dataset in the pre-explanation and post-explanation phases. Overall, workers took part in this study. explanattnn
Evaluation details. Firstly, during pilot study, we identified that workers found it difficult to in- terpret the meanings of different quantifiers. As a result, following the confidence values in Srivas- tava et (2018), we reverse map the quantifiers in MaNtLE explanations to confidence values and present them to the turkers. For example, given 'if Education not equal to Dropout, then Income is cer- tainly >5K'', we convert it to '95% of the time, the Income is > >5K if Education not equal to Dropout'. Secondly, workers preferred high-confidence expla- nations and often rated other explanations poorly. To make fair evaluation of our explanations, we re- strict human evaluations to settings where MaNtLE explanations are at least 85% confident of their ex-
"hhhen we mention the classification performance of hu- man workers, we refer to the number of times they can predict the same lahel as the classitier, whose explanations they see during the learning phase.

---

Instructions
Task Desclabio nhe sidee thow too haut noure Acoue ecicced on teln Cataa caliac lours wnsk Hecmie
Worlly utliliiity of difforent iss methodo cutegarcing dita.
the codaponize
scamples for categetcation
es.
tourhh peak, you categoron newr using information suamples
explanation.
Palss
HOrE: rournedd to click en 'tave Arsoorr for each table to reghtter your cheice and commpheas the connectly
timw the bwitacee Coppiil Suerec
You will not be able revisit page once cllii on NNeatt
Stan Descriotion: thes bée we show some rows from the Audut: necen datuset untructe foom the 1994
Cansu ditabaree Gr van so stribuuoe gloe hours per woek, education status, hocone-groug etc the to
ofher pessonmalkc incom over BOK
Add
1400
Marke
Cookul
Moar's
Vadelen
Codded
Wetter
Og Siorort
Sahival
Mcreed
Madio
34
Diosett
Staro
6hd
Normal
NOTE: Please Sarve Answers' before dilccing "Sbmiit..
NOTE Wasse click Seve clicting
1/4
Niei
Figure 13: Template for performing the perturbation and simulation experiments using examples from the adul t dataset.
Explanation Evaluation
to cagiaa wofte rony to unddestaadd
vepan exmber srom the Comrotet rodiirate
stfcc sody the
exgler adion cass
Necenal modenajun maarg
Very Neparforaaiv compire Mecy opptestion
95% of the the Income equal Dropout
cxplaaaion
Cbick Fmchhe rod
witsi coloeatioe
Figure 14: Template for performing the subjective evaluation of different explanations 1-5 Likert scale for understandability, informativeness, and overall preference.

---

hore
IF Marital Status Married AND Hours per week > 48.00 AND Education = High School grad AND Workclass Private THEN PREDICT income 1S >50K
He Snatu maried
of the time, the Income >50K if Marital Status equal to Married
Nopelie Oan
(a) Anchors
(b) LIME
(c) MaNtLE
Figure 15: Explanation format used in human evaluation experiments to investigate the faithfulness and simulatability of explanations generated by different methods.


Reviews:
[{"review_id": "SnK3npB3HF", "reviewer": "Reviewer_P3Ea", "summary": "", "questions": "1. no other natural language explanation method is compared with?\n2. the datasets compated (UCI income, recidivism, travel-insurance) are all tabular datasets? I dont see a strong case that in those datasets, natural language explanations would be better than attributon based method. If so, what is the main reason for that? also, for tabular dataset, if I have an attribution based method, it would be very easy to convert it into natural language based explanations.\n", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments on our draft. We are particularly encouraged that the reviewer finds our experimentation and evaluation “comprehensive” and “interesting”. Below, we address some of the concerns raised by the reviewer:\n\n- **Natural Language Explanation (NLE) Baselines**: This is a question raised by other reviewers as well (reviewers:  P3Ea, tLzw, CbXw). We appreciate an opportunity to address this and clarify two fundamental points (which make these improper comparisons, and which guided our decision not to include these baselines):  \n   - **MaNtLE explains other classifiers, rather than performing classification unlike other NLE baselines**: With NLE baselines, such as CAGE or WT5, the classifier jointly predicts ***both*** a label and a supporting explanation. The primary contribution of these methods is the induction of knowledge in explanations to improve classification. \\\n\\\n   In contrast, MaNtLE, as well as methods like LIME or Anchors, operates separately from the label-predicting classifier. It functions as a post-hoc explanation technique, deliberately dissociating the processes of classification and explanation. Hence, MaNtLE can be applied in a model-agnostic fashion for any classifier. The primary function of these methods is to explain classifiers rather than improve them. This distinction also accounts for the absence of LIME or Anchors as baselines in prior NLE works. (Rajani et al. 2019, Narang et al. 2020) \n\n   - **Explaining Subset of Examples vs. Single-example Explanations**: The focus of CAGE and WT5 is to explain the predictions made by a classifier for a *single-example*. In contrast, MaNtLE explains *subsets* of examples and extracts more general patterns. To compare fairly with MaNtLE, we require a method that can aggregate multiple single-example predictions. Deducing general patterns from multiple single-example explanations raises interesting research questions that can be a separate avenue of exploration. However, addressing this aspect goes well beyond the scope of our current study, and would overload the introduction of new methods.  \n\n\n- **When and why would MaNtLE perform better and be preferred over attribution-based methods on tabular datasets?** As mentioned in 416-429, LIME and Anchors require access to predictions from the model for multiple perturbations of an example. However, if the classifier we applied for a particular tabular dataset were GPT-3, then querying this classifier would have a substantial monetary and environmental cost. In settings that necessitate the control of model calls, results in Table 3 conclusively indicate that MaNtLE would be the preferred method for explaining classifiers. \\\n\\\nFurther, suppose we have tasks with some inherent noise in concept occurrence. In that case, MaNtLE can accurately capture such ambiguity by using quantifiers unavailable in attribution-based methods. \\\n\\\nFinally, we would like to point out that natural language explanations for tabular datasets are essential for the accessibility of interpretability tools to people from diverse domains (e.g., medical doctors) who may find it challenging to interpret explanations provided in the form of attribution scores.  Similar requirements for natural language explanations have been noted in Lakkaraju et al. (2022).\n\n\n- **Fidelity over faithfulness**: Thanks for the suggestion! We will take this into account and update the next version appropriately.\n\n**References**:\n- Rajani et al. 2019: Explain Yourself! Leveraging Language Models for Commonsense Reasoning, ACL 2019\n- Narang et al. 2020: WT5?! Training Text-to-Text Models to Explain their Predictions, arxiv 2020\n- Lakkaraju et al. 2022 : Rethinking Explainability as a Dialogue: A Practitioner’s Perspective, arxiv 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. the way constructing a synthetic dataset is interesting and smart. Though the setting is still artificial and simple. \n2. comprehensive experiments. \n3. the evaluation on CLUES-REAL and LaSQuE is interesting. ", "reasons_to_reject": "1. the experiments seem simple. \n2. it would be better to include more discuss on in which cases natural language explanations would be more helpful. I dont think tabular dataset would make a strong case of this. ", "questions_to_authors": "1. no other natural language explanation method is compared with?\n2. the datasets compated (UCI income, recidivism, travel-insurance) are all tabular datasets? I dont see a strong case that in those datasets, natural language explanations would be better than attributon based method. If so, what is the main reason for that? also, for tabular dataset, if I have an attribution based method, it would be very easy to convert it into natural language based explanations.\n", "paper_topic_and_main_contributions": "MaNtLE is a model-agnostic natural language explainer. It is fine-tuned T5-Large model on thousands of synthetic classification tasks, each paired with natural language explanations. ", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I would prefer fidelity over faithfulness since the later is overloaded and controversial.\n"}, "extra_text": {}}, {"review_id": "SZO9LSIIOJ", "reviewer": "Reviewer_vjwc", "summary": "", "questions": "* Could you provide random examples of explanations generated by each method on Adult, Travel Ins., and Recidivism? \n* On more realistic tasks, how can you run MaNtLE-BS and MaNtLE-PF (to evaluate faithfulness)\n* Is there a difference between explaining a subset of the data versus the entire predictor?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments on our draft. We are particularly encouraged that the reviewer finds our work “very novel” and our method “very efficient at inference”. Additionally, we are thankful for the reviewer’s “appreciation” of the “wide range of [experimental] settings” in our work. The reviewer also raised some thoughtful concerns about our work. Below, we address these concerns:\n\n- **MaNtLE sees thousands of synthetic tasks; Baselines are budget-constrained**: We would like to clarify that pre-training is done just once, and the model is never fine-tuned on new tasks. It's important to note that this pre-trained model will be released upon publication. Thus, pre-training is not an additional demand placed on the user of MaNtLE.\nOur evaluations in Section 5 are geared towards fair computations for each method on the new datasets that have been presented. From the perspective of computation utilized for the new task, the baselines aren’t the only budget-constrained approaches; MaNtLE is budget-constrained, too. To the best of our knowledge, we have applied the LIME and Anchors to their best ability (barring the model call computation) and hence find it a completely fair evaluation of approaches. We can provide a more detailed explanation if the reviewer could kindly articulate their exact concern with respect to computation.\n\n\n- **MaNtLE acts on subsets but evaluates on full**: Note that MaNtLE is not optimized to maximize simulatability. This is an emergent attribute from large-scale pre-training. Ideally, we would like to obtain explanations from MaNtLE that can be applied across all model predictions. However, it is possible that MaNtLE misses out on explaining some tail behavior, as pointed out by the reviewer. Conceptually, this can be alleviated by recursively applying MaNtLE to explain different subsections of the data to improve the coverage of explanations.\n\n\n- **Examples of explanations in Adult/Recidivism/Travel-Insurance**: We would like to point out that Figure 15 (in the Appendix) provides an example of the explanations from the three methods. However, given the graphical nature of LIME explanations (Figure 15), we instead present some of MaNtLE-PF’s explanations explaining the logistic regression classifier in results of Section 5 below:\\\n\\\nAdult:\n1. If Hours per week equal to 40, then Income is often >50K\n2. If Marital Status equal to Married, then Income is definitely >50K\n3. If Education equal to High School grad, then Income is sometimes >50K\\\n\\\nRecidivism:\n1. If Alcohol not equal to Yes, then Recidivism is occasionally commit\n2. If Race equal to White AND Priors equal to 1 to 5, then Recidivism is likely commit\n3. If PrisonViolations not equal to Yes, then Recidivism is seldom commit\\\n\\\nTravel-insurance:\n1. If FrequentFlyer not equal to Yes, then the customer is never interested in travel insurance\n2. If AnnualIncome not equal to Low, then the customer is certainly interested in travel insurance\n3. If Age greater than 26, then the customer is typically interested in travel insurance\\\n\\\nMore examples of explanations generated by all methods will be incorporated in the future version.\n\n- **Intractability of faithfulness/simulatability computation on real-world datasets**: We assume that the question is in relation to the use of a semantic parser in our experiments here. In more general scenarios, where we can have more diverse linguistic elements in explanations, we can assume access to models like LaSQuE (Ghosh et al. 2023) to provide approximate estimates of the utility of different explanations. We also note that we can parallelize the computation of these metrics for each explanation, making this time-efficient.\n\n- **Difference between explaining subset vs. entire predictor**: Great question! Suppose we are interested in understanding how the educational qualifications of individuals in a specific province (let's call it \"Province X\") in a country (\"Y\") influence their earnings. However, we are handed a national database containing information from across the country.  In this situation, our approach would be to filter out the “subset” of examples corresponding to people in province X from the national database and run MaNtLE to interpret the general pattern of educational background among people in the province. Of course, this is a toy scenario trying to explain information from a dataset (or database). A similar analogy can be made to analyze model predictions on different sections of data.\n\n**References**:\n- Ghosh et al. 2023: LaSQuE: Improved Zero-Shot Classification from Explanations Through Quantifier Modeling and Curriculum Learning, Findings of ACL 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper’s approach of fine-tuning explanations over many different tasks seems interesting, and very novel relative to prior work.\n* The authors study explanations in a range of settings (synthetic, zero shot generation, human in the loop, etc.) which I appreciated.\n* Their method, MaNtLE, is very efficient at inference time; unlike existing instance-level methods that require perturbing an input in order to come up with an explanation, MaNtLE simply generates explanations", "reasons_to_reject": "* MaNtLE to do worse on many of the realistic tasks presented, despite some advantages (get to choose the best of 20 explanations, other methods are budget constrained, while mantle sees thousands of synthetic tasks, extra postprocessing for mantle explanations instead of LIME). \n* The authors frame MaNtLE as acting on subsets, but it seems like it’s aiming to evaluate full (via simulatability metric). This means MaNtLE likely misses all tail behavior / doesn’t produce “correct” explanations. And it’s unclear if the MaNtLE explanations are more faithful than just the standard label function (even when the model differs from this due to imperfect accuracy\n* MaNtLE-BS and MaNtLE-PF rely on having a way to quickly test the faithfulness of explanations, which seems intractable on real datasets.\n* Correctness of explanations is measured “functionally”, but if the human (or zero-shot model) isn’t getting perfect accuracy, the explanations are still wrong.\n* It would be helpful to have more motivating examples for what counts as an explanation (and how to parse it) for the synthetic data.", "questions_to_authors": "* Could you provide random examples of explanations generated by each method on Adult, Travel Ins., and Recidivism? \n* On more realistic tasks, how can you run MaNtLE-BS and MaNtLE-PF (to evaluate faithfulness)\n* Is there a difference between explaining a subset of the data versus the entire predictor?", "paper_topic_and_main_contributions": "This paper introduces a new technique to explain model behavior, specifically focusing on explaining a subset of points (as opposed to individual point). The method works by training a T5 model on thousands of synthetic tasks for which an explanation of the labels is known; specifically, they fine-tune a model to predict the explanation (via generation) from 10+ examples with labels. In order to come up with a new explanation, the authors feed in examples (with the constraint that at least 10% of the examples come from each task), then either greedily generates an explanation, generates 20 with a beam search (then chooses the one that most faithfully follows the labels on the chosen points), or generates 20 using a method that favors diversity, then chooses the most faithful again. The authors find that their method generalizes reasonably well to unseen synthetic datasets, outperforms Anchor and LIME on adult, but largely loses out to Anchor on travel insurance and recidivism (even in the “budget-constrained” setting where Anchor gets much fewer samples than general), and in a human study finds that their (postprocessed) explanations are more useful for humans to predict model behavior than those from LIME or anchors (on the adult dataset).", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Perhaps relevant, work finding semantically-coherent subsets models do poorly on (e.g., https://arxiv.org/abs/2203.14960 and followups).", "Typos_Grammar_Style_and_Presentation_Improvements": "Typos\n* Line 211: comma error"}, "extra_text": {}}, {"review_id": "RES3Nlz7t0", "reviewer": "Reviewer_tLzw", "summary": "", "questions": "A. Why are the classification accuracy so low in Table 2? The highest number is 63% on the binary classification dataset. Additionally, where does the 7% on L333 come from?\n\nB. Why do you sample 100 subsets (each with 10 examples) for automatic evaluation? Why do you choose it over having less subsets but each with more examples?\n\nC. Why are there only 10% examples that belong to `{label}` and `not {label}` classes? What are the other classes? I thought all non-target-label classes are already converted to `not {label}` class.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments on our draft. We are grateful for the reviewer’s acknowledgment of the “extensive evaluation” of MaNtLE in our work. Below, we address these concerns raised by the reviewer:\n\n- **Unfair comparison with LIME/Anchors**: Our evaluations in Section 5 are geared towards fair computations for each method on the new datasets that have been presented. From the perspective of computation utilized for the new task, the baselines aren’t the only budget-constrained approaches; MaNtLE is budget-constrained, too. To the best of our knowledge, we have applied the LIME and Anchors to their best ability (barring the model call computation) and hence find it a completely fair evaluation of approaches.\\\n\\\nIt is important to note that in Section 6.2, the experiment displayed LIME and Anchors with full budget allocations, granting LIME access to 5000 perturbations and Anchors access to the entire training set. Consequently, Figure 7 portrays higher numbers of faithfulness and simulatability for LIME. However, MaNtLE remains restricted to the budget constraint, and therefore has lower numbers. Nevertheless, as the number of examples given to each method as input increases, the performance of MaNtLE-PF improves, and closely matches that of LIME.\n\n\n- **NLE Baselines**: Please refer to the point under Reviewer P3Ea (1st review on this page) with the same sub-title.\n\n\n- **Evaluation not different from NLEs**: We feel there is a misunderstanding here. MaNtLE is utilized to explain classifiers per our original goal indicated throughout the paper. Concretely, the experiments in Section 5, 6.2, and Appendix D seek to explain different classifiers (logistic regression, neural networks, decision trees, and XGBoost) trained on three different datasets.\\\n\\\nIn faithfulness and simulatabilty experiments, we evaluate the effectiveness of MaNtLE’s explanations in retrieving the same predictions as made by the classifiers for examples from the input (for faithfulness) and unseen examples (for simulatability). The unseen examples could have been from the (unseen) training, validation, or test set. Our choice of using the test set for simulatability in that sense is arbitrary and bears no relevance to the final conclusions.\n\n\n- **Unclear automatic evaluation of LIME/Anchors**: Firstly, we would like to re-iterate that MaNtLE does not perform any classification (see Figure 1). It explains the predictions made by other classifiers. Aside from T5 forming the base of MaNtLE, we do not explain a T5 model’s predictions anywhere in the paper.\\\n\\\nNow, we would like to clarify the reviewer’s query wrt the evaluation of LIME. LIME (Ribeiro et al. 2016) fits a linear regression module (over feature values) that predicts the probability of a particular example belonging to {label} vs. not {label}. The weights of the linear classifier are used as the feature attributions, as per the original paper. Now, to make predictions using the LIME explanations, we merely use these feature attributions as weights of a linear classifier again and predict on new examples. A similar logic flows for Anchors as well. We hope this clarifies the doubt here.\n\n\n- **Low classification accuracies in Table 2**: Human performance using crowdsourced explanations for these datasets is estimated to be 70% (as per Menon et al. 2022). Hence, assuming some error due to incomplete natural language understanding, having classification accuracies in the ballpark of 60% is expected for models like LaSQuE (Ghosh et al. 2023). \\\n\\\nRegarding the 7% gap mentioned in L333, we regret the confusion caused by this error. We meant that the gap between the *best* MaNtLE explanations and the crowdsourced explanations is within 4% of each other (absolute). \n\n\n- **100 subsets**: The concept of utilizing 100 subsets is analogous to conducting 100 distinct model runs with varied seeds, except that in this case, we evaluate 100 diverse subsets of input data. This was done to provide a more robust estimate of the different metrics of interest in our paper.\\\n\\\nIt's important to mention that the experiment requested by the reviewer has been executed and is detailed in our analysis within Section 6.2. It's noteworthy that the inclusion of additional strategies is essential due to the inherent limitation that not all examples can fit into MaNtLE's input due to restrictions on the maximum token length. For details, please refer to the elaboration in Section 6.2.\n\n- **On 10% examples belonging to label and non-label classes**: We would like to clarify that the statement mentions “at least 10% of the examples belong to either class” (L241-242). Accordingly, all examples passed into MaNtLE are either of the ‘{label}’ or ‘not {label}’ class with *AT LEAST* 10% of each.  \n\n**References**:\n- Ribeiro et al. 2016: “Why Should I Trust You?” Explaining the Predictions of Any Classifier,  SIGKDD 2016\n- R. Menon et al. 2022 : CLUES: A Benchmark for Learning Classifiers using Natural Language Explanations, ACL 2022\n- Ghosh et al. 2023: LaSQuE: Improved Zero-Shot Classification from Explanations Through Quantifier Modeling and Curriculum Learning, Findings of ACL 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Proposes a model-agnostic explainer that generates natural language explanations for structured classification tasks.\n- Evaluated their method extensively: including comparing to popular interpretability tools, conducting both automatic evaluation and human evaluation, and further analyzing the factors affecting the generalizability of MaNtLE.", "reasons_to_reject": "- Unfair comparisons to baselines: The authors trained T5-large on thousands of synthetic tasks, while only limiting LIME and Anchors to use very few perturbations / training examples. They acknowledged that “LIME uses 5k perturbations to fit a good linear model and generated explanations for tabular datasets.” So it is unclear to me whether their results in showing MaNtLE outperforming LIME is because their method is actually effective or because they simply did not provide enough input to LIME.\n- Missing NLE baselines: The authors only compared to feature attribution type of explanations, but their explainer is generating natural language explanations. They should compare with other NLE interpretability tools.\n- Evaluation is not different from NLEs: The authors defined faithfulness and simulatability as classification utility on training (perturbation) and testing set, respectively. This is different from what they claim to be doing differently from previous works that focuses on improving model classification performance with natural language explanations. The authors should seek to evaluate how well MaNtLE can explain model behavior (as claimed in related work to differentiate themselves from the other NLE work), but the current two metrics do not show how they are measuring this.\n- Unclear automatic evaluation for LIME and Anchors: Unclear how LIME and Anchors explanations are being used to predict the label. In particular, LIME generates attribution scores for each feature, so it is not a natural language explanation format. If the authors use T5 to predict the label, then it is unfair because T5 is a LM, which handle natural language input better.", "questions_to_authors": "A. Why are the classification accuracy so low in Table 2? The highest number is 63% on the binary classification dataset. Additionally, where does the 7% on L333 come from?\n\nB. Why do you sample 100 subsets (each with 10 examples) for automatic evaluation? Why do you choose it over having less subsets but each with more examples?\n\nC. Why are there only 10% examples that belong to `{label}` and `not {label}` classes? What are the other classes? I thought all non-target-label classes are already converted to `not {label}` class.", "paper_topic_and_main_contributions": "This paper proposes a new explainability tool (MaNtLE) for machine learning systems. Their explainer targets on explaining structure classification tasks with natural language explanations (NLE). To build such a classifier, they fine-tune T5-large on thousands of synthetic classification tasks with NLE supervision using multi-task training. They evaluated the usefulness of explanations by measuring the model classification-utility of the generated explanations as well as human evaluation (including understandability, informativeness, and overall preference). They showed the MaNtLE outperforms previous interpretability tools LIME and Anchors.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "A lot of papers on NLE are not cited.", "Typos_Grammar_Style_and_Presentation_Improvements": "Typo:\n\n- Missed underscore for M in title.\n- Conventionally, figures should be on the top of the page. Figure 2 is currently below a footnote."}, "extra_text": {}}, {"review_id": "EVyAsyf58z", "reviewer": "Reviewer_CbXw", "summary": "", "questions": "- Do you have the online demo? I am interested to see the demos.\n- Discussion about scaling up to hundreds of attributes? I have the impression that this is mainly limited by the maximum sequence length of transform models.\n- Can this method be extended to embedding features?\n- Does the synthetic datasets include noise? I think this might be important to strengthen the robustness of the model.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their motivating comments on our draft. We are particularly grateful for the acknowledgment of the “simplicity” and “effectiveness” of our work alongside the appreciation for the readability of our work. Accessibility of our work to readers from diverse expertise is paramount to us; hence, this acknowledgment is particularly encouraging. Below, we address some of the queries from the reviewer:\n- **NLE baselines**: Kindly refer to the point mentioned under Reviewer P3Ea (1st review on this page) with the same sub-title.\n\n\n- **Online demo**: Thanks for the suggestion! We will create a website for hosting the demo to improve accessibility and interaction for other researchers. \n\n\n- **Scaling to thousands of attributes**: Indeed, as the reviewer mentions, we are limited by the number of tokens that can be used as input to the transformer model. Incorporating techniques like ALiBi (Press et al. 2022) to extend the inference time input for models would be an exciting future direction to stretch the application of MaNtLE for this scenario.\n\n\n- **Extending to embedding-based features**: If the point of discussion is shifting the attribute values alone from text to embeddings, it may yet be possible to train such a model for categorical tasks. However, suppose we are to replace the structured format altogether with embeddings. In that case, the model may be untrustworthy as we may not completely understand what sections of the input contributed to the explanation. \n\n\n- **Do synthetic datasets include noise?** It would be nice to gain some more clarity on the kind of noise referenced here. Through the use of quantifiers in the synthetic task explanations, we are already introducing some stochasticity in the predictions. Further, we randomize the order of features in every training batch to ensure that the order of feature names does not bias the generated explanations.\n\n\n**References**:\n- Press et al. 2022 : Train Short, Test Long: Attention With Linear Biases Enables Input Length Extrapolation, ICLR 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Simple and effective method.  This method provides a new direction for the language explanations for model predictions, which does not need any human labeling effort. With the rapid development of LLM, I think there are a lot of research ideas to explore in this direction.\n- Well-written paper. Easy to follow.", "reasons_to_reject": "- Based on the categorization of explainability methods discussed in Section 2, I am expecting to compare the proposed method against language explanation methods, such as CAGE or WT5, instead of feature attribution methods, such as LIME and Anchor.  I have concerns about these mismatched baselines.\n", "questions_to_authors": "- Do you have the online demo? I am interested to see the demos.\n- Discussion about scaling up to hundreds of attributes? I have the impression that this is mainly limited by the maximum sequence length of transform models.\n- Can this method be extended to embedding features?\n- Does the synthetic datasets include noise? I think this might be important to strengthen the robustness of the model.", "paper_topic_and_main_contributions": "In this work, the authors propose to use large language models to generate explanations for classifier predictions.\nThe proposed method is simple yet effective.\nSpecifically, this work fine-tunes a T5 model on the thousands of synthetic classification tasks (with natural language explanations). Then, the learned model is used to generate explanations for unseen tasks.\nThe synthetic tasks and explanations are completely automatically generated, which saves a lot of human effort comparing previous methods.\n\nThis work first compares the generated explanations to the human-written explanations on the CLUES-Real benchmark. The results show that the proposed method can generate explanations that are similar quality of human explanations.\nThen,  the proposed method is compared against previous works, namely LIME and Anchors. The results show that the proposed method is better than the baselines in terms of faithfulness and simulatability", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes MaNtLE, a model-agnostic natural language explainer which generates explanations for subsets of examples in structured classification tasks. According to the initial reviews, this paper conducts comprehensive experiments (studying explanations in several settings and performing both automatic and human evaluations). Also, the method is efficient at inference time without requiring input perturbation. However, there were some concerns raised, especially on the evaluation part (e.g., the missing NLE baselines and the unfair comparison to the baselines). During the rebuttal phase, the authors well addressed several questions and concerns of the reviewers. In particular, the authors argued that thousands of synthetic tasks MaNtLE saw (which seem unfair to other baselines) were used for pre-training MaNtLE, which happened only once, and this pre-trained model will be released upon publication so public users can leverage it in an efficient way. The authors also presented additional results comparing MaNtLE to a WT5-style model, served as an NLE baseline. Both models were pre-trained on the same number of synthetic tasks, and MaNtLE still outperformed WT5 in terms of simulatability. It has to be noted that, according to the authors' argument, MaNtLE and WT5 are not directly comparable by nature as the former explains a set of examples while the latter explains an individual example. The newly presented results were possible due to a simple aggregation method the authors used to enable the comparison.
Decision: Accept-Main