text: 330,211,1382432437632790529,1242135548040548352,Xi Ye,"Excited to share our (w/ Rohan Nair, @gregd_nlp ) new pre-print ""Evaluating Explanations for Reading Comprehension with Realistic Counterfactuals"": <LINK> We propose a methodology to evaluate explanations <LINK> An explanation should allow us to understand the reading comprehension model's high-level behavior with respect to a set of realistic counterfactual input scenarios.  As an example, we show explanations for a HotpotQA example (blue) generated by several methods. At first glance, it seems token-level attributions are more plausible than pairwise interactions, as they highlight both movies being documentaries. But does that reflect the model's true reasoning? We profile the model behaviors with the predictions on realistic counterfactual input. The model always makes the same predictions on the counterfactuals  -- and the pairwise explanation actually conveys this more accurately! We annotate realistic counterfactuals on multiple settings (HotpotQA, SQuAD, and synthetic) and evaluate several explanation methods including token-level attribution techniques and pairwise interaction techniques in terms of whether they can give insights about model behavior. Our analysis suggests that pairwise explanation techniques are better suited to analyzing the behavior of RC models, which fundamentally involves complex interaction between questions and contexts. code and data: <LINK>",https://arxiv.org/abs/2104.04515,"When a model attribution technique highlights a particular part of the input, a user might understand this highlight as making a statement about counterfactuals (Miller, 2019): if that part of the input were to change, the model's prediction might change as well. This paper investigates how well different attribution techniques align with this assumption on realistic counterfactuals in the case of reading comprehension (RC). RC is a particularly challenging test case, as token-level attributions that have been extensively studied in other NLP tasks such as sentiment analysis are less suitable to represent the reasoning that RC models perform. We construct counterfactual sets for three different RC settings, and through heuristics that can connect attribution methods' outputs to high-level model behavior, we can evaluate how useful different attribution methods and even different formats are for understanding counterfactuals. We find that pairwise attributions are better suited to RC than token-level attributions across these different RC settings, with our best performance coming from a modification that we propose to an existing pairwise attribution method. ","Connecting Attributions and QA Model Behavior on Realistic