text:   Counterfactuals",6,"['Excited to share our (w/ Rohan Nair, @gregd_nlp ) new pre-print ""Evaluating Explanations for Reading Comprehension with Realistic Counterfactuals"":\n<LINK>\n\nWe propose a methodology to evaluate explanations <LINK>', ""An explanation should allow us to understand the reading comprehension model's high-level behavior with respect to a set of realistic counterfactual input scenarios.\xa0\nAs an example, we show explanations for a HotpotQA example (blue) generated by several methods."", ""At first glance, it seems token-level attributions are more plausible than pairwise interactions, as they highlight both movies being documentaries. But does that reflect the model's true reasoning?"", 'We profile the model behaviors with the predictions on realistic counterfactual input. The model always makes the same predictions on the counterfactuals \xa0-- and the pairwise explanation actually conveys this more accurately!', 'We annotate realistic counterfactuals on multiple settings (HotpotQA, SQuAD, and synthetic) and evaluate several explanation methods including token-level attribution techniques and pairwise interaction techniques\xa0in terms of whether they can give insights about model behavior.', 'Our analysis suggests that pairwise explanation techniques are better suited to\xa0analyzing the behavior\xa0of RC models, which fundamentally involves complex interaction between questions and contexts.\n\ncode and data: https://t.co/4uxg53GX6r']",21,04,1398