SlowGuess/ACL_2020_OCR huggingface dataset:
Textual reasoning Prior work (Gupta et al., 2020) proposed heuristic methods to extract supervision for the find-num and find-date modules in DROP. On top of the end-to-end objective, they use an auxiliary objective that encourages these modules to output the "gold" numbers and dates according to the heuristic supervision. They show that supervising intermediate module outputs helps
improve model performance. In this work, we evaluate the effect of such supervision on the faithfulness of both the supervised modules, as well as other modules that are trained jointly.
4.3 Decontextualized word representations
The goal of decomposing reasoning into multiples steps, each focusing on different parts of the utterance, is at odds with the widespread use of contextualized representations such as BERT or LXMERT. While the utterance attention is meant to capture information only from tokens relevant for the module's reasoning, contextualized token representations carry global information. For example, consider the program filter [red] (find [car]) for the phrase red car. Even if find attends only to the token car, its representation might also express the attribute red, so find might learn to find just red cars, rather than all cars, rendering the filter module useless, and harming faithfulness. To avoid such contextualization in Visual-NMN, we zero out the representations of tokens that are unattended, thus the input to the module is computed (with LXMERT) from the remaining tokens only.
We first introduce the datasets used and the experimental setup for measuring faithfulness (§ 5.1). We demonstrate that training NMNs using end-task supervision only does not yield module-wise faithfulness both for visual and textual reasoning. We then show that the methods from §4 are crucial for achieving faithfulness and how different design choices affect it (§ 5.2). Finally, we qualitatively show examples of improved faithfulness and analyze possible reasons for errors (§ 5.3).
Please see Appendix C for further detail about the experimental setups.
Visual reasoning We automatically generate gold program annotations for 26,311 training set examples and for 5,772 development set examples from NLVR2. The input to this generation process is the set of crowdsourced question decompositions from the BREAK dataset (Wolfson et al., 2020). See Appendix C.1 for details. For module-wise faithfulness evaluation, 536 examples from the development set were annotated with the gold output for each module by experts.
5599
Model	Performance (Accuracy)	Overall Faithful. (↑)	Module-wise Faithfulness F1(↑)
Prec.	Rec.	F1	find	filter	with-relation	relocate
LXMERT	71.7
Upper Bound	1	0.84	0.89	0.89	0.92	0.95	0.75
NMN w/ Layer-count	71.2	0.39	0.39	0.11	0.12	0.20	0.37	0.27
NMN w/ Sum-count	68.4	0.49	0.31	0.28	0.31	0.32	0.44	0.26
NMN w/ Graph-count	69.6	0.37	0.39	0.28	0.31	0.29	0.37	0.19
NMN w/ Graph-count + decont.	67.3	0.29	0.51	0.33	0.38	0.30	0.36	0.13
NMN w/ Graph-count + pretraining	69.6	0.44	0.49	0.36	0.39	0.34	0.42	0.21
NMN w/ Graph-count + decont. + pretraining	68.7	0.42	0.66	0.47	0.52	0.41	0.47	0.21
Table 1: Faithfulness and accuracy on NLVR2. "decont." refers to decontextualized word representations. Precision, recall, and  $\mathrm{F}_{1}$  are averages across examples, and thus  $\mathrm{F}_{1}$  is not the harmonic mean of the corresponding precision and recall.
Model	Performance (F1 Score)	Overall Faithful. (cross-entropy* ↓)	Module-wise Faithfulness* (↓)
find	filter	relocate	min-max†	find-arg†
Text-NMN w/o prog-sup
w/ extract-answer	63.5	9.5	13.3	9.5	3.5	2.6	9.9
w/o extract-answer	60.8	6.9	8.1	7.3	1.3	1.7	8.5
Text-NMN w/ prog-sup
no auxiliary sup	65.3	11.2	13.7	16.9	1.5	2.2	13.0
w/o sorting & comparison	63.8	8.4	9.6	11.1	1.6	1.3	10.6
w/ module-output-sup	65.7	6.5	7.6	10.7	1.3	1.2	7.6
Table 2: Faithfulness and performance scores for various NMNs on DROP. *lower is better. †min-max is average faithfulness of find-min-num and find-max-num; find-arg of find-num and find-date.
Textual reasoning We train Text-NMN on DROP, which is augmented with program supervision for 4,000 training questions collected heuristically as described in Gupta et al. (2020). The model is evaluated on the complete development set of DROP which does not contain any program supervision. Module-wise faithfulness is measured on 215 manually-labeled questions from the development set, which are annotated with gold programs and module outputs (passage spans).
5.2 Faithfulness evaluation
Visual reasoning Results are seen in Table 1. Accuracy for LXMERT, when trained and evaluated on the same subset of data, is  $71.7\%$ ; slightly higher than NMNs, but without providing evidence for the compositional structure of the problem.
For faithfulness, we measure an upper-bound on the faithfulness score. Recall that this score measures the similarity between module outputs and annotated outputs. Since module outputs are constrained by the bounding boxes proposed by Faster-RCNN (§2.1), while annotated boxes are not, perfect faithfulness could only be achieved by a model if there are suitable bounding boxes. Upper Bound shows the maximal faithfulness score
conditioned on the proposed bounding boxes.
We now compare the performance and faithfulness scores of the different components. When training our NMN with the most flexible count module, (NMN w/ Layer-count), an accuracy of  $71.2\%$  is achieved, a slight drop compared to LXMERT but with low faithfulness scores. Using Sum-count drops about  $3\%$  of performance, but increases faithfulness. Using Graph-count increases accuracy while faithfulness remains similar.
Next, we analyze the effect of decontextualized word representations (abbreviated "decont.") and pre-training. First, we observe that  $NMN w/ Graph-count + \text{decont}$ . Increases faithfulness score to  $0.33 F_1$  at the expense of accuracy, which drops to  $67.3\%$ . Pre-training ( $NMN w/ Graph-count + \text{pretraining}$ ) achieves higher faithfulness scores with a higher accuracy of  $69.6\%$ . Combining the two achieves the best faithfulness ( $0.47 F_1$ ) with a minimal accuracy drop. We perform a paired permutation test to compare  $NMN w/ Graph-count + \text{decont}$  + pretraining with  $NMN w/ Layer-count$  and find that the difference in  $F_1$  is statistically significant ( $p < 0.001$ ). Please see Appendix D.1 for further details.
Textual reasoning As seen in Table 2, when trained on DROP using question-program supervision, the model achieves  $65.3\mathrm{F}_1$  performance and a faithfulness score of 11.2. When adding supervision for intermediate modules (§4.2), we find that the module-wise faithfulness score improves to 6.5. Similar to Visual-NMN, this shows that supervising intermediate modules in a program leads to better faithfulness.
To analyze how choice of modules affects faithfulness, we train without sorting and comparison modules (find-max-num, num-compare, etc.). We find that while performance drops slightly, faithfulness deteriorates significantly to 8.4, showing that modules that perform atomic reasoning are crucial for faithfulness. When trained without program supervision, removing extract-answer improves faithfulness  $(9.5\rightarrow 6.9)$  but at the cost of performance  $(63.5\to 60.8\mathrm{F}_1)$ . This shows that such a black-box module encourages reasoning in an opaque manner, but can improve performance by overcoming the limitations of pre-defined modules. All improvements in faithfulness are significant as measured using paired permutation tests  $(p < 0.001)$ .
Generalization A natural question is whether models that are more faithful also generalize better. We conducted a few experiments to see whether this is true for our models. For NLVR2, we performed (1) an experiment in which programs in training have length at most 7, and programs at test time have length greater than 7, (2) an experiment in which programs in training have at most 1 filter module and programs at test time have at least 2 filter modules, and (3) an experiment in which programs in training do not have both filter and with-relation modules in the same program, while each program in test has both modules. We compared three of our models - NMN w/ Layer-count, NMN w/ Sum-count, and NMN w/ Graph-count + decont. + pretraining. We did not observe that faithful models generalize better (in fact, the most unfaithful model tended to achieve the best generalization).
To measure if faithful model behavior leads to better generalization in Text-NMN we conducted the following experiment. We selected the subset of data for which we have gold programs and split the data such that questions that require maximum and greater-than operations are present in the training data while questions that require com
puting minimum and less-than are in the test data. We train and test our model by providing gold-programs under two conditions, in the presence and absence of additional module supervision. We find that providing auxiliary module supervision (that leads to better module faithfulness; see above) also greatly helps in model generalization (performance increases from  $32.3\mathrm{F}_1 \rightarrow 78.3\mathrm{F}_1$ ).
Model comparisons We analyze outputs of different modules in Figure 3. Figures 3a, 3b show the output of find[llamas] when trained with contextualized and decontextualized word representations. With contextualized representations (3a), the find fails to select any of the llamas, presumably because it can observe the word eating, thus effectively searching for eating llamas, which are not in the image. Conversely, the decontextualized model correctly selects the boxes. Figure 3c shows that find outputs meaningless probabilities for most of the bounding boxes when trained with Layer-count, yet the count module produces the correct value (three). Figure 3d shows that find fails to predict all relevant spans when trained without sorting modules in Text-NMN.
Error analysis We analyze cases where outputs were unfaithful. First, for visual reasoning, we notice that faithfulness scores are lower for long-tail objects. For example, for dogs, a frequent noun in NLVR2, the execution of find [dogs] yields an average faithfulness score of 0.71, while items such as roll of toilet paper, barbell and safety pin receive lower scores (0.22, 0.29 and 0.05 respectively; example for a failure case for safety pin in Fig. 3e). In addition, some objects are harder to annotate with a box (water, grass, ground) and therefore receive low scores. The issue of small objects can also explain the low scores of relocate. In the gold box annotations used for evaluation, the average areas for find, filter, with-relation, and relocate (as a fraction of the total image area) are 0.19, 0.19, 0.15, and 0.07, respectively. Evidently, relocate is executed with small objects that are harder to annotate (tongue, spots, top of), and indeed the upper-bound and model scores for relocate are lowest among the module types.
NMNs were originally introduced for visual question answering and applied to datasets with syn
5601
find[llamas] utt:"the llamas in both images are eating
find[people]
utt: "there are three people"
(c)
find[safety pin] utt: "at least one safety pin is not embellished."
(d)
(e)
Figure 3: Comparison of module outputs between NMN versions: (a) Visual-NMN with contextualized representations, (b) Visual-NMN with decontextualized representations, (c) model using a parameter-rich count layer (Layer-Count), (d) Text-NMN trained without sorting module produces an incorrect find output (misses 2-yard rushing TD), and (e) Visual-NMN failure case with a rare object (of w/ Graph-count + decont. + pretraining)
thetic language and images as well as VQA (Antol et al., 2015), whose questions require few reasoning steps (Andreas et al., 2016; Hu et al., 2017; Yang et al., 2018b). In such prior work, module-wise faithfulness was mostly assessed via qualitative analysis of a few examples (Jiang and Bansal, 2019; Gupta et al., 2020). Yang et al. (2018b) did an evaluation where humans rated the clarity of the reasoning process and also tested whether humans could detect model failures based on module outputs. In contrast, we quantitatively measure each module's predicted output against the annotated gold outputs.
A related systematic evaluation of interpretability in VQA was conducted by Trott et al. (2018). They evaluated the interpretability of their VQA counting model, where the interpretability score is given by the semantic similarity between the gold
label for a bounding box and the relevant word(s) in the question. However, they studied only counting questions, which were also far less compositional than those in NLVR2 and DROP.
Similar to the gold module output annotations that we provide and evaluate against, HOTPOTQA (Yang et al., 2018a) and CoQA (Reddy et al., 2019) datasets include supporting facts or rationales for the answers to their questions, which can be used for both supervision and evaluation.
In concurrent work, Jacovi and Goldberg (2020) recommend studying faithfulness on a scale rather than as a binary concept. Our evaluation method can be viewed as one example of this approach.
We introduce the concept of module-wise faithfulness, a systematic evaluation of faithfulness in neural module networks (NMNs) for visual and textual reasoning. We show that naive training of NMNs does not produce faithful modules and propose several techniques to improve module-wise faithfulness in NMNs. We show how our approach leads to much higher module-wise faithfulness at a low cost to performance. We encourage future work to judge model interpretability using the proposed evaluation and publicly published annotations, and explore techniques for improving faithfulness and interpretability in compositional models.
We thank members of UCI NLP, TAU NLP, and the AllenNLP teams as well as Daniel Khashabi for comments on earlier drafts of this paper. We also thank the anonymous reviewers for their comments. This research was partially supported by The Yandex Initiative for Machine Learning, the European Research Council (ERC) under the European Union Horizons 2020 research and innovation programme (grant ERC DELPHI 802800), funding by the ONR under Contract No. N00014-19-1-2620, and by sponsorship from the LwLL DARPA program under Contract No. FA8750-19-2-0201. This work was completed in partial fulfillment for the Ph.D degree of Ben Boin.
5602
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Learning to compose neural networks for question answering. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1545-1554, San Diego, California. Association for Computational Linguistics.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378.
Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. 2020. Neural Module Networks for Reasoning over Text. In International Conference on Learning Representations (ICLR).
Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415.
Minghao Hu, Yuxing Peng, Zhen Huang, and Dongsheng Li. 2019. A multi-type multi-span network for reading comprehension that requires discrete reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1596-1606, Hong Kong, China. Association for Computational Linguistics.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 804-813.
Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6700-6709.
Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? In Proceedings of the 2020 Conference of the Association for Computational Linguistics.
Yichen Jiang and Mohit Bansal. 2019. Self-assembling modular networks for interpretable multi-hop reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4473-4483, Hong Kong, China. Association for Computational Linguistics.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. 2016. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1988-1997.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-73.
Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1516-1526, Copenhagen, Denmark. Association for Computational Linguistics.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. 2019. The neurosymbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In International Conference on Learning Representations.
Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. Compositional questions do not necessitate multi-hop reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4249-4257, Florence, Italy. Association for Computational Linguistics.
E.W. Noreen. 1989. Computer-Intensive Methods for Testing Hypotheses: An Introduction. Wiley.
Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Proceedings of the 28th International Conference on
5603
Neural Information Processing Systems - Volume 1, NIPS'15, pages 91-99, Cambridge, MA, USA. MIT Press.
Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Velez. 2017. Right for the right reasons: Training differentiable models by constraining their explanations. In *IJCAI*.
Howard Seltman. 2018. Approximations for mean and variance of a ratio.
Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. 2017. A corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 217-223, Vancouver, Canada. Association for Computational Linguistics.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6418-6428, Florence, Italy. Association for Computational Linguistics.
Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641-651, New Orleans, Louisiana. Association for Computational Linguistics.
Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5099-5110, Hong Kong, China. Association for Computational Linguistics.
Alexander Trott, Caiming Xiong, and Richard Socher. 2018. Interpretable counting for visual question answering. In International Conference on Learning Representations.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Dan Ventura. 2007. CS478 Paired Permutation Test Overview. Accessed April 29, 2020.
Sarah Wiegrefe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language
Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11-20, Hong Kong, China. Association for Computational Linguistics.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question understanding benchmark. Transactions of the Association for Computational Linguistics, 8:183-198.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018a. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018b. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Computational Linguistics.
Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. In Proceedings of the 18th conference on Computational linguistics-Volume 2, pages 947-953. Association for Computational Linguistics.
Yan Zhang, Jonathon Hare, and Adam Prgel-Bennett. 2018. Learning to count objects in natural images for visual question answering. In International Conference on Learning Representations.
5604
We list all modules for Visual-NMN in Table 3.
For Text-NMN, as mentioned, we use all modules are described in Gupta et al. (2020). In this work, we introduce the (a) addition and subtraction modules that take as input two distributions over numbers mentioned in the passage and produce a distribution over all possible addition and subtraction values possible. The output distribution here is the expected distribution for the random variable  $Z = X + Y$  (for addition), and (b) extract-answer that produces two distributions over the passage tokens denoting the probabilities for the start and end of the answer span. This distribution is computed by mapping the passage token representations using a simple MLP and softmax operation.
B Measuring Faithfulness in Visual-NMN
B.1 Numerators of Precision and Recall