SlowGuess/ACL_2020_OCR huggingface dataset:
Sanjay Subramanian\*1 Ben Bogin\*2 Nitish Gupta\*3 Tomer Wolfson\*1,2 Sameer Singh4 Jonathan Berant\*1,2 Matt Gardner
$^{1}$ Allen Institute for AI  $^{2}$ Tel-Aviv University
$^{3}$ University of Pennsylvania  $^{4}$ University of California, Irvine
{sanjays,mattg}@allenai.org, {ben.bogin,joberant}@cs.tau.ac.il, nitishg@seas.upenn.edu, tomerwol@mail.tau.ac.il, sameer@uci.edu
Neural module networks (NMNs) are a popular approach for modeling compositionality: they achieve high accuracy when applied to problems in language and vision, while reflecting the compositional structure of the problem in the network architecture. However, prior work implicitly assumed that the structure of the network modules, describing the abstract reasoning process, provides a faithful explanation of the model's reasoning; that is, that all modules perform their intended behaviour. In this work, we propose and conduct a systematic evaluation of the intermediate outputs of NMNs on NLVR2 and DROP, two datasets which require composing multiple reasoning steps. We find that the intermediate outputs differ from the expected output, illustrating that the network structure does not provide a faithful explanation of model behaviour. To remedy that, we train the model with auxiliary supervision and propose particular choices for module architecture that yield much better faithfulness, at a minimal cost to accuracy.
Models that can read text and reason about it in a particular context (such as an image, a paragraph, or a table) have been recently gaining increased attention, leading to the creation of multiple datasets that require reasoning in both the visual and textual domain (Johnson et al., 2016; Suhr et al., 2017; Talmor and Berant, 2018; Yang et al., 2018a; Suhr et al., 2019; Hudson and Manning, 2019; Dua et al., 2019). Consider the example in Figure 1 from NLVR2: a model must understand the compositional sentence in order to then ground dogs in the input, count those that are black and verify that the count of all dogs in the image is equal to the number of black dogs.
"All the dogs are black."
find [dogs]
Basic-NMN
Faithful-NMN
find [dogs]
filter[black]
count count
1.4 1.6   
|
#
equal
False (57%)
False (98%)
Figure 1: An example for a visual reasoning problem where both the Basic and Faithful NMNs produce the correct answer. The Basic NMN, however, fails to give meaningful intermediate outputs for the find and filter modules, whereas our improved Faithful-NMN assigns correct probabilities in all cases. Boxes are green if probabilities are as expected, red otherwise.
filter[black]
count count
0.9 2
equal
Both models that assume an intermediate structure (Andreas et al., 2016; Jiang and Bansal, 2019) and models without such structure (Tan and Bansal, 2019; Hu et al., 2019; Min et al., 2019) have been proposed for these reasoning problems. While good performance can be obtained without a structured representation, an advantage of structured approaches is that the reasoning process in such approaches is more interpretable. For example, a structured model can explicitly denote that there are two dogs in the image, but that one of them is not black. Such interpretability improves our scientific understanding, aids in model development, and improves overall trust in a model.
5594
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5594-5608
two dogs are touching a food dish with their face
Figure 2: An example for a mapping of an utterance to a gold program and a perfect execution in a reasoning problem from NLVR2 (top) and DROP (bottom).
with-relation [is touching]
relocate [face]
find [dog]
find [food dish]
number [two]
True 2
[2, 5]
[1, 4]
[3, 6]
Who threw the longest touchdown pass in the second half?
In the first quarter, the Texans trailed early after QB Kerry Collins threw a 19-yard TD pass [1] to WR Nate Washington. Second quarter started with kicker Neil Rackers made a 37-yard field goal, and the quarter closed with kicker Rob Bironas hitting a 30-yard field goal. The Texans tried to cut the lead with QB Matt Schaub getting a 8-yard TD pass [2] to WR Andre Johnson, but the Titans would pull away with RB Javon Ringer throwing a 7-yard TD pass [3]. The Texans tried to come back into the game in the fourth quarter, but only came away with Schaub [4] throwing a 12-yard TD pass [5] to WR Kevin Walter.
relocate[who threw]
find-max-num
filter [the second half]
find [touchdown pass]
Schaub [4]
[2, 3, 5]
[1, 2, 3, 5]
Neural module networks (NMNs; Andreas et al., 2016) parse an input utterance into an executable program composed of learnable modules that are designed to perform atomic reasoning tasks and can be composed to perform complex reasoning against an unstructured context. NMNs are appealing since their output is interpretable; they provide a logical meaning representation of the utterance and also the outputs of the intermediate steps (modules) to reach the final answer. However, because module parameters are typically learned from end-task supervision only, it is possible that the program will not be a faithful explanation of the behaviour of the model (Ross et al., 2017; Wiegrefe and Pinter, 2019), i.e., the model will solve the task by executing modules according to the program structure, but the modules will not perform the reasoning steps as intended. For example, in Figure 1, a basic NMN predicts the correct answer False, but incorrectly predicts the output of the find[dogs] operation. It does not correctly locate one of the dogs in the image because two of the reasoning steps (find and filter) are collapsed into one module (find). This behavior of the find module is not faithful to its intended reasoning operation; a human reading the program would expect find[dogs] to locate all dogs. Such unfaithful module behaviour yields an unfaithful explanation of the model behaviour.
Unfaithful behaviour of modules, such as multiple reasoning steps collapsing into one, are undesirable in terms of interpretability; when a model fails to answer some question correctly, it is hard to tell which modules are the sources of error. While recent work (Yang et al., 2018b; Jiang and Bansal,
2019) has shown that one can obtain good performance when using NMNs, the accuracy of individual module outputs was mostly evaluated through qualitative analysis, rather than systematically evaluating the intermediate outputs of each module.
We provide three primary contributions regarding faithfulness in NMNs. First, we propose the concept of module-wise faithfulness - a systematic evaluation of individual module performance in NMNs that judges whether they have learned their intended operations, and define metrics to quantify this for both visual and textual reasoning (§3). Empirically, we show on both NLVR2 (Suhr et al., 2019) and DROP (Dua et al., 2019) that training a NMN using end-task supervision, even using gold programs, does not yield module-wise faithfulness, i.e., the modules do not perform their intended reasoning task. Second, we provide strategies for improving module-wise faithfulness in NMNs (§4). Specifically, (a) we demonstrate how module architecture affects faithfulness (§4.1), (b) propose supervising module outputs with either a proxy task or heuristically generated data (§4.2), and (c) show that providing modules with uncontextualized token representations improves faithfulness (§4.3). Figure 1 shows an example where our approach (Faithful-NMN) results in expected module outputs as compared to the Basic-NMN. Last, we collect human-annotated intermediate outputs for 536 examples in NLVR2 and for 215 examples in DROP to measure the module-wise faithfulness of models, and publicly release them for future work. Our code and data are available at https://github.com/allenai/faithful-nmn.
2 Neural Module Networks
Overview Neural module networks (NMNs; Andreas et al., 2016) are a class of models that map a natural language utterance into an executable program, composed of learnable modules that can be executed against a given context (images, text, etc.), to produce the utterance's denotation (truth value in NLVR2, or a text answer in DROP). Modules are designed to solve atomic reasoning tasks and can be composed to perform complex reasoning. For example, in Figure 1, the utterance "All the dogs are black" is mapped to the program equal(count(find[dogs]), count.filter[black](find[dogs])) The find module is expected to find all dogs in the image and the filter module is expected to output only the black ones from its input. Figure 2 shows two other example programs with the expected output of each module in the program.
A NMN has two main components: (1) parser, which maps the utterance into an executable program; and (2) executor, which executes the program against the context to produce the denotation. In our setup, programs are always trees where each tree node is a module. In this work, we focus on the executor, and specifically the faithfulness of module execution. We examine NMNs for both text and images, and describe their modules next.
2.1 Modules for visual reasoning
In this task, given two images and a sentence that describes the images, the model should output True iff the sentence correctly describes the images. We base our model, the Visual-NMN, on LXMERT (Tan and Bansal, 2019), which takes as input the sentence  $x$  and raw pixels, uses Faster R-CNN (Ren et al., 2015) to propose a set of bounding boxes,  $\mathcal{B}$ , that cover the objects in the image, and passes the tokens of  $x$  and the bounding boxes through a Transformer (Vaswani et al., 2017), encoding the interaction between both modalities. This produces a contextualized representation  $t \in \mathbb{R}^{|x| \times h}$  for each one of the tokens, and a representation  $v \in \mathbb{R}^{|B| \times h}$  for each one of the bounding boxes, for a given hidden dimension  $h$ .
We provide a full list of modules and their implementation in Appendix A. Broadly, modules take as input representations of utterance tokens through an utterance attention mechanism (Hu et al., 2017), i.e., whenever the parser outputs a module, it also predicts a distribution over the utterance to-
kens  $(p_{1},\ldots ,p_{|x|})$ , and the module takes as input  $\sum_{i = 1}^{|x|}p_i t_i$ , where  $t_i$  is the hidden representation of token  $i$ . In addition, modules produce as output (and take as input) vectors  $\pmb{p}\in [0,1]^{|\mathcal{B}|}$ , indicating for each bounding box the probability that it should be output by the module (Mao et al., 2019). For example, in the program filter[black](find[dog]), the find module takes the word 'dog' (using utterance attention, which puts all probability mass on the word 'dog'), and outputs a probability vector  $\pmb{p}\in [0,1]^{|B|}$ , where ideally all bounding boxes corresponding to dogs have high probability. Then, the filter module takes  $\pmb{p}$  as input as well as the word 'black', and is meant to output high probabilities for bounding boxes with 'black dogs'.
For the Visual-NMN we do not use a parser, but rely on a collected set of gold programs (including gold utterance attention), as described in §5. We will see that despite this advantageous setup, a basic NMN does not produce interpretable outputs.
2.2 Modules for textual reasoning
Our Text-NMN is used to answer questions in the DROP dataset and uses the modules as designed for DROP in prior work (Gupta et al., 2020) along with three new modules we define in this work. The modules introduced in Gupta et al. (2020) and used as is in our Text-NMN are find, filter, relocate, count, find-num, find-date, find-max-num, find-min-num, num-compare and date-compare. All these modules are probabilistic and produce, as output, a distribution over the relevant support. For example, find outputs a distribution over the passage tokens and find-num outputs a distribution over the numbers in the passage. We extend their model and introduce additional modules; addition and subtraction to add or subtract passage numbers, and extract-answer which directly predicts an answer span from the representations of passage tokens without any explicit compositional reasoning. We use BERT-base (Devlin et al., 2019) to encode the input question and passage.
The Text-NMN does not have access to gold programs, and thus we implement a parser as an encoder-decoder model with attention similar to Krishnamurthy et al. (2017), which takes the utterance as input, and outputs a linearized abstract syntax tree of the predicted program.
5596
3 Module-wise Faithfulness
Neural module networks (NMNs) facilitate interpretability of their predictions via the reasoning steps in the structured program and providing the outputs of those intermediate steps during execution. For example, in Figure 2, all reasoning steps taken by both the Visual-NMN and Text-NMN can be discerned from the program and the intermediate module outputs. However, because module parameters are learned from an end-task, there is no guarantee that the modules will learn to perform their intended reasoning operation. In such a scenario, when modules do not perform their intended reasoning, the program is no longer a faithful explanation of the model behavior since it is not possible to reliably predict the outputs of the intermediate reasoning steps given the program. Work on NMNs thus far (Yang et al., 2018b; Jiang and Bansal, 2019) has overlooked systematically evaluating faithfulness, performing only qualitative analysis of intermediate outputs.
We introduce the concept of module-wise faithfulness aimed at evaluating whether each module has correctly learned its intended operation by judging the correctness of its outputs in a trained NMN. For example, in Figure 2 (top), a model would be judged module-wise faithful if the outputs of all the modules, find, relocate, and with relation, are correct - i.e. similar to the outputs that a human would expect. We provide gold programs when evaluating faithfulness, to not conflate faithfulness with parser accuracy.
3.1 Measuring faithfulness in Visual-NMN
Modules in Visual-NMN provide for each bounding box a probability for whether it should be a module output. To evaluate intermediate outputs, we sampled examples from the development set, and annotated gold bounding boxes for each instance of find, filter, with-relation and relocate. The annotator draws the correct bounding-boxes for each module in the gold program, similar to the output in Figure 2 (top).
A module of a faithful model should assign high probability to bounding-boxes that are aligned with the annotated bounding boxes and low probabilities to other boxes. Since the annotated bounding boxes do not align perfectly with the model's bounding boxes, our evaluation must first induce an alignment. We consider two bounding boxes as "aligned" if the intersection-over-union (IOU) between them
exceeds a pre-defined threshold  $T = 0.5$ . Note that it is possible for an annotated bounding box to be aligned with several proposed bounding boxes and vice versa. Next, we consider an annotated bounding box  $B_{A}$  as "matched" w.r.t a module output if  $B_{A}$  is aligned with a proposed bounding box  $B_{P}$ , and  $B_{P}$  is assigned by the module a probability  $> 0.5$ . Similarly, we consider a proposed bounding box  $B_{P}$  as "matched" if  $B_{P}$  is assigned by the module a probability  $> 0.5$  and is aligned with some annotated bounding box  $B_{A}$ .
We compute precision and recall for each module type (e.g. find) in a particular example by considering all instances of the module in that example. We define precision as the ratio between the number of matched proposed bounding boxes and the number of proposed bounding boxes assigned a probability of more than 0.5. We define recall as the ratio between the number of matched annotated bounding boxes and the total number of annotated bounding boxes.  $\mathrm{F}_1$  is the harmonic mean of precision and recall. Similarly, we compute an "overall" precision, recall, and  $\mathrm{F}_1$  score for an example by considering all instances of all module types in that example. The final score is an average over all examples. Please see Appendix B.2 for further discussion on this averaging.
3.2 Measuring faithfulness in Text-NMN
Each module in Text-NMN produces a distribution over passage tokens (§2.2) which is a soft distributed representation for the selected spans. To measure module-wise faithfulness in Text-NMN, we obtain annotations for the set of spans that should be output by each module in the gold program (as seen in Figure 2 (bottom)) Ideally, all modules (find, filter, etc.) should predict high probability for tokens that appear in the gold spans and zero probability for other tokens.
To measure a module output's correctness, we use a metric akin to cross-entropy loss to measure the deviation of the predicted module output  $p_{\mathrm{att}}$  from the gold spans  $S = [s_1, \ldots, s_N]$ . Here each span  $s_i = (t_{\mathrm{s}}^i, t_{\mathrm{e}}^i)$  is annotated as the start and end tokens. Faithfulness of a module is measured by:  $I = -\sum_{i=1}^{N} \left( \log \sum_{j=t_{\mathrm{s}}^i}^{t_{\mathrm{e}}^i} p_{\mathrm{att}}^j \right)$ . Lower cross-entropy corresponds to better faithfulness of a module.
<sup>1</sup>The numerators of the precision and the recall are different. Please see Appendix B.1 for an explanation.
5597
4 Improving Faithfulness in NMNs
Module-wise faithfulness is affected by various factors: the choice of modules and their implementation (§ 4.1), use of auxiliary supervision (§ 4.2), and the use of contextual utterance embeddings (§ 4.3). We discuss ways of improving faithfulness of NMNs across these dimensions.
4.1 Choice of modules
Visual reasoning The count module always appears in NLVR2 as one of the top-level modules (see Figures 1 and 2). We now discuss how its architecture affects faithfulness. Consider the program, count.filter[black](find[dogs])). Its gold denotation (correct count value) would provide minimal feedback using which the descendant modules in the program tree, such as filter and find, need to learn their intended behavior. However, if count is implemented as an expressive neural network, it might learn to perform tasks designated for find and filter, hurting faithfulness. Thus, an architecture that allows counting, but also encourages descendant modules to learn their intended behaviour through backpropagation, is desirable. We discuss three possible count architectures, which take as input the bounding box probability vector  $\pmb{p} \in [0,1]^{|B|}$  and the visual features  $\pmb{v} \in \mathbb{R}^{|B| \times h}$ .
Layer-count module is motivated by the count architecture of Hu et al. (2017), which uses a linear projection from image attention, followed by a softmax. This architecture explicitly uses the visual features,  $\mathbf{v}$ , giving it greater expressivity compared to simpler methods. First we compute  $\pmb{p} \cdot \pmb{v}$ , the weighted sum of the visual representations, based on their probabilities, and then output a scalar count using:  $\mathrm{FF}_1\left(\mathrm{LayerNorm}(\mathrm{FF}_2(\pmb{p} \cdot \pmb{v})\right)$  where  $\mathrm{FF}_1$  and  $\mathrm{FF}_2$  are feed-forward networks, and the activation function of  $\mathrm{FF}_1$  is ReLU in order to output positive numbers only.
As discussed, since this implementation has access to the visual features of the bounding boxes, it can learn to perform certain tasks itself, without providing proper feedback to descendant modules. We show in §5 this indeed hurts faithfulness.
Sum-count module on the other extreme, ignores  $v$ , and simply computes the sum  $\sum_{i=1}^{|B|} p_i$ . Be-
ing parameter-less, this architecture provides direct feedback to descendant modules on how to change their output to produce better probabilities. However, such a simple functional-form ignores the fact that bounding boxes are overlapping, which might lead to over-counting objects. In addition, we would want count to ignore boxes with low probability. For example, if filter predicts a  $5\%$  probability for 20 different bounding boxes, we would not want the output of count to be 1.0.
Graph-count module (Zhang et al., 2018) is a middle ground between both approaches - the naive Sum-Count and the flexible Layer-Count. Like Sum-Count, it does not use visual features, but learns to ignore overlapping and low-confidence bounding boxes while introducing only a minimal number of parameters (less than 300). It does so by treating each bounding box as a node in a graph, and then learning to prune edges and cluster nodes based on the amount of overlap between their bounding boxes (see paper for further details). Because this is a light-weight implementation that does not access visual features, proper feedback from the module can propagate to its descendants, encouraging them to produce better predictions.
Textual reasoning In the context of Text-NMN (on DROP), we study the effect of several modules on interpretability.
First, we introduce an extract-answer module. This module bypasses all compositional reasoning and directly predicts an answer from the input contextualized representations. This has potential to improve performance, in cases where a question describes reasoning that cannot be captured by pre-defined modules, in which case the program can be the extract-answer module only. However, introducing extract-answer adversely affects interpretability and learning of other modules, specifically in the absence of gold programs. First, extract-answer does not provide any interpretability. Second, whenever the parser predicts the extract-answer module, the parameters of the more interpretable modules are not trained. Moreover, the parameters of the encoder are trained to perform reasoning internally in a non-interpretable manner. We study the interpretability vs. performance trade-off by training Text-NMN with and without extract-answer.
Second, consider the program find-max-num (find[touchdown]) that aims to find the longest touchdown. find-max-num
2Top-level modules are Boolean quantifiers, such as number comparisons like equal (which require count) or exist. We implement exist using a call to count and greater-equal (see Appendix A), so count always occurs in the program.
5598
should sort spans by their value and return the maximal one; if we remove find-max-num, the program would reduce to find[touchdown], and the find module would have to select the longest touchdown rather than all touchdowns, following the true denotation. More generally, omitting atomic reasoning modules pushes other modules to compensate and perform complex tasks that were not intended for them, hurting faithfulness. To study this, we train Text-NMN by removing sorting and comparison modules (e.g., find-max-num and num-compare), and evaluate how this affects module-wise interpretability.
4.2 Supervising module output
As explained, given end-task supervision only, modules may not act as intended, since their parameters are only trained for minimizing the end-task loss. Thus, a straightforward way to improve interpretability is to train modules with additional atomic-task supervision.
Visual reasoning For Visual-NMN, we pre-train find and filter modules with explicit intermediate supervision, obtained from the GQA balanced dataset (Hudson and Manning, 2019). Note that this supervision is used only during pre-training - we do not assume we have full-supervision for the actual task at hand. GQA questions are annotated by gold programs; we focus on "exist" questions that use find and filter modules only, such as "Are there any red cars?".
Given gold annotations from Visual Genome (Krishna et al., 2017), we can compute a label for each of the bounding boxes proposed by Faster-RCNN. We label a proposed bounding box as 'positive' if its IOU with a gold bounding box is  $>0.75$ , and 'negative' if it is  $<0.25$ . We then train on GQA examples, minimizing both the usual denotation loss, as well as an auxiliary loss for each instance of find and filter, which is binary cross entropy for the labeled boxes. This loss rewards high probabilities for 'positive' bounding boxes and low probabilities for 'negative' ones.