SlowGuess/NAACL_2021_OCR huggingface dataset:
MULTIPROVER: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning
Swarnadeep Saha Prateek Yadav Mohit Bansal
{swarna, prateek, mbansal}@cs.unc.edu
We focus on a type of linguistic formal reasoning where the goal is to reason over explicit knowledge in the form of natural language facts and rules (Clark et al., 2020). A recent work, named PROVER (Saha et al., 2020), performs such reasoning by answering a question and also generating a proof graph that explains the answer. However, compositional reasoning is not always unique and there may be multiple ways of reaching the correct answer. Thus, in our work, we address a new and challenging problem of generating multiple proof graphs for reasoning over natural language rule-bases. Each proof provides a different rationale for the answer, thereby improving the interpretability of such reasoning systems. In order to jointly learn from all proof graphs and exploit the correlations between multiple proofs for a question, we pose this task as a set generation problem over structured output spaces where each proof is represented as a directed graph. We propose two variants of a proof-set generation model, MULTIPROVER. Our first model, Multilabel-MULTIPROVER, generates a set of proofs via multi-label classification and implicit conditioning between the proofs; while the second model, Iterative-MULTIPROVER, generates proofs iteratively by explicitly conditioning on the previously generated proofs. Experiments on multiple synthetic, zero-shot, and human-paraphrased datasets reveal that both MULTIPROVER models significantly outperform PROVER on datasets containing multiple gold proofs. Iterative-MULTIPROVER obtains state-of-the-art proof F1 in zero-shot scenarios where all examples have single correct proofs. It also generalizes better to questions requiring higher depths of reasoning where multiple proofs are more frequent.
Formal reasoning over explicit multi-sentence knowledge (Newell and Simon, 1956) has often
proved to be challenging (Musen and Van Der Lei, 1988), owing to the difficulty in creating logical forms from such sentences, thereby restricting the application of semantic parsers (Zettlemoyer and Collins, 2005; Berant et al., 2013; Berant and Liang, 2014). Thus, in a recent work, Clark et al. (2020) bypass the creation of intermediate logical forms and show that transformers (Vaswani et al., 2017) can act as "soft theorem provers" by answering questions over natural language (English) rule-bases, consisting of facts and rules. In order to reliably interpret these predicted answers, Saha et al. (2020) propose PROVER, a transformer-based model that generates the corresponding proof graph, thus emulating formal reasoning closely. Consider the two example rule-bases with two questions and corresponding proofs in Figure 1, where a proof is a directed graph consisting of the relevant facts and rules from the corresponding rule-base.
PROVER shows good single-proof generation accuracy but is designed and trained in a way to generate only a single proof for each question. This is not ideal because formal proofs are not always unique and there may be multiple correct ways of arriving at the answer. For example,  $Q_{1}$  and  $Q_{2}$  in Figure 1 have three and four correct proofs respectively. Hence, in order to enhance the human-interpretability of linguistic formal reasoning systems, it is desirable to develop methods that can generate multiple proofs, each providing a different rationale for the predicted answer. Such interpretable methods, while possessing the flexibility of operating over natural language, can also aid in verifying claims when constructing proofs from scratch is tedious or infeasible.
We find that PROVER (Saha et al., 2020), when trained on all proofs as independent training examples (Eq. 2) and extended to generate top- $p$  proofs during inference (Eq. 3), fails drastically, achieving a low proof precision of  $34\%$ . The subsequent proofs are often incorrect because it is not
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3662-3677 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: Diagram showing two rule-bases with rules, facts, questions, answers and all possible proofs. The first question has three correct proofs while the second question has four correct proofs. MULTIPROVER answers both questions correctly and also generates all the corresponding proofs accurately for each question.
trained jointly with all proofs and hence, is unable to exploit the inter-proof correlations and also does not learn the correct number of proofs for a question. Thus, we propose MULTIPROVER, a transformer-based model that can generate a set of proof graphs with appropriate cardinality for a given question. Since multiple proofs can be generated in any arbitrary order, we pose this task as a set generation problem over graphs and train MULTIPROVER jointly with a permutation-invariant Hungarian Loss (Zhang et al., 2019a,b) over all proofs.
A proof graph is generated through a node module which selects the relevant facts and rules as part of the proof and an edge module which determines the edges between the chosen nodes. Similar to PROver, we first enforce multiple structural constraints during training and inference to ensure that a generated proof is valid. Next, in order to generate a set of proofs jointly, we propose our first model, Multilabel-MULTIPROVER, a multi-label classification framework which performs implicit conditioning among the proofs and predicts  $p$  binary labels for each node and edge, denoting its presence or absence in each of the  $p$  proofs that we want to generate. It is efficient in terms of number of parameters and training time and also achieves a better proof F1 than PROVER. However, the lack of explicit conditioning between the proofs is not ideal because a question with multiple proofs often has certain common sub-graphs across the proofs. E.g., all the 3 proofs for  $Q_{1}$  in Figure 1 have the sub-graph  $\{F_{10} \to R_1\}$  common. Thus, in order to exploit these correlations which Multilabel-MULTIPROVER cannot capture explicitly, we further propose an improved variant of MULTIPROVER, named Iterative-MULTIPROVER, which generates appropriate number of proofs by stacking multiple node and edge encoders, each of which
generates one proof at each time step by conditioning on the previously generated proofs. This enables the model to better learn the correlations between multiple proofs for a given question. To capture the set-based nature of the task, we train MULTIPROVER using a permutation-invariant Hungarian Loss (Sec. 3.5), which solves an assignment problem between a set of predicted and gold proofs.
Empirical evaluation on synthetic and human paraphrased QA rule-bases (Clark et al., 2020) show that both of our MULTIPROVER models achieve a significantly higher proof F1 compared to PROVER while retaining the QA accuracy. Further, on a challenging hand-authored zero-shot dataset, where all examples have single gold proofs, Iterative-MULTIPROVER achieves state-of-the-art proof F1. It also generalizes better to questions requiring higher depths of reasoning with more multiple proofs. Overall, our contributions are:
- We address a new and challenging problem of generating a set of multiple logical proof graphs for reasoning over natural language rule-bases by proposing two set-based joint models, Multilabel-MULTIPROVER and Iterative-MULTIPROVER.
- Iterative-MULTIPROVER's joint training and explicit conditioning helps it to better learn the relative importance of rules and facts for a particular question and uncover common subgraphs across multiple proofs. Thus, compared to Multilabel-MULTIPROVER and PROVER, it is able to transfer well in zero-shot settings because it learns to assign a soft prior over the rule-base.
- Iterative-MULTIPROVER's conditional generation also enables it to generalize better to questions requiring higher depths of reasoning where the presence of multiple proofs is frequent.
1Our code and models are publicly available at https: //github.com/swarnaHub/multiProver.
The task of rule reasoning (Clark et al., 2020) is related to other recently proposed tasks on QA (Weston et al., 2015; Yang et al., 2018; Lin et al., 2019; Tafjord et al., 2019; Richardson et al., 2020) and NLI (MacCartney and Manning, 2014). However, most of these tasks require implicit reasoning rules as opposed to explicit ones and the focus is either on broad language understanding or on single rule application. Below we discuss MULTIPROVER's relation to multiple areas of NLP and ML.
Structured Explanations: There is useful previous work on developing interpretable and explainable models (Doshi-Velez and Kim, 2017; Rudin, 2019; Hase and Bansal, 2020; Jacovi and Goldberg, 2020) for NLP. Explanations in NLP take three major forms – (1) extractive rationales or highlights (Zaidan et al., 2007; Lei et al., 2016; Yu et al., 2019; DeYoung et al., 2020) where a subset of the input text explains a prediction, (2) free-form or natural language explanations (Camburu et al., 2018; Rajani et al., 2019; Zhang et al., 2020; Kumar and Talukdar, 2020) that are not constrained to the input, and (3) structured explanations that range from semi-structured text (Ye et al., 2020) to chain of facts (Khot et al., 2020; Jhamtani and Clark, 2020; Gontier et al., 2020) to explanation graphs (based on edges between chains of facts) (Jansen et al., 2018; Jansen and Ustalov, 2019; Xie et al., 2020).
Generating Multiple Outputs: Generating a set of proofs can be viewed as a task of generating multiple structured outputs (Prasad et al., 2014). Multiple prior studies focus on generating diverse unstructured texts (Gimpel et al., 2013; Dai et al., 2017; Xu et al., 2018; Raffel et al., 2020). which broadly span two categories - (1) using improved decoding techniques like beam search with intersibling ranking penalty (Li et al., 2016), iterative beam search (Kulikov et al., 2018), diverse beam search (Vijayakumar et al., 2018), and sentence codes (Shu et al., 2019), (2) varying the hidden representations or using multiple decoders (Dai et al., 2017; Jain et al., 2017; Shen et al., 2019). Our baseline, PROVER-top- $p$ , which extends PROVER to generate top- $p$  proofs during inference falls in the first category while MULTIPROVER falls in the second category, where the multiple node and edge encoders vary the node and edge representations for generating multiple proofs.
Machine Learning over Sets: Set-based ML models (Zaheer et al., 2017; Lee et al., 2018; Zhang et al., 2019a; Kosiorek et al., 2020) have a wide range of applications including generating multiple image captions (Vinyals et al., 2015), generating diverse translations (Cho et al., 2014; Bahdanau et al., 2015), enumerating rules in a logical inference system (Gao et al., 2019). Set problems are challenging because the number of valid solutions for a set of size  $n$  are  $n!$ , which increases faster than exponential in  $n$  and ignoring the set structure produces sub-optimal solutions (Zhang et al., 2019a). Thus, we use a set-based Hungarian Loss (Zhang et al., 2019a,b) for capturing the permutation-invariant nature of generating a set of proofs.
3.1 Task Description and Notations
The input to our task is a tuple of the form  $(\mathcal{C},\mathcal{Q})$  where  $\mathcal{C}$  is a rule-base context and  $\mathcal{Q}$  is the question. We want to predict a binary answer  $\mathcal{A} \in \{\text{True},\text{False}\}$  for the question and generate a set of proof graphs  $P = \{\mathcal{P}_1,\dots,\mathcal{P}_p\}$ , each of which provides a diverse rationale for the answer (see Figure 1). The context  $\mathcal{C}$  consists of a set of facts and rules, denoted by  $\mathcal{F}$  and  $\mathcal{R}$  respectively. Facts  $\mathcal{F} = \{F_1,\ldots F_f\}$  are unambiguous statements, while rules  $\mathcal{R} = \{R_1,\ldots R_r\}$  are logical statements, which can be used in conjunction with the facts to arrive at a logical conclusion. Each proof  $\mathcal{P}_i = (\mathcal{V}_i,\mathcal{E}_i)$  is a directed graph, with a set of nodes  $\mathcal{V}_i \subseteq \mathcal{N}$  and a set of edges  $\mathcal{E}_i \subseteq \mathcal{V}_i \times \mathcal{V}_i$  where  $\mathcal{N} = \mathcal{F} \cup \mathcal{R} \cup \{\mathbf{NAF}\}$  and  $k = |\mathcal{N}|$ . If a statement (E.g. "Anne is big") cannot be deduced from the context, then Negation as Failure (NAF) contains the negation of that statement (E.g. "Anne is not big"), which is considered true in a closed-world assumption. See appendix for more details of the syntax of proof graphs.
3.2 Baseline PROVER Model
PROVER (Saha et al., 2020) builds on top of RoBERTa (Liu et al., 2019) and consists of a question answering (QA) module, a node module and an edge module where the node and edge modules are used to predict a single proof graph. The input to RoBERTa is the concatenation of the facts, rules and the question. The QA module takes in the representation of the [CLS] token and predicts a binary label for the question. The node module computes the node embeddings  $\mathbf{N} \in \mathbb{R}^{k \times d}$
Figure 2: Plot showing the percentage of samples with  $p > 1$  proofs for different training datasets, DU0-DU5.
consisting of the representations of each fact, rule and NAF where  $d$  is the embedding dimension. The  $i^{th}$  row  $n_i$  of  $\mathbf{N}$  denotes the embedding of node  $i$ . A node classifier takes in these embeddings to output the node probabilities  $np_i \in \mathbb{R}^k$  for each fact, rule and NAF being present in the proof. The edge module computes the edge embeddings  $\mathbf{E} \in \mathbb{R}^{k^2 \times 3d}$  for every edge  $(i,j)$  through the function  $\phi(i,j) = [n_i; n_j; (n_i - n_j)]$  where ; is the concatenation operation and outputs probabilities  $ep_{i,j} \in \mathbb{R}^{k^2}$  of each edge being present in the proof. PROVER is trained using the joint cross-entropy loss over the QA, node and edge modules. The authors pose inference as a Integer Linear Program (ILP). Given a set of nodes and the edge probabilities from the trained model, the following global score over the edge probabilities is maximized, subject to multiple structural constraints  $\mathcal{S}$  that ensure the validity of a proof graph (like checking for graph connectivity).
$$
\underset {e _ {i, j} \in \{0, 1 \}, s \in S} {\arg \max } \sum_ {i, j, i \neq j} e p _ {i, j} * e _ {i, j} + (1 - e p _ {i, j}) * (1 - e _ {i, j}) \tag {1}
$$
Extending PROVER to Generate Proof-Sets: Since Saha et al. (2020) focus on generating one proof per question, they also train their model with one gold proof per question. For multiple proof generation, an obvious extension is to treat each proof for a question as a separate training example. Formally, for each sample  $l$ , given a context  $\mathcal{C}^l$ , a question  $\mathcal{Q}^l$ , an answer  $\mathcal{A}^l$  and a set of gold proofs  $\mathcal{P}_i^l$ , where  $i \in \{1, \dots, p_l\}$ , the extended training dataset can be defined as:
$$
\mathcal {D} = \bigcup_ {l = 1} ^ {L} \left\{\left(\mathcal {Q} ^ {l}, \mathcal {C} ^ {l}, \mathcal {A} ^ {l}, \mathcal {P} _ {i} ^ {l}\right) _ {i = 1} ^ {p _ {l}} \right\} _ {l} \tag {2}
$$
Once PROVER is trained with this dataset, during inference, we generate top-  $p$  proofs by first selecting the top-  $p$  node sets according to Eqn. 3 and then choosing the corresponding edge sets us
Figure 3: Multilabel-MULTIPROVER.
ing the optimization function in Eqn. 1.
$$
\underset {\mathbf {v} \in \{0, 1 \} ^ {k}} {\arg \max } \sum_ {i = 1} ^ {k} n p _ {i} * \mathbf {v} _ {i} + (1 - n p _ {i}) * (1 - \mathbf {v} _ {i}) \tag {3}
$$
The top- $p$  solutions of Eqn. 3 are  $\mathbf{v}^1, \ldots, \mathbf{v}^p$  which indicate a node's presence or absence in the proofs. Although simple, this approach has two major issues. First, the lack of coupling between the proofs can potentially confuse the model as there are multiple possible proofs for the same (question, context) pair. Second, inference is inflexible and always generates a fixed number of proofs for every example, thus leading to the generation of many incorrect proofs (Section 5.1). As shown in Figure 1, certain questions can have multiple possible proofs. Figure 2 demonstrates this phenomenon statistically – the datasets we experiment with (Clark et al., 2020) contain up to  $13\%$  of the samples with  $>1$  correct proof. Thus, in the light of PROVER's limitations, we propose two novel architectures of a proof-set generation model, MULTIPROVER.
3.3 Multilabel-MULTIPROVER
As described in the previous section, a desired property for generating a set of proofs is to have the proofs conditioned on each other as opposed to treating them independently. Thus, we propose Multilabel-MULTIPROVER (see Figure 3), which poses the problem of generating a set of proofs as a multi-label classification task over all the nodes and edges corresponding to the set of  $p$  proofs. Each training example is a tuple  $(\mathcal{Q}^l,\mathcal{C}^l,\mathcal{A}^l,\{\mathcal{P}_i^l\}_{i = 1}^{p_l})$
Figure 4: Iterative-MULTIPROVER.
consisting of a set of gold proofs  $\{\mathcal{P}_i^l\}_{i = 1}^{p_l}$  per example. It consists of a QA module, a node module and an edge module. Following PROVER (Section 3.2), we obtain the node representations  $\mathbf{N} \in \mathbb{R}^{k \times d}$  by mean-pooling over the constituent RoBERTa representations. These are then passed through a multi-label node classifier, which consists of two linear layers and produces the probabilities  $np_i \in \mathbb{R}^p$  of a node being present in the  $p$  proofs. The node embeddings  $n_i$  and  $n_j$  for a pair of nodes are transformed by the function  $\phi(i,j)$ , described in Section 3.2, to output the edge embeddings  $\mathbf{E} \in \mathbb{R}^{k^2 \times 3d}$ . We also have a multi-label edge classifier, which takes in the edge embeddings to generate the probabilities  $ep_{i,j} \in \mathbb{R}^p$  of an edge  $(i,j)$  being present in the  $p$  proofs. Lastly, a question answering module predicts a binary answer for the question. Following PROVER, during training, we mask certain impossible edges like fact to fact, rule to fact and non-nodes. Given the outputs from the three modules, we train our model jointly over all proofs using a set-based Hungarian Loss.
This model is advantageous because there is implicit conditioning between the proofs as all the proofs are generated in parallel from the same node embeddings and edge embeddings. Thus, it has no additional time or memory overhead while also generating proof-sets better than PROVER (Section 5.1). However, it suffers from two major drawbacks. First, since the proofs are generated in parallel, the model is trained by padding empty proof graphs. Hence for higher values of  $p$ , the model has to learn more empty proofs, which makes the
Figure 5: Plot showing the percentage of samples in DU5 with at least one common node, common edge or both between the proofs for varying number of proofs.
learning problem harder. Second, the proofs are not explicitly conditioned on each other. This motivates us to propose Iterative-MULTIPROVER.
3.4 Iterative-MULTIPROVER
As a motivating example for why explicit conditioning among proofs is necessary, consider the proofs for  $Q_{1}$  in Figure 1 where the sub-graph  $\{F_{10} \to R_{1}\}$  is common across all the proofs.  $F_{10}$  and  $R_{1}$  are essential for answering the question and hence conditioning on the previously generated proofs will help the model adjust the relevance of nodes and edges in the subsequent proofs. Quantitatively, we find that about  $75\%$  of the samples with 4 proofs have at least one node and one edge common across all the proofs (see Figure 5). Thus, we propose Iterative-MULTIPROVER (see Figure 4), which broadly consists of a base PROVER architecture, as in Figure 3 and an additional  $p$  node and edge encoders for generating a maximum of  $p$  proofs. The proofs are generated iteratively until an empty graph is generated to denote the end.
Base PROVER architecture computes the first level of node embeddings  $\mathbf{N}^1\in \mathbb{R}^{k\times d}$  and edge embeddings  $\mathbf{E}^1\in \mathbb{R}^{k^2\times d}$ . These are passed respectively through a node and edge classifier to generate the node probabilities  $np^1\in \mathbb{R}^k$  and edge probabilities  $ep^1\in \mathbb{R}^{k^2}$ , corresponding to the first proof. In the next iteration, two transformer encoders generate the node and edge embeddings corresponding to the second proof. Specifically, we condition the generation of the next node embeddings  $\mathbf{N}^2$  on the previous node  $(\mathbf{N}^1)$  and edge  $(\mathbf{E}^1)$  embeddings simultaneously. Conditioning on both is crucial because  $\mathbf{N}^1$  captures the relevance of nodes for the first proof, while  $\mathbf{E}^1$  contains information about the strength of the connections between these nodes. We condition  $\mathbf{E}^2$  only on  $\mathbf{E}^1$ , because the edge embeddings corresponding to the nodes predicted by
$\mathbf{N}^1$  are already updated in  $\mathbf{E}^1$ . Formally,
$$
\begin{array}{l} \mathbf {T} ^ {1} = W ^ {(1)} \mathbf {E} ^ {1} W ^ {(2)}, W ^ {(1)} \in \mathbb {R} ^ {k \times k ^ {2}}, W ^ {(2)} \in \mathbb {R} ^ {3 d \times d} \\ \mathbf {N} ^ {\prime} = [ \mathbf {N} ^ {1}; \mathbf {T} ^ {1} ] W ^ {(3)}, W ^ {(3)} \in \mathbb {R} ^ {2 d \times d} \\ \mathbf {N} ^ {2} = \text {T r a n s f o r m e r} (\mathbf {N} ^ {\prime}); \quad \mathbf {E} ^ {2} = \text {T r a n s f o r m e r} (\mathbf {E} ^ {1}) \\ \end{array}
$$
These next set of embeddings, when passed through the respective node and edge classifiers, predict the node probabilities  $np^2 \in \mathbb{R}^k$  and edge probabilities  $ep^2 \in \mathbb{R}^{k^2}$ , denoting the likelihood of their presence in the second proof. We repeat this process of stacking up node and edge encoders for generating a maximum of  $p$  proofs. Given the node and edge probabilities corresponding to each proof and a QA probability from the QA module, we train Iterative-MULTIPROVER jointly with all proofs using the Hungarian Loss, described below.
3.5 Permutation-Invariant Hungarian Loss
Unlike words in text generation, proofs can be generated in any arbitrary order. Consequently, computing cross-entropy loss between the  $i^{th}$  predicted proof and the  $i^{th}$  gold proof,  $i \in \{1, \dots, p\}$  will be sub-optimal. Thus, we use a permutation-invariant Hungarian Loss (Zhang et al., 2019a,b) which finds the most optimal assignment between the predicted proofs and the gold proofs such that the overall loss is minimized. Formally, the Hungarian loss  $\mathcal{L}_H$  and total loss  $\mathcal{L}$  are denoted as follows:
$$
\begin{array}{l} \mathcal {L} _ {H} = \min  _ {\pi \in \Pi} \sum_ {i = 1} ^ {p} C E \left(n p ^ {i}, y _ {n} ^ {\pi (i)}\right) + C E \left(e p ^ {i}, y _ {e} ^ {\pi (i)}\right) \\ \mathcal {L} = \mathcal {L} _ {Q A} + \mathcal {L} _ {H} \\ \end{array}
$$
where  $CE(.,.)$  is the cross entropy loss,  $np^i$  and  $ep^i$  are the respective node and edge probabilities for the  $i^{th}$  predicted proof while  $y_n^{\pi(i)} \in \{0,1\}^k$  and  $y_e^{\pi(i)} \in \{0,1\}^{k^2}$  are the respective true node and edge labels for the gold proof  $\pi(i)$ , where  $\pi$  is the most optimal permutation. The Hungarian Loss is implemented by first summing the node and edge cross-entropy loss matrices  $\mathcal{L}^n \in \mathbb{R}^{p \times p}$  and  $\mathcal{L}^e \in \mathbb{R}^{p \times p}$  respectively, each entry  $(i,j)$  of which corresponds to the proof loss between the  $i^{th}$  predicted proof and  $j^{th}$  gold proof (see Figures 3 and 4). Then we find the best assignment between the gold and predicted proofs through the Hungarian algorithm (Kuhn and Yaw, 1955). Our final loss sums the Hungarian proof loss and the QA loss.
3.6 Integer Linear Program (ILP) Inference
Following PROVER, we generate valid proofs during inference using an ILP, subject to multiple