Does Entity Abstraction Help Generative Transformers Reason?
Authors: Nicolas Gontier
Siva Reddy
Christopher Pal
Venue: Accepted by TMLR ()

Abstract: We study the utility of incorporating entity type abstractions into pre-trained Transformers and test these methods on four NLP tasks requiring different forms of logical reasoning: (1) compositional language understanding with text-based relational reasoning (CLUTRR), (2) abductive reasoning (ProofWriter), (3) multi-hop question answering (HotpotQA), and (4) conversational question answering (CoQA). We propose and empirically explore three ways to add such abstraction: (i) as additional input embeddings, (ii) as a separate sequence to encode, and (iii) as an auxiliary prediction task for the model. Overall, our analysis demonstrates that models with abstract entity knowledge performs better than without it. The best abstraction aware models achieved an overall accuracy of 88.8% and 91.8% compared to the baseline model achieving 62.9% and 89.8% on CLUTRR and ProofWriter respectively. However, for HotpotQA and CoQA, we find that F1 scores improve by only 0.5% on average. Our results suggest that the benefit of explicit abstraction is significant in formally defined logical reasoning settings requiring many reasoning hops, but point to the notion that it is less beneficial for NLP tasks having less formal logical structure.

Does Entity Abstraction Help Generative Transformers Rea- son?
Nicolas Gontier Quebec Artificial Intelligence Institute (Mila), Montreal, Canada Polgtechnique Montreal, Canada
gontiern@milln quebee
ServiceNow Research
Siva Reddy
siva. Eirreila quebee
Quebec Artificial Intelligence Institute (Mila), Montreal, Canada McGill University, Montreal, Canada
Facebook CIFAR AI Clwir
ServiceNow Research
Christopher Pal
christopher pitisoopher-pul@mmlla.. quebbe
Quebec Artificial Intelligence Institute (Mila), Montreal, Conada Polytechnique Montreal, Canada
Canada CIFAR AI Chair
ServiceNow Research
Reviewed on OpenReview: https: openreview. net/ ptt/formm? id= 9nhmKuLAWV
Abstract
We study the utility of incorporating entity lype abstractions into pre-trained Transformers
and test these methods on four NLP tasks requiring different forms of logical reasoning: (1)
compositional language understanding with text-based relational reasoning (CLUTRR), (2)
abductive reasoning (ProofWriter). (3) multi-hop question answering (HotpotQA), and (4)
conversational question answering (CoQA). We propose and empirically explore three ways to add such abstraction: (i) as additional input embeddings, (ii) as a separate sequence to encode, and (iii) an auxiliary prediction task for the model. Overall, our analysis demonstrates that models with abstract entity knowledge performs better than without it. The best abstraction aware models achieved an overall accuracy of 88.8% and 91.8% compared to the baseline model achieving 62.9% and 89.8% on CLUTRR and ProofWriter respectively. However, for HotpotQA and CoQA, we find that F1 scores improve by only 0.5% on average. Our results suggest that the benefit of explicit abstraction significant in formally defined logical reasoning sottings requiring many reasoning hops, but point the notion that is less beneficial for NLP tasks having less formal logical structure.
1 Introduction
Transformer language models (TLMs; Vaswani et al. 2017) have enabled rapid progress in natural language processing (NLP). When pre-trained large corpora (such as the web) to predict the next tokens set of masked tokens from input sequence, TLMs capture linguistic knowledge (Peters al. 2018 Goldberg 2019: Tenney et al. 2019b). and yicld state-of-the-art performance on many NLP tasks with little to no lask supervision (Devlin et. al. 2019 Radford el. al. 2018 2019 Brown et al. 2020). However, is not clear these models can capture higher level knowledge such as reasoning skills that can re-used in
arbitrary contexts, and in ways that leverage the compositionality of those skills (Lake & Baroni 2018 Liška
et al.) 2018), something logical reasoners can do relatively well on smaller scale (De Raedt al. 2007
Fierens al.] 2015). Simple compositional tasks such SCAN [aake & Baroni 2018 CLUTRR
(Sinha

---

et al. 2019 and ProofWritter Clark et al. 2020 Tafjord et al. 2021 can help diagnose the compositional general generalization ion behavior of language models. Recent work on some of these datasets showed that TLMSs still struggle to learn reasoning strategies that can be re-used in out-of training distribution settings (Lake & Gontier et al.) 2020).
Baroni 2018
Humans do abstraction to simplify reasoning and become very efficient. This is particularly true in mathe- matics when manipulating variables instead of numbers. Manipulating abstract concepts allows humans to generalize knowledge across domains. If we look at how logical reasoners operate, we find that they have important abstraction component (going from grounded entities to higher level concepts) before logical reasoning can start (De Raedt et al. 2007). Going from an original text sequence to its higher-order meaning is an important part of the NLP pipeline (part of it being entily type tagging). Similarly in mathematios, the introduction of generic variables allows to progress in a logical reasoning process without keeping track of every (grounded) atomic entity. Overall, this idea that we call abstraction, seems to be an important part of logical reasoning. Recent work suggests that incorporating external knowledge about grounded en- tities could improve language models' abilities to reason and generalize {Ji et. al. 2017 Zhang et. al. 2019: Moosavi et al.) 2020 Rosset et al. 2020). However the empirical effect of incorporating generic entity types remains unclear, especially with recent studios suggesting that pre-trained models alroady encode some of that linguistic knowledge in their parameters (Hewitt & Manning, 2019: Liu et al. 2019a Clark et al.) 2019
Tenney et al. 2019a). In this work, we study the effect of explicitly providing entity type abstraction in addition to the original imput to pre-trained Transformers.
We explore and evaluate different ways to incorporate entity type abstraction and observe that some methods are more effective than others. To construct the abstract representation incorporated into TLMs, we leverage entity type information given by fixed pre-trained models. This allows for automatic and reproducible data processing. In general, our approach is the following: given an input sequence, we use an entity tagger to label entity types in the sequence. We then use these labels to construct copy of the original sequence in which all entitics are replaced by their corresponding entity types. This new sequence can then be used as extra input (Sections 3.1 & 3.2 or as extra training signal to the model (Sections 3.3). In particular, we explore three different ways to augment pre-trained Transformers with this abstract knowledge:
1. by combining token embeddings from both the original and the abstract sequence before encoding (Soc- lion 3.1. (Figure 1 & D).
2. by encoding both the original and the abstract sequence and combining them before decoding the target output (Section 3.2h (Figure I & Il).
3. by adding a second language model head on top of the Transformer decoder to predict the abstract
sequence (Section 3.3 (Figure IP).
A series of controlled experiments on two synthetic datasets show that models having access to abstract knowledge about entity types yield better performance at inference time both when interpolating and ex- trapolating to unseen lengths of reasoning chains. Since no (non-synthetic) natural language dataset system- atically and explicitly requires long chains of reasoning, we used synthetic datasets in which we can control for the degree of compositional generalization required. helesth in order to understand if the benefits observed could also be applicable in more realistic settings, we ran a series of experiments on two question answering datasots requiring some degrees of multi-hop reasoning. Results on these more natural language datasets show that abstraction aware models are not signifieantly better than baseline models. We conclude that these "real-world" problems do not have strong enough logical structure benefit from the abstraction technique and that the pre-trained weights of large language models seem to be "enough" for such natural language tasks, confirming previous results (Goldberg) 2019: Tenney et al. 2019a).. It is only when tasked on logical problems that explicitly require reasoning depths unseen during training that abstraction becomes significantly beneficial.
Overall our work contributions are the following:
1. we introduce and compare empirically different ways to incorporate abstraction into pre-trained TLMs. 2. we show that incorporating abstract knowledge can significantly improve compositional generalization to unseen lengths of reasoning chains in multi-step reasoning tasks.
2

---

3. we show that abstraction aware models may not benefit much when language is more natural and less procedural.
We hope that our work will inspire future research in the field to look for simple inductive biases that can complement pre-trained models in their quest to achieve logical reasoning at scale.
2
Related Work
Augmenting neural language models with knowledge about entities has been a popular method to improve their functionality. Ji et al. (211)) trained an entity neural language model to predict sequences of entities with an LSTM (Hochreiter & Schmidhuber 1997). At each sampling step, they predict the next word alongside a categorical variable indicating the current token's 's entity ID. They obtained lower perplexity and better results on co-reference resolution and entity prediction tasks than a variety of baselines. Similarly, Rosset al. (2020) trained a GPT2 model (Radford et al.) 2019) by giving it access entity knowledge at the input level and as additional pre-training loss. Their model achieved better factual correctness benchmarks such LAMA (Petroni et al.) 2019), and performed better than a baseline GPT2 model in various question answering tasks. Inspired by this work and motivated by the goal of building better reasoning language models, we instead focus on the prediction of entity types rather than entity idenlifiers taken from a fixed list of entities. This allows our solution to be robust to new entities. In addition, we explore and compare different ways to incorporate the entity knowledge in an encoder-docoder architecture.
Besides entity knowledge, other types of explicit information has also been given to language models. Prior work by [Swayandipta et al. (2018); Eriguchi et al. (2017); Nädejde et al. (0077) tried incorporating syntax information into language models by introducing els trained to also predict syntactic information achieved stronger performances on various tasks such as PropBank semantics and Neural Machine Translation. Inspired by this work, we also introduce an auxiliary loss but to predict entity types, and with an application towards reasoning tasks. Also working with syntax information, /Sundararaman et al. (2019) incorporated POS tags into the input embedding of a BERT model. Results show improved BLEU score on machine translation and higher accuracy than baselines on the GLUE benchmark (Wang et al., 2018). Similarly, |Sachan et al. (2021) augmented pro-trained BERT model with syntax graph neural network in order to encode syntax trees. Their results show that the quality of the trees are highly tied to the performance boost observed. Levine et al. (2020) trained a BERT-like model to learn word senses. They gave their model access to WordNet supersenses at the input level and as an additional training loss. They achiove better performance than other basclines on the SemEval Word Sense Disambiguation Lask (Raganato et al., 2017]. Moosavi et al. (2020) propose to improve robustness to data biases by augmenting the training data with predicate-argument structures. They train a BERT-base model (Devlin et al. 2019) with PropBank-style semantic role labeling (Shi & Lin 2019) on MultiNLI (Williams) et al. 2018) and SWAG (Zellers et al. 2018) datasets. Their results show that incorporating predicate- argument structure in the input sequence (only during training) makes the model more robust to adversarial examples in MultiNLL. More recently, |Poaaaa et al.] (2021) extended a RoBERTa model (Liu et al. 2019b) with hypernym abstraction based on WordNet to evaluate the plausibility of events. Their model is able to better predict human plausibility judgement than other RoBERTa baselines. Although different in application, all these prior works leverage the general idea of explicitly giving more abstract knowledge to language models, hence showing how flexible and generic this strategy can be. We take similar approach with entity types, but in the hope of improving the reasoning skills of our baseline model.
auxiliary loss to the model. Results show that mod-
A flurry of recent work has also examined ways to augment TLMs with entities from external knowledge bases (hhang et al. 2019 Peters et al. 2019 Févry et al. 2020: Verga et al. 2020). However, most of the time, these solutions rely on external components such as knowledge graphs with pre-trained entity embeddings, and/or or an additional memory. While they often use entity linking as a way to perform co-reference resolution, they do not incorporate higher level of abstractions such as entity types like we do here.
3

---

Table 1: Different architectures for different abstraction strategies. (blue) the original sequence embeddingg Xs (green) is the embedding of the simplified sequence with entities replaced by their entity type tags, "ENC" is the '5 encoder, H. (blue) is the contextualized representation of sequence X, H. (green) is the contextualized representation of sequence Xs, "DEC" is the T5 decoder, and Y is the target sequence to predict.
ENC
DEC
X,
X
ENC
(a) emb sum architecture
(b) catb-cat architecture
X
ENC
X
ENC
H
X,
ENC
H.
X
ENC
enc-sum architecture
(d) enc-cat architecture
Win
x
Wabs
dec-loss architecture
3 Introducing Abstraction Inductive Biases
In this section we describe five different ways to incorporate abstraction into pre-trained encoder-decoder model. Given input sequence X, use spacy named entity tagger] to make simplified copy X, of the input. This is a more generic copy of X.
We run the spacy recognizer on X to extract entity tags such as PERSON, ORG, GPE, etc... For each entity type, we create n additional vocabulary entries (with randomly initialized embeddings) such as |PERSON_1, ..RSS.,,,E..EN..... n, ORG_1R numbered) entity tag to make the simplified sequence X,, If the same entity is present multiple times in X, each occurrence will be replaced by the same entity tag in Xs- If entity is found for token in X, the original token's text is kept in X. (e.g. "Bob Smith has cat that he loves. Bob also loves Alezandra." " would be transformed into " *PERSON 11 has cal that he loves. PERSON 11 also loves PERSON 3.").
ORG ORG_n..... ..... Every token in X then replaced by their (randomly
We select n greater than one as we believe that for some tasks it is important to differentiate between two entities of the same type. At the same time we select n to be much smaller than the total number of entities in the dataset, thus forcing abstraction. value such that each distinct entity within the same example gets a different entity tag. This is different for each dataset depending Appendix A If the same entity appears more than once in a single example, it will gel the same tag every time within that example. Since we re-use the same finite set of entity tags across all examples, each entity tag will be used for different entity tokens, thus after seeing many examples, entity tags of the same type will likely have a similar embedding. We discuss some result highlighting this phenomenon in Appendix B
particular, the hyper-parameter n is set to the smallest, possible
the number of unique entities per example. Individual values can be scen in
In order to better understand the influence of having multiple tags for the same entity type, we also ran experiments in which we set n 1, thus forcing all entities of the same type to be mapped the same embedding. However noticed both more variance and weaker performance of our models, so the rest of this work will focus on the n > 1 setting described above. Results can be seen in Appendix C
In the following subsections, we describe different strategies to incorporate X, into an encoder-decoder Transformer model.
3.1 Abstraction as an additional embedding
Our first strategy is to combine X, with X at the embedding level. To do that, we construct X, to be of the same length of X. If a spacy entity spans over multiple tokens (e.g. A"Aeex", " "andra", " "Smith" "is", "the". "wife", " "of", " Bob")), we copy its entity tag at each sub-token positions (e.g. [*PERSON 3", "PERSO 3".
Inttps:/ http:///spacc.../mooeeeeeee
4

---

"PERSON 3", "is", "the", "wife", "of", "PERSON_ 11").. For each token within both sequences either sum (emb-sum experiments) concatenate (emb-cat experiments) their respective token embeddings.
sum. In emb-sum experiments (Figure I if tokens do not have entity tags associated with them, we ignore their embedding to avoid summing the same embedding twice for non-entity tokens. We only sum embeddings of tokens that do have an abstract tag associated with them. This to ensure that we only modify pre- trained embeddings that correspond to entity tokens. This is done by masking out the tokens X, that are the same in X. The imput given to the model's encoder is then emb(X) + mask x emb(X,) + positional with emb() being the embedding matrix, mask defined the X # X, binary tensor, and positional being the regular Transformer positional embedding. This resembles the setting used by Rosset et al. (2020), however their knowledge--ware embedding comes from a sequence of entities from a dictionary lookup, rather than a sequence of entity types from a pre-trained tagger. The advantage of our method is that it is robust to unseen entities of the same type.
concat. In emb-cat experiments (Figure [p), if tokens do have an entity tag associated with them, we replace their embedding with generic (learnable) "<grounded>" token embedding. This ensures that all non-entity token embeddings gets modified in the same way compared to entity tokens. We eventually resize the concatenated embeddings with a learnable matrix W1 R2exe with being the model's embedding size. The input given to the model's encoder is then [emb(X); emb((.)) W1 + posilional with emb() being the embedding matrix, [:] defined as the concatenation operator, and positional being the regular Transformer positional embedding. In this setup the model has 2e² e more parameters.
3.2 Abstraction as an additional sequence encode
Our second strategy is combine X, with X the encoding level. To do that, we again construct X, be of the same length of X. Similarly as above, if spacy cntity spans over multiple tokens, we copy its entity lag al each sub-token positions. We then encode both X and X, with the same encoder weights to have two contextualized encodings: H and H, respectively. For each token within both sequences either sum (enc-sum experiments) or concatenate (enc-cat experiments) their respective contextualized encodings.
sum. For enc-sum experiments (Figure 1 1 the input given to the model's decoder becomes H + H Unlike in Section 3.1 in these experiments we do not mask any position because the encodings are contextualized over the entire sequence. All token representations were influenced by all other tokens bocause of the Transformer encoder attention mask. Thus even non-entity token representations were influenced by entity tokens.
concat. For enc-cat experiments (Figure I],, we introduce a learnable matrix W₂ E R with d being the model's encoding size in order to resize the concatenated encodings, similarly to Section 3.1 The input given to the model's decoder is then [H; H.] W2 with [t] defined as the concatenation operator. In this setup the model has 2d more parameters.
3.3 Abstraction as an auxiliary task
Our third strategy is to incorporate X, into the model with an additional cross entropy loss (dec-loss experiments). The model will now be tasked to predict both the target output Y as well as the abstracted input X,. To do that, we introduce second language model head W Wase Rddxoccu (with d being the model's decoder size) on top of the model's decoder, initialized to have the same weights than the original language model head Wim, and fine-tuned during training with second cross-entropy loss. The final model's 's loss is then the average between the two cross entropy losses:
IX. P(X,); log Wabs)):
0.5 *
X,
1=0
+0.5 *
P(Y); * log(softmax(Frtec Wim))j;
Y
5

---

with Hdec being the output tensor of the model's decoder. Note that X, and Y have different lengths ([xa) and |Y] respectively), which is why we differentiate indices between the two sums. By sharing the decoder weights (except for the additional language model head), we make sure that most of the network parameters are influenced by the additional cross-entropy loss. This acts as a regularizer and forces the model to "know" about entity types within its original parameters. In this setup, the model has * vocab more parameters. Figure I illustrates this strategy. While not the scope of our initial experiments, we evaluated the performance of this model to predict abstract sequences with its additional language model head. Curious readers can refer to Appendix B for more details.
4 Experiments
In this section we describe the experiments we ran on various datasets. We start with the CLUTRR bench- mark (Sinha et al. 2019) in Section 4.1 as controlled experiments in which we know how much compositional
generalisation required. We then test our models on the ProofWriter &Clark et al., 2020 Tafjord et al, 2021) dataset in Section 4.2 This allows to verify if entity type abstraction is beneficial in formally defined logical reasoning environments with simple language. We next report experiments the multi-hop question answering task HotpotQA (Yang et. al. 2018) in Section 4.3 This allows to Lest if entity lype abstraction is beneficial in two-hop question answering settings with more natural language which is by nature more noisy. Eventually, we also report results on the convorsational question answering task CoQA (Reddy et al.) 2019) in Section 4.4 This allows to test if entity type rastroction is beneficial in conversational seltings which the entity being discussed may be originally introduced much earlier in the conversation, thus requiring some entity linking before answering.
For all experiments in addition to baseline model fine-tuned without any abstraction knowledge. We used the AllenNLP library (Gardner et al.) (017)) with the HuggingFace transformers library (Wolf et al.) [201)] PyTorch implementation of T5-small with 16-bit floating point precision. Each experiment was run on tesla V100 32gb GPUs with early stopping and a patience of 10 epochs on the validation set (defined as a 10% split of the training set). We report all pypem-parameters and library versions in Appendix A for reproducibility
trained one model for each of the 5 different strategies presented in Section 3
purposes.
4.1 Compositional generalization with CLUTRR
Although synthetic, CLUTRR is used because it allows for controlled experiments in which we can clearly mcasure both interpolation and extrapolation performance of our model. We generated 390, 000 examples that were roughly split. 77/23 between training and lesting. Each example consists of a unique (non-cyclic) family graph. The goal of this task is to infer the type of edge (family relation) between two nodes (two entities) that are the further apart in the input graph. The bigger the graph, the more hops are required to infer the missing edge. We express each family graph, along with its question and answer in text using a simple "fe 1) is the [rel] of fe_ 2)" template. Some examples of /out put sequence pairs can be seen in Appendix D
We evaluate the generated answer accuracy. The answer is defined as the first sentence in the generated sequence. Since all answers during training were expressed using simple template, we inverse this template to extract the rél, ē2) triple from the generated answer. If the extraction fails, we consider the generated answer wrong. We then compare the extracted triple to the ground truth provided by the CLUTRR dataset. If the reference solution is (el, (el,rel, e2), we accept both (el, rel, e2) and (e2, inc rél, as valid solutions. with inv rel boing the inverse relation of rel. For instance, the inverse relation of "father" can be "son" or daughter", we accept both.
4.1.1 Testing Compositional generalization
We carefully divided train and test sets to force the model to generalize to both unseen graph sizes (i.e.: unseen reasoning depth) and unseen (el, rel, e2) triples. Specifically, the training set is made of graphs with 2, 4 and 6 relations between 3, 5 and 7 entities respectively, while the test set is made of graphs with up to
6

---

Table 2: Prediction accuracy on CLUTRR test set for all difficulty levels. Models have been trained on levels 4,
6 with only 9.58% of all the (el, rel, e2) triples present in the test set. Per-level performance is colored in shades of green for better visualisation. The red boxed area indicates test problems at depths unseen during training. CLUTRR abstraction
2
3
4
5
8
9
10
avg.
100%
84.4%
64.8%
61.5%
54.2%
56.6%
45.5%
42.6%
56.8%
62.9%((#0.18)
cmb-sum
100%
85.3%
74.6%
59.3%
64.3%
67.7%
65.1
58.9%
68.4%
71.5%(11.13)
emb-cat
94.6%
60.0%
35.6%
42.0%
40.2%
74.3%
78.2%
77.9%
80.6
64.8%((10.2)
enc-sum
100%
94.8%
86.9%
89.9%
85.6%
87.2%
85.1%
84.3%
85.5%
88.8%((4.005)
enc-cat
100%
86.1%
72.6%
70.2%
66.7%
69.1%
63.8%
61.9%
74.0%
73.8%((0.12)
dec-loss
100%
74.7%
64.9%
59.3%
56.2%
61.3%
52.8%
47.8%
61.3%
64.2%((+0.15
10 relations botween 11 entities. In addition, all possible entities and relations are seen during training but only 9.58% of (el, rel,rel, triples from the lest set are also in the training set. This small overlap makes the test set harder than originally designed and allows to analyse the compositional generalization capacity of our model.
We fine-tuned a T5-small model on 300, 000 training examples of levels 2, 4 and 6 and evaluate the model on 9 test sets of 10, 000 examples each for all levels from to 10. The level is defined as the number of edges (relations) in the graph. The higher the level is, the bigger the graph is, the further apart the two entities to link are, and the greater the number of hops required to answer the query is. Specifically, we test levels 2, and 6 for compositional generalization, levels 3 and 5 for interpolation, and levels 7, 8, 9 and 10 for extrapolation.
4.1.2 Results
Table 2 shows answer accuracy on each test set level for models trained with n = 20 Lokens per entity lype. As mentioned in Section g this helps keeping track of "who's who" in the abstracted sequence, which is important for solving a task such as CLUTRR. We see that the best model (enc-sum) strongly outperforms all other models with an average score of 88.8% compared to 62.9% for the baseline.
sum -vs- concatenation. We see that for both the embedding strategies (emb-cat & emb-sum) and the encoding strategies (enc-cat & enc-sum), summing represenlations together yields better performance than feeding their concatenation through a linear layer. One hypothesis is that in the emb-cat model, all pre- trained embeddings are modified by either a entity type token embedding or the "<grounded>" embedding, whereas the emb-sum model keeps most of its pre-trained embeddings unchanged. This result also suggests that sometimes, simpler methods are more ellective.
embedding -vs- encoding. From Table 2 we also see that, on average, processing the abstract sequence through the encoder yields better performance than only processing it through the token embedder. This is expected as more layers of the Transformer can process the abstracted sequence.
learning the abstraction. In the last line of Table 2 the abstraction is given as output to the model. This experiment tests if learning how to abstract along-side the original task helps the model. We can see that learning to predict the abstract sequence can indeed help generalize as the average score increases from 62.9% for the bascline to 64.2% for the dec-loss model. However it not the most effective method this case. We believe this is because we test longer reasoning lengths (up to 10 steps), making the input sequence much longer than what is seen during training (up to 6 steps). This is because in CLUTRR, to make long reasoning chains the input family story gets longer. This penalizes the dec-loss model since it is designed to generate the abstracted version of the input, regardless of its length.
Overall, we can see that abstraction aware models can all extrapolate better than the baseline model, sug- gesting that entity abstraction does help pre-trained Transformers to compositionally generalize. To verify that this is not just feature of the CLUTRR dataset, we perform the same analysis on another synthetic dataset in the next section.
7

---

Table 3: Prediction accuracy on different slices the ProofWriter D5 test set for all our models and the originally reported numbers by Cllrk et al. (2020). Models have been trained on depth DO, D1, D2. Models are trained in the "open-world" assumption (OWA), except for the original Clark et al. (2020) model which was trained the "closed-world" assumption (CWA). Per-depth performance is colored in shades of green for better visualisation. The red boxed area indicates test problems at depths unseen during training.
RoBERTa-large
ProofWriter
Clark et 2020
abstraction OWA
emb-sum OWA
emb-cat OWA
enc-sum OWA
enc-cat OWA
dec-loss OWA
CWA
Overall
83 X3.9%((+..2)
10.1%
90.9%(+0.07
90.9%(++000))
88. 08.4%
90. 00.1%((=.007
(18.07+0.077)
DO
100.0%
99.5%
99.2%
99.3%
98 0%
99.2%
90 90.4%
D1
98.8%
95.6%
93.5%
95.3%
89 89.5%
91.8%
95.1%
D2
88.8%
87.9%
81.0%
87.1%
75.3%
83.5%
86.6%
D3
71.1%%
83.7%
8-1.6%
85.9%
81.7%
85.2%
87.4%
D4
43.4%
77.3%
87.4%
82 2%
84.7%
84.1%
85.0%
D5
37.2%
70.0%
85.3%
74.8%
84.0%
79.6%
80.6%
4.2
Abductive Reasoning with ProofWriter
Similarly to CLUTRR, the ProofWriter dataset (Tafoord et al. 2021 is a collection of synthetic facts and rules with derived conclusions. The goal of this task is to infer the truth value of an unknown statement given a series of known facts and 1-hop inference rules. Each example is made of a different set of facts, rules, and an unknown statement. The model then has to predict if the unknown statement is " "True", " "False, or "Unknown" according to the input knowledge. Each fact and rule is expressed in simple templated language given by grammar. Some examples of input/output pairs can be seen in Appendix D
The dataset also provides the required chain of inference required to arrive at the final answer. The number of such 1-hop inference steps is considered the "depth" " of the example. For instance depth-0 (DO) example simply requires to see if the unknown statement. is present in the input list of fact or not; a depth-1 (D1) example requires to apply one inference rule to one fact to arrive at the answer; etc... The dataset contains examples of up to depth-5 (D5) inference chains. We test for compositional generalisation by training on examples of up to depth 2 reasoning chains (DO, D1, D2) and testing on examples for each depth from DO to D5.
We fine-tuned T5-small models on the official training and development set from the depth <= 2 data folder and tested it on the test set from the depth <= 5 data folder; consisting of 70,076 training examples and 20,030 testing examples. We trained one model for cach the 5 different strategics presented in Section 3 in addition to a baseline model fine-tuned without any abstraction knowledge. Unlike in all other experiments, for ProofWriter examples, we did not use the generic spacy named entity tagger because it did not support the entity types covered by ProofWriter examples. Instead, used the real abstraction labels provided by the grammar files and defined the following abstraction tokens: "PERSON", "ATTRIBUTT "ANIMAL", "RELTTION".
Table shows the prediction accuracy on different slices of the D5 test set for all our models. Although trained on an older version of the dataset ("closed-world" assumption CWA), we also report the original performance by Clark et al. (202)). While not directly comparable because of the different version of the dataset, we can still see that our T5- T5-small experiments all extrapolate (D3-155 better than the original RoBERTa-large model despite having less parameters. RoBERTa-large performs better at depths seen during training (D0-D2), which may be due to the model size being bigger (hence having greater capacity to model questions of previously seen depths), however, without abstraction, the larger model struggles to generalize to unseen reasoning depths, unlike our models.
Most importantly, if we compare with our "no abstraction" baseline model, Table also shows that our abstraction methods help extrapolate to unscen reasoning depths. While our bascline model performs 83. 83.7%, 77.3%, and 70% on examples from D3, D4, D5 respectively; all other abstraction-aware models extrapolate better, with the best overall model (dec-loss) performing 87, 87.4%, 85%, and 80.6% D3, D4, D5 examples respectively.
8

---

Table 4: Average test performance for models on HotpotQA, Average and standard deviation computed with
random seeds.
HOTPOTQA
ExactMatch
F1
Precision
Recall
no abstraction
54.7 (=0.01)
68.9 (#0.01)
72.4 (#0.01)
69.1 (#0.01)
emb-sum
54.3 (10.01)
68.4 (10.01)
72.0 (10.01)
68.6 (10.01)
emb-cat.
52.6 (+0.01)
66.5 (+0.01)
69.7 (+0.01)
67.0 (+0.01)
enc-sum
54.2 (10.01)
68.5 (4:0.11)
72.0 (100.1)
68.7 (40.01)
enc-cat
52.9 (+0.01)
67.2 (+0.01)
71.0 (+0.01)
67.3 (#0.01)
dec-loss
55.7 (+0.02)
69.8 (+0001)
73.3 (+0.01)
69.9 (+0.01)
We also note that, unlike in CLUTRR, in this dataset the depth of reasoning (D0-D5) not tied to the input length. In ProofWriter, all examples have similar average input length, regardless of the depth of reasoning required to predict the answer. Thus the dec-loss model is not penalized like it was the case in CLUTRR, which result in it being the best model overall.
Overall, we achieve new state-of-the-ar results on the ProofWriter dataset when trained only on examples from D0-D2. This is suggesting that entity abstraction does help pre-trained Transformers to composi- tionally generalize to unseen reasoning chains. One important thing to note however is that both CLUTRR and ProofWriter sentences are relatively simple to abstract: 100% of the entitics are correctly abstracted in both datasets. In the next sections we will see if explicit abstraction is still beneficial on more realistic but less formally defined tasks.
4.3 Multi-hop Question Answering with HotpotQA
In this section report experiments on the multi-hop question answering (HotpotQA) dataset (Yang al, 2018). HotpotQA contains natural language, making it more diverse and harder to get abstraction labels than CLUTRR. In addition, Ho has 2-hop inference chains both in its training and testing data splits, thus we are not able to test generalisation to longer reasoning chains. Nevertheless, we believe it is a good compromise between natural language and multi-hop reasoning.
Used in the distractor setting, each example consists of list of 10 Wikipedia paragraphs, a question that requires the model to combine information from two paragraphs, and the answer. Since concatenating all of the 10 paragraphs would result in context size much larger than what regular Transformer models allowed (512 or 1024 tokens), we instead only took the two golden paragraphs as context, plus the question. While this beats the original purpose of retrieving the useful paragraphs, we are not interested in achieving state- oh-the art on this benchmark. We are rather interested in using it to compare the usefulncss of our approach on a more natural multi-hop question-answering setup. An example of input/out sequence pair can be seen in Appendix D
Because the official test set is not public, used the official validation set models and fine-tuned T5-small model on 90% of the training set while keeping the remaining 10% our custom validation set for early stopping. We trained one model for each of the 5 different strategies presented in Section 3 in addition to a baseline model fine-tuned without any abstraction knowledge.
test set to compare
Table shows exact match, F1 scores, Precision and Recall on our test set. We can see from these results that the best of our model is the abstraction-awwacc dec-loss model (trained to predict both the answer and the input in its abstract form) with an Fl score of 69.8% against 68.9% for the baseline model n" abstraction" " row). However, the baseline model is a strong candidate and the abstraction does not always benefit the model depending on how it is incorporated. This may be due to the fact that entity abstraction labels are harder to predict on natural language, and that entity type abstraction may not be required in problems with little to no formal logical structure. We further discuss this in Section
In an effort to contextualize these results, we note that our models performance is slightly above the "Query Focused Extractor" (Nishida et al., (201)) performance of 53.86 Exact Match and 68.06 F1. At the time of writing, the SOTA model AA is the "From Fasy to Hard" model (Li et al. 2022 with Fxact
9

---

Table 5: Average test performance for models on CoQA, Average and standard deviation computed with random seeds.
CoQA
ExacttMatch
F1
no abstraction
66.0%(+0.001)
74.4%((#0.000)
emb-sum
65.7%((10.002)
74.2%((10.002)
emb-cat
63.8%((+0.00))
72.3%((+0.00))
enc-sum
65.8%(++0.00))
74.1%((+0.002)
enc-cat
65.2%(+00002)
73.8%((0..022)
dec-loss
66.4%((+0.001)
74.9%(#0.001)
Match of 71.89 and Fl score of 84.44. We note however that the test & training data our setting is slightly different, so these are not perfectly comparable results. In addition, we are more interested in evaluating the effect of entity abstraction by comparing our model variants.
4.4 Conversational Question Answering with CoQA
Eventually; motivated by the of generative model, we test the same abstraction strategy in a con- versational setting. For that, leveraged the conversational question answering datasct CoQA (Roddy et al., 2019). The task presented by this dataset is to understand a text passage and answer series of inter-connected questions in a conversation. The conversation aspect introduces follow-up questions that forces the model to keep track of what entity is currently boing referred to and to look back at previous interactions. Examples of input/output sequence pairs can be found in Appendix D The dataset does not always explicitly forces multi-hop reasoning steps, but it could still happen (í.e. see the last. CoQA example of Appendix D in which the model must perform a substraction between all subjects and the ones already mentioned). In addition, the conversational nature of this dataset often introduces co-reference step to be made before fetching the information from the paragraph in context. In this setting we will thus test if abstraction can help in this multi-step information retrieval procedure.
Similarly to HotpotQA, because the official test set is not public, we used the official validation set as our test set to compare our models and fine-tuned a T5-small model on 90% of the training set while keeping the remaining 10% as our custom validation set for early stopping. We trained one model for each of the 5 different strategies presented in Section 3 in addition baseline model fine-tuned without any abstraction knowledge.
Table shows the exact match and F1 score our test set. Although by a small margin, we can see that the best of our model is again the abstraction aware dec-loss model (trained to predict both the answer and the imput in its abstract form) with an Fl score of 74.9% against 74.4% for the baseline ("no abstraction" row). The baseline model is quite strong already and the benefit abstraction questionable in this case. We further discuss this result in the following section. As in previous experiments, the worst performing model emb-cat and one of the best is the enc-sum model (after dec-loss in this case).
In an effort to contextualize these results, note that our models performance is better than DrQA + seq2seq with copy attention (Reddy et al. 2019h and + (Yatskar) 2018) with an F1 score of 67.0 and 71.6 respectively. The next. closest model in the CoQA leaderboard is the FlowQA model (Huang et al. 2019) with an F1 score of 76.3. At the time of writing, the SOTA model on CoQA is RoBERTa+ KK (Ju et al.] 2019) with an Fl score of 90.9. We note however that the test data in our setting slightly different, so these are not perfectly comparable results. In addition, we are more interested in evaluating the effect of entity abstraction by comparing model variants.
5 Discussion
In this section we aim to analyse further the results presented above by comparing them and measuring some key characteristic about each datasets.
10

---

Let's first summarize the results from all experiments. Overall, in all datasets (except for CLUTRR) the best model on average was the dec-loss model, which is trained to predict both the target output sequence and the imput sequence in its abstracted form. As discussed previously, we believe that the reason why the dec-loss model did not perform as good as the other models in CLUTRR is because input sequence length is tied to the reasoning depth required to answer the question. Indeed, a question of level n (2 S n S 10) will have exactly n sentences in its input. Thus, during testing, the model must generate sequences of unseen lengths. Previous work showed that length generalization is common weakness of classical language models (Murray & Chiang 2018 Clark et al. 2020 Gontier et al. 2020 Anil et al. 2022; Press et al.) 2022. Overall, our results suggest that when input sequence length is relatively stable across all examples, the dec-loss abstraction strategy is beneficial to multi-step reasoning tasks.
We then investigate why results on the two "natural", less procedural tasks (HotpotQA & CoQA) do not yield strong conclusions like in the synthetic cases of CLUTRR & ProofWriter. The first major difference between these datascts that CLUTRR and ProofWriter are designed explicitly to test for compositional generalisation and reasoning extrapolation. In both datasets, the model is tested on examples of longer reasoning chains than what is observed during training. The language vocabulary is limited and the required reasoning depth is controlled. This is possible when working with tasks that are formally defined in a logical reasoning setting. On the other hand, HotpotQA and CoQA are designed from human written text (Wikipedia. for Hotpot, human conversations for CoQA). This natural setting implies two things. (1) Tt is harder to control the reasoning depth required: all HotpotQA examples require to combine two pieces of information in order to answer the question. CoQA examples require to solve long co-reference resolution chains, however the test set does not contain longer chains of reasoning.
(2) The language vocabulary is more noisy, making it harder to extract the useful information from piece of text. These distinctions suggest that entity type abstraction is mostly beneficial for formally defined logical tasks in which the model must reason at unseen depths during inference time.
A second differentiating factor between these datasets is the quantity and quality of entily lags provided to the model. In an effort to estimate the influence of the tagger accuracy on our results, compare each dataset in terms of the amount of en- tity tags they contain, as well as the accuracy of these tags. The reported performance of the entity tagger we used is 0.85 F To further validate this metric, for each dataset, we estimate (i) the percent- age of tokens tagged as entities, (ii) the correctness of the tagged entities (precision), and (ii) the amount of correctly tagged entities out of all the entities that should have been tagged (recall). We estimate these metrics by manual inspection of a random set of examples for each dataset and report the F1 metric in Table G We can see that the percentage of tokens tagged as entities is roughly the same across datasets, except for CoQA which has fewer entity tokens. This is likely due to the fact that a lot of entities are referred to by co-reference in conversational QA. The average quality of entity lags for Hotpoi and CoQA is in line with the reported 85% F1 performance of the tagger. However, we note that all entities in CLUTRR and ProofWriter are correctly labeled. This distinction suggests that the quality of the entity tagger can influence the benefits of performing entity type abstraction.
Table 6: Percentage of tokens being tagged as entities
and estimated F1 score of the tagger on cach datasct.
Dataset CLUTRR
Entity Tokens
F1
22.2% (+00119)
100% (=))
ProofWriter
36.0% (+0.036)
100% (+0)
HotpotQA CoQA
37.0%
(+00085)
88.5% (+00092)
17.7%
(+0.095) 88.4% (+0.085)
To further estimate this influence, we ran experiments the CLUTRR and ProofWriter benchmarks in which we added noise to the abstract labels, thus artificially simulating a weaker tagger. In particular, we modified 25%, 50%, and 75% of the entities by either replacing their entity type or by deleting the tag altogether. We report in Figure the average answer accuracy of our models across extrapolation reasoning levels of the test set for both benchmark dataset (Ivl 7-10 for CLUTRR and D3-D5 for ProofWriter). Overall, we can see that as the tagging noise increases, all models see their performance reduced and sometimes under- performing the baseline when the performance of the tagger is too weak. For instance once the noise reaches 50% or more, all models on ProofWriter and almost all models on CLUTRR (except enc-sum & enc-cat) become weaker than the baseline. This confirus that as the tagger accuracy decreases, the overall
http:///thnynllcom///coorwwwwg
11

---

CLUTRR
ProofWriter
75%
75%
RoBERTO
507
50%
0%
0%6
en0-cat 1004
enc-sum
020
Figure 1: Average answer accuracy on extrapolation reasoning levels for models trained with 0%, 25%, 50% and 75% entity tagger noise on CLUTRR and ProofWriter. The baseline model (no abstraction) is represented by a black dotted line. Previous work on Proo/Writer represented by a black solid line.
performance of the model does so to. In addition, it also shows that it is better for the model to not have additional complications (abstraction in this case) if the additional parameters needed for that are used on noise more than 50% of the time.
In an effort to better characterize the failure cases of our models on the more natural datasets we report in mopendixFmmddll predictions on various datasots. In particular, in which abstraction models answered correctly but not the baseline (no-abstraction) model and vice-versa. Overall, we observe that many examples in pot don't. require multiple reasoning hops, which can be additional source of noise beside the tagging accuracy.
(.. examples of HotpotQA
Overall, we believe three factors are influencing our results on the more natural tasks: (i) the effect of
a weaker entity tagger performance as shown in Table and Figure [ (ii) the depth of reasoning being relatively shallow and the same across training and testing sets as observed in Appendix D and F.2 (iii) the natural and conversational language being further away from a formal language. These factors result in "real-world" problems not having strong enough logical structure which can benefit from the abstraction tochnique. It is only when tasked on more logical problems explicitly requiring reasoning depths unseen during training that abstraction becomes significantly beneficial.
6 Conclusion
Conclusion. We presented various ways to incorporate abstract knowledge into Transformer Language Models. Focusing on entity types, this work evaluated model performance on reasoning tasks requiring compositional generalization and multi-hop reasoning. Overall our results demonstrate three things: (i) incorporating abstract knowledge significantly improves reasoning and compositional generalization in both interpolation and extrapolation when the environment is formally defined in a logical reasoning setting; (ii) different ways to incorporate abstraction yields different performance boosts: enc-sum and dec-loss are generally performing better than others; (iii) abstraction is not beneficial when the task at hand more natural, less procedural, and not requiring long reasoning chains. This last result is due to the noisy entity lagging from "off-the-shelf" taggers, and due to the nature of the task at hand.
Limitations. One limitation of our work is that the method we present requires annotated data which is not always available. Furthermore, this additional data procossing can take time and may not scale well to larger datasets if implemented naively. Overall, we did not find significantly longer training time for all approaches except for the dec-loss model that was training on average 2 times longer than the other methods. The most important factor in training time was the dataset used rather than the method used. Models trained on natural language datasets CoQA and HotpotQA took days to train while models trained on CLUTRR. and ProofWriter took few hours.
12

---

Future Explorations. One hypothesis that results from our work is the following question: could pre- Iraining Transformer models with additional abstraction data result in stronger performance when fine-tuned with abstraction data like we do in this work? Although we could not train from-scrutch a T5 model on its original C4 dataset (Raffel et al. 2020), we believe that augmenting C4 with abstract annotations like we do on a smaller scale and training T5 from scratch on this augmented dataset could potentially yield a stronger language model.
Another interesting future direction worth exploring would be the ethical benefit of incorporating explicit abstraction into large language models. Previous work showed that pre-trained language models can have some undesired socictal biases (Henderson et al. 2018 Shwartz et al. 2020 Bender et al. 2021). Although not explored in this work, we believe that giving abstract entity types like we do in this work could have a positive societal impact on language models, potentially alleviating some of these biases. For instance, through abstraction, a model can be exposed to female and male names both being " "PERSON"s and that a "PERSON" can equally be a "manager" or an "assistant" regardless of its gender. Similar exposure could be achieved with other abstraction lypes such as " "RELIGION", " "JOB", "COUNTRY", " "NATIONALITY" etc... Benchmarks such StereoSet [Nadeem et al. 2021) could be used to measure the beneficial impact that explicit abstraction can have.
Acknowledgments
The authors acknowledge support from ServiceNow Research for providing computational resources which were used to run the experiments in this work. The first author is partially funded by scholarship from the Fonds de Recherche du Quebee Nature et Technologic (FRQNT). We thank CIFAR for their support through the CIFAR AT Chairs program. We also thank NSERC and PROMPT for their support.
References
Cem Anil, Yuhuai Wu, Anders Andreassen, Ailor Lewkowyez, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. arXiv preprint arXiv:220700490 2022.
Emily M Bender, Timnit Gebru, Angelina McMillannMajorr and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference Fairness, Accountabilily, and Transporency, pp. 610 623, 2021.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprinl ar Xic:20000 14165, 2020.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning, What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackbozNLP: Analyzing and Interpreting Neural Networks for NLP,, pp. 276-286 286, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/ /v/61///11199422, URL hutps://accaattoology www.ooloooyooo.v/eeo. wrtg/19-4/222222
Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. In Christian Bessiere (ed.), Proceedings of the Twenty- -inth International Joint Conference on Artificial Intelligence, International Joint Conferences Artificial Intelligence Organization, 7 2020. doi: 10.24963 00.24963/ijcai.. (jcaii.2200/5333 2020/537. URL https://dei org/10. 2000.24963ijjcce 2020/5337 Main track.
IJCAI-20, pp.
Lue De Raedt, Angelika Kimmig, and Hannu Toivonen. Problog: A probabilistic prolog and its application in link discovery. In Proceedings of the Twentieth International Joint Conference on Artificial Intelligence, International Joint Conferences on Artificial Intelligence Organization, 2007.
IJCAT-07, pp.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pp. 4171-4186, Minncapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10.18633/v1/NIB 1423. URL
https://www aclweb. www..eeb.rr//aatoolooylll-9-1---
13

---

Akiko Eriguchi, Yoshimasa Tsuruoka, and Kyunghyun Cho. Learning to parse and translate improves neu- ral machine translation. In Proceedings of the 55th Annual Meeling of the Association for Compulational Linguistics (Volume 2: Short Papers), pp. 72 78, Vancouver, Canada, July 2017. Association for Compu- tational Linguistics. doi: 10.18653/vvi 1111186611111122112 URL
Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. Entities experts: Sparse memory access with entity supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natura Language Processing (EMNLP), pp- 4937-4951 Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020..mr eat::00mmmllmmmma. main.400. URL https://aclanntoolpgg org/202 emnlp-mane main. m000
Daan Fierens, Guy Van den Broeck, Joris Renkens, Dimitar Shterionov, Bernd Gutmann, Ingo Thon, Gerda Janssens, and Luc De Raedt. Inference and learning in probabílistic logic programs using weighted boolean formulas. Theory and Practice of Logic Programming, 15(3):358 401, 2015.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafford, Pradeep Dasigi, Nelson F. Liu, Matthew Pe- ters, Michael Schmitz, and Luke S. Zettlemoyer. Allennlp: deep semantic natural language processing platform. 2017.
Yoav Goldberg. Assessing bert's syntactic abilities. arXiv preprint Xix: 1001:990..229 2019.
Nicolas Gontier, Koustuv Sinha, Siva Reddy, and Christopher Pal. Measuring systematic generalization in neural proof generation with transformers. In Advances in Neural Information Processing Systems 33. Curran Associates, Inc., 2020. URL htttps://arxiv.org///
14786. pdff
Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gonin Nan Rosemary Kc, Genevieve Fried, Ryan Lowe, and Joelle Pineau. Ethical challenges in dala-driven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 123 129, 2018.
John Hewitt and Christopher D. Manning. A structural probe for finding syntax word representations. Tn Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4129-4138, Min- neapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 10.18653 (0.18653/11/N19-141 URL htap:ss///aclaattooogg.ogg////// 1419.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98)) 1997.
Hsin-Yuan Huang, Funsol Choi, and Wen tau Yih. FlowQA: Grasping flow in history for conversational machine comprehension. In International Conference on Learning Representations, 2019. URL https: //openreview htt.:ernnntttttooommmdd--ttttaaa
Yangfeng Ji, Chenhao Tan, Sebastian Martschat, Yejin Choi, and Noah A. Smith. Dynamic entity repre- sentations in neural language models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1830-1839, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10. 11151//1//11-11111
URL httt::///ccaanthoogggooo///////
Ying Ju, Fubang Zhao, Shijie Chen, Bowen Zheng, Xuefeng Yang, and Yunfeng Liu. Technical report on conversational question answering. arXiv preprint or Xrxi::1999, 10772, 2019.
Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to--sequeece recurrent networks. In ICML, pp. 2879 2888, 2018. URL http://proceedingg. mlr. 2ttr.eress/v001akella.a html
Yoay Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwarlv, Amnon Shashua, and Yoav Shoham. SenseBERT: Driving some sense into BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 6656-4667 Online, July 2020. Association for Computational Linguistics. doi: 10.11663/11/2220.aa 423. URL https://aclanthologyy org/
2020. aww.amalmm.....m... 423
14

---

Xin- Yi Li, Wei-Jun Lei, and Yu-Bin Yang. From easy to hard: Two-stage selector and reader for multi-hop question answering. arXiv preprint or arXi::205.. 1 1729, 2022.
Adam Liška, Germán Kruszewski, and Marco Baroni. Memorize or generalize? searching for a compositional rmn in a haystack. aiv preprint arXiv:1802.04467 2018.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 1073 1094, Minneapolis, Minnesota, June 2019a. Association for Computational Linguistics. doi: 10. (8.18653/ v1 1112-1112. URL https://acllatthollgg 0ww.:/9-rg/111911111
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint Xrxiv:1907. 1 1669, 2019b.
Nafise Sadat Moosavi, Marcel de Boer, Prasctya Ajic Utama, and Iryna Gurevych. Improving robustness by augmenting training sentences with predicate-argumeut structures. arXiv preprint or arXiv:2010.125: 2020.
Kenton Murray and David Chiang. Correcting length bias in neural machine translation. In Proceed- ings of the Third Conference on Machine Translation: Research Papers, pp. 212 223, Brussels, Bel- gium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1 (118653/v1//LL 111111116622 URL https:/acclantoolog..oom org hatgg//18--22222 2/1/
Moin Nadeem, Anna Bethke, and Siva Reddy. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1 /20211. acl-long.416 URL
Maria Nädejde, Siva Reddy, Rico Sennrich, Tomasz Dwojak, Marcin
Philipp Koehn, and Alexandra Birch. Predicting target language CCG supertags improves neural machine translation. In Proceedings of the Second Conference Machine Translation, pp. 68-79, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 2211853/v1/W117-22 URL https: //aclanthology..omm wwwp:///ooooo///7//7700/
Kosuke Nishida, Kyosuke Nishida, Masaaki Nagata, Atsushi Otsuka, Itsumi Saito, Hisako Asano, and Junji Tomita. Answering while summarizing: Multi-task learning for multi-hop QA with evidence extraction. Tn Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2335 2345, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18663/v1/P19-1222 URL
htw.aaattooooogooor///ps.// 1225
Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word em- beddings: Architocture and ropresentation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1499 1509, Brussels, Belgium, October-November 2018 Association for Computational Linguisties. Tiniaaatioonnnnnniiiiiiissss doi: 10. (0.18653/111D18-111 URL htttps://www.aclweebb. www.::wwwwwwwwwwwwwolwloo.. arg//nnhology/
D18-1179
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. Knowledge enhanced contextual word representations. In Proceedlings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLLP), pp. 43-5 13-54, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. (0.18853/vl /1///11/1111 1-119-9-100 1005. URL
http////latttooooyoorrr////--o00 org/D19-1005
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexan- der Miller. Language models as knowledge bases? Tn Proceedings of the 2019 Conference on Empirioal
15

---

http:///arrrrooeor///////uuuuaee Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- gunge Processing (EMNIP-IJCNTP)), pp. 2463-2473, Hong Kong, China, November 2019. Association for 1250. URL http::/wwww.ccceee./rr/tttttt//a
Computational Linguistics. doi:
D19-1250
lan Porada, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. Modeling event plausibility with consistent conceptual abstraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistios: Human Language Technologies, pp. 1732-1743, Ouline, June 2021. Association for Computational Linguistics. doi: 10. URL https://acllatthooogg.goo
naacl maacn-main.naaaaaaaaaamaaaaaaana 138.
naacl-main. 138
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022. URL https:
//
Alec Radford, Karthik Narasimhan, Tim Salimans, and Tlya Sutskever. Improving language
understanding by generative pre-training, URL http:///S-uswweskt amazonaws. com/openat understanding paper. pdf, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(1), 2020.
Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. Word sense disambiguation: A unified evaluation framework and empirical comparison. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Lënguistics: Volume 1, Long Papers, pp. 99 110, 2017.
Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249 20099266,44 2019.
Corby Rosset, Chenyan Xiong, Minh Phan, Xia Song, Paul Bennett, and Saurabh Tiwary, Knowledge-aware language model pretraining. arXiv preprinl wxXi:2007.00655, 2020.
Devendra Sachan, Yuhao Zhang, Peng Qi, and William L. Hamilton. Do syntax trees help pre-trained transformers extract information? In Proceedings of the 16th Conference of the Europeun Chapter of the Association for Computational Linguistics: Main Volume, pp. 2647-2661, Online, April 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/20 eecl- 228. URL /TR//attpp::/aclaaohoollgy org/
2021. maac-mmann.. 228
Peter Shaw, Ming-Was Wei Chang, Panupong Pasupat, and Kristina Toutanova. Compositional generaliza- tion and natural language variation: Can a semantic parsing approach handle both? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Interna- tional Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 922-938, On- line, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2m5 2222//111/221//// lcl- URL https//acclattoology www.aatthooooooooo////.e/ acl-long. 75
Peng Shi and Jimmy Lin. Simple bert models for relation extraction and semantic role labeling. arXiv preprint arXie:1904.0522 2019.
Vered Shwartz, Rachel Rudinger, and Oyvind Tafjord. "you are grounded!": Latent name artifacts in pre- trained language models. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP). pp. 6850 6861, Online, November 2020. Association for Computational Linguistics. doi: 10. 1111663//v1/0020.mmmm-mmmmmm
URL htttp://acllattoooogg..r//////// wrgp:/o0.oog//0000 wttp//00.eemmpmmmmii...... emnlp-main.556
16

---

Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Proceedings of the 2019 Conference on Empirioul Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lenguage Process- ing (EMNLP-JJCNLP), pp. 4506-4515, Hong Kong, China, November 2019. Association for Computa- tional Linguistics. doi: 10.18653/v / 1458. BRLhhttss:: http://wwwwwwwww..../.. www.:awwww......ntooooooee otg//anthology//1--1445
Dhanasckar Sundararaman, Vivek Subramanian, Guoyin Wang, Shijing Si, Dinghan Shen, Dong Wang, and Lawrence Carin. Syntax-infused transformer and bert models for machine translation and natural language understanding. arXie preprint arXiv:1911.06155 2019.
Swabha Swayamdiptta, Sam Thomson, Kenton Lee, Luke Zeitlemoyer, Chris Dyer, and Noah A. Smith. Syntactic scaffolds for semantic structures. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3772-3782, Brusscls, Belgium, October- 2018. Association for Computational Linguistics. doi: 10.
1412.
Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. ProofWriter: Gencrating implications, proofs, and ab- ductive statements over natural language. In Findings of the Associalion for Compulaliional Linguistics: ACL-IJCNLP 2021, pp. 3621-3634, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1 /2021 fin.....nningg-ac.3311 URL https: ptttps://aclaanthooogooo www.gyy..rg/22222. f mandings-acl.311
lan Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 4593-4601, Florence, Italy, July 2019a. Association for Computational Linguistics. doi: 10.18653/v1/P11 1452. URL https: //aclanthology.oo wtw.aaoooooo.or/11111111900 org/P19-1452
lan Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Ben- jamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? prob- ing for sentence structure in contextualized word representations. In Proceedings of the 2019 International Conference Learning Representations, 2019b. URL https::/ htttg:://ppparrrrrseroee//ooorrs
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Process- ing Systems 30, pp.
Curran Associates, Inc., 2017. URL http://papers nips cc/paper/
7181-attention as-alll------ wwlllllllleeee yeu--eedd pdf
Pat Verga, Haitian Sun, Livio Baldini Soares, and William W Cohen. Facts as experts: Adaptable and interpretable neural memory over symbolic knowledge. ar preprint arXiv:2007.008) 2020.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue:
A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07/4 2018
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122, 1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 18653/ 15653/v1/N18 1101. URL 1101
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: Stale-of-the-art natural language processing, Arxiv, abs/ 191000990.7771, 2019.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 2369- 2380, Brussels, Belgium, October-November 2018 Association for Computational Linguistics. doi: 10. 10. N0.18653/v1/D1s-1 1259. URL httpps://acclatthooogg..ooogg -01./lo0/0or///1--55555-
17

---

Mark Yatskar. A qualitative comparison of coqa, squad 2.0 and quac. arXiv preprint arXiv:1809.1073 2018.
John M Zelle and Raymond Mooney. Learning parse database queries using inductive logic programming. In Proceedings of the thirteendh national conference artificial intelligence, volume 2, pp- 1050-1055, 1996.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 93 13-104, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1, /1/11/1001 URL httpes///ccaaattooooogoog//////-
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Compulational Lénguistics, pp. 1441 1451, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19- 1139. URL http:://occammboooogyyoo/////322
18

---

A Hyperparameters
We used the default T5-small hyperparameters from the HuggingFace library (Wolf ct al. 2019 We present in Table 7 below the library version we used and the model hyperparameters used for all experiments.
The only difference between each dataset the number same-type entity tags allowed in each input soquence n. These values were chosen be the smallest possible number while still being able to identify all the different entities in one sequence. For CLUTRR the maximum number of same-type entities was 20, for ProofWriter it 10, and for HotpotQA and CoQA it was 100. This is due to the larger context size and the diversity of natural language texts. Results on CLUTRR when = I can be found below in Appendix C
for CLUTRR for ProofWriter
20
10
for HotpolQA
100
for CoQA
100
AlPenNLP
version 2.2.0
Transformers
version 4.4.2
Spacy
version 2.3.5
batch size
256
16-bit floating point
True
dim embedding
512
dim feedforward
2048
dim key-value
64
dropout
0.1
max length
512
#of heads
8
#of layers
6
optimizer
AdamW
learning rate
I.00E-05
betas
[0.9, 0.999]
epsilon
1.00E-08
gradient norm
1.0
sampler
top-p
0.9
temperature
1.0
Table 7: Library version and model ppper-parameters..
B On the abstraction accuracy of the dec-loss model
After training the dec-loss models for each lask, one question we might ask is whether the model is able to correctly predict abstract tokens with its dedicated language model head.
After generating abstract sequences for all test sets, we mcasured that the model is correctly predicting entity types 100% of the time for CLUTRR and ProofWriter, and around 70% of the time for HotpotQA and CoQA compared to the types predicted by "off-tho-shelve" tagger.
One interesting finding though, is that the model is not consistent with entity numbers across the same example. While it can correctly predict the type of an entity ("PERSON" vs "LOCATION"), it is almost impossible to stay consistent with entity IDs of that type ("PERSON _1" vs "PERSON __3"____""_______ This suggests that at inference time, the distribution of abstract tokens belonging to the same entity type is very close to uniform, which also suggests that all these token embeddings are close each other.
19

---

C CLUTRR results when n =
To better understand the effect of the hyper-parameter n in our models, we ran additional experiments on the CLUTRR benchmark for different values of n. When n = 20 all entities in each example gets & unique abstract token ID. When n = 1, all entities of the same type are mapped to the same abstract token. order to keep track of which entity is related to which, token identity must be preserved. When n 1, the only way preserve token identity for the model use the original (non abstracted) sequence. We report below the average answer accuracy of our models CLUTRR for different values of The average is taken across all test levels from 2 to 10.
20
abstraction
62.9%
emb-sum
71.5%
75.0%
emb-cat
64.8%
30.4%
enc-sum
88.8%
85.3%
enc-cat 73.8%
36
dec-loss
64.2%
75.7%
Table 8: Average CLUTRR answer accuracy for models trained with 20 and token per entity tag,
We can see that the emb-cat and enc-cat models perform much worse in the setting, while the other models are relatively similar in both settings. This suggests that concatenating representations together and then performing matrix multiplication to resize the representation does not keep entity identity as well the other methods.
20

---

D Input Output examples
CLUTRR.
"question How is Anne related to Gary ? context
1v1.2 input:
Brett is Anne 's father Gary is a son to Brett ." "ansuer Anne has a brother named Gary ."
output:
"question: What the family connection between Patricia Timothy context : May the aunt Doris Patricia a daughter called Doris Timothy is Charles 's brother May has called Charles
1v1.4 input:
output:
"answer Timothy is the nephew of Patricia ."
ProofWriter.
"context: The cow is round. The cow needs the lion. The cow needs the rabbit. The con sees the lion. The con visits the rabbit. The lion round. The rabbit is kind. The rabbit visits the tiger. The tiger is big. The tiger kind. The tiger sees rabbit. The tiger visits the rabbit. If something is kind and it visita the rabbit then it is young. If something sees the tiger and it visits the lion then it sees the rabbit. If something is big and young then it sees the lion. If something visits the rabbit then the rabbit needs the lion. If something is big then it visits the rabbit. If something sees the tiger then it is rough. rabbit and is kind then rabbit needs the lion. something rough and kind then visits the lion. something needs the lion then it is big. question The tiger visits the rabbit."
Depth-0
input:
something visits the
Depth-0
"ansver True"
output:
"contert: Anne is nice. Charlie is blue. Charlie is furry. Charlie is green. Charlie is kind. Charlie is nice. Charlie red. Fiona is furry. Fiona is green. Harry furry. Harry kind. Harry nice. nice people are rough. someone is red and furry then they blue. Rough, kind people furry. If Charlie is furry then Charlie is nice. All furry people are nice. If someone is rough then they are kind. If someone is red and nice then they are blue. All furry people are red. question Harry is not blue."
Depth-2
input:
Depth-2
"answer : False"
output:
HotpotQA.
Which magazine was started first Arthur's Magazine or First for Women context : Arthur's Magazine (1844-1846) was an American literary periodical published in Philadelphia century. Edited by T.S. Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, and others. In May 1846 it merged into "Godey's Lady's Book". First for Women is woman's magazine published by Bauer Media Group in the USA. The magazine was started in 1989. It is based in Englevood Cliffs, New Jersey. In 2011 the circulation of the magazine was 1,310, 696 copies." "answer : Arthur's Magazine"
"question :
19th
input:
output:
21

---

CoQA.
"context The Vatican Apostolic Library, more commonly called the Vatican Library or simply the Vat, is the library of the Holy See, located in Vatican City. Formally established in 1475, although it is much older, is of the oldest libraries in the world and contains of historical texts. It has 75,000 codices from throughout history, as well 1.1 million printed books, which include some 8,500 incumabula. The Vatican Library is research library for history, law, philosophy, science and theology. The Vatican Library open anyone who can document their qualifications needs." " [...] "Only handful of volumes survive from this period, though
of the most significant collections
context:
research
very significant." "
input:
context above + "question When was the Vat formally opened?" "answer It was formally established in 1475.
output:
context above + "question : When was the Vat formally opened? answer It was formally established in 1475. question what is the library for? answer research. question for what subjects?" answer: history, and law."
input:
output:
context above "question When the Vat formally opened? It was formally established in 1475. question what is the library for? answer research. question for what subjects? answer history, and law. question what else ?" "philosophy, science and theology."
input:
oulput:
22

---

E GeoQuery Results
In this scction we report results of our different models on the GeoQuery (eele & Mooncy 1996) benchmark. In particular, we use two versions described by Shaw et al. (2021), focusing on compositional generalization: a train/test split based on query length and a train/test split based on Target Maximum Compound Divergence (TMCD), Each train and test set contains 440 examples.
We trained our models on both of these splits and evaluated the exact match of the predicted output of our models. Below
average results based on random seeds:
LENGTH split
TMCD split
no-abstraction
24.24% (4- 0.001)
36.06% (+- 0.001)
emb-sum
27.05% (+- 0.000)
34.47%
(+- 0.001)
emb-cat
15.91% (+- 0.002)
18.56%
(+- 0.001)
enc-sum
18.64%
(+- 0.002)
35.08%
(+- 0.001)
enc-cat
17.50%
(+- 0.000)
27.58%
(+- 0.001)
dec-loss
24.09%
(+- 0.000)
30.38%
(+- 0.001)
We can see that for the length split, one abstraction model performs better than the baseline (no-abstraction) but the TMCD split, models with abstraction are not better than the baseline. However, believe this dataset may not be a good candidate to showcase if abstraction is useful for 2 reasons:
First, all examples in the dataset have few number of entities (2 maximum):
entities per example
examples
percentage of the dataset
0
207
23.5%
1
667
75.8%
2
6
0.7%
Total
880
100%
This makes each abstract input very similar to the original input. Thus the abstract sequence provides very little information to the model.
Second, the dataset as a whole contains a few number entities. 12% of tokens per input are tagged as entities. This relatively small number compared to the other datasets as we can see below.
Fraction of tokens per input in each dataset identified entities by spacy NER:
Data
avg (+-std)
CLUTRR
0.22 (+- 0.019)
Proofwriter
0.36 (+- 0.036)
HotpotQA
0.37 (+- 0.085)
CoQA
0.18 (+- 0.095)
Geoquery
0.12 (+- 0.080)
We can say from these results that in order for the abstraction technique to be useful, the dataset must contain at least more than 2 entities per input sequence.
23

---

F Prediction examples
F.1 CLUTRR
In this section we present output examples of our models on the CLUTRR level test set.
Table 9: CLUTRR test set level output examples. Correct answers are in green and wrong answers are in red for better visibility.
input
abstraction
emb-sum
emb-cat
enc-sum
enc-cat
dec-loss
Jonathan is the father of Anne Anne is aunt to Stephanie Stephanie the danghter of Veronica How Veronica related to Jonathan ?
Jonathan
Jonathan
Jonathan
Jonathan
Jonathan
Jonathan
is Veronica father
Veronica 's father
Veronica 's uncle
Veronica father
Veronica
Veronica father
8 father
Stephanie is the granddaughter of Jonathan Bryant is Stephanie father Eric of Bryant How are Jonathan and Eric related to each other Joyce has daughter called Stephanie Betty grandmother to Stephanie. Paul has wife who is Joyce Paul, who is Betty ? Bryant is son of Jonathan Jonathan is the husband of Betty Craig is son of Bryant Who is Craig from the point of relation of Betty ?
Eric Jonathan grand- son
Eric is Jonathan grand- son
Eric is Jonathan grand- son
Eric is Jonathan 's grand- son
Eric is Jonathan grand- son
Eric is the grandson of Jonathan
Betty is Paul 's
Betty is Paul 's mother- in-law
Betty is Paul mother- in-law
Betty is Paul 's mother- in-law
Betty Paul 's mother- in-law
Betty is Paul 's aunt
mother- in-law
Betty has grandson who Craig
Betty has grandson who Craig
Craig the son- in-law of Betty
Craig is
Craig is Betty "s grandson
Craig is Betty 's son
Betty son
Stephanie a granddaughter to Betty Anne Bryant sister Bryant brother of Stephanie Who Anne from the point of relation of Betty
Anne Betty 's
Anne is Betty grand- daughter
Anne Betty 's grand- daughter
Anne is Betty 's grand- daughter
Anne is Betty 's grand- daughter
Anne is Betty grand- daughter
grand- daughter
score
4/5
5/5
3/5
5/5
4/5
4/5
score on 1vl.3 test reported in Table 2
84.4%
85.3%
60.0%
94.8%
86.1%
74.7%
Out these examples, the emb-cat model makes mistakes (linel & line3), the no-abstraction, enc-cat, dec-loss models make mistake (line4) and the emb-sun and enc-sum models make 0 mistake.
The emb-cat model mistakes (linel & line3) are due to the prediction of uncle/aume relations instead of motteee--neer--law respectively. It is interesting to note that these relations are similar in the sense that they all link one generation to the one just above.
It is also interesting to note that the example in which the ao-abstraction, enc-cat and dec-loss models fail (lino4), the emb-cat model on the other hand answers correctly. In this specific example the correct relationship "grandson" which links one generation generations below, however the mistakes made by the failing models only link generation below (son, son, son-in-law).
24

---

F.2 HotpotQA
In this section we present Ao examples in which all our abstraction models correctly answered the question but not the baseline (no abstraction) model.
Question: Are both Elko Regional Airport and Gerald R. Ford International Airport facts
located in Michigan?
"Elko Regional Airport (IATA: EKO, ICAO: KEKO, LID: EKO) formerly Elko Municipal Airport, is mile west of downtown Elko, in Elko County, Nevada.", "Gerald R. Ford International Airport (IATA: GRR, ICAD: KGRR, FAA LID: GRR) is a commercial airport in Cascade Township approximately 13 mi southeast of Grand Rapids, Michigan. The facility is owned by the Kent County Board of Commissioners and managed by an independent authority. The Federal
Aviation Administration (FAA) National Plan of Integrated Airport Systems for 2017-2021 categorized it as small hub primary commercial service facility."
Answer
baseline: yes embsum
embcat no
encsum no
enccat no
decloss no
This a ycs/no question with many acronyms and entities. The abstraction can be beneficial here in order to simplify the context.
Question:
What act for Innocent Records achieved Platinum sales and shares its name with a primary color in the color model?
facts
"Blue is the colour between violet and green the spectrum of visible light.
Human eyes perceive blue when observing light with wavelength between 450 and 495 nanometres. Blues with a higher frequency and thus a shorter wavelength appear more violet, while those with a lower frequency and a longer wavelength gradually appear more green. Pure blue, in the middle, has a wavelength In painting and traditional colour theory, blue is one of the three primary colours of pigments, along with red and yellow, which can mixed to form wide gamut of colours. Red and blue mixed together form violet, blue and yellow together form green. Blue is also primary colour in the RGB colour model, used to create all the colours on the screen of a television or computer monitor.",
470 nanometres.
"Innocent Records was pop record label created to cater to for EMI's Virgin Records more pop oriented acts. Following the success of the Spice Girls, Virgin Records decided delve into the pop market. In doing so they poached Hugh Goldsmith from multi-
RCA Records (famous for steering Take That's initial flagging sales, platinum act). They let him launch his Virgin Records offshoot. His first signing Billie Piper, followed by Martine McCutcheon, along with several dance acts Todd Terry to name one. The label continued to thrive well into the mid-2000s with Atomic Kitten and Blue achieving Platinum sales."]
Answer Blue
baseline: Atomic Kitten
embsum Blue
embcat Blue
25

---

encsum
Blue
enccat Blue
decloss Blue
This is a question with long paragraphs in the context, abstracting entities could help simplify the context to better answer the question. Nevertheless, note that we assume the model already knows that blue is a color (or that Atomic Kitten not a color), then the second paragraph is enough answer the question.
Question: William Cammisano was part which Mafia family?
facts
"William "Willie Rat" Dominick Cammisano Sr. (April 26, 1914 January 26, 1995) was a Kansas City, Missouri, mobster and enforcer for Nicholas Civella\'s 's Kansas City crime family.",
"The Kansas City Crime Family, also known as Civella crime family (pronounced ] ) is a Mafia family based in Kansas City, Missouri.")
Answer Kansas City crime family
baseline: Nicholas Civella embsum Kansas City Crime Family
embcat Kansas City Crime Family
encsum Kansas City Crime Family
enccat Kansas City Crime Family
decloss Kansas City Crime Family
Here the baseline model predicted the owner of the mafia family, which is also used to refer to the mafia group according the second paragraph. Note again, the question can be answered with the first paragraph.
Question: Who is older out of Bob Saget, the American comedian, Indian director S. Shankar?
facts
"Shankar Shanmugam (born 17 August 1963), credited mononymously Shankar, is an Indian film director and producer who predominantly works in Tamil cinema. He was identified by S. A. Chandrasekhar. Recognized for directing high budget films, he is also a pioneer of vigilante movies in Tamil. He made his directorial debut in "Gentleman" (1993) produced by K. T. Kunjumon, for which he was awarded Filmfare Best Director Award and the Tamil Nadu State Film Award for Best Director. He is the highest paid film-maker in India among his contemporaries.."
"Robert Lane "Bob" Saget (born May 17, 1956) is an American stand-up comedian, actor, and television host. His television roles include Danny Tanner on the ABC sitcom "Full House" (1987-95) and its Netflix sequel "Fuller House", and hosting "America\'s s Funniest Home Videos" from 1989 to 1997. Saget is also known for his adult-oriented stand-up routine.
also provided the voice the future Ted Mosby on the CBS sitcom "How I Met Your Mother" from 2005 to 2014."] Answer Robert Lane "Bob" Saget
baseline: Ted Mosby
embsum Robert Lane
embcat Robert Lane Bob
Saget
encsum Robert Lane
enccat Robert Lane " Bob " Saget
decloss Robert Lane " Bob " Saget
26

---

This question asks to compare two different entities of the same type. The baseline model answers with person's 8 name but not the correct one. Since there are lots of entities in this example, abstraction can be beneficial to simplify the context and select the correct person in the end.
Next, we present HotpotQA examples in which all our abstraction models incorrectly answered the question but not the baseline (no abstraction) model.
Question: What city does Paul Clyne and David Soares have in common? facts [
"Paul Clyne was the District Attorney of Albany County, New York from January 2001 through December 2004. A graduate Albany Law School, spent about 14 years as an assistant district attorney, before he was tapped by local politicians to replace the retiring District Attorney, Sol Greenberg. He was defeated for re-election by David Soares, first in the Democratic Party primary election in September 2004, and then in the general election in November 2004, in which he ran an independent line. After stint teaching at the New York Prosecutors Institute, he went into private practice as a criminal defense attorney in 2007, with an office in Albany, New York.",
"P. David Soares (born October 26, 1969, Brava, Cape Verde) the Albany County, N.Y. District Attorney. He is Democrat.")
Answer New York
baseline: New York
embsum Albany
embcat Albany
encsum Albany
enccat Albany
decloss Albany
In this example the abstraction models all answered 'Albany', which also valid answer In fact, the question asks for a city and not a state. addition, the second paragraph doesn't mention New York City, so Albany should probably have been the correct answer.
Question: what producer of The Real Housewives of Orange County also hosts "Watch What Happens Live with Andy Cohen"?
facts
"Andrew Joseph "Andy" Cohen (born June 2, 1968) an American talk show radio host, author and producer. Cohen hosts the Bravo nightly series "Watch What Happens Live with Andy Cohen". He is the first openly gay host of an American late-night talk show. After being head of development at Bravo for more than 10 years, Cohen resigned in November 2013. He continues to serve as an executive producer of "The Real Houserives" franchise.".
"The eleventh season of "The Real Housewives of Orange County", an American reality television series, is broadcast on Bravo. It aired June 20, 2016, until November 21, 2016, and is primarily filmed in Orange County, California. Its executive producers are Adam Karpel, Alex Baskin, Douglas Ross, Gregory Stewart, Scott Dunlop, Stephanie Boyriven and Andy Cohen."] Answer Andy Cohen baseline: Andy Cohen
embsum Adam Karpel
embcat Adam Karpel
encsum Adam Karpel
enccat Adam Karpel
27

---

decloss Adam Cohen
In this example the abstraction models all answer with the first entity listed as being producer of The Real Housewives, eventhough it is not the person who hosts What Happens Live. Note that this example contains the answer in the question, which may be confusing the abstraction models.
Question: What Actor whose birth name was Charles Dennis Buchinsky, was part Leslie Nielsen comedy?
facts [
"Charles Bronson (born Charles Dennis Buchinsky; Lithuanian: "Karolis Dionyzas Bučinskis" November 3, 1921 - August 30, 2003) was an American actor.", ., "Allan A. Goldstein (born May 23, 1949) is an American film director and screenwriter, perhaps best known for directing the Charles Bronson vehicle and the Leslie Nielsen comedy".
Answer Charles Bronson
baseline: Charles Bronson
embsum Allan A. Goldstein
embcat Allan A. Goldstein
encsum Allan A. Goldstein
enccat Allan A. Goldstein
decloss Allan Goldstein
In this example the answer can be answered directly from the first paragraph since we are asking for the name the actor born CCharles Denis Buchinsky'. Here the abstraction models all answered with the person from the sccond paragraph. This is probably due the fact that a two hop question, starting with entity 'Charles Dennis Buchinsky', and asking for person would first link the first and second paragraph with the entity CCharles Bronson', and then answer with the entity 'Allan A. Goldstein'
28


Reviews:
[{"review_id": "h8uc9ZXtfs", "reviewer": "Reviewer_THgz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n\nStrengths\n\n- The research question is interesting to the community of entity-centric reasoning and logical reasoning. Injecting entity-type info into neural language models has been tried in many places and there is no such detailed analysis.\n\n- The authors create a universal framework for comparing different injecting methods and different tasks for Transformer models. I believe the codebase for this evaluation framework can be used for other people to continue exploring other methods and evaluating more extensively. This is also the first time we can get a comprehensive understanding of the potential benefits of entity-type info for reasoning. \n\n- The paper is well-written and introduces the background knowledge, motivation, methods, and evaluation with a nice presentation for most of the parts.\n\nWeakness\n- I would like to see more analysis with real case studies on how the injection of entity-type info really changes the model behaviors, and thus enhance the reasoning performance. However, the main experiment section is mostly empirical results. Also, I think it is more beneficial if the authors can include some categorical analysis on how these behavior changes are related to the injected entity types, such that the paper can provide more insights for future research. \n\n- The overlap across the injection methods is not analyzed. Given the above suggestion, once there are more detailed, categorical case studies on the model behavior changes, this should be easier to analyze. Then, we can clearly see the similarities and differences between these injection methods. \n\n- The error analysis is also important. Before and after the injection of entity-type info, do the models correct their errors in an expected way? Or do they correctly predict answers to some new examples but make mistakes on even simpler questions? Without detailed error analysis and tracking of the model behavior, the empirical results themselves are not convincing to me at all. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper,  the authors investigate the research question that whether or not injecting the entity-type information can help reasoning tasks in NLP with generative Transformer models. They use three straightforward methods to inject the entity-type information: 1) use them as additional input embeddings, 2) encode them with additional individual sequences separately from the token sequences, and 3) create auxiliary tasks predicting the entity types for joint training. The authors found that entity-type info can only benefit the reasoning tasks that require/have formal logic constraints (e.g., CLUTRR and ProofWriter), while they are not very useful for tasks like HotpotQA and CoQA which do not have formal logic elements. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Section 4 can include more case studies and qualitative analysis. (See more in the Weakness part.) \n- Focus on tracking the model behavior changes before and after the injection of entity-type info. This can largely improve the analysis section and provide more insights to the readers. \n- Use more visualization to discuss the results so that the analysis and conclusions are more accessible to readers.\n", "broader_impact_concerns": "No. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KuJ5ZoMcUE", "reviewer": "Reviewer_Bo31", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1. The paper addresses an interesting and important research question: Given the difficulty of reasoning-based tasks for generative Transformer models, what sort of additional information can help improve them? More broadly, answering this question enables the community to better understand whether, and to what extent, features that are extracted by \"classical\" NLP pipelines (e.g. syntax, coreference and entity information, semantic information, etc.) are still relevant in the current era of large, end-to-end Transformer models, and if so, how we should best incorporate these types of information.\n\n2. The paper observes substantial improvements for tasks that involve formally defined reasoning settings, which can have downstream implications (e.g. writing new proofs and theorems, or validating existing ones, and maybe also for programming language & semantic parsing domains, etc.).\n\n3. The paper is generally clear, well-written, and easy-to-follow.\n\n4. The proposed approaches are simple and intuitive, which could lower the barrier for more widespread adoption.\n\n# Weaknesses\n\n1. There is a strong possibility that the more minor improvements within the NLP tasks are due to coreference resolution and/or entity type tagging errors, which can introduce and propagate noise into the Transformer. Even the current best coreference resolution and/or entity type taggers are imperfect within their domain (and also potentially bounded by a limited sequence length, e.g. 512 for BERT-based ones, and thus unable to condition on the entire relevant past information such as the whole document). It is also reasonable that their performance would be substantially worse when annotating out-of-domain datasets, which is the case here in this work. The paper makes an attempt to address this in the \"Discussion\" section on page 10, but in my opinion it falls short because: (i) it only examines annotation errors on the CLUTRR dataset, which is not exactly natural language (where entity tagging is much more difficult and ambiguous), and on which the model performs well already (unlike, CoQA or HotpotQA); (ii) it does not discuss or explore any techniques for mitigating these automatic annotation errors, such as tri-training, which has been widely used in the syntax & structure literature within NLP (Weiss et al., 2015; Choe & Charniak, 2016). \n\n2. The paper also does not try another, potentially better method that has been successfully used for injecting syntactic structure information into Transformers, and enables cross-attention between the word sequence information and the entity type sequence information (unlike the paper's second approach, that **separately** encodes the entity type sequence and the word sequence). This technique is done by simply combining the entity type information and the word in a single sequence, in the same fashion as the linearised syntactic tree model of Choe and Charniak (2016), which has been more recently explored in generative Transformers by Sartran et al. (2022), leading to improvements over the standard, word-level Transformers. A straightforward application to the entity type information case would be \"Alex PERSON_23 Andra PERSON_23 Smith PERSON_23 is [non-entity] the [non-entity] wife [non-entity] of [non-entity] Bob PERSON_11\". This would naturally result in a longer sequence, but given (i) its effectiveness in another related domain (syntax), and (ii) the ability of the model to do cross-attention between word and entity-level information, and (iii) overall simplicity & straightforward extension over the paper's methods, it is worth trying.\n\n3. It would be great to add statistical significance tests and/or reporting mean and standard deviation performance to account for noise in the reported performance, as the improvements for the NLP tasks are rather small.\n\n4. There is a lack of comparison with numbers from **external** work (e.g. external/previously published models that have comparable model size & training/test setups as this work), which would improve the credibility of the findings and the author's implementations.\n\n5. Although the approaches are simple and easy-to-implement, they are not particularly novel, and in fact have been used in other domains, e.g. the auxiliary task has been done in the case of syntactic scaffolds (Swayamdipta et al., 2018) and joint parsing and machine translation (Eriguchi et al., 2017); the input embedding approach has been explored --- also in the context of syntax --- by Sundaraman et al. (2019). \n\nMore minor presentational suggestions are provided in the \"Requested Changes\" section below.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# Summary of Contributions\nThe main contribution of this paper is investigating whether --- and to what extent --- injecting entity type information is useful for improving the reasoning capability of generative Transformers. To that end, this paper proposes three intuitive approaches for injecting entity type information: (i) as additional input embeddings; (ii) as a separate, extra sequence to encode (in addition to the standard word sequences); and (iii) as an auxiliary prediction task at training time, which --- unlike the other two approaches --- only requires the entity type information at training time, and not at test time. \n\nExperiments to assess the benefits of entity type information are done on two domains with formally defined logical reasoning settings (CLUTRR & ProofWriter), and on two reasoning-related natural language datasets (HotpotQA & CoQA). The main findings suggest that entity type information is much more beneficial for the first kind of tasks (i.e. for tasks with formally defined logical reasoning settings), whereas the same techniques only achieve minor gains for the reasoning-related natural language datasets. All in all, this finding contributes to a better understanding of where exactly a more explicit modelling of entity type information is beneficial for different types of reasoning tasks.\n\n# Overall Assessment\nOverall, the paper addresses an interesting and important research question: Given the difficulty of reasoning-based tasks for generative Transformer models, what sort of additional information can help improve them? This paper proposes a reasonable hypothesis that including entity type information can help improve reasoning ability, and then proceeds to validate this research hypothesis through 3 different methods on 4 diverse tasks. The paper is also generally clear and well-written. \n\nDespite the strengths, I have two major concerns regarding the paper in its current form. These issues are elaborated more in the strengths and weaknesses section below, but can be briefly described as follows. \n1. First, there is a strong possibility that the more minor improvements within the natural language tasks are due to coreference resolution and/or entity type tagging errors, which can introduce and propagate noise into the Transformer. Even the current best coreference resolution and/or entity type taggers are imperfect within their domain (and also potentially bounded by a limited sequence length), and their performance could be substantially worse for out-of-domain datasets. The paper makes an attempt to address this in the \"Discussion\" section on page 10, but it falls short because: (i) it only examines annotation errors on the CLUTRR dataset, which is not exactly natural language (where entity tagging is much more difficult and ambiguous), and on which the model performs well already; (ii) it does not discuss or explore any techniques for mitigating these errors (see point 1 of the weakness section below for some suggestions).\n2. The paper also does not try other potentially better methods that have been successfully used for injecting syntactic structure information into Transformer models, and enable cross-attention between the word sequence information and the entity type sequence information (see point 2 of the weakness section below).\n3. This is more minor, but having statistical significance tests and/or reporting mean and standard deviation of the performance on each task would improve the credibility of the findings, particularly on the natural language tasks where the gains are fairly small.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. **Critical**: A more detailed analysis about potential tagging errors, especially in the context of the natural language tasks (CoQA, HotpotQA) where entity tagging is highly ambiguous, prone to errors, and potentially requires long-range context information. This can be done by: (i) manually inspecting a small subset of the entity tagger's outputs, and getting an estimate of the tagger's performance; and/or (ii) using established techniques such as tri-training in order to get high-confidence tagging predictions.\n\n2. **Strongly recommended**: Try the approach of Choe and Charniak (2016); Sartran et al., (2022) that jointly aims to predict the linearised sequence of words and tags (see point 2 of the weakness above). This would enable cross-attention between the word information and the entity information, unlike the current approaches.\n\n3. **Critical**: Add comparisons with the performance of previously published models (ideally that have comparable model sizes and training setups with the proposed models). Furthermore, add statistical significance and/or report mean & standard deviations to all reported numbers. Both of these (external comparison and statistical significance testing) can improve the credibility of the findings.\n\n4. **Strongly recommended**: add the following links to the relevant prior work: (i) emergent reasoning ability of very large scale language models, e.g. Palm (Chowdhery et al., 2022); (ii) prior work on multi-task auxiliary loss for injecting syntax into LMs & other downstream tasks, e.g. (Swayamdipta et al., 2018; Eriguchi et al., 2017; Nadejde et al., 2017, and others); (iii) adding syntactic information to the input embedding of Transformers (Sundaraman et al., 2019).\n\n5. Clarification questions (**strongly recommended** to clarify in subsequent versions of the paper):\n- Comparing Tables 2 and 3, it seems that the best-performing variant in each table is different from one another. Do you have insights or conjectures on why this is the case?\n- Would an unsupervised learning approach that aims to **jointly** induce the entity type information and the standard generative modelling loss be a promising path for future work? This would eliminate the need to have a \"pipeline\" system that predicts the entity type information first, and thus mitigate the risk of the wrong entity type information being propagated to the Transformer model.\n- I find the following sentence at the end of page 3 difficult to read and interpret, please consider rephrasing it and/or provide illustrative examples: \"Although this may look similar to ... such ID across examples and across epochs.\"\n- In Section 3.2, it seems that the encoder is shared across the 2 different sequences (word sequences & entity type sequences). What happens if you use two separate encoders instead? This means less information sharing, but it might be compensated by an implicit ensembling effect and an increase in the model capacity (by virtue of having more parameters).\n\n6. Typos:\n- In page 3, \"in order to encode **syntaxt** trees\"\n- In page 8, \"etc...\" -> should there be only 1 period here?", "broader_impact_concerns": "No broader impact concern from my end.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pfboVnRT4Z", "reviewer": "Reviewer_7k7T", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* The paper is generally well written. There is a lot of detail on the methods and results (but some details are missing or presented a bit late, see weaknesses)\n\n* The research question is interesting, and the comparison of many different methods to use entity type information is interesting.\n\n* It is good that experiments on natural data were included in addition to ones on synthetic data. The authors have been very honest about the results.\n\n* Clarity: the methods to integrate the entity types were not entirely clear: initially I thought there are some intermediate pre-training tasks on unlabeled text, but I think what is done end-task finetuning while integrating entity type information in various ways – as features, or as an auxiliary prediction task.\n\n* Clarity:  In section 3.3 Abstraction as auxiliary task, if this is done at task fine-tuning time, Y and X_s will have different lengths. I believe the equation should be updated to indicate sums /averages of per-position cross-entropies , including introducing notation for the lengths.\n\n* Datasets used in experiments: there are some more natural datasets that have been the standard for compositional generalization evaluations, such as ones based on compositional splits of GeoQuery. It is hard to trust the validity of the conclusions given the selected datasets.\n\n* For the experimental conditions where most gains are seen (synthetic data), it seems that the entity types used convey extra knowledge about the synthetic data generation process, which is not available otherwise. Providing extra task information such as giving the generating grammar for SCAN to language models as an input has already been shown very useful before.\n\n* Ablations: for the auxiliary loss method, it would be interesting to compare to using an auxiliary task that uses self-supervised training on the task unlabeled data. Such task adaptive pre-training has been found useful before and it is would be interesting to see whether predicting types is more useful than just predicting text from the task, especially for the tasks based on natural data.\n\n* Computational expense: some of the methods increase the inference cost about 2x  – e.g. the method computing contextualized embeddings for two sequences. This is also the method that often performs well. A natural question would be whether moving to a larger pre-trained model would have a similar or larger effect.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "* The paper uses several methods to integrate information about entity types for four tasks, focusing on compositional generalization and reasoning – two are synthetic tasks and two are non-synthetic.\n\n* The methods are very similar to ones used to integrate different types of annotations in combination with pre-trained LMs before, for example entity linking annotations or pos tags/syntax. \n\n* There are substantial improvements on two synthetic tasks – CLUTRR and ProofWriter – requiring reasoning and compositional generalization. There are small to no improvements on the real NLP tasks of HotPotQA and ConQA, which also require some reasoning.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "* I suggest evaluation on a non-synthetic task of compositional generalization, such as semantic parsing on GeoQuery, also without providing to the system entity types derived from the logical form (without giving extra oracle information), or non-synthetic tasks of solving math word problems that require multiple steps of reasoning\n\n* I suggest discussing the computational cost of the methods to integrate entity types\n\n* I suggest addressing the clarity concerns outlined in the previous section\n\n* (minor) P.2 “ … Observe that some mehtods are more efficient than others” → do you mean effective?\n\n* (minor) Sec 3, first para: “Use existing tools such as spacy” → if you used spacy please say directly “We used the spacy named entity tagger …” Otherwise one may think you used a different tool that is similar to spacy.\n\n\n", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PcZ4JptAY7", "reviewer": "Reviewer_M3sQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The motivation for this work is solid, and the paper is overall written clearly such that it is easy to follow. \n\nWeaknesses: \nI see little novelty in the method. All architectures are fairly standard, embedding features and concatenating them, and summing them in multiple configurations. \n\nI'm unconvinced why this is about \"abstraction\" as they are learning a unique embedding for each entity, similar to the entity as experts (Fevry et al, EMNLP 2022) paper?  -- this was my misunderstanding, thanks for the clarification! \n\nThe pipelined nature of the proposed model poses a weakness — as it relies on the accuracy of entity tagger (which should be reported and understood in the domains they study). More generally, if the entity tagger was able to tag the entity means the information is present in the original text, which end-task application should also be able to exploit them directly?  \n\nThe experiments on Proofwriter seems broken to me — as they are using metadata (entity types from the grammar file which is _used_ to generate the data). It’s hard to claim state-of-the-art results in this setting, where their model is using more information than other baseline methods. \n\nThe entity types that are considered in this work are fairly coarse. I hope they provide at least a few motivating examples where such coarse-grained entity types can be useful for end tasks. It might be interesting to see if more fine-grained entity types can be helpful.\n\nThe entity indexing makes me question what happens to entities that are newly arising at test time. I'm also unclear how much of this is \"abstraction\", if the model is learning a separate embedding for each unique entities? I also find it hard to find that the number of same-type entities in hotpotQA and coQA is only 100..?? What is the dataset it is using exactly? \n\nThe results should be in context with existing work, not just a baseline of its own. Except for Table 3 (experiments on Proofwriter), all other results are not in context with prior work, which makes it harder to evaluate this proposed method’s effectiveness. \n\nWe should also compare the model capacity across different configurations, as far as I understand, the proposed model have increased parameter counts and that is potentially why it outperforms baseline. \n\nI’m not sure the gains in Table 4 are meaningful — some study on variation will be necessary. \n\nThe basic statistics such as the percentage of tokens tagged as entities, length of entity spans, etc, should be reported to understand the results better (esp. for natural dataset, HotpotQA and CoQA). More analysis should follow as well — for example, how does the entity tagger performance affect the performance of the final model? \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on presenting a new architecture for incorporating abstract entity types into NLP tasks. The paper aims for a general method, covering four NLP tasks (relational reasoning, adductive reasoning, multi-hop QA, conversational QA). They try multiple model variants for injecting entity abstractions. The results are somewhat mixed — while it shows improvements in more synthetic datasets, the gain almost dissppears when moving to the natural language dataset. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "See the weaknesses section above. Overall, I think the paper is pretty weak in its current form. ", "broader_impact_concerns": "I do not have broader impact concerns with this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents an analysis of different ways of incorporating entity information in transformer models either through architectural changes or loss modifications. Apart from the different ways of incorporating entity information, the authors also consider a range of evaluations benchmarks covering:
* logical reasoning (CLUTRR and ProofWriter) that stress-test compositional reasoning through synthetically-generated examples 
* multi-hop reasoning (HotpotQA) and 
* conversational reasoning (CoQA). 

The main findings are that entity abstraction helps for compositional reasoning but provides little to no benefit on realistic use cases that require perhaps more world knowledge. While all reviewers, myself as well, agree this is an important question, I believe there are a couple of things missing currently from the manuscript that would help complete the picture – I recommend this as a minor revision case.

The minor revisions should cover the following things:
1) As the method relies on the availability and performance of an entity-tagger, it is important to simulate performance and comparison to baseline with varying degrees of noise. 
2) While it is true that authors do not wish to obtain SoTA, it is difficult to assess the extent of improvements wrt previous work. It is important for each dataset to make an effort to contextualize these results as much as possible.
3) An important question is why the entity abstraction does not help in the more realistic datasets. The authors provide a couple of hypotheses, but it is important to perhaps make an effort to provide a more in-depth answer, in the context of this being an analytical study. For example, what happens if one looks at the predictions that are the same/differ with respect to the baseline, can some pattern be observed? Or, how does the accuracy of the tagger look like in a small subset of data, can this explain the difference? Even if a definate answer is not easy to be reached, perhaps some hypotheses can still be tested and eliminated.

Decision: Accept (poster)