SETI: Systematicity Evaluation of Textual Inference
Year: 2023

Abstract:
We propose SETI (Systematicity Evaluation of Textual Inference), a novel and comprehensive benchmark designed for evaluating pre-trained language models (PLMs) for their systematicity capabilities in the domain of textual inference. Specifically, SETI offers three different NLI tasks and corresponding datasets to evaluate various types of systematicity in reasoning processes. In order to solve these tasks, models are required to perform compositional inference based on known primitive constituents. We conduct experiments of SETI on six widely used PLMs. Results show that various PLMs are able to solve unseen compositional inferences when having encountered the knowledge of how to combine primitives, with good performance. However, they are considerably limited when this knowledge is unknown to the model (40-100 % points decrease). Furthermore, we find that PLMs can improve drastically once exposed to crucial compositional knowledge in minimalistic shots. These findings position SETI as the first benchmark for measuring the future progress of PLMs in achieving systematicity generalization in the textual inference.

heading: 1 Introduction
text: Natural Language Inference (NLI) determines whether a hypothesis follows from a premise (Dagan et al., 2013; Bowman et al., 2015; Williams et al., 2018) and has been explored for decades. Existing large pre-trained language models (PLMs) have shown remarkable performance on this task (Devlin et al., 2019; Raffel et al., 2019; Lan et al., 2020). To better assess the true capabilities of models to perform NLI, various associated tasks and benchmarks have been proposed. These works concentrate on exploring how models make predictions, e.g. by establishing ‘hard’ NLI datasets (Koreeda and Manning, 2021) or asking models to ‘explain’ their predictions through highlighting (Camburu et al., 2018), or by generating plausible
explanations (Bhagavatula et al., 2020). But little is known about how well such models are able to address compositional generalization.
Compositional generalization focuses on how to combine primitive units to predict larger compounds (Hupkes et al., 2020). A key property underlying compositional generalization is systematicity (Fodor and Pylyshyn, 1988), a hallmark of human cognition. Systematicity concerns the ability of (re)combining known constituents and composing rules. For example, humans who understand ‘red apple’ and ‘green train’ are able to conceptualize ‘red train’ by recombining ‘red’ and ‘train’ into a new concept. Similar effects of systematicity (generalization) can be studied in Natural Language Understanding (NLU) (Lake and Baroni, 2018; Kim and Linzen, 2020). Since PLMs have achieved results on par with human performance by fitting NLI training data (Wang et al., 2019), we aim to evaluate to what extent these models can master different types of systematicity in textual inference.
We propose a novel benchmark SETI (Systematicity Evaluation of Textural Inference), which extensively explores systematicity in NLI. SETI contains three interrelated yet independent tasks covering various types of systematicity: 1) Task1: primitives → compositions aims to evaluate if models can perform compositional inference if primitive constituents of the given inference task have been learned independently. 2) Task2: compositions → compositions aims to evaluate if models can perform novel compositional inferences if
4101
their constituents have been learned in other compositions. 3) Task3: primitives and compositions → compositions aims to evaluate if models can perform novel compositional inferences if one primitive constituent has been learned independently, while the other has only been encountered in compositions. SETI can be used to explore systematicity in NLI comprehensively since it considers all possibilities of how to construct a novel composition from known constituent types, derived from the ‘permutation and combination’1 theory acting between primitives and compositions. We introduce these tasks in detail in Section §3. To make the instantiations of systematicity covered in SETI easily accessible, we indicate three analogous visual tasks in Table 1. They test if models can understand: i) a novel compositional concept red apple – given the primitive concepts 2red and apple have been learned independently; ii) a novel compositional concept red train – given constituent concepts red and train have been encountered in compositions red apple and green train; iii) a novel compositional concept red train – given red has been learned independently, and train in the compositional concept green train.
To apply SETI in practice, we define veridical inference (Karttunen and Peters, 1979; Ross and Pavlick, 2019) and natural inference as primitives, and their combinations as compositions. For each systematicity task setting, we provide two instantiations: trivial and non-trivial, depending on the variety of instances presented to the model in training. While both settings fulfill the given task requirements, the non-trivial setting is more challenging because the compositional inference knowledge of how to combine constituents is not seen in training.
We evaluate six well-known PLMs on all SETI tasks. They show good performance in trivial settings, but inferior results in non-trivial settings, for all tasks. This indicates that models can generalize well to unseen compositions when constituents and compositional knowledge are known, while they are limited when they lack knowledge about how to compose constituents. Hence, we further explore whether, and to what extent we can enhance the systematicity capabilities. Our experiments indicate
1While permutations make sense in a setting that deals with grammaticality, this does not hold for inference, hence we do not consider permutation order for SETI.
2In the visual domain, primitive concepts such as colors and object properties seldomly occur independently of objects, instead they occur in composition with objects. Here, they are only used for task clarifications.
that all PLMs benefit greatly from being exposed to minimal doses of relevant compositional instances.
Our main contributions are as follows:
i) We introduce SETI (Systematicity Evaluation of Textual Inference), which to our knowledge is the first benchmark to comprehensively evaluate the systematicity capabilities of PLMs when performing NLI. ii) We provide datasets for three NLI challenge tasks that evaluate systematicity, with controlled splits for seen vs. unseen information. iii) We conduct experiments for six widely used PLMs. The results indicate that models generalize well to unseen compositions if they have previously acquired relevant compositional inference knowledge, but are limited when lacking such knowledge.

heading: 2 Related Work
text: Textual Inference Natural Language Inference (NLI) involves reasoning across a premise and hypothesis, determining the inferential relationship holding between them (Dagan et al., 2013; Bowman et al., 2015; Williams et al., 2018). As one of the major tasks for establishing Natural Language Understanding (NLU), NLI has been widely explored for decades. Recently, large pre-trained language models (Devlin et al., 2019; Raffel et al., 2019; Lan et al., 2020) exhibit remarkable performance on NLI tasks, on par with humans. To better explore the true NLI capabilities of models, various associated tasks and benchmarks have been proposed. Some work has probed NLI models by constructing hypothesis-only baselines (Glockner et al., 2018; Liu et al., 2020), finding that models capture undesired biases. McCoy et al. (2019); Zhou and Bansal (2020); Gubelmann et al. (2022) reveal that models rely on heuristics, e.g., lexical overlap, subsequence heuristics, etc. Nie et al. (2020); Chien and Kalita (2020) evaluate models in adversarial settings and show robustness improvement by training on additional adversarial data. Others focus on explainable NLI, such as highlighting input words that are essential for the label(Camburu et al., 2018), or generating plausible explanations (Bhagavatula et al., 2020). In this work, we focus on exploring the compositional generalization abilities of PLMs when performing textual inference.
Systematicity Systematicity is a crucial property of compositionality, which was first introduced in
cognitive science (Fodor and Pylyshyn, 1988) and recently formalized in Hupkes et al. (2020). It is the ability to make use of known concepts to produce novel concept combinations that have not been encountered before. Recently, systematicity has been widely explored in domains such as image caption generation (Nikolaus et al., 2019), visual attribute recognition (Misra et al., 2017; Li et al., 2020), question answering (Keysers et al., 2020; Liu et al., 2022) and semantic parsing (Lake and Baroni, 2018; Finegan-Dollak et al., 2018; Kim and Linzen, 2020; Zheng and Lapata, 2022). In this work, we focus on systematicity in the domain of textual inference.
Existing works that evaluate systematicity in textual inference only focus on one specific type. For example, Yanaka et al. (2021) evaluates systematicity by testing the transitivity of inference relations. Others conduct experiments on novel compositions involving specific linguistic phenomena, such as systematicity of predicate replacements and embedding quantifiers (Yanaka et al., 2020), systematicity when combining lexical entailment and negation (Geiger et al., 2020), and systematicity of quantifiers, negation and concerning the order between premises and hypotheses (Goodwin et al., 2020).
Compared to prior work, we propose a comprehensive systematicity evaluation benchmark SETI, which: i) covers the full spectrum of systematicity; ii) evaluates various PLMs; and ii) showcases how PLMs can overcome limitations in systematicity.

heading: 3 Reasoning Tasks for Systematicity
text: We now define primitive and compositional inferences and introduce three NLI systematicity tasks.

heading: 3.1 Primitive and Compositional Inferences
text: Among various textual inference types, we select veridical inference and natural inference as two primitive inference tasks3, since they can be flexibly scaled to compositional inferences. Table 2 shows relevant notation and corresponding examples of the two primitive inference types.
Veridical Inference Veridical inference is strongly determined by the lexical meaning of sentence embedding verbs. In the context of a veridical verb we can infer that the proposition it takes as complement is taken to hold true. By contrast, in the context of a non-veridical verb, we can not
3We adopt binary labels (entailment/non-entailment) by grouping contradiction and neutral as non-entailment.
infer that the proposition it takes as complement is taken to hold true. (Karttunen, 1971; Ross and Pavlick, 2019). PIver in Table 2 shows examples of both verb classes. The verb “realize” in the premise “Someone realizes that a man is eating pizza” is veridical in relation to the embedded proposition “A man is eating pizza”, since speakers cannot say the premise unless they believe the latter proposition to be true. In contrast, “hope” is non-veridical, since the premise “Someone hopes that a man is eating pizza” does not license the equivalent conclusion towards the hypothesis “A man is eating pizza”. In our work, we emphasize veridicality in verb-complement constructions and formulate their inference potential in an NLI setting, as premise-hypothesis pairs, as established by Ross and Pavlick (2019). Specifically, the premises of all veridical inference samples follow the template “Someone fv/fnv that s”, where fv and fnv represent veridical and non-veridical complement embedding verbs, respectively. We denote samples of entailed vs. non-entailed veridical inferences as fv(s) → s and fnv(s) ↛ s, respectively.
Natural Inference A pair of sentences is considered a true entailment if we are able to infer the hypothesis based on the premise. PInat in Table 2 shows examples. We categorize natural inference samples into two groups: 1) lexicallybased inferences typically build on lexical inference knowledge captured in lexical meaning relations, e.g., hypernymy boy → kid in “A boy is jumping into the water” → “A kid is jumping into the water”. 2) structure-based inferences involve structural changes, e.g., from active to passive voice and vice versa, as in “The detective follows the man” → “The man is being followed by the detective”. We restrict natural inferences to these two types to facilitate controlled data creation. We denote entailed and non-entailed samples from these two groups as: s lex−−→ s′, s Xlex−−→ s′ and s stru−−→ s′, s Xstru−−→ s′.
Composing veridical and natural inference To evaluate the compositional generalization ability of models, we construct compositional inferences CIver_nat (CI) by combining primitive veridical inference PIver and natural inference PInat, following Yanaka et al. (2021) (see Table 3).
For such compositions to be valid, the hypothesis of a veridical inference must match the premise of a natural inference. This matching condition serves as a crucial link to perform transitive inference. Table 3 shows how a compositional inference
Primitive Inference Types Examples (premise → hypothesis)
sample ‘He realizes a boy is jumping into the water’ → ‘A kid is jumping into the water’ is composed from PIver ‘He realizes a boy is jumping into the water’ → ‘A boy is jumping into the water’ and PInat ‘A boy is jumping into the water’ → ‘A kid is jumping into the water’. This reasoning process we denote as: fv(s) → s ∧ s → s′ ⇒ fv(s) → s′.
In this way, we construct four types of compositional inferences CI from primitive PIver and PInat inferences, where Boolean logical rules (Table 3, col. 3) decide the label of CI , i.e., whether it yields entailment or non-entailment. In case both veridical PIver and natural inference PInat resolve to True, CI yields entailment, given the Boolean logic rule True ∧ True → True (rule ①). By contrast, if PInat yields non-entailment, the compositional veridical inference CI will fail (rule ②). However, compositional inference with nonverdical verbs invariably yields non-entailment, no matter whether PInat resolves to True or False. This is again due to Boolean logic (rules ③, ④): False ∧ (True ∨ False) → False. In conclusion, the first two cases of CI are more complex, since models need to follow Boolean logic, while a model could exploit shortcuts and invariantly predict nonentailment with non-entailing verbs in PIver.

heading: 3.2 SETI Tasks
text: Having characterized the two types of primitive inferences we will use in our experiments, along with ways of composing them, we will now spell out i)
how to define increasingly difficult generalization tasks targeting systematicity, with ii) appropriate specifications of train and test settings, to guarantee proper assessment of a model’s generalizing capacities. Table 4 presents examples.
Task1: primitives → compositions aims to evaluate whether a model can perform a compositional inference CI if its (primitive) constituent inferences PIx and PIy, have been learned independently, while their combination is unseen in training. Hence, Train and Test sets (Dtrain|test) consist of instances e and ẽ:
Dtrain = {e | e ∈ PIx ∨ e ∈ PIy} Dtest = {ẽ | ẽ ∈ CI}
(1)
We select veridical inference and lexically-based natural inference as primitive inferences, and combinations of these two primitives as compositional inferences, as formally specified below:
PIx = PIver = {fv(s) → s, fnv(s) ↛ s} PIy = PIlex = {s lex−−→ s′, s Xlex−−→ s′}
CI = {fv(s) lex + −−−→ s′, fv(s) Xlex − −−−→ s′,
fnv(s) Xlex + −−−→ s′, fnv(s) Xlex − −−−→ s′}
(2)
Here, sentences (s and s′) of composed inferences CI are constrained to match sentences of their primitive constituents PIver∨lex. This is a trivial setting since the challenge is restricted to classifying compositional inference from seen primitive inferences.
However, overlaps of words between PInat and CI bear a risk of shortcuts (Sanchez et al., 2018). Hence, we also evaluate compositional inferences in a non-trivial setting, where sentences used in compositional inferences in Dtest are constrained to differ from sentences used in primitive constituents in Dtrain. This is doable if we guarantee that instances from PInat and CI share the same inference rules lexx. For example, we provide ‘A boy is jumping into the water → A kid is jumping into the water’ in PInat; and ‘Someone fv/fnv a boy is playing in the mud → A kid is playing in the mud’ in CI . In this way, models can retain the knowledge of PIlex by using the same inference rules, e.g., rule x : boy → kid, while we inhibit shortcuts by using different contexts in the test set.
Task2: compositions → compositions aims to evaluate if a model is able to predict unseen compositional inferences CItest whose constituting primitives have been encountered in other compositional inferences CItrain in training. Train and Test sets (Dtrain|test) consist of instances e and ẽ:
Dtrain = {e | e ∈ CItrain} Dtest = {ẽ | ẽ ∈ CItest}
(3)
We construct specific types of compositional training instances by combining veridical inference with lexical natural inference, and non-veridical with structural natural inference, see (4). To evaluate if models can generalize to novel compositions, we switch the constituents (primitive inference types) seen in training to unseen compositional inferences in testing. I.e., we evaluate veridical inference with structural natural inference, and non-veridical inference with lexical natural infer-
ence. CItrain and CItest are specified as:
CItrain ={fv(s) lex + −−−→ s′, fv(s) Xlex − −−−→ s′,
fnv(s) Xstru + −−−−→ s′, fnv(s) Xstru − −−−−→ s′}
CItest ={fv(s) stru + −−−−→ s′, fv(s) Xstru − −−−−→ s′,
fnv(s) Xlex + −−−→ s′, fnv(s) Xlex − −−−→ s′}
(4)
This is a trivial setting, given that four composition rules (①②③④ in Table 3) have been instantiated in the training samples. The challenge is restricted to correctly classifying novel compositions from known primitives.
To further explore if models can generalize to novel compositions based on unseen composition rules we propose a non-trivial setting. Here, a model must combine entailed veridical inference with entailed natural inference, and non-veridical inference with non-entailed natural inference. With this, only rules ① and ④ are instantiated by the training samples. In testing we confront the model with composition instances unseen in training, by switching constituents, so that we test for the unseen rules ② and ③: we compose entailed veridical with non-entailed natural inference, and nonveridical with entailed natural inference. CItrain and CItest are defined as:
CItrain ={fv(s) nat + −−−→ s′, fnv(s) Xnat − −−−→ s′}
CItest ={fv(s) Xnat − −−−→ s′, fnv(s) Xnat + −−−→ s′} (5)
We expected this to be an intractable challenge, since models are now required to classify novel compositions, where identical primitives have been encountered in training compositions, but the required composition rules of tested compositions are not instantiated in the training data.
Task3: Primitives and Compositions → Compositions aims to evaluate whether a model is
able to predict an unseen compositional inference CItest whose one primitive inference PI has been learned independently, while the other has only been encountered in a compositional inference CItrain in training. Hence, Train and Test sets (Dtrain|test) consist of instances e and ẽ:
Dtrain = {e | e ∈ PI ∨ e ∈ CItrain} Dtest = {ẽ | ẽ ∈ CItest} (6)
We could choose either veridical or natural inference as a primitive inference PI . Here, we select natural inference as the PI (veridical inference works analogously). Specifically, we construct CItrain by combining entailed veridical inference with lexically-based natural inference, and define structure-based natural inference as PI . To evaluate if models can generalize to novel compositional inference, we substitute the lexically-based natural inference component in CItrain with structurebased natural inference to form CItest instances, as stated below:
PI = PIstru = {s stru−−−→ s′, s Xstru−−−→ s′}
CItrain = {fv(s) lex + −−−→ s′, fv(s) Xlex − −−−→ s′}
CItest = {fv(s) stru + −−−−→ s′, fv(s) Xstru − −−−−→ s′}
(7)
This is again a trivial setting, given that the composition rules (①②) required in testing have been exemplified by training samples. That is, the challenge is restricted to correctly classifying novel compositions, where their primitives and the composition rules are known.
Analogous to Task2, we introduce a further nontrivial setting to evaluate if models can generalize to novel compositions that test for unseen composition rules. The primitive inference PI could be either veridical or natural inference.
Hence in one variant, we choose i) veridical inference (-Rver) as the PI , and construct the training compositions by combining entailed veridical with entailed lexical natural inference, while the primitive inference is non-veridical inference. For testing, we replace the veridical inference in training compositions with independent non-veridical inferences. This setting is defined below:
PI = {fnv(s) ↛ s′}
CItrain = {fv(s) lex + −−−→ s′}
CItest = {fnv(s) Xlex + −−−→ s′}
(8)
This setting should be challenging, since models are required to evaluate novel compositions that
correspond to the compositional rule ③, while they have only encountered rule ① in training.
As alternative variant ii), we choose natural inference (-Rnat) as the primitive inference PI . We construct training data for compositions by combining entailed veridical with entailed lexical natural inference, and define non-entailed lexical natural inference as the primitive inference. For testing, we replace the entailed lexical inference in training compositions with independent non-entailed lexical inferences. This is defined below:
PI = {s Xlex−−→ s′}
CItrain = {fv(s) lex + −−−→ s′}
CItest = {fv(s) Xlex − −−−→ s′}
(9)
This setting is challenging, since models are required to evaluate novel compositions according to rule ②, while having only seen rule ① in training.

heading: 4 Experimental Setup

heading: 4.1 Dataset
text: To evaluate the systematicity capabilities of PLMs on the series of SETI tasks we established above, we construct controlled datasets with instances chosen from established NLI datasets. For primitive inference: 1) veridical inference, we select 30 verbs (15 veridical, 15 non-veridical) that appear in both the MegaVeridicality2 (White et al., 2018) and the verb veridicality dataset of Ross and Pavlick (2019), as Yanaka et al. (2021) do (cf. Appendix.A for details). 2) natural inference, we extract instances from the SICK dataset (Marelli et al., 2014) that use lexical inferences s lex−−→ s′ where sentence pairs
are formed from lexical relations, e.g., synonymy and hyponymy. In addition, we select structural inferences s stru−−→ s′ where sentence pairs are derived from each other using the active-passive diathesis. Examples are shown in Table 2. For compositional inferences, we construct instances following §2.1. We combine premises fv/nv(s) from veridical inferences with hypotheses s′ from natural inference. Boolean logic rules are used to assign labels for these compositional inference instances.
Based on the constructed pool of inference data, we design three Compositional Generalization task datasets to evaluate the systematicity of PLMs. Specifically, primitive and compositional inferences data is divided for training Dtrain and testing Dtest in a controlled way, as outlined in Section §3. This ensures that the evaluated models will be exposed to specific types of inference instances in training, while being evaluated on unseen compositional inferences. That is to say, the testing data is out of distribution from the training data. In addition, we provide corresponding In-Distribution task datasets for comparison. Here, the data is divided into D
′ train and D ′ test by producing random
splits from D = Dtrain ∪Dtest. Hence, the evaluated models will, during training, encounter instances of the kind that will be presented in testing. In other words, the testing data is In-distribution of the training data. In-Distribution data makes it possible to confirm whether the failure of Compositional Generalization is due to intractable compositional inference tests or a lack of systematicity. Table 5 shows detailed data statistics for both configurations. For further details see Appendix B.

heading: 4.2 Evaluated Models
text: We choose six well-known PLMs for evaluation, of which three are masked language models (encoderonly): BERT (Devlin et al., 2019), RoBERTa (Liu
et al., 2019) and ALBERT (Lan et al., 2020); two are denoising autoencoder models (encoderdecoder): T5 (Raffel et al., 2020) and BART (Lewis et al., 2020); and one auto-regressive model (decoder-only): GPT-2 (Radford et al., 2019). We use standard accuracy as the evaluation metric.
For all PLMs we have chosen Large models, with checkpoints from the Hugging Face implementation (Wolf et al., 2020)4. We finetuned these models using the Adam Optimizer with batch size of 16. The maximum input token number is limited to 128. For each of the seven task settings, we perform five runs for each PLM, using different seeds. Further details are provided in Appendix C.

heading: 5 Experiments

heading: 5.1 Overview Results
text: Fig.1 illustrates the performance of six well-known PLMs on the SETI benchmark across two data configurations: Compositional Generalization and In-Distribution. Among seven different task settings, we find the test accuracy of PLMs in InDistribution to be close to 100% in most cases, with a drop to ≥ 80% in Task1 and Task2, nontrivial, but always stable across five rounds. This indicates that compositional inferences of various types are feasible for the evaluated PLMs if they have seen relevant instances in training. Compositional Generalization shows comparable results in trivial settings of Task2 (Fig.1.b) and Task3 (Fig.1.c), but inferior results for most of the remaining settings. This suggests that the evaluated PLMs are lacking systematicity capabilities when encountering unseen compositional inference problems, while achieving remarkable performance in In-Distribution by fitting training data.
4https://huggingface.co/models
Comparing trivial and non-trivial settings across three tasks, we find: 1) In Task1, the test accuracy of In-Distribution slightly decreases in the nontrivial setting, which confronts the models with novel contexts for PInat inferences within the compositional test cases CI . This shows that the nontrivial setting is more challenging. We also find that the performance of Compositional Generalization drops in the non-trivial setting. However, the encoder-only models ALBERT and RoBERTa outperform others substantially, showing strong systematicity generalization ability in both settings. 2) In Task2, the test accuracy of all generalizationtested models declines sharply to 50% in the nontrivial setting, no matter how well a model performs in the trivial setting. And this finding holds across different rounds of each PLM, indicating that novel compositional inferences are equally challenging for all evaluated models. 3) In Task3, generalization-tested models also show inferior results in the non-trivial setting, while non-trivialRver (Fig.1.f) is an exception (100% accuracy). This is expected since in this setting the PLMs can solve unseen compositional problems by exploiting superficial characteristics during training, rather than by generalization, i.e., the capability of systematicity. Specifically, the non-trivial-Rver task evaluates fnv(s) Xlex + −−−→ s′ given fv(s) lex +
−−−→ s′ and fnv(s) ↛ s′. In this task setting, nonveridical verbs fnv are only seen in non-veridical inference, which may lure models to predict nonentailment for compositional inferences containing non-veridical verbs, yet without considering the entailment class of the embedded lexical inference.
Across the different tasks, the evaluated PLMs show diverse performance for generalization testing. Task1 is almost solved by ALBERT and RoBERTa, highlighting that some models are capable of combining different primitive inferences (learned independently) in unseen compositional inferences. However, along with all other PLMs,
none of the two remaining Tasks can be reliably solved in the controlled, non-trivial “Compositional Generalization” setting: i) predicting unseen compositions, the components of which have been learned during training (Task2) and ii) determining a novel composition, where one primitive is learned independently, while the other has been encountered in a composition during training (Task3).

heading: 5.2 Few-shot Evaluation
text: We conclude from the results shown in Fig 1 that the evaluated PLMs are incapable of performing compositional generalization if they have not encountered crucial compositional inference knowledge during training. Hence, we aim to explore whether, and to what extent we could enhance the systematicity capabilities of the evaluated PLMs, by exposing them to small doses of relevant instances. Specifically, we select three non-trivial sub-tasks that expect models to solve compositional inferences without encountering the required inferential knowledge in training. For each such task, we construct a few-shot dataset Dfew where each sample (compositional inference, CI) is constructed following §4.1. Dfew and Dtest contain different data, i.e., Dtest ∩ Dfew = ∅. For each task, we evaluate few shot samples from 0 to 128, and each model is fine-tuned for three epochs. By doing so, we expect the models to learn the underlying compositional inference knowledge from the samples given in Dfew, so they can finally solve Dtest.
Figure 2 shows the few-shot experiment results. Across different tasks, we find that all evaluated PLMs benefit from few-shot samples that teach the model relevant compositional inference knowledge. In Task1, most PLMs show a significant performance increase with only four CI samples in Dfew. This finding is consistent with the fact that solving Dtest requires four different compositional rule types, as shown in definition Table 3. Similarly, Dtest from Task2 and Task3 require two and one
samples illustrating required, but previously unseen compositional inference knowledge, respectively. We find most evaluated PLMs in Fig 2.b to drastically improve their performance with only two samples, and in Fig.2.c with just a single sample. An exception is BERT, which requires more shots than the number of unseen inference cases.
The above experiment suggests that the evaluated PLMs can greatly benefit from few-shot settings to enhance their systematicity capabilities. It is compelling that the number of samples in Dfew needed to reach substantial task performance corresponds to the number of inference knowledge types required to make correct inference predictions, i.e., making it possible to evaluate novel compositions. It will be interesting to study how to identify potentially missing types of compositional inference knowledge for existing PLMs, and how to inject this knowledge in an efficient, data-free method.

heading: 6 Conclusion
text: We propose the first comprehensive systematicity evaluation benchmark, SETI, applied to Natural Language Inference. Experiments on six widely used PLMs show that they can distinguish novel compositions with known primitives and composing knowledge with high accuracy, but limited when lacking such knowledge. Moreover, we show that models can quickly acquire missing inferential knowledge for systematicity by being presented with unique samples representing each missing case of inferential knowledge, in a few-shot setup.

heading: 7 Limitation
text: SETI only considers veridical inference and natural inference (including both lexically-based inference and structure-based inference). However, our benchmark SETI can be flexibly extended to more varied reasoning patterns, such as negation, quantifiers, or others. In addition, we evaluate the systematicity capabilities of PLMs on semi-synthetic datasets, which are limited in language variance. Extending our benchmark on manually annotated compositional inference datasets might be a promising future work.
Recently, Hupkes et al. (2020) dissect the notion of compositionality and define five theoretically grounded tests for generalization, in a taskagonistic manner. Our work is limited to evaluating the systematicity of PLMs in textual inference. While the systematicity test is one of the most im-
portant tests, the remaining ones (e.g., productivity and localism) are still worth to be explored in future works.

heading: 8 Acknowledgments
text: We are grateful to three anonymous reviewers for their valuable comments that have helped to improve this paper. This work has been supported through a scholarship provided by the Heidelberg Institute for Theoretical Studies gGmbH.

heading: B Data Stastics
text: Since we use 30 verbs to construct premises fv/nv(s) for primitive veridical (PIver) and compositional (CI) inferences from the premises (s) of natural inferences (PInat), the number of these two inference types is 30 times the amount of PInat, respectively. To avoid data biases in composition training, we guarantee the two major types from Dtrain are balanced by downsampling the extensive inference type. For example, in Task1 trivial setting, we downsample PIver to ensure the training data of PIver and PInat is balanced.

heading: C Evaluated Pre-train Language Models
text: We evaluate SETI across six well-known PLMs. Table 7 shows the training objective and parameters of each model. Detailed information and training parameters of each model is:
BERT (Devlin et al., 2019) is a bidirectional transformer pre-trained model, trained with masked language modeling and next sentence prediction objectives on a large corpus. We fine-tuned the baseuncased-large version, with the default setting.
ALBERT (Lan et al., 2020) build on BERT, and presents two parameter-reduction techniques to lower memory consumption and increase the training speed. We fine-tuned the ALBERT-large version, with the default setting.
RoBERTa (Liu et al., 2019) builds on BERT, but is trained without the next-sentence prediction objective and uses much larger data. We fine-tuned a RoBERTa-large version, with the default setting.
GPT-2 (Radford et al., 2019) is a decoder-only model, pre-trained on a large corpus of English
data in a self-supervised fashion. We fine-tuned the GPT2-large version, with the default setting.
BART (Lewis et al., 2020) is an encoder-decoder model. The pretraining task involves randomly shuffling the order of the original sentences and a novel in-filling scheme, where spans of text are replaced with a single mask token. We fine-tuned the BART-large version, with the default setting.
T5 (Raffel et al., 2020) is an encoder-decoder model which is pre-trained on a multi-task mixture of unsupervised and supervised tasks. Each task is in a text-to-text format. We fine-tuned the T5-large version, with the default setting.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7
7 A2. Did you discuss any potential risks of your work? We do not find any risk in our work so far. We use an open-sourced NLI dataset.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: In the abstract and introduction sections.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4.1

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.1
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The datasets used in this paper are open-sourced.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Both artifacts are used for research purposes.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The SICK dataset does not contain information about names or uniquely identifies individual people or offensive content.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.1
C 3 Did you run computational experiments?

heading: Appendix C

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix C
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix C, Section 4.2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.2, Section 5.1
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Extending our benchmark on manually annotated compositional inference datasets might be a promising future work. Recently, Hupkes et al. (2020) dissect the notion of compositionality and define five theoretically grounded tests for generalization, in a taskagonistic manner. Our work is limited to evaluating the systematicity of PLMs in textual inference. While the systematicity test is one of the most im-
portant tests, the remaining ones (e. g. , productivity and localism) are still worth to be explored in future works