Reverse-engineering NLI: A study of the meta-inferential properties of Natural Language Inference
Authors: Rasmus Blanck
Bill Noble
Stergios Chatzikyriakidis
Venue: COLM 2025 (2025)
Keywords: computational semantics
natural language inference
logic
TL;DR: We perform a comprehesive analysis of NLI under three different logical interpretations of its labels and test the meta-inferential behavior of models trained on SNLI to better understand the logical properties of the task encoded by the dataset.

Abstract: Natural Language Inference (NLI) has been an important task for evaluating language models for Natural Language Understanding, but the logical properties of the task are poorly understood and often mischaracterized. Understanding the notion of inference captured by NLI is key to interpreting model performance on the task. In this paper we formulate three possible readings of the NLI label set and perform a comprehensive analysis of their respective _meta-inferential_ properties. Focusing on the SNLI dataset, we exploit (1) NLI items with shared premises and (2) items generated by LLMs to evaluate models trained on SNLI for meta-inferential consistency and derive insights into which reading of the logical relations is encoded by the dataset.

Published as a conference paper at COLM 2025
Reverse-engineering NLI: A study of the meta-inferential properties of Natural Language Inference
Rasmus Blanck & Bill Noble
The Centre for Linguistic Theory and Studies in Probability (CLASP), Department of Philosophy, Linguistics and Theory of Science, University of Gothenburg (rasmus. blanck, bill bill..obeeleeeeeeeeeeeeeee égu.se
Stergios Chatzikyriakidis
Computational Linguistics and Language Technology Lab (UCRC),
Department of Philology,
University of Crete stergios. chatzikyriakiddiseuoo....
Abstract
Natural Language Inference (NLI) has been important task for evaluat- ing language models for Natural Language Understanding, but the logical properties of the task are poorly understood and often mischaracterized. Understanding the notion of inference captured by NLI is key to inter- preting model performance on the task. In this paper we formulate three possible readings of the NLI label set and perform a comprehensive analysis of the meta-inferential properties they entail. Focusing on the SNLI da taseet, we exploit (1) NLI items with shared premises and (2) items generated by LLMs evaluate models trained on SNLI for meta-inferential consistency and derive insights into which reading of the logical relations is encoded by the dataset.
1 Introduction
Natural Language Inference (NLI) has provided important benchmarks for assessing the Natural Language Understanding (NLU) capacity of computational models. The importance of these benchmarks comes from the theoretical role of inference in linguistic semantics, wherein what can be inferred from given sentence is determined by its meaning. The function learned by an NLI model takes two natural language sentences as input and returns label from set of possible inferential relations between those two sentences. The fact that the two inputs are of the same type creates opportunities to investigate the "internal consistency" of the model; that is, the properties of the learned function relation to how we think it should theoretically behave.
In a number of papers, the internal consistency of NLI models is investigated by means of their meta-inferential properties. For example, if we believe that contradiction is symmetric ("a contradicts 5" is equivalent with "b contradicts a"), then we may demand that any NLI model labeling the sentence pair (a, b) as contradiction must label the pair (b,a) a) in the same way, in order to be internally consistent with regards to our external understanding of contradiction. Here, the relation Cab that holds when contradicts b is example of inference relation, wheras the postulate that Cab -> Cba is a meta-inferential relation.
A problem with many logical approaches to NLI in the literature is that (1) the NLI relations are left undefined so that it is not clear what the relations are or in which logic they can be expressed (if at all); and (2) it is not made clear from which logic the meta-inferential intuitions are coming. This is problematic since which meta-inferential relations that hold between inference relations depends on the definition of the inference relations.

---

Published as a conference paper at COLM 2025
In this paper, we first review related work, partly in order to start disentangling different interpretations of the NLI relations (Sec. 3), then establish a framework for analyzing the NLI relations (Sec. 4), and moreover propose three different readings of those relations, analyze their meta-inferential properties, and hypothesize that one of the readings better captures the mode of inference that is implicit in the NLJ dataset. We generate new data (Sec. 5) and experimentally confirm its suitability for testing the main hypothesis (Sec. 6).
Background
The SLNI corpus (Bowman et al., 2015), and its derivatives (such as MNLI (Williams et al., 2018), e-SNLI (Camburu et al., 2018), MedNLI (Romanov & Shivade, 2018), XNLI (Conneau et al., 2018), ANLI (Nie al., 2020)), has been used extensively as proxy for the NLI task. This is three-way classification task, where sentence pairs (a,b) of a premise a and a hypothesis are to be classified using one of the labels Entailment, Neutral and Contradiction. 2 SNLI premises are all captions from the Flickr30k corpus (Young et al., 2014), and the hypotheses are constructed by human workers given the following instructions:
We will show you the caption for a photo. We will not show you the photo.
Using only the caption and what you know about the world:
Write one alternate caption that is definitely true description of the photo. [...]
Write one alternate caption that might be a true description of the photo. [...]
Write one alternate caption that is definitely false description of the photo. [...]
(Bowman et al., 2015, p. 634, emphasis in original)
Williams al. (2018) give similar instructions for constructing the MNLI corpus, along with the gloss that the hypothesis of an entailment is "necessarily true or appropriate whenever the premise is true", a contradiction "necessarily false or inappropriate whenever the premise is true" and neutral "neither condition applies". This suggests that the three- way classification should cover the whole space of sentence pairs, and that neutral pairs are neither entailments nor contradictions sentiments that are repeated by Nie et al. (2020).
Models trained on SNLI-style datasets learn to classify sentence pairs according to the NLI labels. The classification induces relations E, C, N that hold sentence pairs classified by the respective label. Externally, our logical intuition is the norm according to which we expect the models to behave. But, salient question is whether we can actually expect the models to behave according to that norm, given the data from which the inference relations are induced. In other words, what inference relations could possibly be induced from the data provided to the models? If those relations are not compatible with our intuitions, then perhaps the models are not so much to blame as the datasets.
A preliminary inspection shows that the NLI relations are not definable in propositional logic (in strong sense, see Sec. 3.2). The classical material conditional is also well known to be inadequate for formalizing natural language conditionals (see, e.g., Lewis & Langford 1959). Moreover, the NLI annotator instructions exhibit strong modal language ("neces- sary"/" pary"/"defiinn /deffinitllyy true"), which suggests that the NLI relations could lend themselves to modal treatment. The strict conditional of Lewis & Langford (1959) is definable in a modal language, and is better suited for formalizing natural language conditionals, even though it suffers from shortcomings of its own when it comes to counterfactual conditionals.
There is another crucial observation to be made: Every SNLI premise comes along with the implicit assumption that the premise is possible again a glaringly modal notion. The reason is that each SNLI premise is caption of a photo of "everyday activities, events and
1 For data and experimental code see: https: / //g com/GU-CLAS /tw.s///rrrreereeennneerrnnnnnn 2his three-way classification appears early with the FraCaS test suite (Cooper et al., 1996), and became standard with RTF-4 (Giampiccolo et al., 2009).
2

---

Published as a conference paper at COLM 2025
scenes". If the caption is actually accurate, then is describing situation that is indeed possible. We may understand the existence of the photo as implying that there is a model satisfying the caption, namely the situation that the photo is photo of. This means that the data, if constructed precisely according to the instructions, do not contain any examples of vacuously true entailments that, for example, classical propositional logic would validate. A model trained on such data is, for example, unlikely to learn the principle of explosion, and would therefore not be compatible with certain classical modes of inference.
Our approach is to define the NLI relations in a modal language that can adequately express the notions used in the NLI instructions. We consider two such readings, as well as reading based on the "default" material conditional. The various meta-inferential principles appearing in the literature are scrutinized through this lens. We observe that certain " "obvious" meta-inferential principles are valid on some of the readings but not on others. We leverage this observation show to what degree different readings of the NLI relations are compatible with the predictions of models trained on NLI data.
3 Related work
3.1 Modalities and non-contradictory premises
Gubelmann et al. (2024) take note of the modal language ("definitely" aeafiiitely"""neeessaat"t /"necessary"") of Bowman et al. (2015) and Williams et al. (2018), but without explicitly recognizing it as being modal. Instead, they conclude that the SNLI and MNLI are best "conceived as aiming at deductive validity". They leave unspecified what notion of deductive validity to consider.
Holliday et al. (2024) investigate the reasoning capabilities of LLMs on linguistic material containing conditionals and modal operators. Their approach is different from ours, as they are concerned with modalities in the object language and the relations between sentences containing such linguistic material, whereas we are using the language of modal logic metalanguage in which to reason about inference relations.
Feng & Hunter (2024) briefly consider sentence pairs where a premise entails both b and the negation of They observe that in classical propositional logic, this can only hold if is contradictory in itself. They confirm that in random sample of 1000 sentence pairs from eSNLI, each premise is non-contradictory.
3.2 Inference relations
Wang et al. (2019) formalize the NLI inference relations in propositional logic by:
E:a->b b C:a - b N:alb
where 1 means that there is "no clear relation between a and b". A similar definition is given by Banerjee et al. (2024). There is crucial oversight in both accounts: If the implication is classical, then F. and together exhaust the whole space of sentence pairs, and there is no room for any pair being classified as neutral. The reason is that the sentence (a + b) A (a + bb) is a contradiction in classical logic. So the NLI relations can not be meaningfully expressed in classical propositional logic.
Sia et al. (2023) seem to recognise that classical propositional logic will not do, in that their
neutral relation is expressed as "I 3 x", where "3 indicates trufh-nalucless". It is unclear exactly how their application of free logic NII is supposed to work, as they never use the non-denoting singular terms that are the distinguishing feature of free logic. Their claim that classical logic "requires each singular term to denote a Boolean variable in the domain" " suggest mix-up of terms and formulas.
MacCartney & Manning (2014) express the three-way formulation of NLI with a for entailment, F= -b for contradiction, and H b AA / -b for neutral, where F is some semantic notion of entailment. By contrast, Tian et al. (2021), in their study the capabilities of language models to reason in first-order logic, use notion of "syntactical consequence"
3

---

Published as a conference paper at COLM 2025
supposedly meaning derivability in first-order logic. They use the syntactic consequence relation to define four inference relations, where E is defined as a H b A a K -b, C as a K b Aa + -b, N as a K b A a K -b, plus version of the principle of explosion that they call "Paradox": a A F -b. These definitions entail that the standard NLI relations can only hold for pairs in which the premise is non-contradictory. They also observe that the "Paradox" scenario is rare in texts as well as in NLI tasks.
3.3 Meta-inference relations
Minervini & Riedel (2018) postulate that certain meta-inferential relations can be deduced to hold between NLI sentence pairs when the order of the sentences are swapped. They postulate that Eab - -Cba, where E and C are taken to be binary predicates of first-order logic that are axiomatized by rules this form. The validity this rule depends on how the relation E is understood. For example, it is not compatible with the understanding of Wang et al. (2019) that Eab can be defined in propositional logic as a + b and Cab as a + -b. Under this reading, the relation postulated by Minervini & Riedel (2018) holds iff is true.
Li et al. (2019) take note of five meta-inferential rules, including symmetricity of con- tradiction, and Eab A Cbc Eab A Cac + Cbc is a valid principle. But under modest assumptions on the interpre- tation of the NLI relations, this principle can only hold under very specific conditions that drastically restrict the relations between a, b and (see Table 2 for details).
Cac. Based these rules, Jang et al. (2022) claim that
Srikanth Rudinger (2025) investigate the internal consistency of NLI models by means of atomic hypothesis decomposition. They prompt an LLM to break down NLI hypotheses into "atoms", each one judged by the model to be entailed by the original hypothesis. Then, e.g., for an inference Eab, with b decomposed into atoms by,..., bn, the inference Eab; must hold for each S л for model to be internally consistent. Similar meta-inferential relations can be identified for the other NLI labels. Their results show that high accuracy on NLI pairs is not indicative of internal consistency, and that incorrectly labeled examples give rise to more inconsistencies between the original NLI pair and the generated subproblems.
The modal conception of meta-inference
We propose a reading of the NLI relations that takes inspiration from classical Tarskian semantics, and the treatment of quantification over logical structures as a modal operation." 3 define the NLI relations is weak classical set theory. We
The metalanguage in which leave the object language, as well as the notion of structure and satisfaction unspecified. This means that we are treating the NLI sentences as structurally unanalyzed, as we are only concerned with the meta-relations between NLI relations. We are never concerned with concepts such as what makes a true or false, nor the internal logical form (if any) of a, but only with whether, for example, Eab implies in the metalanguage that -Cba.
The standard Tarskian compositional clause for implication is the basis for our modal interpretation of meta-inference. Tentatively, we define
Eab iff
This can be more succinctly and suggestively expressed in the language of propositional modal logic as [a > b), where is understood as universally quantifying over accessible (possible) "worlds" or situations (Goldblatt, 1992). This is the definition of strict implication given by Lewis & Langford (1959). We remain agnostic as to which kind of worlds the modal operators quantify over, and also leave the accessibility relation unspecified. If F is first-order satisfaction and D quantifies over all first-order structures, then Eab coincides with first-order logical consequence. If the symbols given some other interpretation, Fab may or may not coincide with the consequence relation of some other logic. Since NLJ inferences allow for the use of world knowledge, it seems reasonable to allow only worlds
3This style of semantics originates with Tarski (1933/1986) and is outlined in just about any modern textbook in mathematical logic, such Mendelson (2015).
4

---

Published as a conference paper at COLM 2025
Eab
Cab
Nab
MC SC
a + b
a + - a + -b) Q A ( + b)
(a b) A + bb
[a + b)
o b) 0 (a -b)
El
Qa (a b)
(C-av ((aA)) A (0 (an
Table 1: Three different versions of the NLI relations
that are similar enough to the real world to be accessible. We assume that the interpretation of the modal operators are kept fixed to allow comparing different examples, and that they conform with the axioms of the minimal normal modal logic K (Goldblatt, 1992).
Having observed that SNLI premises are logically possible, we also consider a version of NLI relations based on the following modified version of the Tarskian clause:
Eab iff MM F a) WM a M b).
The corresponding modal expression is Qa A (a + b), where the diamond is understood as existentially quantifying over accessible worlds.
Table shows the three different readings each of the NLI relations. The first is based on the material conditional (MC) of classical propostiion logic. The second corresponds to the strict conditional (SC) of Lewis Langford (1959). The third (EI) takes into account the fact that NLI premises logically possible, giving the operator D existential import, similar to the universal premises of Aristotle's syllogistics (Lewis & Langford, 1959, 62ff).
Under both modal readings, modal logic K proves that, for all propositional atoms and b, Eab Cab Nab, so that the relations together cover the whole space.* For the SC reading, we can prove Eab V -Cab V -Nab and Nab + -aa A -Cab, meaning that the relations are not fully mutually exclusive: There might be overlap between E and The El reading, on the other hand, gives full K-provable trichotomy:
(Eab A -Cab -Nab) V V( (-Eab A Cab A -Nab) V (-Eab A -Cab A Nab).
Using these definitions of the NLI relations, we establish number of meta-inferential relations. They are presented in Table 2.5 We observe that the reading of the NLI relations with existential import stand out from the others. It validates certain principles that the others invalidate, and vice versa. Therefore it provides a handle for us to compare the principles with the relations induced by the NLI data and learned by the NLI models.
Data
To test meta-inferential relations, we require NLI data that has overlaps in the sentences appearing as the premise or hypothesis of multiple different items. We achieve this in two ways. First, we use the SNLI (Bowman et al., 2015) dataset, where each premise appears in up to three different items (one with each of the three inference relation labels). Second, we generate new NLI items with SNLI hypotheses as the premise. From SNLI and the generated items, we create a third dataset of inferred items which combine up to two items from the other two datasets to form a new gremis-hyyothhsit pair. While we don't have ground-truth labels for these items, the NLI label of the original items allow to infer (or rule out) labels depending on the logical reading of the inference relation (see Table 2).
5.1 SNLI
SNLI (Bowman et al., 2015) a large dataset of human seemaateeeeerted NLT items, which has been a staple in the field of Computational Linguistics since its release. As premises, the
4 4Proofs are verified using the Tree Proof Generator (Schwarz, 2025).
For brevity, assume that the precedence of logical operators are A, V, ->, 50 that, for example
-a b V is equivaleett to ((-a) A b) V c).
5

---

Published as a conference paper at COLM 2025
MC
SC
EI
Paper
Eaa
Caa
-Qa
1
Naa
1
Oa
Cab Cha
/
-a V
Ob
+,
Cab + -Eba
b
Eab -> -Cha
O
/
Nab + -Cba
Ø
/
Qa V
Q
Eab A Ebc Eac
Eab A Chc + Cac
V
Nab A Ebc Cacc
Nab A cbc + Eac
Cab A Nbc + -Eca
Ø
a
Eab A Ebc + -Cca
Oa
Eab A cbc + Cca
V
Eab A Cbc + -Eca
a b V
Man-bva
Nab A Ebc -Cea
Eab A Cac -Eb
a bA
((avbA-c)
Eab A Cac + Cbe
a V -V-bV
Eab A Nac
Nbc
Nab A Eac + CCh
Ø
Nab A Cac > Ebc
Table 2: Meta-inferential relations. We v, and Ø to denote, respectively, that the relation is valid, contradictory, and vacuously true. When a formula, such as Qa, is listed, the relation is equivalent over K that formula. The formulas denoted by Greek letters are: ( -a A ( -c), f D(~aV V -b) ( -b cb),), ( - -c) ) V O V bΛ -c) (a e), 8 (a -b V (-b Vc) and 8V V c). The paper by Minervini & Riedel (2018) is denoted Li et al. (2019) as t, and Jang et al. (2022) as
dataset uses the visually-descriptive image captions of the Flickr30k corpus (Young et al., 2014). For each caption, crowdsourced annotators were asked write three hypothesis sentences corresponding to each of the three NLI relations, using the instructions described in Sec. of this paper. Because of this procedure, most premise sentences are repeated in three different items in the dataset. The full dataset consists of about 570k items including development and test splits of 10k each. About 10% of the data was validated by up to four additional crowd workers who independently annotated the gremise-hypother pair with one of the NLI labels.
5.2 Generated NLI datasets
To generate NLI items, we provided an LLM with prompt similar to the instructions given to the SNLI annotators. Rather than an image caption, used the hypothesis of an SNLI item as the premise and instruct the LLM to generate hypothesis sentence for each of the three NLI labels. In addition to the instructions, prompted the model with 10 few-shot examples drawn from the collection of "perfect items" in the SNLI train set; that is, captions for which all four of the secondary annotators agreed with the original label all three items.6 The hypotheses in these examples were lightly edited to remove anaphoric references to the premise (see Appendix C for the full prompts). We repeated this procedure for 25k items sampled from the SNLI train and all of the test set items, resulting in dataset of 75k train items and 30k test items for each LIM.
'There are only 19 captions for which there is full inter-annotator agreement on all three corre- sponding SNLI items (one for each NI.I label). Of these 19 we sampled 10 captions, and used the corresponding 30 SNLI items.

---

Published as a conference paper at COLM 2025
We experimented with datasets generated by three different LLMs: Llama3.2 (3b parameters), Llama3.3 (70b parameters), and the distilled DeepSeek-RI model that uses the Llama3.3 architecture (70b parameters). These three models were chosen because they are open access and have quantized versions that are possible to run with commercially available cloud GPUs (see Appendix D for implementation details). For our purposes, it is important that the model can generate NLI items that aren't 't too far out of distribution from SNLI, which validate in Section 6.1.
We refer to the generated datasets as LI3.2, LI3.3 and DS-R1, respectively. We also use combined training sets SNLI+L13.2, for example, refers to the SNLI train set combined with the dlama3.2-generee
train set, and SNLI#LI3,2 is the same, but with 75k randomly sampled items from SNLI removed to maintain size parity with the original SNLI train set.
5.3 Inferred NLI test set
Using the SNLI and generated test sets as input, we create an inferred test set according to the following rules:
Add (b, a) for each SNLI items (a, (1).
Add (b, c) for each pair of SNLI items (a, b,4, and (a, 12)
Add (a, c) for each SNLI item (a, b, and generated item (b, c2,))
Add (c,a) for each SNLI item (a, and generated item (b,c,
While these items are presented as unlabeled here, certain values of / and l2 allow us to infer a restricted set of possible labels, depending the meta-inferential properties of how the entailment relations are interpreted, as detailed in Table 2.
5.4 Manual validation
To validate the generated and inferred data, a small-scale manual annotation was carried out 100 items. A total of 9 items generated and 5 inferred) had expected labels inconsistent with the label chosen by the annotators, although the inherent subjectivity of the task can explain many of those discrepancies. The validation is described in detail in Appendix A.
6 Experiments
We proceed with two experiments. In Sec. 6.1 evaluate the suitability of three LLMs for generating NLI data and select a dataset to proceed with. In Sec. 6.2 we use the inferred test and recently state-of-the-art NLI model to investigate the meta-inferential properties encoded in SNLI.
For each model, we also train hypothesis-only baseline model, which trained predict the NLI label based on the hypothesis alone (Poliak et al., 2018). In the full model, the premise and hypothesis are concatenated as text input with the appropriate sentence-separator token (specific to to the base model architecture) between the two sentences. Additional training details can be found in Appendix E.
6.1 Suitability of the generated NLI data
We conduct the first experiment primarily with BERT (Devlin et al., 2019) since its perfor- mance on SNLI very well-studied. We experiment with five different training sets: The standard SNLI and SNLI augmented with each of the three generated train sets, removing randomly sampled 75k items from the SNLI train set to maintain parity with the vanilla SNLL For LI3.2, also experiment with not removing the sampled items.
We observe that the vanilla SNLI model (first row of Table 3) performs about a bit worse LI3.2 and DS-R1 and bit better LI3.3 in comparison to the SNLI test set. All models
7

---

Published as a conference paper at COLM 2025
Model
Train set
SNLI
L13.2
LI3.3
DS-R1
SNLI
77.9
15.0
73.6
6.1
81.0
2.1
73.3
6,4
SNLI+DS-R1
76.7
12.4
76.5
6.8
82.9
0.6
76.1
4.6
SNLI+LL3.2
77.6
14.3
79.3
8.2
82.0
72.9
5.9
BERT
SNLI+LL3.3
78.1
15.4
76.3
7.3
86.0
3.3
76.2
8.7
SNLI=DS-RI
78.6
14.7
77.7
8.0
84.7
3.0
78.5
7.2
SNLI+LL3.2
77.1
14.3
79.9
7.7
83.5
76.0
7.3
LLL3
79.1
16.3
77.0
7.9
86.9
4.6
78.0
8.8
SNLI=DS-RI
90.7
20.9
87.5
17.0
97.4
10.9
92.4
15.3
RoBERTa+SE
LLL3
91.2
21.3
84.9
14.2
98.0
7.7
89.5
15.5
Table 3: Macro-average Fi-score (and percentage point increase over the hypothesis-only
model) on the standard SNLI and LLM-generated test sets. Rows correspond to different model architectures and training sets, including the standard SNLI training set and SNLI augmented with generated data, as described in $5.2.
perform well on the original SNLI data (first column of Table 3), with the LL3.3 and DS-R1- trained models providing small boost over the unaugmented training data, even when the overall training set size was maintained.
Overall, these results suggest that the generated NLI data, especially that of the two 70b parameter models, adequately replicates the distribution of the human-generated SNLI dataset. While including the LI3.3 data the train set provides the largest performance boost on the SNLI test set, DS-RI is not far behind in this regard. Moreover, we note that the difference between the performance of the full and hypothesis-only models is smaller for LL3.3 in comparison to DS-R1, suggesting that DeepSeek-R -11 R1 is less prone to "give away" the NLI label in how it generates the hypothesis in comparison to LLama3.3, For this reason, we conduct the next experiment with inferred items based on the SNLI DS-RI test sets.
6.2 Meta-inferential consistency
For the main experiment, we the inferred test set to investigate the meta-inferential properties that model learns from SNLI. The strength of the conclusions we can draw depends somewhat on the model's NLI performance, since predictions follow some internal logic that is learned from the data. For this reason, the main experiment is conducted with RoBERTa+ SE (Sun al., 2020), span-based "self- explaining" model based on RoBERTa (Zhuang et al., 2021), which has recent state-of-the- performance on SNLL The model is trained on SNLI+DS-R1. Please refer to Table for the model's performance on the SNLI and generated test sets.
assume that the model
We test the model on novel "inferred" items that combine sentences from standard SNLI and DS-R1 test sets, as described in Sec. 5.3. Table 4 shows results on items for which the model correctly labeled the items it was based on, since these results are most useful for assessing the model's meta-inferential consistency (see Appendix B for unfiltered results).
7 Discussion
We have presented two different modal readings of the NLI relations that validate different collections of meta-inferential relations, and have compared them with the labels predicted by the model. The results show that, overall, the model performs very well on the meta- inference task, as most scores are in the high 90's. As a general pattern, whenever hypothesis is labeled with a label I. in the input pattern and the same reappears as the hypothesis of the target pair, the model prediction skewed towards the label L.
We use the training set that includes generated data to mitigate the possibility that idiosyncrasies in the L.M-genertted data would skew results on inferred items that include generated sentences.
8

---

Published as a conference paper at COLM 2025
input items
item
count
E
N
SC
SCV
ETV
Cab
3023
0.4
79.5
20.1
Cba
79.5
-Eba
99.6
Eab
ba
3094
8.4
2.4
89.1
Cba
97.6
Nab
2800
12.8
8.7
78.5
-Caa
91.3
Eab A Ebc
2975
96.8
0.8
2.4
Eac
96.8
Eac 96.8
Eab
Cbe
3001
0.7
98.3
1.0
Cac
98.3
Cac
98.3
Nab A Ebc
8
2635
59.0
3.0
38.0
Cac
97.0
CCac
97.0
Nab A Chc
2670
0.3
87.1
12.6
Eac
99.7
-Eac
99.7
Cab A Nbc
2614
0.5
60.6
38.9
-Eca
99.5
Eab Ebc
2975
1.3
4.4
94.3
-Cca Eca
95.6
Eab A Che
ca
8
3001
0.1
74.3
25.5
Cca
74.3
99.9
Nab A Ebc
2635
2.3
11.7
86.0
-Cca
88.3
Eab Cac
2824
0.4
82.1
17.5
-Ebc
99.6
Eab A Nac bc
h
2408
0.6
3.8
95.6
Nbc
95.6
Nbc
95.6
Nab A Eac
2408
57.2
6.7
36.1
-Cb
93.3
-Ch
93.3
Nab A Cac
2455
0.8
84.3
14.9
- Ebc
99.2
- Ebc
99.2
Table 4: Meta-inferential results for RoBERTa+SE trained on SNLI=DS-RI (filtered). Results are disaggregated by the labels of the item(s) they are based on (input items) and the premise-hypothes pair provided to the model (item). All and b sentences come from SNLI-Flickr30 captions and human-generated hypotheses, respectively. The next column shows the origin of the sentence (h SNLI hypothesis; g = LLM-
hypothesis). The number of items included is listed under count; E, Cand give the percentage model predictions for each label; SC shows what we infer (if anything about the expected label under the strict conditional reading and SCV is the percentage of predictions consistent with the possible label set implied by the labels of the input items under that reading (see table 2); similarly for columns EI and EIV under the existential import reading.
If, for given relation, the labels inferred under one of the modal readings agree with the model prediction, we take that as evidence for that reading being compatible with the prediction of the model. If one of the readings exhibit stronger agreement, this suggests that that reading more compatible with the model's notion of inference. We discuss the following three items: (1) inferences where the SC and the El readings predict different labels; (2) inferences scoring below 96; and (3) inferences that score surprisingly high.
First, consider the meta-inference Cab - Cba, which is valid under SC, but not under EL. The SC reading agrees with the model prediction in 79.5% of the cases. By contrast, the meta-inference Cab Ebaa is invalid under SC, but valid under EI. The EI reading agrees with the model in 99.6% of the cases. Similarly, the meta-inference Eab ^ Chc - -Eca, which is valid under EI (model scores 99.9), but where SC instead predicts Cca (score 74.3). This initially suggests that the El reading of the NLI relations is more compatible with the model predictions.
We manually analyze 100 randomly sampled items corresponding to the meta-inferences Cab Cba/-Eba discussed in the previous paragraph. The first group draws 50 items for which both ab and ba were labeled C, and the second 50 items where ab were labeled C but ba were labeled N which is correct under the El reading, but not under SC. The second group contains 25 items exhibiting "backwards anaphora" ("The man A man ..." in hypothesis). There are items where both ab and ba can be construed as neutral. Three premises contain an explicit "not", and one premise is in the imperative. By contrast, the first group contain no mislabelled items, only examples of backwards anaphora, and none of the other artifacts. This suggests that artifacts in the data may contribute to the relatively high number of items being labeled N for this meta-inference.
in premise and
There five other meta-inferences where the model scores less than 96. The lowest score, 88.3, is for the meta-infern inference Nab Ebc -Cca. Interestingly, this pattern is not valid on the El reading, but only under the MC (and vacuously so) and SC readings. The second

---

Published as a conference paper at COLM 2025
lowest score, 91.3 is for the pattern Nab + -Cba, which is also invalid under the El reading. Since it is not possible to infer a non-trivial set of possible labels for these inferences under the El reading, there is clear-cut comparison to be made. By contrast, the meta-inferenee Eab A Ebc + -Cca scores the relatively low 95.6, but is only valid under the El reading.
The remaining two low-scoring meta-inferences are Eab A Nac Nbc, scoring 95.6, and Nab A Eac + -Cbc, scoring 93.3. They are both valid under the EI reading. Note that the first of these meta-inferences is equivalent over K to Nab A Eac - Ncb, which together with the meta-inference Nab + -Cba would make it possible to deduce the latter meta-inference But since Nab + -Cba is invalid on the EI reading, this deduction is not possible. This seems to suggest that the EI reading is more compatible with the model predictions. On the other hand, the inference Cab Nbc -Eaa scores 99.5, even though is invalid under ET.
All but one of the meta-inferences for which the model scores high are either valid under both readings, or valid only under EI. Two of the lower scores occur in connection the meta-inference being invalid under either the SC or El reading. The remaing two lower scores are at least in part related to a meta-inference that is invalid under El. So case there is difference between the two readings, validity under El is correlated with a high degree of agreement with the model prediction. Note that there is only one meta-inference that is valid only the SC reading that scores higher than the EI reading.
All in all, this suggests that El is indeed more compatible with the model prediction than SC is. It also suggests that it is not yet possible to completely disentangle the contributions of the different readings, since invalidity under any of the readings correlates with low scores.
8 Future work
One of the long-standing problems in NLI research is the type of inference relations that are captured in different NLI datasets. It is well known that different NLI datasets capture different aspects of inference, e.g., compare the stricter logical entailment relations as these encoded in the FraCaS test suite to the more common-sense inference type relations as these are found in the RTE datasets. We believe that our method can potentially lead to a principled disentaglement of these relations We aspire to provide an attempt at this disentaglement via logics in our future work. For example, the guidelines for RTE-5 (TAC2009, 2009) force hypotheses to be possibly false, which gives rise to yet another different modal notion of entailment.
the basis of solid logical underpinnnings.
We would like to further examine the implicit cloled-world assumption of the SNLI premises that they describe situation exhaustively enough. This is likely to affect which hypotheses would be considered contradictions, and how those are constructed by human workers, as well as having bearing on how to resolve coreference on premise-hypothesi pairs. Relatedly, we would like to conduct more in-depth investigation of possible discourse effects between premise and hypothesis. This study assumes that the same sentence always expresses the same proposition and, while we took steps to reduce the use of anaphora in generated hypotheses, we know that the original SNLI data does contain anaphora.
Another more general avenue that we want to pursue the use of logics to probe/ /under- stand/ approximate better what modern NLU models learn from given task. We want to argue that whatever your stance on the use logics for NLP is, they are at least useful as a generic, robust framework for probing into the reasoning capabilities of these models and as such serious engagement with various well-founded logic systems is helpful in making sense of these capabilities.
Acknowledgments
This work was supported by grant 2014-39 from the Swedish Research Council (VR) for the establishment of the Centre for Linguistic Theory and Studies in Probability (CLASP) at the University of Gothenburg. Stergios Chatzikyriakidis gratefully acknowledges funding /GRNET
from Amazon/ GRNETT for the project Neural-Symbolic Integration for Enhanced Natural Language Processing (NTELS).
10

---

Published as a conference paper at COLM 2025
References
Sourav Banerjee, Anush Mahajan, Ayushi Agarwal, and Eishkaran Singh. First train to generate, then generate train: UnitedSynT5 for few-shot NLI, 2024. URL https:
//arxiv. wwwxiv.orgg/aie http/abs/2422..92/
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Lluís Mårquez, Chris Callison-Burde and Jian Su (eds.), Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, Pp. 632-642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/v /1//3/11D1-110/55 URL https: hatpss///caathooogyy. org/D15-1075/.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blun- som. e-SNLI: Natural language inference with natural language explanations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, Pp- 9539-9549. Curran Associates, Inc., 2018. URL http:/ /tpp:pr//ppp..////////pp nips cc/paper/ saaee--nnnnnaaraall. -aaiiinnaaral-laaaageeee Language-inference- ieeeeennffeeeeeeeeee nitu-naturall nnttr www.taataaaaaaalaaaaaa---------e wxppggaaeeellllaatiioo. pdf.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingua sentence representations. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun Tsujii (eds.), Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, Pp. 2475-2485, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653 200/0/33/1D11111///
URL httttps:/aalantttie htw.lllggoorgg//ee1 org/D18-1269/
Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, Steve Pulman, Ted Briscoe, Holger Maier, and Karsten Konrad. Using the framework. Technical Report LRE 62-051 D-16, The FraCaS Consortium, 1996.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186. Association for Com fottttati Linguistics, 2019. doi: 00.18633vv//N1/ 1423. URL https://www.
aclweb. org/ oet/oorg///////// anthology/N19-1101
Xuyao Feng and Anthony Hunter. Identification of entailment and contradiction relations
between natural language sentences: A neurosymbolic approach, 2024. URL https: //aaxiv.org//abs/24444/... 02244.5.5229.. 01259.
Danilo Giampiccolo, Hoa Dang, Bernardo Magnini, Ido Dagan, Elena Cabrio, and Bill Dolan. The fourth PASCAL recognizing textual entailment challenge, 2009. URL https: //tsapps. nist. gov/ httpssttooopppblccttioo//eeeeeee get pdf. cfmppub id=902260.
Robert Goldblatt. Logics of Time and Computation. Number 7 in CSLI lecture notes. CSLI
Publications, Stanford, CA, second edition, 1992.
Reto Gubelmann, Ioannis Katis, Christina Niklaus, and Siegfried Handschuh. Capturing the varieties of natural language inference: A systematic survey of existing datasets and two novel benchmarks. Journal of Logic, Language and Information, 33:21-48, 2024. doi: doi. org/10.1007
URL https:://oi.ooo org/10.
Wesley H. Holliday, Matthew Mandelkern, and Cedegao E. Zhang. Conditional and modal reasoning in large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Nat- ural Language Processing, pp. 3800-3821, Miami, Florida, USA, November 2024. Asso- ciation Computational Linguistics. doi: 10.18653/ v1/ 2024.emmnlp-min 222. URL https://aclantholooy.. org/222 emnlp-main. 222/.
11

---

Published as a conference paper at COLM 2025
Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. BECEL: Benchmark for consis- tency evaluation language models. In Nicoletta Calzolani, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung- Hoon Na (eds.), Proceedings of the 29th International Conference on Computational Linguistics, pp. 3680-3696, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. URL https: http:////laaatloooyy.y. o 2022. col ing 1. 1.34//.
Diederik Kingma, Jimmy Ba, and Amsterdam Machine Learning lab (IVI, FNWI). Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR). arXiv.org, 2015. URL https://hdl. handle.net//111 505366
C. I. Lewis and C. H. Langford. Symbolic Logic. Dover publications, 2nd edition, 1959.
Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. A logic-driven framework for consistency of neural models. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pp. 3924-3935, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/vi /11////111111//// URL htt.ttss///cc.aannoooooooor/////
1485/.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularizto.. Published as conference paper at TCLR 2019, 2019. URL http://arkiv.oor/abb////t
Bill MacCartney and Christopher D. Manning. Natural logic and natural language inference.
In Harry Bunt, Johan Bos, and Stephen Pulman (eds.), Computing Meaning: Volume 4, pp. 129-147. Springer Netherlands, Dordrecht, 2014. ISBN 978-944-007-728447. doi:
8. URL https://doi.
Elliott Mendelson. Infroduction to Mathematical Logic. CRC Press, Boca Raton, sixth edition, 2015.
Pasquale Minervini and Sebastian Riedel. Adversarially regularising neural NLI models to integrate logical background knowledge. In Anna Korhonen and Ivan Titov (eds.), Proceedings of the 22nd Conference on Computational Natural Language Learning, Pp. 65- 74, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653 /1////11111/1// 1007. URL https: http://cclnnntolloggggr////---/0 org//18-18077/
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLL: A new benchmark for natural language understanding. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Compulational Linguistics, pp. 4885-4901, Online, July 2020. Association for Computational Linguistics. doi: (0.18653//1/2220.aaae https://aclanthon
URL
org/2020 acl mao1-mmin..41...
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines in natural language inference. In Malvina Nissim, Jonathan Berant, and Alessandro Lenci (eds.), Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pp. 180-191, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653 111.885//11//11111 /v//s11-21212..... URL https:/ http:://acaaanhooo.oooroo wtt.//rg/////// 2023//
Alexey Romanov and Chaitanya Shivade. Lessons from natural language inference in the clinical domain. In Ellen Riloff, David Chiang, Julia Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1586-1596, Brussels, Belgiuu, October-November 2018. Association for Computational Linguistics. doi: 10.18653/ w1t/////1/DD111///// URL / Drgg/D8-11 1187/.
Wolfgang Schwarz. Tree Proof Generator. https: http://gtttuuu..oo///ttt/t 2025. URL https: //g thub. com/wo/tpg.
12

---

Published as a conference paper at COLM 2025
Suzanna Sia, Anton Belyy, Amjad Almahairi, Madian Khabsa, Luke Zettlemoyer, and Lambert Mathias. Logical satisfiability of counterfactuals for faithful explanations in NLL. In Proceedings of the Thirty-Seoenth AAAI Conference on Artificial Intelligence and Thirty- Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposiun Educational Advances in Artificial Intelligence, AAAI23 /TAAA//3 D//////EAAT'S) /EAAI'23. AAAI Press, 2023. ISBN 978-1-57735-880-D doi: 10.1609/aaaie aaai. URL https: o/doi.org/// 00066699880 2009/aaai... w3718. 26174.
Neha Srikanth and Rachel Rudinger. NLI under the microscope: What atomic hypoth- esis decomposition reveals. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Pro- ceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Pp. 2574-2589, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-18944 doi: 10.18653/v1 /ttp///112222../// 2025.naacl-lon.. .rg.org////// 2025. naacl-long. 130/.
URL
Zijun Sun, Chun Fan, Qinghong Han, Xiaofei Sun, Yuxian Meng, Fei Wu, and Jiwei Li. Self-Explaining Structures Improve NLP Models, 2020. URL / 01786.
TAC2009. RTE-5 main task guidelines.
httpss//tac.iist.oo/////////
Guidelines, sadiiineee......deeee 2009.
Alfred Tarski. The concept of truth in formalized languages. In John Corcoran (ed.), Logic, Semantics, Metamathemattic, pp. 152-278. Hackett, Indianapolis, second edition, 1933/1986 expanded version, English translation by J. H. Woodger.
Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. Diagnosing the first-order logical reasoning ability through LogicNLI. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, PP. 3738-3747, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1 / wt1//221.emllppppp .mnnl- m03n URL https: //apaanto//cllthoooo//// org/ gegg/221... emnlp-main pm-minn. 303/.
Haohan Wang, Da Sun, and Eric P. Xing. What if we simply swap the two text frag- ments? A straightforward yet effective way to test the robustness of methods to con- founding signals in nature language inference tasks. In Proceedings of the Thirty-Thire Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Infelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intellignce, (AAI119/AATTI 10AA9//9/AAAAAA /EAAT11. AAAT Press, 2019. ISBN 978- 978-1-57735-8099 doi: 10.1609 wtt.:/a.........33000003 033017133333 URL https: / /ooii org/10. 11t.:..60//aaaaa. v33101 v33101.33817113.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp.
1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. URL https: hatp///aaaannnoyo...ooaooooo B/5/N1881111111
doi: 10.18653 10118863/11/N18111
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014. doi: 10.1162/
tacl a 00160.06666.00 URL https::t//// / http///alllltttt http://hlaaaatooggo..oo.o org/Q14- 1-10/111111 1006/.
Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. A robustly optimized BERT pre-training approach with post-training. In Sheng Li, Maosong Sun, Yang Liu, Hua Wu, Kang Liu, Wanxiang Che, Shizhu He, and Gaoqi Rao (eds.), Proceedings of the 20th Chinese National Conference on Computational Linguistics, pp. 1218-1227, Huhhot, China, August 2021. Chinese Information Processing Society of China. URL https: /tte://acllaiii http://hllaaathog.....o.o org/ 2021.cc1-1 108/.
13

---

Published as a conference paper at COLM 2025
generated
ab; ba
ab, ac; bc
ab, bc;
ab, bc; ca
annotator disagreements
3
1
5
substantial disagreements (EI)
0
0
0
substantial disagreements (SC)
2
0
inconsistent consensus label
4
0
5
0
total annotated items
20
20
20
20
20
Table 5: Counts for the annotated items by category. The substantial disagreements rows show disagreements where least one of the annotated labels was not among the item's inferred possible labels under the respective reading. The inconsistent consensus label row shows counts of items for which the annotators' consensus label was outside of the possible label set for the inferred or generated item.
A Manual validation
In order to validate the generated and inferred items, a small-scale manual annotation was carried out by two of the authors. Both annotators assigned an NLI label to each of 100 items - 20 items generated by Llama3.3 and 20 from each of the four inference patters described in Section 5.3. The annotators then discussed disagreements and decided a consensus label. The items were shuffled prior to annotation, and the source and expected labels were unknown at the time of annotation.
Results of the annotation can be found in Table 5, The agreement between the annotators was good overall. A total of 14 items required discussion to reach consensus. Of the disagreements, several represented inferred items where both of the annotated labels were in set of possible labels. L.e. they were not substantial disagreements under one or both of the meta-inferential readings.
We found total of (4 generated and 5 from the inference pattern ab, bc;ac) where the consensus annotated label is not consistent with the expected label set (either the label used to prompt the model or the inferred possible labels). Table shows those items, the expected labels, and how they were annotated. We note that in two of the four generated items, one of the annotators initially chose the label intended by the model. In nearly all cases, the inconsistency appears to hinge on either the precise interpretation of lexical item or degree of plausibility of the hypothesis highlights the subjective nature of the task, but overall gives us confidence that the data is of similar enough quality to SNLI for the present purposes.
conjunction with the premise. This observation
B Additional results
This section contains additional results for the experiment in Sec. 6.2. In particular, Table 7 shows the results without filtering out items for which the model made an incorrect prediction on one more of the anticedent input items; and Tables 8 and 9 show the same results for the Llama3.3-generated data.
As in Table 4: Results are disaggregated by the labels of the item(s) they are based on (input items) and the premise-hypothest pair provided to the model (item). All a and sentences come from next column shows the origin of the sentence (h SNLI hypothesis; g LLM-generated hypothesis). The number of items included is listed under count; E, C and N give the percentage of model predictions for each label; SC shows what we can infer (if anything) about the expected label under the strict conditional reading and SCV is the percentage of predictions consistent with the possible label set implied by the labels of the input items under that reading (see table 2); similarly for columns EI and EIV under the existential import reading.
Flickr30k captions and human-generated hypotheses, respectively. The
14

---

Published as a conference paper at COLM 2025
source
item
Ex
Cn
A2
A man is teaching his granddaughter to swim.
H: A grandfather instructing girl on swimming techniques.
N
E
E
E
generated
P: People near a lot of reading materials.
C
N
N
N
H: People are playing video games
P: The track runners all black ethniticity
E
N
E
N
H: The runners are people of African descent
they are singing in christmas program
E
N
E
N
H: People performing stage.
P: A man in green shirt is getting his feet wet in
front wall showing closeup picture face.
E
H: There's man outdoors.
ab, be; ac
Three people posing for camera
C,N
E
E
E
H: All individuals are completely still.
P: A lady in pink long-sleeved blouse is holding a
pink bag in her lap as she reaches something to the guy next her.
E,N
C
C
c
H: a woman in pink shakes hands with a man group of young boys wearing track jackets stretch their legs on gym floor as they sit a circle.
C
N
N
N
H: The group is sitting down quietly.
One man, shirtless, and woman, clothed, jumping in the water.
C N
They are completely dry.
Table 6: Error analysis for generated and inferred items. Column Ex shows the expected labels, which either the label Llama3.3 was prompted to generate or the possible labels inferred from the labels of the original items. Note that the five ab, be; ac items had the same inferred labels under both the SC and El readings. Cn shows the consensus label, and A1 and A2 show the labels selected by the two annotators. This table shows all annotated items for which the consensus label was not among the expected labels (i.e., where Cn does not appear in Ex).
15

---

Published as a conference paper at COLM 2025
input items
item
count
C
N
SC
SCV
El
ETV
Cab
3237
0.9
76.2
22.9
Cha
76.2
Eba
99.1
Eab
3368
8.7
3.1
88.2
-Caa
96.9
ba
Nab
3219
12.0
10.0
78.0
CCaa
90.0
Eab A Ebc
3368
93.0
1.7
5.3
Eac
93.0
Eac 93.0
Eab Cbc
3368
1.1
96.3
2.6
Cac
96.3
Cac
96.3
Nab A Ebc
g
3219
58.2
4.5
37.4
Cac
95.5
-Cac 95.5
Nab A Cbc
3219
0.7
85.3
13.9
Eac
99.3
-Eac
99.3
Cab Nbc
3237
0.5
59.3
40.3
EEca
99.5
Eab Ebc
3368
1.2
5.3
93.4
-Cca -Eca
94.7
Eab A Cbc
ca
3368
0.1
73.5
26.4
Cca
73.5
99.9
Nab A Ebc
3219
2.1
12.6
85.3
-Cca
87.4
Eab Cac
3247
0.7
78.9
20.3
Ebb
99.3
Eab A Nac
2858
2.7
6.6
90.7
Nbc
90.7
Nbc
90.7
bc
h
Nab A Eac
2858
53.1
8.2
38.6
-Cb
91.8
-Cbe
91.8
Nab A Cac
2951
1.5
81.4
17.1
Ebc
98.5
Ebc
98.5
Table 7: Meta-inferential results for RoBERTa+SE trained on SNLI=DS-RI (unfiltered).
input items
item
count
E
C
N
SC
SCV
EI
EIv
Cab
3029
0.4
87.3
12.3
Cba
87.3
Eba
99.6
Eab
ba
3096
7.7
3.3
89.0
-Cba
96.7
Nab
2834
12.0
9.3
78.7
-Cba 90.7
Eab Ebc
2953
96.9
0.8
2.3
Eac
96.9
Eac
96.9
Eab A Cbe
2975
1.3
97.1
1.5
Cac
97.1
Cac 97.1
Nab A Ebc ac
g
2678
58.6
3.3
38.1
CCac
96.7
-Cac 96.7
Nab A Chc
2680
0.6
82.8
16.6
- Fac
99.4
- Eac
99.4
Cab Nbc
2300
0.6
67.8
31.7
Eca
99.4
Eab A Ebc
2953
1.2
7.0
91.8
-Cca -Eca 99.9
93.0
Eab Che
ca
g
2975
0.1
82.4
17.5
Cca
82.4
Nab A Ebc
2678
2.5
14.1
83.4
-Cca
85.9
Eab Cac
2834
0.5
82.9
16.6
-Ebc 99.5
Eab Nac
2439
0.6
3.9
95.5
Nbc
95.5
Nbc
95.5
Nab A Eac bc
h
2439
55.7
7.5
36.8
-Cbo
92.5
-Ch
92.5
Nab A Cac
2497
1.1
82.3
16.6
Ebc
98.9
Ebc 98.9
Table 8: Meta-inferential results for RoBERTa+SE trained on SNLI+L13.3 (filtered).
16

---

Published as a conference paper at COLM 2025
input items
item
count
E
N
SC
SCV
ETV
Cab
3237
0.8
83.7
15.5
Cba
83.7
-Eba
99.2
Eab
ba
3368
8.2
4.0
87.7
Cba
96.0
Nab
3219
11.3
10.9
77.7
-Caa
89.1
Eab A Ebc
3368
92.6
1.6
5.7
Eac
92.6
Eac 92.6
Eab
Cbe
3368
1.7
94.7
3.6
Cac
94.7
Cac
94.7
Nab A Ebc
g
3219
58.2
4.6
37.2
Cac
95.4
CCac 95.4
Nab A Chc
3219
1.1
81.8
17.1
Eac
98.9
Eac
98.9
Cab A Nbc
3237
0.6
65.5
33,9
Eca
99.4
Eab Ebc
3368
1.2
7.7
91.1
-Cca Eca
92.3
Eab A Che
ca
8
3368
0.1
80.9
18.9
Cca
80.9
99.9
Nab A Ebc
3219
2.3
15.2
82.5
-Cca
84.8
Eab Cac
3247
0.9
79.8
19.3
-Ebc
99.1
Eab A Nac
2858
2.9
6.1
91.0
Nbc
91.0
Nbc
91.0
Nab A Eac bc
h
2858
52.6
8.7
38.7
-Cbe
91.3
-Ch
91.3
Nab A Cac
2951
1.3
80.9
17.9
Ebc
98.7
Ebc
98.7
Table 9: Meta-inferential results for RoBERTa+SE trained on SNLI=LI3.3 (unfiltered).
C LLM prompts for generating NLI data
This section contains the full text of the prompt provided to the LLMs (also available in the code repository). Here, we display system; user: and assistant: tags for legibility, but the exact formatting of the prompt text depends on the language-model specific template. The {(premise) } tag was filled in with the supplied premise and was the only aspect of the prompt that varried between items.
system: You are native speaker of English and an expert annotator of various syntactic and semantic linguistic phenomena.
have been asked to help create a Natural Language Inference (NLI) dataset. Given caption of photo, your task to write an alternate caption that relates the original caption according to each of the three NLI Labels ("entailment", contradiction", and "neutral").
Ne show you the caption
a photo. will not show you the photo. Using only the caption and what you know about the world:
1. [entailment]: Write one alternate caption that is definitely true description of the photo.
Example: For the caption "Two dogs are running through field." " you could write "
There are animals outdoors."
2. [neutral]: Write alternate caption that might true description of the photo. Example: For the caption "Two dogs are running through field." " you could write " Some puppies are running to catch stick."
3. [contradiction]: Write alternate caption that is definitely false description of the photo.
EExampl: For the caption "Two dogs are running through field. " you could write
The pets are sitting on a couchl"
This is different from the maybe correct category because it's impossible for the dogs to be both running and sitting.
When you get the caption from the user, please format your response as a JSON object as follows:
17

---

Published as a conference paper at COLM 2025
"contradiction": "your 1st caption", "entailment": "your 2nd caption", "neutral": "your 3rd caption"
user: Premise: A smiling holds small child wearing pink and red. assistant:
"contradiction": "A man is hugging his dog.", "entailment": "A is smiling.", "neutral": "A is holding his daughter."
user: Premise: Two teams, one in pink and one in white, play lacrosse on field
assistant:
"contradiction"; "Swimmers race to the finish.", "entailment": "Teams compete outdoors.", "neutral": "The pink team is winning."
user: Premise: Asian boy orange, white, and black coat celebrating St. Patric k's day with a soda and green balloon.
assistant:
"contradiction": "A boy dressed "entailment": "a young child holds a sode and balloon for st. pats day", "neutral": "a young christian celebrates the holidays"
black hanging from a jungle gym.",
user: Premise: An Indian dancer actress is performing stage in the spotlig ht, seated on the floor dramatic pose and wearing a blue dress.
assistant:
{
"contradiction": "Jugglers performing for group of children.",
"entailment": performer sitting under a spotlight.",, ",
"neutral": "An actress or dancer are being critiqued their audition."
user: Premise: A brown dog is running across the snowy ground.
assistant:
"contradiction": "A dog is asleep.", "entailment": "A dog is outside.", "neutral": "A lab is in the snow."
user: Premise: A older lady is working with a mix master while two young boys st and next to her with aprons on helping her.
assistant:
"contradiction"; "A lady and two boy eating waffles their kitchen table." "
"entailment": "A lady mixing something." "
"neutral": "A lady teaching her grandchildren how to bake cake."
user: Premise: Young female examining a specimen in microscope.
assistant:
{
"contradiction": "A young female is asleep",
"entailment": young female looking into a microscope",
"neutral": "A young female is looking at a blood sample"
user: Premise: A girl in ethereal, blue costume holding decorated hula hoop gazes intently into the camera.
assistant:
"contradiction": "a girl is dead", "entailment": "a girl in costume",
18

---

Published as a conference paper at COLM 2025
"neutral": "a girl is five"
user: Premise: two women sitting at a table, one looking down and the other on laptop, with banner that says sweden in spanish behind them. assistant:
"contradiction". "two people are staring directly at each other.", "entailment": "A couple of women are at a table.", "neutral": "Two women are part of language club in their college."
user: Premise: A person wearing bright pink wig and yellow shirt, facing y looking at the ocean.
assistant:
"contradiction": "A person wearing shirt that is red.", "entailment": person is standing beside the ocean.", " "neutral": "A person stares at the ocean as they contemplate the drastic dec isions they just recently had to make."
user: Premise: ((premise))
D Generative LLM details
All three LLMs were run using the Ollama API (version 0.5.10). Model details are as follows:
architecture
parameters
quantization
Llama3.2
Llama
3.21B
Q4_K.M
Llama3.3
Llama
70.6B
Q4 K M
DeepSeek-R1
Llama
70.6B
Q4 K M
E NLI model training details
The BERT models were trained with batch size of 64 using the Adam optimizer (Kingma et al., 2015) with learning rate of 0.001, ß1 =0.9 and B2 =0.999, e=le-08, and zero weight decay (default PyTorch settings). The models were given simple linear classification head for the NLI task, which took the sequence [CLS] token as input. Only the final two layers of the BERt base were trained while the first 10 layers were kept frozen.
The RoBERTa+SE models were trained with a batch size of 32. Following Sun et al. (2020), we used the AdamW, optimizer (Loshchilov & Hutter, 2019) with a learning rate of 2e-5, B =0.9, $2=0.98, e=10e-8, and weight decay of 0.01 (except for on bias and layer norm parameters which received weight decay).
All models were trained for 10 epochs and the checkpoint of the best validation epoch (based on the vanilla SNLI dev set) was used for testing.
19


Reviews:
[{"review_id": "lGtevrMpBS", "reviewer": "Reviewer_UYbC", "summary": "The paper aims to further understand logical properties in the NLI task. To do so, it uses LLMs to generate examples from SNLI, specifically using hypotheses from SNLI as premises and asking LLMs to generate new NLI hypotheses from existing hypotheses. The paper then propagates NLI labels from the existing examples to the new examples.", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper uses LLMs to generate a dataset.", "reasons_to_reject": "While I am very partial to the idea of analyzing NLI datasets, this paper does not seem relevant to COLM. I fail to see how understanding the NLI task fits under \"COLM is an academic venue focused on the study of language modeling, broadly defined, with the goal of creating a community of researchers with expertise in different disciplines, focused on understanding, improving, and critiquing the development of LM technology.\" I am 100% for \"we formulate three\npossible readings of the NLI label set and perform a comprehensive analysis of the meta-inferential properties they entail\" but this seems to belong at a *ACL or StarSem.\n\nNo manual verification/validation is done on the new dataset. While there is justification for the rules used to propagate labels, this should still be verified and reported. What percentage of sampled examples is the propagated label correct/incorrect? How does this breakdown for the different NLI labels?", "questions_to_authors": "Why is there no RoBERTa baseline trained on just SNLI in Table 3?\n\nHow exactly does the performance in Table 3 demonstrate that the \"generated NLI data adequately replicates the distribution of the human-generated SNLI dataset\"?\n\nHow many  “perfect items” in the SNLI train set are there (line 218)?\n\nIn 6.1, how can we conclude that BERT \"performs about a bit worse on Ll3.2 and DS-R1 and a bit better on Ll3.3 in comparison to the SNLI test set” if the data isnt annotated? Are the labels just propogated based on 234-237? If so, what verification/manual sampling was done to verify these rules for propogating labels.\n\nIn Table 3, in each row, the E, C, N values sum to one. But what are we comparing to? Are about 1/3 of the examples supposed to be E, C, N?\n\nWhat is gained by reporting \"percentage point increase over the hypothesis-only model”?\n\nAdd citations for hypothesis-only models in the context of NLI:\nhttps://aclanthology.org/N18-2017/\nhttps://aclanthology.org/S18-2023/\nhttps://aclanthology.org/L18-1239/", "extra_scores": {}}, {"review_id": "dq7CvPmLmA", "reviewer": "Reviewer_yGJG", "summary": "The paper gives the first side-by-side formal semantics for the three SNLI labels: (i) material conditional, (ii) strict conditional, and (iii) existential import strict conditional and turns those definitions into a meta-inference test suite that measures a model’s logical self-consistency.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting formalization of the three SNLI labels under three competing logics. Mapping entailment/contradiction/neutral to three explicit logics reveals which meta-rules (symmetry, transitivity, trichotomy, etc.) are valid under each reading\n- Evidence that SNLI and models trained on it are implicitly adopting an existential-import modal semantics rather than classical or pure strict implication", "reasons_to_reject": "- The work focuses on SNLI, whose artifacts and shallow heuristics are well documented. Analysis on other datasets (e.g., MNLI) would have been beneficial. Similarly, including zero-shot results with contemporary LLMs would have strengthen the claims\n- This work seems better aligned with *CL or TACL that emphasize formal semantics", "questions_to_authors": "- Can the LLM that was used to generate data affect the results if its own heuristics align with the target logic?", "extra_scores": {}}, {"review_id": "iZzT7Wj8W8", "reviewer": "Reviewer_NiGL", "summary": "This paper revisits the interpretation of Natural Language Inference (NLI), a widely used task for evaluating language models' understanding of inference. While NLI plays a role in natural language understanding benchmarks, the logical properties of the task are not always clearly defined. The authors outline three possible interpretations of the NLI label set (material conditional, strict conditional and existential import) and examine their meta-inferential implications. They offer an extensive review on various meta-inferential inferences and test their proposed interpretations, and point out a gap in the existing literature: the “NLI relations are left undefined so that it is not clear what the relations are or in which logic they can be expressed.” However, the authors stop short of analyzing these relations in terms of standard relational properties such as transitivity, reflexivity, or symmetry (though an example in the introduction alludes to the symmetry of contradiction).  Using the SNLI dataset, they analyze examples with shared premises and supplement this with data generated by language models to evaluate the consistency of model behavior.  The findings offer a view of how different interpretations of inference may be reflected in NLI data and model responses. The discussion of future work offers directions for refining the task and enhancing its interpretability.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a well-written paper that offers an extensive theoretical review, effectively supporting the identified gaps in the existing literature. The generated data tests the introduced premises.", "reasons_to_reject": "While this may not be a reason for rejection, one limitation of the paper is the lack of discussion regarding the choice of language models. The authors do not provide a rationale for why these specific models were selected for the tasks, or why they might be particularly suitable for assessing meta-inferential properties. It would be helpful to know whether similar behavior would be expected from other models, especially given the generality of the claims.", "questions_to_authors": "See above", "extra_scores": {}}, {"review_id": "bo7w5ZDKn7", "reviewer": "Reviewer_ndSY", "summary": "This paper investigates how NLI models interpret inference relations and whether these interpretations align with formal logical notions. The authors propose three logical readings of NLI labels (material conditionals (MC), strict conditionals (SC), and existential import (EI)) and examine which best aligns with model behavior. They also curate datasets using the SNLI corpus, newly generated NLI items from large language models (LLaMA3.2, LLaMA3.3, and DeepSeek-R1), and an inferred test set created by composing premise–hypothesis chains from the previous datasets. NLI models (BERT and RoBERTa+SE) are evaluated after being trained on various combinations of these datasets. Results show that models perform well on meta-inference tasks overall, with RoBERTa+SE trained on SNLI and DeepSeek-R1 data showing the highest alignment. The findings support the EI reading as the most compatible with the behavior of current NLI models.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides clear and well-supported explanations for the problem. It offers a much-needed critical analysis of widely used NLI datasets such as SNLI and discusses its logical inconsistencies that have been overlooked in prior work.\n\nThe paper clearly explains and compares three ways to understand NLI labels which makes the analysis more solid and connects the results to formal logic.\n\nThe use of examples generated by LLMs and the construction of inferred test sets based on compositional relations is novel.\n\nThe authors test model performance on different combinations of datasets, which helps show how data sources affect the models’ reasoning. This also creates useful directions for future research to build on.\n\nSince NLI continues to be a benchmark task in NLP, the paper’s focus on inference about inference is relevant for understanding what NLU models are actually learning.", "reasons_to_reject": "The study is an analysis of existing  models.", "questions_to_authors": "You use LLMs to generate new NLI examples. How do you make sure these examples follow the same reasoning or assumptions as the original human-written ones? Did you do any quality checks or human evaluation to confirm this? It would have been helpful to include more details on this in the paper.\n \nThe paper introduces three readings of NLI relations (MC, SC, and EI), but in the discussion, it only refers to two modal readings. I understand that the MC reading is set aside, but this could be confusing for readers. Could you clarify why only two readings are discussed at that point?", "extra_scores": {}}]

Author rebuttal:

Meta-review: First, it is my opinion that the paper’s contributions are well within the scope of COLM. LLMs are both fine-tuned and evaluated on NLI-style tasks, so it seems to me that a paper which uses LLMs as a tool to further scientific understanding of NLI should be welcome at this conference. Similarly, the paper draws on formal semantics to organize its analysis and experiments, with the aim of furthering understanding of NLI datasets and model behavior, not as contributions to formal semantics itself. Concerns about the paper’s fit within COLM notwithstanding, overall reviews are positive; the AC recommends the paper for publication at COLM.

The paper offers two primary contributions:

(1) A clarifying discussion of the possible logical interpretations of NLI inference relations, as material implication (MI), strict conditional (SC), and existential import (EI), along with the meta-inferential rules that each interpretation supports.

(2) A thoughtful set of experiments to empirically test which of these interpretations best explains the behavior of models trained on NLI datasets, as well as LLMs prompted in a similar fashion to original NLI instructions.

Suggestions

(1) Reviewers ndSY and UYbC both raise concerns about lack of manual validation/analysis of generated items. Because the data is generated for descriptive analysis purposes rather than prescriptive evaluation purposes, I believe inclusion of a small-scale manual analysis and discussion would be adequate to address this concern.

(2) In the AC’s opinion, the work would benefit from some qualitative analysis or examples. E.g. from Table 4, what is an example of a contradicted pair (a,b) that is neutral when flipped (b,a), i.e. row Cab, column N? Does this example support the hypothesis that the model is taking an EI approach rather than SC? Or could there be some other explanation of the data? (E.g., is it possible that b’s interpretation depends on a in some way such that the meaning of these sentences changes when their order is swapped?)

(3) Lastly, a very relevant recently published paper the authors may wish to include in their discussion: https://aclanthology.org/2025.naacl-long.130/ (This is not a weakness of the current submission since the related work is contemporaneous.)
Decision: Accept