Conditional Natural Language Inference
Authors: Youngwoo Kim
Razieh Rahimi
James Allan
Venue: EMNLP 2023 Findings (2023)
Keywords: Natural language inference
NLI
explanation
contradictory aspect
token-level explanation;interpretable model

Abstract: To properly explain sentence pairs that provide contradictory (different) information for different conditions, we introduce the task of conditional natural language inference (Cond-NLI) and focus on automatically extracting contradictory aspects and their conditions from a sentence pair. Cond-NLI can help to provide a full spectrum of information, such as when there are multiple answers to a question each addressing a specific condition, or reviews with different opinions for different conditions. We show that widely-used feature-attribution explanation models are not suitable for finding conditions, especially when sentences are long and are written independently. We propose a simple yet effective model for the original NLI task that can successfully extract conditions while not requiring token-level annotations. Our model enhances the interpretability of the NLI task while maintaining comparable accuracy. To evaluate models for the Cond-NLI, we build and release a token-level annotated dataset BioClaim which contains potentially contradictory claims from the biomedical domain. Our experiments show that our proposed model outperforms the full cross-encoder and other baselines in extracting conditions. It also performs on-par with GPT-3 which has an order of magnitude more parameters and trained on a huge amount of data.

Conditional Natural Language Inference
Youngwoo Kim, Razieh Rahimi and James Allan University of Massachusetts Amherst [youngwookim, rahimi, allan @cs. umass edu
Abstract
Question: In patients with advanced diabetes, does treat- ment with antihyprrennsvves improve renal function pro- against cardiovascular incidents?
To properly explain sentence pairs that provide contradictory (different) information for different conditions, we introduce the task of conditional natural language inference (Cond-NI. Cond-NL.I) l and fo- cus automatically extracting contradictory as- pects and their conditions from sentence pair. Cond-NLI can help to provide a full spectrum of information, such when there are multiple answers to question each addressing specific condition, reviews with different opinions for different conditions. We show that widely-used feature-atte aatribbtttiiiii explanation models not suit- able for finding conditions, especially when sen- tences long and written independently. We propose simple yet effective model for the NLI task that can successfully extruct conditions while not requiring token-level annotations. Our model enhances the interpretability while main- taining comparable accuracy. evaluate Cond- NLI, present token-level annotated dataset B oC which contains potentially contradic- tory claims from the biomedical articles. Experi- ments show that our model outperforms full cross-encoder and other baselines extracting conditions. It also performs on-par with GPT-3 which has an order magnitude more parameters and trained on huge amount of data.
Claim
Interpretation Losartan prevents more
cardiovascular morbidity and death than atenolol for similar reduction in blond pressure and is hetter tolerated. [Ans: Yes]
Claim2: Although bedtime dose of doxazosin can significantly lower the blood pressure, it can also increase left ventricular diameter, thus increasing the risk of congestive heart failure. [Ans: No]
Table 1:
An example from the BioClain im dataset. To- kens in red indicate opposite outcomes (contradiction), and yellow ones indicate different conditions (neutral).
contradict each other, they address different condi- tions such as patient groups or treatments.? 2 Given this difference, the two claims are not actually con- tradictory, despite reporting contradictory answers. Neither claim is entailed by the other, thus the most suitable NLI category for this pair is neutral. Nev- ertheless, classifying this claim pair as neutral in- troduces challenge in providing comprehensive range of answers for given question. This is be- cause unrelated claims are also classified as neutral, and mining a large set of neutral-labelled claims to provide a broad spectrum of answers is not effi- cient.
1 Introduction
We introduce the task of Conditional Natural Lan- guage Inference (Cond-NLI) that extends the tradi- tional natural language inference (NLI) to be more suitable for finding a full spectrum of information. The original NLI task involves inferring an entail- ment contradiction relationship between pair of sentences: premise and a hypothesis. This is typically modeled as three-way classification, predicting if the premise entails, contradicts, or remains neutral to the hypothesis.
We develop modeling framework to capture the relationship between pair of sentences that pro- vides different answers under diverse conditions. Such sentence pairs are henceforth referred to as since none of the entail- ment, contradiction, or neutral classes of NLI pre- cisely describes their relationship.
Table shows an example of two claims from biomedical articles (Dahlöf et al., 2002) and (Mat- sui et al., 2008), which are included in the Poten- tially Contradictory Claims (PCC) corpus (Alamri and Stevenson, 2016). Although the claims seem to
Cond-NLI includes two token-level tasks is to identify contradictory tokens that embody tradictory aspects and the second identify neu- tral tokens that indicate conditions that are not en-
OOur code and dataset are available at https:// gi tiub.b ctchuuu.ccmm//// wtt.///c//yyoooeeee www.:/ooogwooooomommo cond=nli
Losartan and doxazosin are hoth antihypertensives.

---

tailed by the other sentence. The focus of this study is determine different conditions in a pair of conditionally-compattie sentences. For the ex- ample pair in Table I, the segments highlighted in yellow represent the condition tokens. Contrary to NLI, where an ordering is specified between paired sentences via the roles of premise and hypothesis, paired sentences in Cond-NLI do not require such order because the contradiction holds in both directions.
includes conditionally-comppan sentence pairs. Such pairs require the identification of neutral to- kens in the presence of contradictory tokens. Com- pared to other token-level explanation datasets such as e-SNLI and MNLITag (Camburu et al., 2018; Kim et al., 2020), which are built on NLI cor- pora (Bowman et al., 2015; Williams et al., 2018), BioClaim has longer hypothesis sentences. This characteristic introduces additional complexity in the selection of non-entailed tokens.
Automatic identification of different conditions in conditionally-compaae
Perturbation-base methods (Ribeiro et al., 2016; Kim et al., 2020) have shown to be effective identifying tokens that contribute to contradiction or neutral labels when evaluated on e-SNLI (Cam- buru et al., 2018) or MNLITag (Kim et al., 2020), However, show that these perturbation- explanation models face challenges in accurately identifying condition tokens when hypothesis sen- tences are long and contain a large number of non- entailed tokens (conditions in Cond-NLI).
sentence pairs allows us to summarize and provide a full spectrum of an- swers in a form where users are not overloaded with excessive information. This is of particular practi- cal importance as it has shown that there are usually multiple answers to user's question in different do- mains, such as biomedical (Alamri and Stevenson, 2016), e-commerce (Santos al., 2011), and fac- toid dnuestion-nswwere (Min et al., 2020), where the difference between answers/opinions is their provided conditions.
Extensive experiments on the BioCla lm and SeiEntsBank (Dzikovska et al., 2013) datasets show that our PAT significantly outperforms strong and state-of-the-art baseline models. Against Intruct- GPT (Ouyang al., 2022) and ChatGPT (Ope- nAI, 2022), PAT shows better performance on the SciEntsBank dataset and comparable performance on the BioClain dataset, while PAT has a sig- nificantly smaller number of parameters. While our PAT model slightly underperforms the cross- encoder BERT model on the original NLI task, its enhanced interpretability enables effective fine- grained token-level inference required for Cond- NLI.
We propose Aattial-Ateettion model, PAT, a sim- ple yet effective model for natural language infer- ence that can address the Cond-NLI task. PAT predicts an NLI label for a sentence pair from the intermediate labels for their partitions. The inter- mediate labels for partitions of sentences can be subsequently used attribute these labels into the token-level.
The NLI token-level attributions align closely with the objective of Cond-NLI. Different condi- tions in a claim pair would cause an NLI model to predict the pair to be neutral. Thus, identify- ing the tokens responsible for triggering neutral labels could serve technique to detect different condition tokens in Cond-NLL Similarly, contra- dictory tokens of Cond-NLI can be attained from attributing contradiction label in NLI. Finally, PAT effectively solves Cond-NLI through training with sentence-level NLI data, without requiring task- specific token-level annotations.
2
Related Work
2.1 NLI corpora
Many NLI datasets (Giampiccolo et al., 2007; Bow- man et al., 2015; Williams al., 2018; Thorne et al., 2018) have limited diversity in contradictory pairs as hypothesis sentences are crafted by anno- tators, fails to reveal potential challenges (Zhou and Bansal, 2020). Our sioolall a dataset (Sec- tion 3.2) differs in that both sentences of (poten- tially) contradictory pairs are written independently, resulting in paired sentences that have large lcxical differences even when they have high semantic sim- ilarity. Moreover, most of the potentially contradic- tory sentence pairs contain a large amount of infor- mation that is not mentioned in both. While some methods (Ribeiro et al., 2016; Kim et al., 2020;
To evaluate different models for Cond-NLI, we build (and make publicly available) the BioClaim dataset, an extension of an exist- ing corpus initially built to assist systematic reviews (Alamri and Stevenson, 2016). The Bioclaim dataset provides a challenging bench- mark for the NLI models. contrast to the Sci- EntsBank dataset (Dzikovska et al., 2013), which lacks contradictory sentence pairs, BioClain

---

Wu al., 2021) have shown token-level prediction capabilities on e-SNLI (Camburu et al., 2018), we found them less effective on the BioClaim dataset that presents new challenges.
not suitable for solving the Cond-NLI task. Wu et al. (2021) proposed the Explainable Phrasal Reasoning (EPR) model, which aligns phrases ex- tracted from the premise and hypothesis, predicts phrase-level labels, and subsequently combines them to predict the sentence-level NLI label. Stacey et al. (2022) proposed span-level reasoning (SLR) model, which partitions hypothesis into multiple pieces and predicts labels for each of the hypoth- esis partitions and the premise. Feng et al. (2022) also segment a hypothesis into spans and predict seven logical relations on each of them, which are used for the final sentence level predictions.
2.2 Explaining II. models
One approach to explain neural NLI models is to select a subset of the input tokens that are important for the model decision as a rationale. A token being important to the model decision is justified by how it is connected (gradient or active weights) to the model decision (Gilpin et al., 2018) how its exis- tence (or removal) affects the model decision (Lei et al., 2016; Dououg et al., 2020).
These models use static partitioning for the premise hypothesis, which constrains the gran- ularity of fine-grained information the predeter- mined span boundaries. In contrast, we propose training model on randomly partitioned hypothe- ses for enhanced granularity, allowing label predic- tions across diverse boundaries.
In the existing NLI datasets (Bowman et al., 2015; Williams et al.. 2018), the hypothesis with a neutral label has mostly small number of to- kens that are not entailed by the premise. In these samples, individual not-entailed (neutral) tokens are critical to the NLI model decision and can be easily identified by feature-attribution explana- tions (Ghaeini et al., 2018; Liu et al., 2018; Kim et al., 2020). However, we show that these explana- tion models are not effective in identifying neutral tokens on BioClaim. We discuss the reasons for the failures in subsection 5.2.
Moreover, for Cond-NLL, partitioning the hy- pothesis into more than two segments unneces- sary. Partitioning into two enables using an ag- gregation function based on matrix multiplication, which is not only simpler than the fuzzy logic- based technique (Wu et al., 2021; Stacey et al., 2022) but also demonstrates higher accuracy for NLI.
Another direction for explaining NLI is to gen- crate natural language explanation (NLE) (Kumar and Talukdar, 2020; Zhao and Vydiswaran, 2021). often trained in a supervised manner using a dataset such as e-SNLI (Camburu et al., 2018). These models are not capable of identifying not-entailed tokens when the prediction for a sentence pair is contradiction. When hypothesis contains both contradictory information and not-entailed infor- mation in relation to premise, the assigned NLI label is contradiction. Some NLE models (Cam- buru et al., 2018; Kumar and Talukdar, 2020) could have limitations in capturing all the not-entailed tokens in low similarity sentence pairs.
Li and Srikumar (2019) utilized fine-grained logic for sentence-level NLI but didn't show its use for partial-entailment or token-level predic- tions. Levy et al. (2013) adapted textual entail- ment models for facet level partial entailment us- ing lexical and syntactic matches. This approach, however, does not apply to NLI models based on context-sensitive architectures such as Transform- ers (Vaswani al., 2017).
Cond-NLI task and datasets
Attention-based explanations (Thorne et al., 2019; Jiang et al., 2021) can identify which to- kens are closely related, but are not calibrate well for identifying not-entailed contradictory tokens. Both a non non-entailed and entailed token of hypoth- esis can have high attention the most related token in the premise, complicating the distinction using attention scores.
3.1 Task definition
Our Conditional Natural Language Inference (Cond-NLI) is formally defined as a token-level classification task, aligning with the definition of the existing task of partial entailment (Levy et al., 2013). Given a pair of claims (p, h)) and span s from h, the goal is to classify s as either neutral or contradictory to p. Note that, neutral tokens are considered equivalent to condition tokens.
2.3 Interpretable NLI models
There are a few existing models with interpretable architectures for the NLI task, however they are

---

3.2 BioClaim
was built assess student answers, and formatted as an entailment task by taking a student answer as a premise and a reference answer as hypothesis. annotated a facet of the hypothesis is entailed by the premise, where a facet is tuples consisting of two words. Following data filtering process similar to one used in SemEval-2013 (Dzikovska et al., 2013), the test split contained 9,974 'Ex- pressed' and 10,516 'Unaddressed' facet-level an- notations.
To evaluate our model, we built the BioCla a dataset by adding token-level annotations to isting corpus of potentially contradictory claims (PCC) (Alamri and Stevenson, 2016). PCC con- sists 24 closed-form research questions and a total of 259 claims relevant to the questions. The claims are aligned with the relevant questions and are also annotated with their answer (Yes or No) to the relevant questions. Claim pairs relevant to the same question with different answers the ques- tion are potentially contradictory or conditionally- compatible.
4 Partial-Attention NLI Model
The typical effective approach for text-pair classi- fication, such as the NLI task, using Transformer- based language models such as BERT (Devlin al., 2019), by concatenating the text pair as input, which we refer to as full cross-encoder BERT. Specifically, cross-encoder BERT takes the con- catenation of premise p and hypothesis h, denoted by po h, taking the [CLS] token vector sentence representation, and outputs classification probabil- ity y
From 24 question groups, we selected pairs with opposite answers (Yes-No). Since each group has different numbers of Yes or No labeled claims, the combinations of opposite-answed pairs range from 3 to several hundred. We limit the maximum num- ber of pairs from each group to 20, prioritizing those with greater term overlap when sampling.
Annotators were given sampled claim pair and asked to annotate tokens that indicate opposite out- comes (corresponding to the contradiction label) and tokens that indicate different conditions in the two claims (corresponding to the neutral label). While NLI has three classes, we only annotated tokens that are related to contradiction and neu- tral, as the entailment tokens are expected to be the remaining tokens that are not contradiction nor neutral.
y= f(po h).
Output y in the NLI task is a 3-dimensional vec- tor representing the probabilities of the entailment, neutral, and contradiction classes.
P1
Agg
We employed nursing college students as annota- tors. The resulting dataset consists of 14,915 anno- tated tokens, including 1,862 contradiction tokens and 145 neutral tokens, all of which are derived from 285 claim pairs. Using Cohen's Kappa (Co- hen, 1960), we observed a moderate agreement score of 0.46. Out of all the claim pairs, 195 re- ceived multiple annotations; we randomly selected two annotations from these pairs to measure agree- ment.
BERT
BERT
h1
h2
Gross nutional saving has increased over the past 5 years.
Gross national saving
[MASK]
this year.
highest
Figure I: The architecture of proposed PAT model. p represents the tokens of the premise. h and hy are subsets hypothesis tokens. Agg combines two inter- mediate output @1 and P2 as in Eq.
In the evaluation of Cond-NLI using BioClaim, each claim pair generates multiple Cond-NLI prob- lems. This occurs for every token in the claim pair (tokenized by spaces) and for each token-level class, namely neutral and contradiction.
We propose the Partial-ATtention model, PAT, that predicts the NLI label for and h based on two intermediate NLI labels for two subsequences of h. Specifically, the hypothesis h is partitioned into two subsequences h and h2g Premise p is sepa- rately concatenated with h and h2 and fed into the encoder f', which outputs intermediate predictions Pi and p2, respectively. Each intermediate output is
3.3 SciEntsBank
We also used SciEntsBank (Dzikovska et al., 2012), a dataset with fine- mra-grai entailment annotations, for our evaluation due to its task similarity with neu- tral token classification in CoLd-NLI. SciEntsBank

---

P1
MultiNLI
SNLI
SciTail
Entailment
Neutral
Contradict
Cross Encoder
0.829
0.887
0.925
Entailment
Entuilment
Neutral
Contradict
PAT
0.793
0.870
0.889
P2
Neutral
Neutral
Neutral
Contradict
+ fuzzy logic 0.763
0.844
0.860
Contradict
Contradict
Contradict
Contradict
four segments
0.744
0.831
0.818
Table Logical behavior for combining the intermedi- ate NLI decisions. Gray cells show the final NLI label.
Table Classification accuracy of the cross-encoder baseline, proposed PAT, and alternative architectures (ablation study) for sentence-pair NLI.
a probability distribution over three classes. The in- termediate NLI labels are then aggregated to obtain the final NLI label for the pair p and h:
and Tijk otherwise. The final NLI label y is obtained by the matrix multiplication:
(p. h) Agg(p1; P2)
(2)
Agg(p1 p2) pl -T -
(5)
P1 = f(po
P2 f
(3)
Cond-NLI. Once the PAT model is trained, the intermediate decision predictor I' can be used to predict labels for any arbitrary subsequence s within hypothesis, as it would treat o s similarly to either p h1 p h2-
where f' has the same architecture as the function f in Eq. I, but is trained to be robust to partial text segments. The function Agg(-) combines the intermediate outputs to predict the final NLI label. Figure shows the PAT architecture.
While our goal is to predict label for an individ- ual token of h, only feeding one token to the model not ideal due lack of contextual information. Instead, we consider longer spans that contain the token in h. The tokens' final label determined by combining the labels of these spans.
Partitioning hypothesis. For training, h is par- titioned by randomly selecting two indices ig and ies where ig S le- h1 is built from tokens ig to ie of h. h2 is built by concatenating two segments of h with a [MASK] token between them: token I to i, and token i. 1 to the last token of h. Combining intermediate decisions. The pected logical behavior of the aggregation func- tion, when each intermediate decision discrete, is shown in Table 2. For example, when both inter- mediate decisions, P1 and P2, are entailment (the probabilities for entailment are close to I), the final decision should be entailment (entailment proba- bility is close to 1). If one of the two intermediate decisions, for instance p1, is neutral (contradiction) while the other is entailment, then the combined decision inherits the label of P1. If one is neutral and the other contradiction, the final decision should be contradiction. This is similar to the meth- ods proposed by Wu et al. (2021) and Stacey et al. (2022), which are motivated by fuzzy logic.
Specifically, we used sliding windows of size I, 3 and 6 tokens with stride of 1. Let S; denote the set of subsequences that contain the i-th token of h. The probability vector G indicating three NLI classes of i-th token of with respect to p is predicted as:
C
Σ f'(oo))
(6)
S
BES.
where I'(p 8) is probability vector of three classes from the intermediate predictions of PAT.
Experiments
Experimental Settings. Both the full cross- encoder NLI (Eq. 1) and the PAT models are trained by fine-tuning the BERT-base model (Devlin et al., 2019) on the MultiNLI dataset (Williams et al.. 2018) for one epoch, more epochs expected result in over-fitting and lower performance on the BioClaim dataset. For perturbations and token-level enumerations, sentences are tokenized by spaces instead of BERT's subword tokenizer.
To implement this logical behavior, we first model Table 2 with an integer matrix
0
2
M
1
2
2
2
where entailment, neutral and contradiction are rep- resented 0, 1 and respectively. Based on this matrix, we then build a one-hot representation T. T' a rank 3 tensor where Tisk- = 1 if M,j =
5.1 NLI sentence-pair classification
We compare the accuracy of the full cross- BERT and PAT for the original NLI task over

---

three datasets MultiNLI, SNLI (Bowman et al., 2015), and SciTail (Khot al.. 2018). Both mod- els are separately trained and tested on each of the datasets.
match or embedding similarity approaches in clas- sifying neutral/entailed tokens?
Neutral tokens are the ones not entailed by the other sentence in pair. If a token pair from two sentences has similar meanings (high semantic sim- ilarity), one can expect that the tokens are less likely to be neutral. Thus, we consider exact match and word2vec (Mikolov et al., 2013) as baselines to predict neutral/entailed tokens
Table 3 summarizes the accuracy of the PAT and the full cross-encoder models. PAT shows 2% 4% lower accuracy than the full cross-encoder model, however intermediate decisions enhance the interpretability of its predicted NLI class.
We also used the accuracy of the NLI task for an ablation study to compare different design as- pects of our PAT model, higher NLI accuracy is likely to result in good performance for Cond- NLI under similar data distributions. Table in- cludes the accuracy of ablated versions of the PAT model. The "+ fuzzy logic" model replaces the our aggregation function in Eq. 5 with the one from EPR (Wu et al., 2021), a phrased-based NLI model. The "+ four segments" model, in addition to the previous change, splits the hypothesis into four pieces instead of two in PAT. This is based on the observation that EPR model splits hypothesis into an average of four pieces in the SNLI. We observe that replacing our strategies with those used in the existing models results in lower accuracy over all datasets.
A token's entailment score with respect to sen- tence is determined by its highest similarity to the other sentence's 's tokens. For this purpose, we build a similarity matrix S (p/-1a] where Siy indicates the similarity of the i-th token in p to the j-th token in h. In case of exact match, Sij is a binary value indi- cating whether the two tokens are the same or not. With word2vec, Sig indicates the cosine similarity between embeddings of p; and hy. The entailment score of the j-th token in h, hy. with respect to is computed as max; Sty. The neutral score is computed as one minus the entailment score.
In SciEntsBank, a facet s is composed of two tokens of h and we compute the span entailment score as an average of two tokens' entailment score.
RQ2 Is PAT more effective than adapting the existing models for solving Cond-NLI?
5.2 Evaluation Metrics for Cond-NLI
First, we investigate if the (eature-attributie ex- planation models (Ribeiro et al., 2016; Zeiler and Fergus, 2014: Kim et al., 2020) can solve Cond- NLI. These methods assign an importance score to each input feature based on its contribution to the predicted class probability. Given a premise- hypothesis pair and an NLI model, we use feature attribution explainers to obtain importance scores of input tokens the predicted probability for the neutral class by the NLI model. Interpreting these importance scores as tokens' neutral scores, feature attribution explainers can solve the Cond-NLI task.
We report accuracy and Fl score as the main met- ries for the evaluation of Cond-NLI. For SciEnts- Bank, we report macro-averaged Fl which av- erage of Fl scores for each of 'Expressed' and "Unaddressedd labels (Dzikovska et al., 2013).
Many of the baseline methods such LIME SLR, assign (importance) scores to tokens, and do not provide binary class labels. To perform a meaningful comparison that demonstrates the po- tential of each method, we convert token scores into binary class labels by applying threshold crite- rion; tokens are assigned to a specific class depend- ing on whether their scores exceeds the predefined threshold. The threshold is determined through evaluating multiple candidate values. The chosen threshold for each model the one that maximizes the model's performance on the validation set.
We include the following perturbation- based methods that are either widely-used for explanation of a black-box classifier or specifically designed for explanation of the NLI task. LIME (Ribeiro et al., 2016) is a widely-used explanation method which attributes the model's prediction to input features (tokens in the NII task). Occlusion (Zeiler and Fergus, 2014) removes one token at time and mca- sures the output changes to score the importance of the removed token. SE-NLI (Kim et al., 2020) is an explanation model that generates leven- explanations for the NLI task. It uses BERT token representation as a feature to predict the impor-
5.3 Baseline methods
We address three research questions in our evalua- tion. Baseline methods are selected and described based on the research question we aim to address. RQ1 Is PAT more effective than the lexical

---

tance score for each token. The training objective for importance prediction is predict the change in the NLI scores when the token is deleted.
Neutral
Contradiction
F1
Acc
F1
Acc
Similarity-bawee
SLR (Span-Level Reasoning) (Stacey et al.. 2022) is an NLI model that makes explicit span- level predictions. However, its span granularity is restricted because it divides hypothesis into spans at noun phrase boundaries. Nevertheless, to demon- strate the limitations of SLR, we converted their span predictions into token or facet level by using method similar Equation 6.
Exact match
0.647
0.538=
word2vec
0.645
0.575F
NLJ MLs-based
Co-attention
0.644
0.538
LIME
0.639i
0.538
0.277
0.872
Occlusion
0.632
0.538
0.246
0.859
SENLI
0.632
0.541
0.292
0.866
SLR
0.624
0.538F
0.280
0.859:
Token-entail
0.638
0.538
0.248
0.8667
Beyond attribtnatri explanation methods, we consider adapting the full cross-encoder NLI model for solving Cond-NLI. The assumption is that if a hypothesis span premise p. then the NLI model would predict neu- tral on (p. s), where span 8 alone treated as a hypothesis. This baseline can demonstrate the ad- vantage of function /' in Eq. over function / in Eq. 1. We refer to this baseline as Token-entail. Token-entail is different from our PAT model in two ways; it uses the full cross-encoder model in Eq. 1 with only single token hypothesis while our model uses sub-sequences of variable length as hypothesis. We did not compare against the full cross-encoder model when the hypothesis is a sub- sequence of longer length, because cross-encoder is not robust to such sub-sequences as input and its performance drops significantly.
PAT
0.657
0.622
0.414
0.871
Large language model
InstructGPT ChatGPT 0.624
0.593
0.673
0.435
0.856
neutral against
0.657
0.459
0.8461
Table 4: Cond-NLI: neutral token and contradiction ken classification results on BioClaim. and 1 indicate that the difference between the method and PAT is sig- nificant p < 0.01 and p < 0.05.
formance in many downstream tasks. To solve Cond-NLI, used the task instruction used for BioClain annotation and claim pair build prompt the LLMs. The LLMs are asked generate words that correspond to either neutral or contradiction (Figure 2) For SciEntsBank, we included student answer, a reference answer, and facet word pair in the prompt (Figure 3) and then asked the L.MMs to determine if the facet is entailed by the student's answer.
We developed the Co-attention baseline in- spired by the work of Jiang et al. (2021). Co- attention uses the attention scores from Trans- former encoder intuition is that in NLI trained model, high attention score between token pair across two sentences indicates that the tokens are likely se- mantically similar, which makes their representa- tions can be compared through attention. Thus, a token that is neutral is likely to have small atten- tion scores to the tokens of the other sentence. The normalized attention scores of token to the tokens of the other sentence averaged over all self- attention heads in all layers. The obtained scores are used as similarity matrix S, similar to the exact match baseline.
The implementation details of the baseline meth- ods are described in Appendix A.
token similarity proxy. The
5.4 Results
Tables 4 and 5 show the performance of all compared methods on the Cond-NLI over the BioClain and SciEntsBank (Dzikovska et al., 2013) datasets. On both datasets, the proposed method, PAT, outperforms other NLI-based meth- ods with the only exception of LIME on contra- diction in terms accuracy. However, this gap is not statistically significant and Cond-NLI largely outperforms LIME when evaluated with F1.
We suggest the following reasons for the poor performance of explanation models LIME, Occlu- sion, and SE-NLI on the Cond-NLI, especially for the neutral class. First, the hypothesis contains many tokens that are not entailed. Perturbing a small number of tokens is likely to lead to the partial removal of neutral tokens. Such pertur-
RQ3 How does PAT compare against GPT-3 based models?
InstructGPT (Ouyang et al., 2022) and Chat- GPT (OpenAI, 2022), which are fine-tuned ver- sions of the large language model GPT-3 (Brown et al., 2020)), have shown good zero-shot per-

---

UA
UD
UQ
Mean
in terms of Fl scores on BioClaim and Scilints- Bank. However, they cannot be used to predict contradicting tokens, thus their performance for contradiction not listed. They outperform PAT in Unseen Domain (UD) split of SciEntsBank.
Similarity-han -hased
Exact match
0.733
0.792
0.753
0.759
word2vec
0.753
0.780
0.756
0.763
NLI-based
Co-attention
0.746
0.700
0.817
0.754
On BioClaim, PAT shows comparable perfor- mance to InstructGPT and ChatGPT, since the su- periority between them varies depending on the metrics and token classes. Note that GPT-3 has 175 billion parameters (Brown et al., 2020), which is more than 1,000 times larger than our proposed model having 1 10 million parameters (BERT-base) On SciEntsBank, ChatGPT is not effective, possi- bly due to the difficulty in connecting a word pair (facet) to the student answer and reference answer. This format might not be frequent in the data that ChatGPT was trained on.
LIME 0.635
0.673
0.663*
0.657
Occlusion
0.494
0,488
0.404
0.462
SENLI
0.542F
0.547t
0.600
0.563
SLR
0.722
0.713
0.6981
0.711
Token-entail
0.714
0.721
0.713
0.716
PAT
0.763
0.778
0.826
0.789
Large language model
ChatGPT
0.655F
0.687+
0.680
0.674
Table 5: Macro-averaged Fl score on the partial entail- ment dataset SciEntsBank. UA (Unseen Answers), UD (Unseen Domain), and UQ (Unseen Question) are splits of the test set. indicates that the difference between the method and PAT is significant at < 0.01.
In BioClaim, the improvements of PAT over all other methods are statistically significant at p-value of 0.01, except for similarity-buse based methods based on FI, where the significance level is at 0.05, Note that none of the NI.I-based method outperformed PAT with statistically significance. The statistically significance was measure by the paired t-test for accuracy and bootstrapping test for the F1 score.
bations would cause negligible changes in model predictions. Simultaneously removing all neutral tokens is also unlikely to have desirable impact on the model decision large removal increases the chance of out-of-distribution inputs and thus unreliable model decision for explanation (Hase et al., 2021).
We also evaluate our PAT on e-SNLI (Cam- buru et al., 2018) and MNLIEx (Kim et al., 2020), two leven-level annotated datasets, to eval- uate its robustness. Although these datasets lack conditionally sentence pairs, limiting their use for comparing models on the Cond-NLI task, they measure the robustness of PAT across diverse datasets. Our model shows competitive per- formance with explanation methods such as LIME and SE-NLI, demonstrating the generalizability of our model. Detailed results are in the appendix C.2 (Tables 7 and 8).
Second, many of annditionally-comaatiin pairs are predicted as contradictory by the NLI models despite the existence of tokens that indicate differ- ent conditions. In this case, identifying different conditions becomes more challenging as the neu- tral probability predicted by the NLI model is very small and effect of not-entailed tokens for the neu- tral probability cannot be observed.
SLR (Stacey al., 2022) also underperformed PAT due to its fixed span segmentation, limiting its ability to infer entailment information for arbi- trary tokens. The performance of the token-entail method, which is based on the full cross-encoder, is not as good as PAT. We further inspected its outputs and found that the token-entail method pre- dicts high neutral scores for functional and generic words, such 'patient', 'study', and 'factors', that are implicitly entailed. These failure examples im- ply that the full cross-encoded model is not robust to partial hypothesis segments and cannot provide meaningful predictions for them.
Conclusion
We proposed PAT, a partial attention model, ca- pable of attributing the model decision into the parts of input. Using PAT, we address the Cond- NLI task, token-level prediction task that ex- plains conditionally-compatiblel claims. We built the Bioclain dataset for Cond-NLI The pro- posed method shows the accuracy up to 8% higher than the best NLI-based baseline method in predict- ing condition tokens.
Exact match and word2vec outperform other NLI-based methods for predicting neutral tokens

---

Acknowledgment
Limitations
This work was supported in part by the Center for Intelligent Information Retrieval and in part by a UMass Amherst Manning/IALS Innovation grant. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor.
The proposed PAT model has limitations compared the full cross-encoder. First limitation is lated to combining intermediate predictions in Eq 5. When intermediate predictions of neutral and con- tradiction are combined, the current model pre- dicts contradiction while neutral predictions are also possible. Similarly, when both intermediate predictions are contradiction, it is possible that they cancel out each other double negation and have entailment or neutral as the gold label. As identi- fied in Appendix C.3, partitioning hypothesis has potential issues that may happen with different fre- quencies in different datasets.
BioClaim dataset has few limitations. Anno- tators have different preferences toward including tokens that could be chunked as syntactic units. For example, one annotator selected "stenosis and hy- pertension" as neutral tokens, while another select "stenosis" and "hypertension", excluding "and"" This could prohibit any system from achieving very high scores.
Due the unbalanced distribution of claims an- swering question with "yes" versus those with "no", some claims are repeated more frequently than others. While this does not affect our evalua- tions, could potentially serve as limiting factor for certain applications.
References
Abdulaziz Alamri and Mark Stevenson. 2016. A corpus of potentially contradictory research claims from car- diovascular research abstracts. Journal of biomedical semantics. 7(1):1-9.
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher Manning. 2015. A large annotated corpus for learning natural ence. In Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, pages 632-642.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Oana-Maria Camburu, Tim Rocktäschel. Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natu- ral language inference with natural language explana- tions. In Proceedings of the 32nd Annual Conference on Neural Information Processing Systems, NIPS 2018, Montreal, Canada, December 3-8, 2018

---

Jacob Cohen. 1960.
coefficient of agreement for nominal scales. Educational and psychological mea- surement, 1033)374
Paraphrasing, RTE '07. pages 1-9, Stroudsburg, PA, USA. Association for Computational Linguistics.
Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Ba- jwa, Michael Specter, and Lalana Kagal. 2018. Ex- plaining explanations: An overview of interpretabil- ity of machine learning. In 2018 IEEE 5th Interna- tional Conference on Data Science and Advanced Analytics (DSAA), pages 80-89. IEEE.
Björn Dahlöf, Richard B Devereux, Sverre E Kjeld- sen, Stevo Julius, Gareth Beevers, Ulf de Faire, Frej Fyhrquist, Hans Ibsen, Krister Kristiansson, Ole Lederballe-Pederser et al. 2002. Cardiovascular morbidity and mortality the losartan intervention for endpoint reduction in hypertension study (life): randomised trial against atenolol. The Lancet, 359(93111:995-1033
Peter Hase. Harry Xie, and Mohit Bansal. 2021. The out-of-distribution problem in explainability and search methods for feature importance explanations. Advances in Neural Information Processing Systems. 34.
Jacob Devlin, Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Zhongtao Jiang, Yuanzhe Zhang, Zhao Yang. Jun Zhao, and Kang Liu. 2021, Alignment rationale for natu- ral ence. In Proceedings of the 59th Anmual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers). pages 5372-5387.
Jay Young. Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. ERASER: A benchmark evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 4443 4458, Online. Association for Computational Linguistics.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SeiTail: A textual entailment dataset from science question answering. AAAI
Youngwoo Kim, Myungha Jang, and James Allan, 2020. Explaining text matching on neural natural language inference. ACM Transactions on Information Sys- tems (TOIS). 38(4):1-23.
Myroslava O Dzikovska, Rodney Nielsen, and Chris Brew. 2012. Towards effective tutorial feedback for explanation questions: dataset and basclines. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 200-210.
Sawan Kumar and Partha Talukdar. 2020. Nile: Natural language inference with faithful natural language ex- planations. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties. pages 8730-8742.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods Natu- ral Language Processing, pages 107-117.
Myroslava O Dzikovska, Rodney Nielsen, Chris Brew, Claudia Leacock, Danilo Giampiccolo, Luisa Ben- tivogli, Peter Clark, Ido Dagan, and Hoa Trang Dang. 2013. Semeval-2013 task 7: The joint student re- sponse analysis and 8th recognizing textual entail- ment challenge. In Second joint conference on lexi- cal and computational semantics (* SEM). Volume 2: Proceedings of the seventh international work- shop on semantic evaluation (SemEval 2013), pages 263-274.
Omer Levy. Torsten Zesch, Ido Dagan, and Iryna Gurevych, 2013. Recognizing partial textual entail- ment. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Vol- ume 2: Short Papers), pages 451-455.
Tao Li and Vivek Srikumar. 2019. Augmenting neu- networks with first-order logic. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 292-302.
Yufei Feng, Xiaoyu Yang, Xiaodan Zhu, and Michael Greenspan. 2022. Neuro-symbolt natural logic with introspective revision for natural language inference. Transactions of the Association for Computational Linguistics, 10:240-256.
Shusen Liu, Zhimin Li, Tao Li, Vivek Srikumar, Valerio Pascucci, and Peer-Timo Bremer. 2018 Nlize: A perturbation-driven visual interrogation tool for an- alyzing and interpreting natural ence models. IEEE rransactions on visualization and computer graphics. 25( 1660.
Reza Ghacini, Xiaoli Fern, and Prasad Tadepalli. 2018. Interpreting recurrent and attention-baan based neu- ral models: case study on natural ence. arXiv preprint arxiv: 1808:0388..
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third pascal recognizing textual entailment challenge. Proceedings of the ACL-PASCAL Workshop on Textual Entailment and
Yoshio Matsui, Kazuo Eguchi, Seichhi Shihasaki, Joji Ishikawa, Satoshi Hoshide. Thomas G Pickering, Kazuyuki Shimada, Kazuomi Kario, JMS-1 study group, et al. 2008. Effect of doxazosin on the left

---

ventricular structure and function in morning hyper- tensive patients: the japan morning surge study. Journal of hypertension, 26(7): 1463-1
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. broad- broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume (Long Papers), pages 1112-1122.
Tomas Mikolov, Iyaa Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositionality. In Advances in neural information processing sys- tems, pages 311 111199
Zijun Wu, Atharva Naik, Zi Xuan Zhang, and Lili Mou. 2021. Weakly supervised explainable phrasal reasoning with neural fuzzy logic. arXiv preprint 109002200..0897..
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and
Luke Zettlemoyer. 2020. Ambigqa: Answering biguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 5783- 5797.
Matthew Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In Euro- pean conference computer vision, pages 818-833. Springer.
OpenAl. 2022. Introducing chatgpt. https:/ operai.com. 06-23.
wwtpbcccg///ooccocccoccc...ooocc http//lllgcccat.....///e Accessed: 2023-
Xinyan Zhao VG Vinod Vydiswaran. 2021. Lirex: Augmenting ence with relevant explanations. In Pro- ceedings of the AAAI Conference Artificial Intelli- gence, volume 35, pages 14532-14533
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow in- structions with human feedback. arXiv preprint arXiv:2203.02155.
Towards robus- Xiang Zhou Mohit Bansal. 2020.
tifying nli models against lexical dataset biases. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8759- 8771.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should trust you?: Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135-1144. ACM.
Rodrygo LT Santos, Craig Macdonald, and ladh Ounis. 2011. Aggregated search result diversification. In Conference on the Theory of Information Retrieval, pages 2610-261.000 Springer.
Joe Stacey, Pasquale Minervini, Haim Dubossarsky, and Marek Rei. 2022. Logical reasoning with span-level predictions for interpretable and robust nli models. In Proceedings of the 2022 Conference on Empiri- cal Methods Natural Language Processing, pages 3809-3823.
James Thome Andreas Vlachos, Christos Christodoulopoulls,, and Arpit Mittll. 2018. Fever: a large-scale dataset fact extraction and verification. In NATCL-HLT.
James Thoree, Andreas Vlachos. Christos Christoddoulopoulo,, and Arpit Mittal. 2019. Generating token-level explanations for natural language inference. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pages 963-9699
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In NIPS

---

A Implementation details
A.1 Baselines
Neutral token prediction with similarity scores
Neutral tokens are the ones that are not entailed by the other sentence in a pair. If a token pair from two sentences has similar meanings (high semantic similarity), one can expect that the tokens are less likely to be neutral. For this purpose, we first build similarity matrix S E RR pix hi where Sij indicates the similarity of the i-th token in p the j-th token in h. In case of exact match, Siy a binary value indicating whether the two tokens are the same or not. With word2vee, Sij indicates the cosine similarity between embeddings of p; and hj.
The entailment score of the j-th token in h, hj.
with respect p computed as.
Entail(p, h, hy) max Sig
(7)
The neutral token score is computed as one mi- nus the entailment score:
Neutral(p, hy) Entail(p, hy hy)
(8)
Word2vec computes the word embedding sim- ilarity scores between all token pairs across two sentences. We adopted the widely used version sereeeewww-eettos-nggaavvve0
For each token, negative of the maximum similarity score was used as score for mismatch. If there no embedding for the word, we check for the exact match and assign score there is the exact match. Otherwise, we assign the score of O.
Co-attention
To extract attention scores we used the same cross- encoder NLI model as used for other NLI-based methods. Co-attention scores tokens based on the attention scores between tokens. The scoring is done in the following steps.
1. We collect normalized attention probabilities. As result we get four dimension tensor of W E R2XLxMxx where L is the sequence length, M is the number of layers, and H is the number of attention heads in each layer. Was denote the attention probability for the i-th token to attend to the j-th token in the k-th attention head the /-th layer.

---

2. We average W over the last two dimensions, which corresponds to different layers and heads, and get a two-dimensional matrix A.
the contribution of the corresponding i-th token T; to the NLI model's neutral probability. This attri- bution score is subsequently used as a token score for Cond-NLI prediction.
Ag ΣΣ W I
(9)
We used the python library version of LIME and used 500 the number of samples.
3. Let [p| be the number of tokens in the premise and [h] to be the number of tokens in the hy- pothesis. When [CLS] token and [SEP] tokens are combined with the premise and hypothesis tokens, the premise tokens are lo- cated from the second token to (Ipl 1)-th token, and the hypothesis tokens are located from (lp) + 3) token (Ip) + [|]] 2)-th.
SE-NLI
SENLI (Kim et al., 2020) is an NLI model which generates token-level explanations (attribu- tion scores) using BERT's token representation. It trained via multi-task learning alongside the text- pair classification model. We used the publicly available implementation of the SE-NLI model." For the experiments on BioCla iim, we trained with the default hyperparmeters, For MNLIEx and e- e-SNLI, listed the numbers reported by Kim et al. (2020), after checking that our implementa- tion shows scores similar to their reported ones. SENLI model generates score for each of sub- word tokens of BERT. If a word contains multiple subword tokens and we averaged the scores.
indicates the averaged probability that premise to- kens to attend to hypothesis tokens, and [M) indicates averaged probability that hypothesis tokens attend to premise tokens.
Then, 1p
4. By transposing the latter matrix and adding it to the first, we obtain S. In this resultant ma- trix, Sij indicates degree of attention between the i-th token of the premise and j-th token of the hypothesis. Finally, we compute entail- ment and neutral scores using Equations 7 and 8.
Span-Level Reasoning (SLR)
The Span-Level Reasoning (SLR) is an NLI model that makes explicit span-level predictions, and has demonstrated more robust performance on NLI datasets (Stacey al., 2022). Unlike PAT model, the spans of SLR are split only by noun phrases as boundaries, limiting the granularity of the information. Nevertheless, to demonstrate the limitations of SLR, we applicd the scoring method similar to Equation 6.
Note that Co-attention method is also not capable of predicting contradiction tokens.
LIME
LIME (Ribeiro et al., 2016) is a widely used expla- nation method that attributes the model's prediction to input features (tokens).
We trained the SLR model On the MultiNLI dataset using the publicly released code. The trained model showed accuracy of 0.81 on the de- velopment set, which we accepted as successful training.
The LIME method begins by tokenizing hy- pothesis sentence into a sequence denoted by Ti, T,.,,,,, .... Tn. Subsequently, some of the tokens are perturbed (removed). This perturbation rep- resented by binary vector X; if T; is removed, X; 0 T. remains, X; = The tokens remain- ing form a perturbed hypothesis, which is then concatenated with the premise and fed into the NLI model. This model outputs probabilities for the three NLI labels. If the goal is to predict neu- tral tokens, the neutral probability is selected as y. This process results in an (X, v) pair for each perturbation. LIME collects samples of these pairs and trains linear regression model represented by y = Wx x + b. The coefficient vector W serves as a feature attribution vector, where each W, indicates
Let h2 denote the [CLS] token representation acquired by encoding i-th span by BERT. h4 is en- coded by the MLP produce an,i and ā These are termed "unnormalized attention weights" for neutral and contradiction labels, respectively. ān.s and बe.i are subsequently combined with other span-level scores to decide the sentence-level pre- dictions for the evaluation. Similar to Equation 6, compute token-level score for Cond-NLI the mean of the span-level scores containing the
'https://gf http://gtt.ttb.coooooooo///oo///
SENLI

---

token. A token i's score for neutral (n:) and contra- diction (c₂) are computed
512 to reduce the computational costs. The other hyperparameters and setups are the same as the publicly available implementation BERT (De- (10) vlin et al., 2019).
1
n4 |S
ān,s
sES.
A.3 GPU Hours and Infrastructure
Q: S
0o,31 (11)
We used GPUs, mostly GTX 1080 Ti or similar capacity devices which have less than 12GB of VRAM per each. All of our training took less than hours.
sES;
where St denotes the set of indices for the spans that include token i.
For the entailment score for SciEntsBank, we used - n, as the entailment score for the token i.
B BioClaim
InstructGPT
In this section, describe how dataset BioClaim constructed and how is intended to used. This section is the extension subsec- tion 3.2
We used OpenAl's fine-tuned GPT-3 model, text- davinci-003 which has 153 billion parameters. We used different prompt instructions for predictions of the neutral and contradiction tokens. The prompt instructions are built by modifying the instructions that were provided to the annotators for building our BioClain dataset. Figure 2 shows the ex- ample the prompt given to the model to predict different condition (neutral) tokens. To identify where the model generated words appear in the claim, we used word level exact match. The model outputs are automatically parsed. When the pars- ing failed due to slightly different formats of model output, we manually modified the format to parse it. We did not apply InstructGPT over the SciEnts- Bank dataset, as we found ChatGPT outperforms that over the Bi BioCla im dataset.
B.1 License
BioClaim will be released under Creative Com- mons Atribution-Vomcommrria--- We follow the license of the PCC dataset (Alamri and Stevenson, 2016) that we used, which is released under Creative Commons Attribution- Noncommercial-Share Alike 2.0 UK England & Wales License.
Alike.
B.2 Intended use
BioClaim is mainly annotated to evaluate the performance of a NLP system on token-level task of Cond-NLI. The dataset can be used for train- ing of token-level predictions sentence-pair NLI classification task.
ChatGPT
assist the
We utilized ChatGPT, specifically the "gpt-3.5- turbo" version. As did not have access to the primary choice.
GPT4 API, this version We provided ChatGPT with instructions similar to those given to InstructGPT. The only difference is that we specified output format be JSON, as ChatGPT has demonstrated the ability for generat- ing output in the JSON format. To apply ChatGPT for the facet-level partial entailment task of SciEnts- Bank, we used the prompt illustrated in Figure 3.
B.3 Dataset building
Our annotation spans 23 topics (question groups). We allocated 12 topies for the development set and 11 topics for the test set. The split information will be released along with the corpus. We recruited undergraduate students in nursing college as notators.
To resolve the different token-level annotations from annotators, we selected the annotation with more tokens annotated, with expectation that miss- ing conditions tokens would be more likely than including non-condition ones. The disagreements often appeared by different tendency of including neighboring tokens of the main tokens, such as an- notating "indicate a significant relationship" versus only "significant relationship".
A.2 Experiment setups
A.2.1 Hyperparameters for NLI model training
For both of the cross-encoder and PAT models, used the same hyperparameters: batch size of 32, learning rate of 10 and used 10% of the total training steps as warming up steps. We used the maximum sequence length of 300 instead default

---

In each of the examples, two claims extracted from research paper abstracts will be shown. The given two claims seem to be contradictory as they are implying opposite results about the same question. Precisely though, the two claims may have been obtained for different population intervention details that make it possible that both claims to be true. We want annotate the tokens (words) that express different conditions.
Claim I: We conclude that women with preeclampsia, prolonged dietary supple- mentation with l-arginine significantly decreased blood pressure through increased endothelial synthesis and/or bioavailability of NO. Claim 2:
Oral L-arginine supplementation did not reduce mean diastolic blood pressure after days of treatment compared with placebo in pre-eclamptic patients with gestational length varying from 28 to 36 weeks.
Condition tokens Claim 1: women, preeclampsia, prolonged, dictary supplementation, l-arginine, increased, endothelial synthesis, bioavailability, NO
Condition tokens Claim 2: pre-eclample patients, gestational length, 28 to 36 weeks
Figure 2: An example of the prompt given to InstructGPT model to solve Cond-NLI neutral token prediction. The text that is colored with yellow generated by the model.
Student answer: By letting it sit in a dish for a day. Reference answer: The water was evaporated, leaving the salt. Facet: (evaporated, water)
The facet is relation extracted from the reference answer. In the example above, does the student answer entail the given facet? Answer with Yes/No
Figure 3: An example of the prompt given to the ChatGPT model solve partial entailment task for SciEntsBank dataset.

---

Contradictory information in medical claims
Task instruction
Task instruction
How to use the page:
You can click on the buttons to select words.
You click the selected word again to exclude it from the selection.
Switch the modes between conflict mismatch by clicking the buttons next 'Mode:".
The page will automatically generate the text that contains the indices of the selected words for each of categories.
After annotation, copy-and-pase paste the indices 'Selected the provided sheets.
Task #1 1
Claim Supplementation during pregnancy with medical food containing L-arginine and antioxidant vitamins reduced the incidence of pre-eclampsia a population at high risk of the condition.
Claim 2: Oral L-arginine supplementation did not reduce mean diastolic blood pressure after days of treatment compared with placebo in pre-eclamptic patients with gestational length varying from 36 weeks.
Mode:
conflict
Claim 1
Supplementation
during
pregnancy
medical food containing L-arginine antioxidant
reduced
the
incidence
of pre-eclampsia
in diid
population at high risk of the condition.
Claim 2
Oral L-arginine
supplementation with placebo
reduce mean diastolic blood pressure days pre-eclamptic patients with gestational length varying
treatment
compared
from 28 to 36
weeks,
range select D
claim1 selected mismatch tokens:
claim1 selected conflict tokens:
claim2 selected mismatch tokens:
claim2 selected conflict tokens:
Selected all :...
Copy
Figure 4: Web annotation interface

---

Additional Experiments
cases, possible to guess how the PAT model interprets each partition. We classified the failure cases that we could interpret into four categories as follows with some examples shown in Table 9.
C.1 BioClaim evaluation more metrics
Table 6 shows the experiments result on BioClaim that additionally contain precision, recall and MAP (Mean average precision) as metric. MAP is com- puted by taking the mean of average precision for each sentences.
hypothesis segment con- tradicts the premise, however this contradic- tion is negated by an additional negation ex- pression present in the missing context of the hypothesis.
Double negation:
C.2 e-SNLI and MNLIEX
e-SNLI (Camburu et al., 2018) and MNLIEx (Kim et al., 2020) are two token-level annotated datasets built on top of the SNLI and MultiNLI datasets, respectively. These datasets do not include conditionally-compatbbe sentence pairs, thus they cannot be used to compare the performance of dif- ferent models on the Cond-NLI task. However, we applied our PAT model on these datasets to mea- sure its robustness in other datasets. For MNLIEx, we used the models trained on MultiNLI and for e-SNLI, we used the models trained on SNLI. Table 7 and 8 show the performance of PAT and baseline models on token-level explanation datasets e-SNLI and MNLIEx. For this evaluation, we use the metrics and categories that are used the pre- vious works (Thorne et al., 2019; Kim et al., 2020). Perturbation-baase explanation models, LIME and SE-NLI, achieve high performance on these two datasets. The results demonstrate that our PAT does not significantly underperform the explana- tion model SE-NLI that is designed for and trained on the NLI datasets.
Alignment: an expression in hypothesis seg- ment is aligned with a wrong segment of the premise, where the missing hypothesis con- text would allow the correct alignment.
Disambiguation: expression in hypoth- esis segment is interpreted in a wrong sense, mostly with incorrect parts of speech, where the missing context would enable correct dis- ambiguation.
Contextual:
The cases that cannot be catego- rized into the above three categories.
This failure analysis provides insights into po- tential strategies for improving PAT's performance. For example, the first two categories of failures could be mitigated without changing the model or current random partitioning. This could be achieved by incorporating limited additional in- formation into the hypothesis partitions, such as syntactic structure of the missing contexts (first of Table 9) or the presence of negation quantifiers (second case). These modifications be made while keeping the PAT's advantage of hav- ing model decisions attributable to input segments.
C,3 Hypothesis partitions that cause error
To get insights about the potentials and possible improvements of PAT, we analyze how limited con- texts in hypothesis partitions impact the model be- havior. For this analysis, we compare PAT with the full cross-encoder the sentence-pain NLI classi- fication over the MultiNLI dataset. We define the failure of PAT as cases whose labels are correctly predicted by the full cross-encoder, but not by PAT. training and evaluation of PAT for the NLI task, we use one random partitioning of a hypothesis. However, failure cases are not very common with single random partitions. Thus, we enumerated all possible partitions of the hypothesis of an instance for collecting failure cases.
We manually analyzed failure cases under- stand how the absence of full hypothesis context results different predictions by the PAT com- pared to the full crosse. In most failure

---

Neutral tokens
Contradiction tokens
Prec
Recall
F1
Acc
MAP
Prec
Recall
F1
Acc
MAP
Random
0.462
1.000
0.632
0.538
0.517
0.140
1.000
0.246
0.860
0.251
Similarity-bar -based
Exact match
0.518
0.863
0.647
0.565
0.597
word2vec
0.518
0.856
0.645
0.575
0.634
basee
Co-attention
0,484
0.962
0.644
0.538
0.612
LIME 0.476
0.971
0.639
0.538
0.528
0.187
0.537
0.277
0.872
0.441
Occlusion
0.462
1.000
0.632
0.538
0.478
0.140
1.000
0.246
0.859
0.301
SENLI
0.462
1.000
0.632
0.541
0.513
0.180
0.763
0.292
0.866
0.462
SLR
0.462
0.963
0.624
0.538
0.587
0.220
0.384
0.280
0.859
0.429
Token-entail
0.471
0.988
0.638
0.538
0.624
0.164
0.510
0.248
0.866
0.363
Proposed
0.505
0.939
0.657
0.622
0.711
0.401
0.429
0,414
0.871
0.543
Large Language Model
InstructGPT 0.699
0.515
0.593
0.673
0.686
0.483
0.396
0.435
0.856
0.530
ChatGPT 0.631
0.616
0.624
0.657
0.644
0.453
0.467
0.459
0.846
0.541
Table 6: Full experiments of token-level inference performance of prodicting neutral tokens and contradicting tokens
from the claim pairs of BioClaim. This table is an extended version of Table 4, while including additional metrics. Precision and Recall used the same threshold which optimized for the score. Mean average precision (MAP) is ranking metric and independent of the threshold.
Conflict
Match
Mismatch
Method
P@
MAP
Acc
P@1
MAP
Acc
P@
MAP
Acc
LIME
0.637
0.618
0.799
0.905
0.777
0.597
0.735
0.731
0.601
SE-NLI
0.750
0.723
0.800
0.965
0.903
0.760
0.817
0.830
0.714
Token Entail
0.662
0.628
0.757
0.930
0.842
0.692
0.723
0.733
0.597
PAT
0.696
0.700
0.770
0.918
0.868
0.753
0.850
0.851
0.682
Table 7: Token prediction evaluated MNLIEx (Kim al., 2020) It show precision at (P@) mean average precision (MAP), accuracy (Acc).
Premise
Hypothesis
Method LIME 0.376
Precision
Recall
F1
Precision
Recall
F1
1
0.547
0.46
0.834
0.593
SE-NLI 0.525
0.726
0.609
0,492
0.66
Token-entail
0.422
1,000
0.560
0.515
1,000
0.649
PAT
0.443
0.939
0.562
0.562
0.959
0.664
Table 8: Token prediction evaluated on e-SNLI (Camburu et al., 2018) It show precision, recall, Fl on each of premise and hypothesis, All three labels are averaged without differentiation.

---

Texts
Category
Gold
Prodiction
P: yeah well you're a student right
Disambiguation
Neutral
Contradiction
H: Well you're a mechanics student right?
P: smiled back.
Double negation
Neutral
Contradiction
H: She was so huppy she couldn't stop smiling.
P. He turned and smiled Vrenna.
Alignment
Neutral
Contradiction
smiled at Vrenna who was walking slowly behind him with her mother.
P: One of our number will curry your instructions minutely.
Contextual
Entailment
Neutral
A member my teum execute your orders with immense precision.
Table 9: Examples of four frequent categories of PAT's failures. The background colors on the hypothesis indicate how the hypothesis partitioned what were predictions each the parts. Red color indicates contradiction, blue is for entailment, and yellow is for neutral. Note that the hypothesis is divided into two partitions and one partition can have two text segments in it.
Predicted
E
N
C
E
23.0
14.0
37.0
Gold
N
8.4
28.2
36.6
C
8.4
18.0
26.4
8.4
23.0
42.2
Table 10: Confusion matrix on the instances are considered hypothesis partition errors. E is for entail- ment, N is for neutral, and is for contradiction. The numbers are percentages among the all error cases


Reviews:
[{"review_id": "LRmmm5XaWc", "reviewer": "Reviewer_kinb", "summary": "", "questions": "A. I would ask authors to please reason about the first reason to reject, especially the \"inconsistencies\" I felt were shown in the example of Table 1.\n\nB. Can authors please discuss the second reason to reject? Especially why performances on the baseline seem to differ so much compared to previous work and how could one test or justify the PAT model is better at explaining.\n\nC. Is the dataset going to be openly released? Abstract seems to indicate so, but want to make sure.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments. We are glad to hear that our work is considered as an important (kinb), reasonable (gcJS), practically-motivated (496z) novel explanation task, provides a novel dataset (kinb,496z) that can benefit the research community (kinb) by addressing the limitations of existing ones (496z), performs a thorough set of experiments (496z, gcJS) with good results (kinb,gcJS), and is flexible enough to be extended even to other tasks (kinb). We respond to the questions from the review below.\n\n# Regarding Reasons to Reject 1 / Question A\n\nWe highlight that the provided example is the result of annotating two types of tokens: 1) those signifying opposite outcomes, and 2) different conditions (line 290). We hypothesize that tokens leading to contradictions often signal opposing results. Therefore, Conditional NLI can be solved by attributing NLI decisions to the token level.\nThe token labels in Table 1 can be the result of the following justification. \n\n\nThe token ‘Losartan’ is considered a neutral token because it contains information (specific drug named Losartan) that is not inferable from the other, while it also contains entailed information (being antihypertensive), thus works with the verb ‘prevents’ to contradict with the other claim.\nConsider the following simplified example.\n\n> Claim 1: \"Losartan prevents (symptom).\"\n>\n> Claim 2: “Doxazosin increase the risk of heart failure”\n\nClaim 1 can be split into two sentences :\n\n> Claim 1-1: \"The antihypertensive prevents (symptom).\"\n>\n> Claim 1-2: \"The antihypertensive is Losartan.\"\n\nIn this example, the claim 1-1 contradicts claim 2, whereas the claim 1-2 doesn't draw any inference from claim 2. Yet, \"The prescribed antihypertensive\" can be inferred from claim 2.\n\nTo reiterate, the actual annotation process involves identifying opposing outcomes, without requiring the intricate reasoning outlined above.\n\n\n\n# Regarding Reasons to Reject 2 / Question B\nThe goal of our experiments on SNLI is to evaluate if and how partial attention, instead of full cross-attention, impacts the performance of the regular NLI task. \nWe were not able to replicate the reported accuracy on SNLI as the SLR paper, although we obtained the same accuracy on MultiNLI as reported in the BERT paper. \nOur conclusion is that PAT has 3% lower accuracy on MultiNLI compared to the baseline BERT, but it outperforms baseline BERT when this MultiNLI trained model is used to solve Cond-NLI. Our experiments comparing PAT and BERT on SNLI and SciTail show that the difference in NLI accuracy between PAT and BERT is reduced on these datasets compared to MultiNLI. Note that even if BERT's performance on SNLI is 90.77 instead of 0.887, and PAT's accuracy remains unchanged, it does not impact our main claim about PAT’s superiority on Cond-NLI.\n\n\n\nComparison on explainable NLI datasets: The explanation task in e-SNLI differs from the Cond-NLI explanation task. ​​This difference is also clear when examining the performance results of perturbation-based explanation methods (Lines 121-130): while these methods show strong performance on e-SNLI, their effectiveness significantly drops on BioClaim. Therefore, we did not compare the performance of PAT and baseline models on this dataset, and we do not make any claims about PAT's performance on the explanation task in e-SNLI.\n\nAlthough not evaluated, SLR has restricted capacity in generating the token-level predictions required by e-SNLI. This is because SLR merely splits sentences into spans by taking noun chunks as boundaries, preventing it from getting labels for individual tokens in a span. For instance, SLR will consider the span “increase left ventricular diameter” (from Claim 2 of Table 1) as a unit for inference. Thus, SLR would only generate an overall prediction for the entire span as neutral, failing to provide different labels for “increase” and “left ventricular diameter”. However, in Cond-NLI, these tokens would be labeled differently.\n\nAnother dataset for explainable NLI is the partial entailment dataset SciEntsBank, which better aligns with Cond-NLI. We evaluated PAT on SciEntsBank and the results show superior performance of PAT compared to state-of-the-art baselines (Table 5). \n\n\n# Question C\nYes, we will definitely release the dataset and the codes upon the paper acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A new annotated dataset for the task. Manual annotation is a hard endeavour and an appreciated contribution, especially for such an important task as NLI. I believe the research community may benefit from such a resource.\n2. The model proposed is shown to perform better than other explainability approaches or flexible models with more complex training schemes such as SLR. The setup is flexible enough to be extended even to other tasks, albeit it requires of a \"logical matrix\".", "reasons_to_reject": "1. I am not entirely convinced of the framing of the task. While previous work has also followed similar setups in which all non-neutral or contradicting spans are considered entailed, pushing it to the token level raises at least some questions. \n\nTake the example from Table 1:\nWhy are \"Interpretation Losartan\" and \"cardiovascular morbidity\" annotated as neutral? If the guidelines considered that they should be annotated as neutral since they do not appear in the other sentence, the verb \"prevents\" is not necessarily contradicting, since its subject and object are neutral. If the annotator should consider the token \"prevents\" as contradictory, if implies it considers Interpretation Losartan as \"antihypertensives\" that prevents a cardiovascular incident, which contradicts the fact \"doxazosin\" increases the risk of \"congestive heart failure\", as doxazosin is an \"antihypertensives\" and increases the risk of a cardiovascular incident. Therefore, if there are contradictory tokens in these pair of sentences, shouldn't the subject and object be entailed? Or the opposite, if the subjects and objects of those verbs are considered neutral, why is there a contradiction?\n\nI do not believe this is an issue with this annotation itself but just a consequence of the inherent shortcomings of reframing NLI as a token-level task, with no regard for semantic structures. \n\n2. The presented model is shown as better than more complex approaches, as pointed out before, but I would have appreciated more evaluations on already existing benchmarks for explainable NLI or comparisons on regular NLI with some of the models tested. Taking the SLR paper as an example, the baseline (BERT) reported a much higher performance on SNLI (90.77 vs 0.887), which is 0.870 for the proposed model PAT. Then some results are relegated to the Appendix for explainable NLI datasets (such as e-NLI), but compared to much older systems. To be fair, it seems the field is lacking proper comparisons and benchmarks as the SLR paper does not compare e-SNLI on the same setup as previous systems seemed to. But it would be nice to have a better evaluation to check whether PAT, while simple, is better at providing explanations for NLI, besides performing better on BioClaim or Cond-NLI.\n\n3. The annotation seems a bit underwhelming. If I understand correctly, only a few samples were annotated by more than one annotator (which were the ones used to compute agreement). Given the fact kappa was not that high, perhaps multiple annotations per sentence would have been desirable, albeit I understand that always add to the overall cost.", "questions_to_authors": "A. I would ask authors to please reason about the first reason to reject, especially the \"inconsistencies\" I felt were shown in the example of Table 1.\n\nB. Can authors please discuss the second reason to reject? Especially why performances on the baseline seem to differ so much compared to previous work and how could one test or justify the PAT model is better at explaining.\n\nC. Is the dataset going to be openly released? Abstract seems to indicate so, but want to make sure.", "paper_topic_and_main_contributions": "The paper presents a reframing of the NLI task at a token level called Conditional NLI. Here, instead of three classes, a set of tokens is annotated as neutral or contradictory to a given sentence. Notice how here there is a drop of the premise-hypothesis \"hierarchy\" but rather pairs of sentences where individual tokens are annotated to whether they contradict the other sentence or tackle different conditions, while unannotated tokens are considered entailed.\n\nWhile this would be how the task is defined, there are no available datasets to test this new task. Therefore the authors annotate some sampled sentences from an existing dataset (PCC, Alamri and Stevenson, 2016) to encourage their relevance as possibly contradictory. They call this dataset BioClaim.\n\nSimilarly, there are no models to predict arbitrary spans for this framing of the task. Therefore the authors propose Partial-Attention NLI Model which leverages existing NLI datasets but reframes the task by splitting the hypothesis into two arbitrary spans, predicting their entailment to the hypothesis separately, and then back-propagating through a combination of both with the overall label. This is a simpler approach than previous systems (such as SLR, Stacey et al., 2022) but proves to be more effective at Conditional NLI.\n\nTherefore the contributions are:\n- A new token-level framing of NLI called Cond-NLI.\n- A new dataset.\n- A new training approach that provides arbitrary span predictions while trained on traditional NLI datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ukMPjMyA2r", "reviewer": "Reviewer_496z", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments. We are glad to hear that our work is considered as an important (kinb), reasonable (gcJS), practically-motivated (496z) novel explanation task, provides a novel dataset (kinb,496z) that can benefit the research community (kinb) by addressing the limitations of existing ones (496z), performs a thorough set of experiments (496z, gcJS) with good results (kinb,gcJS), and is flexible enough to be extended even to other tasks (kinb). We respond to the questions from the review below.\n\n# Regarding reasons to reject 2\n\nAs suggested, there are two different concepts, one is the Cond-NLI task and the other is token-level attribution of NLI. Our hypothesis is that token-level attribution of NLI can be used as an effective solution for Cond-NLI. \nCond-NLI aims to identify different conditions, and neutral tokens are most likely to represent those conditions. Similarly due to the nature of the dataset, where a claim pair gives opposing answers to the same question, the tokens indicating contradiction are expected to correspond with tokens signifying opposing results.\n\nWe will provide further clarity in the manuscript regarding how we formulated the solution to the Cond-NLI task through identifying tokens that lead to contradiction, entailment, or neutrality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and easy to follow. \n- Explaining model decisions is an important question and the proposed PAT architecture seems to do that well for NLI.\n- The proposed dataset is interesting because it addresses limitations of prior NLI datasets, e.g. high lexical overlap between hypothesis and premise. I am also intrigued by the practical motivation of the task through assisting systematic reviews.\n- The authors provide a thorough set of experiments. I find it especially interesting that the GPT models perform so well in a zero-shot setting on a novel dataset.", "reasons_to_reject": "- The inter-annotator agreement for the proposed dataset is rather low with a Cohen's kappa of 0.46. I think the impact of this on the comparisons could be more extensively discussed, especially since the authors provide results of significance tests.\n- The authors define the Cond-NLI task as having the goal to extract contradictory aspects and their conditions from text. I don't fully see how this maps to the formulation of finding contradicting/entailing/neutral tokens.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Edit after rebuttal: Thank you for your response. It doesn't change my positive evaluation substantially, so I'll leave the scores unchanged.\n\nThe paper \"Conditional Natural Language Inference\" proposes the new task of conditional NLI (Cond-NLI), which extends the original NLI task with token-level decisions on which tokens of the hypothesis contradict/entail/are neutral towards the premise. The authors release a new manually annotated English dataset from the biomedical domain for this task. Additionally, they propose a new architecture, PAT, that can learn to perform Cond-NLI without requiring any span-label labels. During training, PAT randomly divides the hypothesis into two chunks and predicts the NLI label for both of them individually and then combines both scores to arrive at the final prediction. This allows PAT to assign a prediction to each token at test time. The authors the span-level predictions of PAT to various X-AI methods, as well as to Instruct- and ChatGPT, on two datasets. They find that PAT sacrifices some performance on the NLI task but outperforms all X-AI baselines and performs on par with the much larger GPT models on the token-level task.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YVXwtiVR9I", "reviewer": "Reviewer_gcJS", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments. We are glad to hear that our work is considered as an important (kinb), reasonable (gcJS), practically-motivated (496z) novel explanation task, provides a novel dataset (kinb,496z) that can benefit the research community (kinb) by addressing the limitations of existing ones (496z), performs a thorough set of experiments (496z, gcJS) with good results (kinb,gcJS), and is flexible enough to be extended even to other tasks (kinb). \nWe respond to the concerns and questions of the review below. We hope this clarification can address concerns about the paper. \n# Regarding reasons to reject 1\nWe appreciate your feedback. We will thoroughly review and refine the sections mentioned to ensure clarity and ease of understanding for our fellow researchers in the community.\n\n# Regarding reasons to reject 2\n The C-STS (Conditional Semantic Textual Similarity) paper [1] was only published on Arxiv in May 2023. It is considered contemporary to our work based on the submission guidelines (https://2023.emnlp.org/calls/main_conference_papers/). We kindly request a reconsideration of the novelty criticism comments based on contemporary works.\n\n\nOur approach to the problem distinctly diverges from that of the CSTS work. We address the Cond-NLI task without requiring large-scale annotations for the target task. Specifically, our PAT model for Cond-NLI is trained by leveraging existing datasets for the regular NLI task. On the other hand, CSTS is treated as an independent task in [1], one that cannot be addressed using STS datasets, and depends exclusively on CSTS-specific training data. Given these distinctions, our work offers unique insights and has a valuable contribution in its own right.\n\n\nOur approach has the advantage of using sentence-level annotations to generate predictions at the token level. Sentence-level annotations are available on a larger scale than fine-grained annotations, as they can be extracted from naturally occurring text pairs, such as in our BioClaim dataset, or obtained through a less complex annotation process. Thus, we believe that the proposed novel method can be valuable to the academic community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Conditional NLI is reasonable, method changes are simple, but there are some improvements.\n2. Good results\n3. A lot of experiments and meaningful additional experiments.", "reasons_to_reject": "1.  Writing confuses me, especially the content settings, introduction, methods, and experimental parts in the chapters are all confusing to varying degrees, which will increase the difficulty of reading for other researchers in the community\n2. The idea of NLI under conditional constraints is generally not new, [1] have done research on it, and I think they provide a more general approach. Furthermore, NLI is quite expensive when constructing conditional constraints. Is it possible to achieve the same or better results by means of data construction? \n3. Overall, I think this work is limited in innovation and confusingly described, and may not have a significant impact in this field, and I think researchers in the NLP community should get more high-quality input from EMNLP\n\n[1]  Deshpande, Ameet , et al. \"CSTS: Conditional Semantic Textual Similarity\"", "questions_to_authors": "", "paper_topic_and_main_contributions": "The author believes that NLI is considered to be an ambiguous task. Judging their relationship is constrained by the scene, and their relationship changes as the scene changes. This paper proposes a conditional attention model PAT, which can attribute model decisions to the input various parts. At the same time, based on the above task definition, this paper also contributed a task-related dataset BioClaim. The proposed method outperforms the best NLI-based baseline method by 8% in predicting conditional markers.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The majority of the reviewers agree that the work is both sound and exciting. I don't find the objection raised by the third reviewer convincing. The fact that a paper was published on Arxiv in May 2023 should not necessarily diminish the novelty of this work.
Decision: Accept-Findings