WiCE: Real-World Entailment for Claims in Wikipedia
Authors: Ryo Kamoi
Tanya Goyal
Juan Diego Rodriguez
Greg Durrett
Venue: EMNLP 2023 Main (2023)
Keywords: natural language inference
textual entailment
fact-checking

Abstract: Textual entailment models are increasingly applied in settings like fact-checking, presupposition verification in question answering, or summary evaluation. However, these represent a significant domain shift from existing entailment datasets, and models underperform as a result. We propose WiCE, a new fine-grained textual entailment dataset built on natural claim and evidence pairs extracted from Wikipedia. In addition to standard claim-level entailment, WiCE provides entailment judgments over sub-sentence units of the claim, and a minimal subset of evidence sentences that support each subclaim. To support this, we propose an automatic claim decomposition strategy using GPT-3.5 which we show is also effective at improving entailment models' performance on multiple datasets at test time. Finally, we show that real claims in our dataset involve challenging verification and retrieval problems that existing models fail to address.

WICE: Real-World Entailment for Claims in Wikipedia
Ryo Kamoi R Tanya Goyal 9 Juan Diego Rodriguez Greg Durrett .
& Department of Compuler Science, The University of Texas at Austin Applied Research Laboratories, The University of Texas at Austin ryokamoi@utexas. edu
Abstract
Claim
Evidence: cited urticle [3]
The Société de transport de Monirial (STM) 747 Shutle repiaced the "Aerobus" that was privately operated by Groupe La Québéccuise 1!
... The route is the 747 Express bus, which finally provides direct. non-stop link
Textual entailment models are increasingly ap- plied in settings like fact-checking, presupposi- tion verification in question answering, or sum- mary evaluation. However, these represent a significant domain shift from existing entail- ment datasets, and models underperform result. We propose WICE, a new fine-grained textual entailment dataset built natural claim and evidence pairs extracted from Wikipedia. In addition to standard claim- level entailment, WICE provides entailment judgments sentence units of the claim, and minimal sub- set of evidence sentences that support each sub- claim. To support this, propose auto- matic claim decomposition strategy using GPT- 3.5 which we show also effective at improv- ing entailment models' performance on multi- ple datasets at test time. Finally, we show that real claims in our dataset involve challenging verification and retrieval problems that existing models fail to address.'
between downtown and Dorval Pterre Elliot Trudeou International Airport... also replaces La Québécoise's Acrbbus shatile service between the bus station and the airport that used cun every holf hour and cost $16...
Subclaim
The Société de transport de Montréal (STM) 747 Shutile Bus replaced the "derobus." "
Entailment label: SUPPORTED Supporting senteace indices: 9,
Subclaim 2
The "Aerobus" " waas privotely operated by Groupe la Quéhicolse.
sub-
Entailment lubel: PARTIALIN-SUPPORTED
Supporting sentence indices:
Not- Tokens: privitelly, Groupe
WICE annotation for claim in Wikipedia and its cited evidence. Claims are automatically broken into subclaims. WICE provides entailment labels and indices of evidence sentences that support a subelaim. Real-world claims often partially supported (sub- claim 2); unsupported tokens are annotated here.
Figure I:
1 Introduction
et al., 2021) and WANLI (Liu et al., 2022). As result, existing frameworks for document-level entailment are built upon aggregating local entail- ment scores (Zhou et al., 2019; Laban et al., 2022) or using retrieval stage (Nie et al., 2019; Schuster et al., 2022). There are a few exceptions such DocNLI (Yin et al., 2021), but it features large amount of synthetic negative data. This highlights the second weakness: the lack of ccologically valid negative examples. The process by which contra- dictory cases are authored leads to spurious cor- relations, including single-word correlations (Gu- rurangan et al., 2018; Gardner et al., 2021), syn- tactic heuristics (McCoy al., 2019), lack of reliance on the input (Poliak al.. 2018). Third, these datasets lack fine fine-grained annotations of what parts of a claim are supported or not.
Textual entailment (Dagan et al., 2005) and natu- ral language inference (MacCartney and Manning, 2009; Bowman al., 2015; Williams et al., 2018) are longstanding problems in NLP that take many forms. The SNLI dataset has stated purpose to use NLI "as a tool for the evaluation of domain- general approaches to semantic representation" (Bowman et al., 2015). However, this is far from how NLI is used today, e.g., to validate QA system outputs (Chen et al., 2021), evaluate generated summaries (Falke et al., 2019; Laban et al., 2022) under- stand goooledge-grounded dialog (Honovich et al., 2021; Gupta al., 2022: Dziri et al., 2022).
There are some major gaps when applying mod- ern entailment systems to these tasks. First is the fact that many NLI datasets target short premises, often single sentences, such as VitaminC (Schuster
Our work addresses these shorteomings by col- lecting WICE (Wikipedia Citation Entailment). a dataset for verification of real-world claims in
'Our data is available at: ryokamoi/wice

---

Wikipedia. Given a sentence in Wikipedia and the corresponding article(s) cites, we annotate the entailment label, a list of sentences the cited arti- cle(s) that support the claim sentence, and tokens in the claim that are unsupported by the article(s) (see Figure 1). We show that the claims in WICE involve challenging verification and retrieval prob-
Segment the followwnng souronee into indnuchal facts: [in-contest examples of decompusition; see Appendis)
Original Sentence"
The main altar houses 77th-century freseoler fieures interietine with the framed 15th century jion of the Madonna [ (6388, painted by Mano Balassi.
GPT-3.5
Predicted
The main altur houses a 17th century fresco.
The fresco is of figures interacting with the framed 17th-century
lems beyond the scope of current NLI datasets.
of Madonnn.
Madonna was painted by Balassi 1638.
To aid the construction of WICE and pro- vide fine-grained annotations, we introduce Claim-Split, method of decomposing hypothe- ses into subclaims using few-shot prompting with GPT-3.52 (Brown et al., 2020; Ouyang et al., 2022), shown in Figure 2. This decomposition resembles past frameworks derived from OpenIE (Stanovsky et al., 2018; Ernst et al., 2021) or Pyra- mid (Nenkova and Passonneau, 2004; Shapira et al., 2019; Zhang and Bansal, 2021), but avoids relying on annotated data and achieves greater flexibility by using GPT-3.5. By operating at the subclaim level, we simplify both our annotation process and the final entailment prediction task for automatic models. We also show that Claim-Splic t can im- prove the entailment classification performance of off-the-shelf models by simplifying long claims.
Figure 2: Cpaim-Split automatically breaks claims into subclaims using LLM (GPT-3.5 in this work).
2.1 Claim-Split using LLMs
A key idea we argue for in this paper is claim de- composition. Real world claims, such are those in Wikipedia that constitute WICE, often consist of multiple related pieces of information, each of which may may not be supported by the evi- dence. We show example of this in Figure I where the subclaims, derived from the com- plex claim, have different entailment labels. By decomposing claims prior to annotation, and col- lecting annotations at the subelaim level, we can offer a more fine-graine tne-ggraind view into which parts of claims are supported by cited documents.
We evaluate range of systems including existing short-paragraph entailment mod- els "stretched" to make document-level entail- ment judgment out of short-paragrapp judgments (Laban et al., 2022; Schuster et al., 2022). We show that chunk-level processing of the long evidence and retrieval- based approach are strong starting point for future systems, although current systems still perform below human level on this dataset and retrieval performance poor.
dataset,
First, we describe our claim decomposition strat- egy, called claim-Split. Then, we establish the validity of Claim-Splis as a pre-annotation step by manually verifying the generated subclaims.
Claim-Split method Our method prompts an instruction-tuned GPT-3.5 model (text- davinci-002) (Brown et al., 2020; Ouyang et al., 2022) in a few-shot setting to automatically decompose a claim into multiple subclaims: Claim-Split(c) = (c.)...Cm.. The prompt used includes K example splits along with the instruc- tion "Segment the following sentence into individ- ual facts." These examples are designed such that the subclaims provide complete coverage over the corresponding input claims (see prompt used for WICE in Appendix D.1). Figure 2 shows an exam- ple of decomposed claims using our approach.
2 The WICE Dataset
We aim to annotate claims that are: (1) naturally- occurring; we extract claims from Wikipedia and its cited documents, where the noise in citations gives realistic negative examples, (2) in-context with surrounding text; this mirrors real use cases where claims occur in discourse context, and (3) fine-grained; we break down complex claims into multiple subclaims with Claim-Split and pro- vide entailment judgments for both granularities. We also provide token-level annotation for non- supported tokens.
Claim-Split generated subclaims correctly and subclaims completely cover the information in the source claim. We recruited Mechanical Turk workers to annotate subclaims decomposed by Claim-Split for 700 Wikipedia claims, for two necessary criteria: (1) Completeness: subclaims
*GPT-3.5 was the strongest model available at the time we collected the data.
"We explain how claim sentences are chosen in Section 2.3.

---

{ci,... ..m must cover all information in e, and (2) Correctness: all subclaims c must faithfully present a part of the information in c.
by automatically projecting subclaim annotations using a natural set of rules described in Section 2.3.
2.3 Dataset Collection
We found that 92.3% of the claims satisfy the completeness criterion and 97.7% of the generated subclaims satisfy the correctness criterion. Analyz- ing the errors fürther, found that one-third were relatively trivial such as disregarding parentheses in the original claim, and could be solved with tar- geted prompts. Other errors were more complex and do not have straightforward solutions; see Ap- pendix A for more analysis. In total, only 8.6% of the claims included one of the two types of er- rors. These results justify our strategy of seeking annotations at the subclaim level and show that building systems at the subclaim level is viable path for further efforts.
Base Data We use the same base set of Wikipedia claims as the SIDE dataset (Petroni et al., 2023). For cach claim, we re-retrieve the cited web ar- ticle(s) from Common Crawl+ and segment into sentences. 5 This gives our base set of claim- evidence pairs (c, (2,,,),...
Claim-Split Next, we split each claim into subelaims {21, .2....CCr. using Claim-Split, as described in Section 2.1. We use few-shot prompt- ing with six examples (Appendix D.1). Also, we filter claims that are decomposed into either only one or more than six subclaims." 6 For the develop- ment set, this filters 19.1% of examples.
The set of annotated claims from this experiment constitutes the complete dev and test set of WICE (details later in Section 2.3); therefore, we man- ually fix the errors detected in this annotation to provide high quality evaluation dataset.
Additional Filtering We use NLI model (RoBERTa-Large trained on DocNLI (Yin et al.. 2021) to remove datapoints (c, E) for which all subclaims 0j E Claim-Split(c) are classi- fied as entailed." By using a relatively weaker RoBERTa-Large model, we remove trivially en- tailed claims but avoid making a datasct that is ad- versarially difficult for larger models (e.g.. T5-3B),
2.2 Tasks in WICE
Let claim (analogous to the hypothesis in standard NLI terminology) be sentence in Wikipedia ar- ticle, and evidence E (ci...., cn) (analogous to the premise) refer to sentences from web ar- ticle(s) cited as a reference for the claim c. Let Claim-Split(c) = {ci,.... } be the automati- cally decomposed subelaims from c.
We retain 16.3% of the claims after applying all the filtering steps above. Despite this, we observe diverse entailment phenomena in the remaining subset, which we analyze further in Section 2.5.
Human Annotation We recruited Mechanical pairs for each of the three tasks outlined in Section 2.2. First, ran qualification task with 3 examples (chosen include challenging annotations) and qualified 23 workers based on Annotators were paid $0.75 per HIT. Each HIT involved annotation of all subclaims corresponding to a single claim.
The human-annotate data we collect supports the following three tasks:
Turk workers to annotate
1. Entailment Classification: Given a claim c (or subclaim ej) and evidence document E, is the claim (or subelaim) entailed by the document? We annotate three-way entail- ment: (SUPPORTED, PARTTALLY-SUPPORTEN NOT-SUPPORTEDD.
We collect annotations from 5 unique workers for each example in the development and test set, and from 3 for the train set. We observe reason- able inter-annotator agreement for the entailment
2. Evidence Retrieval: Given claim e (or sub- claim eÿ) and evidence document E, which sub- set of evidence sentences (e) C E
support or partially support e (or c;)?
"ttpps: / ///p///Cooooaaalllc.a org//
"We re-retrieved the cited web articles because SIDE only
3. Detecting Non-Supported Tokens:
Given sub-
contains one evidence web article even when there are multiple
on the original Wikipedia page.
claim ej and evidence document E, which to- kens (t1....,tp))
"We removed the claims that cannot be decomposed by Claim-Split because we intend to focus on complex claims. In addition, we removed the claims that are decomposed into more than six sub-claims because typically they are not inter- esting cases; they often just include list of examples.
tp) C Cj are not supported by E?
For each of these three sub-tasks, we only col- lect human annotations at the subclaim-level (as shown in Figure 1). Claim-level labels are obtained
i 'Detail of how we split E into chunks and combine chunk- subclaim entailment scores are in Appendix B.

---

Statistic
Subclaim
Claim
WICE
Category
FEVER
VHC
Subel
Claim
# Datapoints Dev
Train
3,470
1,260
949
349
Compression
10
20
3.9
4
Test
958
358
w/ contextualization
28
11
14.2
4
# Tokens
12.1
27.4
Paraphrase
Direct
34
24
26.0
16
# Supporting Sents
1.9
3.1
Calculation
0
30
0.0
0
# Subclaims/ Claim
3.0
+ Inference
24
8
52.8
68
Background Knowledge
2
0
3.1
8
Evidence Sentences Datapoint # Tokens/ Evidence Sentence # Tokens/ / Claim's Context
119.5
14.0
Annotation Mistake
2
0.0
0
122.5
Table 3: Distribution (%) of verification problems
Table 1: Statistics of the WICE dataset.
timated from annotation for 50 claims in each dataset (127 subclaims WICE), We evaluated claims labeled entailed in FEVER and VitaminC, and claims labeled as supported or partially supported WICE.
Supported
Partially Supp.
Not Supp.
Claim Subclaim 55.8
33.0
54.7
12.3
18.2
25.9
Table Entailment label distribution (%) of claims and subclaims in the development set of WICE.
typical NLI dataset, both PARTIALLY-SUPPORETE and NOT-SUPPORTED would simply be labeled as neutral, which is not very useful for sys- tem designer attempting verify a fact. For PARTIALLY-SUPPOTEE subclaims, 25.2% of kens were identified as not supported.
classification task; Krippendorff's a 0.62 on the development set. We describe how we aggregate annotations from these workers in Appendix B.
Deriving claim labels from subclaim labels For entailment classification, if all subclaims are SUPPORTED NOT-SUPPORTED, we assign that as the claim-level label. Else, assign PARTTALLY-SUPPORTED For supporting sentences, we take the union of subclaim level supporting sen- tences as the claim level annotation. When there are multiple sets of supporting sentences for each subclaim, we take the union of all combinations.
2.5 Analysis of Phenomena
We compare with FEVER and VitaminC to show that the in-the-wild claims and cited articles in WICE constitute diverse and challenging verifi- cation problems. We bucket verification problems from these datasets into the following categories. First, we define Compression to include cases where the claim/subclaim appears almost verba- tim in the evidence document with trivial phrasal deletions. Relatedly, the category Compression w/ Decontextualization includes cases in which pronouns, VP ellipsis, other similar effects need to be resolved appropriately contextualize the claim. These are typically not challenging for language models. We also define four cases we broadly call Paraphrasing. Within this, Direct cases are those where the evidence document infor- mation has been restated in the claim, but in a way that is relatively transparent and simple to follow (e.g., g., synonym substitution). Require Calculation includes cases that require numerical calculation comparison. Require Inference captures slightly more complicated where inferences about the sit- uation are needed. Require Background Knowl- edge represents cases where additional background knowledge or world knowledge is required (c.g., knowledge of entity aliases or the ability to recog- nize causal relationships between events).
2.4 Dataset Statistics
Table shows the overall statistics for the final WICE dataset. On average, claims were decom- posed into 3.0 subclaims by the Claim-Spli it method. Our dataset contains approximately 5.9K subclaim level (or 2K claim level) examples for entailment classification task. Each subclaim (or claim) is supported by an average of 1.9 (or 3. I) evidence sentences when the label is SUPPORTED or PARTIALLY-SUPPORTED.
Table shows the distribution of entailment la- bels. Roughly 56% of the subclaims labeled SUPPORTED. This percentage much lower at the claim level (33%) since all subclaims must be SUPPORTED. Consequently, majority of the claims fall into the PARTIALLY-SUPPORIE
category. In
"There can be multiple sets of supporting sentences for each claim/subclaim because different annotators can annotate different sets of supporting sentences that include sufficient in- formation to support (or partially support) the claim/subclaim.
We manually annotate 50 claims randomly se-

---

Claim
Subclaim Fl
Stretching NLI for document-level entailment WiCE's evidence articles are generally much longer than the input length limits of NLI mod- els. Therefore, we adopt the "stretching" technique from prior work (Laban et al., 2022; Schuster et al.,
Model
Train Data
Acc
Acc
Majority Label
49.6
33.0
71.6
$5.8
Sentence-level
RoBERTa-Large
SNLI*
47.31
31.0
73.31
66.87
RoBERTa-Large
MNLI-
46,51
35.8
71.9
57.97
2022) for all our entailment models.
ALBeRT-xLarge
VitaminCa
49.8
58.91
74.1
67.8
ALBERT-XLarge
ViCC+MNLI
52.71
61.5t
77.0*
73.0
We divide the evidence document into mul- tiple partitions Pp. These can be individual sen- tences (e:) or chunks (contiguous e4's). We re- strict maximum chunk size to 256 tokens. For each subclaim/claim and partition pair, we use the NLI model to get "local" entailment score: se(c,,) P) = P(y = entailed G. Then, as the first method of the "stretching" technique, we derive document-level score by taking the maxi- mum local score: sc(c,, E) maxpePe [se(e), p)]. We call this the MAX entailment strategy.
T5-38
ViC+MNNI
48.3
61.7*
77.1
73.4
T5-3B
ANLI
48.0
41.9
77.6
71.8
Chunk-level
T5-38
DocNLI
56.41
62.8
79.7t
74.91
T5-Large T5-38
ANLI ANLI
61.8
70,71
78.3
64.3
75.1
83.4
79.6
Human
83.3
92.0
94.4
Table 4: Off-the-shelf binary entailment classification performance of existing NLI models on WICE using the MAX strategy to combine local entailment scores. I; Dataset-model configurations from Laban et al. (2022) and 2: from Schuster al. (2021). Similar Honovich et al. (2021), we observe the best performance using models trained on ANIII, although there re- mains gap between these and human performance. 1 Worse than T5-3B trained on ANLI (chunk-level) with p-value < 0.05 according to a paired bootstrap test.
Off-the-shelf models We report the perfor- mance of RoBERTa-Large (Liu et al., 2019), ALBERT-xLarge (Lan al., 2020), T5-Large and T5-3B (Raffel al., 2020) fine-tuned on SNLI (Bowman et al., 2015), MNLI (Williams al., 2018), DocNLI (Yin et al., 2021) or ANLI (Nie et al., 2020). We choose model-dataset pairs that have been used in prior work and have released weights, but default to T5-3B for our new settings. The PARTIALLY- and NOT-SUPPORTED labels in WICE correspond to the neutral (and sometimes contradiction) category in SNLI, MNLI, and ANLI. On the other hand, DocNLI only includes binary categories (entailed or not entailed). To evaluate models trained on these datasets on WICE, we consider binary classification task: SUPPORTED or not. We use the predicted probability for entailment as the predicted score for the SUPPORTED class.
lected from the development sets with these cate- gories for the three datasets. 9 In WICE, we anno- tated 127 subclaims in 50 claims. Table 3 shows the estimated distribution. We can see that natural claims in WICE involve difficult entailment clas- sification problems often requiring some kind of inference even at the subelaim level. con- trast, relatively few claims VitaminC involve inference, but mostly require narrower types of reasoning such as calculation.
3 Experiments on WICE
We evaluate GPT-3.5 and GPT-4 in Section 3.4 on the oracle retrieval dataset (explained later); the stretching approach requires invoking the model for every (document chunk, subclaim) pair, which becomes very expensive to test with large models.
We have three main questions for our dataset. (I) How well do existing NLI models perform off- the-shelf when using the "stretching" paradigm? dataset improve accu- racy? (3) Would being able retrieve the relevant supporting sentences improve accuracy further?
(2) Does fine-tuning
Models fine-tuned on WICE WICE consists of entailment labels corresponding to entire evi- dence documents and also supporting sentences for SUPPORTED and PARTIALLY SUPPORTTD cases. To train models that be stretched as described above, we derive sentence- and chunk-level entail- ment labels from these WICE annotations (details are in Appendix F.2). Although we evaluate the performance on the binary classification task, we fine-tune tune models on the three-way classification
3.1 Entailment Classification on WICE
We benchmark the performance of NLI models on WICE in both off-the-shelf and fine-tuned settings.
"If more than one category applies, we assign the most "difficult" category (latest in our list). Examples are given in Table 13 in the appendix.

---

Claim
Subelaim
Claim
Subelaim
Unit
Train Data
Train Data
Acc
Acc
F1
sent
ANLI+WiCE
58,01
62.8'
81.2
77.31
BM25
15.7!
9.01
98.6
13.21
7.81
96.7
chunk
WICE ANLI+WiCE
65.3t
77.1
85.1
82.71
ANLI
24.8f
27.11
39.21
41.91
39.51
55.51
chunk
72.1
79.1
87.3
85,0
WICE
49.01
48.4*
67.0
49.31
46.31
67.6
Human
83.3
92.0
94.4
94.4
ANLI+WICE
62.01
61.01
76.91
58.5
54.7
79.6
w/ evidence context
Table 5: Binary entailment classification WICE with fine-tuned T5-3B models. Further fine-tuning on ANLI- tuned models improves performance and chunk-leve outperforms sentence-level. 1; Significantly worse than T5-3B fine-tuned on ANLI+ WICE (chunk-level) with p-value < 0.05 according to paired bootstrap test.
WICE 67.4
65.0
81.7
58.6
54.9
75.7
ANLI+WICE
64.81
62.5
77.81
56.6
49.0
86.4
Huraa 90.9
92.2
92.6 1
91.6
93.2
92.5
Table 6: Performance of T5-3B on the evidence re- trieval task of WICE. f; Worse than T5-3B finetuned on WICE (w/ context) with p- p-value < 0.05 in paired bootstrap test. Human performance is on 50 random claims. 4: For claim level human performance, the union set of retrieved sentences at subclaim level.
labels in WICE.
take
Human Performance We manually annotate 50 randomly selected test claims and report that as human performance. Similar to crowd annotation, we annotated the subclaim level and aggregated them to obtain claim-level judgments.
for prior NLI datasets and p(SUPPORTED) + 0.5 x MHARTIALLY-SUPPORII
for WICE retrieval scores. Evidence sentences with scores larger than a threshold т are predicted as supporting sen- tences.
Results Table 4 outlines off-the-shelf perfor- mance on WICE for the binary entailment clas- sification task. shows that predicting scores at the chunk-level works better than sentence-level using the MAX strategy. Overall, T5-3B trained on ANLI performs best, though it is still substantially lower than human performance (64.3 vs 83.3 Fl at the claim-level). This shows that the realistic claims and document-level setting WiCE differs substantially from previous NLI datasets.
However, we saw in Table that sentence- level NLJ models perform significantly worse than chunk-level models on classification, suggesting that single sentence without context insuffi- cient for entailment evaluation. Therefore, we also train another sentence-level variant that includes additional evidence context (128 tokens) input, in format of "claim <SEP> evidence-context <SEP> siddence-sentence""
For fine-tuning, we evaluate two settings: only fine-tuning on WICE or further fine-tuning a T5-3B model already fine-tuned on ANLI. Results are in Table 5. It shows that the chunk-level T5-3B model fine-tuned on WICE after ANLI achieves the best performance both granularity levels. Although it improves over off-the-shelf results in Table 4, it is still substantially lower than human performance. This suggests that WICE is challenging even for fine-tuned models.
Metric We evaluate supported partially- supported claims and subclaims, which include at least one gold supporting sentence. As there can be multiple gold sets supporting sentences for each clain/subclaim in WICE, 12 we report the maximum F1 score over the gold sets for each claim/subclaim: max; F1(S,, S:) where S- is the predicted set and S; is the i-th gold set. We choose the threshold that gives the best Fl score (calcu-
3.2 Evidence Retrieval on WICE
First, we benchmark the performance of sentence- level NLI models fine-tuned on WiCE on the re- trieval task: given claim/subclaim c, retrieve all supporting sentences from evidence E. Then, we evaluate if retrieve-then-preeee pipeline can im- prove the performance of entailment classification.
1"The intuition behind this formula is that we want to pre- fer the sentences that receive high scores for the supported category, while also accepting partially supporting sentences. We did observe performance improvement by includ- ing context for claims. This likely because the evidence in our dataset already relates to the entity or event the claim. Therefore, decontextualization less critical here, but would likely be crucial retrieving supporting documents.
Retrieval using NLI models Our strategy as follows: derive a score for each evidence sen- tence and claim/subclaim pair. We use p(entailed)
Difffrrent annotators may select different sets of support- ing sentences that contain equally sufficient information to support (or partially support) a claim. WICE includes all of them as gold sets of supporting sentences.

---

Claim
Subelaim
Claim Acc
Subclaim
Setting
Model
Acc
Acc
F1
Acc
MAX best (Table 5)
72.1
79.1
87.3
85.0
TS-3B (ANLD
61.8
74,0
83.9
85.0
T5-3B (ANLI+WICE)
77.8
88.0
88.7
89.0
top-k
71.1
79.3
86.6
84.2
lop-k (w/ context)
72.9
79,9
88.4
87.0
GFT-3.5
39.3
32.0
73.3
73.3
oracle
78.0
84.4
88.7
87.7
GPT-4
61.0
77.0
91.1
92.0
Human
83.3
92.0
94.4
94.4
Human
83.3
92.0
94.4
94.4
Table 8: Binary entailment classification performance of the GPT models (few-shot) on WICE with the oracle retrieval (100 claims/subelaims)
Table 7: Binary entailment classification performance of the retrieve-then-preditt pipeline using chunk-level T5-3B on WICE. I; Worse than top-k (w/ context) with p-value < 0.05 according to paired bootstrap test.
egy using the retrieval model without evidence con- text does not work well. However, adding context improves performance significantly. This mirrors evidence retrieval results from Table 6. As an upper bound, we report results in the oracle setting. i.e., if gold set of supporting sentences is provided as input to the NLI model."? 17 We see substantially improved performance the oracle setting (71.1 vs 78.0 in oracle). The large gaps suggest that better retrieval can improve the entailment classification performance.
lated as above) the development set."
Results: Including context from evidence sen- tences improves retrieval performance. Table 6 reports the performance of the baseline BM25, best off-the-shelf model from Table 4 (T5-3B on ANLI), and fine-tuned entailment models. Performance of models fine-tuned with evidence context on WICE is shown in the bottom half of the table. We find that these latter category of models perform best, with the best performance reported by the T5-3B model fine-tuned on WICE. 14
3.4 Entailment Classification by GPT
3.3 Entailment Classification using Retrieval Here, we use serrive-tteen-rediie rather than the MAX "stretching" strategy from Section 3.1.
Table shows the entailment classification perfor- mance of GPT-3.5 (Brown et al., 2020; Ouyang et al., 2022) and GPT-4 (OpenAI, 2023) on WICE with oracle retrieval, which uses the gold set supporting sentences. 18 We use few-shot prompt with three examples (Appendix H). We see that GPT-4 is stronger at entailment classification than our best fine-tuned subclaim-level model, but claim- level classification, which is expected to be more complex, is still challenging for simple few-shot prompting even with the oracle retricval.
Setup We retrieve the top-k (= 7, in our exper- 15 sentences using the sentence-level re- trieval scores in Section 3.2. 16 These sentences are concatenated to construct a new premise/evidence; this is used by the chunk-level NLI model to make a document-level judgment: se(c, F) = P(y entailed C; Ei,1 ... Ei,)), where e,,1 ... 6i, the top-k retrieved sentences. This strategy is simi- lar Nie al. (2019).
Although we evaluate the GPT models with oracle retrieval, we believe that future work explore how to scale GPT-4 work over long- document entailment settings (e.g., tradeoffss of us- ing stretching vs. feeding in contexts up to the max- imum size allowed by GPT-4) and reduce its cost so it can be practically deployed for fact verification in real workflows.
Results We report the performance of the chunk- level 3B-3B model fine-tuned on ANLI+ MIL+WICE in Table 7. It shows that the retrieve-then- then-predict strat-
When no supporting sentence is retrieved for a claim/subclaim, we define the precision, recall, and Fl for this case as zero.
"In the ANCF+WICF setting, pre-trained the model on ANLI, which does not have the evidence context, and further trained WICF, which has the evidence context. This mismatch may be reason for the drop in performance.
'To avoid biases caused by the input length, we add or remove sentences in the oracle setting make input evidences up to but longer than 256 tokens. For nop-suppot cases with no gold set supporting sentences, we default to the MAX setting from Section 3.1. Refer to Appendix G for details the oracle retrieval.
5988 of claims in the development set have equal to or less than 7 gold supporting sentences.
16 "For retrieval, we use T5-3B finetuned on ANLI+WICE w/o evidence context and T5-38 finetuned on WICE w/ evi- dence context, which are the hest models in each setting.
We use first 100 claims/subclaims from the oracle retrieval datasct hecause the GPT-4 evaluation is costly.

---

broken shackle and chain lay her us she walks forward, commenooraing the recent national abolition of slavery.
Fine-tuned T5-3B models on WICE
Test Data
Train Data
Standard
w/ Spain
Claim-Split
Subclaim
Claim
only WICE
82.0
88.0
WICE
+ ANII
88.2
89.7
broken shackle and chain lay at her feet.
0.80
Harmonic Mean
NLI Model
Off-the-shelff models trained on ANLI
This is commemorating the recent national abolition of slavery.
0.10
+
0.24
Test Data
Model
Standard w/ Clain-Split
T5-Large T5-3B
79.2 80.2
83,3"
She walks forward.
0.80
WICE
83,2
Figure 3: Entailment classification using Claim-Split. We decompose a claim into subclaims, predict subclaim- level entailment scores, then aggregate these get the score for the claim.
VitaminC (long)
T5-Large T5-3B
80.2
87.2"
90.6
92.8"
PAWS (long)
T5-Large T5-3B
83.7
86.1*
89.5
88.8
FRANK
T5-Large T5-3B
86.7
89.7
(XSum)
93.2
93.7
4 Is Claim-Split useful for entailment classification?
Table 9: Comparison of AUROC scores for claim-level entailment classification task using the standard and "w/ Claim-Split" method. Table 15 includes results in Fl and accuracy. *: improvement from the standard method is statistically significantly with p-value < 0.05 according to paired bootstrap test.
We introduced Claim-Split as a method to provide fine-grained annotation by decomposing claims into simpler subclaims, and we used it with human verification to ensure the quality of our dataset. However, human evaluation in Section 2.1 shows that Claim-Split makes mistakes less than 10% of the time, indicating its robustness. In this section, we aim demonstrate the potential impact of Claim-Split beyond the dataset construction and its applicability to diverse domains. Specifi- cally, we test the hypothesis that Claim-Split makes entailment classification easier and im- proves the performance of NLI models. To show this, we evaluate a strategy of entailment prediction by aggregating the subclaim level entailment score four datasets.
Test Data In addition to WICE, we report re- sults the test sets of three datasets from Hon- ovich al. (2022): VitaminC (fact-verification) (Schuster et al., 2021), PAWS (paraphrase) (Zhang et al., 2019), and FRANK-XSum (summariza- tion) (Pagnoni et al., 2021). We evaluate the 500 longest claims in VitaminC and PAWS, using length as proxy for complexity that Claim-Split is designed for. 20 To decompose claims using Claim-Split,
unique prompt for each dataset that includes 2-4 dataset-specific examples (Appendix D.1).
Setup Given a claim-evidence (c, pair and target entailment label y, we compare the perfor- mance of two configurations that use identical en- tailment models for prediction:
Models
We evaluate the performance of T5 mod- els fine-tumed on ANLI2 for off-the-sheil settings and on WICE for fine-tune settings. For WICE, the MAX setting described in Section 3.1. Note that use the same trained models when comparing standard and w/ Claim-Split settings.
1. Standard: As is the typical mode of inference with these models, we directly predict the entail- ment label for the original claim c. The experi- ments in Section 3 are also in this setting.
Results We report the AUROC metric for the en- tailment classification task in Table 9. For most model-dataset pairs, "w/ Claim-Split" method outperforms the standard method of using off-the- shelf models. For the smaller T5-Large model. we observe statistically significant improvements
2. w/ Claim-Split: We first predict subclaim- level entailment scores for each subclaim ey E Claim-Split(c).. These are combined into a claim-level entailment score using harmonic mean." 19 Figure 3 describes this configuration.
This increases the mean length VitaminC from 15.5 to 37.5, and PAWS from 21.0 to 30.1. For FRANK, we use the XSum subset to restrict our analysis one sentence claims.
19 AAlthough more intuitive, we found that aggregating using min is quite sensitive to mistakes made by the entailment models or the Claim-Split method. Harmonic mean was good balance between min and arithmetic mean.
21 Honovich et al. (2022) show that T5-11B trained on ANLI worked best for many off the--fh settings, hut we found that we could achieve competitive performance with T5-38.

---

using Claim-Split for three datasets. This is in- tuitive as reducing the complexity of the problem likely benefits models with lower capability. Our results show that despite the introduction of noise (discussed in Section 2.1), Claim-Split is effec- tive at simplifying the entailment classification task and improving performance. We expect that the use of better prompts and aggregation methods would lead to further improvement.
F'act-verfficttion datasets A separate line of datasets designed for multi-hop reasoning comes from fact verification (Thorne et al., 2018; Schuster et al., 2021). I). However, in practice, claims in these datasets rarely require multiple evidence sentences (Thorne et al., 2018, FEVER) or are skewed to- wards statements about quantities (Schuster et al., 2021, VitaminC). In recent work, Petroni et al. (2023) looked at the attribution task in the con- text of Wikipedia citations, but only at the coarse level of finding better supporting document.
5 Discussion: Outstanding Challenges
Explainable entailment. Our end goal is an ex- plainable document-level entailment system that able to localize non-factuality within claims, as our subclaims and token-level annotation allow. This requires surfacing the right evidence, which we show remains a hard problem.
Hypothesis Decomposition In summarization, the Pyramid method (Nenkova and Passonneau, 2004) and its recent automated variants (Shapira et al., 2019; Zhang and Bansal, 2021; Liu et al.. 2023) decompose summary into semantic con- tent units, but is primarily aimed at understanding what content is covered rather than the reliabil- ity of that content. More recent frameworks have looked at breaking statements down into propo- sitions (Stanovsky et al., 2018; Ernst et al.. 2021; Chen et al., 2022, 2023); our approach similar but does not rely on supervised judgments and is not re- stricted to token extraction. Also, purely extractive methods are not suitable for use with off-the-shelf entailment models compared to Claim-Split Work on factuality in summarization has looked at entailment of sub-sentence units like depen- dency ares (Goyal and Durrett, 2020, 2021) or us- ing question-answer pairs to isolate specific pieces of information (Wang et al., 2020; Durmus et al., 2020; Scialom et al., 2021). However, these and related frameworks like QA-SRL (He al., 2015) fine-grained for annotation scheme.
Unsupported token detection. Although, we did conduct experiments on this, we believe that localization of errors is an important problem and difficult to address with existing methods (Kamoi et al., 2023). This problem should be further stud- ied in context of decomposition techniques like Claim-Split.
Better understanding of contextualization. Fi- nally, believe that the nature of contextualiza- tion remains major unsolved problem. While decontextualizing claims an appealing possibil- ity (Choi et al., 2021), we found that not all claims were easy to succinctly decontextualize. For exam- ple, The fresco is figures... in Figure 2 theoreti- cally requires specifying quite lot of information to understand exactly what fresco is being referred to, which removes some of the benefits of subclaim splitting, Our view that subelaims-in-cotte is natural unit explore, but further work is needed to substantiate this experimentally.
7 Conclusion
We collect WICE, a new NLI dataset constructed from Wikipedia. By comparing sentences in Wikipedia against cited evidence documents, we find rich set of real-world entailment phenomena distinct from those in prior NLI datasets. We also show that decomposing complex claims into sub- claims can be a valuable pre-process pre-processing step for both annotation and entailment prediction.
6 Related Work
Short-paragraph entailment The majority of NLI datasets have short premises and hypothe- ses, i.e., single-sentence (Bowman et al., 2015; Williams et al., 2018; Liu et al., 2022) or short paragraphs (Nic et al., 2020, ANLI), and involve less multi-step reasoning. There are few excep- tions; however, ContractNLI (Koreeda and Man- ning, 2021) is restricted to a single domain (con- tracts), and DocNLI (Yin et al., 2021) uses syn- thetic negative data (e.g., word replacement).
Limitations
Scope and Diversity of the Dataset Although we propose WICE to evaluate and improve models for evaluating real-world entailment, this dataset only includes claims in English Wikipedia articles

---

and evidence in the cited websites. We observe that WICE includes diverse claims and evidence, but there are many types of real-world claims that are very different from claims in WICE in both style and content, such as political claims and social media posts.
or different frescoes. These mismatches would likely be more prevalent if we used automatic re- trieval to find relevant documents. Instead, the linked documents from Wikipedia that perform the basis of our dataset are implicitly about the same entities as in the claims, so our models do not need to understand the context as thoroughly to evalu- ale the entailment relationships. Further work can explore broadening our findings to open-domain settings, including an open -omain version of our dataset.
Furthermore, almost all recent language models are pre-trained on Wikipedia articles. As a result, our dataset cannot evaluate truly zero-shot perfor- mance, given that models have been exposed to this text before. However, note that pretraining does not necessarily enable a model to know whether this fact Wikipedia is supported by this partic- ular document; we believe that many unsupported claims in our dataset are true, just not supported by the particular evidence documents. The fact that all models in our experiments are pre-trained on Wikipedia, yet they do not all perform uniformly well, supports this point. Developing datasets that are based on brand-new texts is promising di- rection for future work, in order to evaluate the performance in truly zero-shot condition.
A Acknovledgments
This work was principally supported by funding from Applied Research Laboratories, The Univer- sity of Texas at Austin, and UT Austin's Creating Connections program. This work was also partially supported by NSF CAREER Award IIS- 2145280, Good Systems, 22 a grant from Open Philanthropy. a gift from Salesforce, Inc., and a gift from Adobe.
References
The experiments in this paper are mainly conducted on T5-38, which is smaller than recent large language models (LLMs). Al- though we evaluate GPT-3.5 and GPT-4 on the ora- cle retrieval dataset in Section 3.4, we do not evalu- ate L.Mss on the full-pipeline experiments (retrieve supporting sentences from evidence articles for en- tailment classification, or feed the whole evidence articles to the models). Nevertheless, our dataset can provide realistic testbed for experiments eval- uating the ability of LLMs on long documents.
Baseline Models
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference Empiri- cal Methods Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, 2020. Language Models are Few-Shot Learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Context for Claims Although many existing NLI datasets target short and independent claims and evidence, claims and evidence in the WICE dataset are in-context with the surrounding text. We experimentally show that the context for ev- idence sentences would be useful for supporting sentence retrieval. However, our experiments do not show that providing the context of claims im- proves the entailment evaluation performance on the WICE dataset, in spite of our observation that some claims require anaphora resolution. We hy- pothesize that this observation can be attributed to the nature of our dataset, where only relevant, cited evidence documents are used. Context would be much more important if in a case like Figure we were attempting to substantiate these claims based on evidence documents discussing different altars
Jiangjie Chen, Qiaoben Bao, Changzhi Sun. Xinbo Zhang, Jiaze Chen, Hao Zhou, Yanghua Xiao, and Lei Li. 2022. Loren: Logul-regul reason- ing for interpretable fact verification. Proceedings of the AAAI Conference on Artificial Intelligence. 36(10): 10482-1091.
Jifan Chen, Eunsol Choi, and Greg Durrett. 2021. Can NLI models verify QA systems' predictions? In Findings of the Association for Computational Lin- guisties: EMNLP 2021, pages 3841-3854, Punta
Phttpss//goodyyttems.eees wtw.ws.eeee.......o... eduu/

---

Cana, Dominican Republic, Association for Compu- tational Linguistics.
Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, and Noah A. Smith. 2021. Competency problems: On finding and removing artifacts in language data. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 1801-1813, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, Dan Roth, and Tal Schuster. 2023. PropSegmEnt: A large- scale corpus for proposition-level segmentation and entailment recognition. In Findings of the Associa- tionfor Computational Linguistics: ACI. 2023, pages 8874-8892 8893, Toronto, Canada. Association for Com- putational Linguistics.
Timnit Gebru, Jamie Morgenstern, Briana Vec- chione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for Datascts. Commun. ACM, 64(12):86-92.
Eunsol Choi, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. 2021. Decontextualization: Making sen- tences stand-alone. Transactions of the Association for Computational Linguistics, 9:447-461.
Tanya Goyal and Greg Durrett. 2020. Evaluating Fac- tuality in Generation with Dependency- En- tailment. In Conference on Empirical Methods in Natural Language Processing (EMNLP), page 3592-3603.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Proceedings of the First Inter- national Conference on Machine Learning Chal- lenges: Evaluating Predictive Uncertainty Visual Ob- ject Classification, and Recognizing Textual Entail- ment, CLC500 page 177-190, Berlin, Heidelberg. Springer-Verte Verlag.
Tanya Goyal and Greg Durrett. 2021. Annotating and modeling fine-grained factuality in summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 1449-1462, Online. Association for Computa- tional Linguistics.
Sarkar Snigdha Sarathi Das. Chirag Shah. Mengting Wan, Jennifer Neville, Longgi Yang, Reid Ander- sen, Georg Buscher, and Tara Safavi. 2023. S3- dst: Structured open-domain dialogue segmentation and state tracking in the era of llms. arXiv preprint arXiv:2309.08827.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. FALTE: A Toolkit for Fine-graiee Fine-grained Annotation for Long Text Evaluation. In Proceedings of the The 2022 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pages 351-358, Abu Dhabi, UAE Association for Computational Linguistics.
Esin Durmus, He He, and Mona Diab. 2020. FEQA: A
question answering evaluation framework for faith- fulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 5055- 5070, Online. Association for Computational Lin- guistics.
Prakhar Gupta, Chien-Sheng Wa, Wenhao Liu, and Caiming Xiong. 2022. DialFact: A benchmark for fact-checking in dialogue. 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 3785-3801, Dublin, Ireland. Association for Compu- tational Linguistics.
Proceedings of the
Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2022. Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark. Transactions of the Association for Computational Linguistics, 10:1066-1083 1083.
Suchin Gururangan. Swabha Swayamdipta. Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language infer- ence data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Short Papers), pages 107-112, New Orleans. Louisiana. Association for Computa- tional Linguistics.
Ori Ernst, Ori Shapira, Ramakanth Pasunuru, Michael Lepioshkin, Jacob Goldberger, Mohit Bansal, and Ido Dagan. 2021. Summary-soure proposition-leww alignment: Task, datasets and supervised baseline. In Proceedings of the 25th Conference on Computa- tional Natural Language Learning, pages 310-222, Online. Association for Computational Linguistics.
Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015. Question-answer driven semantic role labeling: Us- ing natural language to annotate natural language. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 643-653, Lisbon, Portugal. Association for Compu- tational Linguistics.
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Rank- ing generated summaries by correctness: An interest- ing but challenging application for natural language inference. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 2214-2220, 2220, Florence, Italy. Association for Computational Linguistics.
Or Honovich, Roce Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and

---

Yossi Matias. 2022. TRUE: Re-evaluating factual consistency evaluation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3905-3920, Seattle, United States. Association for Computational Lin- guistics.
Bill MacCartney and Christopher D. Manning, 2009, An extended model of natural logic. In Proceed- ings of the Fight International Conference on Com- putational Semantics, pages 140-156, Tilburg, The Netherlands. Association for Computational Linguis- tics.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuris- tics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages Italy. Association for Computational Linguisties.
Or Homovich, Leshem Choshen, Roce Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. q²: Evaluating factual consistency in knowledge- grounded dialogues via question generation and ques- tion answering. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 7856-7870, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Florence.
Ani Nenkova and Rebecca Passonneau. 2004. Evaluat- ing content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chap- ter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145-152, Boston, Mas- sachusetts, USA. Association for Computational Lin- guistics.
Ryo Kamoi, Tamya Goyal, and Greg Durrett. 2023. Shortcomings of Question Answering Based Factual- ity Frameworks for Error Localization. In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, pages 332-14,, Dubrovnik, Croatia. Association for Computational Linguistics.
Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining Fact Extraction and Verification with Neural Semantic Matching Networks. In Proceed- ings of the Thirty-Third AAAI Conference on Arti- ficial Intelligence and Thirty-First Innovative Ap- plications of Artificial Intelligence Conference and Ninth AAAI Symposium Educational Advances in Artificial Intelligence, AAAI Press.
Yuta Koreeda and Christopher Manning. 2021. Con- tractNLI: A dataset for document-level natural lan- guage inference for contracts. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1907-1919. Punta Cana, Dominican Re- public. Association for Computational Linguistics.
1911A111AAIIII 19.
Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hcarst. 2022. SummaC: Re-visiting NLI- based models for inconsistency detection in summa- rization. Transactions of the Association for Compu- tational Linguistics, 10:16 163-177.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language under- standing. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Lànguistics, pages 4885-4901, Online, Association for Computa- tional Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: Lite BERT for Self-supervised Learning of Language Representations. In Interna- tional Conference on Learning Representations.
OpenAL. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens. Amanda Askell, Peter Welinder. Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-27744. Curran Associates, Inc.
Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. Wanli: Worker and ai collaboration for natural language inference dataset creation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretrain- ing Approach. arXiv preprint 197.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstrac- tive summarization with FRANK: A benchmark for factuality metries. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciution for Computational Linguistics: Human Lan- guage Technologies, pages 4812-4829, Online. As- sociation for Computational Linguistics.
Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Liny- ong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. 2023. Revisiting the gold standard: Ground- ing summarization evaluation with robust human evaluation. In Proceedings of the 61 Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). pages 4170- Toronto, Canada, Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor

---

Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, Perffrmmnee Deep Learn- ing Library. In Advances in Neural Information Pro- cessing Systems, volume 32. Curran Associates, Inc.
(Long and Short Papers), pages 682-687, Min- neapolis, Minnesota. Association for Computational Linguisties.
Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer. and Ido Dagan. 2018. Supervised open information extraction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Tech- nologies, Volume I (Long Papers), pages 885-895, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Fabio Petroni, Samuel Broscheit, Aleksandra Pik- tus. Patrick Lewis, Gautier Izacard. Lucas Hos- seini, Jane Yu. Maria Lomeli, Timo Schick, Michele Bevilacqua, Pierre-Emmanuel Mazaré, Ar- mand Joulin, Edouard Grave, and Sebastian Riedel. 2023. Improving wikipedia verifiability with ai. Na- ture Machine Intelligence, 5(10): 11-1
James Thorne, Andreas Vlachos, Christos Christoddollppulls,, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Pupers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguisties.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language infer- ence. In Proceedings of the Seventh Joint Confer- ence on Lexical and Computational Semantics, pages 180-191, New Orleans, Louisiana. Association for Computational Linguistics.
Robert Truswell. 2019. The Oxford handbook of event structure. Oxford University Press.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Lim- its Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21:1-67.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the fac- tual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 10085000, Online. Asso- ciation for Computational Linguistics.
Tal Schuster, Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, and Donald Metzler. 2022. Stretching Sentence-pair -arr NLI Models to Reason over Long Doc- uments and Clusters. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Adina Williams. Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long Papers), pages Louisiana. Association for Computational Linguis- tics.
New Orleans.
Tal Schuster, Adam Fisch, and Regina Barzilay. 2021.
Get your vitamin C. robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 624-643, 643, Online. As- sociation for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue. Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- ic7, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite. Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-ther thehhhart natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Thomas Scialom, Paul- Paul-Alexis Dray, Sylvain Lamprier. Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. Quest[vval: Summariza- tion asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 6594-6604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Wenpeng Yin, Dragomir Radev, and Caiming Xiong. 2021. DocNLI: A large-scale dataset for document- level natural language inference. In Findings of the Association for Computational Linguistics: ACI- IJCNLP 2021, pages 4913 4992,, Online. Association for Computational Linguistics.
Ori Shapira, David Gabay, Yang Gao, Hadar Ronen, Ra- makanth Pasunuru, Mohit Bansal, Yacl Amsterdamer, and Ido Dagan. 2019. Crowdsourcing lightweight pyramids for manual summary evaluation. In Pro- ceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume
Shiyue Zhang and Mohit Bansal. 2021. Finding a bal- anced degree of automation for summary evaluation.

---

Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 6617-6632, Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Error Category
%
Missing Intermediate Clause
16.7
Missing Details
13.3
Missing First Clause
10.0
Completeness
Mistake Parsing
10.0
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 1298 1308,, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Remove Parentheses
6.7
Missing "And"
6.7
Completeness and/or Correctness
Over-splitting
13.3
Table 10: Error analysis of Claim-Split on WICE. We annotated 30 mistakes in the development set. Each example can be assigned more than one category or left uncategorized (33.0%).
Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. GEAR: Graph-basee based evidence aggregating and rea- soning for fact verification. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pages 892-901, Florence, Italy. Association for Computational Linguistics.
echelon of women's 's tennis" missing in the de- composed sentences.
Original Sentence:
Before they established themselves the up- per echelon women's tennis, Dominique Van Roost was the only player Belgian history to ranked the top ten the ATP or WTA rank- ings, a mark she did not achieve until 1998 after Clijsters and Henin turned professional.
A Human Evaluation of Claim-Split
In Section 1, we evaluated the subclaims obtained using Claim-Split for completeness and correct- ness criteria. Figure 9 shows the instructions pro- vided to the crowd annotators for this task. It in- cludes three examples, one correct split of claims into subelaims, one example where the subclaims fail the completeness criterion and one where they fail the correctness criterion.
Decomposed Sentencos:
Dominique Van Roost was the only player in Belgian history to be ranked in the top ten of the ATP or WTA rankings. Dominique Van Roost achieved this mark in 1998.
Clijsters and Henin turned professional before Van Roost achieved this mark.
Given these instructions, each HIT asks anno- tators to verify 4 claims (and their corresponding subclaims from Claim-Split). For each tuple of context, claim, and decomposed subclaims, we ask the following questions: (1) Do all decomposed sentences correctly convey the information in the original sentence? [Yes/No]. (2) If you selected No: Which decomposed texts include mistakes? [Free Text]. (3) Decomposed sentences cover ALL information in the original sentence. [Yes/No]. (4) If you selected No: What information is missing? [Free Text]. We include (2) and (4) to improve and check the annotation quality, but we do not use the answers for these questions in the analysis.
Another relatively simple mistake is removing parentheses (6.7%). In the following example, de- composed sentences ignore "(Center for Predictive Engineering and Computational Sciences)" in the original sentence.
Original Sentence:
In 2009 he was appointed deputy director of the PECOS center (Center for Predictive Engineer- ing and Computational Sciences) at the Univer- sity of Texas.
Decomposed Sentences:
In 2009 he was appointed deputy director of the PECOS center.
The PECOS center is at the University of Texas.
Characterization of Claim-Split errors We manually characterized the Claim-Split errors in 30 dev examples of WICE. These statistics are shown in Table 10.
We expect the above mistakes and "Missing And" mistakes, which ignore some words or clauses con- nected by "and", could be solved by changing the prompt to include examples featuring these for- mats.
Some of the mistakes relatively simple and we expect that better prompts (few-shot examples) can fix them. For example, we observe that 30% of the mistakes are caused by removing the first or intermediate clauses. In the following example, "Before they established themselves in the upper
However, we also find errors that would be dif- ficult to solve. For example, in the following ex- ample, the fact that Howard lives with his wife Cerys is missing although decomposed sentence

---

Howard's wife Cerys is a doctor mentions his wife. Claim-Split with our prompt sometimes make mistakes when multiple decomposed sentences should be generated for a specific span in the origi- nal sentence; for his wife Cerys, decomposed sen- tences about two facts Howard lives with his wife Cerys and Howard's wife Cerys is doctor should be generated.
main article body; however, we included them in WICE because real-world entailment classification often requires dealing with noisy data.
Note that for claim sentences with multiple cita- tions, we only include claims with I or 2 citations positioned at the end of the sentence. Cases with larger numbers of citations mately 8. 8.)%) and typically represented either lists or multiple articles all in support of the same basc Howard lives in Camden, London with his wife Cerys, doctor, and their dog, Jack Russell Terrier named Archie.
infrequent (approxi-
Original Sentence:
fact.
Additional Filtering In Section 2.3, outlined additional filtering using RoBERTa-Large model to filter out trivially entailed claims, i.c., those for which all subclaims are predicted as entailed. Here, we provide more details of our process.
Decomposed Sentences:
Howard lives Camdon, London.
Howard's 's wife Cerys is a doctor.
Howard Cerys have dog named Archic.
Archie is a Jack Russell Terrier.
We use a pre-trained model fine-tuned on the DocNLI dataset provided by the authors of the dataset. 23 To deal with the long WICE evidences, split the documents into chunks of less than 256 tokens each. We predict entailment scores for each chunk- subblaim pair using the NLI model. For aggregation across chunks, we classify subclaim as entailed if it classified as entailed by at Ieast one chunk.
Another challenging type of mistake is caused by over-splitting. the following example, as a result of decomposition, the information of "after" is lost. In this case, a candidate of correct decomposition is not to decompose the latter two sentences as "Shine Limited was set up by former BSkyB executive Elis- abeth Murdoch after she quit broadcaster".
Original Sentence:
The production company that selected Shine Limited, which was set up by former BSkyB exccutive Elisabeth Murdoch after she quit as hroadcaster.
Task Interface Figure shows our annotation interface. The left panel shows the evidence articles which are split into sentences and numbered. The right panel shows the claim along with its preceding context. In the bottom half of the right panel, the sublclaims derived using Claim-Spl Salm-Split it are shown; all annotation is performed for these.
Decomposed Sentenees:
The production company that was selected was
Shine Limited.
Shine Limited was set up by former BSkyB executive Elisabeth Murdoch.
Each HIT includes annotation of one claim, i.c.. 2-6 subclaims. The median work time for each HIT was about 5 minutes ($9/h). 24 For each subclaim, the annotators first select the entailment classifi- cation label and, if applicable, the supporting sen- tences. If they select "Partially Supported / Not Supported" in the first step, the annotation inter- face for non-supported tokens is shown them (see Figure 5). For these cases, we consider subclaim as NOT-SUPPORTED if the annotator highlights all subclaim tokens, else PARTIALLY-SUPPON "Confirm" " button allows annotators to move to the next subclaim.
Elisabeth Murdoch quit as broadcaster.
We note that we have manually fixed these mis- takes in the dev and test set of the WICE dataset.
B Additional Data Collection Details
Base Dataset As mentioned in Section 2.3, we use the same articles-claims pairs from Wikipedia as the SIDE dataset (Petroni et al., 2023) but do not use their annotations or any other aspects of their pipeline. We re-retrieve the citations from Wikipedia directly because SIDE only contains one supporting evidence even when there are multiple in the data. For our dataset, use the Au- gust 2019 version for both Wikipedia and Common Crawl. We automatically parse the cited articles' HTML to extract the article text. This process is often quite noisy and may include extrancous sen- tences like "Click for more" that are not part of the
The
We build our annotation interface based on the FALTE annotation tool (Goyal et al., 2022).
Phttps: / thub. wwm./salesforcee/eeeeeooooe 20We aimed for $15/hr but workers took longer than we expected on the task. Annotators viewed our task favorably and completed it promptly, but we will strive to calibrate our pay estimates better in the future.

---

Titlr Whodd coren d (JRR)
Blst
Secties Hoztrry cofarecments
aph: begh 2C18, bnlowed start recomit yoior staion Fast earty 2016 Second peidin sico exuardedd port and parking lo replaced with pedosttian everpess with two slevetors. (CITATIO))
(PREV ciT)
Precs enarto Hight saucch
Prijec
Facis (decomppaed santense):
The dargercca prade ore naiinng Meen Stavel wea dowedd The pedeostion ceepoee aid hove two ewanos
Maen Street Crede Creaing Dimiutee
cuappas.
Current Past 11/
of abse land loin lin donng mimithe par ofaaa live expancion Pacc Mictrinn
in
di fact apported Whb oiinoo??
gasee
Suppotia
aooci c Edil sk diven, pedcations and zustones
fed
Par-di teeerea
lingity commuse othere Lif curbras
Chocoe seraen
support
crooing prize ane the posdice nouhh perconal
buring AMl and Pi hours
vostue the
fold carrwe crowing
For oober wnderee covilii tor come ietoraatio
doss iupppriii de anille cniia ialones samenciaa
talee with the Vilage Minwa war culer drooi
trude pormonitty
Fieect recaton
lalign
velcala tollice boing divetal
Aile Keemau
Thi ty ei erowing dvoggne ef renurator www.oooooooooooooooooooo wi yiggg Meodle and cefence
Figure Annotation interface for WICE annotation.
3: Highlight Supported Spans
Filtering Step
%removed #post-filtering
Before Filtering
1,545
pedestrian overpess would have fao alevators.
Missing Wikipedia pages
7.1
4,222
Hapil iit searc that are not apperted the senterens se you selected STEP
highlght
Bullet points
7.5
3, 9,905
thore are contrad itoos botwoon the fool and the atod intioo tig lightght iokons cormied::
Lte uentannne baneyee you setociad as STEP
Cite not at sentence end
17.1
3, 238
Prev cite not at sentence end
13.4
2, 2,805
#Citations >=
8.1
2,577
Missing in Common Crawl
24.3
1,951
Figure Annotation interface for highlighting non- supported tokens.
HTML postprocessing failed
14.9
1, 666
#subclaims or > RoBEKTa-large classified
19.1
1,343
all suhclaims as entailed
45.0
739
Filtering Steps We perform filtering at different stages of our dataset collection process, e.g., while retrieval of evidence articles from Common Crawl, cleaning the base dataset or during filtering using the NLI model. Table 11 shows the number of data points removed each step of the filtering process for the development set.
Table 11: Statistics of the filtering of the development
set. The original size is 545 and the final size is 739. Note that did not annotate all these data and the number of claims in development set is 349.
label selected in the above step). We prefer this over union as there can be multiple different sets of sentences with identical information (c.g., the date can sometimes be ascertained from several different sentences). We found that this subset of workers chose the exact same set of supported sen- tences for 56.1% of SUPPORTED cases and 34.4% of PARTIALLY-SSUPORTTI cases, which shows high inter-annotator agreement.
Aggregation of subclaim level labels from different workers For entailment classifica- tion, we take majority vote between worker labels. If no majority exists but 2 anno- tators each select PARTIALLY-SUPPORTTE NOT-SUPPORTED for the dev test set, choose PARTIALLY-SUPPORTED as the final label. In all other scenarios, we remove the subclaim (and the corresponding claim c) from our dataset as these cases tend to be quite subjective. This filters out 12.5% of the claims in the development set.
and
To aggregate unsupported tokens in subclaims, take token-level union of all workers who chose
as the entailment label. For this task, we remove data points if any annotator disagrees with the final set of tokens by more than three tokens (25.3% subclaims in the develop-
For supporting evidence sentences, we retain individual sets of supporting sentences by all work- ers who chose the majority entailment label (the

---

Full WICE
Supported
Partially Supported
Sub-Cliim Cai
sub-Cleim
Sub Galm
Gaim
Glim
0.4
0.2
10
Supporting Sentences
Figure 6: Distribution of the number of supporting sentences for each claim and subclaim in the development set of WICE. The figures shows this distribution for PARTIALLY- SUPPPORTED, SUPPORTED and the combined set.
Train
Dev
Test
PORTT REPORITT SUPNOATO
# Partially Supp. subclaims
374
124
105
# Tokens/ / subclaim
12.8
13.0
14.4
Table 12: Statisties of WICE for non-supported token annotation. These annotations are for partially sup- ported subclaims with high inter-annotator agreement. Each subclaim in the development set includes 3.3 non- supported tokens (25.2% of tokens) on average.
(a) W WICE (Claim)
(b) WICE (Sub Claim)
NOT PETTI Am
ment set).
15
Additional Dataset Statistics and Analysis
(c) VitaminC real
(d) FEVER
Figure 7: Word overlap (recall of bigrams) between claims and evidence.
Supporting Sentences Figure shows the dis- tribution of the number of supporting sentences f PARTIALLY-SUPPOTTED. SUPPORTED and the com- bined set in WICE. Supported and partially sup- ported subclaims have almost the same number of supporting sentences annotated (1.9 on average), but supported claims have more supporting sen- tences annotated compared to partially supported claims (averages are 3.4 and 2.9).
subclaim level. This suggests that WICE does not suffer from spurious biases.
Analysis of Phenomenon In Section 2.5, we cat- egorized entailment relationships between claims and evidence into several categories. We provide examples for each category in Table 13. Note that the FEVER example, character is a person, rep- resents a typical type of the inferences required in this dataset, which are simple hypernymy.
Non-Supported Tokens After filtering out data points with low inter-annotator agreement on the annotation for non-supported tokens, there are 374 partially supported subclaims in the training data with an average of 12.8 tokens per subclaim in (Ta- ble 12). Partially supported subclaims have 3.3 non- supported tokens 2% of tokens in subclaims) on average in the development set.
Additionally, we also characterize distribution of semantic roles that form each claim. We use taxonomy drawn from Davidsonian event seman- tics (Truswell, 2019) and semantic roles to char- acterize what each claim about: this consists of events, properties (attributes describing partici- pants in an event; these include name, occupation, nationality, quantity, ordinal), location, time, rea- son (why something happened), manner, and ev- identials. Each claim may carry multiple labels. and subclaims consist of necessarily disjoint subsets of these categories, e.g., "Jones buttered his toast slowly" (event, manner) and "Jones but- tered his toast in the bathroom" (event, location). Table shows this distribution.
Word Overlap Figure 7 shows the word overlap between claims and evidence (the recall of claim bigrams that are also in the evidence) for WICE, VitaminC, and FEVER. Although VitaminC manu- ally created claims so that this overlap is lower, we observe that the real claims in WICE have compet- itively low claim-evidence overlap. Furthermore, examples corresponding to different entailment la- bels have similar word overlaps, especially at the

---

Verification Category
Dataset
Claim
Evidence
And cutting the ribbon will be Brian Wilkins, 82, who has been using the beach since 1938 and been enthusiustic supporter of the campaign to get back open.
Brian Wilkins had been using the beach since 1938.
Compression
WICE
First Lieutenant John R. Fox distinguished himself by extraordinary heroism at the risk of his own life 26 December 1944 in the Serchio River Valley Sector, in the vicinity of Sommocolonia, Italy
His actions were on Decem- ber 26, 1944, in the vicinity Sommocolonia, Italy.
Compression
WICE
w/ contextualizztion
Chicago has one of the world's 's largest and most diversi- fied economies, with more than four million employees and generating annual gross regional product (GRP) of over $609 billion.
The Chicago Board of Trade is the largest and most diverse derivatives market globally.
Paraphrase- Direct
WICE
As of 22 March, more than 336,000 cases of COVID-19 have been reported in over 189 countries and territories, resulting in more than 14,400 deaths and 96,000 recov- cries.
Cases of COVID-19 have been confirmed more than 185 countries.
Paraphrase Require Calculation
VitC
Two of MMA's top 335-pound female fighters will collide a five-round title fight Jackson's MMA Series on April 9th in Albuquerque, New Mexico. Both fighters recently took part in the ill-fated UI- timate Women Challenge reality show competition last year, though results from the fights that took place during filming have yet to be released.
Young faced Kedzie again in five-round title rematch.
WICE
Paraphrase Require Inference
Wilhelmina Vivian Slater born Wanda Slater fictional character American dramedy series Ugly Belty.
FEVER Wilhelmina Slater a person.
Paraphrase Require
Vic Groves went close equaliser but the 'Busby Babes' held out for famous 5-4 victory.
WICE
United defeated Arsenal 5-4.
Background Knowledge
Table 13: Examples categories of entailment classification problems Table 3.
What is being asserted?
%
Sentence: <example claim>
Facts:
Event
52.9
<example subclaim>
Time
3.6
<example subclaim>
Location
10.7
Manner
6,4
Reason
7.9
Sentence: <input claims
Property
Name
6,4
Facts:
Occupation
5.7
Quantity
12.9
For the experiments in Section on VitaminC, PAWS, and FRANK, we use the following instruc- tion with three or four examples to generate sub- claims that are suitable for the off-the-shhele evalua- tion by entailment classification models:
Ordinal
3.6
Other
10.0
Others
2.9
Table 14: Estimated distribution of subclaims types in WICE. Each subclaim may have multiple properties.
Please decompose the following sentence into decontextualized sentences while ensuring all information is retained and the wording is as unchanged possible (please return the original sentence if cannot be decomposed):
D Claim-Split
This section provides further details and results of Claim-Split.
Full prompts with few-shot examples are pro- vided in our GitHub repository.
D.1 Claim-Split Prompts
We used the following prompt template with six examples:
D.2 Claim-Split Aggregation Performance
Table 15 shows additional results for Claim-Split aggregation experiments in Section 4. We observe
Segment the following sentence into individual facts:

---

FI
Accuracy Claim-Split
AUROC
Dataset
Model
Train Data
Original
Claim-Split
Original
Original
Claim-Split
TS-large
ANLI
61.8
64.5
70.7
70.1
79.2
83.3*
T5-3B
ANLI
64.3
64.1
75.1
71.5
80.2
83.2
WICE
T5-3B
DocNLI
56.4
62,44
62.8
70.7"
72.5
77.5
T5-3B
WiCE
65.3
72.7*
77.1
82.4"
82.0
88.0"
T5-3B
ANII+WiCR
72.1
71.4
79.1
80.7
88.2
89.7
VitC
T5-large TS-3B
ANLI
79.2
81.0
78.0
81,4"
86.7
90.4"
(long)
ANLI
86.0
85.2
85.8
85.6
91.6
92.6
PAWS
TS large TS-3B
ANLI
70.6
73,2
72.8
79.6"
81.7
87.2"
(long)
ANLI
74.1
75.2
77.2
79.0
86.2
89.4"
FRANK
Tarlerge
ANLI
40.0
32.2
95.7
86.1
86.6
86.0
(XSum)
T5-3B
ANLI
38.0
46.6"
87.9
92,1
91.1
91.4
Table 15: Claim-level entailment classification performance by aggregating scores for the subclaims generated by Claim-Split. Claim-Split improves entailment classification performance, especially on smaller T5-large models. * p-value 0.05 in paired bootstrap test.
improvement in almost all models and metrics. This result suggests that Claim-Spli effectively reduces the complexity of the claims.
partitions, i.e., (pi, c, Up,) Here, we describe how we derive these gold labels Up, using the supporting sentences annotation included in WICE.
For (pi, c), if the partition p; does not include any supporting sentence of c, we label y/, NOT-SUPPORTED. the partition includes all sup- porting sentences for с and the original entail- ment label in WICE is SUPPORTED, we label Vp. as SUPPORTED. For all other cases, we set Up, as PARTIALLY-SUPPORET
E Implementation Details
We use PyTorch (Paszke et al., 2019) and Hug- ging Face Transformers (Wolf et al., 2020) libraries in our implementation. We use machine with NVIDIA Quadro RTX 8000.
E.1 Baseline Models
E.3 Fine-tuning T5 Models
When available, we use NLI models provided by prior works and available on the HuggingFace 2ub for experiments in Section 3.1. The model names provided in Table 16.
Input Format The input format to our T5 mod- els is "entailment: claim [SEP] evidence". The T5 models are trained to generate the follow- ing single-character tokens corresponding to entail- ment labels: (entailed supported), p (partially supported), (not supported or neutral), and (con- tradiction). For example, fine-tuning on WICE uses e, p. and As the model places a probability distribution over the whole vocabulary, we normal- ize over our target set of labels to get classification probabilities.
We fine-tune T5 models VitaminC + MNLI using the dataset provided by Schuster al. (2021). For fine-tuning on ANLI and DocNLI, we use the datasets provided by the authors.
For evaluating the BM25 performance in Table 6, we use the rank_bm25 library."
E.2 Training Data of WICE
The WICE dataset provides entailment labels y for each cvidenee-clain/suullii pair, i.e.. (E, ..,.) y). However, due to the size of the docu- ment, we need to partition it into multiple partitions {P1. P2, Pm } (either individual sentences or con- catenation of sentences of at most 256 tokens) and pairs. To
We fine-tuned T5 (T5-large and T5-3B) for 25K steps with learning rate of 10 and batch size of 32. We compute the accu- racy on the development set after every 1, 000 steps and save the best model checkpoint. We use a max- imum input length of 512 tokens during training, but feed all input tokens during inference.
Hyperparameters
train on these cewiiiiion-cliimsuuubbboo do this, need entailment labels for each of these
Training with input size 512 is suitable for DocNLI, which includes many premises longer than 512 tokens. We use the input size 512 fol- lowing Yin al. (2021). Although this the best
hhtttps://huggiii
co/nodels
16 ANLL: https://gi thub. con/facebookresearch anli. DocNLI: https://gtt thub. con/ com/salesforce/Deekk hhhttps://githhee com/ dor wwwiiannnbbaaaana helbboow/rannkbhae bm25_

---

Model
Train Data
Model Name
RoBERTa-large
SNLI
boychaboySSMLL_rreetta-BBB
RoBERTa-large
MNLI
roberta-llage--mil
ALBERT-xlarge ALRERT-xlaree
VitaminC
tals//abberttxxlaree-itttaii
VitaminC+MNII
Table 16: Publicly available models in Hugging Face Hub used in our experiments in Section VitaminC models are provided Schuster et al. (2021). These models also used in experiments in Laban al. (2022).
supporting sentences." We note that there can be multiple oracle chunks for each claim/subclaim be- cause different annotators can annotate different sets of supporting sentences, which include suffi- cient information to support (or partially support) the claim/subclaim.
FI 0. Accuracy
0.75
Clain
Accuracy
Sub Clain
0.70
F1
Sub-Clain
0.65
Accuracy
To avoid biases caused by the number of sup- porting sentences (e.g. supported claims may have a larger number of supporting sentences than partially-supported claims), we add randomly se- lected sentences from the evidence article to the oracle chunks. Specifically, we add randomly lected sentences until the size of the chunks reaches 256 tokens (the chunks equal shorter than 256 tokens).
Claim in Train Data
Figure Relation between the size of train set and performance the entailment classification task of WICE on TB-3B.
setting in their experiments, this is a possible reason for the lower performance of the model finetuned DocNLI compared to the ANLI model.
For non-supported cases, which do not have any gold supporting sentences, we provide chunks as in the MAX setting in Section 3.1. The chunks in this setting also include at most 256 tokens in most cases."
F Effect of Training Data Size on Entailment Classification Performance
Figure 8 shows the performance of T5-3B fine- tuned on different amounts of training data from WICE. This result suggests that the performance of fine-tuning WICE is saturating with this train- ing procedure and models used in this paper, that much larger amounts of training data are needed improve the performance.
Finally, avoid biases caused by the number of oracle chunks, we randomly select three ora- cle chunks for each claim/subclaim. straightfor- ward way of performing entailment classification on the oracle retrieval dataset is to evaluate the entailment score for every oracle chunk for each claim/subclaim and take the maximum entailment Therefore, clains/subclains with large number of oracle chunks are likely to receive higher entailment scores. To avoid this bias, make ev- ery claim/subclaim in the oracle retrieval dataset has the same number of oracle chunks.
Oracle Retrieval Dataset
WICE is a dataset that is designed for evaluating entailment classification on long evidence articles, it requires supporting sentence retrieval as first step for language models that cannot evaluate very long inputs. To solely evaluate the entailment classification capability of language models, we make the oracle retrieval dataset, which is designed to simulate the situation in which we have an ideal retrieval model. This dataset provides a gold set of supporting sentences as input to NLI models. The oracle retrieval dataset is used in experiments in Section 3.3 and 3.4.
H Entailment Classification by GPT
We provide details of the entailment classification experiment on GPT-3.5 and GPT-4 in Section 3.4.
Whee the oracle chunk becomes longer than 256 tokens only with gold supporting sentences, we remove gold supporting sentences to make the chunk shorter than equal 256 tokens.
2We do not split sentences into sub-sentences when we make chunks, Therefore, when an evidence article includes sentence longer than 256 tokens, we include the sentence in the chunk as is.
The oracle retrieval dataset consists of oracle chunks that include all sentences in a gold set of

---

Models We use 300-3.5-turbo-065 and gpt-4-0613 for this experiment.
Datasheet for WICE
In this section, we provide a datasheet (Gebru et al., 2021) for the WICE dataset.
Dataset We use the first 100 claims/subclaims of the oracle retrieval dataset (Appendix G) for this experiment.
1.1 Motivation for Datasheet Creation
The information regarding the individuals or orga- nizations who created or funded the dataset will be included in the camera-ready version.
Prompts We provide the prompt used in this ex- periment below. We use XML as an output format in (Das al., 2023) make post-processing easy. Our prompt includes three examples from the development set of WICE, but we omitted two ex- amples and evidence sentences the first example, Our GitHub repository includes the full prompt.
For what purpose was the dataset created? There some major challenges when applying modern entailment models to measure real-world attribution and factuality consistency. Specifically, existing natural language inference (NLI) models and datasets target relatively short claims and evi- dence, negative examples are often artificially cre- ated, and fine- rve-ggained labels have not been studied well. We create the WICE dataset to address these limitations in the existing NLI datasets.
Your task is to evaluate if claim is supported by provided evidence article snippet.
You need to present your explanation first, and then choose your conclusion from the options [supported, partialll_sppoorte not_ s.ppprttedd.
We provide several examples. Your response must be in the same format as the XML in the exam- ples.
1.2 Dataset Composition
What are the instances? Each instance in WICE is group of subclaims derived from claim sentence Wikipedia article) and evidence (cited websites).
Examples: <imput>
<claim>On August 22, 2017, Richard Amardi was selected to play for the Canadian Senior Men's National Team to compete in the FIBA AmeriCup 2017 in Argentina.c/claime <evidence>
Is there a label or target associated with each instance? The annotation for each subelaim in- cludes the entailment label (supported, partially- supported, not-supported), supporting sentences (a subset of evidence sentences that support or par- tially support the subclaim), and non-supported to- kens (tokens in the subclaim that are not supported by the evidence).
<sentence_ <sentence_3V>>Values 39>-val
Vision and
Mis-
sionsc/sentece__55
<sentence_401ss 40b# SENIOR MEN'S NATIONAL
TEAM ANNOUNCES FIBA AMERICUP 2017 ROSTER</senlence__e ~0>
(Evidence Sentences are Omitted)
<devidence>
</input>
<!- Your explanation and answer should be writ-
How many instances are there? WICE includes 1, 260, 349, and 358 claims in the train, develop- ment, and test data, which are decomposed into 3, 170, 949, and 958 subclaims. Detailed dataset statisties are provided Table 1.
ten below ->
<output>
<explanation-Senttmnee 41 says that the final ros- ter for the Senior Men's National Team set to compete at the FIBA AmeriCup 2017 in Ar- gentina was announced on August 22, 2017. Sen- tence 77 shows that Richard Amardi is on the list, list://explanation>> <unserpsuppootell/asswwwr
Does the dataset contain all possible instances or is it a sample of instances from a larger set? The claims in WICE are a sub-set of sentences in Wikipedia articles. The sentences are randomly se- lected from those used in the SIDE dataset (Petroni et al., 2023).
</output>
(Two examples are omitted)
Here is your task:
<input>
(claim>(claim)) </llaim>>
(clam
<evidence>
Is the dataset self-contained? Yes, all resources are included in our release.
[evidence]
</evidence>
</input>
<!- Your explanation and answer should be writ-
I.3 Data Collection Process
ten below ->
How was the data associated with each instance acquired? We acquired claims from Wikipedia
30
openai. awtenai.com/dccseee. com/docss/mdeess

---

and evidence from Common Crawl (the August 2019 version).
Who was involved in the data collection process and how were they compensated? The labels (the entailment classification, supporting sentences, and non-supported tokens) are annotated by work- recruited using Mechanical Turk. Each instance is annotated by five workers. Annotators were paid $0.78 75 per claim annotation.
Over what timeframe was the data collected? Claims and evidence are collected from the Au- gust 2019 version of Wikipedia and Common Crawl. The annotation by workers conducted in November and December 2022.
1,4 Data Preprocessing
What preprocessing / cleaning was done? We automatically parse the cited articles' HTML to extract the article text. addition, we decom- pose claims (sentences in Wikipedia) by using Claim-Split (Section 2.3). Details of the filter- ing process are described in Appendix B.
What software was used to preprocess the data? We use Beautiful Soup to extract sentences from the HTML of the retrieved articles. We use GPT- 3.5 (Brown et al., 2020; Ouyang et al., 2022) to decompose claims into subclaims. Specifically, we use the text-davinci-002 model through the OpenAl API in November 2022.
1.5 Dataset Distribution
How will the dataset be distributed? The WICE dataset is available in GitHub repository.
Who will supporting and maintaining the dataset? This dataset will be maintained by the authors of this paper.

---

that information you complex you CORRECTLY convey complex COVER complex senteence, judge
complex
Were, the 1at sentence further decompossed, but do not
problem.
Context:
On the next day, 1 more coffee shop was opened. The first Starbucks store in Blovakia opened in Aupark Shopping Center Bratislave on May 81, 2016, with two more stores contirmdd open in Bratislava by the and of 2016. February 2016, Howard Schulzz announced the oponiin stores Itay The first Italian Starbucks store was raauguatted Milan on Saptember& 2018. After Tasta Hoidingg acquired outlet licsnsing for South Aftican stores, Starbucks opened its first store in Bouth Affica in Rosebonk, Johhanesbuug on Thursday, April 21, 2016, and its second in the country at the end of April in Mall of Africa In May 2017, Starbucke announced it was commencing operations Jamaica, where the first store to open in the resort city of Montegoo Bay.
Original Sentence:
company
Boach, offering views
Decomposed
The company announced that its first stoww would be incated the shores of the worid--arouu Dector's Cáve Beach The store would ofter views the Carlabean See.
Bad Example
Hore, the 'sl senteece is incomedt becouse she was MOT the only member the Weshington Health Bereilt Exchange although sie war the only one who voled against the salary increase.
Context:
Taraaa Mosquada American politician poation
Seartle, Wishington,
the City Council represent at-linge
Origina
November 2013,
only member the Washington Health Benefit Exchange incressing health CEO
18%.
Decomposed Sentences:
In Novamber 2013, she was the only mambar of the Wisshington Health Benaft Exchange. 2. She voled against incressing the salary of the health sachange's a CEO by 13%.
Bad Example
Heme, indivdusal clain cornect but do al the information from the complex sentence. Here, cines not inclndle the fect that Hoveard Nvee wilh Ris wile.
Contect:
(nc contexti
Origina Sentenze:
Howard Cander, London with wife Carya. dog, Rusoall Terrier Archin.
Decompased Sentences:
1. Howard lvees Camden, London.
2. Howard's wite Cerys is doctor
3. Howard and Darys have a namad Archie.
4, Archiiele a Jack Russell Tameer
Figure Instructions given crowd annotators evaluate the correctness and completeness of subclaims generated using the Claim-Spli it method.


Reviews:
[{"review_id": "Z4x5QhIhy6", "reviewer": "Reviewer_SxkM", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed comments and valuable suggestions!\n\n> First, the contributions of the paper are not clearly described. I guess they are (1) the new entailment dataset that has the above three features, (2) the Claim-Split method, and (3) the quantitative analyses and experiments.\n\nYes, that is a precise summary of our contribution. We will make it clearer in the final version of the paper.\n\n> Second, it should be justified why the authors did not directly use GPT-3.5 for the three tasks\n\nThank you for your suggestion. In this work, we focused on evaluating the models fine-tuned on NLI datasets because prior work (e.g., Gao et al., 2021; Wei et al., 2022; Yue et al., 2023) showed that LLMs often do not work well on related entailment tasks in a zero-shot or few-shot condition, as we mentioned in the limitations section (line 610). In addition, using LLMs for very long context inputs would be costly.\n\nHowever, since we finalized the version of this work submitted to EMNLP, OpenAI has continued to reduce prices for its models, and also increased the maximum input lengths. We will conduct benchmarking of LLM methods in any future version. \n\n> Third, 50 samples might be too few to draw any conclusion about human performances for the tasks. Using only 50 samples should have been justified too.\n\nNote that in Tables 4 and 5, the gap between human performance and model performance is quite large. With 50 samples and p = 0.92, the standard deviation of a binomial distribution is roughly 2, or roughly 4%. Even allowing for this error, the human performance is still significantly higher than model performance.\n\nThese examples are time-consuming to annotate (as they require reading entire cited documents), and we wanted to conduct the annotation carefully ourselves to establish a realistic human upper bound.\n\nWe also note that we annotated 144 sub-claims because we annotated 50 claims in WiCE, so the number of subclaims is substantially higher. \n\n> Fourth, I wonder why AUROC was used for the entailment classification results in Table 8 while F1 and accuracy were used for the entailment classification results in Tables 4 and 5. This needs explanation.\n\nFor the experiment in Table 8, we used the TRUE benchmark (Honovich et al., 2022; mentioned in line 479).  Honovich et al. (2022) used AUROC in their main tables to accommodate datasets that don’t have separate development sets for tuning a classification threshold, so we used this metric here to make the results more directly comparable.\n\nHowever, since the datasets we used in Table 8 have both the development and open test sets, we can also calculate F1 and accuracy. We agree that it is better to also show F1 and accuracy and will add them in the final version of our paper.\n\n> What does it mean for a token to be supported?\n\nAnnotators selected tokens that correspond to a sub-part of a claim that is semantically not supported by the evidence. These frequently correspond to modifiers (“in 1998”, if this is not stated) or propositions as suggested here. Our annotators had a consistent view of what this means: we observe high inter-annotator agreement, as mentioned in line 1247, and we ensure the quality of the dataset by only using the cases with high inter-annotator agreement (line 1253).\n\n> What do you mean by \"ecologically valid\"? Do you mean naturally occurring negative examples?\n\nYes, we are referring to naturally occurring negative examples which are not artificially created. We will clarify our usage of the term.\n\n> \"In total, only 8.6% of the claims included one of the two types of errors.\" ... What are the two types?\n\nIt refers to the Completeness and Correctness errors, which are described in line 142.\n\n> \"we re-retrieve the cited web article(s)\" ... Why did you need to re-retrieve them?\n\nWe re-retrieved the cited web articles because SIDE only contains one evidence web article even when there are multiple on the original Wikipedia page (line 1166).\n\n> \"Also, we filter claims that are decomposed into either only one or more than six subclaims.\" ... Why did you need this filter? Why six?\n\nFirst, we removed the claims that cannot be decomposed by the Claim-Split because we intend to focus on complex claims. Second, we removed the claims that are decomposed into more than six sub-claims because typically they are not interesting cases; they often just include a list of examples. Six was a threshold where we observed that the examples became very unnatural. \n\n> In Table 3, WiCE Subcl and WiCE Claim do not add up to 100%.\n\nWe annotated 50 cases but claims/subclaims that are annotated as non-supported are excluded from this table because there is no entailment category for them. Therefore, the percentage for each category is not always an integer and does not necessarily add up to 100%. We will explain this clearly in the final version of this paper.\n\n> \"p(SUPPORTED) + 0.5 × p(PARTIALLY-SUPPORTED)\" ... How was this derived?\n\nThe intuition behind this formula is that we want to prefer the sentences that receive high scores for the \"supported\" category, while also accepting the partially supporting sentences. The parameter 0.5 was arbitrarily selected. We will explain this point in the final version of the paper.\n\n> \"As there can be multiple gold sets of supporting sentences for each claim/subclaim in WICE,\" ... I could not understand this clause.\n\nIn the data collection process of WiCE, we retain individual sets of supporting sentences by all workers who chose the majority entailment label (line 1239).\n\nFor example, when annotators provide the following annotations,\n* Annotator 1: entailment=supported, sentences=[1, 5, 9]\n* Annotator 2: entailment=supported, sentences=[1, 5, 18]\n* Annotator 3: entailment=supported, sentences=[1, 5, 9]\n* Annotator 4: entailment=supported, sentences=[1, 3, 9]\n* Annotator 5: entailment=partially_supported, sentences=[1, 5]\n\nThe aggregated annotation for this sub-claim will be entailment=supported, supporting sentences={[1, 5, 9], [1, 5, 18], [1, 3, 9]}\n\nWe use this strategy because there can be multiple different sets of sentences that correctly support the claim. In this example, it is possible that sentences [1, 5, 9], [1, 5, 18], and [1, 3, 9] all include sufficient information, but taking the union of these would be redundant. We will improve the description in the final version of our paper.\n\n> In Table 6, why was ANLI+WiCE worse than WiCE in the \"w/ evidence context\" setting, while it was better in the other setting?\n\nIn this experiment, we pre-trained the model on ANLI, which does not have the evidence context, and further trained it on WiCE, which has the evidence context. This mismatch may be a reason for the drop in performance.\n\n> I could not understand footnote 7.\n\nIn the oracle setting, our goal is to emulate a scenario where we have an ideal retrieval model and, subsequently, assess the maximum potential performance on the entailment classification task. However, if we simply provide the gold supporting sentences, it can introduce undesirable biases. For example, if there are many supporting sentences, the model can infer that the claim is supported without reading them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The new entailment dataset will be useful for textual entailment recognition research.\n\nA series of the quantitative analyses and experiments are informative for RTE researchers.\n", "reasons_to_reject": "The weakness of this paper is the lack of description and justification for some important aspects of the paper.\n\nFirst, the contributions of the paper are not clearly described.\nI guess they are (1) the new entailment dataset that has the above three features, (2) the Claim-Split method, and\n(3) the quantitative analyses and experiments.\n\nSecond, it should be justified why the authors did not directly use GPT-3.5 for the three tasks (entailment\nclassification, evidence retrieval, and non-supported token detection). The authors used GPT-3.5 for Claim-Split.\nTheir problem settings therefore allow to use GPT-3.5. It should be a reasonable baseline given that we can use it\nrelatively easily.\n\nThird, 50 samples might be too few to draw any conclusion about human performances for the tasks.\nUsing only 50 samples should have been justified too.\n\nFourth, I wonder why AUROC was used for the entailment classification results in Table 8 while\nF1 and accuracy were used for the entailment classification results in Tables 4 and 5. This needs explanation.\n\nLast, there are many descriptions that need more detailed explanations or justification.\nI understand that there may be many details that cannot be written in the paper.\nBut some of them seem important information or key research decisions. Below are some examples:\n\np.1\n\n* What does it mean for a token to be supported? A fact or a proposition can be supported.\nBut I don't really understand how a token is supported. Do you simply mean that a token is supported\nif the token is written in an evidence text?\n\n* What do you mean by \"ecologically valid\"? Do you mean naturally occurring negative examples?\n\np.3\n\n* \"In total, only 8.6% of the claims included one of the two types of errors.\"\n... What are the two types?\n\n* \"we re-retrieve the cited web article(s)\"\n... Why did you need to re-retrieve them?\n\n* \"Also, we filter claims that are decomposed into either only one or more than six subclaims.\"\n... Why did you need this filter? Why six?\n\np.4\n\n* In Table 3, WiCE Subcl and WiCE Claim do not add up to 100%.\n\np.6\n\n* \"p(SUPPORTED) + 0.5 × p(PARTIALLY-SUPPORTED)\"\n... How was this derived?\n\n* \"As there can be multiple gold sets of supporting sentences for each claim/subclaim in WICE,\"\n... I could not understand this clause.\n\n* In Table 6, why was ANLI+WiCE worse than WiCE in the \"w/ evidence context\" setting, while\nit was better in the other setting?\n\np.7\n\n* I could not understand footnote 7.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about a new entailment dataset created from Wikipedia.\nTheir dataset has three key features compared to existing similar datasets;\n(1) its premises are long;\n(2) negative examples are natural ones, i.e. not synthetic ones; and\n(3) its annotation is fine-grained and allows to identify which parts of claims are not supported by evidence text.\n\nQuantitative analyses in Table 3 shows that their dataset is more challenging than previous datasets (FEVER and VitaminC).\n\nThe dataset allows to conduct three entailment-related tasks;\n(a) entailment classification;\n(b) evidence retrieval; and\n(c) non-supported token detection.\n\nThe authors demonstrated that these three tasks defined on their dataset were difficult by comparing\nthe performances of several competitive models with those of humans.\n\nThe authors also proposed Claim-Split, which automatically splits a claim into sub-claims using GPT-3.5.\nTheir experiments showed that Claim-Split was effective for entailment classification.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "p.2\n\n\"Real world claims, such are those\"\n-->\n\"Real world claims, such as those\"\n\np.6\n\n\"Performance of models fine-tuned with evidence context on WICE is supported in the bottom half of the table.\"\n-->\n\"Performance of models fine-tuned with evidence context on WICE is reported in the bottom half of the table.\"\n\n"}}, {"review_id": "kdeWxPxVlE", "reviewer": "Reviewer_rRJ9", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time in reading and reviewing our paper.\n\n> The data comes from Wikipedia, which makes it difficult to test the model's capabilities, as many models are pre-trained on the Wikipedia.\n   \nAs you correctly pointed out, most language models have been exposed to the Wikipedia articles used in WiCE during pretraining. However, the models have not necessarily been exposed to the evidence articles, and even if they were, pretraining does not necessarily enable a model to know whether this fact on Wikipedia is supported by this particular document. Therefore, we believe that pre-training on Wikipedia articles does not strongly influence their performance on our dataset, except insofar as Wikipedia data is comfortably in domain for pre-trained LMs. In fact, our experiments showed that our dataset is challenging for the T5 models.\n\nWe agree that developing datasets that are based on brand-new texts is a promising direction for future work, in order to evaluate the performance in a truly zero-shot condition. \n\n> [1] Logic-Regularized Reasoning for Interpretable Fact Verication. AAAI 2022.\n\nThank you for pointing out the missing reference. We agree that this work in interpretable fact verification is related to our work, particularly with regard to Claim-Split and non-supported tokens detection. We will include this reference in the final version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n1. Fine-grained textual entailment is more realistic and deserves further research.\n2. Paper is well-written and the experiments are solid.\n\n", "reasons_to_reject": "\n1. The data comes from Wikipedia, which makes it difficult to test the model's capabilities, as many models are pre-trained on the Wikipedia.\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes WICE, a new fine-grained textual entailment dataset built on natural claim and evidence pairs extracted from Wikipedia.\nIt provides entailment judgments over subsentence units of the claim, and a minimal subset of evidence sentences that support each subclaim.\nExperiments show that decomposing complex claims into subclaims can be a valuable pre-processing step for both annotation and entailment prediction.\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "\n[1] Logic-Regularized Reasoning for Interpretable Fact Verification. AAAI 2022."}, "extra_text": {}}, {"review_id": "1AwSqgb2mG", "reviewer": "Reviewer_rBnT", "summary": "", "questions": "- what is the average hourly for m-turkers if you pay 0.75$ per HIT?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time in reading and reviewing our paper.\n\n> what is the average hourly for m-turkers if you pay 0.75$ per HIT?\n\nWe aimed for \\\\$15/hr but workers took longer than we expected on the task. The median work time per HIT was about 5 minutes, so the median payment was approximately \\\\$9/hr (line 1207). We calculated the median because some workers took a very long time and the average is affected by these outliers. Annotators viewed our task favorably and completed it promptly, but we will strive to calibrate our pay estimates better in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- good presentation and well written\n- paper identifies use-cases of models trained on Textual entailment and finds that the underlying datasets are not suited for such use cases. Hence, they propose a new dataset for this.\n- I think such a dataset is a useful and interesting contribution to the community\n- good amount of relevant experiments conducted\n", "reasons_to_reject": "- None really", "questions_to_authors": "- what is the average hourly for m-turkers if you pay 0.75$ per HIT?", "paper_topic_and_main_contributions": "This paper introduces a TE dataset built on natural claim and evidence pairs extracted from Wikipedia; especially annotations over sub-sentence units. The authors conduct sufficient NLP experiments to support claims about shortcomings of prior NLI datasets and the usefulness of the dataset.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers agree on the fact that the paper provides a valuable contribution to the domain. In particular, the proposed entailment dataset extracted from Wikipedia will be useful for textual entailment recognition research, to complement existing resources. The experiments are solid, and provide interesting insights for research on RTE.
The authors have addressed the reviewers' concern in the rebuttal phase (in particular the fact of clarifying the contributions of the paper, as well as some technical aspects and justification of choices in the experimental setting), providing detailed descriptions of the unclear aspects in the paper.
Decision: Accept-Main