WiCE: Real-World Entailment for Claims in Wikipedia
Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett

Automated Fact-Checking of Complex Claims
Claim
Climate change isn’t real
because Florida has had
119 hurricanes since 1850

Supported?
Not Supported?

2

Automated Fact-Checking of Complex Claims
Climate change doesn’t cause hurricanes

Claim
Climate change isn’t real
because Florida has had
119 hurricanes since 1850

Supported?
Not Supported?

[...] Climate change doesn’t cause hurricanes.
[...] hurricanes have been present in the
Atlantic for at least the past few thousand
years, and probably longer [...]
Climate change made 2020 hurricanes rainier[...] Climate change made the record-smashingdeadly 2020 Atlantic hurricane season
noticeably wetter, a new study says. [...]
Example from AP FACT CHECK

3

Automated Fact-Checking of Complex Claims
Climate change doesn’t cause hurricanes

Claim[...] Climate change doesn’t cause hurricanes.
[...] hurricanes have been present in the
Atlantic for at least the past few thousand
years, and probably longer [...]

Climate change isn’t real
because Florida has had
119 hurricanes since 1850

Climate change made 2020 hurricanes rainierSupported?
Supported?
Not Supported

NLI Model(e.g., Nie et al., AAAI 2019)[...] Climate change made the record-smashingdeadly 2020 Atlantic hurricane season
noticeably wetter, a new study says. [...]
Example from AP FACT CHECK

4

Automated Fact-Checking of Complex Claims
Climate change doesn’t cause hurricanes

Claim[...] Climate change doesn’t cause hurricanes.
[...] hurricanes have been present in the
Atlantic for at least the past few thousand
years, and probably longer [...]

Climate change isn’t real
because Florida has had
119 hurricanes since 1850

Climate change made 2020 hurricanes rainierSupported?
Supported?
Not Supported

NLI Model(e.g., Nie et al., AAAI 2019)[...] Climate change made the record-smashingdeadly 2020 Atlantic hurricane season
noticeably wetter, a new study says. [...]
Example from AP FACT CHECK

We need datasets suitable for training and analyzing fact-checking systems

5

How to Create Useful Dataset for Fact-Checking?
It is challenging to construct a dataset of real fact-checking examples

6

How to Create Useful Dataset for Fact-Checking?
It is challenging to construct a dataset of real fact-checking examples

In WiCE, we use claims and evidence in Wikipedia articles to approximate the fact-checking

7How to Create Useful Dataset for Fact-Checking?
It is challenging to construct a dataset of real fact-checking examples

In WiCE, we use claims and evidence in Wikipedia articles to approximate the fact-checking
Sentence in Wikipedia article
Claim… She was the only memberof the Washington Health
Benefit who voted against
increasing the salary of thehealth exchange's CEO.[7]…
8How to Create Useful Dataset for Fact-Checking?
It is challenging to construct a dataset of real fact-checking examples

In WiCE, we use claims and evidence in Wikipedia articles to approximate the fact-checking
Sentence in Wikipedia article
Claim… She was the only memberof the Washington Health
Benefit who voted against
increasing the salary of thehealth exchange's CEO.[7]…

Websites cited for the claim
Evidence: Cited Article [7]… All of the board members
in attendance voted in favorof the pay raise except one.
Teresa Mosqeda voted against
the motion. …
9Why WiCE is Suitable for Real-World Fact-Checking?
We need realistic testbeds and good training data for fact-checking systems
Existing Datasets●

Short premise○●

Synthetic negative cases○●

DocNLI (e.g., word replacement)

Lack of fine-grained annotation○●

SNLI, MNLI, WANLI, ANLI

ContractNLI includes fine-grained
annotation, but only for a single domain

Easy Verification Problems○

FEVER, VitaminC

10

Why WiCE is Suitable for Real-World Fact-Checking?
We need realistic testbeds and good training data for fact-checking systems
Existing Datasets●

Short premise○●

DocNLI (e.g., word replacement)

Lack of fine-grained annotation○●

SNLI, MNLI, WANLI, ANLI

Synthetic negative cases
○

●

WiCE

ContractNLI includes fine-grained
annotation, but only for a single domain

Easy Verification Problems○

FEVER, VitaminC

11

Why WiCE is Suitable for Real-World Fact-Checking?
We need realistic testbeds and good training data for fact-checking systems
Existing Datasets●

Short premise○●Long premise○

Includes 120 sentences on average

DocNLI (e.g., word replacement)

Lack of fine-grained annotation○●●

Synthetic negative cases○●

SNLI, MNLI, WANLI, ANLI

WiCE

ContractNLI includes fine-grained
annotation, but only for a single domain

Easy Verification Problems○

FEVER, VitaminC

12

Why WiCE is Suitable for Real-World Fact-Checking?
We need realistic testbeds and good training data for fact-checking systems
Existing Datasets●

Short premise○●●

DocNLI (e.g., word replacement)

Long premise○●

Includes 120 sentences on average

Natural negative cases○

Mistakes by Wikipedia authors

Lack of fine-grained annotation○●●

SNLI, MNLI, WANLI, ANLI

Synthetic negative cases○

WiCE

ContractNLI includes fine-grained
annotation, but only for a single domain

Easy Verification Problems○

FEVER, VitaminC

13

Why WiCE is Suitable for Real-World Fact-Checking?
We need realistic testbeds and good training data for fact-checking systems
Existing Datasets●

Short premise
○

●

●

●

ContractNLI includes fine-grained
annotation, but only for a single domainLong premise○●

DocNLI (e.g., word replacement)

Lack of fine-grained annotation○●

SNLI, MNLI, WANLI, ANLI

Synthetic negative cases○

WiCE

Natural negative cases○●

Includes 120 sentences on average

Mistakes by Wikipedia authors

Fine-grained annotation○○○

Sub-claim level annotation
Supporting sentences
Non-supported tokens

Easy Verification Problems○

FEVER, VitaminC

14

Why WiCE is Suitable for Real-World Fact-Checking?
We need realistic testbeds and good training data for fact-checking systems
Existing Datasets●

Short premise
○

●

●

●

○

●
●

FEVER, VitaminC

Mistakes by Wikipedia authors

Fine-grained annotation
○
○
○

●

Includes 120 sentences on average

Natural negative cases○

ContractNLI includes fine-grained
annotation, but only for a single domain

Easy Verification Problems

Long premise○

DocNLI (e.g., word replacement)

Lack of fine-grained annotation○●

SNLI, MNLI, WANLI, ANLI

Synthetic negative cases○

WiCE

Sub-claim level annotation
Supporting sentences
Non-supported tokens

Challenging Verification Problems

15

Our Contributions●

We collect a document-level textual entailment dataset based on Wikipedia○

1,967 claims (5,377 subclaims) in total○

Fine-grained annotation●

We propose a method to automatically decompose claims into subclaims●

We evaluate fact-checking frameworks on this dataset○

Fine-tuned T5-3B and few-shot GPT-4
16

OverviewConstruction of WiCE●

Sub-claims generated by Claim-Split●

Indices of supporting sentences●

Non-Supported Tokens

Experiments on WiCE●

How well do existing NLI models
perform on WiCE?

●

How well does GPT-4 perform?

17Fine-Grained Annotation on WiCE
Claim
… The STM 747 Shuttle Bus
replaced the "Aerobus" thatwas privately operated byGroupe La Québécoise.[7]…

Evidence: Cited Article [7]… The route is the 747 Express
bus, which … It also replaces
La Québécoise's Aérobus
shuttle service between the bus
station and the airport …

●

Claim (hypothesis)○●

Sentence in Wikipedia article

Evidence (premise)○

Websites cited for the claim

18Fine-Grained Annotation on WiCE
Claim
… The STM 747 Shuttle Bus
replaced the "Aerobus" thatwas privately operated byGroupe La Québécoise.[7]…

Evidence: Cited Article [7]… The route is the 747 Express
bus, which … It also replaces
La Québécoise's Aérobus
shuttle service between the bus
station and the airport …

●

Claim (hypothesis)○●

Sentence in Wikipedia article

Evidence (premise)○

Websites cited for the claim

The STM 747 Shuttle Bus replaced the "Aerobus."●

Subclaim-level annotation○

The "Aerobus" was privately operated by Groupe La Québécoise.

We split the original claim into
multiple independent sentences
19Fine-Grained Annotation on WiCE
Claim
… The STM 747 Shuttle Bus
replaced the "Aerobus" thatwas privately operated byGroupe La Québécoise.[7]…

Evidence: Cited Article [7]… The route is the 747 Express
bus, which … It also replaces
La Québécoise's Aérobus
shuttle service between the bus
station and the airport …

Annotation for each subclaim

The STM 747 Shuttle Bus replaced the "Aerobus."

The "Aerobus" was privately operated by Groupe La Québécoise.

20Fine-Grained Annotation on WiCE
Claim

Evidence: Cited Article [7]… The STM 747 Shuttle Bus
replaced the "Aerobus" thatwas privately operated byGroupe La Québécoise.[7]…

… The route is the 747 Express
bus, which … It also replaces
La Québécoise's Aérobus
shuttle service between the bus
station and the airport …

Annotation for each subclaim●

3-way entailment label○

The STM 747 Shuttle Bus replaced the "Aerobus."

Supported, Partially-Supported,
Not-Supported

Entailment Label: SUPPORTED

The "Aerobus" was privately operated by Groupe La Québécoise.
Entailment Label: PARTIALLY-SUPPORTED
21

Fine-Grained Annotation on WiCE
Claim

Evidence: Cited Article [7]… The STM 747 Shuttle Bus
replaced the "Aerobus" thatwas privately operated byGroupe La Québécoise.[7]…

… The route is the 747 Express
bus, which … It also replaces
La Québécoise's Aérobus
shuttle service between the bus
station and the airport …

Annotation for each subclaim●

3-way entailment label○

The STM 747 Shuttle Bus replaced the "Aerobus."
Entailment Label: SUPPORTED
Indices of Supporting Sentences: 9, 11●

Supported, Partially-Supported,
Not-Supported

Indices of supporting sentences

The "Aerobus" was privately operated by Groupe La Québécoise.
Entailment Label: PARTIALLY-SUPPORTED
Indices of Supporting Sentences: 1122

Fine-Grained Annotation on WiCE
Claim

Evidence: Cited Article [7]… The STM 747 Shuttle Bus
replaced the "Aerobus" thatwas privately operated byGroupe La Québécoise.[7]…

… The route is the 747 Express
bus, which … It also replaces
La Québécoise's Aérobus
shuttle service between the bus
station and the airport …

Annotation for each subclaim●

3-way entailment label○

The STM 747 Shuttle Bus replaced the "Aerobus."
Entailment Label: SUPPORTED
Indices of Supporting Sentences: 9, 11
The
The "Aerobus""Aerobus" waswas privately
privately operated by GroupeGroupe La
La Québécoise.
Québécoise.

Supported, Partially-Supported,
Not-Supported●

Indices of supporting sentences●

Non-supported tokens in claims(for Partially-Supported cases)

Entailment Label: PARTIALLY-SUPPORTED
Indices of Supporting Sentences: 1123

Claim-Split

The Société de transport de Montréal (STM) 747 Shuttle
Bus replaced the "Aerobus" that was privately operated
by Groupe La Québécoise.

Subclaims
The Société de transport de Montréal (STM) 747 Shuttle
Bus replaced the "Aerobus."
The "Aerobus" was privately operated by Groupe La
Québécoise.

24Claim-Split
PromptSegment the following sentence into individual facts:[in-context examples]
Original Sentence:
The Société de transport de Montréal (STM) 747 Shuttle
Bus replaced the "Aerobus" that was privately operated
by Groupe La Québécoise.

●

Use LLM with few-shot prompting(GPT-3.5 in our experiments) toautomatically split a claim into subclaims

LLM
Subclaims
The Société de transport de Montréal (STM) 747 Shuttle
Bus replaced the "Aerobus."
The "Aerobus" was privately operated by Groupe La
Québécoise.

25

Claim-Split
PromptSegment the following sentence into individual facts:[in-context examples]
Original Sentence:
The Société de transport de Montréal (STM) 747 Shuttle
Bus replaced the "Aerobus" that was privately operated
by Groupe La Québécoise.

LLM●

Use LLM with few-shot prompting(GPT-3.5 in our experiments) toautomatically split a claim into subclaims●

Subclaims are also complete sentences,making them easy to be used forannotation and downstream applications

Subclaims
The Société de transport de Montréal (STM) 747 Shuttle
Bus replaced the "Aerobus."
The "Aerobus" was privately operated by Groupe La
Québécoise.

26

Claim-Split
PromptSegment the following sentence into individual facts:[in-context examples]
Original Sentence:
The Société de transport de Montréal (STM) 747 Shuttle
Bus replaced the "Aerobus" that was privately operated
by Groupe La Québécoise.

●

Use LLM with few-shot prompting(GPT-3.5 in our experiments) toautomatically split a claim into subclaims●

Subclaims are also complete sentences,making them easy to be used forannotation and downstream applications●

GPT-3.5 makes only 8.6% of errors(We manually fixed the mistake for WiCE)

LLM
Subclaims
The Société de transport de Montréal (STM) 747 Shuttle
Bus replaced the "Aerobus."
The "Aerobus" was privately operated by Groupe La
Québécoise.

27

Supporting Sentences●

Evidence in WiCE includes
120 sentences in average●

Each claim has, on average,
3.1 supporting sentences28

Supporting Sentences●

Evidence in WiCE includes
120 sentences in average●

Each claim has, on average,
3.1 supporting sentences29Supporting Sentences●

Evidence in WiCE includes
120 sentences in average●

Each claim has, on average,
3.1 supporting sentences

30

Wikipedia Includes Complex Verification Problems
Claim-Evidence pair that can be verified by Direct Extraction (in VitaminC)
Panjaa received negative reviews .

Panjaa received largely negative reviews from critics.

Claim-Evidence pair that Requires Reasoning for verification (in WiCE)
She was the only member of the
Washington Health Benefit Exchange
who voted against increasing thesalary of the health exchange's CEO.

... All of the board members in attendance voted infavor of the pay raise except one. Teresa Mosqeda,
legislative and policy director for the Washington State
Labor Council, voted against the motion. ...
31

Wikipedia Includes Complex Verification Problems

More
Complex

Verification Problems in Fact-Verification Datasets (%)

●

FEVER

VitaminC

WiCE (claim)

Extraction

10

20

4

Easy Paraphrase

34

24

16

Require Reasoning

24

8

68

Many claims in existing datasets (FEVER and VitaminC) can be verified by simply
extracting corresponding information from evidence

32

Wikipedia Includes Complex Verification Problems

More
Complex

Verification Problems in Fact-Verification Datasets (%)
FEVER

VitaminC

WiCE (claim)

Extraction

10

20

4

Easy Paraphrase

34

24

16

Require Reasoning

24

8

68

●

Many claims in existing datasets (FEVER and VitaminC) can be verified by simply
extracting corresponding information from evidence●

Many claims in WiCE require reasoning to verify them, in addition to selecting
correct information in evidence
33

OverviewConstruction of WiCE

Experiments on WiCE●

Fine-grained annotation on
sub-claims generated by Claim-Split●

How well do existing NLI models
perform on WiCE?

●

Indices of supporting sentences●

How well does GPT-4 perform?

34

How to Evaluate Long Evidence?
Evidence Article
The Main Street grade crossing:posed a safety risk to drivers,
pedestrians and LIRR customers
contributed to noise and air pollution
caused lengthy commutes for bothdrivers and LIRR customers
contributed to noise
and air pollution...

●

Evidence in WiCE contains about 1600 tokens in average●

Many NLI models cannot accommodate long evidence

35How to Evaluate Long Evidence?
Evidence Article
The Main Street grade crossing:posed a safety risk to drivers,
pedestrians and LIRR customers

ChunksThe Main Street grade crossing:posed a safety risk to drivers, pedestrians
and LIRR customers
contributed to noise and air pollution

contributed to noise and air pollution
caused lengthy commutes for bothdrivers and LIRR customers
contributed to noise
and air pollution...

●

Evidence in WiCE contains about 1600 tokens in average●

Many NLI models cannot accommodate long evidence

36How to Evaluate Long Evidence?
Evidence Article
The Main Street grade crossing:posed a safety risk to drivers,
pedestrians and LIRR customers
contributed to noise and air pollution
caused lengthy commutes for bothdrivers and LIRR customers

ChunksThe Main Street grade crossing:posed a safety risk to drivers, pedestrians
and LIRR customers
contributed to noise and air pollution
posed a safety risk to drivers, pedestrians
and LIRR customers
contributed to noise and air pollution
caused lengthy commutes for both drivers
and LIRR customers

contributed to noise
and air pollution...

●

Evidence in WiCE contains about 1600 tokens in average●

Many NLI models cannot accommodate long evidence

37How to Evaluate Long Evidence?
Evidence Article
The Main Street grade crossing:posed a safety risk to drivers,
pedestrians and LIRR customers
contributed to noise and air pollution
caused lengthy commutes for bothdrivers and LIRR customers
contributed to noise
and air pollution

ChunksThe Main Street grade crossing:posed a safety risk to drivers, pedestrians
and LIRR customers
contributed to noise and air pollution
posed a safety risk to drivers, pedestrians
and LIRR customers
contributed to noise and air pollution
caused lengthy commutes for both drivers
and LIRR customers...

contributed to noise and air pollution
caused lengthy commutes for both drivers
and LIRR customers
caused lengthy commutes for both drivers
and LIRR customers●

Evidence in WiCE contains about 1600 tokens in average●

Many NLI models cannot accommodate long evidence

38How to Evaluate Long Evidence?
Evidence Article
The Main Street grade crossing:posed a safety risk to drivers,
pedestrians and LIRR customers
contributed to noise and air pollution
caused lengthy commutes for bothdrivers and LIRR customers
contributed to noise
and air pollution

ChunksChunk Scores

The Main Street grade crossing:posed a safety risk to drivers, pedestrians
and LIRR customers
contributed to noise and air pollution

0.9

posed a safety risk to drivers, pedestrians
and LIRR customers
contributed to noise and air pollution
caused lengthy commutes for both drivers
and LIRR customers

NLI Model

Claim
The dangerous grade
crossing would be closed...

contributed to noise and air pollution
caused lengthy commutes for both drivers
and LIRR customers
caused lengthy commutes for both drivers
and LIRR customers●

Evidence in WiCE contains about 1600 tokens in average●

Many NLI models cannot accommodate long evidence

39How to Evaluate Long Evidence?
Evidence Article
The Main Street grade crossing:posed a safety risk to drivers,
pedestrians and LIRR customers
contributed to noise and air pollution
caused lengthy commutes for bothdrivers and LIRR customers
contributed to noise
and air pollution

ChunksChunk Scores

The Main Street grade crossing:posed a safety risk to drivers, pedestrians
and LIRR customers
contributed to noise and air pollution

0.9

posed a safety risk to drivers, pedestrians
and LIRR customers
contributed to noise and air pollution
caused lengthy commutes for both drivers
and LIRR customers

Claim
0.7

NLI Model

The dangerous grade
crossing would be closed...

contributed to noise and air pollution
caused lengthy commutes for both drivers
and LIRR customers
caused lengthy commutes for both drivers
and LIRR customers●

Evidence in WiCE contains about 1600 tokens in average●

Many NLI models cannot accommodate long evidence

40How to Evaluate Long Evidence?
Evidence Article
The Main Street grade crossing:posed a safety risk to drivers,
pedestrians and LIRR customers
contributed to noise and air pollution
caused lengthy commutes for bothdrivers and LIRR customers
contributed to noise
and air pollution

ChunksChunk Scores

The Main Street grade crossing:posed a safety risk to drivers, pedestrians
and LIRR customers
contributed to noise and air pollution

0.9

posed a safety risk to drivers, pedestrians
and LIRR customers
contributed to noise and air pollution
caused lengthy commutes for both drivers
and LIRR customers...

contributed to noise and air pollution
caused lengthy commutes for both drivers
and LIRR customers
caused lengthy commutes for both drivers
and LIRR customers

Claim
0.7

The dangerous grade
crossing would be closed

NLI Model
0.2●

Evidence in WiCE contains about 1600 tokens in average●

Many NLI models cannot accommodate long evidence

41How to Evaluate Long Evidence?
Evidence Article
The Main Street grade crossing:posed a safety risk to drivers,
pedestrians and LIRR customers
contributed to noise and air pollution
caused lengthy commutes for bothdrivers and LIRR customers
contributed to noise
and air pollution

Chunk Scores

Doc Score

The Main Street grade crossing:posed a safety risk to drivers, pedestrians
and LIRR customers
contributed to noise and air pollution

0.9

posed a safety risk to drivers, pedestrians
and LIRR customers
contributed to noise and air pollution
caused lengthy commutes for both drivers
and LIRR customers

0.7

0.9

contributed to noise and air pollution
caused lengthy commutes for both drivers
and LIRR customers
caused lengthy commutes for both drivers
and LIRR customers

0.2

The maximum
chunk-levelentailment score

...

...

Chunks●Evidence in WiCE contains about 1600 tokens in average●

Many NLI models cannot accommodate long evidence

42Results: Maximum Chunk-level Scores of T5-3B●

We evaluate binary classification(Supported vs. Partially- or Not-Supported)43

Results: Maximum Chunk-level Scores of T5-3B●

We evaluate binary classification(Supported vs. Partially- or Not-Supported)●

Subclaim-level is much easier(Claim-Split is useful for improving inference)

44

Results: Maximum Chunk-level Scores of T5-3B●

We evaluate binary classification(Supported vs. Partially- or Not-Supported)●

Subclaim-level is much easier(Claim-Split is useful for improving inference)●

Fine-tuning on WiCE achieves
the best single-dataset performance45

Results: Maximum Chunk-level Scores of T5-3B●●

We evaluate binary classification(Supported vs. Partially- or Not-Supported)●

Fine-tuning on WiCE achieves
the best single-dataset performance●

Using multiple datasets is beneficial

Subclaim-level is much easier(Claim-Split is useful for improving inference)46

Results: Maximum Chunk-level Scores of T5-3B●●

We evaluate binary classification(Supported vs. Partially- or Not-Supported)●

Fine-tuning on WiCE achieves
the best single-dataset performance●

Using multiple datasets is beneficial●

Human Performance is much better

Subclaim-level is much easier(Claim-Split is useful for improving inference)47

Results: Few-Shot GPT-4
Performance (F1) with Oracle Retrieval●

Few-shot GPT-4 works well at
subclaim-level but not very well atclaim-level●

Much smaller fine-tuned model(T5-3B) is better than GPT-4 atclaim-level

WiCE is challenging for GPT-4 even with oracle retrieval (perfect evidence retrieval)48

Takeaways●●

WiCE is suitable for training and analyzing realistic fact-verification systems○

Long premises○

Natural negative cases○

Fine-grained annotation

WiCE is challenging for fine-tuned T5 and few-shot GPT-4

Thank you!
Please reach out us for any questions
ryokamoi@utexas.edu

49

Dataset Statistics

50

Verification Categories

51

Retrieve-then-Predict: Claim Verification by NLI
Evidence Article
The grade crossing being eliminated.
A pedestrian bridge will be constructed.

...
It posed a safety risk to drivers.

Claim

●

Evidence in WiCE contains about
1600 tokens in average●

Many NLI models cannot
accommodate long evidence●

We retrieve useful evidence
sentences and feed the retrievedsentences to NLI models

The dangerous grade
crossing would be closed

Retrieval Model

The grade crossing being
eliminated. ... It posed asafety risk to drivers.

NLI Model(Nie et al., AAAI 2019)52

Results: Retrieve-then-Predict

Entailment Classifier is T5-3B
fine-tuned on ANLI + WiCE

T5-based Retrievers●

Better retrieval model improves theentailment classification performance53

Results: Retrieve-then-Predict

Oracle Retrieval

T5-based Retrievers●

Better retrieval model improves theentailment classification performance

Entailment Classifier is T5-3B
fine-tuned on ANLI + WiCE●

Provide gold supporting sentences●

The improvement suggests thatthe retrieval is important54