Enhancing Structured Evidence Extraction for Fact Verification
Authors: Zirui Wu
Nan Hu
Yansong Feng
Venue: EMNLP 2023 Main (2023)
Keywords: fact verification; evidence extraction

Abstract: Open-domain fact verification is the task of verifying claims in natural language texts against extracted evidence. FEVEROUS is a benchmark that requires extracting and integrating both unstructured and structured evidence to verify a given claim. Previous models suffer from low recall of structured evidence extraction, i.e., table extraction and cell selection. In this paper, we propose a simple but effective method to enhance the extraction of structured evidence by leveraging the row and column semantics of tables. Our method comprises two components: (i) a coarse-grained table extraction module that selects tables based on rows and columns relevant to the claim and (ii) a fine-grained cell selection graph that combines both formats of evidence and enables multi-hop and numerical reasoning. We evaluate our method on FEVEROUS and achieve an evidence recall of $60.01\%$ on the test set, which is $6.14\%$ higher than the previous state-of-the-art performance. Our results demonstrate that our method can extract tables and select cells effectively, and provide better evidence sets for verdict prediction. Our code is released at https://github.com/WilliamZR/see-st

Enhancing Structured Evidence Extraction for Fact Verification
Zirui Wu, Nan Hu, Yansong Feng"
Wangxuan Institute of Computer Technology, Peking University, China (ziruiwu, hunan, (engyansong))epee edu. cn
Abstract
1958
fooriell games,
against
Open-domain verification the task of ver- ifying claims in natural language texts against extracted evidence. FEVEROUS is a bench- mark that requires extracting and integrating both unstructured and structured evidence to verify a given claim. Previous models suffer from low recall of structured evidence extrac- tion, i.c., table extraction and cell selection. In this paper, we propose a simple but effective method to enhance the extraction of structured evidence by leveraging the row and column se- manties of tables. Our method comprises two components: (i) coarse-grained table extrac- tion module that selects tables based on rows and columns relevant the claim and (ii) fine-grained cell selection graph that combines both formats of evidence and enables multi- hop and numerical reasoning. We evaluate method on FEVEROUS and achieve an evi- dence recall 60.01% % on the test set, which is 6.14% higher than the previous state-of-the the- art performance. Our results demonstrate that our method can extract tables and select cells effectively, and provide better evidence sets for verdict prediction. Our code is released at ittps://gll thub. wwwoo///wllii LawZR/see-eeeeee
Label: SUPPORTS
Evidence
Structured: 1959 Eastem Michigan Hurons football team, Laabe
Data
Opponent
Bita
Result
Attandance
September
Youngetoww Stste
Ypusionn, MI
October?
Illolss
Ypsiland,
W 13-7
3,500
October 14
Northon Michigan
Marquetto,
L6-39
October
Central illno's
Ypsiland, MI
LB:21
5,000
October 24
Eastern lllnols
Creriesoon, I.
L6
October
Ypailans,
November 7
Northam liiriaa
DuKall, L0-34
No.9
Untructured: (1959 Eastem Michigan Hurons football team, sss__sseeeeeeee In the second game of the season, the Hurons defeateed Illinii State 13-7 then lost remaining
Figure Example of FEVEROUS with gold evidence.
The gold cell evidence highlighted in orange.
tables enable efficient look-up of individual val- ues, while the arrangement of rows and columns allows for the extraction of high-order semantics through various operations, such as comparisons, filtering, arithmetic calculations, and the identifi- cation of minimum and maximum values. The two-dimension structure and high-order semantics of tables make them more difficult extract com- pared with texts. Regarding the previous method extract structured evidence. Aly et al. (2021) treats tables as text sequences neglecting the table struc- ture itself and extracts them only based on lexical matching. This approach is also used in many fol- lowing works (Bouziane et al., 2021; Saeed et al., 2021; Hu et al., 2022, 2023). Consequently, these methods struggle extract tables with little word overlap with the claim. Alternatively, Bouziane et al. (2021) and Gi et (2021) convert each table cell into a text sequence and perform cell retrieval directly. Kotonya et al. (2021) construct a reason-
1 Introduction
Open-domain fact verification is the task of veri- fying a factual claim based on evidence extracted from a knowledge base(Guo et al., 2022; Hardalov et al., 2022). A complete evidence set provides necessary information for verification module to support refute the claim. In the real world, fact-checkers could use both unstructured and struc- tured data as evidence.
Tables are structured form of evidence that provides an organized representation of informa- tion that facilitates effortless comparison and cross- referencing. By presenting information in cells,
"Corresponding Author.

---

Retrieved Pages
Claim: Esalem Michigan foolbadl except against
Selection
Dootee
A/9
Date
Opp
3ile
Resut
Allenfence
Column Nodes
Sepsenter 10
Youggisan Siate
Ype 14
L3-21
4,127
Table
October
Inoto Stane
Ytokrkk
W1ST
2,000
Row Nedes
Orvieer
Notths Mich
Fide
Cukkie Hako
Choleoon
1palbrt
Novmmer?
hortwm linds
Delabl I.
LG-34
Noweroorr 14
Ro.9 Rosten Bince
Maorrb.
Le:2
Sentence Nodes
Sentennee Extraction
Evidence Graph
15-7 then nemsinnng
Figure 2:
Overview of proposed approach SEE-ST. It extracts structured evidence based retrieved pages and extracted sentences, SEE-ST includes two modules: coarse coarse-grained table extraction ($2.1) and fined- mhed-grained cell selection based on an evidence graph($2.2).
ing graph of sentences and cells to extract evidence. Yet these methods overlook the context within the table and fail capture high-order semantics from rows and columns.
tion from the Date and Opponent columns and the 3rd row. Overall rows and columns have compa- rable semantic granularity with the claim and are the most suitable fundamental unit for both table extraction and cell selection.
Figure presents an example from FEVER- OUS (Aly et al., 2021), a benchmark on open- domain fact verification using both sentences and table cells as evidence. The verification model is expected extract both unstructured and struc- tured evidence verify the claim. The structured table can be interpreted in several ways for the model: a complete table, a combination rows or columns, or a composition of isolated cells. The entire table encompasses substantial amount of information. During table extraction, the Site and Atlendance columns are irrelevant and only intro- duce confusion for both lexical and semantic meth- ods (Herzig et al., 2021). Cells typically contain short phrases and numbers, which are often insuffi- cient for conveying comprehensive information. In contrast, rows and columns possess a similar vol- ume of information to the claim, thereby exhibiting comparable semantic granularity. The extraction process should prioritize the information from the 3rd row and the Result column. During the cell selection step, the selection module must capture the high-order semantics in the Result column to verify the statement... ... lost all games except for one... Moreover, it needs to identify the cells October and Illinois State by integrating informa-
In this paper, we propose Structured Evidence Extraction with Row and Column Semanties of Tables (SEE-ST), novel approach to enhance both table extraction and cell selection modules for fact verification by leveraging the semantics of rows and columns. This method aligns the seman- tic granularity of structured evidence with claim and sentence evidence, enabling more precise ev- idence extraction. SEE-ST starts by extracting ta- bles from retrieved Wikipedia pages and focuses on identifying the most relevant rows and columns for the claim, thereby reducing confusion arising from irrelevant cells. Since a cell represents the intersection of a row and a column, its selection can be conducted by analyzing both dimensions. We integrate rows and columns within a graph neu- ral network to facilitate fine-grained cell selection while preserving high-order semantics. Addition- ally. incorporate extracted sentence evidence into the graph to merge information from evidence of both formats at a similar semantic granularity.
Our contributions can be summarized follows: (i) We propose to leverage the row and column semantics to enhance table extraction and cell se- lection. (ii) We design an evidence graph for cell

---

selection. It enables rows and columns to interact with sentences at comparable semantic granularity while still maintaining high-order semantics from table evidence. (iii) Our proposed method SEE-ST achieves state-of-the-ar performance in evidence extraction on the FEVEROUS dataset, with better evidence sets contributing the increased accuracy of the verification module.
Each row and column in the evidence table is rep- resented the average of all the token embeddings in the same row or column. Then we predict a prob- ability score for each row and column separately. The selective score for the table is computed as the multiplication of the maximum probability score for the rows and columns.
P(t) max (Prow) X max (Pea))
2 Our Model
The training set is constructed with all the tables that contain gold cell evidence as positive exam- ples. As for negative examples, we use top tables retrieved by DrQA (Chen et al., 2017) which do not contain cell evidence but have lexical overlap with the claim. The ratio of positive and negative exam- ples is I at every mini-batch in training. During training, the model is trained to select table rows and columns that contain gold cell evidence. The loss function is computed with a cross-entropy loss on each row and column.
FEVEROUS is a benchmark on open-domain fact verification (Aly et al., 2021). The task is to ver- ify claim based on Wikipedia. We follow the three-step pipeline in open-domain fact verification (Thorne et al., 2018; Aly et al., 2021). The three steps are document retrieval, evidence extraction, and verdict prediction. The first step retrieves rip relevant pages from the Wikipedia dump. Then the second step extracts sentence evidence S and cell evidence C' from the retrieved pages. At last, the model predicts the veracity label ý of the claim based on sentence and cell evidence.
= an Row + BeL Cal
SEE-ST enhances the extraction of structured evidence in the second step of the pipeline. It oper- ates on the same conceptual framework, utilizing row and column semantics to enhance evidence ex- traction (Figure 2). SEE-ST performs structured ev- idence extraction in two stages: (i) Coarse-gras grained table extraction that retrieves top n, tables for the second step. It scores relevance between rows or columns with the claim and ranks tables accord- ingly and (ii) Fine-grained cell selection that re- trieves cell evidence from complex tables. It con- structs a graph neural network of rows, columns, and sentences at comparable semantic granular- ity. It utilizes graph attention network to enable delicate information integration between different pieces of evidence. The cells selected as the intersection of rows and columns.
I, Row and I, C,Col are cross-entropy loss functions of rows and columns. These rows and columns that have gold cell evidence are labeled as select while rows/columns in the same table but do not contain cell evidence and rows/columns in negative examples are labeled nol select.
At inference, a selective score is computed for each table in retrieved pages. Top n, tables are se- lected for the following fine-grained cell selection.
2.2 Fine-grained Cell Selection
Following the idea of leveraging row and column semantics, we build an evidence graph E)) to select cell evidence. The first two types of nodes to -., UM and 11 UM WN encode ta- ble rows and columns. Shown as yellow nodes in Figure 2. M and N are the total number of table rows and columns. The third type of node is sentences VM-N, tK, shown as blue nodes in Figure 2. Overall, the evidence graph has K nodes of unstructured and structured evidence at a comparable semantic granularity.
At the verification step, we utilize DCUF (Hu et al., 2022), a method that converts evidence into dual-channel encodings to verify the claim.
2.1 Coarse-grained Table Extraction
In this section, we introduce our table extraction module. First, the claim and table pair are fed to TAPAS, pre-trained table model aware of table (tructu et al., 2020). Instead of using embedding of [CLS] token to predict the relevance between the claim and table. our module predicts the relevance based on row and column semantics.
Graph edges E are constructed between related evidence following three rules: (i) Cross-Format Related: i; and vj are different format evidence of sentence and table, and in the same Wikipedia page. (ii) Structure-Bautt
2 and by are nodes from the same table and (iii) Entity-Be -Bsee:: 2s and ug have

---

common entities and hyperlinks sentences on the same Wikipedia page.
Train
Dev
Test
Total
Supported Refuted
41,835
3,908
3.372
49,115
We feed each claim-sentence pair to a RoBERTa model and use the embedding for [CLS] token to initialize the sentence nodes in our graph. And nodes of table rows and columns are initialized using the same method as in $2.1.
27.215
3,481
2,973
33,669
NEI
2.241
501
1,500
4,242
Sentences
31,607
3,745
3.589
38,941
Cells
25,020
2.738
2.816
30,574
Sentences+Colls
20,865
2,468
2,062
25,395
Total
71,291
7,890
7,845
87,026
Then we apply layer of Graph Attention Net- work (GAT) (Veličković al., 2018) with resid- ual connection update representations of each node. dynamically allocates weights between different nodes evidence, allowing the graph to capture semantic connections between evidence.
Table I: Quantitative characteristies of FEVEROUS.
3.1 Evaluation
Evidence extraction crucial in fact verification. provides the verdict prediction module with the necessary evidence to verify the claim. FEVER- OUS uses precision, recall and FI score to measure the quality of retrieved evidence. E is the collec- tion of gold evidence sets and É is the retrieved evidence set. Recall of specific type of evidence is computed for instance
Finally, we use two-layer feed-forward net- work to predict the retrieved probability for each node separately. The selective score for cell evi- dence is the multiplication of the probability of the row node and column node that contains the cell. Top n. cells are extracted for verdict prediction. The loss function is computed as the weighted sum of the loss on nodes and cells:
LAUUE Recall E) IUE;
L == OCe: Co + Br. L. Roo th Ce
LS, [Col Row and L are cross-entropy loss computed on sentences, table columns, table rows, and cells. Og.. Be and "e are coefficients to ad- just weights between nodes and cells. Nodes of sentences in the gold evidence set are labeled as positive. Nodes of rows and columns are labeled positive if they contain the gold cell evidence.
El; is the number of i type evidence in set E. Evidence type i could be sentence, table or cell. A retrieved evidence set is considered complete iff at least gold evidence set is included. The recall of evidence set is defined for an instance follows:
Recall(E, E) = {{
EEEE:
3 Experiments
otherwise
We utilize FEVEROUS as the test bed for our ap- proach, as it is, to our knowledge, the only open- domain fact verification benchmark that combines both unstructured and structured evidence. FEVER- OUS aims to accomplish two objectives: first, to ex- tract sentence and table cell evidence from English Wikipedia, which contains over 95.6 million sen- tences and 11.8 million tables; and second, pre- dict the veracity label a given claim. The possi- ble veracity labels include SUPPORTS, REFUTES, and NOT ENOUGH INFO (NEI). Some claims can be verified using either sentences or tables alone, while approximately 59% of instances need struc- tured evidence for claim verification. 27% of the instances need combine evidence from unstruc- tured and structured evidence verify. The quan- titative characteristies of the benchmark are pre- sented Table I. Detailed experiment settings can be found in Appendix
The number of retrieved evidence is limited to 5 sentences and 25 cells maximum (Aly al., 2021). the following discussion, recall refers to evidence recall unless specified for an evidence type.
The evaluation for verdict prediction uses label accuracy(Acc.) and FEVEROUS score(F.S) its metrics. They are defined for instance
Acc. 0) = 1 y y=
otherwise
F.S = Acc. 1) x Recall
is the gold label and ğ is the predicted label. FEVEROUS score considers both the correct pre- diction of the veracity label and the completeness of the retrieved evidence set.

---

Models
Table
Sentence
Cell
Evidence
The accuracy on the test set is generally about 10% lower than the accuracy on dev set. The main reason is the unbalanced instances of NEI claims different splits. Our analysis of verdict prediction results shows that DCUF performs poorly in NEI instances. This explains the accuracy gap between the dev and test set.
Baseline
56
53
30
FaBULOUS
56.6
34.2
40.4
DCUF
75.59
62.54
58.41
43.22
UnifEE 75.59
75.36
67.44
55.08
Our Model
80.86
75.50
77.16
61.43
Table 2: Recall of different formats of evidence on the development set.
3.3 Ablation Study
Table Extraction We assess the effectiveness our table extraction model through the following ablation experiments: (i) w/o Column Semantics: We train a model that only considers the relevance between rows and the claim for table extraction, scoring tables based the criterion max (P (rom). (ii) w/o Row Semantics: Analogously, we score ta- bles using max cot). (iii) w/o Row and Column: We eliminate the design of table extraction and instead utilize the embedding of the [CLS] token in claim-table encodings for table extraction. The recall of the top 3 tables is presented in Table 4.
3.2 Main Results
Evidence Extraction Results Table2 shows the extraction results of our model the development set for different categories of evidence. All the pre- vious methods restrict their evidence set to at most 5 sentences, tables, and 25 cells. Official base- line (Aly et al., 2021) and FaBULOUS (Bouziane et al., 2021) use weaker document retrieval mod- ule which results in error propagation and low evi- dence recall. Our model uses the same document retrieval module as DCUF (Hu et al., 2022) and UnifEE (Hu et al., 2023), which obtains a recall of 85.20% for Wikipedia pages. Therefore recall of each format of evidence can be used to com- pare the performance of evidence extraction di- rectly. SEE-ST leverages the row and column se- mantics of tables and increases the recall of tables by 5.27%. SEE-ST also increases the recall of cells by 11.65% and 9.27% compared with DCUF and UnifEE. This suggests the two stages of SEE-ST, table extraction and cell selection can both improve the performance of structured evidence extraction. Overall Results Table 3 shows overall perfor- mance on the dev and test set based on our idence extraction results. We achieve 60.01% % evidence recall on the blind test set, which is 6.14% higher than the previous state-of-the-art UnifEE (Hu et al., 2023). Following the verifica- tion method proposed in Hu et al. (2022), we obtain accuracy 74.68% 16% and FEVEROUS score 49. 49.73% dev/test set. This demonstrates enhancement structured evidence extraction provides verification with more accurate evidence so that could make the right predictions.
In the setups without column or row semantics, the recall of the top tables decreases by 1.03% and 1.76%, respectively. In the w/o Row and Column setting, table recall drops significantly by 17.75%, resulting in a recall of 63.11%, which is even lower than the recall achieved by DrQA (Table 2). These results suggest that our table extraction module in SEE-ST, which combines semantics from both rows and columns, effectively captures the coarse- grained relevance between claims and tables.
Cell Selection To evaluate the effectiveness of our evidence graph for cell selection, we design ab- lation experiments with evidence graph con- structed with the top tables. The experimental settings are as follows: (i) w/o Structured Edges; We introduce additional edges between rows and columns from the same Wikipedia pages but not the same table. (ii) w/o Edge Pattern: We directly connect every node in the evidence graph to each other. (iii w/o Row Semanties: Following Hu et al. (2023), build evidence graph of sentences and columns, and separately, a graph of table cells, using columns as intermediates to select cells. (iv) w/o Evidence Graph: We follow the table extrac- tion pipeline with an added term for the cell loss function, selecting each cell based solely on the context within the same table.
Since SEE-ST obtains the maximum number of evidence in extraction, also experiment UnifEE without its threshold selection mechanism compare its performance with SEE-ST more fairly. SEE-ST still improves the evidence recall by a large margin. Our choice discarding threshold selection is further analyzed in Appendix B.
The results of the ablation experiments are pre- sented in Table 5. When increasing the number of tables in the evidence graph from to 5, cell and

---

Dev
Test
Models
F.S
Acc.
E-P
E-R
E-F1
F.S
Acc.
E-P
E-R
E-F1
Official Baseline
19
53
12
30
17
17.73
48.48
10:17
28.78
15.03
EURECOM
19
53
12
29
17
20.01
47.79
13.73
33.73
19.52
Z team
22.51
49.01
7.76
42.64
13.12
CARE
26
63
7
37
12
23
53
7
37
11
NCU
29
60
10
42
17
25.14
52.29
9.91
39.07
15.81
Papelo
28
66
25.92
57.57
7.16
34.60
11.87
FaBULOUS
30
65
43
14
27.01
56.07
7.73
42.58
13.08
DCUF
35.77
72.91
15.06
43.22
22.34
33.97
63.21
14.79
44.10
22.15
UnifEE
44,86
73.67
19.04
55.08
28.30
41.50
65.04
18.35
53.87
27.37
UnitEF"
46.13
73.14
12.48
56.22
20.42
Our Model
49.73
74.68
10.60
61.43
18.07
44.75
65.16
9.81
60.01
16.89
Table 3: Model performance the development set and test set. F.S is FEVEROUS score and Acc. is the accuracy of veracity labels. E-R, E-P and E-FI are recall, precision and Fl computed based on the evidence set. All metrics are averaged on all instances. UnitEE" uses maximum number of evidence as the constraint for the evidence set.
Models
@ecall@3
and columns, and another with cells. Cell recall and evidence recall decline by 1.04% and 0.92% in this setting, indicating that SEE-ST's utilization of row and column representations captures cell information more precisely. Moreover, compared to the two graphs of sentences and cells, SEE-ST is more computationally efficient with only one evidence graph. Finally, the w/o Evidence Graph setting loses all connections between sentences and tables, as well shared entity information among evidence. Each node is predicted based on its own information, resulting in significant drop in cell and evidence recall. This highlights the importance of the design of evidence graph that enables inter- actions between texts and tables.
SEF-ST
80.86
w/o Column Semanties
79. .38
w/n Row Semantics
79.10
w/o Row and Column
63.11
Table 4: Ablation study of coarse coarse-graines grained table extrac- tion. Recall@3 is the recall of top extracted tables.
Recall
Cell
Evidence
Top5 Tables
SEE-ST
77.16
61.43
Top3 Tables
SEE-ST
74.20
60.16
w/o Structured Edges
72.30
58.61
w/o Edge Pattern
72.22
58.56
Analysis
w/o Row Semantics
73.16
59.24
w/o Evidence Graph
68.37
56.08
4.1
Error Analysis
Here we perform a detailed error analysis for evi- dence extraction.
Table 5: Ablation study of fine-gras ine-grained cell selection.
Error Source Analysis The error source for ev- idence sets is defined the format of evidence that is not extracted completely in the three-step pipeline ($2). We use Page denote that document retrieval fails to retrieve all the pages containing evidence. For those instances that have a complete document set, the error source is defined according to the format of evidence fail to extract: Unstruc- tured (sentences), Structured (tables or cells) and Both. The instances with complete evidence set are denoted Complete. About 1% of instances have other evidence types such lists and table captions. These instances are excluded during error analysis. The impact of fignorance of other evidence types discussed in the Limitations section.
evidence recall rise by 2.96% and 1.38%, respec- tively, achieving recall rate of 77.16% for cell evidence and 61.43% for evidence recall. In the w/o Structured Evidence setting, rows and columns from different tables are connected they are on the same Wikipedia page, resulting in a decrease of 1.98% in cell recall and 1 1.53% in evidence re- call. This demonstrates that the Structured Edges designed in $2.2 are beneficial for selecting cells as intersections of and columns. Connecting all cells in the graph further compromises the perfor- mance of structured evidence. the w/o Row Se-
mantics setting, we build two evidence graphs fol- lowing Hu et al. (2023): one containing sentences

---

UnifEE SEEST
50-
Multi-hop
6
Numerical reasoning
Entity Disambiquation
toms claim
Combining Table and Texts
Propartion
Proportion
30
10-
10
Page
Unstrictured
Structured
Bott
Complete
Page
Unstrctured
Both
Figure 3: Error analysis of extracted evidence for the dev Left: Overall error source analysis. Right: Error source proportions of claims with different reasoning challenges.
The result of the error source analysis is shown Figure 3. The document retriever fails to retrieve complete documents for 15.8% instances in the dev set. Our proposed method decreases the proportion of cases missing structured evidence from 14.28% to 9.34%. It shows the effectiveness of SEE-ST in structured evidence extraction.
also improves the evidence extraction performance of numerical reasoning. The rate of Structured er- rors is 34.82% for UnifEE (Hu et al., 2023). SEE- ST increases the recall rate by 8.25% and decreases the rate of Structured errors by 7.35%. Yet nu- merical reasoning remains the main challenge in structured evidence extraction.
Analysis based on challenge types fact-checker system would face a specific reasoning challenge when extracting evidence and verifying claims for FEVEROUS. The challenges include multi-hop rea- soning, entity disambiguation, search term not in claim, combining tables and texts and numerical reasoning. The challenges that do not belong to those five types categorized other.
Case Study
In this section, we demonstrate two cases for ev- idence extraction Figure 4. For the claim on 1991 Waterford City Council election, our table ex- traction module successfully extracts the table of election statistics from the Wikipedia dump. The main challenge for this case is numerical reason- ing. Verifying ... six parties all of which have at least two seats requires high-order semantics in Seats column. Such semantics maintained in coarse gaariee table extraction and fine-grained cell selection. SEE-ST extracts cells in Party and Seats columns evidence. It shows the numerical reasoning ability of SEE-ST.
We further evaluate the performance of SEE-ST in each category to demonstrate its ability to re- trieve evidence for claims with different reasoning challenges. SEE-ST achieves great extraction per- formance over claims that need to combine tables and texts with a recall rate of 69.27% showing that the unified graph neural network effectively in- tegrate information from both formats of evidence.
As for the evidence extraction for the claim of The Castle of Iron, demonstrates the ability of SEE-ST to perform multi-hop reasoning. The claim states that was published by a large-scale pub- lishing company in New York, and refuted by sentence evidence Gnome Press was American small-press publishing company... However, the name of the publisher is not mentioned in the claim. It is the structured evidence that links the claim with the sentence evidence. The design of our evi- dence graph could enable interaction between sen- tence and table cells to achieve multi multi-hop reasoning over different formats of evidence.
The recall rates of claims that need multi-hop rea- soning, entity disambiguation or search terms are lower than 50%. But the analysis in Figure 3 shows that the main error source of incomplete evidence extraction is document retrieval. It fails to retrieve complete document sets for each challenge type rate of 38.6%, 28.4%
listed above with and 54.6% while only about 7% instances in these challenges lose complete evidence sets because of unsuccessful structured evidence extraction. This demonstrates the ability of our model to retrieve evidence more challenging scenarios. Our model

---

Claim: 1981 Waterford City Council election was planed to take piace on 27 June 1991 with six partie all of which have at least two seats.
Claim: The Castle of iron ie fanisey novel written by Sprague Camp and Fletccher Pratt and was published by large-scale publishing company in New York.
Label: SUPPORTS
Challenge: Numberical Reasoning
Label: REFUTES
Challenge: Multihop Reasoning
Extracted Evidence
Structured:
Extracted Evidence
Party
Structured:
Fiarna Fai
3
-2
3,166
19.01%
Author
Samaue de Camp
Publisher
Gnama Press
Fletcher
Fire Gae!
2
-2
2,200
13.54%
Cover artist
Harnes Bok
Publication date
1941, 1950
Labour
3
41
3,415
20.6%
Gountry
United Starees
Media type
Print
Workens's
3
3,868
20.18%
Language
English
Pages
224
Parly
Progreesciv
2
1,66
10.01%
Series
Haroic Shea.
Preceded by
The Incomplete Erchartar
Democrale
Indepandent
2
2,300
14.2%
Genre
Fantaie
Followed by
Wall Serpents
Totals
18
16,645
100%
Untructured:
Untructured:
Gnome Press was an American small-press publishing company primarily known for publishing many solence fiction classics.
An election to Waterford City Council took place on 27 June 1991 that Irish local elactions. Waterford City Cauncil Comhairte Cathrach Phort Lairge) was the authority responsible for local governments the waterford.
fantasy novella by American Sprague de Camp and Fietcher Pratt, and of the novel into which it was later expanded by
Demonstrations extracted evidence set for claims in FEVEROUS. Gold evidence highlighted orange. The underlines indicate the cells selected by SEE-ST and table headers marked with bold texts.
Figure 4:
6 Related Work
tations of row and column to rank tables but over- looks the context of other cells. TaBERT (Yin et al., 2020) utilizes content-snapshov mechanism to only encode the relevant rows thus may lose the high-order semantics of columns.
Open-domain fact verification extracts evidence from knowledge base (KB) verify the claim. In the real world, unstructured and struc- tured evidence can be used to verify the claim. FEVER (Thorne et al., 2018) and HoVer (Jiang et al., 2020) focus on extracting unstructured ev- idence i.e. sentences from the Wikipedia dump. And the research of structured evidence mainly fo- cuses on tables. TabFact (Chen et al., 2019) and InfoTabs (Gupta et al., 2020) verify claims with tables and infoboxes from Wikipedia. SEM-TAB- FACT (Wang al., 2021) and SciTab (Lu al., 2023) are constructed on claims and tables from scientific papers. Yet these datasets all provide the table evidence for verification as in closed- domain setting. To our knowledge, FEVEROUS is the only open-domain fact verification dataset that considers table evidence (Aly al., 2021). Therefore, work uses FEVEROUS testbed for table extraction and cell selection.
Gi et al. (2021) converts each cell into text sequence to select cell evidence, bypassing the ble extraction step. Acharya (2021) selects cells through dependency parsing. Jindal et (2021) combines single-cell NLI task with cell-wise rel- evance to extract evidence at cell-level semantic gramularity. These methods are performed at the se- mantic granularity of cells. Approaches in question answering have proposed to select cells the in- tersection of rows and columns (Glass et al., 2021; Pan al., 2021). All these above methods do not consider the interaction between evidence.
The idea evidence graph has been explored previous works. Kotonya et al. (2021) linearizes cells into sentences to construct fully connected evidence graph with texts. UnifEE (Hu et al., 2023) proposes to use column nodes as intermedi- ates between sentence-level and cell-level evidence graphs. Our method more computationally effi- cient with only one evidence graph. Our design en- ables rows and columns interact with sentences at comparable semantic granularity while still main- taining high-order semantics from tables.
As for table extraction, the lexical matching ap- proach is widely used in fact verification (Aly et al., 2021: Hu et al., 2022, 2023). Chen et al. (2020) encodes tables with BERT and concatenate em- bedding with structured features to rank tables. TAPAS enhances table encoding with additional table-aware positional embeddings (Herzig et al., 2021). (Pan et al., 2021) also leverages represen-

---

7 Conclusions
2021), pages 1271-1275, Online, Association for Computational Linguistics.
This paper presents SEE-ST, a novel method for enhancing the extraction of structured evidence. By leveraging both row and column semanties, SEE- ST can operate with a comparable evidence granu- larity, enabling both coarse-grained table extraction and fine-grained cell selection. Experimental re- sults demonstrate the effectiveness of SEE-ST, as it significantly improves the recall of structured evidence compared to existing methods in fact ver- ification. Moreover, SEE-ST outperforms sfate-of- the-art methods in evidence recall and FEVEROUS score on FEVEROUS benchmark. These findings underline the impact of SEE-ST in enhancing struc- tured evidence extraction.
Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull,
James Thome, Andreas Vlachos, Christos Christodoolopoulos, Oana Cocarascu, and Arpit Mittal. 2021. The fact extraction and VERification over unstructured and structured information (FEVEROUS) shared task. In Proceedings of the Fourth Workshop on Fact Extraction and VERifica- tion (FEVER), pages 1-13, Dominican Republic. Association for Computational Linguisties.
Mostafa Bouziane, Hugo Perrin, Amine Sadeq, Thanh Nguyen, Aurélien Cluzeau, and Julien Mardas. 2021. Fabulous: Fact-checking based on understanding of language unstructured and structured informa- tion. In Proceedings of the Fourth Workshop on Fact Extraction and VERification (FEVER), pages 31-39.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia answer open- domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1870-1877
Limitations
Our work explores the extraction of structured ev- idence in enhancing table extraction and cell se- lection. In the real world, fact checkers also use other structured evidence as knowledge graphs and databases to verify the claim. FactKG is a fact veri- fication dataset on knowledge graphs (Kim et al.. 2023). The verification model is required to re- trieve subgraphs as evidence for verdict prediction. SEE-ST can not be directly applied to this task. We will try to build a unified evidence extraction mod- ule that could retrieve both tables and subgraphs for fact verification in our future work.
Wenhu Chen, Hongmin Wang. Jianshu Chen, Yunkai Zhang, Hong Wang. Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. Tabfact: large- scale dataset for table-based fact verification. arXiv preprint arXiv: 1909.02064
Zhiyu Chen, Mohamed Trabelsi, Jeff Heflin, Yinan Xu, and Brian Davison. 2020. Table search using a deep contextualized language model. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 189-598. ACM.
Our work only enhances the extraction of struc- tured evidence, while the whole evidence extraction process still suffers from eтror propagation from document retrieval. The error propagation in the three-step pipeline makes it extremely difficult for the model to retrieve complete evidence sets and predict the right verdict label. A new pipeline for open-domain fact verification is needed to solve this issuc.
In-Zu Gi. Ting- Yu Fang, and Richard Tzong- Han Tsai, 2021. Verdict inference with claim and retrieved elements using roberta. Fourth Workshop on Fact Extraction and VERifica- tion (FEVER), pages 60 65.
Proceedings of the
Michael Glass, Mustafa Canim, Alfio Gliozzo, Sa- neem Chemmengath, Vishwajeet Kumar, Rishav Chakravarti, Avi Sil, Feifei Pan, Samarth Bharadwaj. and Nicolas Rodolfo Fauceglia. 2021. Capturing row and column semantics transformer based question answering over tables. In Annual Conference of the North American Chapter of the Association for Com- putational Linguistics.
Acknowledgements
This work is supported by NSFC (62161160339) We would like to thank the anonymous reviewers for their helpful comments and suggestions. For any correspondence, please contact Yansong Feng.
Zhijiang Guo, Michael Schlichtkrull, and Andreas Vla- chos. 2022. A survey on automated fact-checking. Transactions of the Association for Computational Linguistics, 10:178-206
References
Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. INFOTABS: Inference on tables as semi-structured data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2309-2324, Online. Association for Computational Linguistics.
Kaushik Acharya. 2021. KaushikAAhaan at SemFival- 2021 task 9: Candidate generation for fact verifica- tion over tables. In Proceedings of the 15th Interna- tional Workshop on Semantic Evaluation (SemEval-

---

Momehil Hardalov, Arnv Arora, Preslav Nakov, and Isabelle Augenstein. 2022. A survey on stance de- tection for mis-and disinformation identification. In Findings of the Association for Computational Lin- guistics: NAACL 2022, pages 1259-1277.
Neema Kotonya, Thomas Spooner, Daniele Magazzeni. and Francesca Toni. 2021. Graph reasoning with context-aware linearization for interpretable fact ex- traction and verification. In Proceedings of the Fourth Workshop on Fact Extraction and VERifica- tion (FEVER), pages 21-30.
Jonathan Herzig, Thomas Müller, Syrine Krichene, and Julian Eisenschlos. 2021. Open domain question answering over tables via dense retrieval. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 112-519, Online. Association for Computational Lin- guistics.
Xinyuan Lu, Liangming Pan, Qian Liu, Preslav Nakov, and Min-Yen Kan. 2023. Scitab: A challenging benchmark for compositional reasoning and claim verification on scientific tables. arXiv preprint 23iv:2305.13186.
Feifei Pan. Mustafa Canim, Michael Glass, Alfio Gliozzo, and Peter Fox. 2021. Cltr: An end-to-end, transformer- bassed system for cell-level table retrieval and table question answering. In Proceedings of the 59th Annual Meeting of the Association for Compu- rational Linguistics and the 11th International Joint Conference Natural Language Processing: System Demonstrations, pages 202-209.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Mueller. Francesco Piccinno, and Julian Eisensch- los. 2020. Tapas: Weakly supervised table parsing via pre-training, In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 4333..
Mohammed Saeed, Giulio Alfarano, Khai Nguyen, Due Pham, Raphael Troncy, and Paolo Papotti. 2021. Neural re-rankers for evidence retrieval in the FEVEROUS task. Proceedings of the Fourth Work- shop on Fact Extraction and VERification (FEVER). pages 108-112, Dominican Republic. Association for Computational Linguistics.
Nan Hu, Zirui Wu, Yuxuan Lai, Xiao Liu, and Yansong Feng. 2022. Dual-channel evidence fusion for fact verification over texts and tables. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, pages 5232-5242
James Thorne, Andreas Vlachos, Christos Christoddoulppoulss, and Arpit Mittal. 2018. FEVER: large-scale dataset for fact extraction and VERification. In Proceedings of the 2018
Nan Hu, Zirui Wu, Yuxuan Lai, Chen Zhang, and Yan- song Feng. 2023. Unifee: Unified evidence extrac- tion for fact verification. In Proceedings of the 17th Conference of the European Chapter of the Asso- ciation for Computational Linguistics, pages 1142- 1152.
Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguisties.
Yichen Jiang. Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. 2020. Horer: A dataset for many-hop fact extraction and claim verification. In Findings of the Conference on Empirical Methods Natural Language Processing (EMNLP).
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations.
Aditya Jindal, Ankur Gupta, Jaya Srivastava, Preeti Menghwani, Vijit Malik, Vishesh Kaushik, and Ashutosh Modi. 2021. BreakingBERT @ IITK at SemEval-2021 task 9: Statement verification and evidence finding with tables. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemFval-20011) pages 327-337, Online. Association for Computational Linguistics.
Nancy X. R. Wang. Diwakar Mahajan, Marina Danilevsky, and Sara Rosenthal. 2021. SemEval- 2021 task 9: Fact verification and evidence finding for tabular data in scientific documents (SEM-TAB- FACTS), In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-20211 pages 317-326, Online. Association for Computa- tional Linguistics.
Jiho Kim, Sungjin Park, Yeonsu K Kwon, Yohan Jo. James Thoree, and Edward Choi. 2023. FactKG: Fact veri- fication via reasoning on knowledge graphs. In Pro- ceedings of the 61st Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 161 1690066, Toronto, Canada. As- sociation for Computational Linguistics.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Se- bastian Riedel. 2020. Tabert: Pretraining for joint understanding of textual and tabular data. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413-8426.
A Implementation Details
In our approach, we retrieve np 5 pages from Wikipedia for each claim following Hu et al. (2022). And we extract n. = sentences and n, = 5 tables
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint TrXiv:1412.6980

---

from the pages construct our evidence graph. As for the cell selection, limit the selected cells to maximum of ne 25 cells. For fair com- parison with baselines, the source tables of cells are constrained to the top 3 tables. The tables are re-ranked by the highest cell selective score at cell selection.
verification model used in this work is robust threshold selection within the same extracted evi- dence set, and verdict prediction achieves the high- est FEVEROUS score without threshold selection. The annotations the gold evidence set may not be exhaustive of all possible evidence, which could result in underestimation of precision due to the presence of correct evidence that is not annotated as gold evidence. In summary, SEE-ST employs the maximum number of sentences and cells as constrain for evidence sets for optimal verdict pre- diction performance.
For table extraction, we utilize TAPAS-base to encode the claim-table pair. Our model is trained with Adam optimizer (Kingma and Ba, 2014) with a batch size of 8. The learning rate is 10-7 for TAPAS and 10-5 for the classifier. at and 3r are assigned a value of 1.
Regarding evidence graph, employ Spacy² to extract common entities from evidence connect relevant nodes. RoBERTa-base and TAPAS-base are chosen as the encoders for node initialization. The model is optimized using the Adam optimizer (Kingma and Ba, 2014) with a batch size of and a peak learning rate of 10-6, We set the weights for different coefficients of loss functions as follows: de Be 2 and "Ye 1. The model takes 14 hours to train for 3 epochs and 34 minutes select cells in the dev on a single NVIDIA A100 GPU.
Threshold
E-P
E-R
F-FI
FS
Acc.
0
10.60
61.43
18.07
49.73
74.68
0.001
12.99
60.17
21.36
48.38
74.25
0.005
15.85
59.60
25.04
48.04
74.03
0.01
17.88
58.54
27.49
47.28
74.22
0.05
23.39
50.78
32.04
42.56
73.99
Table 6: Effects of evidence extraction verdict pre-
diction.
Verdict
Recall
NEI
41.72
SUPPORTS
61.38
B Evidence Precision Analysis
REFUTES 61.84
With the constraint of maximum number of sen- tences and cells in the evidence set, there are two ways to construct evidence sets. The previous SOTA method UnifEE (Hu et al., 2023) uses a threshold to construct an evidence set with high precision at the expense of a slight decrease in evi- dence recall. Other baselines use evidence as much fit the constraints to construct the evidence sets resulting in higher evidence recall. In this section, we aim to analyze which strategy achieves higher accuracy and FEVEROUS score at the verdict pre- diction step. We filter out unnecessary evidence with different thresholds and verify claims based on the filtered evidence sets. The results are presented in Table 6.
Table 7: The performance of evidence extraction for claims with different veracity labels.
C Performance on Different Verdict
We further analyze the performance of evidence extraction and verdict prediction for claims with different veracity labels, as shown in Table Gen- crally, extracting evidence for REFUTES claims is more challenging, as the information contained in the evidence is not always explicitly included in the claim, making it difficult to retrieve based on relevance alone. However, our model achieves comparable extraction performance for both SUP- PORTS and REFUTES types, with recall rates of 61.38% and 61. 184%, respectively. The recall for NEI instances is approximately 20% lower than for the other types. NEI claims cannot be verified based on evidence from Wikipedia alone; only the most relevant information is listed in the gold evi- dence set, making it difficult to extract a complete evidence set for NEI claims.
The results indicate that as the evidence thresh- old increases, evidence precision improves at the cost of reduced evidence recall. However, label accuracy remains largely unaffected by changes the evidence set. Threshold selection leads to a decrease in the FEVEROUS score. Overall, the
"nttps://huggingfface http:/oooooooooooooeeeeeeeeee/ee btppe//eeeeeeeeooooooo/////aeaae "https://spacy.ut io/ wtt.:ooorrrrrbbaeeeee.eeeee


Reviews:
[{"review_id": "ugskjcWOnC", "reviewer": "Reviewer_CvBB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your feedback on our paper. We appreciate your time and effort in reviewing our work. We have read all the comments and suggestions. Below, we respond to the raised.\n\nQ1: The proposed method is very specialized...\n\nA: Our work focuses on a challenging task that extracts evidence by combining texts and tables. This is a more realistic setting compared with other fact verification tasks focus on only one format of evidence. Our proposed method leverages row and column semantics to allow structured evidence to interact with unstructured evidence at a comparable semantic granularity. The same idea could be applied to open-domain question answering or information retrieval over other types of structured evidence.\n\nQ2:The performance success of the proposed method hasn't been fully explained...\n\nA: Previous works retrieve texts and tables separately without considering potential connection between evidence. The evidence graph allows interaction between texts and tables. *w/o evidence graph setting* in Table 5 indicates that the evidence graph is the main reason our method outperforms other baselines, especially for challenging claims that need multi-hop reasoning and combining tables and texts (shown in Figure 3).\nThe proposed method also benefits a lot from leveraging row and column semantics. Rows and columns interact with sentences at comparable semantic granularity while still maintaining high-order semantics from tables. As shown in Figure 1, the *Result* column suggests *...lost all its games except for the one...* and redundant sentence and row evidence about *October 7 Match* corroborates each other.\n\nI hope this address your concerns. Thank you again for your time and consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to follow.\n2. The proposed method is simple yet effective.\n3. The proposed method is well-ablated and analyzed.", "reasons_to_reject": "1. The proposed method is very specialized, and it is unclear how it can be applied to tasks other than evidence extraction.\n2. The performance success of the proposed method hasn't been fully explained in the context of the previous works, e.g., is the input semantic granularity difference the main reason that the proposed method outperforms baselines?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a performant model for evidence extraction, the middle step of the three-step pipeline of open-domain fact verification. Unlike the previous evidence extraction methods, the proposed method first retrieves top tables based on the relevance score between the claim and the table rows or columns, then selects the relevant cells via a carefully designed graph neural network. The empirical results on the FEVEROUS benchmark demonstrate the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "abwk0xbdSw", "reviewer": "Reviewer_W4G6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank for your feedback on our paper. We appreciate your time and effort in reviewing our work. We have read all the comments and suggestions. Below, we response to the concerns raised in the review.\n\nQ1: ...Motivation behind the proposed method...\n\nA: Please refer to Answer for Q2 for Review#1 CvBB. It explains the motivation and the advantages of our proposed method.\n\nQ2: ...the proposed method underperforms the baseline methods on E-F1...\n\nA: The idea is that for fact verification, the ultimate goal is the complete evidence set and accurate verdict prediction. Previous SOTA UnifEE uses a threshold to select evidence from top 5 sentences and top 25 cells which results in high F1. Such a method decreases recall of the complete evidence set. As shown in Figure 7, label accuracy remains unaffected by the change of evidence threshold (which affects precision, recall and F1). Therefore we choose to employ a maximum number of evidence as a constraint instead of the threshold to achieve optimal label accuracy and Feverous score at the final step.\n\nQ3: Code sharing\n\nA:  Sure, we will release our code along with the improved version of our paper soon.\n\nI hope this address your concerns. Thank you again for your time and consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The method proposed in this paper is simple but effective. The ablation study and error analsysis help to better understand the advantages of the proposed method.", "reasons_to_reject": "1. I may not fully understand the motivation behind the proposed method. What's the limitation of the existing method and how you address the issues? What's the advantages of your method and why? The paper can be improved if authors provide more insights in the introduction part.\n\n2. From Table3, we can see that the proposed method underperforms the baseline methods on E-F1. Some explanations should be provided. \n\n3. No code was provided. No code sharing promised.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on open-domain Fact Verification and proposed a framework that leverages row and column semantics of tables. The proposed method gains good results on benchmark dataset.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KicQwaPEZC", "reviewer": "Reviewer_cEAj", "summary": "", "questions": "- Wikipedia tables commonly also contain a table caption. Has the table caption been used? Was it useful at all to consider the caption for additional context?\n- Whether a table cell is a header of the table or not is only considered for the TAPAS representation or also elsewhere (or maybe even nowhere)?\n- Table 4 shows that removing either column or row semantics decreases performance only marginally. This is quite surprising to me, as I would expect that both semantics would be more complementary: i.e. for some tables, the semantics of the row is more important, for other tables the column (with a tendency towards the column being more important). Why is this not the case?\n- I would have liked to see the standard error in the results to confirm the robustness of the approach. Can the experiments be rerun and mean + standard error be reported?\n- Are the authors releasing their source code for reproducibility?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank for your feedback on our paper. We appreciate your time and effort in reviewing our work. We have read all the questions for experiments and suggestions for writing. Below, we response to the concerns raised in the review.\n\nQ1: ...Has the table caption been used? ...\n\nA: As in the case we show in our paper, evidence extraction for tables is usually focused on specific cells which are only a small proportion of the table. And the table caption is more of a general of description of the table content. We haven't considered table captions yet. However, the use of table captions could be useful.\n\nQ2: ...table header only considered for TAPA representation?...\n\nA: Yes, the annotation of table headers is only used for TAPAS representation. A special token [H] is added in the cell to indicate it is a header. As for the evidence graph construction and cell selection, no specific manipulation for headers.\n\nQ3: ... Table 4 ...removing either column or row semantics decreases performance only marginally...\n\nA: I have to admit, this is surprising to me as well. I do not have solid explanations for this. I think the use of the whole table as input makes row or column representation still have context from other rows or columns. So the decrease is not as large as we expect.\n\nQ4: ...mean + standard error be reported?\n\nA: We have checked our logs and the mean value and standard error for evidence recall on dev set is 61.35 and 0.39(computed over 5 experiments). This describes the robustness of the crucial metrics. We would provide detailed standard errors for other metrics and experiments if the paper gets accepted\n\nQ5: code release:\n\nA: Sure, we will release our code along with the improved version of our paper soon.\n\nQ for Error Analysis in Reasons to Reject:\n\nA: We appreciate you pointing out the gap between the analysis of the verifier and our story. The verdict classifier sensitivity aims to explain why use the maximum number as the constraint for our evidence set (Please refer to Q2 for Reviewer #2 W4G6 for details). And we will provide more quantitative examples for error analysis of evidence extraction.\n\nI hope these answers address your concerns. We thank you again for your time and effort in reviewing our work. The suggestions for typos and presentation style are really helpful. We would definitely give the paper a proper pass for improvements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors propose a simple and effective method for semi-structured evidence retrieval for fact-verification which improves over previous results substantially. While previous efforts have deployed both tabular and graph models for retrieval, the authors demonstrate that incorporating context beyond individual cells, but of entire rows and columns, is of essence. Tabular/Cell evidence retrieval is one of the key challenges of the FEVEROUS dataset and the authors provide a strong solution for this. \n- Their approach is well motivated through selected examples. Figure 1 is well-chosen and demonstrates intuitively why a cell/row level extraction is beneficial.\n-  The error analysis, broken down into challenging claims of different classes (e.g. Multi-hop Reasoning, Numerical Reasoning, etc.), highlights current gaps in evidence retrieval and motivates potential future directions.", "reasons_to_reject": "- While the paper is clear on a high level, it is in part difficult to read and verbosely written. For instance, as mentioned above, Figure 1 showcases the advantages of a row/column-focused retrieval system due to the entire \"Result\" column being of relevance and the second-row providing additional context when considered in its entirety, yet the text does not concisely describe this.\n- While the error analysis is interesting, parts of the analysis section are less of interest to me and appear less relevant to the story of the paper. The verdict classifier sensitivity regarding the evidence threshold seems a bit out of place since the paper does not propose the verdict prediction module itself. Similarly, the prediction performance for different veracity classes does not fit very well. The observations simply confirm previous work and the authors do not attempt to explain them (e.g. very low performance for NEI). Instead, I would wish the authors to focus more on the error analysis and provide qualitative examples of where their system fails.", "questions_to_authors": "- Wikipedia tables commonly also contain a table caption. Has the table caption been used? Was it useful at all to consider the caption for additional context?\n- Whether a table cell is a header of the table or not is only considered for the TAPAS representation or also elsewhere (or maybe even nowhere)?\n- Table 4 shows that removing either column or row semantics decreases performance only marginally. This is quite surprising to me, as I would expect that both semantics would be more complementary: i.e. for some tables, the semantics of the row is more important, for other tables the column (with a tendency towards the column being more important). Why is this not the case?\n- I would have liked to see the standard error in the results to confirm the robustness of the approach. Can the experiments be rerun and mean + standard error be reported?\n- Are the authors releasing their source code for reproducibility?", "paper_topic_and_main_contributions": "The authors are concerned with open-domain fact verification, specifically with the scenario where evidence to verify a claim has to be retrieved from a knowledge base consisting of both text and tables. The paper presents a method to extract of tables and associated cells by leveraging the row and column semantics of tables through a two-step pipeline: (i) the coarse grain extraction of tables from selected documents by computing a score over encodings of a table's rows and columns using a pre-trained tabular model, (ii) the fine-grained cell selection by building an evidence graph from the selected tables and separately extracted evidence sentences. The authors evaluate their system on FEVEROUS The authors show that their extraction module outperforms previous systems and when integrated into a full verification pipeline, they achieve new state-of-the-art performance on the FEVEROUS leaderboard.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- I strongly recommend the authors give the paper a proper pass for improvements in clarity and conciseness:\n- Figure 3 is difficult to read. It is confusing that \"Complete\" is not an error class, but the other four ones are. This needs to be clarified in the figure and in the caption. Moreover, the numbers in 4.1 do not all match up with the Figure. l.448 mentions that the error rate for numerical reasoning is decreased to 10.3%, where is that in the figure? It shows 29.0% to me.\n- l. 532 to l.534. I do not understand the connection of the sentence \"The lexical matching approach [...] widely used in fact verification\" to the previous one about question answering.\n- l.544 and l.557 regarding Kotonya et al. (2021) are almost identical in content.\n- Typos: l.56 construct, l.099 \"most relevant rows and columns [for] the claim\"\n- Several sentences where necessary articles are omitted, resulting in ungrammatical sentences. I recommend passing through the paper with spell-checking software.\n- Several lines with only a single word (e.g. l.110), should be avoided. Through slight rewording, the authors can be much more space efficient. "}}]

Author rebuttal:

Meta-review: Summary: The authors of the paper focus on verifying factual claims in open domains, specifically when evidence needs to be retrieved from a knowledge base consisting of text and tables. Their proposed approach involves extracting tables and the associated cells using a two-step pipeline. Firstly, they use a pre-trained tabular model to compute a score over encodings of a table's rows and columns to extract tables from selected documents. Secondly, they build an evidence graph using the selected tables and evidence sentences to select the cells. The authors demonstrate the effectiveness of their system by evaluating it on the FEVEROUS dataset.

Strengths: All the reviewers unanimously agree that this is a solid contribution with interesting implications. The approach used in this study is simple yet effective, yielding substantial improvements over previous models. The comprehensive experimentation conducted in this study greatly contributed to understanding the mechanism of the proposed method.

Weaknesses: I don't think there are any major weaknesses -- some of the weaknesses have been addressed during the discussion phase. The authors should take the feedback into account when preparing for the final version of the paper. For example, concerns about the baseline raised by Reviewer cEAj have not been addressed in full.
Decision: Accept-Main