SlowGuess/NAACL_2022_OCR huggingface dataset:
Nan Hu\*, Zirui Wu\*, Yuxuan Lai\*, Xiao Liu\*, Yansong Feng\*
$\spadesuit$ Wangxuan Institute of Computer Technology, Peking University, China
$\diamond$ The MOE Key Laboratory of Computational Linguistics, Peking University, China
Department of Computer Science, The Open University of China
{hunan, ziruiwu, erutan, lxlisa, fengyansong}@pku.edu.cn laiyx@ouchn.edu.cn
Different from previous fact extraction and verification tasks that only consider evidence of a single format, FEVERIOUS brings further challenges by extending the evidence format to both plain text and tables. Existing works convert all candidate evidence into either sentences or tables, thus often failing to fully capture the rich context in their original format from the converted evidence, let alone the context information lost during conversion. In this paper, we propose a Dual Channel Unified Format fact verification model (DCUF), which unifies various evidence into parallel streams, i.e., natural language sentences and a global evidence table, simultaneously. With carefully-designed evidence conversion and organization methods, DCUF makes the most of pre-trained table/language models to encourage each evidence piece to perform early and thorough interactions with other pieces in its original format. Experiments show that our model can make better use of existing pre-trained models to absorb evidence of two formats, thus outperforming previous works by a large margin. Our code and models are publicly available<sup>1</sup>.
The task of fact extraction and verification aims to extract evidence and verify a given claim. Previous efforts focus on dealing with text format evidence from unstructured documents (Nie et al., 2019; Zhong et al., 2020; Kruengkrai et al., 2021) or evidence from a single given table (Chen et al., 2020; Yang et al., 2020; Eisenschlos et al., 2020). Recently, Aly et al. (2021) propose a new realistic setting, FEVERIOUS, i.e., fact extraction and verification over unstructured and structured information. In FEVEROUS, models should not only extract evidence sentences/table cells from millions
Figure 1: An excerpt example from FEVERIOUS.
of passages, but also combine the evidence in different formats to verify a given claim.
Previous works on FEVEROUS generally convert all evidence pieces into either plain text (Aly et al., 2021; Saeed et al., 2021; Malon, 2021) or several tables (Bouziane et al., 2021). However, format conversions inevitably lose rich context information for the converted evidence, thus may mislead the subsequent encoding and interaction steps. For example, in Figure 1, the entire top two rows are indispensable to understand the table cell Won. It is difficult to identify all related context cells and design a general conversion method to render them into sentences, but these connections can be easily caught by pre-trained table models (Herzig et al., 2020; Yin et al., 2020). On the other side, identifying/re-organizing crucial elements in a sentence to construct a table is also challenging. Simply inserting a whole sentence in a table cell (Bouziane et al., 2021) will make the new cells much larger (and unique) than normal ones, thus can not make the most of general pre-trained table models (Herzig et al., 2020; Yin et al., 2020) as we
'https://github.com/lanlanabcd/dualannel_feverous
Human Language Technologies, pages 5232 - 5242
expect.
Considering the inevitable expense in format conversion, we believe that each evidence in its original format can contribute necessary information to final verification, thus should be better encoded in its original format. This further indicates that we should design both sentence-to-table and table(cell)-to-sentence conversion methods to obtain all evidence in both formats, and maintain two parallel encoders to absorb the two formats, respectively. An advantage of doing so is to maximally encourage early interaction, which proves more effective than pair-wise encoding (Tymoshenko and Moschitti, 2021; Jiang et al., 2021)
When converting table evidence into sentences, previous works either convert table cells to a concatenation of key-value pairs (Aly et al., 2021; Malon, 2021), or construct sentences in a coordinate-description style (Kotonya et al., 2021a). They pay less attention to the conventional organization of tables structures. We observe that, in a table, the column headers usually represent the types/properties and the row headers often denote entities or scopes. We argue that one should consider these conventions to convert a table cell evidence into more natural sentences, and later pretrained language models will be able to better capture the contextualized semantics of the table cells from generated sentences. On the other hand, existing pre-trained table models are trained to analyze one table at one time, while previous evidence conversion methods produce several small tables for one instance. It would be necessary to properly organize all evidence in one table so that pre-trained table models can allow the most interactions among all evidence pieces.
In this paper, we propose a dual channel unified format verification model (DCUF) to allow each evidence piece encoded in its original format and maximally maintain its original rich context, while encouraging further interactions with other evidence. DCUF converts each evidence into both textual and tabular formats in respective channels, and apply corresponding pre-trained models to learn the representations for the final verification. With the dual channel setting and carefully designed evidence conversion methods, DCUF makes better use of pre-trained language/table models to perform early and thorough interactions among all evidence and also between the claim and evidence.
In summary, we make the following contribu
tions in this paper: (1) we propose DCUF, a novel model to maintain various evidence in two unified formats and allow each evidence piece to interact with other evidence in its best form. Experiments show that DCUF outperforms all previous works in literature. (2) we propose a context-aware evidence conversion method that can properly organize evidence of different formats, which fits current pre-trained language/table models hence take their most advantage to obtain accurate and focused representations.
The FEVERIOUS task can be formalized as, given a claim  $q$  and Wikipedia dump, a model is asked to find the evidence set consisting of sentences  $S$  and table cells  $C$ , and predict the veracity label of the claim accordingly. The veracity label set includes "SUPPORTS", "REFUTES" and "NOT ENOUGH INFORMATION".
2.1 Model Overview
Following the widely adopted fact verification pipeline (Thorne et al., 2018; Aly et al., 2021), we take three steps to solve the FEVERIOUS task (i) retrieving pages from the Wikipedia dump; (ii) extracting evidence from the retrieved pages, and (iii) verifying the claim according to extracted evidence.
Specifically, for the document retrieval step, we narrow the search space with an information retrieval model DRQA (Chen et al., 2017) and then rerank the retrieved pages. For the evidence retrieval step, we design a multi-turn cell selector to extract sentence evidence and table evidence respectively, and select evidence cells from tables. Finally, we propose a Dual Channel Unified Format verification model (DCUF, shown in Figure 2) for the verification step. DCUF converts evidence to a unified table/sentence format with carefully-designed evidence conversion and re-organization methods in each channel, and combine dual-channel encodings to make the final prediction.
2.2 Document Retrieval
An efficient and effective document retriever is required since the Wikipedia dump containing millions of pages. We first narrow the search space to several hundred pages  $(m_0)$  with an efficient information retrieval method based on TF-IDF, namely, DRQA (Chen et al., 2017). A RoBERTa-based
Figure 2: The architecture of our verification model. Orange lines show how we arrange the evidence into a text format, while blue ones show how to arrange into a table format.
re-ranker (Saeed et al., 2021) and a BM25-based re-ranker are then applied in parallel to re-rank the  $m_0$  document candidates. We combine the results of two re-rankers and keep top  $m$  documents since BM25 focuses more on entity matching and RoBERTa-based re-ranker pays more attention to the overall sentence structure. The document scores are calculated as the sum of their rankings in the two re-rankers. Documents with lower scores have higher priority.
We further notice that the first several words of a claim always contain the page titles needed. We therefore derive a position-aware sub-sequence matching to strengthen the page retriever. We also remove pages with a long Wikipedia title starting with a specified year that is not contained in the claim.
2.3 Evidence Retrieval
We use DRQA (Chen et al., 2017) to extract  $k$  sentences  $S = \{s_i\}_{i=1}^k$  and  $n$  tables  $T = \{t_i\}_{i=1}^n$  from the retrieved pages, respectively. Then we select cells from the extracted tables. Many instances in the FEVEROUS dataset require evidence cells from more than one table, and each retrieved table has different relevance score to the claim. However, the widely-used cell extractor (Aly et al., 2021) reserves cells from only one table in their implementation.
We thus propose a Multi-turn Cell Selector (MCS), which retrieves cells from all evidence tables and consider the importance of the retrieved tables. A basic cell selector concatenates a given claim  $q$  and a flattened candidate evidence table
$t_i$  and feeds it into a sequence tagger to decide whether each cell in the table should be selected. MCS implements this procedure in a multi-turn manner, since each table has a different relevance score to the given claim. In the first turn, MCS selects the table most related to the given claim and feeds it to the basic cell selector. All cells with a selection score larger than the threshold  $g$  will be added to the cell evidence set  $C$ . In the second turn, all tables ranked second in  $T$  are the input to the basic selector. If the number of cells in  $C$  has not reached the upper limit, MCS adds the newly selected cells in the second turn to  $C$ . MCS repeats this procedure for  $n$  loops and we will get the cell evidence set  $C = \{c_i\}_{i=1}^{o_j}$ .  $o_j$  is the number of cells selected as evidence for the  $j^{th}$  instance.
2.4 Unified Format Encodings
Since the evidence can be of two formats, textual format and tabular (or cell) format, we convert each evidence of one format to another, so that we will get a unified representation of all evidence and could easily assemble them. We propose two conversion methods, i.e., cells-to-sentences and sentences-to-tables for the original tabular evidence and textual evidence, respectively. We carefully design the evidence conversion and re-organization methods to make converted evidence natural, thus take better advantage of the pre-trained language/table models.
2.4.1 Text Format Encoding
We consider the table conventions, i.e., row headers in general tables usually represent attribute types,
and convert table cells into natural sentences, thus make better use of pre-trained language models to perform early interaction over the claim and all evidence.
There are two types of tables on the Wikipedia pages, (i) general tables and (ii) Infoboxes. For general tables, we ignore header cells selected by the evidence retriever and convert content cells selected into text format. For each cell, we identify its row header cell and column header cell. We find that most general-typed tables are column tables, which means they only contain column headers. However, the first column of a table, in many cases, indicates the object name and others are attributes. Therefore, if a cell does not have an explicit header cell, we choose the first cell of the same row to be its row header cell. We observe that the row header cell for a general table always indicates the object name, the column header cell indicates the attribute type, and the cell itself is the attribute value for the object. We thus form the corresponding text for a cell in a general table as “<column header> for <row header> is <cell value>.” For Infoboxes, it is a different story. The row header is always the attribute type, the column header is the field that the attribute belongs to, and the Wikipedia title is the object name. Therefore, the text representation of each cell in an Infobox is formulated as “<column header> : <row header> of <Wikipedia title> is <cell value>”. As shown in Figure 3, “21-24 minutes” is a selected cell, its row header “Running Time” is the attribute type, and the header cell “Production” is the field which the attribute “Running time” belongs to. Thus, the cell will be converted to “Production: Running time of The Simpsons is 21-24 minutes.”
Claim verification on a set of evidence containing many cells often requires operations on cells in the same column, such as maximum and summary (Aly et al., 2021). Therefore, we pack the texts from cells in the same column together. These texts are joined by semicolons and form a piece of column text. Each column is converted into a sentence. As shown in Figure 3, the cells "December 17, 1989" and "October 11, 1990" are jointly converted to one sentence, "Season premiere for 1 is December 17, 1989; Season premiere for 2 is October 11, 1990." The given claim, the extracted text evidence, and the column texts are concatenated together to form the input to the text format evidence encoder, separated by the separator "
" in
the RoBERTa model. Column texts from the same table are adjacent, and sentences from the same Wikipedia page are arranged together.
2.4.2 Table Format Encoding
As shown in the right part of Figure 3, we propose to construct a single evidence table containing all evidence candidates, both textual and tabular, since most existing table pre-training models are designed to and trained to analyze one single table at one time. Making better use of the pre-training models helps to perform early and thorough evidence interaction.
The method of converting text format evidence to tabular format is straightforward. We group the evidence sentences from the same Wikipedia page. If there are  $n$  evidence sentences from the same Wikipedia page, we construct a table of  $n + 1$  rows and 1 column. The only cell in the first row is a header cell containing the Wikipedia title. And for other rows, each row has one cell, and in that cell is a piece of sentence evidence from that page. Assuming there are evidence sentences from  $m$  pages, we will get  $m$  table units after this step.
As the pre-training table model has a capacity limit, we crop irrelevant cells first to compress the extracted tables. To be precise, rows and columns containing no selected cells are removed. After that, we add one cell row to the top of each cropped table, this cell contains the title of the Wikipedia page from which the table is extracted. If there are  $n$  extracted tables, we will get  $n$  cropped table from this step.
We get a global evidence table by stacking the  $m$  tables from sentences and  $n$  tables from tabular evidence, as illustrated in Figure 3. Then, we feed the claim and the global evidence table to a pretrained table model, TAPAS (Herzig et al., 2020), and get the tabular format evidence representation.
2.4.3 Dual-Channel Verdict Prediction
The final verdict prediction is based on the dual channel encoding. Therefore, each evidence can be encoded in its original format while interacting with all evidence pieces and the claim.
We concatenate the text format evidence encoding  $h_{text}$  and the tabular format evidence encoding  $h_{tab}$  to obtain a joint format encoding for prediction. With a feed-forward network and a softmax layer, we obtain the veracity probability distribution of the claim and the predicted label is the one
Figure 3: An example of our evidence conversion methods. Cells with [H] are explicitly specified header cells. Light blue cells are the selected cells. The left part shows how we convert evidence to a unified text format, while the right shows how to convert all evidence candidates into a global evidence table.
with the largest probability:
$$
h = \left[ h _ {\text {t e x t}}; h _ {\text {t a b}} \right] \tag {1}
$$
$$
p (y \mid q, S, T, C) = \operatorname {S o f t m a x} (\operatorname {F N N} (h)) \tag {2}
$$
$$
\hat {y} = \operatorname {a r g m a x} _ {y} p (y | q, S, T, C) \tag {3}
$$
where  $\mathrm{p}(y|S,T,C)$  represents the probability of each alternative label  $y$  given the claim  $q$ , evidence sentences  $S$  and evidence tables  $T$ .
To strengthen the model's ability to predict the veracity label with the evidence set containing irrelevant pieces, we construct two instances for each claim in the training set. One is the claim with the gold evidence provided by the FEVERIOUS dataset, and the other is the claim with extracted evidence pieces from previous evidence extraction steps. We use the cross-entropy loss function:
$$
L = - \frac {1}{N} \sum_ {i = 1} ^ {N} \log (\mathrm {p} (\hat {y} = y _ {i} | q, S, T, C)) \tag {4}
$$
where  $y_{i}$  is the true veracity label of the  $i^{th}$  instance.  $N$  is the size of the training set.
We evaluate our models on the FEVERIOUS dataset, where each claim is annotated with a gold veracity label and several gold evidence sets. Any one of the evidence sets is sufficient to verify the claim. More details about the FEVERIOUS dataset are in Appendix.
The FEVEROUS dataset provides two official metrics, namely label accuracy (Acc.) and FEVEROUS score. Label accuracy calculates the ratio
of the instances whose veracity label is correctly predicted. FEVEROUS score is the ratio of the instances whose veracity label is correct and the extracted evidence set is sufficient. Here, sufficient evidence sets are defined as the evidence sets covering one of the gold evidence sets provided in the FEVEROUS dataset. Note that there are at most 25 table cells and 5 sentences to calculate the scores.
In the document retrieval step, the number of pages retrieved by the BM25-based retriever  $m_0$  is 150. We keep the top 5 pages for evidence extraction after page re-ranking. For evidence retriever, the top  $k = 5$  sentences and top  $n = 2$  tables are extracted from the retrieved pages. The gate of cell selection  $g$  is 0.25 and a maximum of 25 cells are selected in total for each claim.
We use an Adam optimizer (Kingma and Ba, 2015) with a linear learning rate scheduler. The rate of warm-up steps is  $20\%$ . The peaking learning rate for parameters in pre-trained language models is  $10^{-5}$  and  $10^{-3}$  for other parameters. The batch size is 24, implemented with gradient accumulation techniques. DCUF takes about 5 hours to run on a single NVIDIA A100 Tensor Core GPU (40GB).
The RoBERTa-based re-ranker is initialized from the hugging face checkpoint without further fine-tuning. The TAPAS checkpoint is fine-tuned with a table fact verification task, Tabfact (Chen
$^{2}$ https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-12-v2
Models	Development	Test
Feverous	Acc.	E-P	E-R	E-F1	Feverous	Acc.	E-P	E-R	E-F1
Official Baseline	19	53	12	30	17	17.73	48.48	10.17	28.78	15.03
EURECOM	19	53	12	29	17	20.01	47.79	13.73	33.73	19.52
Z team	-	-	-	-	-	22.51	49.01	7.76	42.64	13.12
CARE	26	63	7	37	12	23	53	7	37	11
NCU	29	60	10	42	17	25.14	52.29	9.91	39.07	15.81
Papelo	28	66	-	-	-	25.92	57.57	7.16	34.60	11.87
FaBULOUS	30	65	8	43	14	27.01	56.07	7.73	42.58	13.08
DCUF	35.77	72.91	15.06	43.22	22.34	33.97	63.21	14.79	44.10	22.15
et al., 2020)  $^{3}$ . Same as the baselines, the sentence evidence encoder is RoBERTa-large tuned with several NLI and verification tasks $^{4}$ .
Document Retriever Details For BM25 re-reranker, all page candidates of every 200 adjacent instances are merged to build the BM25 index for these instances. Each document in the Wikipedia dump is represented by the concatenation of its title and the first 64 words of its content for the BM25 re-ranker. We use NLTK<sup>5</sup> to remove stop words and lemmatize the remains. For position-aware entity matching, if a sub-sequence, with more than two words, in the first ten words of a given claim is a page title in the Wikipedia dump and it is not in the  $m$  documents we replace the page of the lowest priority with it.
The overall performance of our model on the development set and the test set are shown in Table 1. We get an increase of  $5.77\%$  on the FEVERIOUS score and  $7.91\%$  on the accuracy over the previous best model FaBULOUS (Bouziane et al., 2021) on the development set. For the test set, the increase is  $6.96\%$  and  $7.14\%$  in Feverous score and label accuracy, respectively. These results suggest the effectiveness of our proposed DCUF model. The evidence format of a global evidence table is consistent to the input of pre-trained table models. Thus, DCUF can make better use of the internal ability of pre-trained models than previous works which concatenate linearized tables or max-pool lots of claim-table pair encoding (Bouziane et al.,
2021). Moreover, DCUF also performs better than another well-performing model, CARE (Kotonya et al., 2021b). DCUF converts cells to meaningful sentences that are similar to the inputs of PLMs pre-training stage, which makes better use of the PLMs ability.
We also conduct experiments with the gold evidence to investigate the effectiveness of our verification model. The results are shown in Table 2. DCUF obtains an increase of  $2.88\%$  on accuracy over the previous best result and  $3.68\%$  over RoBERTa-based models. With all evidence candidates in the same format and preserving context information, our system can make better use of the pre-trained language/table models and perform early and thorough interactions among evidence and between the claim and evidence.
Table 1: Model performance on the development set and test set. Acc. is Accuracy. E-P, E-R and E-F1 is Evidence Precision, Evidence Recall and Evidence F1, respectively.
Models	Accuracy
Fabulous-Max	80
Papelo-RoBERTa	82.9
NCU	84
Fabulous-Joint	84
Papelo-T5	84.8
DCUF	87.68
Table 2: Model accuracy on the development set with gold evidence as model input.
Evidence Extraction Results The document retrieval results are shown in Table 3. In Table 3, experiments show that both BM25 re-ranker and RoBERTa re-ranker can improve the document retrieval quality to a great extent compared to vanilla DrQA page retriever, with whole evidence set recall improvement of  $8.63\%$  and  $12.56\%$  respectively. The combination of them can further enlarge this gap to  $16.22\%$ . We find that the average number of pages in the merged set of top-5 BM25 re-ranked
$^{3}$ https://huggingface.co/google/  
tapas-large-finetuned-tabfact  
 $^{4}$ https://huggingface.co/ynie/  
roberta-large-snli_mnli_fever_anli_R1_  
R2_R3-nli  
 $^{5}$ https://www.nltk.org
Models	MAP	Rec-single	Rec-set
DrQA@150	91.13	90.00	89.29
DrQA	69.32	66.91	65.50
BM25 RR	77.98	75.32	74.13
RoBERTa RR	82.56	78.65	78.06
Combined RR	85.13	82.29	81.72
+ Rule Enhancement	88.11	85.20	84.82
Table 3: Document retrieval results on the development set. RR means re-ranker. Rec-single is the recall of single evidence. Rec-set considers all evidence for a instance as a whole. It is the recall with top-5 pages retrieved without special statement.
Models	Table	Sent	Cell	All
Official Baseline	56	53	28.70	30
FaBULOUS	-	56.6	34.2	40.4
MCS	75.59	62.54	58.41	43.22
results and top-5 RoBERTa re-ranked results is 7.75 in the development set. It proves that these two re-rankers tend to focus on different aspects when evaluating the correlation of the given claim and a page candidate. The rule-based enhancement methods, namely matching position-aware entities and removing pages with unmatched year, bring a further improvement of  $3.10\%$ . It indicates that document retrievers should take the word positions in the given claim into consideration. Without any training procedure or Dense Retriever (which is time-consuming), we get a whole set recall of  $84.82\%$  when retrieving 5 pages for each claim.