From Relevance to Utility: Evidence Retrieval with Feedback for Fact Verification
Authors: Hengran Zhang
Ruqing Zhang
Jiafeng Guo
Maarten de Rijke
Yixing Fan
Xueqi Cheng
Venue: EMNLP 2023 Findings (2023)
Keywords: Utility; Evidence Retrieval; Fact Verification

Abstract: Retrieval-enhanced methods have become a primary approach in fact verification (FV); it requires reasoning over multiple retrieved pieces of evidence to verify the integrity of a claim. To retrieve evidence, existing work often employs off-the-shelf retrieval models whose design is based on the probability ranking principle. We argue that, rather than relevance, for FV we need to focus on the utility that a claim verifier derives from the retrieved evidence. We introduce the $\textbf{feedback-based evidence retriever} (FER)$ that optimizes the evidence retrieval process by incorporating feedback from the claim verifier. As a feedback signal we use the divergence in utility between how effectively the verifier utilizes the retrieved evidence and the ground-truth evidence to produce the final claim label. Empirical studies demonstrate the superiority of FER over prevailing baselines.

From Relevance to Utility: Evidence Retrieval
with Feedback for Fact Verification
Hengran Zhang!? 1.2 Ruqing Zhang 1.2 Jiafeng Guo 1.2
Maarten de Rijke²
Yixing Fan 1.2 Xueqi Cheng!? 1.2
School of Computer Science and Technology, University of Chinese Academy of Sciences; "CAS Key Laboratory of Network Data Science and Technology,
Institute of Computing Technology, Chinese Academy of Sciences; 'University of Amsterdam. (zhanghengran224, zhangruqing, guojiafeng, fanyixing, exq) @ict.ac.en, m.derijke uva.nl
Abstract
based on their likelihood of being relevant the claim. However, the sentences retrieved in this way are assumed to be consumed by humans, and this may not align with a retrieval-enhancee verification framework (Zamani et al., 2022): the top-ranked sentences produced by existing retrieval models do not always align with the judgments made by the claim verifier regarding what counts evidence.
Retrieval-enhanced methods have become pri- mary approach in fact verification (FV); it re- quires reasoning over multiple retrieved pieces of evidence to verify the integrity of a claim. To retrieve evidence, existing work often employs off-the-sheeff retrieval models whose design is based on the probability ranking principle. We argue that, rather than relevance, for FV we need to focus the utility that a claim verifier derives from the retrieved evidence. We in- troduce the feedback baadback-basedd evidence retriever (FER) that optimizes the evidence retrieval pro- cess by incorporating feedback from the claim verifier. As feedback signal vergence in utility between how effectively the verifier utilizes the retrieved evidence and the ground-truth evidence produce the final claim label. Empirical studies demonstrate the superiority FER over prevailing baselines."
We argue that when designing evidence retrieval models for FV, the notion of relevance should be reconceptualized as the utility that the verifier de- rives from consuming the evidence provided by the retrieval model, a viewpoint that aligns well with task based perspectives on IR (Kelly et al., 2013). Hence, we assume that when training evidence re- trievers, it is advantageous obtain feedback from the claim verifier as signal to optimize the trieval process.
the di-
Therefore, we propose shift in emphasis from relevance to utility in evidence retrieval for FV. We introduce the feedback- baaedd evidence retriever (FER) that incorporates a feedback mechanism from the verifier to enhance the retrieval process. FER leverages a coarse-to-fine strategy. Initially, it identifies candidate set of relevant sentences the given claim from the large-scale corpus. Sub- sequently, the evidence retriever is trained using feedback from the verifier, enabling re-evaluation of evidence within the candidate set. Here, feed- back is defined as the utility divergence observed when the verifier evaluates the sentences returned by the retriever, compared to when it consumes the ground-truth evidence for predicting the claim label. By measuring the utility criterion between the two scenarios, we can optimize the retriever specifically for claim verification.
1 Introduction
The risk of misinformation has increased the de- mand for fact-checking, i.c., automatically assess- ing the truthfulness of textual claims using trust- worthy corpora, e.g., Wikipedia. Existing work on fact verification (FV) commonly adopts retrieval- enhanced verification framework: an evidence re- triever is employed to query the background corpus for relevant sentences, to serve as evidence for the subsequent claim verifier. High-quality evidence is the foundation of claim verification. Currently, pre- vailing approaches to identifying high-quality evi- dence typically adopt off-the-shelf retrieval models from the information retrieval (IR) field for evi- dence retrieval (Wan et al., 2021; Jiang et al., 2021; Chen et al., 2022a; Liu et al., 2020). These models are usually based on the probability ranking prin- ciple (PRP) (Robertson, 1977), ranking sentences
Experimental results on the large-scale Fact Extraction and VERification (FEVER) dataset (Thore et al., 2018) demonstrate a 23.7% Fl per- formance gain over the SOTA baseline.
Reseaach conducted when the author was at the Univer- sity of Amsterdam. 'Our code and data are available at https://github com/ict-B bi gdggalab//FFE

---

2 Background
Caediles relsoo (10accae Nikitine Li toms some love ditillionna com (idensmm fortoo (3
For the FEVER task (Thorne et al., 2018), au- tomatic FV systems need to determine a three- way veracity label, i.e., SUPPORTS, REFUTES or NOT ENOUGH INFO, for each suman-generaan claim based a of supporting evidence from Wikipedia. The FV system can be split into docu- ment retrieval, evidence retrieval, and claim verifi- cation phases. The first phase is to retrieve related documents from the corpus. The second phase aims to extract evidence from all sentences of the recalled documents. The final phase aggregates the information of the retrieved evidence to predict claim label. In this work, contributions focused the evidence retrieval step. The related work and more extensive discussions are included in Appendix A.I.
Haeso Clt 100losnn an has crpoell
Retieved
G0 (14)
(4)
Pine grained nitieved
Chiiivetller
04
Gramd-arthh
Figure I: Architecture of FER.
beginning of each sentence to indicate its presence. The training objective & for Ro consists of two parts, i.c., evidence classification L and utility divergence Lutis denoted as:
3 Our Approach
== + BL
Based on the related documents (i.e., Wikipedia pages) obtained by a common document retrieval method with entity linking (Hanselowski et al., 2018; Liu et al., 2020), FER comprises two com- ponents: (i) coarse retrieval: using a bRs-b retrieval model to recall set of candidate sen- tences to the claim from the related documents; and (ii) fine-grained retrieval: selecting evidential sen- tences from the candidate based on the feedback from the claim verifier. The overall architecture of FER is illustrated in Figure 1.
where a and / are coefficients.
Evidence classification. We first leverage the ground-truth evidence to boost plausibility. Specif- ically, we input the concatenated sequence into BERT and leverage the hidden state of cach [CLS] classify the corresponding sentence through lin- ear layers and ReLU activation functions. Here, E {0, 1} is the sentence label where or rep- resents whether a sentence is the ground-truth ev- idence or not. Then, the classification loss is de- fined as a measure of the diffcrence between the predicted evidence and the ground-truth evidence via binary cross-entropy loss, i.e.,
3.1 Coarse retrieval
For given claim, first retrieve set S of candi- date sentences from the related documents based on the PRP. Following (Zhou al., 2019; Liu et al., 2020), we employ the base version of BERT for coarse retrieval. We utilize the hidden state of the "[CLS]" token to represent the claim and sentence pair. To project the "[CLS]" hidden state to rank- ing score, we employ linear layer followed by tanh activation function. Lastly, optimize the BERT-based ranking model using typical pair- wise loss function. During inference, given a test claim, we take the top-K K ranked sentences to form the candidate set
Leda
Σ golog(p(s)) + (1 g) log(1 - p())), SCB
where is sentence in the candidate set S and p(s) is the predicted probability of by R& the ground-truth. During training, this loss function Leaa encourages the retriever select evidence sentences that are deemed plausible, representing ground-truth evidence.
Utility divergence based on the claim verifier. To further ensure that the retrieved evidence im- proves verification performance, we propose to get feedback from the claim verifier as a signal for op- timizing the evidence retriever. Here, we leverage the utility that the verifier obtains by consuming the retrieved evidence as the feedback. Specifically, for each claim c, we can measure the divergence utility between the verifier's judgements based on the ground-truth evidence and those based on the
3.2 Fine-grained retrieval with feedback Given the candidate set S, train the fine- fine-grained evidence retriever Rg with the feedback from the claim verifier. We leverage the base version of BERT to implement Rg. The claim and sen- tences in S are concatenated as a single input se- quence, with a special token [CLS] added at the

---

Table Comparisons of the evidence retrieval performance achieved by FER and baselines. significant improvements over all the baselines (p-value 0.05).
indicates statistically
Dev
Test
Model
P
F1
P
R
F1
ID-ID (Thorne et al., 2018)
17.20
11.28
47.87
18.26
ColumbiaNLP (Chakraburty et al., 2018)
78.04
23.02
75.89
35.33
UKP-Athene (Hanselowski et al., 2018)
87.10
23.61
85.19
36.97
GEAR (Zhou al., 2019)
24.08
86.72
37.69
23.51
84.66
36.80
NSMN (Nie et al., 2019a)
36.49
86.79
51.38
42.27
70.91
52.96
KGAT (Liu et al., 2020)
27.29
94.37
42.34
25.21
87.47
39.14
DREAM (Zhong et al., 2020)
26.67
87.64
40,90
25.63
85.57
39.45
DQN (Wan et al., 2021)
79.92
64.98
$2.24
77.93
62.55
GERE (Chen et al.. 2022a)
79.61
67.40
54.30
77.16
63.74
Stammhach (Stammhach, 2021)
71.25
83.21
76.72
FER w/o Leta
80.261
81.62
80,931
75.76
75.97
75.86"
FER w/o Lucs FER
75.69t
87.38
81.121
71,84*
80.87
76.08
84.04!
84.59
84.31
79.35'
78.34
78.84
sentences provided by the retriever when predicting the claim label, i.c.,
prises 185,455 claims with 5,416,537 Wikipedia documents from the June 2017 Wikipedia dump. All claims are annotated "SUPPORTS", "RE- FUTES "NOT ENOUGH INFO".
D'lc, y Dole, Ro(e, S)), = -
where is the given claim, F denotes the ground- truth evidence and y" denotes one-hot indicator vector of the ground-trutt truth claim label. Ro(c, S) represents the sentences selected by the evidence retriever, which takes the candidate set and the claim as input. (-(-) is the probability distribu- tion predicted by the claim verifier. The details of loss gradient passed to the fine-grained retriever can be found in the Appendix A.2.
For evaluation, leverage official evaluation metrics, i.e., Precision (P), Recall (R), and FI for evidence retrieval, and the FEVER score and Label Accuracy (LA) for claim verification. We prioritize the Fl as our primary metric for evidence retrieval because it directly reflects the model performance in terms of retrieving precise evidence.
The candidate set size K is set to 25; both and 3 are set to 1. All paramepass on the development set. We include detailed de- scriptions of the implementation details, evaluation metrics, and baselines in Appendix A.3.
tuned
Here, we also BERT's base version as the verifier in FER. During training, the verifier per- forms classification on the concatenated input, i.e., [CLS] + + [SEP] + E + [SEP], and leverage the hidden state [CLS] to classify c into three cate- gories via a linear layer and softmax function. After training, the verifier is fixed and we directly com- pute the probability prediction vector (-(-) based F. and Ro(c, S), respectively. Note that leverage other advanced claim verifiers to provide feedback (Section 4.4). The loss function Luti en- courages the retriever to prioritize the selection of sentences that are crucial for claim verification.
4.2 Results on evidence retrieval
We select several representative evidence retrieval models as our baselines and conduct experiments on both the development and test set following the common setup in FEVER. Based on the results presented in Table 1, we find that: (i) FER signifi- cantly outperforms the statc-of-the-art methods in terms of P and FI, demonstrating its superiority in providing supporting evidence for claim verifi cation based on utility. (ii) The superior precision achieved by FER comes at the expense of lower recall when compared to baselines. The reason might be that FER aims to retrieve more precise and concise sets of evidence for each claim. That is, FER results in smaller amount of retrieved ev- idence compared to the baselines. Similarly, DQN, GERE and Stammbach also aim identify precise evidence. However, their P and Fl scores are sub- stantially lower than those for FER, indicating that
At inference time, given a test claim, we leverage the optimized retriever to retrieve evidence (Sec- tion 4.2). Then, we can directly input the retrieved evidence to several advanced verification models verify the claim (Section 4.3).
4 Experiments
4.1 Experimental settings
We conduct experiments on the FEVER bench- mark dataset (Thorne et al., 2018), which com-

---

Table 2: Performance of different claim verification
4.4 Quality analysis
models on the test using evidence from the original paper vs. evidence retrieved by FER.
Feedback using different verifiers. Here, we ex- plore the potential of utilizing off-the-shelf claim verifiers to offer feedback to Rg in our FER. From Table 2, KGAT (Liu et al., 2020) and GAT exhibit promising verification performance, making them suitable choices for providing feedback. BERT Pair cannot be employed for providing feedback to FER, since it produces multiple independent probabilities instead of a probability distribution encompassing the final claim labels. In future work, we will explore alternative feedback formulations to incorporate additional verifiers. The experiments are reported on the test set; see Table 3. Similar findings have been obtained the development set; see Appendix A.4.1. Obtaining feedback from other verifiers could also show promising evidence retrieval performance; this result showcases the adaptability of the proposed FER method, which effectively adjusts to different verifiers. Besides, good retrievers and verification models can mutu- ally benefit from each other's s strengths.
Model
LA
FEVER
BERT Concat (Zhou et al., 2019)
71.01
63.64
BERT Concat + FER
72.46
68.16
BERT Pair (Zhou et al., 2019)
69.75
65.18
BERT Pair + FER
72.12
68.03
GEAR (Zhou et al., 2019)
71.60
67.10
GEAR FER
72.54
67.49
GAT (Liu al., 2020)
72.03
67.56
GAT+FER
72.85
69.43
KGAT(BERT Basc) (Liu et al., 2020)
72.81
69.40
KGAT+FER
73.34
69.61
considering the utility of evidence contributes to re- trieving evidential sentences for FV. (iii) FER with- out Cela outperforms FER without Cun terms of P. indicating that feedback from the verifier plays a crucial role in aiding the retriever to identify precise evidence that is crucial for the verification process. FER without Lcia retrieves a smaller number of sentences than FER without Luti, i.e., its perfor- mance in terms of R and Fl is poorer. Some case studies can be found in Appendix A.4.2.
85
80
75.
4.3 Results on claim verification
To better understand the effectiveness of evidence retrieved by FER, we choose several advanced claim verification models, and provide them with evidence retrieved by FER and evidence obtained from the original paper, respectively. The experi- ments are reported on the test set; see Table 2. Sim- ilar findings can be obtained the development set; see Appendix 1. All the claim verifica- tion models leveraging evidence retrieved by FER outperform their respective original versions based on PRP. By conducting further analyses, we find that the evidence retrieved based on the likelihood of being relevant to the claim in the original pa- pers may not be always useful for the verification process or contain conflicting pieces. FER is able to select more precise evidence for claim verifica- tion, contributing to the verification outcome and leading to improved results.
70-
F1
60
20
Figure 2: Evidence retrieval performance with different
values of K on the test set.
Effect of the candidate set size K. In Figure 2, we plot the effect of the number of candidate sentences K obtained by the coarse retrieval components on the final evidence retrieval performance. Recall is high for all values of K. The reason might be that on average, each claim is associated with 1.86 relevant sentences in FEVER, while K is set to be larger than 5. Furthermore, lower value of K im- plies that less information will be available for the fine-graing -reiggained retrieval model, which, in turn, hurts precision. As K is increased, more information is provided to the model, which amplifics the impact of feedback, potentially causing slight fluctuations in the recall rate.
Table 3: Evidence retrieval performance with feedback from different verifiers on the test set.
5 Conclusion
Verifier
Fl
In this paper, we proposed FER (feedback-based evidence retrieval) for FV, novel evidence re- trieval framework that incorporates feedback from the claim verifier. Unlike traditional approaches
GAT
77.93
79.48
78.70
KGAT
84.38
75.95
79.94

---

based on PRP, FER places an emphasis on the util- ity of evidence that contributes meaningfully the verification process, going beyond mere relevance. Through the integration of feedback from the ver- ifier, FER effectively identifies the evidence that is both relevant and useful for claim verification. The experimental results on the FEVER dataset demonstrated the effectiveness of FER.
Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scien- tific Research, https://hybrid-ii
nl, and project LESSEN with project number NWA.1338 183 of the research program NWA ORC 2020/21, which is (partly) financed by the Dutch Research Council (NWO).
All content represents the opinion of the authors, which is not necessarily shared endorsed by their respective employers and/or sponsors. We would like to thank the reviewers for their valuable feed- back and suggestions.
Limitations
In this paper, we utilized the performance disparity between ground-trum evidence and retrieved evi- dence for claim verification as a form of feedback to train an evidence retrieval model. There are two primary limitations that should be acknowledged: (i) Currently, the retrieval and verification models are optimized independently, lacking conditional optimization or joint end-to-end optimization. Fu- ture research could explore approaches to optimize both the retrieval and verification models in a sin- gle objective function. In this manner, we expect to see improved overall performance. (ii) In this study, we focused solely on investigating the prob- ability distribution generated by the claim verifier compute the utility divergence. However, it is important to note that there exist multiple methods for quantifying the divergence in utility of retrieval results, e.g., the gradients of the verification loss. Additionally, it is worth considering that there may be various forms of feedback that can be incor- porated. We hope that this research will inspire further exploration and attention in this area for future studies. (iii) Finally, we only demonstrated the effectiveness of the proposed FER method on a single dataset, the FEVER dataset, and the evi- dence retrieval process. We encourage future work aimed at the creation of further claim verification datasets and document retrieval process.
References
Chenxin An. Ming Zhong, Zhichao Geng, Jianqiang Yang. and Xipeng Qiu. 2021. Retrievalsum: A re- trieval enhanced framework for abstractive summa- rization. arXiv preprint arXiv:2109.0773 07943.
Tuhin Chakrabarty, Tariq Alhindi, and Smaranda Mure- san. 2018. Robust document retrieval and individual evidence modeling for fact extraction and verification. In Proceedings of the First Workshop on Fact Extrac- tion and VERification (FEVER), pages 127-131.
Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Yixing Fan, and Xueqi Cheng. 2022a. Gere: Generative evidence retrieval for fact verification. In Proceedings of the 45th International ACM SIGIR Conference on Re- search and Development in Information Retrieval, pages 2184-2189.
Jiangui Chen, Ruging Zhang, Jiafeng Guo, Yiqun Liu, Yixing Fan, and Xueqi Cheng. 2022b. Corpusbrain: Pre-train a generative retrieval model for knowledge- intensive language tasks. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022, pages 1000000..000 ACM.
Miaoxin Chen, Zibo Lin, Rongyi Sun, Kai Ouyang, Hai- Tao Zheng, Rui Xie, and Wei Wu. 2022c. Retrieval enhanced segment generation neural network for task- oriented dialogue systems. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6596. IEEE.
Acknowledgments
This work was funded by the National Natu- ral Science Foundation of China (NSFC) un- der Grants No. 62006218 and 61902381, the China Scholarship Council under Grants No. 202 202104910234, the Youth Innovation Promotion Association CAS under Grants No. 2021 100, the project under Grants No. JCKY2022130COSS and 2021QY1701, and the Lenovo-CAS Joint Lab Youth Scientist Project. This work was also (partially) funded by the Hybrid Intelli- gence Center, a 10-year program funded by the
Anton Chernyavskiy and Dmitry Ilvovsky. 2019. Ex- tract and aggregate: A novel domain-independer approach to factual data verification. Proceedings of the Second Workshop on Fact Extraction and VERifi- cation (FEVER).
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACL-HLT 2019, Minneapolis, MN, USA,

---

June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186, Association for Computational Linguisties.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprini 1rXiv
Zhihao Fan, Yeyun Gong. Zhongyu Wei, Siyuan Wang, Yameng Huang. Jian Jiao, Xuanjing Huang, Nan Duan, and Ruofei Zhang. 2020. An enhanced knowl- edge injection model for commonsense generation. In Proceedings of the 28th International Confer- ence on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8- 8-13, 2020, pages 2014-2025, International Committee on Com- putational Linguistics.
Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Wei Chen, and Xueqi Cheng. 2023, On the robustness of generative retrieval models: An out-of-distribttonn perspective. arXiv preprint arXiv:2306. 22755
Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020. Fine-grained fact verification with kernel graph attention network. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7342-7351, On- line. Association for Computational Linguisties.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Pe- ters, Michael Schmitz, and Luke 7eutlemoyer. 2018. AllenNLP: A deep semantic natural language pro- cessing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1-6, Melbourne, Australia, Association for Computational Linguisties.
Ilya Loshchilov and Frank Hutter. 2019, Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018. UKP-athene: Multi sentence textual entailment for claim verification. In Proceed- ings of the First Workshop on Fact Extruction and VERification (FEVER), pages 103-108. Brussels, Bel- gium. Association for Computational Linguistics.
Jing Ma. Wei Gao. Shafiq Joty, and Kam-Fai Wong. 2019. Sentence-level evidence embedding for claim verification with hierarchical attention networks. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2561- 2571, Florence, Italy. Association for Computational Linguisties.
Shwai He. Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, and Dacheng Tao. 2023. Mera: Merging pre- trained adapters for few-shot learning, arXiv preprint arXiv:2308. 15982.
Yixin Nie, Haonan Chen, and Mohit Bansal. 2019 Combining fact extraction and verification with neu- ral semantic matching networks. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 33, pages 6859-6866.
Christopher Hidey and Mona Diab. 2018 Team SWEEPer: Joint sentence extraction and fact check- ing with pointer networks. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 150-155, Brussels, Belgium. Asso- ciation for Computational Linguistics.
Yixin Nie, Songhe Wang. and Mohit Bansal. 2019b. Revealing the importance of semantic retrieval for machine reading at scale. In Proceedings of the 2019 Conference on Empirical Methods in Natu- rul Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNL)), pages 2553-2566. Hong Kong. China. Association for Computational Linguistics.
Xuming Hu, Zhaochen Hong, Zhijiang Guo, Lijie Wen, and Philip S. Yu. 2023. Read twice: Towards faithfully interpretable fact verification by revisiting evidence. volume abs/2205.03507.
Md Rizwan Parvez, Jianfeng Chi, Wasi Uddin Ahmad, Yuan Tian, and Kai-Wei Chang. 2023. Retrieval en- hanced data augmentation for question answering privacy policies. In Proceedings of the 17th Confer- ence of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pages 201-210. Association for Computational Linguistics.
Kelvin Jiang, Ronak Pradeep, and Jimmy Lin. 2021. Ex- ploring listwise evidence reasoning with 15 for fact verification. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 Ithh International Joint Conference on Natu- ral Language Processing (Volume 2: Short Papers), pages 402-410.
Stephen F. Robertson. 1977. The prohability rank- ing principle in IR. Journal of Documentation. 33(4):294-344.
Diane Kelly, Jaime Arguello, and Robert Capra. 2013. NSF workshop on mnsk-based information search sys- tems. In ACM SIGIR Forum, volume 47, pages 116- 127. ACM New York, NY, USA.
Justyna Sarzynska- -aawee,k--Waee.r. Aleksander Wawer, Alek- sandra Pawlak, Julia Szymanowska, Izabela Stefa- niak, Michal Jarkiewicz, and Lukasz Okruszek. 2021. Detecting formal thought disorder by deep contex- tualized word representations. Psychiatry Research. 304: 14135.
Amrith Krishna, Sehastian Riedel, and Andreas Vlachos. 2022. Proofver: Natural logic theorem proving for fact verification. Transactions of the Association for Computational Linguistics, 10: 10:1003-

---

Rishab Sharma, Fatemeh H Fard, and Apurva Narayan. 2021. Retrieval enhanced ensemble model frame- work for rumor detection on micro-blogging plat- forms. In 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 227-232 IEEE
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in nearal information processing systems, 18:17733-17997
Amir Soleimani, Christof Monz, and Marcel Worring- 2020. Bert for evidence retrieval and claim verifi- cation. In Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14-17, 2020, Proceedings, Part II 42, pages 359-366, Springer.
Hamed Zamani, Fernando Diaz, Mostafa Dehghani, Donald Metzler, and Michael Bendersky. 2022. Retrieval-enhance machine learning. In SIGIR '22:
The 45th International ACM SIGIR Conference Research and Development in Information Retrieval, Madrid, Spain, July 11 -15, 2022, pages 2875-2886. ACM.
Yiping Song, Rui Yan, Xiang Li, Dongyan Zhao, and Ming Zhang. 2016. Two are better than one: An ensemble of retrieval-and and generation based dialog systems. arXiv preprint 1610.7744
Jing Zhang, Xiaokang Zhang, Jifan Yu, Jian Tang, Jie Tang, Cuiping Li, and Hong Chen. 2022. Subgraph retrieval enhanced model for multi-hop knowledge base question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pages 5773-5784. Association for Computational Linguisties.
Dominik Stammbach. 2021. Evidence selection a token-level prediction task. In Proceedings of the Fourth Workshop on Fact Extraction and VERifica- tion (FEVER), pages 14-20. Association for Compu- tational Linguistics.
Chen Zhao, Chenyan Xiong, Corby Rosset, Xia Song, Paul Bennett, and Saurabh Tiwary. 2020. Transformer- XH: Multi-evidence reasoning with ex- tra hop attention. International conference on learning representations.
James Thoree, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-IIIT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume I (Long Papers), pages 809-819. Association for Computational Linguistics.
Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2020. Reasoning over semantic-level graph for fact check- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 61 17776180, Online. Association for Computational Linguistics.
Hai Wan, Haicheng Chen, Jianfeng Du, Weilin Luo, and Rongzhen Ye. 2021. A dqn-based approach to finding precise evidences for fact verification. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1030- 1039.
Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. GEAR: Graph-based evidence aggregating and rea- soning for fact verification. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pages 892-901, Florence, Italy. Association for Computational Linguistics.
Han Wang, Yang Liu, Chenguang Zhu, Linjun Shou, Ming Gong, Yichong Xu, and Michael Zeng. 2021. Retrieval enhanced model for commonsense gener- ation. In Findings of the Association for Computa- tional Linguistics: ACL/JICNLP 2021, Online Event. August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages Computational Linguistics.
Qingfu Zhu, Lei Cui, Weinan Zhang, Furu Wei, and Ting Liu. 2019. Retrieval-enhanced adversarial train- ing for neural response generation. Proceedings of the 57th Conference of the Association for Compu- lational Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume Long Papers, pages 3763 3773. Association for Computational Linguis- tics.
Association for
Peilin Yang. Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of lucene for information retrieval research. In Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval, pages 1253-1256.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoe V Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advances in neural informa- tion processing systems, 32.

---

A Appendix
size of machine learning models. However, it im- portant to consider that this approach comes with the trade-off of increased costs in model design and training. Hence, the task of alleviating model memory through retrieval enhancement holds sig- nificant applications in various domains (Sharma et al., 2021: He et al., 2023; Chen et al.. 2022b), e.g.. commonsense generation (Wang et al., 2021; Fan et al., 2020), dialogue systems (Song et al., 2016; Zhang et al., 2022; Zhu et al., 2019; Chen et al., 2022c; Parvez et al., 2023), summarization (An et al., 2021), and fact verification (Thorne et al., 2018).
A.1 Related work
In this section, we briefly review two lines of related work, i.e. fact verification and retrieval- enhanced machine learning.
A.1.1 Fact Verification
The real-world fact verification (FV) task focuses on verifying the accuracy of claims made by hu- mans through the retrieval of evidential sentences from reliable corpora such as Wikipedia (Chen et al., 2022a). Current FV models typically adopt three-step pipeline framework, which includes document retrieval, evidence retrieval, and claim verification.
Retrieval-enhanced machine learning methods could be mainly divided into two categories: (i) The first one is retrieval-only: the retrieval model acts auxiliary tool provide query-re- lated responses to the prediction model. Note that the majority of current fact verification models pri- marily belong to this category. (ii) The second one is retrieval with feedback: the retrieval model is fur- ther trained based on the feedback provided by the prediction model. Very recently, Hu et al. (2023) proposed to utilize the loss value of the prediction model as signal for optimizing the retrieval model. However, this method may not be suitable for large datasets because it relies on all provided sentences to compute the loss. Additionally, without access to ground-truth -tunt evidence for the prediction model, ac- curately capturing the utility of retrieved sentences may be challenging.
In the document retrieval stage, existing works could be broadly categorized into three cate- gories: entity linking (Hanselowski et al., 2018; Chernyavskiy and Ilvovsky, 2019; Zhao et al.. 2020; Liu et al., 2020), keyword matching (Nie et al., 2019a; Ma et al., 2019; Nie et al., 2019b) and feature-based (Hidey and Diab, 2018; Yang al., 2017; Chen al., 2022a) methods.
In the evidence retrieval stage, existing FV works commonly utilize retrieval models that are built upon the probability ranking principle (PRP) (Robertson, 1977) in the field of information re- trieval (IR). These models aim to select the top- ranked sentences evidence based on relevance ranking (Jiang et al., 2021; Soleimani et al., 2020; Zhou al., 2019; Liu et al., 2020; Hanselowski et al., 2018; Wan et al., 2021). Very recently, Chen et al. (2022a); Liu et al. (2023) proposed to retrieve evidence in generative fashion by generating doc- ument titles and evidence sentence identifiers.
A.2 The training process of the fine-grained retriever
In this section, we provide a detailed description of training process of the fine-grained retriever, given the K candidate sentences from the coarse retriever, unfolds as follows:
In the claim verification stage, considerable at- tention has been given to this step in many existing FV works. These mainly include implication-basee inference methods that treat the verification task as an entailment task (Thorne et al., 2018; Nie et al., 2019a; Krishna et al., 2022: Nic et al., 2019b), as well as graph neural network- based methods (Liu
The fine- arei-gr retriever selects sentences from the candidate set.
The claim and K candidate sentences
[$1, [$1,82, S2, 8k] are concatenated with a special token [CLS] at the beginning of each sentence and then provided as input to the fine-grained retriever.
et al., 2020: Zhou et al.. 2019) that frame the veri- fication task as graph reasoning task. It is worth noting that the accuracy of this stage heavily re- lies on the retrieved evidence. Therefore, having precise set of evidence can significantly impact the verification outcome and lead improved results.
Subsequently, the hidden state of each [CLS] undergoes a transformation through a linear layer followed by a ReLU activation, yielding K sets of 2-dimensional vectors, for example, [[0.1, 0.9].
A.1.2 Retrieval-enhanced machine learning
[0.2, 0.8). ..., [0.7, 0.3]].
As mentioned in Zamani et al. (2022), one approach to improve accuracy is by increasing the parameter
Finally, Gumbel Softmax is applied to these K 2-dimensional classification vectors, producing a

---

sequence-length K vector like [1, 1, .... Oj. where indicates the predicted evidence sentence.
MediaWiki APP is utilized to search for page titles that correspond the identified entities. Lastly, the retrieval results are filtered using the entity cover- age limitation to select the appropriate documents. For coarse retrieval, we follow the approach described in (Liu et al., 2020), where we uti- lize the base version of the BERT model (Devlin et al., 2019) from Hugging Face* to implement our graaind retrieval model. We use the "[CLS]" hidden state to represent claim and sentence pairs. Training samples are constructed by using both ground-truth evidence and non non- con-ground-trute sen- tences from document retrieval. The pairwise train- ing method is employed to train the retrieval rank- ing model. The max length of the input is set to 130. We use the Adam optimizer with a learning rate of 5e-5 and warm-up proportion of 0.1. For fine-grained retrieval, contains the fine- grained retrieval model Rg and the verifier. Specif- ically, Rg concatenates the claim and the retrieved candidate sentences and feeds them into the BERT model. The maximum length is set to 512, and the sentences are separated using the special token "[CLS]" The hidden vector corresponding to the "ICLS]" token is then used for evidence classifica- tion. The learning rate for the AdamW (Loshchilov and Hutter. 2019) optimizer is set 2e-5. In the verifier model, the input consists of the concate- nated sequence the claim and the ground-truth evidence. The batch size is set to 5, and the accu- mulate step is also set to 5, The learning rate for the Adam W optimizer is set to 3e-5. The hyper- parameters of a and 33 are set to and 1, respec- tively.
The fine-grained retriever forwards the selected sentence information to the verifier.
The sequence-length K vector is adaptively ex- tended to match the lengths of the K candidate sentences, aligning with the respective sentence lengths. For instance, [len(s)) 1, (s2 1, "....,len(sk)) 0] represents the selected vec- tors, which serve as the input_mast for the BERT tokenizer. Here, len(s1) * signifies the repeti- tion of for the length of s1.
The selected vectors [[en(s)) 1, (32)
1, le.(s)) 0] from the retriever, along with the K discrete sentences [$1, .., sk), are pro- vided as input_mask and input_ids, respectively, to the verifier using the BERT tokenizer.
For the ground-truth evidence, the input_ids ids cor- responds to the true discrete evidence, while the input_mase are represented as the unit vector.
Subsequently, the utility divergence between the verifier's 's judgments based on ground-truth evi- dence and those derived from eetriever-provider sentences is computed, all in the context of pre- dicting the claim label.
The verifier pock-propsgatts the loss gradient to the fine-grained retriever.
Following the computation of the utility diver- gence, gradients are back-propagated to the selected vectors, e.g., [len(s;) * 1, (en(2)
1, len.., + 0].
As the Gumbel Softmax function is differentiable, gradients permeate through the selected vectors, extending back to the retriever.
Furthermore, all models are implemented using the Py PyTorch framework. For online evidence eval- uation, only the initial five sentences of predicted evidence provided by the candidate system are uti- lized for scoring. In order to adhere to the specifi- cations of the online evaluation, the baselines and our FER select the five sentences as the evidence.
A.3 Reproducibility
In this section, we introduce our experimental de- tails. The source code and trained models will be made publicly available upon publication to im- prove the reproducibility of results.
A.3.1 Experimental settings
A.3.2 Evaluation metric
For document retrieval, we adopt the entity link- ing approach (Hanselowski et al., 2018) to retrieve relevant documents, following the methodology of Hanselowski et al. (2018). On average, four docu- ments are retrieved for each claim. Following the traditional entity linking pipeline approach, there are three steps for document retrieval. First, the claims are parsed using AllenNLP (Gardner et al., 2018) to extract multiple entities. Secondly, the
For evaluation on the FEVER benchmark dataset we use the official evaluation metrics, i.... Precision (P), Recall (R), and F1 for evidence re- trieval: FEVER score and Label Accuracy (LA) for claim verification. Following the official evaluation (Thome et al., 2018), we compute P@5, R R@5 and
hnttps://ww..eedaawii.... wttp://www....././/////........
htttps://hugiingface
hhttps://fever. al/dataset/fever.ht

---

Fl @5. FEVER evaluates accuracy based on the condition that the predicted evidence fully covers the ground-truth evidence. LA assesses the accu- racy of the claim label prediction without taking into account the validity of the retrieved evidence.
Table 4: Performance of different claim verification
models the development set using evidence from the original paper vs. evidence retrieved by FER.
Model
LA
FEVER
BERT Concat (Zhou et al., 2019)
73.67
68.89
BERT Concat FER
76.31
69.90
A.3.3 Evidence retrieval baselines
BERT Pair (Zhou et al., 2019)
73.30
68.90
In our work, we consider several advanced evi- dence retrieval baselines.
BERT Pair + FER
76.34
69.35
GEAR (Zhou al., 2019)
74.84
70.69
GEAR + FER
76.24
72.17
TF-IDF (Thore et al., 2018) is traditional sparse retrieval model that combines bigram hashing and TF-IDF matching to return relevant documents.
GAT (Liu et al., 2020) GAT+FER
76.13
71.04
77.75
71.27
KGAT(BERT Basc) (Liu ct al., 2020) KGAT+FER
78.02
75.88
79.02
76.59
ColumbiaNLP (Chakrabarty et al., 2018) ini- tially utilizes TF-IDF to rank the candidate sen- tences and subsequently selects the top 5 sen- tences with the highest relevance. To mitigate the presence of noisy data, ELMo embeddings (Sarzynska- Wawer et al., 2021) are employed to convert the claims and sentences into vectors. Subsequently, the top-3 sentences with the high- est cosine similarity are extracted as the final retrieval results.
Table 5: Evidence retrieval performance with feedback from different verifiers on development set.
Verifier
P
R
F1
GAT KGAT
82.45
85.65
84.02
88.57
81.82
85,06
Deep Q-learning Network (DQN) (Wan et al., 2021) utilizes the RoBERTa for evidence repre- sentation and applies the deep Q-learning net- work to select precise evidence.
UKP-Athene (Hanselowski et al., 2018) intro- duces sentence ranking model utilizing the Enhanced Sequential Inference Model (ESIM)
GERE (Chen et al., 2022a) employs the new paradigm of generative retrieval generate the relevant evidence identifiers.
(Hanselowski al., 2018). This model takes a claim and sentence as input, and the predicted ranking score obtained by passing the last hid-
Stammbach (Stammbach, 2021) employs token- based single-document candidate sentence clas- sification to retrieve evidence, which uses RoBERTa (Liu et al., 2019) and BigBird (Za- heer et al., 2020) to encode candidate sentences. To ensure fair comparison, we adopt the results based on RoBERTa our baseline.
den layer of the ESIM through a single neuron. GEAR (Zhou et al., 2019) enhances the UKP- Athene model by introducing threshold. Sen- tences with relevance scores higher than the threshold (set 0.001) filtered and consid- ered as retrieval results.
Kernel Graph Attention Network (KGAT) (Liu et al., 2020) utilizes both ESIM and BERT (Devlin et al., 2019) to construct evidence re- trieval models, which are trained in a pairwise manner.
A.3.4 Veracity prediction
In our work, we also leverage several advanced claim verification models for verification based on the retrieved evidence by FER.
DREAM (Zhong et al., 2020) employs the con- textual representation models XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019) to assess the relevance of claim to each candidate evi- dence.
BERT-pair and BERT concat (Zhou al., 2019) consider claim-evidence pairs individually, or stitch all evidence and the claim together to predict the claim label.
GEAR (Zhou et al., 2019) considers the influ- ence between evidence using graphical atten- tion network and aggregates all evidence through the attention layer.
NSMN (Nic et al., 2019a) employs a direct traver- sal approach, where all sentences are compared with the claim to calculate relevance scores. Sen- tences with relevance scores higher than the threshold (set to 0.5) are considered evidence.
KGAT (Liu et al., 2020) introduces graph at- tention network measure the importance of evidence nodes and the propagation of evidence among them through node and edge kernels.
'nttps://codalab. lisn, www..cay.........mm..m... ff// competitions/7308

---

F1
Figure 3: Evidence retrieval performance with different K on the development set.
A.4 Experimental results
A.4.1 Performance on the development
As shown in Table 4, we report the performance of different claim verification models on the devel- opment set using the evidence retrieved by FER. As shown in Table 5, we report the evidence re- trieval performance of our method with feedback from different verifiers on the development set. As shown in Figure 3, we report the effect of the size K on the final evidence retrieval performance the development set.
A.4.2 Case study
Table 6 presents two illustrative examples from FEVER development set. We can observe that: (i) The sentences provided by the coarse retrieval method, even though they are ranked highly, do not always offer useful information for claim verifica- tion, specifically in terms of providing ground-truth evidence. For example, in the second instance, of the top-5 ranked sentences, only of them actually consist of ground-truth evidence. (ii) Through fine-graines retrieval, we can effec- tively identify ground-truth evidence that might have initially been ranked low, such as the 14-th sentence in the first instance and the 11tth sentence in the second instance. (iii) In the case of fine- grained retrieval, not all ground-truth evidence successfully identified. For example, in the second instance, the 8-th sentence not identified. How- ever, as observed in the given example, only the 8-th sentence is not retrieved, which does not signif- icantly affect the judgment of the claims. Nonethe- less, the issue of the low recall rate necessitates further resolution in future research.

---

Table 6: Two examples from the FEVER development set using our FER method. For a given claim, the coarse retrieval process returns the top-25 sentences. The ground-truth evidence highlighted in red. The final evidence identified through fine-grad ane-grained retrieval process is {1, 14), and (1, 2, 4, 5, 6, 7, 11), respectively.
Claim: The Love Club EP is the debut extended play of singer Ella Marija Lani Saiich-O'Conno
Ground-truth evidence:
The Love Club EP the debut extended play EP by New Zealand singer Lorde Ella Marija Lani Yelich O' O'Connor born 7 November 1996, better known by her stage name Lorde
Label: SUPPORT
Coarse Retrieval Process:
1. The Love Club EP is the dchut extended play EP by New Zealand singer Lorde.
2. The Love Club is song by New Zealand singer Lorde, taken from her debut extended play
3. An indie rock influenced electronica album. The Love Club EP ...
4. To promote The Love Club EP, Lorde performed during various concerts, and Royals was released ...
5. On 8 March 2013 the record was commercially released by Universal Music ...
6. Upon the release of the EP, the song was well received by music critics
14. Ella Marija Lani Yelich-O's Connor born November 1996
Claim: Pearl Jam is a type of dressing.
Ground-truth evidence:
Since inception, the band's line up has comprised Eddie Vedder lead
Stephen Thomas Erlewine of AllMusic referred to Pearl Jam the most popular American rock roll
The band's 's fifth member is drummer Matt Cameron also of Soundgarden. who been
Pearl Jam is an American rock band formed in Seattle, Washington, in 1990.
Pearl Jam has outlasted and outsold many of its contemporaries from the alternative rock breakthrough To date, the band has nearly 32million records in the United States and an
Label: REFUTES
Coarse Retrieval Process:
1. Pearl Jam is American rock band formed in Seattle, Washington, in 1990.
2. Stephen Thomas Erlewine of AllMusic referred to Pearl Jam the most popular ...
3. Pearl Jam sometimes referred
The Avocado Album or simply Avocado is the eighth studio ...
4. One of the key bands in the grunge movement of the carly 1990s, over the course of the hand's carcer, its
Formed after the demise of Gossard and Ament's previous band, Mother Love Bone, Pearl Jam broke into the
6. Pearl Jam has outlasted and outsold many of its contemporaries from the alternative rock breakthrough ...
7. Since its inception, the band's line up has comprised Eddie Vedder lead
8. The band's fifth member is drummer Matt Cameron also of Soundgarden, who has been
To date, the band has sold nearly 32million records in the United States and


Reviews:
[{"review_id": "zrkI7ChjsM", "reviewer": "Reviewer_tnjn", "summary": "", "questions": "1) Could you provide additional details on how the fine-grained retriever is optimized with feedback? It seems that the retriever predicts probabilities for sentences, which selected sentences are then used as input for the verification model. How is the gradient computed based on discrete sentence input?\n\n2) The paper mentions conducting an online evaluation with the top 5 sentences as the evidence set, but the results are not reported. Could you please share the results of this evaluation?\n\n3) The input of the verification model consists of retrieved sentences. Since the evidence set contains both ground-truth evidence and noise/unnecessary evidence, can the divergence signal force the fine-grained retriever to select the ground-truth evidence?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your invaluable feedback and insightful comments. Your time and dedication in evaluating our work are greatly appreciated. We would like to offer our response as follows.\n\n[Reject 1]: Thanks for pointing out this, and we will provide detailed descriptions in our revision. \n\n1. During training, for the fine-grained retriever, we take advantage of fine-tuning the pre-trained architecture (i.e., we use BERT weights) via the evidence classification loss L\\_{cla} and utility divergency L\\_{uti}. \n2. During inference, given a test claim, we leverage the optimized retriever to construct the final evidence set as follows.  \n   * Firstly, we obtain candidate sentences via the coarse retriever.  \n   * Then, we input the concatenated sequence of the test claim and all the candidate sentences, with a special token [CLS] added at the beginning of each sentence to indicate its presence, into the fine-grained retriever (Line 124-128).  \n   * Finally, we leverage the hidden state of each [CLS] to classify whether the corresponding candidate sentence serves as evidence or not. This classification is accomplished using linear layers and ReLU activation functions (Line 138-142). As a result, we are able to derive the final evidence set. \n   * This way has been used in previous fact checking [1,2] and other areas, such as extractive summarization [3].  \n3. For online evaluation, the FEVER online platform stipulates that the count of evidence must not exceed 5. Regarding the evidence extracted through the above process,  \n   - If the count surpasses 5, we will exclusively consider the top 5 evidence for evaluation.  \n   - Conversely, if the count falls below 5, we will directly incorporate them in their entirety.  \n   - To ensure a fair comparison, we uniformly employ 5 evidence as provided by baselines. \n   - This evaluation way has been leveraged by many previous works, such as our baseline method: DQN, GERE and Stammbach.  \n\n[Reject 2]: Thanks. This is indeed an essential aspect that we have taken into careful consideration.  \n\n1. The baselines we have employed can be broadly categorized into two distinct types:\n   - 5-sentence evidence baseline: TF-IDF, ColumbiaNLP, UKP-Athene, NSMN, KGAT and DREAM follow a process of ranking candidate sentences and subsequently utilize the top 5 ranked sentences as evidence. \n   - Precise evidence baseline: In the cases of DQN, GEAR, GERE and Stammbach, the authors also emphasize the significance of achieving dynamic predictions for relevant sentences that suit different claims, i.e., a precise set of relevant sentences. In this way, the final size of the retrieved evidence set is potentially equal to or less than 5, like our results.  \n2. It is worth noting that the FEVER dataset holds an average ground-truth evidence count of 1.8.\n   - For the 5-sentence evidence baselines, a direct selection of the top 5 sentences could introduce more noise, thereby potentially leading to reduced precision. \n   - In comparison with the 5-sentence evidence baselines, the precise evidence baselines as well as our method demonstrate an enhanced capability in accurately extracting evidence. Specifically, our approach surpasses GERE on the test set, resulting in a 46.1% increase in the precision score. \n3. Therefore, following precise evidence baselines, we believe our approach to comparison is fair. We will include the related discussions in our revision.  \n\n[Question-1] Thanks for pointing out this and we apologize for any confusion caused. To clarify, the sentences selected by the retriever are utilized as input to the verification model in vector form, rather than discrete sentences.\n\nThe training process of the fine-grained retriever, given the K candidate sentences from the coarse retriever, unfolds as follows:\n\n1. The fine-grained retriever selects sentences from the candidate set.  \n   - The claim and K candidate sentences [s\\_1, s\\_2, …, s\\_K] are concatenated with a special token [CLS] at the beginning of each sentence and then provided as input to the fine-grained retriever.  \n   - Subsequently, the hidden state of each [CLS] undergoes a transformation through a linear layer followed by a ReLU activation, yielding K sets of 2-dimensional vectors, for example, [[0.1, 0.9], [0.2, 0.8], ..., [0.7, 0.3]].  \n   - Finally, Gumbel Softmax is applied to these K 2-dimensional classification vectors, producing a sequence-length K vector like [1, 1, ..., 0], where 1 indicates the predicted evidence sentence.  \n2. The fine-grained retriever forwards the selected sentence information to the verifier.  \n   - The sequence-length K vector is adaptively extended to match the lengths of the K candidate sentences, aligning with the respective sentence lengths. For instance, [len(s\\_1)\\*1, len(s\\_2)\\*1, …, len(s\\_K)\\*0] represents the selected vectors, which serve as the input\\_mask for the BERT tokenizer. Here, len(s\\_1)\\*1 signifies the repetition of 1 for the length of s\\_1.  \n   - The selected vectors [len(s\\_1)\\*1, len(s\\_2)\\*1, …, len(s\\_K)\\*0] from the retriever, along with the K discrete sentences [s\\_1, s\\_2, …, s\\_K], are provided as input\\_mask and input\\_ids, respectively, to the verifier using the BERT tokenizer.  \n   - For the ground-truth evidence, the input\\_ids corresponds to the true discrete evidence, while the input\\_mask are represented as the unit vector.  \n   - Subsequently, the utility divergence between the verifier's judgments based on ground-truth evidence and those derived from retriever-provided sentences is computed, all in the context of predicting the claim label. \n3. The verifier back-propagates the loss gradient to the fine-grained retriever.  \n   - Following the computation of the utility divergence, gradients are back-propagated to the selected vectors, e.g., [len(s\\_1)\\*1, len(s\\_2)\\*1, …, len(s\\_K)\\*0].  \n   - As the Gumbel Softmax function is differentiable, gradients permeate through the selected vectors, extending back to the retriever.  \n4. We will incorporate the implementation specifics in our revision and make the source code publicly available upon publication. Thanks again for pointing out this.  \n\n[Question-2] Similar to the response to [Reject 1] and [Reject 2], the results in Table 1 are derived from the top 5 sentences forming the evidence set. The precise evidence baselines, including DQN, GEAR, GERE, and Stammbach, utilize the same comparative approach. We will ensure to clarify this configuration in our revision.\n\n[Question-3] Thanks for your question. \n\n1. The ground-truth evidence set and the retrieved sentences from the fine-grained retriever, are input separately into the verifier, generating two distinct probability distributions for feedback. Although the retrieved sentences may include some ground-truth evidence and noise evidence, the divergence signal can effectively guide the fine-grained retriever to focus solely on predicting the ground-truth evidence set.  \n2. Such an approach, founded on divergence between ground-truth and predicted information, finds its application in various domains, including instances like face recognition [4] and authorship attribution [5].   \n3. In Table 1, FER w/o L\\_cla (only with utility divergence loss) outperforms GERE by 19% in terms of F1 on the test set. This underscores the pivotal role that feedback from the verifier plays in enhancing the retriever's ability to identify precise evidence.  \n4. We will include this discussion in our revision.  \n\nReference:\n\n[1] David Wadden, Kyle Lo, Lucy Lu Wang, Arman Cohan, Iz Beltagy, and Hannaneh Hajishirzi. 2022. MultiVerS: Improving scientific claim verification with weak supervision and full-document context. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 61–76, Seattle, United States. Association for Computational Linguistics.\n\n[2] Hu, X., Hong, Z., Guo, Z., Wen, L., & Yu, P. (2023, July). Read it Twice: Towards Faithfully Interpretable Fact Verification by Revisiting Evidence. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 2319-2323).\n\n[3] Yang Liu and Mirella Lapata. 2019. Text Summarization with Pretrained Encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730–3740, Hong Kong, China. Association for Computational Linguistics.\n\n[4] Wu, H., Xu, Z., Zhang, J., Yan, W., & Ma, X. (2017, October). Face recognition based on convolution siamese networks. In 2017 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI) (pp. 1-5). IEEE.\n\n[5] Fabien, M., Villatoro-Tello, E., Motlicek, P., & Parida, S. (2020, December). BertAA: BERT fine-tuning for Authorship Attribution. In Proceedings of the 17th International Conference on Natural Language Processing (ICON) (pp. 127-137).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) This work proposes a novel method to align evidence retrieval with verifier based on divergence of verdict prediction result of  ground-evidence and retrieved evidence as feedback signal.\n\n2) The method is simple and effective. Experiments on FEVER show that FER improves the F1 score of evidence retrieval by a large margin.", "reasons_to_reject": "1) The explanation of fine-grained retrieval is unclear and could be more concise. The process of constructing the final evidence set is not detailed enough, only mentioning online evaluation in lines 715-716 (Top5 sentences).\n\n2) Lines 222-224 suggest that FER aims for more precise and concise evidence sets, potentially containing less than 5 sentences, resulting in higher precision and lower recall. The comparison with baselines using top 5 sentences for F1 score computation may not be completely fair.", "questions_to_authors": "1) Could you provide additional details on how the fine-grained retriever is optimized with feedback? It seems that the retriever predicts probabilities for sentences, which selected sentences are then used as input for the verification model. How is the gradient computed based on discrete sentence input?\n\n2) The paper mentions conducting an online evaluation with the top 5 sentences as the evidence set, but the results are not reported. Could you please share the results of this evaluation?\n\n3) The input of the verification model consists of retrieved sentences. Since the evidence set contains both ground-truth evidence and noise/unnecessary evidence, can the divergence signal force the fine-grained retriever to select the ground-truth evidence?", "paper_topic_and_main_contributions": "This paper focuses on evidence retrieval in fact verification tasks and argues that the sentences retrieved by existing methods may not align well with a verification model based on the retrieved evidence. The proposed method involves two phases: 1) coarse retrieval, which retrieves a smaller candidate set, and 2) fine-grained retrieval, which selects evidence for verification.\n\nSpecifically, the fine-grained evidence retriever is trained using a combination of classification loss and divergence utility. Its goal is to align the retrieval model with the verification model by incorporating the verdict results from the verifier, both from ground-truth evidence and from retrieved evidence.\n\nExperiments conducted on FEVER demonstrate that the proposed method significantly improves the F1 performance of evidence retrieval.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EsnaZJhc3B", "reviewer": "Reviewer_uHs1", "summary": "", "questions": "- I got confused when comparing the numbers in Table 1 and Table 2. In Table 1, FER outperforms all other baselines by a large margin. For instance, FER achieves 78.84 F1 score on the test set of FEVER while KGAT (Liu et al., 2020) only achieves 39.14 F1 score. However, in Table 2, the improvement on downstream performance (FEVER score) over KGAT is quite marginal: KGAT (69.40) vs. KGAT+FER (69.61). Do you have an explanation for this?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your invaluable feedback and insightful comments. Your time and dedication in evaluating our work are greatly appreciated. We would like to offer our response as follows.\n\n[Reject-1] Yes, many studies have delved into leveraging the loss of the downstream task to enhance the intermediary component. These works primarily differ in two crucial aspects: (1) the design of feedback provided by the downstream task, and (2) the utilization of this feedback in the intermediate component.\n\n1. For feedback design: Previous studies, e.g., Choi et al. (2016) and Lewis et al. (2020), directly leveraged downstream task performance using retrieval results as feedback. Nonetheless, this strategy heavily depends on the prediction accuracy of downstream task models. In contrast, we evaluate the divergence in utility between verifier assessments grounded in ground-truth evidence and those derived from retrieval outcomes. This approach mitigates the extent of reliance on the downstream task models' precision.\n2. For feedback use: In prior studies, such as Choi et al. (2016) and Lewis et al. (2020), employed reinforcement learning techniques for parameter updates. In contrast, we utilize utility divergence as the retriever's training loss directly. While our baseline model, DQN, used reinforcement learning, our approach significantly outperforms it, achieving a 26.0% higher F1 score on the test set. The reason may be that reinforcement learning involves intricate reward function design and optimal policy formulation, which can result in unintended agent behaviors if not done well. Our direct optimization approach could eliminate the need for intricate reward design and policy tuning. \n3. We contribute two major improvements as described above, rather than merely instantiating existing techniques on FEVER.\n4. In the future works, we will explore alternative ways for feedback design and model training. Besides, we will incorporate a discussion on these two aspects within our revision.\n\n[Reject-2] We are sorry for any confusion caused. To clarify, the retriever employs a soft selection approach for sentences, as opposed to a hard one. In other words, the retriever's chosen sentences are used as vector-based input for the verification model, rather than as discrete sentences.\n\nThe training process of the fine-grained retriever, given the K candidate sentences from the coarse retriever, unfolds as follows:\n\n1. The fine-grained retriever selects sentences from the candidate set.  \n   - The claim and K candidate sentences [s\\_1, s\\_2, …, s\\_K] are concatenated with a special token [CLS] at the beginning of each sentence, and then provided as input to the fine-grained retriever.\n   - Subsequently, the hidden state of each [CLS] undergoes a transformation through a linear layer followed by a ReLU activation, yielding K sets of 2-dimensional vectors, for example, [[0.1, 0.9], [0.2, 0.8], ..., [0.7, 0.3]].\n   - Finally, Gumbel Softmax is applied to these K 2-dimensional classification vectors, producing a sequence-length K vector like [1, 1, ..., 0], where 1 indicates the predicted evidence sentence.\n2. The fine-grained retriever forwards the selected sentence information to the verifier.\n   - The sequence-length K vector is adaptively extended to match the lengths of the K candidate sentences, aligning with the respective sentence lengths. For instance, [len(s\\_1)\\*1, len(s\\_2)\\*1, …, len(s\\_K)\\*0] represents the selected vectors, which serve as the input\\_mask for the BERT tokenizer. Here, len(s\\_1)\\*1 signifies the repetition of 1 for the length of s\\_1. \n   - The selected vectors [len(s\\_1)\\*1, len(s\\_2)\\*1, …, len(s\\_K)\\*0] from the retriever, along with the K discrete sentences [s\\_1, s\\_2, …, s\\_K], are provided as input\\_mask and input\\_ids, respectively, to the verifier using the BERT tokenizer.\n   - For the ground-truth evidence, the input\\_ids corresponds to the true discrete evidence, while the input\\_mask are represented as the unit vector.\n   - Subsequently, the utility divergence between the verifier's judgments based on ground-truth evidence and those derived from retriever-provided sentences is computed, all in the context of predicting the claim label.\n3. The verifier back-propagates the loss gradient to the fine-grained retriever.\n   - Following the computation of the utility divergence, gradients are back-propagated to the selected vectors, e.g., [len(s\\_1)\\*1, len(s\\_2)\\*1, …, len(s\\_K)\\*0].\n   - As the Gumbel Softmax function is differentiable, gradients permeate through the selected vectors, extending back to the retriever.\n   \n4. We will include detailed implementation information in our revision and share the source code publicly after publication. For an explanation of the results in Table 1 and Table 2, please refer to the response to [Question-1].\n\n[Question-1] We are sorry for causing any confusion about the experimental results on your part. We further clarify the difference between Table 1 and Table 2 as follows:  \n\n1. Table 1 shows the evidence retrieval results on both the dev set and the online test set, which holds our primary focus in this study.\n   - We have two types of baselines: (1) 5-sentence evidence baseline: For TF-IDF, ColumbiaNLP, UKP-Athene, NSMN, KGAT and DREAM, they directly select the top 5 sentences as evidence for evaluation; (2) precise evidence baseline: For DQN, GEAR, GERE and Stammbach, they also emphasize the significance of achieving dynamic predictions for relevant sentences that suit different claims, i.e., a precise set of relevant sentences. This way often results in a predicted count that is potentially equal to or less than 5, like our results. \n   - In general, precise evidence baselines and our method outperform 5-sentence evidence baselines. Our approach outperforms GERE on the test set, resulting in a 23.7% increase in the F1 score.\n2. Table 2 shows the claim verification results. We choose several advanced (off-the-shelf) claim verification models, provided with the retrieved evidence obtained by our method and baselines respectively, for further comparison. \n   - For these claim verification models, they have separate processes for their retrievers and verification models. Firstly, they leverage traditional retrieval methods, such as BM25, BERT based sentence retrieval and other retrieval methods based on relevance, to obtain relevant sentences; Subsequently, they usually train the verification models using the ground truth data with added noise, rather than using the retriever's retrieval results. \n   - Hence, some verification models’s structure designed for noisy training data might not be very sensitive to retrieved evidence, even though the retrieval performance we've achieved has seen significant improvement (Table 1). For example, KGAT employs a graph attention network to explicitly model the mechanism of importance between evidence with respect to claims. This approach could potentially be robust to noisy evidence on claim judgments. Therefore, the difference between relying on evidence retrieved via previous methods and our approach might not be considerably large.  \n   - The FEVER score's definition varies from the LA score. While the LA score assesses claim verification accuracy independently of retrieved evidence validity, different degrees of LA metric improvement are noticeable. The FEVER metric requires both accurate claim classification and inclusion of all ground-truth evidence. Thus, cases with modest LA score improvements tend to result in constrained improvements in FEVER scores.\n3. Similar findings are also present in other studies on evidence retrieval [1], indicating large improvements on evidence retrieval and marginal improvements on the downstream claim verification given by the retrieved evidence. \n4. Currently, our focus is solely on evidence retrieval, with separate optimization of retrieval and verification models. We intend to investigate methods for jointly optimizing both models within a single objective. In this manner, we expect to see improved FEVER and LA scores. \n5. Thank you for this insightful question. We will polish the related content in our revision. \n\nReference:\n\n[1] Chen, J., Zhang, R., Guo, J., Fan, Y., & Cheng, X. (2022, July). GERE: Generative evidence retrieval for fact verification. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 2184-2189).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper utilized the idea of using downstream task loss to improve the intermediate component, here, the fine-grained retriever, and achieved strong retrieval results on FEVER dataset.", "reasons_to_reject": "- One of the main weakness of this paper is the lack of excitement. The idea of using downstream task loss to improve the intermediate component is not new and has been explored by many prior work. For example, Choi et al. 2016 used QA loss as indirect supervision and using RL to improve the sentence retriever; Lewis et al., 2020 used the two different decoder loss to jointly train the retriever and the decoder. This work is an instantiation of such techniques on FEVER. \n- I got confused by the equation at line 165. Here, if I understand it correctly, $R_\\theta(c, S)$ represents the fine-grained retriever and it returns a \"hard\" sentences selection from $S$. If it's a \"hard\" selection, how is the loss gradient passed to the fine-grained retriever? I also got confused by the results in Table 1 and Table 2. See the following question section for details.\n\nReference:\n- Choi, Eunsol, et al. \"Coarse-to-fine question answering for long documents.\" Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017.\n- Lewis, Patrick, et al. \"Retrieval-augmented generation for knowledge-intensive nlp tasks.\" Advances in Neural Information Processing Systems 33 (2020): 9459-9474.", "questions_to_authors": "- I got confused when comparing the numbers in Table 1 and Table 2. In Table 1, FER outperforms all other baselines by a large margin. For instance, FER achieves 78.84 F1 score on the test set of FEVER while KGAT (Liu et al., 2020) only achieves 39.14 F1 score. However, in Table 2, the improvement on downstream performance (FEVER score) over KGAT is quite marginal: KGAT (69.40) vs. KGAT+FER (69.61). Do you have an explanation for this?", "paper_topic_and_main_contributions": "This paper proposed the feedback-based evidence retriever (FER) that optimizes the evidence retrieval process by incorporating feedback from the claim verifier. The retriever is trained using both the direct supervision from the gold evidence and the indirect supervision from the final task loss. Experimental results on FEVER dataset (Thorne et al., 2018) demonstrated a 23.7% F1 performance gain over the SOTA baseline in terms of evidence sentence retrieval.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CsyflocoFp", "reviewer": "Reviewer_MWUn", "summary": "", "questions": "Question A: Do you use a fixed candidate set size of 25 or is it the higher bound?\nQuestion B: How different is you method with a simple method of changing a Threshold T to modify the classification decision? I was wondering if by adding a second term to the loss function you just balance something like a threshold. My clue to think about this possibility is your results in Table 1: We see that FER impose a balance between recall and precision between FER w/o L_cla and FER w/o L_uti.\nQuestion C: You first use a coarse model and then train an extra fine-grained retrieval model. This clearly result in improving precision and decreasing recall (Table 1). How do you prove the final improvement comes from your extra loss and not just an addition point-wise model on top of your pair-wise results? Do you have the result of the pure coarse model too?\nQuestion D:  L_cls is a point-wise loss function, why do you describe your ranking model as pair-wise in line 117? Do you mean your coarse retrieval is a pair-wise model and the fine-grained one is point-wise? Maybe you should clarify this confusion. ", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your invaluable feedback and insightful comments. Your time and dedication in evaluating our work are greatly appreciated. We would like to offer our response as follows.\n\n[Reject-1] Thanks for pointing out this. It is an important issue we should consider. Similar to the reply to the [Question-C].\n\n[Question-A] The candidate set size is fixed and set to 25, which is tuned on the validation set. In Figure 2 and 3, we also plot the effect of the size of candidate set (5, 10, 15, 20, 25 and 30) on the final evidence retrieval performance. We find that as the size is larger than 25, more noisy/unnecessary information is provided to the fine-grained retriever, potentially hurting precision.  \n\n[Question-B] Thanks, this is an interesting perspective on the proposed loss. \n\n1. A simple method of changing a Threshold has been used in existing works. For instance, the baseline model GEAR initially retrieves sentences with the highest 5 relevance scores. Subsequently, GEAR applies a relevance score filter using a designated threshold, which eliminates sentences with scores below that threshold. Consequently, the retrieved evidence set is equal to or less than 5. However, GEAR's performance in our paper's Table 1 and in the original GEAR paper's Table 4 is not particularly promising, even across various thresholds.\n2. The hard threshold approach is heuristic, relying solely on relevance and disregarding downstream verification tasks, which results in a lack of feedback from them. In contrast, our method provides quantity flexibility and directly guides retrieval based on utility information derived from downstream verification tasks.\n3. Your discovery on the balance between FER w/o L\\_cla and FER w/o L\\_uti is an interesting and profound issue. We think it might that different loss functions might emphasize distinct aspects. Traditional approach of using hard thresholding to filter relevance lacks flexibility. The L\\_{cla} loss and L\\_{uti} loss act like selecting individual appropriate soft thresholds for each independent claim, thus achieving an overall performance improvement. \n4. Thank you again for this intriguing question and welcome further discussions. We will include the discussions in the revision.\n\n[Question-C] Thanks for your valuable comments.\n\n1. For the coarse model used in our work, we follow the retrieval model in KGAT (Line 110). Consequently, the outcome of the pure coarse model aligns with KGAT's results in Table 1. Specifically, the pure coarse model (i.e., KGAT) exhibits rather low precision and F1 scores. Our approach achieves a remarkable 101.4% improvement over KGAT's F1 score on the test set.\n2. In our work, we input the concatenated sequence of the claim and all the candidate sentences given by the coarse retriever, with a special token [CLS] added at the beginning of each sentence to indicate its presence, into the fine-grained retriever (Line 124-128). Then, we leverage the hidden state of each [CLS] to classify whether the corresponding candidate sentence serves as evidence or not. Therefore, our fine-grained retriever is not merely a point-wise training approach. The interaction information between all candidate sentences might also aid in classifying each sentence, which has been used in previous works [1, 2, 3]. \n3. The coarse retriever and fine-grained retriever are kept independent. As you suggested, if we only add the model of fine-grained retriever on top of the pair-wise results, it can be generally regarded as the variance of our model, i.e., FER w/o L\\_uti (only with classification loss). As shown in Table 1, FER w/o L\\_uti outperforms baselines, demonstrating the superiority of the addition model training strategy. However, FER w/o L\\_uti underperforms FER w/o L\\_cla and the full version FER in terms of precision, showing the importance of the novel utility divergency. Therefore, we think the performance improvement stems from changes in the training strategies as well as the introduction of new loss functions.\n4. Thank you again for this insightful question. We will include the discussions in the revision. Should you still have any remaining concerns or queries regarding this matter, please don't hesitate to continue the discussion. Your input is highly valued.\n\n[Question-D] We are sorry for causing any confusion on your part. We further clarify the difference between the coarse retriever and the fine-grained retriever:\n\n1. Yes, our coarse retriever is a pair-wise model. However, the fine-grained retriever is not merely point-wise, which is similar to the reply to the [Question-C]. \n2. The parameters of the coarse-retrieval and the fine-grained retrieval are not shared; they are kept separate and independent.\n   - The development of the coarse retriever involves pair-wise training, which remains grounded in relevance.\n   - Building upon the sentences identified by the coarse retriever, the fine-grained retriever focuses on extracting evidence that offers significant utility to the verifier.\n3. Thanks again for your suggestions. We will provide clearer clarification of this differentiation in the revision. \n\nReference:\n\n[1] Stammbach, D. (2021). Evidence selection as a token-level prediction task. In Proceedings of the Fourth Workshop on Fact Extraction and VERification (FEVER) (pp. 14-20). Association for Computational Linguistics.\n\n[2] David Wadden, Kyle Lo, Lucy Lu Wang, Arman Cohan, Iz Beltagy, and Hannaneh Hajishirzi. 2022. MultiVerS: Improving scientific claim verification with weak supervision and full-document context. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 61–76, Seattle, United States. Association for Computational Linguistics.\n\n[3] Hu, X., Hong, Z., Guo, Z., Wen, L., & Yu, P. (2023, July). Read it Twice: Towards Faithfully Interpretable Fact Verification by Revisiting Evidence. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 2319-2323).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Improving the evidence retrieval and consequently claim verification performance using their proposed new loss function. ", "reasons_to_reject": "It is not clear if the improvement in results stem from an additional point-wise model which is trained over the results of a coarse retrieval model or it is because of the proposed loss function.", "questions_to_authors": "Question A: Do you use a fixed candidate set size of 25 or is it the higher bound?\nQuestion B: How different is you method with a simple method of changing a Threshold T to modify the classification decision? I was wondering if by adding a second term to the loss function you just balance something like a threshold. My clue to think about this possibility is your results in Table 1: We see that FER impose a balance between recall and precision between FER w/o L_cla and FER w/o L_uti.\nQuestion C: You first use a coarse model and then train an extra fine-grained retrieval model. This clearly result in improving precision and decreasing recall (Table 1). How do you prove the final improvement comes from your extra loss and not just an addition point-wise model on top of your pair-wise results? Do you have the result of the pure coarse model too?\nQuestion D:  L_cls is a point-wise loss function, why do you describe your ranking model as pair-wise in line 117? Do you mean your coarse retrieval is a pair-wise model and the fine-grained one is point-wise? Maybe you should clarify this confusion. ", "paper_topic_and_main_contributions": "This paper focus on evidence retrieval for claim verification. They propose to use an additional loss function when training a retrieval model. This additional term comes from the claim verification model. They report improvement over retrieval and verification experiments. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to optimize the evidence retrieval process by utilizing feedback from the claim verifier. The retriever is trained on both ground-truth evidence and retrieved evidence. Fact verification is an important task and the proposed approach is intuitive and effective. The experiments showed good improvement on FEVER. Reviewers raised issues about unclear presentation in some parts of the paper, missing references, and weak ablation studies. One reviewer had a concern about lack of excitement. Overall, I think this paper presents a solid approach to an important task with some convincing results, especially considering it is a short paper. I hope the reviewers’ comments on the weaknesses will help improve the quality of the next version of the paper.
Decision: Accept-Findings