Unsupervised Pretraining for Fact Verification by Language Model Distillation
Authors: Adrián Bazaga
Pietro Lio
Gos Micklem
Venue: ICLR 2024 poster (2024)
Keywords: Unsupervised Learning
Self-supervised Learning
Deep Features
Contrastive Learning
Large Language Models
Knowledge Distillation
Multimodality
Fact Verification
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose a new framework for performing unsupervised pretraining for fact verification by distilling large language models knowledge for claim-evidence matching

Abstract: Fact verification aims to verify a claim using evidence from a trustworthy knowledge base. To address this challenge, algorithms must produce features for every claim that are both semantically meaningful, and compact enough to find a semantic alignment with the source information. In contrast to previous work, which tackled the alignment problem by learning over annotated corpora of claims and their corresponding labels, we propose SFAVEL ($\underline{S}$elf-supervised $\underline{Fa}$ct $\underline{Ve}$rification via $\underline{L}$anguage Model Distillation), a novel unsupervised pretraining framework that leverages pre-trained language models to distil self-supervised features into high-quality claim-fact alignments without the need for annotations. This is enabled by a novel contrastive loss function that encourages features to attain high-quality claim and evidence alignments whilst preserving the semantic relationships across the corpora. Notably, we present results that achieve a new state-of-the-art on FB15k-237 (+5.3\% Hits@1) and FEVER (+8\% accuracy) with linear evaluation.

Published as a conference paper at ICLR 2024
UNSUPERVISED PRETRAINING FOR FACT VERIFICA- TION BY LANGUAGE MODEL DISTILLATION
Adrián Bazaga, Pietro Liò & Gos Micklem University of Cambridge [ar989,p12) gm2 63 cam. ac uk
ABSTRACT
Fact verification aims to verify claim using evidence from a trustworthy knowl- edge base. To address this challenge, algorithms must produce features for every claim that are both semantically meaningful, and compact enough to find a semantic alignment with the source information. In contrast to previous work, which tackled the alignment problem by learning over annotated corpora of claims and their corresponding labels, we propose SFAVEL (Self-supervised Fact Verification via Language Model Distillation), a novel unsupervised pretraining framework that leverages pre-trained language models to distil self-supervised features into high- quality claim-fase -fct alignments without the need for annotations. This enabled by novel contrastive loss function that encourages features attain high-quality claim and evidence alignments whilst preserving the semantic relationships across the corpora. Notably, we present results that achieve a new state-of-the-an on FB 15k-237 (+5.3% Hits@ 1) and FEVER (+8% accuracy) with linear evaluation.
INTRODUCTION
In recent years, the issue of automated fact verification has gained considerable attention as the volume of potentially misleading and false claims rises (Guo al. 2022). resulting in the development of fully automated methods for fact checking (see Thore et al. (2018): Zubiaga et al. (2018): ; Guo al. (2022): Vladika & Matthes (2023): Das al. (2023) for recent surveys). Pioneering research in the field of Natural Language Processing (NLP) has led to the emergence of (large) language models (LMs) (e.g. Raffel al, (2020b); Brown et al. 2020); Radford al. (2019) 2018)). These models have been successful in many applications due to the vast implicit knowledge contained within them, and their strong capabilities for semantic understanding of language. However, issues around fact hallucination have gained considerable attention (Huang al., 2023 Liu et al. 2023) and are major concern in the widespread usage M---aased applications across different settings.
As the world becomes more aware of the issues around information trustworthiness, the importance of developing robust fact verification techniques grows ever more critical. Historically, the design of fact verification methods has been enabled by the creation of annotated datasets, such FEVER (Thorne) jt al. 2018) or MultiFC (Augenstein et al. 2019), of appropriate scale, quality, and complexity in order develop and evaluate models for fact checking. Most recent methods for this task have been dominated by two approaches: natural language inference (NLI) models (e.g.. Si et al. (2021)
Zhu et al. (202T): [Thorne et al. (2018) ; Luken et (2018); Yin & Roth (2188); Ye et (2020)), and knowledge graph- auaph-tugmented methods (e.g. Zhou et al. (2019); 2 et al. 20 Chen et al. (2021a by:
al. (202T)). These proposals mainly leverage the NLİ methods to model the semantic
relationship between claim and evidence, or further make use of the knowledge graph (KG) structure to capture the features underlying between multiple pieces of evidence.
However, these studies have largely relied annotated data for model training, and while gathering data is often not difficult, its labeling or annotation always consuming and costly. Thus, an emerging trend in the literature (Chen et al.) 2020a Caron al. 2020 He et al. 2020 is to move away from unnotation-depeedaam methods and try to learn patterns in the data using unsupervised training methods. With the advent of unsupervised training methods, new avenues have opened for research into leveraging the huge amounts of unlabeled data to achieve better performance more efficiently. Despite significant advancements in the field of unsupervised learning, only handful

---

Published as a conference paper at ICLR 2024
of strategies have been proposed for textual fact verification (e.g. Jobanputra (2019); Kim & Choi (2020); :ooly et al. (2022); Zeng & Gao (2023)) Thus there are still opportunities for the development of unsupervised techniques tailored specifically for such tasks.
Following recent trends in unsupervised methods, we eliminate data annotation requirements and instead, without human supervision, automatically try to identify relevant evidence for fact-checking. Thus, in this paper we present SFAVEL (Self-supervised Fact Verification via Language Model Distillation), which introduces novel self-supervised feature representation learning strategy with well-designed sub-tasks for automatic claim-fact matching. SFAVEL leverages pre-trained features from Language Models and focus on distilling them into compact and discrete structures that attain a high alignment between the textual claims to be verified, and their corresponding evidence in the knowledge graph. In partieular, our contributions are summarized as follows:
We introduce Self-supervised Fact Verification via Language Model Distillation (SFAVEL), novel unsupervised pretraining method tailored for fact verification on textual claims and knowledge graph-based evidence by language model distillation.
We demonstrate that SFAVEL achieves state of the art performance on the FEVER fact verification challenge and the FB15k-237 dataset when compared to both previous supervised and unsupervised approaches.
We justify SFAVEL's design decisions with ablation studies on the main architectural components.
2 RELATED WORK
Fact verification with pre-trained language models Most recent works typically divide the fact verification task into two stages. The first stage retrieves a relatively small subset of evidence from a knowledge source (e.g. knowledge graph) that is relevant to verify given claim. The second stage performs reasoning over the retrieved evidence to discern the veracity of the claim. Such retrieval- and-reasoning approaches aim to reduce the search space, and have proven their superiority over directly reasoning on the whole knowledge graph (Chen et al. 2019 Saxena et al.) 2020). In order to match evidence with claims, typical approach is to devise claim-face similarities using semantic matching with neural networks. Due to the great semantic understanding recently demonstrated by pre-trat trained language models (PLMs), some recent works employ PLMs for addressing the claim-face fact semantic matching task. In this vein, some works exploit the implicit knowledge stored within LMs for performing zero-shot fact checking, without any external knowledge [Lee et al. 2020: Yu et al. 2023). However, such methods are prone to suffer from hallucination incorrect predictions. Other work, such as ReAct (Yao et al., 2022). explores the use of LMs to generate both reasoning traces and task-specific actions over a knowledge base by in-context learning via prompting, overcoming prevalent hallucination issues by interacting with a Wikipedia API. However such an approach is limited by input length making it impractical in complex tasks. SFAVEL, we distill the features of recent pre-trained language models to yield highly-coreelated claim and evidence embeddings. We make use of set of eight language models as backbones because of their quality, but note that SFAVEL can work with any language model features.
explicit evidence retrieval
errors, resulting
Unsupervised pre-training methods for fact verification Learning meaningful features for claim- fact matching without human labels is nascent research direction in fact verification approaches, with recent works relying on self-supervised techniques. For instance, CosG (Chen et al. 2021dh proposes a graph contrastive learning approach to learn distinctive representations for semantically similar claims with differing labels, with the goal of mitigating the over-smoothing issues commonly found in graph-based based approaches. The model incorporates both unsupervised and supervised contrastive learning tasks to train a graph convolutional encoder, enhancing the representation of claim-fact pairs in the embedding space. Mu et al. (2023) presents SSDL, a multi-task learning strategy that initially builds student classifier using both self-supervised and semi-supervised methods, and then fine-tunes the classifier using distilled guidance from a larger teacher network that remains frozen during training. However, this method requires pairs of text claims and corresponding visual information as training data. Chen et al. (202cc) introduces KEGA, a knowledge- knowledge-enhancee eeddge--nhnnedd graph attention network for fact verification, which uses external knowledge bases to improve claim and
2

---

Published as a conference paper at ICLR 2024
evidence representations. It uses contrastive learning loss to capture graph structure features. With BERT as its backbone, the model includes knowledge from WordNet and pre-traines knowledge base embeddings to enrich token representations, while graph attention network (Veličković et al. 2018) and the contrastive loss further enhance the model's ability to reason. LaPraDoR (Xu et al., 2022) introduces pre-trained dense retriever approach with contrastive learning for unsupervised training of query and document encoders. The method is applied in a variety of text retrieval challenges, with FEVER being one of them. However, it shows a significant performance gap when compared against the supervised state-of-the approaches for FEVER. One of the main reasons is the lack of task- specific contrastive functions. In contrast to these works, SFAVEL is designed with task taec-specific self-supervised feature representation strategy, leveraging language model distillation to achieve high-quality claim-fact unsupervised matching on large scale datasets for fact verification.
Knowledge distillation Knowledge distillation seeks to transfer the knowledge from (usually large) model, called the teacher, to another (usually small) model, called the student. This technique is often used for increasing the performance of the small model. One of the first approaches for knowledge distillation was proposed by |iintnn et al. (2015). via minimizing the KL-divergence between the teacher and student's logits, using the predicted class probabilities from the teacher as soft labels to guide the student model. Instead of imitating the teacher's logits, Romero et al. (2013) distilled knowledge by minimizing the L2 distance between the intermediate outputs of the student and teacher. Park et al. (2019) aligned the pair-wise similarity graph of the student with the teacher, and Zagoruyko & Komodakis (2017) used the attention map generated by the teacher to force the student to attend to the same areas as the teacher. More recently, knowledge distillation has been extended for sule-supereieed settings. For instance, (Tian et al.] ( (2022) use the contrastive loss to enforce cross-modality consistency. Xu et al. (2020) and Fang et al. (202T) aim at aligning the features between views of the same instances by computing pair-wise similarities between the student's 's outputs and features kept in a feature memory bank produced by the teacher. In this work, we propose to transfer the semantic knowledge of a pre-trained language model within a new type of self supervised task, fact verification, by leveraging the language model capabilities of language understanding to guide the student to produce high-quality features for claim-fact matching.
OVERVIEW OF THE APPROACH
In this section we present our proposed unsupervised approach, SFAVEL, in detail as illustrated in Figure Da First, we begin with the data processing pipeline. Then, we detail our proposed pre-training methodology, followed by details of the different components our proposed contrastive loss function. Finally, we describe the optional adaptation phase, where we fine-tune the model pre-trained with our framework for a downstream fact verification classification task.
3.1 DATA PROCESSING PIPELINE
Throughout this section, we assume to have sampled a batch of unlabeled claims x = [ with I, being the th claim and N the batch size. In addition, represented as a knowledge graph G(e, R), where E, R are the set of real-world entities associated with name (e.g. Barack Obama, New York City), and the relationship type between entities (e.g. was born in), respectively. We also define F as the set of facts contained in G, represented as triples of subject, relation, object, named as head, relation and tail. Then, G can be defined as {(h, Ti. 4)] fa, t E, E R }, where i E {I...., [F]..
assume access to knowledge base
3.2 PRETRAINING METHOD
As shown in Figure Th our pre-training approach uses a pre-trained language model to obtain a feature tensor for each of the input claims. In SFAVEL, we use knowledge model to embed facts from the knowledge graph, and a scoring module that scores each of such facts conditioned upon specific claim. To tackle the discrepancy in information-level between the features from the pre-trained language model and the knowledge model, we introduce an unsupervised distillation loss that encourages the representation of the claim and its related knowledge facts to be mapped close together in the feature space of the knowledge model. A scoring loss function encourages the scoring module provide higher scores for positive facts than to aondomly-generater negative facts. To
3

---

Published as a conference paper at ICLR 2024
(a) Self-supervised Language Model Distillation Pre-training
(b) (Optional) Fine-tuning
Random sample
"Barack Obama
Claim
Knowledge Base
Negative pool N
Is
Random Ar
Kenyan'
Frozen
Knowledge Model
Language Model
Frozen Language Model
Frozen Knowledge Model
Features
x
x
Clamm Embeedingg
Exndance Embeddings
Scoring Module
XIM
Scoring
Top evidence (K=2)
evidence
0.7
BBarack Obama, bait, Mt HamaR)
iM
Hawall United
Top Exidence Emboddings
Prediction Head
Distillation
Claim-Fact
Intra-Sample
Scoring
REFUTED - 0.36)
Loss
Edismi
intta
sscoring
Figure 1: (a) high-level overview of the SFAVEL framework. Given a textual claim, we use frozen language model (orange box) to obtain its embedding features, X XLM The knowledge base is fed to the knowledge model to produce a knowledge base embedding X Then, the scoring module produces scores for facts in the knowledge base, conditioned upon the claim embedding. The positive sub-graph formed by the top K facts is kept, denoted X=* Next, negative pool instances N. Finally, both the positive and negative sub-graphs are encoded with the knowledge model, obtaining the positive and negative sub-gra -obogrrph embeddings, X and X and their respective scores, St and ST. Grey boxes represent three the different components of our self-supervised loss function used to train the knowledge model. (b) Optional supervised fine-tuning stage downstream task using the pre-trained model.
avoid the network finding trivial solutions where both positive and negative facts are given similar scores, a contrastive loss is used to encourage the separation, within the feature space, of features representing the positive and negative facts.
First, the knowledge model initialized randomly and the backbone is initialized from any off-the- shelf pre-trale mre-trained language model (e.g. T5; Raffel et al. (2020a)). In this work, the backbone fr kept frozen during the entire training and is only used to obtain feature tensor for each claim, denoted as LM which is used for distillation obtain a single tensor representation per claim, we take the global average pooling (GAP) of the backbone features for each claim. For the knowledge model, we utilize Relational Graph Attention Network (Busbridge et al. 2019).
the much smaller knowledge model. In order
Next, the knowledge model, fo G + Rdy, maps the input knowledge graph, G, into X KB TR| V where [ is the number nodes G and dy is the feature space dimensionality. In order to obtain single feature tensor for each fact in F, use multilayer perceptron (MLP) that combines the head and tail embeddings for each fact into single tensor, denoted as X E TR x
Then, given a single claim and fact embedding, denoted as X..M and X,", respectively, we propose score function, fscore- The goal of fecoee is to measure how likely it is that given fact is in the same context as the corresponding claim. Specifically, the calculation of score defined as:
S (xane(X/-1) (XLM X.) dl d(x!MM x!)
(1)
4

---

Published as a conference paper at ICLR 2024
where d(.) is similarity score, which take to be the L2 nonm. Therefore, given the embedding of claim Tú and every fact embedding in the knowledge base, X we can compute the relevance scores S ( (X,.M) .x Vvje (I.....,FFI) [F]}. Then, the set of most relevant facts corresponding to claim 2; is defined as:
F top-rank(S;, K)
where top-rank(., K) returns the indices of top K items a set. We can now obtain the embeddings of the positive facts, X C Cx, and the corresponding scores, S C S, by using the indices of the top K facts according to the scores S.
3.3 GENERATION OF NEGATIVE INSTANCES
In Section 3.2 we have described the process of obtaining both the positive fact embeddings and scores, * and S+, respectively. In this section we explain how to harness the graph structure of the knowledge base to produce corresponding negative signals for contrastive learning.
In order to produce negative set for claim Tiz, herein denoted as N₁, we take inspiration from recent advances in graph contrastive learning (e.g. Xia et al. (2022) Yang et al. (2022);; Rony al. 2022 and propose to generate two sets negative instances: in batbhtch negatives and tn-knowledge be-b base negatives, denoted N/ and N! respectively. Our approach aims generate negative samples that are factually false while preserving the contextual meaning of the entities, so that meaningful negative samples are fetched.
In-batch negatives To generate in-batch negatives we perform random perturbation of the entities in the set of positive facts Fx for a given claim 2;. Formally, let us define the set of triples in the positive set of claim i as T; = {(n, ri, 4)) V Vi (1, 2, .... |P: where h, r, represents head, relation and tail, respectively. Our goal is to generate M negative samples in each given batch B. For each triple rr. in T., we decide a probabilistic manner whether the perturbation is done the head or the tail of the triple. For this, let define random variable perturb head ~ Bern(Pkead) sampled from a Bernoulli distribution with parameter Phead, dictating whether the head of a triple should be perturbed, with probability Phead, the tail otherwise. Then for each triple f, we generate a negative triple l ,,, or ',,,,'' by altering head (Phead 1) or tail (Phead 0), respectively, such that the new head, h', or tail, f', are sampled from e uniformly. To provide semantically meaningful negatives, we enforce the entity type of the randomly sampled head/tail to be of the same type as the one in Up,r.t.
In-knowledge-base negatives Given the nature of the in-batch negative generation process, the negative triples are bounded to be semantically similar to the corresponding positive triples, and hence close by in the feature space. Therefore, this bias leads under-exploration of other parts of the knowledge base feature space. In order to alleviate this issue, we propose to add sundomly-samples facts from the knowledge base into the negative set. Specifically, given the knowledge base G. we sample M triples that are at least H hops away from Fit. This encourages the negative generation procedure to dynamically explore other parts of the knowledge base.
To obtain the final negative set for claim we join both the set of in- negatives Wº and in-knowledge base negatives A2kb as:
N = N U.V.
Finally, we obtain the embeddings the negative set, X from the knowledge model, and the negative scores from the scoring module as S = (X L.M. (.AAM.//1VV D 21.,.... [N,]].
3.4 CLAIM-FACT MATCHING VIA LANGUAGE MODEL DISTILLATION
Once we get positive and negative fact embeddings and scores, they can be used for the distillation process. In particular, seek to learn low-dimensional embedding that "distills" the feature
5

---

Published as a conference paper at ICLR 2024
correspondences of pre-trained language model between textual claims and the knowledge base features produced by the knowledge model. To achieve this, we propose a loss function composed of 3 terms: claim-fact distillation, Ldistitt, intra-sample contrastive loss, Lintras and scoring loss, Lccorrigg
Claim-Fact Distillation To transfer the feature correspondences from the pre-trained language model to the knowledge model, propose Seature-basee (Zagoruyko & Komodakis (017) claim- fact distillation loss. Specifically, we propose the following distillation loss:
FKM |I
FLM
P
Σ JEFT
Cdsatill
(4)
ll2
| 11
where F M and F/-M are respectively the knowledge model (student) and language model (teacher) feature representations, for each fact j. in the positive fact set, F+ p refers to the norm type, and we p 2 for Lg-normalized features. To obtain FLM, we compute the pooled features representation of the verbalized triplet j given by the LM, similarly to Lu et al. (2022). Therefore, the distillation is computed the Mean Squared Error on the normalized fact embeddings.
Intra-Sample Contrastive Loss The intra-sample distillation loss derives from the standard con- trastive loss. The aim of the contrastive loss is to learn representations by discriminating the positive instance, among negative samples. For instance, in MoCo (He et al. (020), two views, x and x of one input image obtained using augmentation, and encoder J, and momentum encoder Ss used to generate embeddings of the positive pairs, such that = (r) and fu(Ø' ). In this case, the contrastive loss can be defined as:
exp(q kt/r) ΣXEN exp(q k;/T)
Leontrastive log
(5)
Similarly Frosst al. (2019), we extend the contrastive loss function by replacing the query, q, in the original formulation with the claim embedding, denoted as X.L.M and extending the formulation account for multiple positive keys. Consequently, we contrast the query with respect each the individual positive (numerator) and negative (denominator) facts, follows:
Cintra
-1 [F
Σ log FEFt
exp(X!.M X! /T) exp LM X2 /T)
(6)
ΣjEF-
Exer-ep(XX
where is the temperature parameter, used during training to smooth the logits distribution. The rationale of Lintra is to pull the positive fact embeddings close to the claim representation while simultaneously pushing away the negative fact embeddings.
Scoring Loss The scoring loss is variant of the conventional pair-wise ranking loss (Chen et al. 2009). Ranking losses are used to evaluate the performance of a learned ranking function. this work, we propose the Lacoring loss function to maximize the scores given by the scoring model to the positive facts, F+, and minimize the scores of the negative facts, F- for a given claim x;. In particular, we minimize the following loss:
Lecoring
ΣΣ
Σ
max (0, + fscare (X. .x,') -soore (X. Xf. ))
iEFt JEF-
where 7 is margin factor. Minimizing Lecoring encourages elements of S (X.MM, XFFt) to be highly ranked and elements of (X. .x." to have low scores. More explicitly, it optimizes the model parameters so that the scores positive facts, (h, r. t) F+, are higher than the scores of negative facts (h'r'', rr'(') F-

---

the official FEVER Published as a conference paper at ICLR 2024
Finally, SFAVEL's full loss is:
Liotal Adistatt Edistal + Xintra Cintra + Macoring_scoring
(8)
where Adistill, Aintras Ascoring E IR. In practice, found that ratio of Xintra Ascoring 2Adistsil led to good experimental results.
EXPERIMENTS
In this section, we present a comparative study of the results of our proposed method on standard benchmarks for fact verification, well ablation studies the most relevant components. We first describe the datasets, evaluation and training settings. Next, we discuss extensive experiments on method for the task of fact verification. Finally, we set of ablation studies evaluate the impact of the most important components of our proposed framework.
4.1 IMPLEMENTATION DETAILS
Datasets and evaluation We the FEVER (Thorne al. 2018) dataset for all our experiments and comparison against previous methods. For pre-training we use the official FEVER training set. For providing the performance comparisons against previous work, testing set. In our ablation studies, we employ the official FEVER development split. To evaluate the learning performance
low-data regime, we randomly sample 1%. 5% 10% of the training data. As knowledge base, we use the Wikidata5m (Wang et al. 202 We provide some examples of claims from FEVER Section A.1 of the Appendix. Hinally, we also compare on the FB15k-231 dataset from Toutanova et al. 2015) in Section A.2 of the Appendix.
Pretraining Eight models with variety of sizes used as pre-trained language models: T5-Small (Raffel et al. 2020a), DeBERTaV3 (He et al. 2023b). XLNet (Yang et al., 2020), RoBERTa (Liu al. 2019). BERT (Devlin al. 2019), Transformer-XL. XL (Dai et 2019) and GPT-2 (Rafford et al. 2019). The pre-trained language models are kept frozen during pre-training with our method. The officially released weights from HuggingFace (Wolf al. 2020 used to initialize the pre-trained language models for fair comparisons. Pre- Pre-traing training is run for total of 1000 epochs. During training we use RGAT the knowledge model with convolutional layers, with hidden size of 512. The projector from node embeddings to triple embeddings is MIP with the same dimensionality as the pre-trained language model sentence embedding size. The model is trained with the SGD optimizer with momentum 0.9 and weight decay 0.0001. The batch size is set 512 over 4 A100 GPUs, and the coefficients for the different losses are Aintra = Ascoring = 1, Adiarill = 2. We set the temperature т 0.1. We use K 5 for the number of facts to keep after scoring. The number of negative instances used the negative pool for contrastive learning is set to M = 4096.
Linear Probe In order to evaluate the quality of the distilled claim-fact matching features, we follow common evaluation protocols (Gansbeke et al., 202T Chen et al.) 2020d) for measuring transfer learning effectiveness. Specifically, we train a linear classifier to perform label assignment to claims (see Figure [ for an example illustration). The classifier trained for 200 epochs, using the SGD optimizer with 20 as the initial learning rate. The only purpose of this linear probe is to evaluate the quality of the features and not part of the SEAVEL pretrainnn procedure.
4.2 RESULTS
We summarize our main results on the FEVER fact verification benchmark in Table Our method significantly outperforms the prior state of the art, both supervised and unsupervised. particular, SFAVEL improves by +9.23% on label accuracy in the test set when using simple linear probe and a frozen backbone pre-trained using our method. Notably, even though our method has been trained without any data annotations, it is capable of outperforming the best supervised method (ProoFVer) by +9.58% label accuracy. These experiments demonstrate the benefits of our task-specific unsupervised pretraining framework for learning rich feature representations for claim-fact matching.
7

---

Published as a conference paper at ICLR 2024
Table 1: Performance on the FEVER benchmark (label accuracy and FEVER score in %) of our proposed pre-training approach after fine-tuning linear classification probe on the FEVER bench- mark. In this experiment we use the GPT-2-XL. backbone. Underlined performances indicate the top score for particular metric, bold indicates the overall best method. We show that our SFAVEL outperforms previous methods, both supervised and unsupervised. Dataset splits are from Thorne
et al. 2018
Dev
Test
Method
Unsupervised
LA
Fever Score
LA
Fever Score
GEAR (Zouu et
2019
X
74.84
70.69
71.60
67.10
KGAT (Lu et I 2021
X
78.29
76.11
74.07
70,38
Di Liello el al. 2022
V
81.21
74.39
GERE Chen el al 2022b
X
79.44
77.38
75.24
71.17
CorelBERT Ye
2020
X
75.96
72.30
DREAM [hhong et al. 2020
X
79.16
76.85
70.60
ProoFVer (Krishna et a
2021
X
80.74
79.07
79.47
76.82
Jobanputra 2019
V
80.20
80.25
SFAVEL (Ours)
90.32
88.10
89.48
86.15
T5 T5-5mal
Tanetsmmnn:
05-
00.37
50-
95.45
24.30
85
Labdl
28.47
72.55
60
52
Labex
Lanelc
10% latelc
05
6.1
6-5
L-20
Figure 2: Low-data experiments by fine- tuning on 1%, 5%, 10% and 15% data over 3 different language backbones.
Tep sevetion
Figure 3: Label accuracy with different K for number of facts selected after scoring.
Furthermore, following previous works in contrastive learning (Chen et al. 2020b), we evaluate the proposed method by distilling 3 different language model backbones (T5- Small, Transformer-XL. GPT-2-XL) and fine-tuning in low-data setting by using 1 %% 5%, 10% and 15% of labeled data. As shown Figure 2 our method capable of achieve on-par performance with recent methods despite only fine-tuning with 1 1% of the data, reaching 74.22% and 76.14% 14% dev set accuracy with Transformer-Xi and GPT-2-XL. backbones, respectively. When using 5% of labelled data, SEAVEL surpasses previous state-of-the-art on the FEVER benchmark. This experiment highlights the high- quality features our framework is capable of learning for claim-fact matching, allowing high accuracy when only few labelled data points are available.
4.3 ABLATION STUDIES
In the following section, we provide several ablation studies for our proposed approach. All experi- ments and results are performed the FEVER development with the GPT-2-XL as language model backbone unless explicitly stated.
To understand the impact pre-trained language model selection for distillation backbone, we perform an ablation study and report the results in Table [ We analyze the effect of using several different language models in SFAVEL, such as T5-Small, DeBERTaV3 XLNet, GPT-2-Small, RoBERTa, BERT, Transformer-Xi and GPT-2-XL. We choose this particular set of language models as they are diverse in terms of their number parameters. The smallest language model in experiments is T5-Small (60 Million parameters), with the biggest LM being GPT-2-XL. (1.5 Billion parameters). This gives some insight into how the language representation capabilities of each of the models affects the distillation effectiveness when using SFAVEL. We find
Pre-trained Language Model
8

---

Published as a conference paper at ICLR 2024
Table Effect of the different components of our loss function. From left to right: language model backbone used, whether claim-fact dis- tillation, intra-sample contrastive and scoring losses are used, respectively.
Table 2: Accuracy of linear classification re- sults on FEVER dev using different pre- trained language models as backhone for dis- tillation.
Scoring Loss
Distill Loss
Cont Loss
Backbone
Params
Accuracy
T5-Small
60M
80.79
DeBERTaV3
86M
82.09
Backbone
Accuracy
XLNet
110M
84.46
T5-Small
56.34
GPT-2-Small
117M
84.92
T5-Small
72.71
RoBERTa
125M
85.31
T5-Small
77.46
BERT
140M
87.92
T5-Small
80.79
Transformer-Xi
257M
89.51
GPT-2-XL
57.20
GPT-2-XL
1.5B
90.32
GPT-2-XL
79.64
GPT-2-XL
87.16
GPT-2-XL
v
90.32
that the GPT-2-XL is the best feature extractor of the list and leads by significant margin in terms of accuracy. However, we note that even the smallest backhone (T5-Small; 60M parameters), although modestly, achieves performance greater than the previous state-of-the-an (+0.54% accuracy).
Influence of in fact selection We inspect the impact of in fact selection after scoring for model performance. As shown in Figure 3 the results are consistent for a range of K (K = 5, 10, 20, 30). particular, we observe a decrease classification accuracy with K 10, 20, 30 compared with K 5. We believe this decrease is caused by the factual noise introduced when K becomes large, where irrelevant information used for verifying the specific claim. contrast, with I, the performance drop is caused by lack of information, as only a single fact is used to check claim. Avoiding this is critical in settings where multiple pieces evidence are required for reasoning, as is for FEVER.
Loss function components We evaluate the different loss functions described in Section 3.4 and provide results in Table 3 particular, we investigate suppressing particular components of the loss function, such as the claim-face distillation loss, intra-sampl contrastive loss, and scoring loss. To do so, we set their respectively factors to O, effectively nullifying their influence during training. We find that these loss components lead to significant performance decreases when removed, therefore justifying our architectural decisions.
5 CONCLUSION
This paper proposes new self-supervised pretraining method based distillation, named SFAVEL, which aims to produce high-quality features for claim-fact matching in the context of fact verification tasks. We have found that modemn self-supervised language model backbones can be distilled into smaller knowledge-awwe aware models to yield state-of-the-art fact verification performance. Our approach achieves this by introducing a novel contrastive loss, that leverages inductive biases in the fact verification task, and exploits them for accurate and entirely unsupervised pretraining for claim-fact matching. show that SFAVEL yields a significant improvement over prior state-of-the-art, both over unsupervised and supervised methods, the FEVER fact verification challenge (+8% accuracy) and the FB15k-233 237 dataset (+5.3% Hits 1). Finally, we justify the design decisions of SEAVEL by performing ablation studies over the most important architectural components. The proposed self-supervised framework is a general strategy for improving unsupervised pretraining for fact verification, and we hope it will guide new directions in the unsupervised learning field.

---

code Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
To guarantee reproducibility of this paper, we
release
the source
at
REFERENCES
Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. Multife: A real-world multi-domain dataset for evidence-based fact checking of claims. CoRR, abs/1909.03242, 2019. URL /ttpppp://a //arxiv
org/abs/1909 03242
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL/ttts::/ppooeediinss neurips wttpeiissscccppaaeeae cc/paper files/ files/papers 22000fffee4455000rr824011118888 paappppppppaaccceeeeaeeearrrrrre -Paper.pdf.
Dan Bushridge, Dane Sherbum, Pietro Cavallo, and Nils Hammerla. Relational graph attention networks. 2019. URL ULL/tta://aaaavv.rrr/bbb//
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 9912-9924. Curran Associates, Inc., 2020. URL htttps::/proceediiiiii www.innggnnnuuurspe neurinu wttpppps.ccppppppp.pppep httppppprrr.llleeppperr///// pss/paper/2220/ 21le/70feb62b6995 (twpaa22229966622222242424aa22//
Chen Chen, Yufei Wang, Bing Li, and Kwok-Yan Lam. Knowledge is flat: A Seq2Seq generative framework for various knowledge graph completion. In Nicoletta Calzolari. Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na (eds.), Proceedings of the 29th International Conference on Computational Linguistics, pp. 4005- 4017, Gyeongju, Republic of Korea, October 2022a. International Committee Computational Linguistics. URL https: htt/://aaaanntooooooooo org/ 2022 www.cconnnnn.o 1.352
Chonghao Chen, Fei Cai, Xuejun Hu, Wanyu Chen, and Honghui Chen. Hhgn: A hierarchical reasoning-bee based heterogeneous graph neural network for fact verification. Inf. Process. Manage., 58(5), sep 2021a. ISSN 0306-4573, doi: prg/10.100
2021..02659, URL https://doii
1pm 2021. 102659
Chonghao Chen, Fei Cai, Xuejun Hu, Jianming Zheng, Yanxiang Ling, and Honghui Chen. An entity- graph based reasoning method for fact verification. Information Processing & Management, 58(3): 102472, 2021b. ISSN 0306-4573, doi: https://bii.rr///oo.oooo...oooop
102477. URL https
// 7/www. sci
www.esccieeeiiic eeceei rect. com http:///oooooeeaaooooo./////////
Chonghao Chen, Jianming Zheng, and Honghui Chen. Knowledge-enhanced graph attention network for fact verification. Mathemattics, 916), 2021c. ISSN 2227-7390. doi: 10,3399/
URL http://pw.../aaan.pp....ooooo///
Chonghao Chen, Jianming Zheng, and Honghui Chen. Cosg: A graph-based contrastive learning method for fact verification. Sensors, 2021d. ISSN 1424-8220. doi: 10.3390/s21 103471.
URL URL https:://///.. wttppppccom/11222222222111111
10

---

Published as a conference paper at ICLR 2024
Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Yixing Fan, and Xueqi Cheng. Gere: Generative
evidence retrieval for fact verification. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, pp. 2184-2189, New York, NY, USA, 2022b. Association for Computing Machinery. ISBN 9781 450387333. doi: 10.1 1414/34474455.3318827
35314777955535511111 3531
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprini htii:2000.110222 2020b.
Wei Chen, Tie-yan Liu, Yanyan Lan, Zhi-ming Ma, and Hang Li. Ranking measures and loss functions in learning to rank. In Y. Bengio, D. Schaurmans, J. Lafferty, C. Williams, and A. Culotta (eds.), Advances in Neural Information Processing Systems, volume 22. Curran Asso- ciates, Inc., 2009. URL httpppp:/pppccceeegss...eeee Paaper..dppppe.pp..
htcpiipp.ccpppppee cc/paper files/paper/
Weijie Chen, Shiliang Pu, Di Xie, Shicai Yang, Yilu Guo, and Luojun Lin. Unsupervised image classification for deep representation learning, 2020c.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, 2020d.
Yu Chen, Lingfei Wu, and Mohammed Zaki. Bidirectional attentive memory networks for question answering over knowledge bases. arXiv preprint arXiv: 1903.0029300888 2019.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe V. Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context, 2019.
Anubrata Das, Houjiang Liu, Venelin Kovatchev, and Matthew Lease. The state of human- centered NLP technology for fact-checking. Information Processing & Management, 60(2): 103219, 2023. ISSN 0306-4573, doi: https://o..orr//0000////pppppmpp 20tp::/.....m..222222.... 103219, URLhttps 1/www. , www.:sccieceeiicom rectt.rrreee...mm cwm/ http:///w...............////////
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Associa- tion for Computational Linguistics, 2019. URL http:p::/pp...eent..ccc...o
www.stsssccc.see......m org/
Co
Luca Di Liello, Siddhant Garg, Luca Soldaini, and Alessandro Moschitti. Paragraph-based trans- former pre-training for multi-sentence inference. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2521 2531, Seattle, United States, July 2022. Association for Computational org/ org
Linguistics. doi: 10. 18631/66vvvl http////2000 1/2022 2022. naac www..annn... 181
181.
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang. Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervised distillation for visual representation, 2021.
Nicholas Frosst, Nicolas Papernot, and Geoffrey Hinton. Analyzing and improving representations with the soft nearest neighbor loss, 2019.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Lue Van Gool. Unsupervised semantic segmentation by contrasting object mask proposals, 2021.
Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. A Survey on Automated Fact-Checking.
Transactions of the Association for Computational Linguistics, 10:178-200 206, 02 2022. ISSN 2307- 387X. doi: 10.
URL httpthttt///tttt org 11 1et.:1111aatce Laaecel____000000 00454
Jiabang He, Liu Jia, Lei Wang, Xiyao Li, and Xing Xu. Mocosa: Momentum contrast for knowledge graph completion with structure- augmented pre-trained language models, 2023a.
11

---

Published as a conference paper at ICLR 2024
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning, 2020.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style embedding sharing. 2023b.
pre-training with
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015.
Xiaowei Huang, Wenjie Ruan, Wei Huang, Gaojie Jin, Yi Dong, Changshun Wu, Saddek Bensalem, Ronghui Mu, Qi, Xingyu Zhao, Kaiwen Cai, Yanghao Zhang, Sihao Wu, Peipei Xu, Dengyu Wu, Andre Freitas, and Mustafa A. Mustafa. A survey of safety and trustworthiness of large language models through the lens of verification and validation, 2023.
Mayank Jobanputra. Unsupervised question answering for fact-checking, In Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER), pp. 52-56, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18663/1//D116660 URL
Shailza Jolly, Pepa Atanasova, and Isabelle Augenstein. Generating fluent fact checking explanations with unsupervised post-editing. Information, 13(10), 2022. ISSN 2078-2489. doi: 10.3390/ info13100500. URL //tp://wwwww........ http://wwpppp.....ooo com/ 20mm//278--a2 21////4/////////
Jiseong Kim and Key-sun Choi. Unsupervised fact checking by counter-weighted positive and negative evidential paths a knowledge graph. Proceedings of the 28th International Confer- ence on Computational Lingwistics, pp. 1677-1686, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10. http://w12222.cooong..o maainng-mii..l 147. 2020. wwwp::000wwo.mmm.....0.......
URL
Amrith Krishna, Sebastian Riedel, and Andreas Vlachos. Proofver: Natural logic theorem proving for fact verification. CoRR, abs/2108. 11357, 2021. URL http://////axwi..o////ass/////// wwtp/aa////aaa///o///
11357
Nayeon Lee, Belinda Z. Li, Sinong Wang, Wen-tau Yih, Hao Ma, and Madian Khabsa. Language models fact checkers? In Proceedings of the Third Workshop Fact Extraction and VERification (FEVER), pp. 36-41, Online, July 2020. Association for Computational Linguistics. doi: 10.18653) v1/2020.fever-I1. URL https: http://aaaaaatooooooorrroo
fever-1.55
Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq. and Hang Li. Trustworthy Ilms: survey and guideline for evaluating large language models' alignment, 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. Fine-grained fact verification with kernel graph attention network, 2021.
Yinquan Lu, Haonan Lu, Guirong Fu, and Qun Liu. Kelm: Knowledge enhanced pre-trained language representations with message passing on hierarchical relational graphs, 2022.
Jackson Luken, Nan-Jiang Jiang, and Marie-Catherine de Marneffe. Qed: A fact verification system for the fever shared task. 2018. Utthttps::// semanticscholi
org/CorpusID
53649146
Michael Mu, Sreyasee Das Bhattacharjee, and Junsong Yuan. Self-supervised distilled learning for multi-modal misinformation identification. In 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2818 18-2827, 2023. doi: 10. 1099NACCV56688..0222. 200.03.0.2220222.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation, 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
12

---

Published as a conference paper at ICLR 2024
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res., (1) jan 2020a. ISSN 1532-4435.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the I. Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140):1-67, 2020b. URL
http::/jmlz:
http:/iwl...rogoooeeeoe wwwppppwpr./ppopoootttttop....at /wteerssvv///// v21 120 074 html
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets, 2015.
Md. Rashad AI Hasan Rony, Mirza Mohtashim Alam, Semab Ali, Jens Lehmann, and Sahar Vahdati. Lemon: Language model for negative sampling of knowledge graph embeddings. 2022. URL
sttt::://ap...eeanttcccsses wwwanteeetttteee www.aotticccco.oo ar wttpaa.oooo//ocoss.o2222220 2521199982
Apoory Saxena, Aditay Tripathi, and Partha Talukdar. Improving multi-hop question answering over knowledge graphs using knowledge base embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4498 1498-4507, 2020.
Jiasheng Si, Deyu Zhou, Tongzhe Li, Xingyu Shi, and Yulan He. Topic-aware evidence reasoning and stance-aware aggregation for fact verification, 2021.
James Thome, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. FEVER: a large- scale dataset for fact extraction and verification. CoRR, abs/1803.0533 2018. URL http:
1/arxiv. wwwxiv.org/abaa. wtwp////aaa///ooooee 05355
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation, 2022.
Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. Representing text for joint embedding of text and knowledge bases. In Lluís Marquez, Chris Bullisoo-Buuchh, and Jian Su (eds.), Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, pp. 1499-1559, Lisbon, Portugal, September 2015. URL https://
Association for Computational Linguistics. doi: 10. 10D15154/
ac acl celanthology.oo wtg//15-111/11
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018.
URL //tpnrvv//peeeevieww net /ost/foooe de-rmmmiddxxXXpiii.
Juraj Vladika and Florian Matthes. Scientific fact-checking: A survey of resources and approaches.
In Anmual Meeting of the Association for Computational Linguistics, 2023.
Bo Wang, Tao Shen, Guodong Long, Tianyi Zhou, Ying Wang, and Yi Chang. Structure-augmented text representation learning for efficient knowledge graph completion. In Proceedings of the Web Conference 2021. ACM, apr 2021a. doi: 10.1 10.1145/344238..0 1.38881340043.... hRRtsstps://dde /rg/
100.1455223444233 2 3450043
Liang Wang, Wei Zhao, Zhuoyu Wei, and Jingming Liu. Simkge: Simple contrastive knowledge graph completion with pre-trained language models, 2022.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang. Zhiyuan Liu, Juanzi Li, and Jian Tang. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Compulational Linguistics, 9:176-194, 2021b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Pro- cessing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguisties. _ httptt:t////....... wwwww//wwww///.o///////////./o web. www..org//attoollgg/////// emn em pe-demos.... 6
13

---

Published as a conference paper at ICLR 2024
Jun Xia, Lirong Wu, Ge Wang. and Stan Z. Li. Progel: Rethinking hard negative mining in graph contrastive learning. In International conference on machine learning. PMLR, 2022.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. LaPraDoR: Unsupervised pretrained dense retriever for zero-shot text retrieval. In ACL 2022 (Findings), 2022.
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self- supervision, 2020.
Yuhao Yang, Chao Huang, Lianghao Xia, and Chenliang Li. Knowledge graph contrastive learning for recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research
and Development in Information Retrieval, SIGIR '22, pp. 1434-144 1443, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450387323. doi: 10.1145/347499955522009 3532005
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.
Liang Yao, Chengsheng Mao, and Yuan Luo. Kg-bert: Bert for knowledge graph completion, 2019.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting language models. arXiv preprint arXiv:2210.03629, 2022.
Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. Corefer- ential reasoning learning for language representation, 2020.
Wenpeng Yin and Dan Roth. Twowingos: A two-wing optimization strategy for evidential claim verification, 2018
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models strong context generators. International Conference for Learning Representation (ICLR), 2023.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. 2017.
Fengzhu Zeng and Wei Gao. Prompt to be consistent better than self-consistent? few-shot and zero-shot fact verification with pre-trained language models. In Annual Meeting of the Association for Computational Linguistics, 2023. htllhttps://eee wwwantticscciee
Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. Reasoning over semantic-level graph for fact checking, 2020.
Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Gear: Graph-based evidence aggregating and reasoning for fact verification, 2019.
Biru Zhu, Xingyao Zhang, Ming Gu, and Yangdong Deng. Knowledge enhanced fact checking and verification. |EEE/ACM Transactions Audio, Speech, and Language Processing, 29:3132-3143, 2021. doi: 10. (0.1109/AASLP22201.1330 1.111333
Arkaitz Zubiaga, Ahmet Aker, Kalina Bontcheva, Maria Liakata, and Rob Procter. Detection and resolution of rumours in social media: A survey. ACM Comput. Surv, (2), feb 2018. ISSN 0360-0300. doi: 10.1 URL
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 EXAMPLES OF CLAIMS IN THE FEVER DATASET
For illustration purposes, in Table we provide some examples of claims, with the annotated evidence and labels, from the FEVER dataset.
Claim
Evidence
Label SUPPORTED
The Rodney King riots took place in the populous county in the USA
(1992 Los Angeles riots, kiso_nnow__as, Rod-
ney King riots), (Rodney King riots, curred in, Los Angeles County), (Los Angeles County, county of, USA), (Los Angeles County, moss_populoos..coutyy.oo USA)
Giada at Home was only available
(Giada at Home, a, Televion Show), (Giada Home, aired on, Food Network), (Food Network, is.a, Television Channel)
REFUTED
DVD
Al Jardine is American rhythm gu-- tarist
(Alan Charles Jardine, is guitarist), (Al Jardine, alias of, Alan Charles Jardine), (Alan Charles Jar-
SUPPORTED
dine, bas_nationality. American)
Bob Ross created ABC drama The Joy of Painting
(Robert Norman Ross, wwww..oooo.nnoo..o..m...mo.o known as, Bob Ross), (Robert Norman Ross, is_a, Painter), (Robert Nor- Ross, is Television host), (Robert Norman Ross, creator.ol .of, The Joy of Painting), (The Joy of Painting, is.a, Instructional television program)
REFUTED
Table Examples from FEVER
(Thone al. 2018) of claims, their annotated evidence and label.
A.2 ADDITIONAL EXPERIMENTS ON THE FB -37-237 DATASET
In addition to our evaluations in Section 4.2 we compare SFAVEL to prior art on the FB F515--237 fact checking task presented in Toutanova al. 2015). We utilize the evaluation metries and dataset processing as (He et al., 2023a). In this setting, in contrast to FEVER evaluation, we obtain the top K facts given claim directly from SFAVEL's unsupervised pre-trained model, without the need for a downstream prediction head. In Table we find that SFAVEL is able to achieve +5.3%, +3.6% and +6.3% for Hits@ 1, Hits and Hits @ I0, respectively, compared the previous state of the art.
Method
Hits@ 1
Hits @3
Hits@ 10
---BERTT Yao ct al
2019
42.0
StAR Wang et al.
2021a
20.5
32,2
48.2
KG-S2S Chen al.
2022a
25.7
37.3
49.8
SimKGC
Wang
2022
24.6
36.2
51.0
MoCoSA He 2023a
29.2
42.0
57.8
SFAVEL (Ours)
34.5
45,6
64.1
Table 5: Performance on the FB15k-237 dataset test split. We use the GPT-2-XL as backbone for distillation. Following previous methods, measure performance using the Hits 1, 3, 10 metric. We obtain the top K facts from our unsupervised pre-trained model, without the need for downstream prediction head.
15


Reviews:
[{"review_id": "PKkxHxsx5P", "reviewer": "Reviewer_bggr", "summary": "This paper focused on unsupervised fact verification --- verifying a claim based on a trustworthy knowledge base without a requirement of direct supervision. The author proposed to train a scoring model on top of embeddings produced by a pre-trained language model to determine whether a given claim can be aligned to a fact from knowledgebase. The scoring model is trained by leveraging positive and negative examples constructed based on triples from a knowledge graph. The author conducted experimental evaluations on FEVER, and showed that their method can yield a significant improvement over SOTA.", "questions": "1. How this method is different from a ranking/retrieval problem? Is fact verification equivalent to ranking/retrieval?\n2. How would this method work on other datasets that are not created based on Wikipedia?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The author's idea of leveraging a knowledge graph to produce positive and negative examples of unlabeled claims to train a scoring model is creative. \n2. The paper is very well-structured, and easy to follow.\n3. The experiments presented promising results on FEVER (~8% improvement on accuracy), and the method can work on a broad set of language models.", "weaknesses": "1. The technique proposed in the paper does not seem to be generalizable. Specifically, the positive and negative examples constructed through triples from knowledge graph are too simple, which makes this method difficult to generalize to more complicated claims. Specifically, triples can only represent who did what, while in reality, a claim can be who did what at where on when for why.  Any wrong information about these factors can make a claim false. While it is not clear to me how the current method can learn a model that can be effectively aware of some more fine-grained factual differences.\n2. The experimental setup is limited. The evaluations are only based on FEVER, which is not convincing. FEVER is created through Wikipedia, and Wikipedia information is closer to triples, which is bias to author's method and training process. At least, an experiment to show the effectiveness of this method on other fact verification dataset would be very helpful.\n3. Ranking may not be the best problem formulation for fact verification. For claim verification, it is important to help people decide whether they should believe the claim or not. Now the author formulates this problem as a ranking problem, which is not very useful from a fact verification perspective. it is not clear what does it mean to the user that a claim can find a piece of evidence with 0.9 score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "liaKzzBmaT", "reviewer": "Reviewer_1Hnq", "summary": "The authors propose a new (contrastive) loss to train models for unsupervised fact verification. This allows to check claims without having to collect annotations, instead relying on unsupervised claim-fact alignment.", "questions": "* Have you considered including larger language models (given that the title mentions \"Large Language Models\")?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The paper is very well written and well motivated.\n* The results presented in the paper are impressive, outperforming FEVER SOTA even for supervised approaches.\n* The authors compare the approach on 7 different models, including a variety of small to medium size models.\n* The paper contains good ablation experiments, in particular analysing the different components of the loss on a small model.", "weaknesses": "* No large models were included, the biggest model tested has 250M parameters. There is no strict definition of LLM, but the authors may overpromise in their title/intro when no model with more than 1B parameters is included.\n* The increase over the SOTA may be exaggerated, given that most of the systems the paper compares to are several years old, and do not include the latest generation of models. (This is not strictly a weakness, but context worth mentioning.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cqTCmzm6YX", "reviewer": "Reviewer_6mXX", "summary": "The paper motivates the necessity of fact-verification specifically in an unsupervised way. Given that the recent works have focused on NLIs, this work focuses on a pre-training objective that includes claim-fact distillation loss, intra-sample contrastive loss, and scoring loss. These losses are determined based on the positive and negative samples and their embeddings from the knowledge-base conditioned on the claim. The overall goal is to pre-train this model in the context of available knowledge base to verify facts. The results show that the model performs well on FEVER dataset. \n\nConcerns of this work: \n1. Size of the models: Given the aspect of large language models where the sizes are in billions, the evaluation is performed on smaller models. Is there some conclusion that can be made with these smaller models instead of just mentioning that the results are good?\n2. Datasets: The work has been evaluated only on one single dataset which begs the question of generalizability. Some works such as [1, 2] have evaluated on other datasets such as UKP and FEVER 2 etc. How does this work compare to those? This is particularly necessary because of the use of Wikidata5m knowledge-base that is used. If it is outside the context of Wikipedia, how can this approach work for other knowledge-bases?\n3. Comparisons to other approaches: The top-k fact retrieval seems to play an important role, given that the recall in number of facts is improved based on the ranking your approach has, is it a fair comparison to other approaches that work on probably the only retrieved fact? If K=1 then the dev numbers are not comparable to any of the approaches mentioned in the paper. \n4. Self-supervision: There is a strong assumption that there is availability of a knowledge base - Wikidata5m and since Fever is derived from it, the losses are carried between facts from Wikidata and Fever claims. Would be good to clarify why the authors think this is self-supervised? \n\n\n[1]: Incorporating External Knowledge for Evidence-based Fact Verification\n[2]: Retrieval-augmented generation for knowledge-intensive nlp tasks", "questions": "In the Summary", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper describes a novel approach for fact-verification\n2. Results show significant gains in comparison to state of the art approaches", "weaknesses": "1. Generalizability of the approach given other knowledge bases\n2. Self-supervision is an ambitious claim\n3. Fever is the only dataset used", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "doSiX2dZiB", "reviewer": "Reviewer_GMHM", "summary": "The paper proposes an unsupervised method called SFAVEL for fact verification. The method aim is to distil knowledge from a language model into a knowledge model so that they produce similar vectors for fact in a formal form and in a natural language form. Besides to make the knowledge model distinguish facts, contrastive learning is employed. The paper shows that SFAVEL remarkably outperforms SoTA fact-verification models on the FEVER dataset (about 8% label accuracy higher than the best model in the literature).", "questions": "In contrastive learning loss, e.g. eq 5, the denominator includes the numerator. However, it's not the case in eq 6. Why is that? What is the impact of excluding the numerator out of the denominator? \n\nIn section 4.2, what would happen if more data are used (rather than only 1, 5, 10%)?\n\nIn table 3, as transformer-XL is used in the end, why the ablation is for T5-small instead?\n\n\n-------\nThe score is updated after reading the authors' response.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The experimental result of SFEVEL on the FEVER dataset is remarkable.", "weaknesses": "First of all, the paper is confusing in using the term \"unsupervised\". The proposed method SFAVEL is unsupervised because it is for learning a knowledge model. However, the fact-verification model reported in section 4 is supervised. The model uses SFAVEL for mapping fact / claim to vectors and then uses a classifier trained in a supervised learning manner. \n\nSecondly, it is unclear about what is the used \"linear probe\". Fig 4b shows that the linear probe takes top evidence as input. But then how can we verify the input claim if we use only evidence (and their scores)? E.g. how knowing \"Obama was born in Hawaii\" and \"Hawaii is in the US\"  can reject a claim without knowing what the claim is? \n\nThirdly, although the performance of the proposed model is remarkable, it is unclear why there's such a big gap between it and the existing models in the literature. What are cases that the proposed model can solve but the others can? Does the model find some crucial factors that the others miss? \n\nLast but not least, the proposed SFAVEL is for learning a knowledge model. But it is unclear whether that knowledge model is useful for other fact-verification cases (like on other FEVER dataset -- FEVER 2.0 for example). Also, whether that knowledge model is also useful for other downstream task requiring fact?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IUmLC4Exat", "reviewer": "Reviewer_kcpN", "summary": "This work proposes an unsupervised framework for unsupervised fact verification. Specifically, three losses are designed to encourage the two models adopted in this framework to produce high-quality features for claim-fact matching. The authors conduct experiments on the FEVER dataet. Experimental results show that the proposed method can achieve good performance on the FEVER dataset.", "questions": "Please refer to the questions above in the weakness. Especially please make it clear how the model works at the beginning when the representations of the LM and the knowledge model are significantly different.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Experimental results on the FEVER dataset show the advantage of the proposed method. The improvement seems very significant.\n\n2. The unsupervised manner of conducting fact verification is encouraged and useful.", "weaknesses": "1. Though the experiments show the effectiveness of the method, I do not get how the framework solves the cold starting. For the scoring module, the claim embeddings from the LM are very different from those of Knowledge from the knowledge model. Then how does the framework pick the top-k evidence at the beginning? How does $L_{distill}$ work at the early iterations? This is an important prerequisite that should be clearly stated in the paper. \n\n2. Why does the paper only evaluate the effect of each loss on a smaller T5 model? Considering the best performance reported in the work is based on Transformer-XL, ablation studies based on it are desired.\n\n3. The annotation in the methodology section makes me really confused. For example but not limited to:\n\n    (1) In Section 3.1, what does the V and each $v_i$ mean? I cannot get it until I read through the whole methodology section. \n\n    (2) The use of subscript and superscript is messed up. I think embedding is presented as $X_F$ in Section 3.2, but it becomes $X^F$ in Section 3.3.\n    \n    (3) Some annotations are not really necessary, e.g., $N_i$ in Equation 3 are not used anywhere else in the paper.\n\n    I encourage the authors to make Section 3 more concise and clear by better formula presenting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)