SlowGuess/NAACL_2022_OCR huggingface dataset:
Given source documents  $D$ , the latent retriever selects relevant sentences as evidence that can be used to predict the veracity for the claim  $c$ . For the  $i$ -th sentence  $x_{i} \in D$ , we obtain the sentence-level embedding  $\mathbf{h}_{i}$  based on a BERT encoder by using the CLS token. Then we can calculate the latent selector  $k_{i}^{\prime \prime}$  by:
$$
k _ {i} ^ {\prime \prime} = s \left(u _ {i}; a _ {i}, b _ {i}, l, r\right),
$$
$$
a _ {i} = f _ {a} \left(\mathbf {h} _ {i}; \phi_ {a}\right) \quad b _ {i} = f _ {b} \left(\mathbf {h} _ {i}; \phi_ {b}\right) \quad u _ {i} \sim \mathcal {U} (0, 1) \tag {3}
$$
where  $f_{a}(\cdot ;\phi_{a})$  and  $f_{b}(\cdot ;\phi_{b})$  are feed-forward networks with softplus outputs  $a_{i}$  and  $b_{i}$ .  $s(\cdot)$  turns the uniform sample  $u_{i}$  into the latent selector  $k_{i}^{\prime \prime}$ . Next, we use the sampled  $k_{i}^{\prime \prime}$  to modulate inputs to the classifier for veracity prediction:
$$
f _ {F} \left(k _ {i} ^ {\prime \prime} \cdot \mathbf {h} _ {i}, \mathbf {c}; \theta_ {F}\right) \tag {4}
$$
where  $\mathbf{c} = f_{\theta^{\prime}}(c)$  denotes the embedding for the given claim  $c$  obtained by using the CLS token through a BERT encoder.  $f_{F}(\cdot ;\theta_{F})$  represents the classifier (e.g. graph-based model). The joint system can be optimized by gradient estimates of  $\mathcal{E}(\phi ,\theta)$  via Monte Carlo sampling from:
$$
\mathcal {E} (\phi , \theta) = \mathbb {E} _ {\mathcal {U} (0, 1)} [ \log P (y \mid X, s _ {\phi} (u, X), \theta) ] \tag {5}
$$
where  $y$  is the label of veracity and  $k_{i}^{\prime \prime} = s_{\phi}(u,X)$  abbreviate the transformation from uniform samples to HardKuma samples.
4.2.2 More Baselines
Apart from the proposed system, we include following baselines for comprehensive comparisons:
Reinforce Instead of using gradient-based training, we follow Lei et al. (2016) by assigning a binary Bernoulli variable to each evidence sentence.
5Please refer to the detailed derivations in Bastings et al. (2019).
Because gradients do not flow through discrete samples, the evidence retriever is optimized using REINFORCE (Williams, 1992). A  $L_{0}$  regularizer is used to impose sparsity-inducing penalties.
Multi-task We also adopt the multi-task learning method proposed by Yin and Roth (2018), which is the state-of-the-art joint model on FEVER dataset (Thorne et al., 2018). The model predicts a binary vector that indicates the subset of sentences as evidence, and a one-hot vector indicates the veracity of the claim. The overall training loss is the sum of these two prediction losses.
5 Experiments and Analyses
Following Augenstein et al. (2019), we computed the Micro F1 and Macro F1 as the evaluation metric. We further reported the mean F1 score and standard deviation by using 5 models from independent runs. For the pipeline system, we used 6 different evidence settings, including evidence sentences retrieved by surface ranker, semantic ranker and hybrid ranker, Google snippets, gold evidence and without using any evidence. For the joint system, we used 2 types of evidence: Google snippets and source documents, where the latent retriever can select sentences from. We used three classifiers for both systems, BERT-based (Schuster et al., 2021), attention-based (Gupta and Srikumar, 2021) and graph-based models (Liu et al., 2020).
The hyper-parameters are chosen based on the development set. In the evidence retrieval step of the pipeline system, we set the retrieved evidence obtained from TF-IDF to be more than 5 words for surface ranker. We use the BERT default tokenizer with max-length as 256 to preprocess data for semantic ranker. We use the default parameters in sklearn.svm.LinearSVC with RBF kernel for hybrid ranker.
In the veracity predication step of the pipeline system, we use the BERT default tokenizer with max-length as 256 and pretrained BERT-base-Chinese as the initial parameter to encode claim and evidence<sup>6</sup>. For BERT-based model, the fully connected network for classification is defined with layer dimensions of  $h_R - h_R / 2$ -verification_labels, where  $h_R = 768$ . We use BertAdam (Devlin et al., 2019) with  $5e - 6$  learning rate, warmup with 0.1 to optimize the cross entropy loss and set the batch
size as 16. For attention-based model, we use BertAdam with  $2e - 5$  learning rate, warmup with 0.1 to optimize the cross entropy loss and set the batch size as 8. For graph-based model, we use BertAdam with  $5e - 5$  learning rate, warmup with 0.1, batch size with 16, dropout with 0.6 and kernel size with 21.
For the joint system, we use Adam (Kingma and Ba, 2015) with  $5e - 5$  learning rate, learning rate decay with 0.5 to optimize the cross entropy loss. We set the batch size as 32. The fully-connected networks  $f_{a}(\cdot ;\phi_{a})$  and  $f_{b}(\cdot ;\phi_{b})$  for two parameters  $a_{i}$  and  $b_{i}$  are defined with layer dimensions of  $h_R = 768$ . We set the dropout rate as 0.5.
Pipeline System: According to Table 5, pipeline systems with evidence including Google snippets, sentences returned by rankers and gold evidence consistently outperform systems without using evidence. These results confirm that evidence plays an important role in verifying real-world claims. On the other hand, systems with retrieved sentences achieve higher scores than systems with Google snippets. Specifically, systems with gold evidence significantly outperform the ones with Google snippets, indicating information that is necessary for verification is missing in the snippets. Moreover, systems with retrieved evidence are more robust in terms of standard deviation. We hypothesize the reason is that irrelevant information is presented in the snippets. When comparing with different rankers, we observed that using contextualized representations to measure the similarity (Semantic Ranker) is generally better than exact string match (Surface Ranker). However, there still exists a large performance gap between the pipeline system with semantic ranker and the system with gold evidence. One potential solution is to develop better retrieval models based on the supervision signal of gold evidence provided by CHEF. Given the evidence sentences, graph-based models tend to have higher scores than BERT-based and attention-based models, which shows the effectiveness of leveraging graph structure to synthesize multiple evidence.
Joint System: Similar to the pipeline systems, joint systems that retrieve evidence sentences from documents achieve better F1 scores than directly use the summary snippets. In order to verify real-world claims, it is necessary to train fact-checking systems that learn how to effectively retrieve evi
<https://huggingface.co/
System / Evidence	BERT-Based Model1	Attention-Based Model2	Graph-Based Model3
Micro F1	Macro F1	Micro F1	Macro F1	Micro F1	Macro F1
Pipeline	No Evidence	54.46±2.89	52.49±2.44	54.87±1.95	53.47±2.82	—	—
Snippets	62.07±2.55	60.61±2.96	62.42±2.31	60.24±2.56	62.78±1.70	61.06±2.59
Surface Ranker	63.17±1.67	61.47±2.02	63.77±1.89	62.65±2.32	64.58±1.45	61.46±1.72
Semantic Ranker	63.47±1.71	61.94±1.66	63.95±1.46	62.80±1.33	64.67±1.54	62.28±1.50
Hybrid Ranker	63.29±1.65	61.80±2.31	63.48±1.22	62.74±1.30	64.37±1.66	62.58±1.43
Joint	Reinforce4	Snippets	63.76±1.52	61.74±1.88	64.06±1.76	61.97±1.04	65.77±1.23	62.34±1.11
Documents	64.37±1.65	62.46±1.72	64.86±1.83	62.66±1.32	66.58±1.45	63.47±1.58
Multi-task5	Snippets	62.78±1.41	61.98±2.59	64.43±1.72	61.58±1.34	66.21±1.57	63.15±1.46
Documents	65.02±1.46	63.12±1.78	65.45±1.59	62.94±2.03	67.46±1.72	64.31±1.81
Latent	Snippets	64.45±1.68	62.52±2.23	65.73±1.75	63.44±1.68	67.81±1.74	64.34±1.57
Documents	66.77±1.43	64.65±1.74	67.62±1.48	64.81±1.26	69.12±1.13	65.26±1.67
Pipeline	Gold Evidence	78.99±0.82	77.62±1.02	79.18±1.07	78.36±1.40	79.84±1.24	78.47±1.17
Schuster et al. (2021)<sup>1</sup>, Gupta and Srikumar (2021)<sup>2</sup>, Liu et al. (2020)<sup>3</sup>, Lei et al. (2016)<sup>4</sup>, Yin and Roth (2018)<sup>5</sup>
dence sentences from full documents on web pages. In addition, joint system outperforms pipeline system consistently with both Google snippets and source documents as inputs. For example, latent retriever with Google snippets are able to achieve an average  $2.74\%$  and  $1.77\%$  Micro/Macro F1 boost compared with the pipeline systems with the same type of evidence. We attribute the consistent improvement of joint system to the explicit feedback to the evidence retrieval via gradient estimation on veracity prediction. Another advantage of the joint system is that the latent evidence retriever is able to dynamically select relevant sentences from documents for each instance, while rankers return a fixed number of evidence.
Compared with the reinforce and multi-task methods, the proposed latent retriever achieves  $1.41\%$  and  $1.98\%$  higher F1 on average with Google snippets and source documents as inputs across various classifiers. When considering standard deviation, reinforce is less robust. We believe the main reason is that latent retriever facilitate training through differentiable binary variables, which leads to robust and generalized model that exhibits small variance over multiple runs.
5.3 Analysis and Discussion
In this section, we further provide fine-grained analyses for baseline systems on CHEF. For brevity, we abbreviate pipeline systems with Google Snippets, Surface Ranker, Semantic Ranker, Hybrid Ranker as GS, Sur, Sem, Hyb, while joint systems with Google snippets and source documents as inputs as JG and JS, respectively. All results are reported
Table 5: Results of pipeline and joint systems on CHEF.
#E	GS	Sur	Sem	Hyb	JG	JS
1	55.24	55.67	56.04	56.72	56.98	57.54
3	58.69	59.24	59.52	59.18	59.89	61.45
5	60.61	61.47	61.94	61.80	62.12	64.65
10	59.12	60.20	60.37	61.24	61.86	64.73
15	55.72	56.31	56.56	57.08	58.69	59.11
Table 6: Effects of evidence: Macro F1 scores on the test set are reported. #E indicates the number of evidence.
based on the BERT-based model. We further provide case study and error analysis on CHEF. Due to limited spaces, we attach them in the appendix.
Effect of Evidence: In Table 6, we varied the numbers of evidence retrieved and reported the Macro F1 on the test set. The fluctuation results indicate that both quantity and quality of retrieved evidence affect the performance. Using fewer evidence will lead to incomplete coverage, which may not provide sufficient information to verify the claims. On the other hand, incorporating more evidence may introduce irrelevant sentences thus propagate errors to veracity prediction. In general, systems with 5 evidence sentences achieves the best performance except the joint system with source documents as inputs. We believe the reason is that the latent retriever maintains a better balance between keeping relevant and removing irrelevant sentences, which helps to achieve higher scores with more evidence sentences.
Performance against Claim Length: We partitioned the test set into 4 classes (<10, 10-19, 20-29, ≥30) based on lengths of the claims and reported
Figure 3: Comparisons against claim lengths.
Figure 4: Per-class results.
the Macro F1 score. For clarity, we choose the best reported pipeline system with semantic ranker to compare with joint systems. As shown in Figure 3, most claims are longer than 10 words. Performance of the systems on short claims (e.g.  $< 10$ ) is lower than other. One reason is that such claims do not contain sufficient information to retrieve evidence and to be verified, based on the observation that the performance of all the systems improve as the length of the claim increase. In general, the joint system outperforms the pipeline system against various claim lengths.
Performance against Classes and Domains: As CHEF is constructed based on real-world claims,
Figure 5: Per-domain results.
most of them are non-factual claims verified by fact-checking websites. Such an imbalance issue poses a challenge to the fact-checking system. Figure 4 shows the performance of models for different veracity labels. The scores of minor classes are much lower than the majority class. This reflects the difficulty of judging SUP and NEI. Informative evidence helps to alleviate this issue. For example, the pipeline system with gold evidence achieves significant improvement on predicting NEI labels when comparing with the system with semantic ranker. Figure 5 shows the performance of different domains. Claims from science, politics and culture domains have fewer training instances as most claims in the dataset focus on the society and public health topics. Again, retrieving informative evidence sentences (JS and Gold) from full documents is beneficial to this data sparsity issue.
We constructed the first Chinese dataset for evidence-based fact-checking. Further, we have discussed the annotation methods and shared some of the insights obtained that will be useful to other non-English annotation efforts. To evaluate the challenge CHEF presents, we have developed established baselines and conducted extensive experiments. We show that the task is challenging yet feasible. We believe that CHEF will provide a stimulating challenge for automatic fact-checking.
We thank the reviewers for their valuable comments. The work was supported by the National Key Research and Development Program of China (No. 2019YFB1704003), the National Nature Science Foundation of China (No. 62021002 and No. 71690231), NSF under grants III-1763325, III-1909323, III-2106758, SaTC-1930941, Tsinghua BNrist and Beijing Key Laboratory of Industrial Bigdata System and Application.
8 Ethical Consideration
Datasets have been collected in a manner that is consistent with the terms of use of any sources and the intellectual property. For each annotator, we compensate based on the number and quality of annotated sentences. More details of our datasets are depicted in Section 3.
Bill Adair, Chengkai Li, Jun Yang, and Cong Yu. 2017. Progress toward "the holy grail": The continued quest to automate fact-checking. In Proc. of the 2017 Computation+Journalism Symposium.
Rami Aly, Zhijiang Guo, M. Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, O. Cocarascu, and Arpit Mittal. 2021. FEVEROUS: Fact Extraction and VERIFICATION over unstructured and structured information. In NeurIPS.
Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019. MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims. In Proc. of EMNLP-IJCNLP, pages 4685-4697, Hong Kong, China. Association for Computational Linguistics.
Ramy Baly, Mitra Mohtarami, James Glass, Lluis Marquez, Alessandro Moschitti, and Preslav Nakov. 2018. Integrating stance detection and fact checking in a unified corpus. In Proc. of NAACL-HLT, pages 21-27, New Orleans, Louisiana. Association for Computational Linguistics.
Jasmijn Bastings, Wilker Aziz, and Ivan Titov. 2019. Interpretable neural predictions with differentiable binary variables. In Proc. of ACL, pages 2963-2977. Association for Computational Linguistics.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. Tabfact: A large-scale dataset for table-based fact verification. In Proc. of ICLR. OpenReview.net.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL-HLT, pages 4171-4186. Association for Computational Linguistics.
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.
Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022. A Survey on Automated Fact-Checking. Transactions of the Association for Computational Linguistics, 10:178-206.
Ashim Gupta and Vivek Srikumar. 2021. X-fact: A new benchmark dataset for multilingual fact checking. In Proc. of ACL-IJCNLP, pages 675-682, Online. Association for Computational Linguistics.
Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. INFOTABS: Inference on tables as semi-structured data. In Proc. of ACL, pages 2309-2324, Online. Association for Computational Linguistics.
Andreas Hanselowski, Christian Stab, Claudia Schulz, Zile Li, and Iryna Gurevych. 2019. A richly annotated corpus for different tasks in automated fact-checking. In Proc. of CoNLL, pages 493-503, Hong Kong, China. Association for Computational Linguistics.
Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. 2020. HoVer: A dataset for many-hop fact extraction and claim verification. In *Findings of EMNLP*, pages 3441-3460, Online. Association for Computational Linguistics.
Jude Khouja. 2020. Stance prediction and claim verification: An Arabic perspective. In Proc. of FEVER@ACL, pages 8-17, Online. Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. of ICLR.
Neema Kotonya and Francesca Toni. 2020a. Explainable automated fact-checking: A survey. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 5430-5443. International Committee on Computational Linguistics.
Neema Kotonya and Francesca Toni. 2020b. Explainable automated fact-checking for public health claims. In Proc. of EMNLP, pages 7740-7754, Online. Association for Computational Linguistics.
Ponnambalam Kumaraswamy. 1980. A generalized probability density function for double-bounded random processes. Journal of hydrology, 46(1-2):79-88.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proc. of EMNLP, pages 107-117.
Chloe Lim. 2018. Checking how fact-checkers check. Research Politics, 5.
Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020. Fine-grained fact verification with kernel graph attention network. In Proc. of ACL, pages 7342-7351, Online. Association for Computational Linguistics.
Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J. Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016. Detecting rumors from microblogs with recurrent neural networks. In Proc. of IJCAI, pages 3818-3824. IJCAI/AAAI Press.
Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Proc. of AAAI.
Jeppe Nørregaard and Leon Derczynski. 2021. Danfever: claim verification dataset for danish. In Proc. of the 23rd Nordic Conference on Computational Linguistics, NoDaLiDa 2021, Reykjavik, Iceland
(Online), May 31 - June 2, 2021, pages 422-428. Linkoping University Electronic Press, Sweden.
Vahed Qazvinian, Emily Rosengren, Dragomir R. Radev, and Q. Mei. 2011. Rumor has it: Identifying misinformation in microblogs. In Proc. of EMNLP.
Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proc. of EMNLP, pages 2931-2937, Copenhagen, Denmark. Association for Computational Linguistics.
Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with contrastive evidence. In Proc. of NAACL-HLT, pages 624-643, Online. Association for Computational Linguistics.
Tal Schuster, Roei Schuster, Darsh J. Shah, and Regina Barzilay. 2020. The limitations of stylometry for detecting machine-generated fake news. Computational Linguistics, 46(2):499-510.
Shaden Shaar, Nikolay Babulkov, Giovanni Da San Martino, and Preslav Nakov. 2020. That is a known lie: Detecting previously fact-checked claims. In Proc. of ACL, pages 3607-3618. Association for Computational Linguistics.
Gautam Kishore Shahi and Durgesh Nandini. 2020. FakeCovid – a multilingual cross-domain fact check news dataset for covid-19. In Proc. of WSM@AAAI.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERIFICATION. In Proc. of NAACL-HLT, pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
Joseph E Uscinski and Ryden W Butler. 2013. The epistemology of fact checking. Critical Review, 25(2):162-180.
Andreas Vlachos and S. Riedel. 2014. Fact checking: Task definition and dataset construction. In Proc. of LTCSS@ACL.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proc. of EMNLP, pages 7534-7550, Online. Association for Computational Linguistics.
William Yang Wang. 2017. "liar, liar pants on fire": A new benchmark dataset for fake news detection. In Proc. of ACL, pages 422-426, Vancouver, Canada. Association for Computational Linguistics.
Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-256.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proc. of EMNLP Demos, pages 38-45. Association for Computational Linguistics.
Wenpeng Yin and Dan Roth. 2018. Twowingos: A two-wing optimization strategy for evidential claim verification. In Proc. of EMNLP, pages 105-114. Association for Computational Linguistics.
Wenxuan Zhang, Yang Deng, Jing Ma, and Wai Lam. 2020. AnswerFact: Fact checking in product question answering. In Proc. of EMNLP, pages 2407-2417, Online. Association for Computational Linguistics.
Xueyao Zhang, Juan Cao, Xirong Li, Qiang Sheng, Lei Zhong, and Kai Shu. 2021. Mining dual emotion for fake news detection. In WWW '21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021, pages 3465-3476. ACM / IW3C2.
Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2020. Reasoning over semantic-level graph for fact checking. In Proc. of ACL. Association for Computational Linguistics.
Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. GEAR: Graph-based evidence aggregating and reasoning for fact verification. In Proc. of ACL, pages 892-901, Florence, Italy. Association for Computational Linguistics.
Arkaitz Zubiaga, Ahmet Aker, Kalina Bontcheva, Maria Liakata, and Rob Procter. 2018. Detection and resolution of rumours in social media: A survey. ACM Comput. Surv., 51(2):32:1-32:36.
A Supplement Materials
A.1 CHEF annotation guidelines
A.1.1 声明抽取和规范化的指引Guidelines for claim extraction and normalization:
标注者首先需要认真阅读事实验证的文章，然后使用一到两句话来概括这篇文章描述的事件作为声明。请标注者直接使用文章中的句子作为声明，比如文章的标题，或者首段的前几句话都可能是这篇文章需要验证的事实。如果没法抽取出相应的句子，可以使用自己的语言总结文章来撰写声明。在撰写声明的时候，有以下注意事项：
- 每个声明必须完整。
- 每个声明不应该存在事实验证偏差。
- 每个声明不应该存在信息泄露。
请仔细阅读以下详细指引和相应的规范化例子：
- 声明中描述的事件缺乏必要的细节,比如:时间和地点。标注者需要加上这些细节让声明完整,才能被验证。比如:今年共有12.08万人参加中考,但招生计划只有4.3万。需要改写声明为:2019年,共有12.08万人参加成都中考,但招生计划只有4.3万。
- 声明中存在特殊符号，需要去除特设符号避免声明中存在偏差。比如：“纯天然”喷雾一喷“秒睡”。需要去除句子中的“”，因为这些特殊符号隐晦地表达了这个声明其实是不实的。模型可以通过特殊符号直接判断一个声明的真实性。这个句子需要改写为：纯天然喷雾一喷秒睡。，这可以避免由于声明中的特殊符号“”带来事实验证偏差。
- 声明中存在信息泄露，需要去除直接指出声明真实性的相关词语。比如：谣言！纯天然喷雾一喷秒睡。句子中使用的“谣言！”已经直接指出该声明是不实的，造成了信息泄露。需要改写声明为：纯天然喷雾一喷秒睡。不能在声明中出现诸如：“谣言”、“错误”，“骗局”等信息泄露词。
- 声明中的反问句需要被改写为陈述句，由于采用反问句形式的声明大部分都是不实的，反问句的形式会造成数据集偏差。比如：别人打了新冠疫苗，我们就可以不打新冠疫苗吗？需要被改写为：别人打了新冠疫苗，我们就可以不打新冠疫苗。
- 不陈述事实的声明需要被丢弃。有两大类的声明是无法进行事实验证的。第一大类为表示推测的声明，比如：明年深圳房价会上涨。第二大类为表示个人意见的声明，比如：我认为特朗普应该连任。
- 声明中如果包含多个声明，需要拆分为多个声明逐一验证。比如：关于新冠疫苗接种的两个事实：第一，别人打了新冠疫苗，自己就可以不打新冠疫苗。其次，新冠疫苗只需要打一针就能具备新冠病毒防护能力。这个声明包括了两个子声明，需要被拆分为：别人打了新冠疫苗，我就可以不打新冠疫苗。第二个声明为：新冠疫苗只需要打一针就能具备新冠病毒防护能力。