# Automatic Fact Checking Using an Interpretable Bert-Based Architecture on COVID-19 Claims
## 5 Experiments and Results  
We trained the relevance evaluation network for three epochs, using a learning rate decay of 0.1 per epoch. BERT blocks were frozen in the first two epochs, and only the extra layers were trained. In the third epoch, BERT blocks were fine-tuned together with the rest of the network. Additionally, using BERT shared parameters was proven to produce better results. The model intended for verification also used BERT shared parameters. We also considered other transformers such as GPT-2 [17], Roberta [18] and Sentence-BERT [16]; however, BERT achieved the best results. As a baseline for relevance evaluation, we used a model implementing SentenceBERT to measure the distance between potential evidence and claim. Distance is computed with cosine similarity. Figure 4 shows this model. We take the top five sentences that are the most closely related to the claim. In this model, the evidence was classified considering its similarity to the claim; if the similarity between the claim and the evidence is less than 0.4, then the evidence is considered relevant; otherwise, it is classified as irrelevant. Figure 3: Classification layer. Table 1 presents the accuracy of the best configurations tested for relevance evaluation task. Our best model for this task was composed of BERT with shared parameters, multiplicative + softmax attention, and a five-layer LSTM for sequence representation. Using other transformers was not beneficial in evaluating relevance. Our model (in bold) underperformed compared with the state of the art [PERSON] et al. [11] for a small difference. In Table 2, we present the recall, precision, and F1 scores considering a retrieval window of five documents. The second model, intended to verify a claim against evidence, was trained in the dataset created with predictions of the best relevance model. The baseline for the verification task is based on the work of [PERSON] et al. [19], who proposed a LSTM neural network intended to detect fake news based on evidence found in Google [[https://www.google.com/](https://www.google.com/)]([https://www.google.com/](https://www.google.com/)) (accessed on 17 October 2022). For comparison, we adapt this model by removing the entries referring to sources' credibility, since the credibility of the evidence is taken for granted in our model [19]. The verification baseline is also trained with the same data as our models. Table 3 present the accuracy of the verification model. The best configuration for this task differs from the best one focused on evaluating relevance, which reflects the different nature of these tasks. The best configuration was SentBERT (Roberta), multiplicative + softmax attention, and average sequence representation. Our best configuration underperformed compared to the state of the art [11], but has considerably better results than the baseline [19]. \begin{table}
\begin{tabular}{l c c c} \hline \hline  & **Model** & **Train** & **Test** \\ \hline Soleimani 2020 [11] & & 0.9298 & **0.8996** \\ [PERSON](shared) + Multiplicative + Softmax & R2 & 0.9261 & 0.8892 \\ [PERSON](shared) + Cosine & R3 & **0.9419** & 0.8858 \\ [PERSON](shared) Multiplicative + Sigmoid & & 0.9241 & 0.8651 \\ [PERSON] (Shared) + Multiplicative + LSTM-5L & R1 & 0.9322 & 0.8937 \\ Sentence-BERT Baseline & & 0.6649 & 0.6553 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Accuracy of the best configurations. Relevance evaluation task. Figure 4: Baseline implemented with SentenceBERT [16] for the relevance evaluation task. The most similar sentences are retrieved. We tested the best configurations of both tasks resulting in nine combinations, and with them, we experimented with the complete three-stage method. For document retrieval, we used the predictions of [9] that achieved a recall of 93% in FEVER. With these predictions, we performed the claim relevance evaluation resulting in a set of evidence. Lastly, both evidence and claim were verified, producing a final tag prediction. Table 4 presents the resulting accuracy using complete method.