# Automatic Fact Checking Using an Interpretable Bert-Based Architecture on COVID-19 Claims
## 3 Interpretable Bert-Based Claim Verification  
BERT [12] and other transformer-based models [13] have state-of-the-art results in many natural language processing (NLP) tasks, including fact verification. However, these models are large, with millions of parameters and many attention heads. These attention heads can be used to interpret the decisions made by the model. Some of the most powerful transformers rely on the size of both parameters and pretraining data. Consequently, the attention mechanisms are numerous, and getting a single interpretation is complex. Currently, research about what is being learned inside transformer networks is still in process [14]. The interpretation of a model is relevant, since it provides an extra output that gives an insight into what aspects of the input had the most influence when producing the  
Figure 1: Fact verification usually is composed of three sequential stages. predictions. In the case of fact verification, having this insight might help to detect and mitigate the impact of false information. The architecture we present in this paper takes advantage of the pretraining power of BERT [12], but simultaneously simplifies the model's interpretation using an attention layer between claim and evidence. Figure 2 depicts our model. The architecture concentrates on relevance evaluation and claims verification tasks, which are the second and third steps of the aforementioned (Section 2) method used to verify descriptions of facts. The architecture codifies claims and evidence independently using a pair of BERT blocks. Each block produces a sequence \(s_{1}\) and \(s_{2}\), respectively. Each sequence is the output of the last BERT codifier layer; all token outputs are used. The sequences of claims and evidence are limited to 128 tokens. With \(s_{1}\) and \(s_{2}\) as inputs, alignment scores are computed in the attention layer, and every element in \(s_{1}\) is evaluated against every element in \(s_{2}\). As a score function, we assessed the next functions:  
* Multiplicative \[A_{i,j}=s_{1i}\cdot s_{2j}\quad\forall\ i,j\] (1)
* Multiplicative + softmax \[A_{i,j}=softmax(s_{1i}\cdot s_{2j})\quad\forall\ i,j\] (2)  
Figure 2: Architecture proposed to evaluate relevance and claim verification. * Multiplicative + sigmoid \[A_{i,j}=sigmoid(s_{1i}\cdot s_{2j})\quad\forall\ i,j\] (3)
* Cosine \[A_{i,j}=\frac{s_{1i}\cdot s_{2j}}{\|s_{1i}\|\|s_{2j}\|}\quad\forall\ i,j\] (4)  
The attention process produces a matrix \(A\) of \(128\times 128\) dimensions which correspond to the sequence size, i.e., the number of tokens in the claim and the evidence. Every item \(a_{ij}\) represents the alignment of item \(i\) in \(s_{1}\) with the item \(j\) in \(s_{2}\). After calculating the attention matrix \(A\), it is multiplied by each of the vector sequences \(s_{1}\) and \(s_{2}\). The aim of matrix \(A\) is to weigh \(s_{1}\) and \(s_{2}\), giving more weight to those elements related between both sequences. This attention map will be useful to comprehend which text spans are considered relevant by the model. The weighting produces two sequences \(u_{1}\) and \(u_{2}\) (see Equations (5) and (6)). \[u_{1}=As_{1} \tag{5}\]  
\[u_{2}=A^{\intercal}s_{2} \tag{6}\]  
Now, it is necessary to obtain a single vector that will represent both the claim and the evidence. For this purpose, a _pooling_ is performed on the vector sequences \(u_{1}\) and \(u_{2}\). The pooling process obtains a vector for each input sentence \(u_{1}\) and \(u_{2}\). The dimension of these vectors corresponds to the size of the representation of the hidden state of _BERT_. For the case of the base version, it is 768 dimensions, and for the large version of 1024. \[p_{i}=\frac{1}{n}\sum_{k=1}^{n}u_{ik}\quad\forall\ i=[1,2] \tag{7}\]  
Using the 128 vectors of the sequence, the centroid of this set is obtained. This centroid is the average of each of the vectors' dimensions. The resulting vector will be Equation (7), where \(n\) is the size of the sequence. Now, each sequence is represented by a vector \(p_{1}\) and \(p_{2}\).