# Synonyms, Antonyms and Factual Knowledge in BERT Heads
## 3 Methodology
### Evaluation Metrics  
Once we have extracted the word pair pairs with the Linker Algorithm, we can compare them to some ground truth relations. Thus, we can verify whether a head possesses some specific knowledge and quantify it through specific evaluation metrics.  
Figure 3: Example of how the Linker Algorithm works on the words Italy and Rome. On the left, we show a visualization of the attention weights of these two words, into which the thickest lines indicate a higher value of attention between the word pairs. On the right, we show the word pairs returned the algorithm, which is the cluster of words with the attention weights extracted with the Mean Shift algorithm [25]. These results can be noisy, and, in this example, we can see that our algorithm can return pairs that are not the ones we want to identify, such as (Italy, country). We use different colours for different tokens.  
For this purpose, we defined an evaluation metric with the same idea of the \(F1\)-Score. We compute this metric as the harmonic mean between precision and recall, which we define this way:  
\[precision=\frac{\#Identified}{\#Pairs} \tag{4}\]  
\[recall=\frac{\#Identified}{\#Dataset} \tag{5}\]  
where \(\#Identified\) is the number of word pairs retrieved by the Linker Algorithm for the head that are correctly identified; \(\#Pairs\) is the total number of pairs retrieved by the Linker Algorithm and \(\#Dataset\) is the number of ground truth pairs present in the dataset. With these metrics, we can compute the \(F1\)-Score as:  
\[F1\,score=\frac{2*precision*recall}{precision+recall} \tag{6}\]  
Another evaluation metric we considered is accuracy, which is the number of word pairs correctly identified (by a head or a group of heads) over the total number of pairs in the dataset. In particular, we considered two different types of accuracy, \(acc_{N}\) and \(acc_{Tot}\), which are computed this way:  
\[acc_{N}=\frac{\#Identified_{N}}{\#Dataset} \tag{7}\]  
\[acc_{Tot}=\frac{\#Identified_{Tot}}{\#Dataset} \tag{8}\]  
into which \(\#Identified_{N}\) is the number of pairs correctly identified by \(N\) heads, and \(\#Identified_{Tot}\) is the number of pairs correctly identified by all the heads in the BERT model.  
The idea behind these two metrics is the following. We consider \(acc_{Tot}\) as a baseline, which represents all the possible knowledge contained in all the heads across the entire BERT architecture. If a relationship between two terms is not identified by any of the heads, that information is contained in other parts of the architecture or not known by the model. Nonetheless, we are not interested in just verifying if that information is present or not but aim to create a selection mechanism into which the Self Metric identifies the most promising heads in terms of semantics and factual knowledge. Therefore, we design also the \(acc_{N}\) metric, into which we consider only the first \(N\) heads identified by the Self Metric. If \(acc_{N}\) is very similar or equal to \(acc_{Tot}\), we can say that our method for selecting the most promising heads performs well because we select those heads that contain all (or almost all) the knowledge of the entire model. Instead, if \(acc_{N}\) is much lower than \(acc_{Tot}\), our Self Metric is not able to identify the most promising heads.  
After some preliminary experiments, as described in Section 5 (and, in particular, in Figure 4), we selected \(N=20\).  
Figure 4: Cumulative Normalized \(F\)-Score progressively considering more heads, which are ordered on the basis of the Self Metric value. These results are obtained considering the Synonyms (**a**) and Capitals (**b**) datasets for all the models we considered.