# Synonyms, Antonyms and Factual Knowledge in BERT Heads
## 5 Experimental Settings and Evaluation
### Robustness to Different Prompts  
We ran an additional experiment to check the behaviour of our approach with different prompts. We created new datasets of synonyms and antonyms, selecting five different prompts and forty entities, for a total of two-hundred samples per domain. We then analysed the results of our techniques, checking that they identified the same entities regardless of the prompt in which they were entered. The prompts we designed for the study on the Synonyms domain are:  
* I could describe it as [X], or, to put it another way, it is [Y].
* If I had to put it into words, I would say it is [X], or, to use a synonym, it is [Y].
* You might call it [X], but I would describe it as [Y]--they are essentially synonyms.
* In my opinion, it is [X], or, to use a similar term, it is [Y].
* If you were to ask me for a word to describe it, I might choose [X], or, to put it synonymously, [Y].  
Similarly, for the Antonyms domain, we designed the following prompts:  
* You used the term [X], but I would argue that it is actually quite the opposite. I would describe it as [Y].
* While you described it as [X], I believe that it is actually the antithesis. I would say it is [Y].
* Your description of it as [X] does not quite fit; I think the antonym is more accurate. I would label it as [Y].
* Although you referred to it as [X], I think there is a more fitting antonym. I would say it is [Y].
* Your characterization of it as [X] does not match my perception; the opposite seems more appropriate. I would describe it as [Y].  
The results of these experiments are available in Table 3. We can see that our method exhibits a remarkable level of robustness with respect to the prompt used, allowing for flexibility and adaptability in various contexts. Despite the diverse prompts employed, our methods obtain stable accuracy in both domains, demonstrating their robustness despite the context in which the entities are presented.  
\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & \multicolumn{10}{c}{**(a) Synonyms**} \\ \hline Model & \multicolumn{2}{c}{Prompt 1} & \multicolumn{2}{c}{Prompt 2} & \multicolumn{2}{c}{Prompt 3} & \multicolumn{2}{c}{Prompt 4} & \multicolumn{2}{c}{Prompt 5} \\ \(acc_{Tot}\) & \(acc_{20}\) & \(acc_{Tot}\) & \(acc_{20}\) & \(acc_{Tot}\) & \(acc_{20}\) & \(acc_{Tot}\) & \(acc_{20}\) & \(acc_{Tot}\) & \(acc_{20}\) \\ \hline bert-base-uncased & 0.9 & 0.8 & 0.9 & 0.83 & 0.9 & 0.88 & 0.9 & 0.85 & 0.9 & 0.88 \\ \hline bert-base-cased & 0.88 & 0.83 & 0.90 & 0.80 & 0.90 & 0.78 & 0.90 & 0.85 & 0.90 & 0.88 \\ \hline bert-multilingual-cased & 0.90 & 0.83 & 0.90 & 0.78 & 0.90 & 0.83 & 0.90 & 0.83 & 0.90 & 0.90 \\ \hline bluebert & 0.88 & 0.75 & 0.90 & 0.90 & 0.90 & 0.78 & 0.90 & 0.85 & 0.88 & 0.88 \\ \hline \multicolumn{10}{c}{**(b) Antonyms**} \\ \hline Model & \multicolumn{2}{c}{Prompt 1} & \multicolumn{2}{c}{Prompt 2} & \multicolumn{2}{c}{Prompt 3} & \multicolumn{2}{c}{Prompt 4} & \multicolumn{2}{c}{Prompt 5} \\ \(acc_{Tot}\) & \(acc_{20}\) & \(acc_{Tot}\) & \(acc_{20}\) & \(acc_{Tot}\) & \(acc_{20}\) & \(acc_{Tot}\) & \(acc_{20}\) & \(acc_{Tot}\) & \(acc_{20}\) \\ \hline bert-base-uncased & 0.9 & 0.8 & 0.9 & 0.83 & 0.9 & 0.88 & 0.9 & 0.85 & 0.9 & 0.88 \\ \hline bert-base-cased & 0.83 & 0.55 & 0.83 & 0.60 & 0.83 & 0.60 & 0.83 & 0.63 & 0.83 & 0.63 \\ \hline bert-multilingual-cased & 0.83 & 0.78 & 0.83 & 0.75 & 0.83 & 0.78 & 0.83 & 0.83 & 0.80 \\ \hline bluebert & 0.80 & 0.80 & 0.80 & 0.78 & 0.80 & 0.78 & 0.80 & 0.78 & 0.80 & 0.80 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Accuracy of pairs correctly identified by all heads and the top 20 heads in the Self Metric for each model on Synonyms and Antonyms domains, on each different prompt, to prove the robustness of our approach.