SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 5: Examples of aligned punctuation marks with varying cosine similarities. The aligned tokens are underlined.
In Figure 4, we break these trends down by individual punctuation marks, focusing only on aligned same words. We look at the most common punctuation marks. Of these punctuation marks, the comma and period show the widest distributions. Even when they play the same role in the paraphrase, they can be given very different embeddings, indicating how highly contextualized these punctuation marks are. The question mark and dash are less contextualized; this is most likely because these punctuation marks are used in more prescribed circumstances. In this dataset, in all but one example, the question mark is the last token; the dash is the first token in all but two examples.
Table 5 shows examples in context of each of these punctuation marks. Looking at the low similarity cases, one common pattern is that the phrase contains a contraction that is expanded in one phrase (e.g., "it is" and "it's").
Conclusion: BERT's representation of punctuation is surprisingly context sensitive, with substantial variation even when we control for meaning.
3.2.3 Polysemy
Previous work has shown that BERT embeddings form clusters based on word senses (Wiedemann et al., 2019). In the context of aligned words in a paraphrase, we would expect even a highly polysemous word to have similar embeddings in the two phrases.
Figure 6: Cosine similarity for different groups of words in the PPDB across all layers of BERT. Decreasing cosine similarity indicates increasing contextualization, and vice versa.
Figure 5: Distributions of cosine similarity for aligned (left) and unaligned (right) same words from the last layer of BERT grouped by the number of senses each word has.
To measure polysemy, we consider the number of WordNet synsets of a word, focusing on same aligned and same unaligned words. In order to have enough data to make a good comparison, we use the 4,000 sampled paraphrases from Section 3.2, as well as an additional random sample of long paraphrases with at least one unaligned same word. We then downsample the aligned same words to get 1,597 instances of both unaligned and aligned same words that are present in WordNet, with up to 52 synsets.[10]
In Figure 5, we show the cosine similarity distri
butions for both aligned and unaligned words with different levels of polysemy across the last layer of BERT. There is not a substantial difference between words with different synsets, which supports our conclusion that BERT successfully captures the semantics of aligned same words in paraphrases. We do see a difference between aligned and unaligned words. Aligned words peak at a high cosine similarity, while unaligned words roughly follow a normal distribution centered around 0.5. Note that for unaligned words with two or three synsets, there is not enough data to draw conclusions about the cosine similarity distributions. Overall, these plots show that even highly polysemous aligned same words have very similar embeddings in the context of a paraphrase.
Conclusion: How polysemous a word is does not substantially impact BERT's ability to consistently represent it.
3.3 Contextualization in BERT Layers
In this section, we consider how context-specific the embeddings in a paraphrase are. Ethayarajh (2019) showed that BERT word embeddings are more context-specific in higher layers. They measure this using the self-similarity of words, defined as the average cosine similarity between a word's contextualized representations across its unique contexts, and show that self-similarity consistently decreases with higher layers of BERT, indicating that the contextualization of words is increasing.
We compare this observation to the paraphrase setting that we have been exploring in this paper. Because there are only two phrases in a paraphrase,
10We look up WordNet synsets using the Python NLTK library (Bird et al., 2009).
we cannot implement the full self-similarity metric. Instead, we measure the cosine similarity between two words aligned in a paraphrase, shown in Figure 6. This revised metric measures is similar to self-similarity.
We see two trends in Figure 6. The first, decreasing cosine similarity, is seen with same words, whether aligned or unaligned, and is similar to what Ethayarajh (2019) report with decreasing self-similarity scores. This trend is stronger with unaligned words than with aligned words, indicating the model is capturing the fact that while these words have the same form, they are being used differently. The second trend that we see is the opposite, increasing cosine similarity, and we see this trend with different words, both aligned and unaligned. This indicates decreasing contextualization.
Conclusion: As seen in prior work, the standard way of using vectors from BERT's layers does not capture the same level of contextualization in all layers. However, in contrast to prior work, when controlling for semantics of the context, it seems that later layers are capturing more of the context, appropriately making words less similar when they are being used in different ways.
Paraphrases with word alignments are a useful tool for studying the behavior of contextual language models. In this paper, we used them to study several contextual models, with a particular focus on BERT. Where possible, we compared our results with prior work, finding patterns that are consistent with the literature. Specifically, our results confirm that BERT consistently represents paraphrases, even for cases with polysemous words, but that individual word representations are overly sensitive to position, particularly for punctuation. One exception is that we found that words in a sentence are more similar to each other in later layers of BERT, in contrast to prior work that did not control for meaning using paraphrases.
The analysis method we introduced opens up new opportunities, such as the comparison of aligned and unaligned, same and different words, which shows the sensitivity of these models to the specific word used. Paraphrases have the potential to inform exploration of other representation methods, showing which way of using the output of language models most accurately captures se
mantics consistently. We hope our findings will inform future work on contextualized models, and the applications that rely on them.
Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 597-604.
Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: Analyzing text with the natural language toolkit. O'Reilly Media, Inc.
Ying-Hong Chan and Yao-Chung Fan. 2019. BERT for question generation. In Proceedings of the 12th International Conference on Natural Language Generation, pages 173-177.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186.
Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 55–65.
Christiane Fellbaum. 1998. WordNet. Wiley Online Library.
Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 758-764.
Aina Gari Soler and Marianna Apidianaki. 2020. BERT knows punta cana is not just beautiful, it's gorgeous: Ranking scalar adjectives with contextualised representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7371-7385.
Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. GlossBERT: BERT for word sense disambiguation with gloss knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3509-3514.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets
of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365-4374.
Somaia Mahmoud and Marwan Torki. 2020. AlexU-AUX-BERT at SemEval-2020 task 3: Improving BERT contextual similarity using multiple auxiliary contexts. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 270-274. International Committee for Computational Linguistics.
Timothee Mickus, Denis Paperno, Mathieu Constant, and Kees van Deemter. 2020. What do you mean, BERT? In Proceedings of the Society for Computation in Linguistics 2020, pages 279-290.
Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168.
Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011. English gigaword fifth edition. Linguistic Data Consortium, Philadelphia.
Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 425-430.
Pushpendre Rastogi, Benjamin Van Durme, and Raman Arora. 2015. Multiview LSA: Representation learning via generalized CCA. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 556-566.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892.
Juan Soler-Company and Leo Wanner. 2017. On the relevance of syntactic and discourse features for author profiling and identification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 681-687.
Charles Spearman. 1910. Correlation calculated from faulty data. British Journal of Psychology, 1904-1920, 3(3):271-295.
Rachael Tatman and Amandalynne Paullada. 2017. Social identity and punctuation variation in the #bluelivesmatter and #blacklivesmatter twitter communities. In The 33rd Northwest Linguistics Conference.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 2214-2218. European Language Resources Association (ELRA).
Yulia Tsvetkov, Manaal Faruqui, Wang Ling, Brian MacWhinney, and Chris Dyer. 2016. Learning the curriculum with Bayesian optimization for task-specific word representation learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 130-139.
Gregor Wiedemann, Steffen Remus, Avi Chawla, and Chris Biemann. 2019. Does BERT make any sense? Interpretable word sense disambiguation with contextualized embeddings. In Konferenz zur Verarbeitung natürlicher Sprache / Conference on Natural Language Processing.
J. Yi and J. Tao. 2019. Self-attention based model for punctuation prediction using word and speech embeddings. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 7270-7274.
Shehel Yoosuf and Yin Yang. 2019. Fine-grained propaganda detection with fine-tuned BERT. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 87-91.
Lang Yu and Allyson Ettinger. 2020. Assessing phrasal representation and composition in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4896-4907.
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. *Bertscore: Evaluating text generation with bert.* In International Conference on Learning Representations.
A Extra breakdowns of results
Table 6 presents an expanded version of Table 3, with results for each layer of BERT. All layers perform better with longer paraphrases, but the improvement is largest for the last layer
Figure 7 shows the specific values for similarity broken down by distance apart of words in the phrases. This shows the pattern of decreasing similarity as words are further away.
Method	Average Length	All
1-2.5	2.5-4	4-6
BERT Layer 1	0.18	0.35	0.47	0.34
BERT Layer 2	0.18	0.35	0.49	0.33
BERT Layer 3	0.18	0.37	0.48	0.31
BERT Layer 4	0.18	0.38	0.48	0.3
BERT Layer 5	0.18	0.39	0.48	0.3
BERT Layer 6	0.19	0.39	0.49	0.29
BERT Layer 7	0.2	0.4	0.49	0.3
BERT Layer 8	0.21	0.4	0.5	0.28
BERT Layer 9	0.21	0.4	0.5	0.28
BERT Layer 10	0.22	0.38	0.48	0.29
BERT Layer 11	0.22	0.36	0.46	0.29
BERT Layer 12	0.1	0.35	0.51	0.16
BERT Concat.	0.2	0.4	0.51	0.31
w2v Average	0.35	0.32	0.41	0.43
PPDB model	0.41	0.50	0.51	0.50
Num. phrases	17,517	5,349	2,870	25,736
Avg. human	2.40	2.94	3.26	2.60
Table 6: This is a version of Table 3 with per-layer results. Spearman's  $\rho$  between human-annotated PPDB paraphrases and different embedding methods (BERT and w2v), broken down by average paraphrase length (the average number of words in each of the two phrases in the paraphrase). At the bottom of the table, we include the length distribution of the human-annotated paraphrases, as well as the average human annotation for each set of grouped lengths.
Figure 7: Cosine similarity using the last layer of BERT for aligned same words broken down by the number of words apart the words are in the two phrases (shown in bar plot and left y-axis). Error bars indicate confidence intervals. The line graph and the right y-axis show how many examples we have for each category.