An Analysis of Italian Contextualized Word Embeddings: Investigating the Representation of Word Senses
Authors: Rocco Tripodi, Alessandro Raganato (Sapienza Università di Roma, Amazon Alexa AI, 2021)
Topic: Distributional Semantics

Abstract
Contextualized embedding models, such as ELMo and BERT, allow the construction of vector representations of lexical items that adapt to the context in which words appear. It was demonstrated that the upper layers of these models capture semantic information. This evidence paved the way for the development of sense representations based on words in context. In this paper, we analyze the vector spaces produced by 11 pre-trained models and evaluate these representations on two tasks. The analysis shows that all these representations contain redundant information. The results show the disadvantage of this aspect.

Introduction
The introduction of contextualized embedding models, such as ELMo <cit.> and BERT <cit.>, allows the construction of vector representations of lexical items that adapt to the context in which words appear. It has been shown that the upper layers of these models contain semantic information <cit.> and are more diversified than lower layers <cit.>. These word representations overcame the meaning conflation deficiency that affects static word embedding techniques <cit.>, such as word2vec <cit.> or GloVe <cit.> thanks to the adaptation to the context of use. The evaluation of these models has been conducted mainly on downstream tasks <cit.>. With extrinsic evaluations, the models are fine-tuned, adapting the vector representations to specific tasks. The resulting vectors are then used as features in classification problems. This hinders a direct evaluation and analysis of the models because the evaluation also takes into account the ability of the classifier to learn the task. A model trained for this kind of task may learn only to discriminate among features that belong to each class with poor generalization. The interpretability of neural networks is an emerging line of research NLP that aims at analyzing the properties of pre-trained language models <cit.>. Different studies have been conducted in recent years to discover what kind of linguistic information is stored in large neural language models. Many of them are focused on syntax <cit.> and attention <cit.>. For what concerns semantics, the majority of the studies focus on common knowledge <cit.> and inference and role-based event prediction <cit.>. Only a few of them have been devoted to lexical semantics, for example, DBLP:conf/nips/ReifYWVCPK19 show how different representations of the same lexical form tend to cluster according to their sense.  In this work, we propose an in-depth analysis of the properties of the vector spaces induced by different embedding models and an evaluation of their word representations. We present how the properties of the vector space contribute to the success of the models in two tasks: sense induction and word sense disambiguation. In fact, even if contextualized models do not create one representation per word sense <cit.>, their contextualization create similar representations for the same word sense that can be easily clustered. 0.65 Model training data vocab. size n. param. vec. dim. objective BERTbase <cit.> 16GB 30K 110M 768 masked language model and next sentence prediction BERTlarge <cit.> 16GB 30K 340M 1024 masked language model and next sentence prediction GPT-2base <cit.> 40GB 50K 117M 768 language model GPT-2medium <cit.> 40GB 50K 345M 1024 language model GPT-2large <cit.> 40GB 50K 774M 1280 language model RoBERTabase <cit.> 160GB 50K 125M 768 masked language model  RoBERTalarge <cit.> 160GB 50K 355M 1024 masked language model XLNetbase <cit.> 126GB 32K 110M 768 bidirectional language model XLNetlarge <cit.> 126GB 32K 340M 1024 bidirectional language model XLMenglish 16GB 30K 665M 2048 language model CTRL <cit.> 140GB 250K 1.63B 1280 conditional transformer language model. Statistics and hyperparameters of the models.

Conclusion
We conducted an extensive analysis of the semantic capabilities of contextualized embedding models. We analyzed the vector space constructed using pre-trained models and found that their vectors contain redundant information and that their first two principal components are dominant. The results on sense induction are promising. They demonstrated the effectiveness of contextualized embeddings to capture semantic information. We did not find higher performances from more complex models, rather, we found that RoBERTa, a model that was developed by simplifying a more complex model, BERT, was one of the best performers. Neither the dimension of the hidden layers, the size of the training data, nor the size of the vocabulary seems to play a big role in modeling semantics. As stated in previous works, inserting an anisotropy penalty to the objective function of the models could improve directly the representations. We also noticed that, even if BERT models and XLNet have different objectives and are trained on different data, they have similar performances. It emerged that these models are less redundant than others. The conclusion that we can draw from our analysis and evaluation is that pre-trained language models can capture lexical-semantic information and that unsupervised models can be used to distinguish among their representations. On the other hand, these representations are redundant and anisotropic. We hypothesize that reducing these aspects can lead to better representations. This operation can be carried out post-hoc but we think that training new models keeping this point in mind could lead to the development of better models.