Learning Trajectories of Figurative Language for Pre-Trained Language Models
Authors: Nicola Arici, Luca Putelli, Alfonso Gerevini, Ivan Serina


Reviewer paper summaries: This paper probes pretrained language models for recognition of four types of figurative language: hyperboles, metaphors, oxymorons, and pleonasms. In order to control for probe complexity, they use minimum description length (MDL) probing. They also analyze checkpoints throughout the training trajectory in order to examine the learning trajectories of the models on recognizing figurative language.

They find that hyperboles and oxymorons are most recognizable from probing, and metaphors and pleonasms are less identifiable. They find that these expressions become recognizable in the early stages of training and do not improve in later stages.

This paper presents a detailed analysis of four types of figurative language—hyperboles, metaphors, oxymorons, and pleonasms—across various layers and training checkpoints of GPT-NeoX, BERT, and RoBERTa. The study reveals that hyperboles and oxymorons are more easily understood by the models compared to metaphors and pleonasms. Additionally, it finds that the central layers of these models possess the most figurative language knowledge, which is acquired during the early stages of training.
Summary of strengths: - Analyzing learning trajectories of figurative language through probing has indeed not been done to my knowledge, and this could be interesting to researchers working on figurative language or natural language understanding in general.
- A new data resource for oxymoron is created, and the authors helpfully gather resources for different types of figurative language, including less-examined ones like pleonasm.

* The paper is well-written, well-explained, and provides a comprehensive review of related work and contextual discussions.
* Employs a more robust probing technique, Minimum Description Length (MDL), compared to conventional methods.
* Comprehensive analysis of different layers and checkpoints of training in 3 model types.
* The authors generated negative instances for the oxymoron dataset.
Summary of weaknesses: - (A) The results, as presented, do not present any takeaways for the reader in terms of either practical considerations (what we should do when deploying/training models on tasks involving figurative language), or detailed scientific understanding of the models. The main result is given in Figure 1, and depicts compression in MDL. However, this result isn’t tied back to the model’s performance on figurative language. For instance, the results for pleonasm are consistently low. Does this mean that all of the models are bad at recognizing pleonasm, or is it that the representations for pleonasm are just organized in a way that is harder to encode, but still effective for downstream tasks? Additionally, we can see that in the last layer, for smaller models the compressibility of oxymoron actually decreases. This finding is not really explained though, does this really mean that these models become worse at recognizing oxymoron after being trained for more tokens? I somehow doubt that this would be the case, which raises the question of what this worsening of compressibility actually means. Section 5.1 simply presents these findings without interpretation, which is not very useful. I have several suggestions to make the findings more applicable for other researchers:
    - (A1) Establish a connection between probing compressibility and task performance (or establish that they are unrelated in this context). For instance, does hyperbole and oxymoron having higher compression values seem to be related to models being better at picking up on these figures compared to others? Does oxymoron recognition actually decrease in later stages of training? 
    - (A2) Comment on the data used during pretraining and how it could relate to the current findings. One question is how the data sources used impact the recognition of these figures of speech, as the models selected are trained on web datasets. Since very small models are part of the analysis, potentially these small models could be trained on literary data such as the books portion of dolma or project gutenberg, and the learning trajectories could be compared. This would be higher effort and perhaps not as important as (A1), but could add some more nuance to the analysis, as the learning trajectory would depend crucially on the dataset a model is trained on and how much figurative language it contains.
- (B) The paper seems to lack a clear hypothesis, which is related to the problem with lack of takeaways. More needs to be done to contextualize why we would be interested in the learning trajectories of figurative language, and what this could tell us. Why does this matter, and what do we expect to see when considering other studies that look at learning trajectories? If we have a hypothesis about what we expect to see in figurative language learning trajectories, what does it mean if the hypothesis is supported or violated?
- (C) Although I’m mindful of compute constraints, larger models could perhaps be used since this is a probing study and the large models aren’t being trained. The models are predominantly pretty small ones, which may have a weaker performance overall at figurative language detection.

* If the classifier size varies based on the representation dimensions (as noted in L273), comparing different models with different sizes might introduce unfairness. A potential solution is to introduce a linear projection layer to standardize all representation sizes to a fixed single size before applying a similarly sized classifier.
* Aggregating the entire sentence can be problematic because figurative language can appear in different parts of a sentence. Some datasets highlight specific words in the sentence with labels indicating whether they are figurative or not. A brief discussion on the potential information loss due to discarding the span of words in the dataset would be beneficial. Justifying the choice of aggregation and addressing its limitations would strengthen the paper.
Comments, suggestions and typos: - Line 002: “is” should be “are”
- Line 009: “full understanding of text” is not very informative, maybe the phrasing could be changed
Perhaps the learning trajectory of the best layer could be presented more prominently, e.g. in figure 1 instead of last layer. In figure 2 we see that oxymoron doesn’t have a decreasing compression in small models when considering the best layer, presenting the last layer raises the question of why the decrease is happening which is not answered
- Some probing papers on figurative language are missed (idioms that I know of, but the authors should also try to do a more thorough search). For example:

Garcia et al 2021, probing for idiomaticity in vector space models

Dankers et al 2022, can transformer be too compositional? Analyzing idiom processing in NMT

Liu et al 2022, are representations built from the ground up? An empirical examination of local compositional in language models

* 081: a hyperbole
* 123: incluiding > including
* 539: shows
* 636: con sidered > considered
Overall assessment scores: 2

3.5