Embeddings Shifts as Proxies for Different Word Use in Italian Newspapers
Authors: Michele Cafagna, Lorenzo De Mattei, Malvina Nissim (Università di Pisa, CNR-ILC, University of Groningen, 2019)
Topic: Fact Checking and Fake News Detection

Abstract
We study how words are used differently in two Italian newspapers at opposite ends of the political spectrum by training embeddings on one newspaper’s corpus, updating the weights on the second one, and observing vector shifts. We run two types of analysis: one top-down, based on a preselection of frequent words in both newspapers, and one bottom-up, using a combination of the observed shifts and relative and absolute frequency. The analysis is specific to this data, but the method can serve as a blueprint for similar studies.

Introduction
Different newspapers, especially if positioned at opposite ends of the political spectrum, can render the same event in different ways. In Example (1), both headlines are about the leader of the Italian political movement “Cinque Stelle” splitting up with his girlfriend, but the Italian left-oriented newspaper *la Repubblica* (rep in the examples) and right-oriented *Il Giornale* (gio in the examples) describe the news quite differently. The news in Example (2), which is about a baby-sitter killing a child in Moscow, is also reported by the two newspapers mentioning and stressing different aspects of the same event. Often though, the same words are used, but with distinct nuances, or in combination with other, different words. In this work we discuss a method to study how the same words are used differently in two sources, exploiting vector shifts in embedding spaces. The two embeddings models built on data coming from *la Repubblica* and *Il Giornale* might contain interesting differences, but since they are separate spaces they are not directly comparable. Previous work has encountered this issue from a diachronic perspective: when studying meaning shift in time, embeddings built on data from different periods would encode different usages, but they need to be comparable. Instead of constructing separate spaces and then aligning them (Hamilton et al., 2016b), we adopt the method used by Kim et al. (2014) and subsequently by Del Tredici et al. (2016) for Italian, whereby embeddings are first trained on a corpus, and then updated with a new one; observing the shifts certain words undergo through the update is a rather successful method to proxy meaning change. Rather than across time, we update embeddings across sources which are identical in genre (newspapers) but different in political positioning. Specifically, we train embeddings on articles coming from the newspaper *La Repubblica* (leaning left) and update them using articles coming from the newspaper *Il Giornale* (leaning right). We take the observed shift of a given word (or the shift in distance between two words) as a proxy for a difference in usage of that term, running two types of analysis. One is top-down, and focuses on a set of specific words which are frequent in both corpora. The other one is bottom-up, focusing on words that result potentially interesting on the basis of measures that combine the observed shift with both relative and absolute frequency. As a byproduct, we also learn something about the interaction of shifts and frequency.

Conclusion
We experimented with using embedding shifts as a tool to study how words are used in two different Italian newspapers. We focused on a pre-selection of high-frequency words shared by the two newspapers, and on another set of words which were highlighted as potentially interesting through a newly proposed methodology that combines observed embedding shifts and relative and absolute frequency. Most differently used words in the two newspapers are proper nouns of politically active individuals as well as places, and concepts that are highly debated on the political scene. Beside the present showcase, we believe this methodology can be more generally used to highlight which words might deserve deeper, dedicated analysis when studying meaning change. One aspect that should be further investigated is the role played by the methodology used for aligning and/or updating the embeddings. As an alternative to what we proposed, one could employ different strategies to manipulate embedding spaces towards highlighting meaning changes. For example, Rodda et al. (2016) exploited Representational Similarity Analysis (Kriegeskorte and Kievit, 2013) to compare embeddings built on different spaces in the context of studying diachronic semantic shifts in ancient Greek. Another interesting approach, still in the context of diachronic meaning change, but applicable to our datasets, was introduced by Hamilton et al. (2016a), who use both a global and a local neighborhood measure of semantic change to disentangle shifts due to cultural changes from purely linguistic ones.