Analysing Word Meaning over Time by Exploiting Temporal Random Indexing
Authors: Pierpaolo Basile, Annalina Caputo, Giovanni Semeraro (Università di Bari Aldo Moro, 2014)
Topic: Distributional Semantics

Abstract
This paper proposes an approach to the construction of WordSpaces which takes into account temporal information. The proposed method is able to build a geometrical space considering several pe-riods of time. This methodology enables the analysis of the time evolution of the meaning of a word. Exploiting this ap-proach, we build a framework, called Tem-poral Random Indexing (TRI) that pro-vides all the necessary tools for building WordSpaces and performing such linguis-tic analysis. We propose some examples of usage of our tool by analysing word meanings in two corpora: a collection of Italian books and English scientific papers about computational linguistics.

Introduction
The analysis of word-usage statistics over huge corpora has become a common technique in many corpus linguistics tasks, which benefit from the growth rate of available digital text and compu-tational power. Better known as Distributional Semantic Models (DSM), such methods are an easy way for building geometrical spaces of con-cepts, also known as Semantic (or Word) Spaces, by skimming through huge corpora of text in order to learn the context of usage of words. In the re-sulting space, semantic relatedness/similarity be-tween two words is expressed by the closeness be-tween word-points. Thus, the semantic similarity can be computed as the cosine of the angle be-tween the two vectors that represent the words. DSM can be built using different techniques. One common approach is the Latent Semantic Analysis (Landauer and Dumais, 1997), which is based on the Singular Value Decomposition of the word co-occurrence matrix. However, many other methods that try to take into account the word order (Jones and Mewhort, 2007) or predications (Cohen et al., 2010) have been proposed. Recursive Neural Net-work (RNN) methodology (Mikolov et al., 2010) and its variant proposed in the word2vect frame-work (Mikolov et al., 2013) based on the con-tinuous bag-of-words and skip-gram model take a complete new perspective. However, most of these techniques build such SemanticSpaces tak-ing a snapshot of the word co-occurrences over the linguistic corpus. This makes the study of seman-tic changes during different periods of time diffi-cult to be dealt with. In this paper we show how one of such DSM techniques, called Random Indexing (RI) (Sahlgren, 2005; Sahlgren, 2006), can be easily extended to allow the analysis of semantic changes of words over time. The ultimate aim is to provide a tool which enables to understand how words change their meanings within a document corpus as a function of time. We choose RI for two main reasons: 1) the method is incremental and requires few computational resources while still retaining good performance; 2) the methodology for building the space can be easily expanded to integrate temporal information. Indeed, the disadvantage of classical DSM approaches is that WordSpaces built on different corpus are not comparable: it is always possible to compare similarities in terms of neighbourhood words or to combine vectors by geometrical operators, such as the tensor product, but these techniques do not allow a direct comparison of vectors belonging to two different spaces. Our approach based on RI is able to build a WordSpace on different time periods and makes all these spaces comparable to each another, actually enabling the analysis of word meaning changes over time by simple vector operations in WordSpaces. The paper is structured as follows: Section 2 provides details about the adopted methodology and the implementation of our framework. Some examples of the potentiality of our framework are reported in Section 3. Lastly, Section 4 closes the paper.

Conclusion
We propose a method for building WordSpaces taking into account information about time. In a WordSpace, words are represented as mathemati-cal points and the similarity is computed accord-ing to their closeness. The proposed framework, called TRI , is able to build several WordSpaces in different time periods and to compare vectors across the spaces to understand how the meaning of a word has changed over time. We reported some examples of our framework, which show the potential of our system in capturing word usage changes over time.