Neural Random Projections for Language Modelling

1 INTRODUCTION . Representation learning is integral to a wide range of tasks from Language Modelling ( LM ) to speech recognition , or machine translation . Better language models frequently lead to improved performance on underlying downstream tasks , which makes LM valuable in itself . The goal of LM is to model the joint probability distribution of words in a text corpus . Models are trained by maximising the probability of a word given all the previous words in the training dataset : P ( w1 , . . . , wN ) = N∏ i=1 P ( wi|hi ) , where wi is the current word and ci = ( w1 , · · · , wi−1 ) is the current word history or context . A Markov assumption is usually used as approximation : instead of using the full word history as context , the probability of observing the ith wordwi is approximated by the probability of observing it in a shortened context of n−1 preceding words . To train a language model under this assumption , a sliding window ( wi , ci ) of size n is applied sequentially across the textual data . Each window is also commonly referred to as n-gram . Neural Network Language Models ( NNLM ) ( Bengio et al. , 2003 ) deal with both data sparseness and generalisation problems , by using neural networks to model the probability distribution for a word given its context , as a smooth function of learned real-valued vector representations for each word . Neural networks thus learn a conditional probability function P ( wi|ci ; θF , θL ) , where θF is a set of parameters representing the vector representations for each word and θL parameterises the log-probabilities of each word based on those learned features . A softmax function is then applied to these log-probabilities to produce a categorical probability distribution over the next word given its context . The resulting probability estimates are smooth functions of the continuous word vector representations , and so , a small change in such vector representations results in a small change in the probability estimation . Energy-based models provide a different perspective on statistical language modelling . Energybased models such as ( Mnih & Hinton , 2007 ) use neural networks to capture dependencies between variables by associating a scalar energy score to each output configuration . The key difference from the previously referred approach to NNLM , is that instead of trying to probability of a target word directly , the models predict the distributed representation for the target word ( embedding ) and attribute an energy score to the output configuration of the model based on how close the prediction is from the actual representation for the target word . In this paper , we describe an encoding mechanism that can be used with neural network probabilistic language models to encode each word without using a predetermined vocabulary of size |V | . Each unique word is encoded using a fixed sparse k− dimensional random vector with k |V | . These random features are designed so that the inner products of the original feature space is approximately preserved ( Achlioptas , 2003 ) . Critically , we show that we can achieve results comparable to the typical NNLM , without learning a unique embedding vector for each word . 2 BACKGROUND AND RELATED WORK . One cornerstone of neural-network-based models is the notion of distributed representation . In a distributed representation , each entity is represented as a combination of multiple factors . Learning distributed representations of concepts as patterns of activity in a neural network has been object of study in ( Hinton , 1986 ) . The mapping between unique concept ids and respective vector representations has been later referred to as embeddings . The idea of using embeddings in language modelling is explored in the early work or Bengio et al . ( Bengio et al. , 2003 ) . This and similar approaches , encode input words using a 1-of-V encoding scheme , where each word is represented by its unique index . The size of the embedding space is thus proportional to the size of the vocabulary , making neural language modelling computationally taxing for large vocabularies . One straightforward method to deal with large vocabularies in LM , is to consider character or subword-level modelling instead of words as the input units such as the work in ( Kim et al. , 2016 ) . Another motivation behind this approach , is the fact that word-based models often fail to capture regularities in many inflectional and agglutinative languages . In this paper we focus on word-based modelling , but the proposed encoding mechanism can be incorporated in models that use for example morphemes instead of words as inputs . Another difficulty with building neural language models lies in the fact that outputting probability distributions require explicit normalisation : we need to consider the output probabilities for all words in the vocabulary to compute log-likelihood gradients . Solutions to this problem include structuring the vocabulary into a tree , speeding-up word probability computations ( Morin & Bengio , 2005 ) , or approximating the log-likelihood gradient using approaches like Noise Constrastive Estimation ( NCE ) ( Mnih & Teh , 2012 ) . In order to analyse the predictive capability of our proposed models , we will use fully normalised models . This means we are still dependent on a known vocabulary to train the model . The normalisation requirements can be alleviated using techniques such as the ones previously mentioned , but , this will be the subject for future explorations . Exploiting low-dimensional representations for structures in high-dimensional problems has become a highly active area of research in machine learning , signal processing , and statistics . In summary , the goal is to use a low-dimensional model of relevant data in order to achieve , better prediction , compression , or estimation compared to more complex ” black-box ” approaches to deal with highdimensional spaces ( Hegde et al. , 2016 ) . In particular , we are interested in the compression aspect of these approaches . Usually , exploiting low-dimensional structure comes at a cost : incorporating structural constraints into a statistical estimation procedure often results in a challenging algorithmic problem but neural network models have been notoriously successful at dealing with these kind of constraints . The use of random projections as a dimensionality reduction tool has been extensively studied before , especially in the context of approximate matrix factorisation –the reader can refer to ( Halko et al. , 2011 ) for a more in depth review of this technique . The basic idea of random projections as dimensionality reduction technique comes from the work of Johnson and Lindenstrauss , in which it was shown that the pairwise distances among a collection of N points in an Euclidean space are approximately maintained when the points are mapped randomly to an Euclidean space of dimension O ( logN ) . In other words , random embeddings preserve Euclidean geometry ( Johnson & Lindenstrauss , 1984 ) . Random projections have also been used in multiple machine learning problems as a way to speedup computationally expensive methods . Examples include the use of random mappings to accelerate the training of kernel machines ( Rahimi & Recht , 2008 ) . In ( Papadimitriou et al. , 2000 ) , random projections are used as a first step for Latent Semantic Analysis ( LSA ) , which is essentially a matrix factorisation of word co-occurrence counts . A similar approach is followed in ( Kaski , 1998 ) : a random mapping projection is applied to document vectors to reduce their dimensionality before an unsupervised clustering algorithm is applied these vectors . The random mappings we use in this paper , are motivated by the work in ( Achlioptas , 2003 ) , which showed that a random projection matrix can be constructed incrementally using a simple sparse ternary distribution . In this paper , we focus on exploring simple feedforward neural network architectures instead of more complex and large recurrent neural network models such as ( Jozefowicz et al. , 2016 ) . This will allow us to iterate quickly on experiments , and better understanding the effects of random projections on network performance . As a starting point to build a baseline model , we chose an architecture similar to ( Bengio et al. , 2003 ) along with the the energy-based formulation in ( Mnih & Hinton , 2007 ) . We chose this approach because it gives us one of the simplest forms of a neural language model . Furthermore , in the future , the implementation of log-likelihood approximations such as NCE becomes straightforward under this formulation . 3 MODEL DESCRIPTION . This section describes our baseline neural language model , along with our alternative formulation using a random projection encoding . The main idea behind the baseline model is to use a neural network to learn lower-dimensional representations for words , while learning a discrete probability distribution of words , similarly to ( Bengio et al. , 2003 ; Mnih & Hinton , 2007 ) . While the baseline model uses a 1-of-V vector to represent each input word , and a |V |×m embedding projection layer ; the neural random projection model encodes each word using a random sparse k − dimensional vector and k ×m embedding layer . 3.1 RANDOM INPUT ENCODING . The distributed representations or embeddings in our model are k ×m subspace , with k < < |V | . A representation for each word as learned by the neural network is the linear combination of ( s × m ) -dimensional basis where s is the number of non-zero entries for each vector . Each unique word is assigned a randomly generated sparse ternary vector we refer to as random index vector . Random indices are sampled from the following distribution with mean 0 and variance 1 : ri = { +1 with probability α/2 0 with probability 1− α −1 with probability α/2 ( 1 ) where α is the proportion of non-zero entries in the random index vector . While similar to the distribution proposed in ( Achlioptas , 2003 ; Kanerva et al. , 2000 ) , we use very sparse random vectors and no scaling factor . Since the input space is an orthogonal unit matrix , the inner product between any two random vectors ( ri , rj ) is expected to be concentrated at 0 . The motivation for this is to make any discrete input indistinguishable from any other input in terms of distance between them . We will use s to refer to number of non-zero entries for randomly generated index vectors . This means that for s = 2 we have exactly one entry with value 1 , one with value−1 and all the rest have value 0 . Figure 1 shows the distribution of inner products between randomly generated vectors with different numbers of non-zero entries . 3.2 BASELINE MODEL . In order to assess the performance of our added random projection module , we build a baseline model without random projections . The architecture for this model is based in the one in ( Bengio et al. , 2003 ) but instead of predicting the probability of each word directly we use the energy-based definition from Mnih ’ s work ( Mnih & Hinton , 2007 ) . The baseline model learns to predict the feature vector ( embedding ) for a target word along with a probability distribution for all the known words conditioned on the learned feature vectors of the input context words . The baseline language model in figure converts each word into a vector of real-valued features using a feature lookup table F . The goal is to learn a set of feature vectors in F that are good at predicting the feature vector for the target word wn . Each word in a context of size n− 1 is converted to n− 1 m-dimensional feature vectors that are then concatenated into a ( n − 1 ) ×m vector and passed to the next layer h that applies a non-linear transformation : hj = σ ( Whf + bh ) ( 2 ) where f is a vector of n− 1 concatenated feature vectors , b is a bias for the layer h and σ is a nonlinear transformation ( e.g . sigmoid , hyperbolic tangent , or rectifier linear unit ( ReLU ) ) . We found ReLU units f ( x ) = max ( 0 , x ) to perform well and at an increased training speed , so we use this transformation in our models . The result of the non-linear transformation in h is then passed to a linear layer ŷ that predicts the feature vector f̂n for the target word wn as follows : ŷn =Wŷh+ bŷ ( 3 ) The energy function E is defined as the dot product between the predicted feature vector f̂n , and the actual current feature vector for the target word fn : En = f̂ T n fn ( 4 ) To obtain a word probability distribution in the network output , the predicted feature vector is compared to the feature vectors of all the words in the known dictionary by computing the energy value ( equation 4 ) for each word . This results in a set of similarity scores which are exponentiated and normalised to obtain the predicted distribution for the next word : P ( wn|w1 : n− 1 ) = exp ( En ) ∑|V | j exp ( Ej ) ( 5 ) The described baseline model can be though of as a typical feedforward model as defined in ( Bengio et al. , 2003 ) with an added linear layer ŷ with dimension m. Furthermore , instead of using two different sets of embeddings to encode context words and compute the word probability distributions , we learn a single set of feature vectors F that are shared between input and output layers to compute the energy values .

Summary: The main idea behind the paper is to use random projections as the initial word representations, rather than the vocab-size 1-hot representations, as is usually done in language modeling. The benefit is that the matrix which projects words into embedding space can then be much smaller, since the space of random projections can be much smaller than the vocab size. The idea is an interesting one, but this work is at too much of a preliminary stage for a top-tier conference such as ICLR. In its present state it would make for a potentially interesting paper at a targeted workshop.