Anchor & Transform: Learning Sparse Representations of Discrete Objects

1 INTRODUCTION . Learning continuous representations of discrete objects such as text , users , and items lies at the heart of many applications including natural language processing , user modeling , and recommendation systems . The standard way to learn these continuous representations involves : 1 ) defining the vocabulary V = { e1 , ... , e∣V ∣ } as the set of all objects , and 2 ) learning a ∣V ∣ × d embedding matrix that defines a d dimensional continuous representation for each object . However , when ∣V ∣ is large ( e.g . million of words in language modeling or millions of users in user modeling ) , this embedding matrix does not scale elegantly and can constitute more than 90 % of all trainable parameters in the model . As a result , there has been large interest in learning sparse representations of these discrete objects rather than the full embedding matrix for cheaper storage as well as faster training and inference . In the following we outline the desiderata of methods to learn sparse representations of discrete objects : 1 ) General-purpose : Good representation learning methods should be as flexible , modular , and as general as possible to be easily adapted for various applications . 2 ) End-to-end trainable : It is advantageous to learn neural representations in an end-to-end manner to directly optimize for the task at hand ( Liu et al. , 2018 ) . Similarly , we would like to enforce sparsity in our representations via task-specific measures . 3 ) Domain knowledge : The user should be able to easily integrate domain knowledge about object relationships . Domain knowledge can include external sources such as WordNet ( Miller , 1995 ) , ConceptNet ( Liu & Singh , 2004 ) , and knowledge graphs ( Pujara & Singh , 2018 ) for text modeling , as well as clusters of users and items for user modeling ( Zhang et al. , 2012 ) . As a step towards the aforementioned desiderata , we propose a general method to learn sparse representations of discrete objects . Our approach , depicted in Figure 1 , consists of two steps : 1 ) ANCHOR : Learn embeddings of a small set of anchor objects that are representative of all objects ( §3.1 ) . The user retains flexibility in defining these anchors and we show strong performance across a suite of initialization strategies including random , frequency , and clustering-based methods . 2 ) TRANSFORM : Learn a sparse transformation from anchors to all objects ( §3.2 ) . Each of the non-anchors should be easily induced by some transformation from ( a few ) nearby anchors . Domain knowledge can be incorporated through transformations between related objects ( §3.3 ) . The anchor embeddings and the sparse transformation are trained end-to-end for task specific learning . We call the resulting method Anchor & Transform , or ANT for short . ANT is scalable , flexible , end-to-end trainable , and allows the user to easily incorporate domain knowledge about object relationships . We also show that ANT recovers several task-specific baselines under certain structural assumptions on the anchors and transformation matrices ( Appendix D ) . In practice , ANT is easy to incorporate into existing models by replacing the EMBEDDING layers in PyTorch or TensorFlow with our newly designed ANTEMBEDDING layers . We demonstrate the effectiveness of ANT on text classification and language modeling tasks ( §4 ) . ANT achieves better performance with fewer parameters as compared to existing vocabulary selection and embedding compression baselines . Code to replicate our experiments is publicly released at < anonymous > . 2 RELATED WORK . Related work on learning sparse representations of discrete structures largely fall into three categories : General purpose matrix compression techniques have been used to reduce the embedding matrix E , such as low rank approximations ( Acharya et al. , 2018 ; Markovsky , 2011 ; Grachev et al. , 2019 ) , quantizing ( Han et al. , 2016 ) , pruning ( Wen et al. , 2016 ; Dong et al. , 2017 ; Anwar et al. , 2015 ) , or hashing ( Guo et al. , 2017 ; Chen et al. , 2015 ; Qi et al. , 2017 ) . However , these methods do not allow the user to integrate domain knowledge about object relationships . Furthermore , their generality implies that they are complementary and can be combined with our proposed methods . Reducing representation size : These methods reduce the dimension d for different objects . For example , Chen et al . ( 2016a ) divides the embedding into buckets which are assigned to objects in order of importance , Joglekar et al . ( 2019 ) learns d by solving a discrete optimization problem with reinforcement learning ( RL ) , and Baevski & Auli ( 2018 ) gradually reduces dimensions for rarer words . These methods typically resort to RL or are difficult to tune with too many hyperparameters . Each object is also modeled independently without information sharing between objects . Task specific methods define sparse embeddings for specific applications . For example , learning representations for common words gives strong performance for language modeling ( Luong et al. , 2015 ; Chen et al. , 2016b ) , and performing dropout-based vocabulary selection retains performance on text classification ( Chen et al. , 2019 ) . Other methods aim to reconstruct pre-trained embeddings using codebook learning ( Shu & Nakayama , 2018 ; Chen et al. , 2018 ) or low rank tensors ( Nam & Quoc , 2017 ; Sedov & Yang , 2018 ) . However , these methods are not general and end-to-end trainable , therefore preventing us from learning representations for general tasks . For example , methods that only model a subset of objects can not be used for retrieval tasks because it would imply never retrieving the dropped objects . Rare objects might be highly relevant to a few users so it might not be ideal to completely ignore them , but rather use only a few parameters to model them . Similarly , subword ( Bojanowski et al. , 2016 ) and wordpiece ( Wu et al. , 2016 ) embeddings , while useful for text , do not generalize to general objects and applications such as item and query retrieval . The concept of using anchors to succinctly represent a large set of points on a manifold or space has been around for a while Mallat ( 1999 ) ; Aharon et al . ( 2006 ) . However , to best of our knowledge , they are mostly limited to cases when a distance or similarity function or a proxy thereof like ranked lists are provided ( Guo et al. , 2017 ; Xu et al. , 2011 ; Liang et al. , 2018 ) . The use of anchors simultaneously with a downstream learning task has been limited to well understood convex objectives with continuous inputs like in sparse Gaussian Process Regression ( Snelson & Ghahramani , 2006 ) , where the kernel acts as the distance function . It is not evident from literature how to apply these anchor tricks to arbitrary learning problem over discrete objects in a flexible manner , like in deep networks . Our main contribution is to demonstrate how these anchors and sparse transformations can be trained jointly with neural models as a general input embedding layer , and how we can obtain better sparse representations using domain knowledge , e.g . if some notion of similarity is provided . 3 ANCHOR & TRANSFORM . We present a general purpose method that is end-to-end trainable and allows us to integrate domain knowledge about object relationships . We suppose we are presented with data X ∈ Rn×∣V ∣ , y ∈ Rn×c drawn from some joint distribution p ( X , y ) where the support of X is over a discrete set V = { e1 , ... , e∣V ∣ } . n is the size of the training set , V is typically known as the vocabulary , and ∣V ∣ the vocabulary size . The entries in X are one-hot realizations of the discrete random variable X while the entries in y can be either discrete ( classification with c classes ) or continuous ( regression with c targets ) . The goal is to learn a d-dimensional continuous representation { e1 , ... , e∣V ∣ } for each discrete object by learning an embedding matrix E ∈ R∣V ∣×d where row i is the representation ei of object i . Learning this representation allows us to embed the discrete data into H =XE and define a neural model fθ with parameters θ to predict ŷ given representations H , i.e . ŷ = fθ ( H ) = fθ ( XE ) . Instead of learning the large embedding matrix E directly , ANT consists of two components : 1 ) ANCHOR : Learn embeddings A ∈ R∣A∣×d of a small set of anchor objects A = { a1 , ... , a∣A∣ } , ∣A∣ < < ∣V ∣ that are representative of all discrete objects . One is free to use different methods ( e.g . random , frequency , or clustering-based initialization strategies ) to initialize/select anchors based on the task . 2 ) TRANSFORM : Learn a sparse transformation T from A to the full embeddings E. In other words , each of the non-anchor objects should be easily induced by some transformation from ( a few ) nearby anchor objects . To ensure sparsity , we want nnz ( T ) < < ∣V ∣ × d. Furthermore , domain knowledge can be incorporated by defining the structure of these transformations between related objects . A and T are trained end-to-end for task specific representation learning . After training , we only store ∣A∣×d+nnz ( T ) < < ∣V ∣×d entries that define the complete embedding , which is much fewer than the ∣V ∣ × d embedding traditionally used . We depict the overall algorithm in Figure 1 . In the following subsections , we explain how these anchor embeddings ( §3.1 ) and sparse transformations ( §3.2 ) are learned , before describing how the user can incorporate domain knowledge into ANT ( §3.3 ) . 3.1 ANCHOR : LEARNING THE ANCHOR EMBEDDINGS A . In the ANCHOR step , we would like to find a small set of anchor objects A = { a1 , ... , a∣A∣ } , ∣A∣ < < ∣V ∣ that are representative of all discrete objects . We describe several methods to select these anchors1 . Frequency and TF-IDF : For tasks where frequency or TF-IDF ( Ramos , 1999 ) are useful for prediction , the objects can simply be sorted by frequency or TF-IDF and the most common objects selected as the anchor points . While this might make sense for tasks such as language modeling ( Luong et al. , 2015 ; Chen et al. , 2016b ) , only choosing the most frequent objects might not cover the space of all objects . As a result , certain rare objects might not be well represented by the common anchors . Clustering initialization : To ensure that all objects are close to some anchor in the representation space , we use k-means++ initialization ( Arthur & Vassilvitskii , 2007 ) . Given a predefined metric space S consisting of n points x1 , ... , xn ∈ Rd with some distance function d ( ) , k-means++ initialization is a randomized algorithm that picks cluster centers to span the entire space . In practice , S is usually a feature space representative of the relationships between objects , such as the Glove ( Pennington et al. , 2014 ) or Word2vec ( Mikolov et al. , 2013 ) embedding space for words or a co-occurrence matrix ( Haralick et al. , 1973 ) for more general objects . Using C to denote the set of cluster centers ( C = { } initially ) , the k-means++ initialization alternates between two steps : 1 . D ( xi ) = minc∈C d ( xi , c ) , and 2. add xi to C with probability ∝D2 ( xi ) . This clustering initial- 1Please refer to Appendix A for more strategies and a comparison of various initialization strategies . Confidential + Proprietary ization method can also be used to augment other strategies , such as initializing anchors using frequent objects followed by clustering iterations to complete the remaining anchors ( Figure 2 ) . Dynamic basis vectors : Randomly initialize A to a set of basis vectors . This simple yet powerful method captures the general case where we have less knowledge about the objects . The random basis A can be viewed as a low-rank representation of the full embedding space E. Mixture of anchors : In general , different initialization strategies may bring about different advantages . For example , using a mixture of random basis vectors has been shown to help model multisense embeddings ( Athiwaratkun et al. , 2018 ; Nguyen et al. , 2017 ) . One can define a set of M anchor embeddings A1 , ... , AM each initialized by different strategies and of possibly different sizes .

Summary: The paper describes a "layer" that aims at producing embeddings for discrete objects by using fewer parameters than classical embeddings layers. Indeed, the model proposes, instead of learning an embedding matrix of size VxN, to learn a matrix of embeddings of anchors (AxN) and a transformation matrix (VxA) such that the embedding of any object can be found by multiplying A with T. On top of that, they propose different regularization techniques to improve the quality of the learned embeddings, and particularly a proximal gradient method over a L1 normalization on T to reduce the number of parameters. They propose also different ways to initialize A and also a method for incorporating a priori information (e.g knowledge) into the model.  They evaluate this model on different tasks: text classification and language modeling and show that they can achieve good performance while using fewer parameters than Sota methods.