Distance-based Composable Representations with Neural Networks

1 INTRODUCTION . We introduce a new deep learning technique to create interpretable and composable representations both for generic classes as well as for individual samples based on distance estimates with respect to contextual information . The generic class representations , which we refer to as ‘ templates ’ , express how samples from a class relate to other classes on average and can be used to efficiently determine class membership . Most neural network-based approaches to representation learning , focus on learning locations of entities in a low dimensional Euclidean vector space . Word2Vec , for example , extracts meaning from the learned location of words in a vector space ( Mikolov et al. , 2013a ; b ) . Other text-based approaches use the hidden state vector of LSTM networks ( Hochreiter & Schmidhuber , 1997 ) that learn useful information while performing sequence-to-sequence learning tasks ( Sutskever et al. , 2014 ) , such as machine translation ( Wu et al. , 2016 ) , or surrounding sentence reconstruction ( Kiros et al. , 2015 ) . For images , relevant features are typically extracted from convolutional neural networks ( CNNs ) ( LeCun et al. , 1998 ) . When they are pre-trained on a classification task , they are considered to contain abstract knowledge that can be used as an input to perform further tasks ( Vinyals et al. , 2015 ; Karpathy & Fei-Fei , 2015 ; Xu et al. , 2015 ) . Such point-based representations have achieved great results for many tasks , but lack flexibility . Word2Vec representations don ’ t change depending on the context . CNN-based representations fail to accurately relay all relevant features for interacting objects in scenes . Recent works in Natural Language Processing ( NLP ) have attempted to make point estimations dependent on the context , for example ELMo ( Peters et al. , 2018 ) and BERT ( Devlin et al. , 2018 ) , yet such representations still lack interpretability . Other approaches have started looking at representations that are essentially probability distributions that are built from the co-occurrence of particular contexts . Singh et al . ( 2019 ) create sentence embeddings by defining a Context Mover ’ s Distance over words occurring in different contexts and Wu et al . ( 2018 ) create text document embeddings with feature maps by estimating the distance of a document to a range of arbitrarily created documents . In this work we will build on the latter approaches by creating a method that is not restricted by modality type . Our approach relies on neural networks to estimate Wasserstein distances between the sample spaces for different classes and the sample spaces for events that we call environments . These environments are defined by the occurrence of any element in a subset of attributes where each subset , limited to a certain size , is selected uniformly from the power set of all attributes . While in this work we limit the definition of attributes to simply refer to ‘ class labels ’ , future work might build on a broader interpretation . In the simplest case , when the size of the subsets is limited to 1 , each environment is given by the occurrence of a singular class label . The goal is to capture some common sense knowledge about how classes relate to a range of differentiating contexts . Such environments are bound to have distinctive features in relation to any given class as they are made up of a combination of arbitrarily chosen factors . This mechanism loosely resembles the associative nature of human memory . Long-term memory storage is believed to rely on semantic encoding that performs better if it can be associated with existing contextual knowledge ( Lepage et al. , 2000 ; Murdock , 1982 ) . Additionally , relating classes to arbitrarily chosen environments increases the probability of encountering distinctive diagnostic features , which should help retrieval , as might be the case with human long-term memory ( Nairne , 2002 ) . We also propose a method to generate template representations of classes such that individual input samples can be interpreted in reference to the template representations . A template representation is thus a vector containing the average distance estimates between the sample distribution of a particular class and those of the available environments . This approach , illustrated in figure 1 , addresses the limitation of Euclidean point based representations where no distinction is made between generic class representations and specific instance representations . With the use of such templates , class membership of individual samples can then easily be determined by computing the similarity between the representation of an individual sample and all template representations . We will show that , for the same neural network , this method performs well on multi-label classification tasks for images , where any image can be assigned several classes ( for example , ‘ baseball ’ and ‘ bat ’ ) . Both the template and individual sample representations are interpretable in the sense that they contain distance estimates along contextual dimensions . Samples that belong to a particular class can be analyzed as to how and to what extent they differ from the generic template representation . A sample that is classified as a ‘ cat ’ might have a smaller distance to the ‘ bear ’ class , suggesting that that particular cat looks more like a bear than the average cat . Another advantage is that we can alter or combine representations in order to modify the information content . The usefulness of these characteristics becomes apparent in retrieval applications . We will demonstrate that it allows one to retrieve an image similar to a particular image but with altered information content . CONTRIBUTIONS • We introduce a technique to build representations based on probability distribution estimates that can be used in downstream tasks . The method , which we call WDVec ( Wasserstein Distance Vector ) , is general in that it can be combined with any neural network . • The obtained representations are interpretable as distance estimates between class and contextual information . They are composable in the sense that they can be modified to reflect different class membership or contextual information . Similar inputs lay near each other in the representation space . • We define template representations for concepts to which individual samples can be compared . This allows efficient class-membership computation and easy manipulation in the representation space . 2 BACKGROUND . We will shortly explain some useful concepts , mostly in relation to distance estimates and how they can be found with neural networks . Such distance estimates will form the basis for our representation learning method further on . We will also refer to recent work that uses similar tactics . Useful concepts Central to our approach is the concept of the ‘ Earth Mover ’ s Distance ’ ( EMD ) ( Rubner et al. , 2000 ) , also known as the Wasserstein distance . It can be understood as the minimal amount of effort that is required to move the mass from one probability distribution to another . Finding the EMD between two distributions is traditionally done by solving the ‘ optimal transport problem ’ ( Hitchcock , 1941 ; Altschuler et al. , 2017 ; Singh et al. , 2019 ) . While some approaches Kusner et al . ( 2015 ) ; Wu et al . ( 2018 ) use some variation of Sinkhorn iterations ( Sinkhorn , 1964 ; Altschuler et al. , 2017 ) to solve the transport problem , neural network-based approximations of the EMD have also been developed in recent years . This is usually in connection to Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) where two neural networks , a generator and a discriminator , are staged against each other . Most recent GAN formulations make use of an Integral Probability Metric ( IPM ) that essentially defines the loss function for a neural network ( referred to as the critic f instead of the discriminator ) that allows it to maximally discriminate between two distributions . Most famously , in the Wasserstein GAN ( WGAN ) formulation ( Arjovsky et al. , 2017 ) , the IPM of the critic attempts to estimate the Wasserstein distance between the distribution of real samples versus the distribution of generated samples . As the generator improves , the distance estimated by the critic should approach zero . The loss function for the critic of the Wasserstein GAN can be defined as follows : dF ( R , G ) = sup f∈F { E x∼R f ( x ) − E x∼G f ( x ) } . ( 1 ) where R represents the distribution of real samples in a dataset and G represents the distribution of fake ( or generated ) samples . F is a set of measurable , symmetric and bounded real valued functions . If F is unbounded , equation 1 will scale without bounds . In order for this approximation to work , F thus needs to be bounded . Usually , the critic network is restricted to be a 1-Lipschitz function . Several methods have been proposed to enforce this constraint , such as weight-clipping ( Arjovsky et al. , 2017 ) , gradient penalties ( Gulrajani et al. , 2017 ) or spectral normalization ( Miyato et al. , 2018 ) . The Fisher GAN formulation takes a different approach and bounds F by construction , that is , by defining a data dependent constraint on its second order moments ( Mroueh & Sercu , 2017 ) . The IPM becomes : dF isherF ( R , G ) = sup f∈F E x∼R [ f ( x ) ] − E x∼G [ f ( x ) ] √ 1/2Ex∼Rf2 ( x ) + 1/2Ex∼Gf2 ( x ) ( 2 ) Equation 2 can be interpreted as the search for a critic function f that maximizes the average discrepancy between two distributions R and G ( thus maximizing inter-class variance ) whilst minimizing the second order discrepancy ( i.e. , limiting intra-class variance ) ( Mroueh & Sercu , 2017 ) . Upon completing training , the numerator thus gives a good Wasserstein distance estimate when inter-class variance is small in comparison to intra-class variance . When this is not the case , the numerator will be large , yet not an exact approximation of the Wasserstein distance . In practice , the Fisher GAN IPM can be estimated with neural network training where the numerator in equation 2 is maximized while the denominator is expressed as a constraint that is enforced with a Lagrange multiplier . Recent work The EMD has been successfully applied to NLP problems . Kusner et al . ( 2015 ) , for example , define the ‘ Word Mover ’ s Distance ’ ( WMD ) , which measures the minimal amount of effort to move Word2Vec based word embeddings from one document to another . Their method is interpretable and outperforms other document distance metrics in text-based classification tasks . Wu et al . ( 2018 ) improve upon their solution by defining a Word Mover ’ s Embedding , an unsupervised feature representation for documents , created by concatenating Word Mover ’ s Distance estimates to arbitrarily chosen feature maps . They then calculate an approximation of the distance between a pair of documents with the use of a kernel over the feature map . The building blocks of the feature maps are thus documents built from an arbitrary combination of words . This idea is based on the Random Features approximation ( Rahimi & Recht , 2008 ) that suggests that randomized feature maps are useful for approximating shift-invariant kernels . Singh et al . ( 2019 ) build on this idea to create unsupervised sentence representations where each entity is a probability distribution based on co-occurrence of words . They embed the distributions in a low-dimensional representation space for text and demonstrate state-of-the-art performance on tasks such as sentence similarity and word entailment . They note that their approach captures uncertainty and allows to interpret the outcome over different contexts . Also relevant is the concept of ‘ classemes ’ in computer vision , where rich features are created out of a combination of visual concepts ( Torresani et al. , 2010 ) . Finally , ‘ Wasserstein Discriminant Analysis ’ ( Flamary et al. , 2018 ) obtains features through dimensionality reduction by maximizing Wasserstein distances between classes while minimizing intra-class discrepancy . Our method borrows some concepts from these works , such as the creation of possibly infinitedimensional feature maps and representations that are built on ( randomized ) contextual information . WDVec can be used with any modality though and features can be found with neural network-based EMD estimations between different sample spaces . Additionally , we will create generic template representations as well as individual sample representations , which allows efficient membership tests and complex retrieval queries . Our representations will be both interpretable and composable .

Summary: The authors proposed a template-based interpretable representation that works based on the earth mover's distance of each class to a number of "environments", which could be taken as union of a few random classes. To achieve this, they train several critics based on Fisher GAN. The method is evaluated based on classification and retrieval tasks.