Compositional Embeddings: Joint Perception and Comparison of Class Label Sets

1 INTRODUCTION . Embeddings , especially as enabled by advances in deep learning , have found widespread use in natural language processing , object recognition , face identification and verification , speaker verification and diarization ( i.e. , who is speaking when ( Sell et al. , 2018 ) ) , and other areas . What embedding functions have in common is that they map their input into a fixed-length distributed representation ( i.e. , continuous space ) that facilitates more efficient and accurate ( Scott et al. , 2018 ) downstream analysis than simplistic representations such as one-of-k . Moreover , they are amenable to one-shot and few-shot learning since the set of classes that can be represented does not depend directly on the dimensionality of the embedding space . Previous research on embeddings has focused on cases where each example is associated with just one class ( e.g. , the image contains only one person ’ s face ) . In contrast , we investigate the case where each example is associated with not just one , but an entire subset of classes from a universe S. The goal is to embed each example so that questions of two types can be answered ( see Figure 1 ( a ) ) : ( 1 ) Is the set of classes in example xa equal to the union of the classes in examples xb and xc ? ( 2 ) Does the set of classes in example xa subsume the set of classes in example xb ? Importantly , we focus on settings in which the classes present in the example must be perceived automatically . We approach this problem using compositional set embeddings . Like traditional embeddings , we train a function f that maps each example x ∈ Rn into an embedding space Rm so that examples with the same classes are mapped close together and examples with different classes are mapped far apart . Unlike traditional embeddings , our function f is trained to represent the set of classes that is associated with each example , so that questions about set union and subsumption can be answered by comparing vectors in the embedding space . We do not assume that the mechanism by which examples ( e.g. , images , audio signals ) are rendered from multiple classes is known . Rather , the rendering process must be learned from training data . We propose two models , whereby f is trained jointly with either a “ composition ” function g ( Model I ) that answers questions about set union , or a “ query ” function h ( Model II ) that answers question about subsumption ( see Figure 1 ( a ) ) . To our knowledge , this computational problem is novel . We see at least two use-cases : ( 1 ) Speaker recognition and diarization ( i.e. , infer who is talking within an audio signal ) with multiple simultaneous speakers : Given an audio signal containing speakers who were not part of the training set and who may be speaking simultaneously , and given one example of each person speaking in isolation ( one-shot learning ) , infer which set of speakers is talking . ( 2 ) Multi-object recognition in images : Given just the embedding of an image xa , answer whether xa contains the object ( s ) in another image xb . Storing just the embeddings but not the pixels could potentially be more space-efficient . Because of the novelty of the problem , it was not obvious to what baselines we should compare . When evaluating our models , we sought to assess the unique contribution of the compositional embedding above and beyond what a “ traditional ” embedding could achieve . Hence , we created baselines by endowing a traditional embedding with some extra functionality to enable it to infer label sets . Modeling assumptions and notation : For generality , we refer to the data to be embedded ( images , videos , audio signals , etc . ) simply as “ examples ” . Let the universe of classes be S. From any subset T ⊆ S , a ground-truth rendering function r : 2S → Rn “ renders ” an example , i.e. , r ( T ) = x. Inversely , there is also a ground-truth classification function c : Rn → 2S that identifies the label set from the rendered example , i.e. , c ( x ) = T . Neither r nor c is observed . We let eT represent the embedding ( i.e. , output of f ) associated with some example containing classes T . Contribution : To our knowledge , this is the first paper to explore how embedding functions can be trained both to perceive multiple objects in the example and to represent the set of detected objects so that set operations can be conducted among embedded vectors . We instantiate this idea in two ways : Model I for set union ( f & g ) and Model II for set containment ( f & h ) . By evaluating on synthetic data , OmniGlot handwritten image data ( Lake et al. , 2015 ) , as well as the COCO dataset ( Lin et al. , 2014 ) , we provide a proof-of-concept that “ compositional set embeddings ” can work . 2 RELATED WORK . Embeddings : We distinguish between two types of embeddings : ( 1 ) “ Perceptual ” embeddings such as for vision ( Facenet ( Schroff et al. , 2015 ) ) and speech ( x-vector ( Snyder et al. , 2018 ) ) where each class ( e.g. , person whose voice was recorded or face was photographed ) may contain widely varying examples across speech content , facial expression , lighting , background noise , etc . ( 2 ) Word embeddings ( word2vec ( Mikolov et al. , 2013 ) , GloVe ( Pennington et al. , 2014 ) ) where each class contains only one exemplar by definition . Within the former , the task of the embedding function is to map examples from the same class close together and examples from other classes far apart . This often requires deep , non-linear transformations to be successful . With word embeddings , the class of each example is already clear and does not need to be inferred ; instead , the goal is to give the embedded vectors geometric structure to reflect co-occurrence , similarity in meaning , etc . Compositional embeddings : Since at least 30 years , AI researchers , cognitive scientists , and computational neuroscientists have explored how the embeddings of multiple elements could be combined to reflect relationships between them or higher-level semantics . However , almost all this work was based on word embeddings , in which perception was not necessary . Some early work investigated how the grammatical structure and/or semantics of an input sentence can be represented ( Pollack , 1989 ) in an efficient manner in neural networks , and how such a network could be trained ( Elman , 1993 ) . Given the advent of word embeddings , deep NLP architectures can combine the word-level semantics , as represented by the embeddings of the individual elements of an input sentence , to infer higher-level attributes , e.g. , sentiment ( Nakov et al. , 2016 ) . Recent work has investigated to what extent contemporary recurrent neural networks can generalize to understand novel sentences ( Lake & Baroni , 2017 ) consisting of known words . Also , in the NLP domain , Joshi et al . ( 2018 ) developed compositional pairwise embeddings that model the co-occurrence relationships between two words given their common context . Probably the most algorithmically similar work to ours is by Lyu et al . ( 2019 ) on compositional network embeddings : the goal is to predict whether two new nodes in a graph , which were not observed during training , are adjacent , using node-based features as predictors . In their approach , two embeddings are used : one to embed the node-based features , and another to aggregate these embedded features into a secondary embedding space . Structurally , their work differs from ours in that ( 1 ) the two embedding spaces in their model do not represent the same universe of objects ; ( 2 ) the embeddings do not capture set relationships . Deep set representations : Our paper is also about how to encode a set of objects with a neural network . One issue is how to ensure invariance to the order in which examples are presented . Vinyals et al . ( 2015 ) proposed an approach based on permutation-invariant content-based attention . For producing sets as outputs , Rezatofighi et al . ( 2017 ) proposed a probabilistic model , within a supervised learning paradigm where all classes are known at training time , that predicts both the cardinality and the particular elements of the set . 3 MODEL I : EMBEDDING f AND COMPOSITION g Given two examples xa and xb that are associated with singleton sets { s } and { t } , respectively , the hope is that , for some third example xc that is associated with both classes ( i.e. , { s , t } ) , we have g ( f ( xa ) , f ( xb ) ) ≈ f ( xc ) Moreover , we hope that g can generalize to any number of classes within the set S. For example , if example xd is associated with a singleton set { u } , then we hope g ( g ( f ( xa ) , f ( xb ) ) , f ( xd ) ) ≈ f ( xe ) where xe is an example associated with { s , t , u } . There are two challenging tasks that f and g must solve cooperatively : ( 1 ) f has to learn to perceive multiple objects that appear simultaneously and may possibly interact with each other – all without knowing the rendering process r of how examples are formed or how classes are combined . ( 2 ) g has to define geometrical structure in the embedding space to support set union operations . One way to understand our computational problem is the following : If f is invertible , then ideally we would want g to compute g ( eT , eU ) = f ( r ( c ( f−1 ( eT ) ) ∪ c ( f−1 ( eU ) ) ) ) . In other words , one ( though not necessarily the only ) way that g can perform well is to learn to perform the following actions ( without knowing r or c ) : ( 1 ) invert each of the two input embeddings ; ( 2 ) classify the two corresponding label sets ; ( 3 ) render an example with the union of the two inferred label sets ; and ( 4 ) embed the result . One-shot learning : Model I can be used for one-shot learning on a set of classes S not seen during training in the following way : We obtain k labeled examples x1 , . . . , xk from the user , where each { si } = c ( xi ) is the singleton set formed from the ith element of S and |S| = k. We call these examples the reference examples . We then infer which set of classes is represented by a new example x′ using the following procedure : ( 1 ) Compute the embedding of x′ , i.e. , f ( x′ ) . ( 2 ) Use f to compute the embedding of each singleton example xi , i.e. , e { i } = f ( xi ) . ( 3 ) From e { 1 } , . . . , e { k } , estimate the embedding of every subset T = { s1 , . . . , sl } ⊆ S according to the recurrence relation : e { s1 , ... , sl } = g ( e { s1 , ... , sl−1 } , e { sl } ) ( 1 ) Finally , ( 4 ) estimate the label of x′ as argmin T ⊆S |f ( x′ ) − eT |22 ( 2 ) Although the number of possible subsets is exponential in |S| , for speaker diarization the number of overlapping speakers is typically small , and thus the iteration is tractable .

Summary: This paper proposes compositional embeddings i.e. embeddings that can be used to infer multiple classes from the data. In particular, the paper deals with two types of composite functions for embeddings, one that computes union of the different classes represented by each embedding vector, and the other where the class of one of the embeddings is subsumed by the class of the other embedding. The actual composition functions are parameterized by neural networks whose parameters are learned from data. Results on synthetic as well as several real-world datasets highlight the superiority of the learned composite embeddings.