LARGE SCALE REPRESENTATION LEARNING FROM TRIPLET COMPARISONS

1 INTRODUCTION . It has been widely recognized that deep neural networks ( DNN ) provide a powerful tool for representation learning ( Bengio et al. , 2013 ) . Representations learned in an unsupervised fashion have been demonstrated to be useful in learning tasks such as classification ( Ranzato et al. , 2007 ; 2008 ; Hinton & Salakhutdinov , 2008 ; Hinton et al. , 2006 ; Bengio et al. , 2007 ) . In the context of supervised learning , representations are typically learned as by-products in neural networks ( Radford et al. , 2015 ) . For example in image classification , low level representations of inputs ( e.g. , rgb values ) are fed to a network , together with class label information , the network is trained to perform some supervised classification . As a by-product it discovers a condensed data representation in the last hidden layers of the network that turns out to be surprisingly successful for other computer vision tasks such as object detection or semantic segmentation ( Girshick et al. , 2014 ; Kümmerer et al. , 2014 ; Long et al. , 2015 ; Ren et al. , 2015 ) . Subsequently , more direct mechanisms have been designed to explicitly learn data representations ( Mensink et al. , 2012 ; Bell & Bala , 2015 ; Schroff et al. , 2015 ) . Again the raw data is fed to a network , this time together with some information about the similarity of objects , and then the network is trained to generate a meaningful data representation . Particularly relevant to our work is the field of contrastive representation learning ( Wang et al. , 2014 ; Hoffer & Ailon , 2015 ; Cheng et al. , 2016 ; Ge , 2018 ; Arora et al. , 2019 ) , where similarity information is provided in terms of contrastive triplets of points ( x , x+ , x− ) : for a given point x , the point x+ and x− are specifically chosen data points that are similar / dissimilar to x . Such approaches have been extremely successful on image and text data , and they are particularly elegant if the similarity information can be extracted from the raw data in some unsupervised manner . Examples are the case of word embeddings ( Mikolov et al. , 2013 ) , where words are considered similar if they occur within the same local neighborhood , or text representations ( Logeswaran & Lee , 2018 ) , where two subsequent sentences are considered similar , or in computer vision ( Wang & Gupta , 2015 ) where pairs of image patches in subsequent frames of videos are considered similar . There are two main ingredients that seem unavoidable in all these approaches : First , one needs an explicit low level representation of the data to feed as input to the DNN architectures . Secondly , one needs information about the similarity of different data points that can be used to train the network parameters . This information can be provided by class membership labels in supervised approaches , or it is extracted from some unsupervised meta-data — for instance the temporal information for frames of video data . In our work , we investigate the problem of learning a data representation in cases where no meaningful input representation or explicit similarity information exists . Instead we assume that we are provided with the answers to a set of general triplet comparisons where , for any arbitrary triplet of items ( xi , xj , xk ) , we know whether xi is more similar to xj or xk . The problem of learning a representation based on such triplet comparisons is called ordinal embedding ( Agarwal et al. , 2007 ; van der Maaten & Weinberger , 2012 ; Kleindessner & von Luxburg , 2014 ; Terada & von Luxburg , 2014 ) . This setting can be of advantage in many cases . Consider learning a representation for a large set of movies . There does not exist an obvious informative low level representation that can be used in a straight forward manner in a neural net , yet one can ask triplet comparisons from users of a movie database . An example of a triplet comparison on movies in shown in Figure 1a . Other use-cases also arise in more scientific contexts . For example , psychophysical scaling aims to find the functional relation of a physical stimulus and human perception . For example , Aguilar et al . ( 2017 ) conducted experiments with triplet comparisons to find out the relation between the angle of a tilted plane and the perceived angle of a human observer . In this study , a number of triplet questions are asked from human observers ( see an example of triplet question in Figure 1b ) . Later , a one-dimensional embedding ( representation ) is learned for the perceived value of angle . A comprehensive study on the application of ordinal embedding in psychophysics is available at Haghiri et al . ( 2019 ) . We design a novel neural network architecture that can learn representations when we are only given the answers to a set of triplet comparisons and no input representation exists . Our approach opens doors into two different worlds : ( 1 ) We use the power of DNNs to approximately solve an NP hard optimization problem with discrete input ( point identifiers and binary comparisons ) . Its widely believed amongst neural network practitioners that the non-convex landscape of deep and wide neural networks is primarily free of sub-optimal local minima . This theory is supported under simplified assumptions by various theoretical findings ( Nguyen & Hein , 2017 ; Choromanska et al. , 2015 ; Kawaguchi & Bengio , 2019 ; Kawaguchi et al. , 2019 ) . These results provide a basis for the hypothesis that deep neural network models offer a tool that can make non-convex , NP-hard optimization problems “ tractable in practice ” . In machine learning , this line of thinking is particularly interesting for unsupervised learning problems such as clustering , dimensionality reduction or representation learning , where the typical objective functions are discrete , often NP hard , or non-convex . The standard approach to solve such problems are convex relaxations . Albeit the resulting relaxed problem then can be solved exactly , there often does not exist any provable guarantee that relates the solution of the relaxed problem to the solution of the original problem . We believe that the ability of DNN ’ s to directly ( approximately ) solve non-convex optimization functions provides an attractive alternative and has not really been explored to solve such problems in ML : very little emphasis has been placed in approaches that use neural networks not as learning machines but rather as toolboxes to solve optimization problems . The same point of view might be valuable for general , even discrete optimization problems . In our paper we demonstrate that DNNs can be successfully applied to such a problem . It will be interesting to see how more generic architectures can be designed to solve more generic discrete , non-convex , NP-Hard optimization problems . ( 2 ) We provide the first , scalable approach for the ordinal embedding problem based on neural networks . The problem of ordinal embedding has been studied extensively in the machine learning literature ( Agarwal et al. , 2007 ; van der Maaten & Weinberger , 2012 ; Kleindessner & von Luxburg , 2014 ; Terada & von Luxburg , 2014 ; Jain et al. , 2016 ) . Optimizing the ordinal embedding objective by traditional means is notoriously difficult ( either algorithms are very slow or lead to unsatisfactory results ) , and for computational reasons it is close to impossible to embed more than 10000 items . Thanks to the fast parallel computations of DNN training , our approach runs significantly faster than traditional methods ( see Subsection 4.4 ) . 2 BACKGROUND ON COMPARISON-BASED MACHINE LEARNING AND ORDINAL EMBEDDING . Assume we are given a set of abstract items X = { x1 , x2 , . . . , xn } , but no explicit representation is available for the items . Even though there exists a dissimilarity function δ ( . , . ) that describes the dissimilarity of pairs of items , we also do not have direct access to this dissimilarity function . Instead , we only get to see answers to triplet questions of the form “ Is item xi more similar to item xj or item xk ? We denote such a triplet question by t = ( xi , xj , xk ) ( or sometimes just by t = ( i , j , k ) ) , and the answer to this triplet question by Rt = { 1 , if xi is more similar to xj than to xk −1 , if xi is more similar to xk than to xj ( 1 ) Machine learning based on such qualitative comparisons has become popular ( Agarwal et al. , 2007 ; van der Maaten & Weinberger , 2012 ; Amid & Ukkonen , 2015 ; Ukkonen et al. , 2015 ; Balcan et al. , 2016 ; Haghiri et al. , 2017 ; 2018 ) . One fundamental approach in this area is the ordinal embedding procedure . Here we assume that a set of triplet comparisons T = { t1 , t2 , . . . tm } is given to us . We consider the passive setting where we do not have any influence on the chosen comparisons ( as opposed to an active setting where we could actively ask particular questions to an oracle ) . The binary answers , according to the above definition , are stored . Ordinal embedding aims to find a d-dimensional representation y1 , y2 , . . . yn ∈ Rd such that the Euclidean distances between these points satisfy as many triplet answers as possible . This can be expressed by the following optimization problem : max y1 , ... , yn∈Rd ∑ t= ( i , j , k ) ∈T Rt · sign ( ‖yi − yj‖2 − ‖yi − yk‖2 ) . ( 2 ) In a real world setting it is typically impossible to satisfy all the input triplets : the dissimilarity function is not necessarily constrained to be a d-dimensional Euclidean metric , and even if so the answers might be noisy . The above optimization problem is discrete , non-convex and NP-hard ( Bower et al. , 2018 ; Pardalos & Vavasis , 1991 ) . There exit various methods that employ relaxations of the this optimization problem : Generalized non-metric multidimensional embedding ( GNMDS ; Agarwal et al. , 2007 ) , stochastic triplet embedding ( STE ; van der Maaten & Weinberger , 2012 ) and local ordinal embedding ( LOE ; Terada & von Luxburg , 2014 ) are among the methods in this category . However , all of them are computationally very demanding , and none of these methods can scale beyond a few thousand input points at most . The only somewhat scalable approach is the one by Anderton & Aslam ( 2019 ) , based on explicit geometric constructions and landmarks . However , this approach requires an active access to triplet answers : rather than just receiving a bag of triplet answers , the algorithm needs to repeatedly ask very specific triplet questions to an oracle . Our approach works in the more general passive setting . The performance of ordinal embedding methods is typically evaluated by the triplet error . Given an estimated representation { ŷ1 , ŷ2 , . . . , ŷn } of the data items and a set T ′ of triplet answers , the triplet error simply counts how many triplet answers are satisfied by the given representation : triplet error ( TE ) = 1 |T ′| ∑ t= ( i , j , k ) ∈T ′ 1 [ Rt · sign ( ‖ŷi − ŷj‖2 > ‖ŷi − ŷk‖2 ) = −1 ] , ( 3 ) where 1 ( · ) equals to one , if the inside expression is true and it is zero otherwise . Note that Rt · sign ( ‖ŷi − ŷj‖2 > ‖ŷi − ŷk‖2 ) = −1 if the estimated embedding is not consistent with the triplet t. Thus , we count the ratio of non-consistent triplets in this way .

Summary: Many prior works have found that the features output by the final layer of neural networks can often be used as informative representations for many tasks despite being trained for one in particular. These feature representations, however, are learned transformations of low-level input representations, e.g. RGB values of an image. In this paper, they aim to learn useful feature representations without meaningful low-level input representations, e.g. just an instance ID. Instead, meaningful representations are learned through gathered triplet comparisons of these IDs, e.g. is instance A more similar to instance B or instance C? Similar existing techniques fall in the realm of learning ordinal embeddings, but this technique demonstrates speed-ups that allow it to scale to large real world datasets.