LARGE SCALE REPRESENTATION LEARNING FROM TRIPLET COMPARISONS

1 INTRODUCTION . It has been widely recognized that deep neural networks ( DNN ) provide a powerful tool for representation learning ( Bengio et al. , 2013 ) . Representations learned in an unsupervised fashion have been demonstrated to be useful in learning tasks such as classification ( Ranzato et al. , 2007 ; 2008 ; Hinton & Salakhutdinov , 2008 ; Hinton et al. , 2006 ; Bengio et al. , 2007 ) . In the context of supervised learning , representations are typically learned as by-products in neural networks ( Radford et al. , 2015 ) . For example in image classification , low level representations of inputs ( e.g. , rgb values ) are fed to a network , together with class label information , the network is trained to perform some supervised classification . As a by-product it discovers a condensed data representation in the last hidden layers of the network that turns out to be surprisingly successful for other computer vision tasks such as object detection or semantic segmentation ( Girshick et al. , 2014 ; Kümmerer et al. , 2014 ; Long et al. , 2015 ; Ren et al. , 2015 ) . Subsequently , more direct mechanisms have been designed to explicitly learn data representations ( Mensink et al. , 2012 ; Bell & Bala , 2015 ; Schroff et al. , 2015 ) . Again the raw data is fed to a network , this time together with some information about the similarity of objects , and then the network is trained to generate a meaningful data representation . Particularly relevant to our work is the field of contrastive representation learning ( Wang et al. , 2014 ; Hoffer & Ailon , 2015 ; Cheng et al. , 2016 ; Ge , 2018 ; Arora et al. , 2019 ) , where similarity information is provided in terms of contrastive triplets of points ( x , x+ , x− ) : for a given point x , the point x+ and x− are specifically chosen data points that are similar / dissimilar to x . Such approaches have been extremely successful on image and text data , and they are particularly elegant if the similarity information can be extracted from the raw data in some unsupervised manner . Examples are the case of word embeddings ( Mikolov et al. , 2013 ) , where words are considered similar if they occur within the same local neighborhood , or text representations ( Logeswaran & Lee , 2018 ) , where two subsequent sentences are considered similar , or in computer vision ( Wang & Gupta , 2015 ) where pairs of image patches in subsequent frames of videos are considered similar . There are two main ingredients that seem unavoidable in all these approaches : First , one needs an explicit low level representation of the data to feed as input to the DNN architectures . Secondly , one needs information about the similarity of different data points that can be used to train the network parameters . This information can be provided by class membership labels in supervised approaches , or it is extracted from some unsupervised meta-data — for instance the temporal information for frames of video data . In our work , we investigate the problem of learning a data representation in cases where no meaningful input representation or explicit similarity information exists . Instead we assume that we are provided with the answers to a set of general triplet comparisons where , for any arbitrary triplet of items ( xi , xj , xk ) , we know whether xi is more similar to xj or xk . The problem of learning a representation based on such triplet comparisons is called ordinal embedding ( Agarwal et al. , 2007 ; van der Maaten & Weinberger , 2012 ; Kleindessner & von Luxburg , 2014 ; Terada & von Luxburg , 2014 ) . This setting can be of advantage in many cases . Consider learning a representation for a large set of movies . There does not exist an obvious informative low level representation that can be used in a straight forward manner in a neural net , yet one can ask triplet comparisons from users of a movie database . An example of a triplet comparison on movies in shown in Figure 1a . Other use-cases also arise in more scientific contexts . For example , psychophysical scaling aims to find the functional relation of a physical stimulus and human perception . For example , Aguilar et al . ( 2017 ) conducted experiments with triplet comparisons to find out the relation between the angle of a tilted plane and the perceived angle of a human observer . In this study , a number of triplet questions are asked from human observers ( see an example of triplet question in Figure 1b ) . Later , a one-dimensional embedding ( representation ) is learned for the perceived value of angle . A comprehensive study on the application of ordinal embedding in psychophysics is available at Haghiri et al . ( 2019 ) . We design a novel neural network architecture that can learn representations when we are only given the answers to a set of triplet comparisons and no input representation exists . Our approach opens doors into two different worlds : ( 1 ) We use the power of DNNs to approximately solve an NP hard optimization problem with discrete input ( point identifiers and binary comparisons ) . Its widely believed amongst neural network practitioners that the non-convex landscape of deep and wide neural networks is primarily free of sub-optimal local minima . This theory is supported under simplified assumptions by various theoretical findings ( Nguyen & Hein , 2017 ; Choromanska et al. , 2015 ; Kawaguchi & Bengio , 2019 ; Kawaguchi et al. , 2019 ) . These results provide a basis for the hypothesis that deep neural network models offer a tool that can make non-convex , NP-hard optimization problems “ tractable in practice ” . In machine learning , this line of thinking is particularly interesting for unsupervised learning problems such as clustering , dimensionality reduction or representation learning , where the typical objective functions are discrete , often NP hard , or non-convex . The standard approach to solve such problems are convex relaxations . Albeit the resulting relaxed problem then can be solved exactly , there often does not exist any provable guarantee that relates the solution of the relaxed problem to the solution of the original problem . We believe that the ability of DNN ’ s to directly ( approximately ) solve non-convex optimization functions provides an attractive alternative and has not really been explored to solve such problems in ML : very little emphasis has been placed in approaches that use neural networks not as learning machines but rather as toolboxes to solve optimization problems . The same point of view might be valuable for general , even discrete optimization problems . In our paper we demonstrate that DNNs can be successfully applied to such a problem . It will be interesting to see how more generic architectures can be designed to solve more generic discrete , non-convex , NP-Hard optimization problems . ( 2 ) We provide the first , scalable approach for the ordinal embedding problem based on neural networks . The problem of ordinal embedding has been studied extensively in the machine learning literature ( Agarwal et al. , 2007 ; van der Maaten & Weinberger , 2012 ; Kleindessner & von Luxburg , 2014 ; Terada & von Luxburg , 2014 ; Jain et al. , 2016 ) . Optimizing the ordinal embedding objective by traditional means is notoriously difficult ( either algorithms are very slow or lead to unsatisfactory results ) , and for computational reasons it is close to impossible to embed more than 10000 items . Thanks to the fast parallel computations of DNN training , our approach runs significantly faster than traditional methods ( see Subsection 4.4 ) . 2 BACKGROUND ON COMPARISON-BASED MACHINE LEARNING AND ORDINAL EMBEDDING . Assume we are given a set of abstract items X = { x1 , x2 , . . . , xn } , but no explicit representation is available for the items . Even though there exists a dissimilarity function δ ( . , . ) that describes the dissimilarity of pairs of items , we also do not have direct access to this dissimilarity function . Instead , we only get to see answers to triplet questions of the form “ Is item xi more similar to item xj or item xk ? We denote such a triplet question by t = ( xi , xj , xk ) ( or sometimes just by t = ( i , j , k ) ) , and the answer to this triplet question by Rt = { 1 , if xi is more similar to xj than to xk −1 , if xi is more similar to xk than to xj ( 1 ) Machine learning based on such qualitative comparisons has become popular ( Agarwal et al. , 2007 ; van der Maaten & Weinberger , 2012 ; Amid & Ukkonen , 2015 ; Ukkonen et al. , 2015 ; Balcan et al. , 2016 ; Haghiri et al. , 2017 ; 2018 ) . One fundamental approach in this area is the ordinal embedding procedure . Here we assume that a set of triplet comparisons T = { t1 , t2 , . . . tm } is given to us . We consider the passive setting where we do not have any influence on the chosen comparisons ( as opposed to an active setting where we could actively ask particular questions to an oracle ) . The binary answers , according to the above definition , are stored . Ordinal embedding aims to find a d-dimensional representation y1 , y2 , . . . yn ∈ Rd such that the Euclidean distances between these points satisfy as many triplet answers as possible . This can be expressed by the following optimization problem : max y1 , ... , yn∈Rd ∑ t= ( i , j , k ) ∈T Rt · sign ( ‖yi − yj‖2 − ‖yi − yk‖2 ) . ( 2 ) In a real world setting it is typically impossible to satisfy all the input triplets : the dissimilarity function is not necessarily constrained to be a d-dimensional Euclidean metric , and even if so the answers might be noisy . The above optimization problem is discrete , non-convex and NP-hard ( Bower et al. , 2018 ; Pardalos & Vavasis , 1991 ) . There exit various methods that employ relaxations of the this optimization problem : Generalized non-metric multidimensional embedding ( GNMDS ; Agarwal et al. , 2007 ) , stochastic triplet embedding ( STE ; van der Maaten & Weinberger , 2012 ) and local ordinal embedding ( LOE ; Terada & von Luxburg , 2014 ) are among the methods in this category . However , all of them are computationally very demanding , and none of these methods can scale beyond a few thousand input points at most . The only somewhat scalable approach is the one by Anderton & Aslam ( 2019 ) , based on explicit geometric constructions and landmarks . However , this approach requires an active access to triplet answers : rather than just receiving a bag of triplet answers , the algorithm needs to repeatedly ask very specific triplet questions to an oracle . Our approach works in the more general passive setting . The performance of ordinal embedding methods is typically evaluated by the triplet error . Given an estimated representation { ŷ1 , ŷ2 , . . . , ŷn } of the data items and a set T ′ of triplet answers , the triplet error simply counts how many triplet answers are satisfied by the given representation : triplet error ( TE ) = 1 |T ′| ∑ t= ( i , j , k ) ∈T ′ 1 [ Rt · sign ( ‖ŷi − ŷj‖2 > ‖ŷi − ŷk‖2 ) = −1 ] , ( 3 ) where 1 ( · ) equals to one , if the inside expression is true and it is zero otherwise . Note that Rt · sign ( ‖ŷi − ŷj‖2 > ‖ŷi − ŷk‖2 ) = −1 if the estimated embedding is not consistent with the triplet t. Thus , we count the ratio of non-consistent triplets in this way .

Summary: The paper proposes to use the triplet loss as a convex relaxation of the ordinal embedding problem. The loss is solved using feed-forward neural network with the input to the network being the ids of the items encoded in binary codes. The benefit of using a deep network is to exploit its optimization capability and the parallelism on GPUs. The experiments presented in the paper include a set of simulation experiments and a real-world task.