An Inductive Bias for Distances: Neural Nets that Respect the Triangle Inequality

1 INTRODUCTION . Many machine learning tasks involve a distance measure over the input domain . A good measure can make a once hard task easy , even trivial . In many cases—including graph distances , certain clustering algorithms , and general value functions in reinforcement learning ( RL ) —it is either known that distances satisfy the triangle inequality , or required for purposes of theoretical guarantees ; e.g. , speed and loss guarantees in k-nearest neighbors and clustering ( Cover and Hart , 1967 ; Indyk , 1999 ; Davidson and Ravi , 2009 ) , or optimality guarantees for A∗ search ( Russell and Norvig , 2016 ) . This also makes the triangle inequality a potentially useful inductive bias for learning distances . For these reasons , numerous papers have studied different ways to learn distances that satisfy the triangle inequality ( Xing et al. , 2003 ; Yang and Jin , 2006 ; Brickell et al. , 2008 ; Kulis et al. , 2013 ) . The usual approach to enforcing the triangle inequality in deep metric learning ( Yi et al. , 2014 ; Hoffer and Ailon , 2015 ; Wang et al. , 2018 ) is to use a Siamese network ( Bromley et al. , 1994 ) that computes a Euclidean distance in the latent space . Specifically , the Siamese network models distance dX : X × X → R+ on domain X by learning embedding φ : X → Rn and computing dX ( x , y ) as ‖φ ( x ) − φ ( y ) ‖2 . Successful applications include collaborative filtering ( Hsieh et al. , 2017 ) , few-shot learning ( Snell et al. , 2017 ) , and multi-goal reinforcement learning ( Schaul et al. , 2015 ) . The use of Euclidean distance , however , has at least two downsides . First , the Euclidean architecture can not represent asymmetric metrics , which arise naturally in directed graphs and reinforcement learning . Second , it is well known that for some metric spaces ( X , dX ) , including large classes of symmetric graphs ( e.g. , constant-degree expanders and k-regular graphs ) , there is no embedding φ : X → Rn that can model dX precisely using ‖ · ‖2 ( Indyk et al. , 2017 ) . A classic example is shown in Figure 1 . In part due to these issues , some have considered non-architectural constraints . He et al . ( 2016 ) impose a triangle inequality constraint in RL via an online , algorithmic penalty . Implementing such a penalty can be expensive , and does not provide any guarantees . An approach that does guarantee 1Code available at https : //github.com/spitis/deepnorms Published as a conference paper at ICLR 2020 ... ... ... ... ... ... Cleaned up version Blah Blah Blah Norm MSE Euclidean , Rn , ∀n 0.057 Deep Norm , R2 0.000 Wide Norm , R2 0.000 Deep Norm Wide Norm Mahalanobis 1 0 11.5 1.0 0.5 0.0 0.5 1.0 1.5 A B C D 1 0 1 1 0 1 A B C D 1 0 1 1.0 0.5 0.0 0.5 1.0 A B C D Fig . 1 : The nodes in the graph ( left ) can not be embedded into any Rn so that edge distances are represented by the Euclidean metric : points φ ( A ) and φ ( D ) must lie at the midpoint of the segment from φ ( B ) to φ ( C ) —but then φ ( A ) and φ ( D ) coincide , which is incorrect . Our models fit the data in R2 ( middle ) . The visualization ( right ) shows learned norm balls in red and embeddings in blue . satisfaction of triangle inequality is to fix any violations after learning , as done by Brickell et al . ( 2008 ) . But this does not scale to large problems or provide an inductive bias during learning . Is it possible to impose the triangle inequality architecturally , without the downsides of Euclidean distance ? In response to this question , we present the following contributions : ( 1 ) three novel neural network architectures , Deep Norms , Wide Norms and Neural Metrics , which model symmetric and asymmetric norms and metrics , ( 2 ) universal approximation theorems for Deep Norms and Wide Norms and modified Input Convex Neural Networks ( Amos et al. , 2017 ) , and ( 3 ) empirical evaluations of our models on several tasks : modeling norms , metric nearness , modeling shortest path lengths , and learning a general value function ( Sutton et al. , 2011 ) . Our models are guaranteed to satisfy the triangle inequality , straightforward to implement , and may be used in place of the usual Euclidean metric should one seek to model asymmetry or increase expressiveness . 2 MODELING NORMS . 2.1 PRELIMINARIES . Our goal is to construct expressive models of metrics and quasi-metrics on domain X . A metric is a function d : X × X → R+ satisfying , ∀x , y , z ∈ X : M1 ( Non-negativity ) . d ( x , y ) ≥ 0 . M2 ( Definiteness ) . d ( x , y ) = 0 ⇐⇒ x = y. M3 ( Subadditivity ) . d ( x , z ) ≤ d ( x , y ) + d ( y , z ) . M4 ( Symmetry ) . d ( x , y ) = d ( y , x ) . Since we care mostly about the triangle inequality ( M3 ) , we relax other axioms and define a quasimetric as a function that is M1 and M3 , but not necessarily M2 or M4 . Given weighted graph G = ( V , E ) with non-negative weights , shortest path lengths define a quasi-metric between vertices . When X is a vector space ( we assume over R ) , many common metrics , e.g. , Euclidean and Manhattan distances , are induced by a norm . A norm is a function ‖·‖ : X → R satisfying , ∀x , y ∈ X , α ∈ R+ : N1 ( Pos . def. ) . ‖x‖ > 0 , unless x = 0 . N2 ( Pos . homo. ) . α‖x‖ = ‖αx‖ , for α ≥ 0 . N3 ( Subadditivity ) . ‖x+ y‖ ≤ ‖x‖+ ‖y‖ . N4 ( Symmetry ) . ‖x‖ = ‖ –x‖ . An asymmetric norm is N1-N3 , but not necessarily N4 . An ( asymmetric ) semi-norm is nonnegative , N2 and N3 ( and N4 ) , but not necessarily N1 . We will use the fact that any asymmetric semi-norm ‖ · ‖ induces a quasi-metric using the rule , d ( x , y ) = ‖x − y‖ , and first construct models of asymmetric semi-norms . Any induced quasi-metric d is translation invariant—d ( x , y ) = d ( x+ z , y+ z ) —and positive homogeneous—d ( αx , αy ) = αd ( x , y ) for α ≥ 0 . If ‖ · ‖ is symmetric ( N4 ) , so is d ( M4 ) . If ‖ · ‖ is N1 , d is M2 . Metrics that are not translation invariant ( e.g. , Bi et al . ( 2015 ) ) or positive homogeneous ( e.g. , our Neural Metrics in Section 3 ) can not be induced by a norm . A convex function f : X → R is a function satisfying C1 : ∀x , y ∈ X , α ∈ [ 0 , 1 ] : f ( αx+ ( 1−α ) y ) ≤ αf ( x ) + ( 1− α ) f ( y ) . The commonly used ReLU activation , relu ( x ) = max ( 0 , x ) , is convex . 2.2 DEEP NORMS . It is easy to see that any N2 and N3 function is convex—thus , all asymmetric semi-norms are convex . This motivates modeling norms as constrained convex functions , using the following proposition . Proposition 1 . All positive homogeneous convex functions are subadditive ; i.e. , C1 ∧ N2⇒ N3 . ... Less misleading : The proof is straightforward ( put α = 12 in C1 and apply N2 to the left side ) . To use Proposition 1 , we begin with the Input Convex Neural Network ( ICNN ) ( Amos et al. , 2017 ) architecture , which satisfies C1 , and further constrain it to be non-negative and satisfy N2 . The resulting Deep Norm architecture is guaranteed to be an asymmetric semi-norm . A k-layer Deep Norm is defined as : for i = 1 ... k , where x is the input , h0 = 0 , W+1 = 0 , the activation functions gi preserve C1 and N2 ( element-wise ) , gk is non-negative , W+i is a non-negative matrix , and Ui is an unconstrained matrix . As compared to the original ICNN architecture , we have omitted the bias terms from Equation 1 , have constrained the gi to preserve positive homogeneity while also allowing them to be any function that preserves element-wise convexity ( this is essential to our universal approximation results ) , and have required gk to be non-negative . It is easy to verify that the set of valid element-wise gi is { gαβ ( x ) = α relu ( x ) + βx |α , β ≥ 0 } . This includes ReLUs and leaky ReLUs . But we do not restrict ourselves to element-wise activations . Inspired by GroupSort ( Anil et al. , 2018 ) , we use activations that depend on multiple inputs ( and preserve element-wise C1 and N2 ) . In particular , we use the pairwise MaxReLU : maxrelu ( x , y ) = [ max ( x , y ) , α relu ( x ) + β relu ( y ) ] , where α , β ≥ 0 ( 2 ) Deep Norms are N2 and N3 . Using the following propositions , we may also impose N1 and N4 . Proposition 2 . If ‖ · | is an asymmetric semi-norm , then ‖x‖ = ‖x|+ ‖ –x| is a semi-norm . Proposition 3 . If ‖ · ‖a is an ( asymmetric ) semi-norm , ‖ · ‖b is a norm ( e.g. , ‖ · ‖b = ‖ · ‖2 ) , and λ > 0 , then ‖x‖a+λb = ‖x‖a + λ‖x‖b is an ( asymmetric ) norm . 2.3 WIDE NORMS . In addition to Deep Norms , we propose the following alternative method for constructing norms : a Wide Norm is any combination of ( asymmetric ) ( semi- ) norms that preserves N1-N4 . It is easy to verify that both ( 1 ) non-negative sums and ( 2 ) max are valid combinations ( indeed , these properties were also used to construct Deep Norms ) , and so the vector-wise MaxMean combination is valid : maxmean ( x1 , x2 , . . . , xn ) = α max ( x1 , x2 , . . . , xn ) + ( 1− α ) mean ( x1 , x2 , . . . , xn ) . Although the family of Wide Norms is broad , for computational reasons to be discussed in Subsection 3.5 , we focus our attention on the Wide Mahalanobis norm . References to “ Wide Norms ” in the rest of this paper refer to Wide Mahalanobis norms . The Mahalanobis norm of x ∈ Rn , parameterized by W ∈ Rm×n , is defined as ‖x‖W = ‖Wx‖2 . It is easily verified that ‖ · ‖W is a proper norm when W is a non-singular ( square ) matrix , and a semi-norm when W is singular or m < n. A k-component Mixture of Mahalanobis norm ( hereafter Wide Norm , or Wide Norm with k Euclidean components ) is defined as the maxmean of k Mahalanobis norms : ‖x‖ = maxmeani ( ‖Wix‖2 ) where Wi ∈ Rmi×n with mi ≤ n. ( 3 ) Wide Norms are symmetric by default , and must be asymmetrized to obtain asymmetric ( semi- ) norms . We use the below property ( Bauer et al. , 1961 ) and propositions ( proofs in Appendix A ) . N5 . ‖ · ‖ is monotonic in the positive orthant if 0 ≤ x ≤ y ( element-wise ) implies ‖x‖ ≤ ‖y‖ . Proposition 4 . If ‖ · ‖ is an N5 ( semi- ) norm on R2n , then ‖x| = ‖relu ( x : : –x ) ‖ , where : : denotes concatenation , is an asymmetric ( semi- ) norm on Rn . Proposition 5 . The Mahalanobis norm with W = DU , with D diagonal and U non-negative , is N5 .

Summary: This manuscript proposes a general framework to learn non-Euclidean distances from data using neural networks. The authors provide a combination of theoretical and experimental results in support of the use of several neural architectures to learn such distances. In particular, the develop “deep norms” and “wide norms”, based either on a deep or shallow neural network. Metrics are elaborated based on norms by combining them with a learnt embedding function mapping the input space de R^n. Theoretical results are mostly application textbook results and intuitive, the overall work forms a coherent line of research bridging theory and applications that sets well justified reference approaches for this topic.