Two Regimes of Generalization for Non-Linear Metric Learning

1 INTRODUCTION . Metric Learning , Bellet et al . ( 2015 ) , is the problem of finding a metric ρ on the space of features , such that ρ reflects some semantic properties of a given task . Generally , the input can be thought of as a set of labeled pairs { ( ( xi , x′i ) , yi ) } ni=1 , where xi , x′i ∈ Rd are the features , and yi is the label , indicating whether xi and x′i should be close in the metric or far apart . For instance , in face identification , Schroff et al . ( 2015 ) , features xi and x′i corresponding to the same face should be close in ρ , while different faces should be far apart . Note that the above metric learning formulation is fairly general and one can convert supervised clustering , or even standard classification problems into metric learning simply by setting yi = 1 if xi and x′i have the same original label and yi = 0 otherwise Davis et al . ( 2007 ) ; Weinberger & Saul ( 2009 ) ; Cao et al . ( 2016 ) ; Khosla et al . ( 2020 ) ; Chicco ( 2020 ) . The metric ρ is typically assumed to be the Euclidean metric taken after a linear or non-linear embedding of the features . That is , we consider a parametric family G of embeddings into a kdimensional space , g : Rd → Rk , and set 1 ρ ( x , x′ ) = ρg ( x , x ′ ) = ‖g ( x ) − g ( x′ ) ‖22 . ( 1 ) As an example , Figure 1 shows tSNE plots , Maaten & Hinton ( 2008 ) , of the classical 20newsgroups dataset . Figure 1a was generated using the regular bag of words representation of the data ( see Sections 5 and D for additional details ) while Figure 1b was generated by applying tSNE to an embedding of the data ( of the form ( 3 ) below ) learned by minimizing a loss based on labels , as described above . Clearly , there is no discernible relation between the label and the metric in the raw representation , but there is a strong relation in the learned metric . One can also obtain similar conclusions , and quantify them , by , for instance , replacing tSNE with spectral clustering . The uniform generalization problem for metric learning is the following : Given a family of embeddings G , provide bounds , that hold uniformly for all g ∈ G , on the difference between the expected value of the loss and the average value of the loss on the train set ; see Section 2 for a formal definition . Such bounds guarantee , for instance , that the train set would not be overfitted . Given a family 1Note that ρ in ( 1 ) is not strictly a metric . Nevertheless , this terminology is common . of embeddings G , consider the family F of functions f : Rd × Rd → R of the form F = { f ( x , x′ ) = ρg ( x , x′ ) | g ∈ G } . ( 2 ) We refer to these as the distance functions , which map a pair of features into their distance after the embedding . Well known arguments imply that one can obtain generalization bounds for F by providing upper bounds on the Rademacher complexity Rn ( F ) of the family of the scalar functions F . Therefore in what follows , we discuss directly the upper bounds on Rn ( F ) for various families G. We refer to Section 2 for formal definitions and details on the relation between G , F , Rn ( F ) , and generalization bounds . 1.1 OVERVIEW OF THE RESULTS . In this paper our goal is to study generalization bounds for neural network type non-linear embeddings g : Rd → Rk given by neural networks of depth L ≥ 1 . Specifically , we consider embeddings of the form g ( x ) = gA ( x ) = φL ( AtL · ( φL−1 ( AtL−1 . . . φ1 ( At1x ) . . . ) ) ) , ( 3 ) where A = ( A1 , . . . , AL ) is a tuple of matrices such that Ai ∈ Rki−1×ki for i = 1 , . . . , L , and ki are the layer widths , with k0 = d and kL = k. The activations φi : R → R are assumed to be Lipschitz , with ‖φ‖Lip ≤ ρi , and act on vectors in Rki coordinatewise . A family of matrix tuples will be generally denoted by A , while the associated embeddings family ( 3 ) will be denoted by G and the associated family of distance functions , ( 2 ) , will be denoted by F . In the context of deep learning generalization guarantees for classification , there exists a large body of work on various complexity bounds related to the family of mappings G. The current strongest uniform bounds were given in Bartlett et al . ( 2017 ) ( see also Neyshabur et al . ( 2018 ) ) . Our goal here is to translate these bounds to bounds on the Rademacher complexity of the family F , derived from G. That is , we study the aspects of the problem that are specific to the metric learning setting . To state the results , we require a few norm definitions : For matrix A ∈ Rs×t , ‖A‖op is the spectral norm , and set ‖A‖2,1 = ∑t i=1 ‖A·i‖2 , where A·i is the i-th column of A . For a family A of matrix tuples , A = ( A1 , . . . , AL ) ∈ A , for i ≤ L we set ‖Ai‖op = sup A∈A ‖Ai‖op and ‖Ai‖2,1 = sup A∈A ‖Ai‖2,1 . ( 4 ) Thus ‖Ai‖op , ‖Ai‖op are the largest respective norms of the component Ai in A . With this notation , our first result is the following bound ( up to logarithmic factors ) : Rn ( F ) ≤ O 1√ n b2 ( L∏ i=1 ρi ‖Ai‖op ) 2 L∑ i=1 ‖Ai‖ 2 3 2,1 ‖Ai‖ 2 3 op 32 . ( 5 ) Here b is an upper bound on inputs , ‖xi‖2 , ‖x′i‖2 ≤ b for i ≤ n. The full statement is given in Section 4 , Theorem 1 . This bound can be regarded as a natural extension of the bounds of Bartlett et al . ( 2017 ) to the metric learning setting . Indeed , it uses the same parameters – the ‖A‖op and ‖A‖2,1 norms . The quantity ( ∏L i=1 ρi ‖Ai‖op ) is the maximal Lipschitz constant of the mappings in G , and it will typically be the dominant term in ( 5 ) . One of the appealing properties of ( 5 ) , inherited from the bounds in Bartlett et al . ( 2017 ) , is that it is dimension free , in the sense that the depth L and the the layer widths ki do not enter the bound explicitly . As noted above , for the case of classification , the bounds in Bartlett et al . ( 2017 ) are currently the strongest known uniform bounds . However , we now show that in the metric learning setting , one can improve the bound in some situations , by using the ‖A‖2 , ∞ = maxi≤k ‖A·i‖2 norm at the last layer . In Theorem 2 , Section 4 , we show the following : Rn ( F ) ≤ O k√ n b2 ( ρL ‖AL‖2 , ∞ L−1∏ i=1 ρi ‖Ai‖op ) 2L−1∑ i=1 ‖Ai‖ 2 3 2,1 ‖Ai‖ 2 3 op + 1 32 . ( 6 ) We refer to ( 5 ) as the sparse bound , and to ( 6 ) as the non-sparse bound . To compare the bounds , and to see the relation to sparsity , let us first consider the simpler single layer setting , L = 1 . In this case , the bounds read Rn ( F ) ≤ O ( b2 ‖φ‖2Lip ‖A‖op ‖A‖2,1√ n ) , ( 7 ) and Rn ( F ) ≤ O ( kb2 ‖φ‖2Lip ‖A‖ 2 2 , ∞√ n ) , ( 8 ) where A is the family of weights matrices , A ∈ Rd×k , φ an activation , and ‖A‖∗ = supA∈A ‖A‖∗ where ‖·‖∗ is one of ‖·‖2,1 , ‖·‖2 , ∞ , ‖·‖op . Consider first the case where A contains non sparse matrices A ∈ A . These are the weights A where all k neurons have roughly the same norm , or equivalently , ‖A‖2,1 ∼ k ‖A‖2 , ∞ . ( 9 ) We refer to this condition as the dense regime . Note that ( 9 ) holds true for neural networks at initialization , and in Section 5 we observe empirically that this also holds for SGD trained networks on MNIST data . Next , when ( 9 ) holds , we also have that ‖A‖op ∼ √ k ‖A‖2 , ∞ . ( 10 ) Indeed , one has ‖A‖op ≤ √ k ‖A‖2 , ∞ for any A , and ( 9 ) implies ‖A‖op ≥ const · √ k d ‖A‖2 , ∞ ( see Lemma 3 in supplementary material A ) . The condition ( 10 ) will be also experimentally verified for MNIST data . Now , substituting ( 9 ) , ( 10 ) into ( 7 ) and ( 8 ) , we obtain that in the non-sparse regime , ( 8 ) is stronger than ( 7 ) by a factor of √ k , a significant improvement . On the other hand , when A is sparse , with most output neurons zeroed out ( e.x . ‖A‖2,1 = ‖A‖op = ‖A‖2 , ∞ ) , ( 7 ) will be much stronger . Such sparse networks can be achieved by adding explicit sparsity regularization terms to the cost . This will be the case for networks trained on the 20newsgroups dataset . Finally , as evident from ( 5 ) and ( 6 ) , for the multilayer case L > 1 , the relation between the sparse and non-sparse bounds is more involved . However , the asymptotics of the dependence on k is similar : With all other ki fixed , ( 6 ) will be better by a factor of √ k for non-sparse last layer AL , and ( 5 ) will be better otherwise . We now discuss an additional direction in which the bounds may be improved : Note that the dependence of the bound on the coefficients of the weights in ( 5 ) and ( 6 ) is quadratic . As we show in Section 4 , this is in general unavoidable , due to the quadratic form of the metric ( 1 ) . However , if the highest activation , φL is bounded , such as for instance the sigmoid φ ( x ) = ( 1 + e−x ) −1 , then one can obtain a linearly homogeneous dependence on the layer weights , thus potentially significantly improving the bounds . These bounds are given in ( 18 ) and ( 21 ) in Theorems 1 and 2 respectively . For comparison , in the single layer case the sparse and non-sparse bounded φL bounds are given by Rn ( F ) ≤ O ( √ kb ‖φ‖∞ ‖φ‖Lip ‖A‖2,1√ n ) and Rn ( F ) ≤ O ( kb ‖φ‖∞ ‖φ‖Lip ‖A‖2 , ∞√ n ) , ( 11 ) where ‖φ‖∞ is the upper bound on the values of φ .

Summary: In this paper, the authors look at the Rademacher complexity of the family of Euclidean metrics learned on a data set via an $L$ layer network where the activations functions are Lipschitz. The idea behind the proof is to use the bounds for $\epsilon$ net for the embedding network from Barlett, Foster, and Telgarsky 2017. This net, then provides a net for space of metrics with only requiring a change in the constants. Using this net, the authors then use standard arguments to bound the Rademacher complexity.   Specifically for the metric learning setting they show that this bound can be improved.