Two Regimes of Generalization for Non-Linear Metric Learning

1 INTRODUCTION . Metric Learning , Bellet et al . ( 2015 ) , is the problem of finding a metric ρ on the space of features , such that ρ reflects some semantic properties of a given task . Generally , the input can be thought of as a set of labeled pairs { ( ( xi , x′i ) , yi ) } ni=1 , where xi , x′i ∈ Rd are the features , and yi is the label , indicating whether xi and x′i should be close in the metric or far apart . For instance , in face identification , Schroff et al . ( 2015 ) , features xi and x′i corresponding to the same face should be close in ρ , while different faces should be far apart . Note that the above metric learning formulation is fairly general and one can convert supervised clustering , or even standard classification problems into metric learning simply by setting yi = 1 if xi and x′i have the same original label and yi = 0 otherwise Davis et al . ( 2007 ) ; Weinberger & Saul ( 2009 ) ; Cao et al . ( 2016 ) ; Khosla et al . ( 2020 ) ; Chicco ( 2020 ) . The metric ρ is typically assumed to be the Euclidean metric taken after a linear or non-linear embedding of the features . That is , we consider a parametric family G of embeddings into a kdimensional space , g : Rd → Rk , and set 1 ρ ( x , x′ ) = ρg ( x , x ′ ) = ‖g ( x ) − g ( x′ ) ‖22 . ( 1 ) As an example , Figure 1 shows tSNE plots , Maaten & Hinton ( 2008 ) , of the classical 20newsgroups dataset . Figure 1a was generated using the regular bag of words representation of the data ( see Sections 5 and D for additional details ) while Figure 1b was generated by applying tSNE to an embedding of the data ( of the form ( 3 ) below ) learned by minimizing a loss based on labels , as described above . Clearly , there is no discernible relation between the label and the metric in the raw representation , but there is a strong relation in the learned metric . One can also obtain similar conclusions , and quantify them , by , for instance , replacing tSNE with spectral clustering . The uniform generalization problem for metric learning is the following : Given a family of embeddings G , provide bounds , that hold uniformly for all g ∈ G , on the difference between the expected value of the loss and the average value of the loss on the train set ; see Section 2 for a formal definition . Such bounds guarantee , for instance , that the train set would not be overfitted . Given a family 1Note that ρ in ( 1 ) is not strictly a metric . Nevertheless , this terminology is common . of embeddings G , consider the family F of functions f : Rd × Rd → R of the form F = { f ( x , x′ ) = ρg ( x , x′ ) | g ∈ G } . ( 2 ) We refer to these as the distance functions , which map a pair of features into their distance after the embedding . Well known arguments imply that one can obtain generalization bounds for F by providing upper bounds on the Rademacher complexity Rn ( F ) of the family of the scalar functions F . Therefore in what follows , we discuss directly the upper bounds on Rn ( F ) for various families G. We refer to Section 2 for formal definitions and details on the relation between G , F , Rn ( F ) , and generalization bounds . 1.1 OVERVIEW OF THE RESULTS . In this paper our goal is to study generalization bounds for neural network type non-linear embeddings g : Rd → Rk given by neural networks of depth L ≥ 1 . Specifically , we consider embeddings of the form g ( x ) = gA ( x ) = φL ( AtL · ( φL−1 ( AtL−1 . . . φ1 ( At1x ) . . . ) ) ) , ( 3 ) where A = ( A1 , . . . , AL ) is a tuple of matrices such that Ai ∈ Rki−1×ki for i = 1 , . . . , L , and ki are the layer widths , with k0 = d and kL = k. The activations φi : R → R are assumed to be Lipschitz , with ‖φ‖Lip ≤ ρi , and act on vectors in Rki coordinatewise . A family of matrix tuples will be generally denoted by A , while the associated embeddings family ( 3 ) will be denoted by G and the associated family of distance functions , ( 2 ) , will be denoted by F . In the context of deep learning generalization guarantees for classification , there exists a large body of work on various complexity bounds related to the family of mappings G. The current strongest uniform bounds were given in Bartlett et al . ( 2017 ) ( see also Neyshabur et al . ( 2018 ) ) . Our goal here is to translate these bounds to bounds on the Rademacher complexity of the family F , derived from G. That is , we study the aspects of the problem that are specific to the metric learning setting . To state the results , we require a few norm definitions : For matrix A ∈ Rs×t , ‖A‖op is the spectral norm , and set ‖A‖2,1 = ∑t i=1 ‖A·i‖2 , where A·i is the i-th column of A . For a family A of matrix tuples , A = ( A1 , . . . , AL ) ∈ A , for i ≤ L we set ‖Ai‖op = sup A∈A ‖Ai‖op and ‖Ai‖2,1 = sup A∈A ‖Ai‖2,1 . ( 4 ) Thus ‖Ai‖op , ‖Ai‖op are the largest respective norms of the component Ai in A . With this notation , our first result is the following bound ( up to logarithmic factors ) : Rn ( F ) ≤ O 1√ n b2 ( L∏ i=1 ρi ‖Ai‖op ) 2 L∑ i=1 ‖Ai‖ 2 3 2,1 ‖Ai‖ 2 3 op 32 . ( 5 ) Here b is an upper bound on inputs , ‖xi‖2 , ‖x′i‖2 ≤ b for i ≤ n. The full statement is given in Section 4 , Theorem 1 . This bound can be regarded as a natural extension of the bounds of Bartlett et al . ( 2017 ) to the metric learning setting . Indeed , it uses the same parameters – the ‖A‖op and ‖A‖2,1 norms . The quantity ( ∏L i=1 ρi ‖Ai‖op ) is the maximal Lipschitz constant of the mappings in G , and it will typically be the dominant term in ( 5 ) . One of the appealing properties of ( 5 ) , inherited from the bounds in Bartlett et al . ( 2017 ) , is that it is dimension free , in the sense that the depth L and the the layer widths ki do not enter the bound explicitly . As noted above , for the case of classification , the bounds in Bartlett et al . ( 2017 ) are currently the strongest known uniform bounds . However , we now show that in the metric learning setting , one can improve the bound in some situations , by using the ‖A‖2 , ∞ = maxi≤k ‖A·i‖2 norm at the last layer . In Theorem 2 , Section 4 , we show the following : Rn ( F ) ≤ O k√ n b2 ( ρL ‖AL‖2 , ∞ L−1∏ i=1 ρi ‖Ai‖op ) 2L−1∑ i=1 ‖Ai‖ 2 3 2,1 ‖Ai‖ 2 3 op + 1 32 . ( 6 ) We refer to ( 5 ) as the sparse bound , and to ( 6 ) as the non-sparse bound . To compare the bounds , and to see the relation to sparsity , let us first consider the simpler single layer setting , L = 1 . In this case , the bounds read Rn ( F ) ≤ O ( b2 ‖φ‖2Lip ‖A‖op ‖A‖2,1√ n ) , ( 7 ) and Rn ( F ) ≤ O ( kb2 ‖φ‖2Lip ‖A‖ 2 2 , ∞√ n ) , ( 8 ) where A is the family of weights matrices , A ∈ Rd×k , φ an activation , and ‖A‖∗ = supA∈A ‖A‖∗ where ‖·‖∗ is one of ‖·‖2,1 , ‖·‖2 , ∞ , ‖·‖op . Consider first the case where A contains non sparse matrices A ∈ A . These are the weights A where all k neurons have roughly the same norm , or equivalently , ‖A‖2,1 ∼ k ‖A‖2 , ∞ . ( 9 ) We refer to this condition as the dense regime . Note that ( 9 ) holds true for neural networks at initialization , and in Section 5 we observe empirically that this also holds for SGD trained networks on MNIST data . Next , when ( 9 ) holds , we also have that ‖A‖op ∼ √ k ‖A‖2 , ∞ . ( 10 ) Indeed , one has ‖A‖op ≤ √ k ‖A‖2 , ∞ for any A , and ( 9 ) implies ‖A‖op ≥ const · √ k d ‖A‖2 , ∞ ( see Lemma 3 in supplementary material A ) . The condition ( 10 ) will be also experimentally verified for MNIST data . Now , substituting ( 9 ) , ( 10 ) into ( 7 ) and ( 8 ) , we obtain that in the non-sparse regime , ( 8 ) is stronger than ( 7 ) by a factor of √ k , a significant improvement . On the other hand , when A is sparse , with most output neurons zeroed out ( e.x . ‖A‖2,1 = ‖A‖op = ‖A‖2 , ∞ ) , ( 7 ) will be much stronger . Such sparse networks can be achieved by adding explicit sparsity regularization terms to the cost . This will be the case for networks trained on the 20newsgroups dataset . Finally , as evident from ( 5 ) and ( 6 ) , for the multilayer case L > 1 , the relation between the sparse and non-sparse bounds is more involved . However , the asymptotics of the dependence on k is similar : With all other ki fixed , ( 6 ) will be better by a factor of √ k for non-sparse last layer AL , and ( 5 ) will be better otherwise . We now discuss an additional direction in which the bounds may be improved : Note that the dependence of the bound on the coefficients of the weights in ( 5 ) and ( 6 ) is quadratic . As we show in Section 4 , this is in general unavoidable , due to the quadratic form of the metric ( 1 ) . However , if the highest activation , φL is bounded , such as for instance the sigmoid φ ( x ) = ( 1 + e−x ) −1 , then one can obtain a linearly homogeneous dependence on the layer weights , thus potentially significantly improving the bounds . These bounds are given in ( 18 ) and ( 21 ) in Theorems 1 and 2 respectively . For comparison , in the single layer case the sparse and non-sparse bounded φL bounds are given by Rn ( F ) ≤ O ( √ kb ‖φ‖∞ ‖φ‖Lip ‖A‖2,1√ n ) and Rn ( F ) ≤ O ( kb ‖φ‖∞ ‖φ‖Lip ‖A‖2 , ∞√ n ) , ( 11 ) where ‖φ‖∞ is the upper bound on the values of φ .

Summary: The paper provides two new generalization bounds for non-linear metric learning with deep neural networks, by extending results of Bartlett et al. 2017 to the metric learning setting. The two bounds have been called the 'sparse' and 'non-sparse' bounds and differ in the norm used for the last layer. Experiments are performed where it is shown that either bound may dominate on real datasets.