Relative stability toward diffeomorphisms indicates performance in deep nets

√ Rf , suggesting that obtaining a smallRf is important to achieve good performance . We study how Rf depends on the size of the training set and compare it to a simple model of invariant learning . 1 Introduction . Deep learning algorithms LeCun et al . ( 2015 ) are now remarkably successful at a wide range of tasks Amodei et al . ( 2016 ) ; Huval et al . ( 2015 ) ; Mnih et al . ( 2013 ) ; Shi et al . ( 2016 ) ; Silver et al . ( 2017 ) . Yet , understanding how they can classify data in large dimensions remains a challenge . In particular , the curse of dimensionality associated with the geometry of space in large dimension prohibits learning in a generic setting Luxburg and Bousquet ( 2004 ) . If high-dimensional data can be learnt , then they must be highly structured . A popular idea is that during training , hidden layers of neurons learn a representation Le ( 2013 ) that is insensitive to aspects of the data unrelated to the task , effectively reducing the input dimension and making the problem tractable Ansuini et al . ( 2019 ) ; Recanatesi et al . ( 2019 ) ; Shwartz-Ziv and Tishby ( 2017 ) . Several quantities have been introduced to study this effect empirically . It includes ( i ) the mutual information between the hidden and visible layers of neurons Saxe et al . ( 2019 ) ; Shwartz-Ziv and Tishby ( 2017 ) , ( ii ) the intrinsic dimension of the neural representation of the data Ansuini et al . ( 2019 ) ; Recanatesi et al . ( 2019 ) and ( iii ) the projection of the label of the data on the main features of the network Kopitkov and Indelman ( 2020 ) ; Oymak et al . ( 2019 ) ; Paccolat et al . ( 2021a ) , the latter being defined from the top eigenvectors of the Gram matrix of the neural tangent kernel ( NTK ) Jacot et al . ( 2018 ) . All these measures support that the neuronal representation of the data indeed becomes well-suited to the task . Yet , they are agnostic to the nature of what varies in the data that need not being represented by hidden neurons , and thus do not specify what it is . Recently , there has been a considerable effort to understand the benefits of learning features for onehidden-layer fully connected nets . Learning features can occur and improve performance when the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . true function is highly anisotropic , in the sense that it depends only on a linear subspace of the input space Bach ( 2017 ) ; Chizat and Bach ( 2020 ) ; Ghorbani et al . ( 2019 , 2020 ) ; Paccolat et al . ( 2021a ) ; Refinetti et al . ( 2021 ) ; Yehudai and Shamir ( 2019 ) . For image classification , such an anisotropy would occur for example if pixels on the edge of the image are unrelated to the task . Yet , fully-connected nets ( unlike CNNs ) acting on images tend to perform best in training regimes where features are not learnt Geiger et al . ( 2021 , 2020 ) ; Lee et al . ( 2020 ) , suggesting that such a linear invariance in the data is not central to the success of deep nets . Instead , it has been proposed that images can be classified in high dimensions because classes are invariant to smooth deformations or diffeomorphisms of small magnitude Bruna and Mallat ( 2013 ) ; Mallat ( 2016 ) . Specifically , Mallat and Bruna could handcraft convolution networks , the scattering transforms , that perform well and are stable to smooth transformations , in the sense that ‖f ( x ) − f ( τx ) ‖ is small if the norm of the diffeomorphism τ is small too . They hypothesized that during training deep nets learn to become stable and thus less sensitive to these deformations , thus improving performance . More recent works generalize this approach to more common CNNs and discuss stability at initialization Bietti and Mairal ( 2019a , b ) . Interestingly , enforcing such a stability can improve performance Kayhan and Gemert ( 2020 ) . Answering if deep nets become more stable to smooth deformations when trained and quantifying how it affects performance remains a challenge . Recent empirical results revealed that small shifts of images can change the output a lot Azulay and Weiss ( 2018 ) ; Dieleman et al . ( 2016 ) ; Zhang ( 2019 ) , in apparent contradiction with that hypothesis . Yet in these works , image transformations ( i ) led to images whose statistics were very different from that of the training set or ( ii ) were cropping the image , thus are not diffeophormisms . In Ruderman et al . ( 2018 ) , a class of diffeomorphisms ( low-pass filter in spatial frequencies ) was introduced to show that stability toward them can improve during training , especially in architectures where pooling layers are absent . Yet , these studies do not address how stability affects performance , and how it depends on the size of the training set . To quantify these properties and to find robust empirical behaviors across architectures , we will argue that the evolution of stability toward smooth deformations needs to be compared relatively to that of any deformation , which turns out to vary significantly during training . Note that in the context of adversarial robustness , attacks that are geometric transformations of small norm that change the label have been studied Alaifari et al . ( 2018 ) ; Alcorn et al . ( 2019 ) ; Athalye et al . ( 2018 ) ; Engstrom et al . ( 2019 ) ; Fawzi and Frossard ( 2015 ) ; Kanbak et al . ( 2018 ) ; Xiao et al . ( 2018 ) . These works differ for the literature above and from out study below in the sense that they consider worst-case perturbations instead of typical ones . 1.1 Our Contributions . ◦ We introduce a maximum entropy distribution of diffeomorphisms , that allow us to generate typical diffeomorphisms of controlled norm . Their amplitude is governed by a `` temperature '' parameter T . ◦ We define the relative stability to diffeomorphisms index Rf that characterizes the square magnitude of the variation of the output function f with respect to the input when it is transformed along a diffeomorphism , relatively to that of a random transformation of the same amplitude . It is averaged on the test set as well as on the ensemble of diffeomorphisms considered . ◦ We find that at initialization , Rf is close to unity for various data sets and architectures , indicating that initially the output is as sensitive to smooth deformations as it is to random perturbations of the image . ◦ Our central result is that after training , Rf correlates very strongly with the test error t : during training , Rf is reduced by several decades in current State Of The Art ( SOTA ) architectures on four benchmark datasets including MNIST Lecun et al . ( 1998 ) , FashionMNIST Xiao et al . ( 2017 ) , CIFAR-10 Krizhevsky ( 2009 ) and ImageNet Deng et al . ( 2009 ) . For more primitive architectures ( whose test error is higher ) such as fully connected nets or simple CNNs , Rf remains of order unity . For CIFAR10 we study 15 known architectures and find empirically that t ≈ 0.2 √ Rf . ◦ Rf decreases with the size of the training set P . We compare it to an inverse power 1/P expected in simple models of invariant learning Paccolat et al . ( 2021a ) . The library implementing diffeomorphisms on images is available online at github.com/pcslepfl/diffeomorphism . The code for training neural nets can be found at github.com/leonardopetrini/diffeo-sota and the corresponding pre-trained models at doi.org/10.5281/zenodo.5589870 . 2 Maximum-entropy model of diffeomorphisms . 2.1 Definition of maximum entropy model . We consider the case where the input vector x is an image . It can be thought as a function x ( s ) describing intensity in position s = ( u , v ) ∈ [ 0 , 1 ] 2 , where u and v are the horizontal and vertical coordinates . To simplify notations we consider a single channel , in which case x ( s ) is a scalar ( but our analysis holds for colored images as well ) . We denote by τx the image deformed by τ , i.e . [ τx ] ( s ) = x ( s − τ ( s ) ) . τ ( s ) is a vector field of components ( τu ( s ) , τv ( s ) ) . The deformation amplitude is measured by the norm ‖∇τ‖2 = ∫ [ 0,1 ] 2 ( ( ∇τu ) 2 + ( ∇τv ) 2 ) dudv . ( 1 ) To test the stability of deep nets toward diffeomorphisms , we seek to build typical diffeomorphisms of controlled norm ‖∇τ‖ . We thus consider the distribution over diffeomorphisms that maximizes the entropy with a norm constraint . It can be solved by introducing a Lagrange multiplier T and by decomposing these fields on their Fourier components , see e.g . Kardar ( 2007 ) or Appendix A . In this canonical ensemble , one finds that τu and τv are independent with identical statistics . For the picture frame not to be deformed , we impose fixed boundary conditions : τ = 0 if u = 0 , 1 or v = 0 , 1 . One then obtains : τu = ∑ i , j∈N+ Cij sin ( iπu ) sin ( jπv ) ( 2 ) where the Cij are Gaussian variables of zero mean and variance 〈C2ij〉 = T/ ( i2 + j2 ) . If the picture is made of n× n pixels , the result is identical except that the sum runs on 0 < i , j ≤ n. For large n , the norm then reads ‖∇τ‖2 = ( π2/2 ) n2T , and is dominated by high spatial frequency modes . It is useful to add another parameter c to cut-off the effect of high spatial frequencies , which can be simply done by constraining the sum in Eq.2 to i2 + j2 ≤ c2 , one then has ‖∇τ‖2 = ( π3/8 ) c2T . Once τ is generated , pixels are displaced to random positions . A new pixelated image can then be obtained using standard interpolation methods . We use two interpolations , Gaussian and bi-linear1 , as described in Appendix C. As we shall see below , this choice does not affect our result as long as the diffeomorphism induced a displacement of order of the pixel size , or larger . Examples are shown in Fig.1 as a function of T and c . 2.2 Phase diagram of acceptable diffeomorphisms . Diffeomorphisms are bijective , which is not the case for our transformations if T is too large . When this condition breaks down , a single domain of the picture can break into several pieces , as apparent in Fig.1 . It can be expressed as a condition on ∇τ that must be satisfied in every point in space Lowe ( 2004 ) , as recalled in Appendix B . This is satisfied locally with high probability if ‖τ‖2 1 , corresponding to T ( 8/π3 ) /c2 . In Appendix , we extract empirically a curve of similar form in the ( T , c ) plane at which a diffeomorphism is obtained with probability at least 1/2 . For much smaller T , diffeomorphisms are obtained almost surely . Finally , for diffeomorphisms to have noticeable consequences , their associated displacement must be of the order of magnitude of the pixel size . Defining δ2 as the average square norm of the pixel displacement at the center of the image in the unit of pixel size , it is straightforward to obtain from Eq.2 that asymptotically for large c ( cf . Appendix B for the derivation ) , δ2 = π 4 n2T ln ( c ) . ( 3 ) The line δ = 1/2 is indicated in Fig.1 , using empirical measurements that add pre-asymptotic terms to Eq.3 . Overall , the green region corresponds to transformations that ( i ) are diffeomorphisms with high probability and ( ii ) produce significant displacements at least of the order of the pixel size . 1Throughout the paper , if not specified otherwise , bi-linear interpolation is employed .

Summary: This paper introduces a new quantity called "relative stability" that strongly correlates with the classification performance of state of the art architectures on standard datasets. This quantity is a ratio between the variation of the features with respect to diffeomorphic transformation of the images and simple additive transformations.  To do so, the authors propose to use a parametrized set of diffemorphisms with bounded L2 norm of the derivative.  The dependence on the training set is highlighted, in particular a given amount of data is necessary to see the "relative stability" decrease with the size training set. Last, a simple synthetic experiment is proposed that illustrates the expected behaviour.