Measuring and regularizing networks in function space

1 INTRODUCTION . A neural network ’ s parameters collectively encode a function that maps inputs to outputs . The goal of learning is to converge upon a good input/output function . In analysis , then , a researcher should ideally consider how a network ’ s input/output function changes relative to the space of possible functions . However , since this space is not often considered tractable , most techniques and analyses consider the parameters of neural networks . Most regularization techniques , for example , act directly on the parameters ( e.g . weight decay , or the implicit constraints stochastic gradient descent ( SGD ) places upon movement ) . These techniques are valuable to the extent that parameter space can be taken as a proxy for function space . Since the two might not always be easily related , and since we ultimately care most about the input/output function , it is important to develop metrics that are directly applicable in function space . In this work we show that it is relatively straightforward to measure the distance between two networks in function space , at least if one chooses the right space . Here we examine L2-space , which is a Hilbert space . Distance in L2 space is simply the expected ` 2 distance between the outputs of two functions when given the same inputs . This computation relies only on function inference . Using this idea of function space , we first focus on characterizing how networks move in function space during optimization with SGD . Do random initializations track similar trajectories ? What happens in the overfitting regime ? We are particularly interested in the relationship between trajectories in function space and parameter space . If the two are tightly coupled , then parameter change can be taken as a proxy for function change . This common assumption ( e.g . Lipschitz bounds ) might not always be the case . ∗aarrii @ seas.upenn.edu Next , we demonstrate two possibilities as to how a function space metric could assist optimization . In the first setting we consider multitask learning , and the phenomenon of catastrophic forgetting that makes it difficult . Many well-known methods prevent forgetting by regularizing how much the parameters are allowed to shift due to retraining ( usually scaled by a precision matrix calculated on previous tasks ) . We show that one can instead directly regularize changes in the input/output function of early tasks . Though this requires a ” working memory ” of earlier examples , this scheme turns out to be quite data-efficient ( and more so than actually retraining on examples from old tasks ) . In the second setting we propose a learning rule for supervised learning that constrains how much a network ’ s function can change any one update . This rule , which we call Hilbert-constrained gradient descent ( HCGD ) , penalizes each step of SGD to reduce the magnitude of the resulting step in L2-space . This learning rule thus changes the course of learning to track a shorter path in function space . If SGD generalizes in part because large changes to the function are prohibited , then this rule will have advantages over SGD . Interestingly , HCGD is conceptually related to the natural gradient . As we derive in §3.2.1 , the natural gradient can be viewed as resulting from constrains changes in a function space measured by the Kullbeck-Leibler divergence . 2 EXAMINING NETWORKS IN FUNCTION SPACE . We propose to examine the trajectories of networks in the space of functions defined by the inner product 〈f , g〉 = ∫ X f ( x ) g ( x ) dµ ( x ) , which yields the following norm : ‖f‖2 = ∫ X |f |2dµ . Here µ is a measure and corresponds to the probability density of the input distribution X . Note that this norm is over an empirical distribution of data and not over the uniform distribution of all possible inputs . The | · |2 operator refers to the 2-norm and can apply to vector-valued functions . While we refer to this space as a Hilbert space , we make no use of an inner product and can also speak of this as any normed vector space , e.g . a Banach space . This norm leads to a notion of distance between two functions f and g given by ‖f − g‖2 = ∫ X |f − g|2dµ . Since µ is a density , ∫ X dµ = 1 , and we can write ‖f − g‖2 = EX [ |f ( x ) − g ( x ) |2 ] . The expectation can be approximated as an empirical expectation over a batch of examples drawn from the input distribution : ‖f − g‖2 ≈ 1 N N∑ i=0 |f ( xi ) − g ( xi ) |2 . The quality of the empirical distance , of course , will depend on the shape and variance of the distribution of data as well as the form of f and g. In section 2.3 , we empirically investigate the quality of this estimator for reasonably sample sizes N . 2.1 DIVERGENCE OF NETWORKS IN L2-SPACE DURING TRAINING We wish to compare at high level how networks move through parameter and function space . Our first approach is to compare a low-dimensional embedding of the trajectories through these spaces . In Figure 1 , we take a convolutional neural network and train three random initializations on a 5000-image subset of CIFAR-10 . By saving the parameters of the network at each epoch as well as the output on a single large validation batch , we can later compute the ` 2 parameter distance and the L2 function distance between the snapshots of network at each epoch . The resulting distance matrix is then visualized as a two-dimensional embedding . In parameter space , the networks are initialized at very different points and proceed to diverge yet further from these points . Despite this divergence , each trajectory yields a network that has learned the training data perfectly and generalizes with ∼ 50 % accuracy to a test set . This illustrates the wide range of parameter settings that can be used to represent a given neural network function . The behavior of the same initializations in function space is quite different . First , note that all three initializations begin at approximately the same point in function space . This is an intriguing property of random initializations that , rather than encoding entirely random functions , random sets of parameters lead on average to the same function ( for related work , see e.g . Giryes et al . ( 2016 ) ) . The initializations then largely follow an identical path for the initial stage of learning . Different initializations thus learn in similar manners , even if the distance between their parameters diverges . During late-stage optimization , random initializations turn from a shared trajectory and begin to diverge in L2 space . These differences underlie the general principle that L2 distances behave differently than ` 2 distances , and that functional regularization could assist training and reduce overfitting . 2.2 COMPARING L2 FUNCTION DISTANCE WITH ` 2 PARAMETER DISTANCE . How well do parameter distances reflect function distances ? The answer to this question is relevant for any method that directly considers the behavior of parameters . Certain theoretical analyses , furthermore , desire bounds on function distances but instead find bounds on parameter distances and relate the two with a Lipschitz constant ( e.g . Hardt et al . ( 2015 ) ) . Thus , for theoretical analyses and optimization methods alike , it is important to empirically evaluate how well parameter distances correspond to function distances in typical situations . We can compare these two situations by plotting a change in parameters ‖∆θ‖ against the corresponding change in the function ‖fθ − fθ+∆θ‖ . In Figure 2 we display this relation for several relevant distance during the optimization of a CNN on CIFAR-10 . There are three scales : the distance between individual updates , the distance between epochs , and the distance from initialization . Note , first , that networks continue to move in function space as well as in parameter space after test error converges , which is around epoch 60 . ( The test error can be seen in Appendix A , along with identical plots colored by test error instead of epoch . ) Their movement relative to initialization slows , but there is still large movement relative to previous iterations and previous epochs . What changes strikingly throughout optimization is the relationship between parameter and function space . There is a qualitative difference in the ratio of parameter distances to function distances that visible at all three distance scales . Early epochs generally see larger changes in L2 space for a given change in parameters . Intriguingly , the ratio of the two distances appears to converge to a single value at late optimization , after test error saturates . This is not because the network ceases to move , as noted above . Rather , the loss landscape shifts such that this ratio become constant . It is also clear from these plots that there is not a consistent positive correlation between the parameter and function distances between any two points on the optimization trajectory . For example , the parameter distance between successive epochs is negatively correlated with the L2 distance for most of optimization ( Fig . 2b ) . The distance from initialization shows a clean and positive relationship , but the relationship changes during optimization . Between successive batches , L2 distance correlates with parameter distance at late epochs , but less so early in optimization when learning is quickest . Thus , at different stages of optimization , the L2/ ` 2 ratio is often quite different . The usage of Batch Normalization ( BN ) and weight decay in this analysis somewhat affects the trends in the L2/ ` 2 ratio . In Appendix A we reproduce these plots for networks trained without BN and without weight decay . The overall message that the L2/ ` 2 ratio changes during optimization is unchanged . However , these methods both change the scale of updates , and appear to do so differently throughout optimization , and thus some trends are different . In Appendix B , we also isolate the effect of training data , by reproducing these plots for a CNN trained on MNIST and find similar trends . Overall , the correspondence between parameter and function distances depends strongly on the context . 2.3 CONVERGENCE OF THE EMPIRICAL ESTIMATOR . It might be worried that since function space is of infinite dimension , one would require prohibitively many examples to estimate a function distance . However , we find that one can compute a distance between two functions with a relatively small amount of examples . Figure 3 shows how the estimated L2 distance converges with an increasing number examples . In general , we find that only a few hundred examples are necessary to converge to an estimation within a few percent . 3 APPLICATIONS . 3.1 COMBATTING CATASTROPHIC FORGETTING IN AN ONLINE LEARNING TASK ( WITH WORKING MEMORY ) . If , after having been trained on a task , a neural network is retrained on a new task , it often forgets the first task . This phenomenon is termed ’ catastrophic forgetting ’ . It is the central difficulty of multitask training as well as applications requiring that learning be done online ( especially in non-IID situations ) . Essentially , new information must be encoded in the network , but the the information pertinent to the previous task must not be overwritten . Most efforts to combat catastrophic forgetting rely on restricting how much parameters can change between tasks . Elastic Weight Consolidation ( EWC ; Kirkpatrick et al . ( 2017 ) ) , for example , adds a penalty to the loss on a new task B that is the distance from the weights after learning on an earlier task A , multiplied by the diagonal of the Fisher information matrix F ( calculated on task A ) : LEWC ( θ ) = LB ( θ ) + λ 2 ∑ i Fi ( θi − θi , A ) 2 This idea is closely related to well-studied approaches to Bayesian online learning , if F is interpreted as a precision matrix ( Honkela & Valpola ( 2003 ) , Opper & Winther ( 1998 ) ) . Other similar approaches include that of Ritter et al . ( 2018 ) , who use a more accurate approximation of the Fisher , and Synaptic Intelligence ( SI ; Zenke et al . ( 2017 ) ) , which discounts parameter change via a diagonal matrix in which each entry reflects the sum contribution of that parameter to the loss . Each of these method discourages catastrophic forgetting by restricting movement in parameter space between tasks , scaled by a ( perhaps diagonal ) precision matrix calculated on previous tasks . Using a function space metric , it is not hard to ensure that the network ’ s output function on previous tasks does not change during learning . In this case , the loss for a new task B is modified to be : L ( θ ) = LB ( θ ) + λ 2 ‖fθA − fθB‖ The regularization term is the L2 distance between the current function fθB and the function after training on task A , fθA . Since our function space metric is defined over a domain of examples , we will store a small set of previously seen examples in a working memory , as well as the output on those examples . This memory set will be used to calculate the L2 distance between the current iteration and the snapshot after training . This is a simple scheme , but novel , and we are not aware of direct precedence in the literature . A working memory approach is employed in related work ( Lopez-Paz et al . ( 2017 ) ; Rebuffi et al . ( 2017 ) ) . Note , however , that storing old examples violates the rules of strict online learning . Nevertheless , for large networks it will be more memory-efficient . EWC , for example , requires storing a snapshot of each parameter at the end of the previous task , as well as a diagonal precision matrix with as many entries as parameters . For the 2 hidden layer network with 400 nodes each that was used in the MNIST task in Kirkpatrick et al . ( 2017 ) , this is 1,148,820 new parameters , or as many pixels as 1,465 MNIST images . When each layer has as many as 2,000 nodes , as in Fig . 3B of Kirkpatrick et al . ( 2017 ) , the extra stored parameters are comparable to 15,489 MNIST images . The working memory approach that is required to regularize function change from old tasks is thus comparable or cheaper in memory .

Summary: This paper proposes a method for functional regularization for training neural nets, such that the sequence of neural nets during training is stable in function space. Specifically, the authors define a L2 norm (i.e., a Hilbert norm), which can be used to measure distances in this space between two functions. The authors argue that this can aid in preventing catastrophic forgetting, which is demonstrated in a synthetic multi-task variant of MNIST.   The authors also show how to regularize the gradient updates to be conservative in function space in standard stochastic gradient style learning, but with rather inconclusive empirical results.  The authors also draw upon a connection to the natural gradient.