Image Functions In Neural Networks: A Perspective On Generalization
Authors: Arushi Gupta
Venue: ICLR 2022 Submitted (2022)
Keywords: generalization
ensembling
algorithmic stability

Abstract: In this work, we show that training with SGD on ReLU neural networks gives rise to a natural set of functions for each image that are not perfectly correlated until later in training. Furthermore, we show experimentally that the intersection of paths for different images also changes during the course of training. We hypothesize that this lack of correlation and changing intersection may be a factor in explaining generalization, because it encourages the model to use different features at different times, and pass the same image through different functions during training. This may improve generalization in two ways. 1) By encouraging the model to learn the same image in different ways, and learn different commonalities between images, comparable to model ensembling. 2) By improving algorithmic stability, as for a particular feature, the model is not always reliant on the same set of images, so the removal of an image may not adversely affect the loss.

Under review as a conference paper at ICLR 2022
IMAGE FUNCTIONS IN NEURAL NETWORKS: A PER- SPECTIVE ON GENERALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we show that training with SGD on ReLU neural networks gives rise to a natural set of functions for each image that are not perfectly correlated until later in training. Furthermore, we show experimentally that the intersec- tion of paths for different images also changes during the course of training. We hypothesize that this lack of correlation and changing intersection may be a fac- tor in explaining generalization, because encourages features at different times, and pass the same image through different functions during training. This may improve generalization in two ways. 1) By encouraging the model to learn the same image in different ways, and Iearn different monalities between images, comparable to model ensembling. 2) By improving algorithmic stability, for particular feature, the model is not always reliant the same set of images, so the removal of an image may not adversely affect the loss.
model use different
INTRODUCTION
Determining why neural networks generalize remains interesting open problem. Training often succeeds even without the use of explicit regularizers like Dropout Srivastava et al. (2014) and even when the model is trained all the way to 100% training accuracy, in the presence of over- aeeepparametriaation ZZang et al.] (021). Existing bounds using capacity control Neyshabur VC dimension difficult analyze when there
et al. 20099, e.g. Rademacher complexity is zero label noise and zero empirical risk Belkin et al. 2018). Several interesting approaches have been explored explain the generalization phenomenon. These include approaches based on
applying algorithmic stability Bousquet & Elisseeti (2000) to SGD Hardt et al. (2016) Kuzborskij
Roy (2017) approaches exploring properties of neural network functions like elasticity He & Su 2019). It has also been hypothesized that neural networks have spectral bias towards low frequency functions Rahaman et al. (2019)
Lampert 2018) PAC Bayes based bounds Dziugate
While discovering neural networks generalize small datasets, Olson et al. (2018) introduced an algorithm to decompose a neural network into a set uncorrelated trees, possibly explaining their ability to perform variance reduction.
For many machine learning models, boosting Schapire (003) and bagging Breiman 1996) are employed to improve generalization performance by either averaging over different trained models, or resampling the dataset so it contains different distributions of data points. The underlying intuition is that different models are unlikely make the same mistake. One can also train models on different subsets of features and take an average to reduce variance. Additionally, one promising line of investigation surrounding neural network generalization that of algorithmic stability, as analyzed by Bousquet & Elissecif (2000) and Hardt et al. (2016) for the case of SGD.
In this work, we consider ReLU networks trained using SGD on image data. We show that training with SGD on ReLU neural networks gives rise to a natural set of functions for each image that are not perfectly correlated until later in training. Furthermore, we show experimentally that the intersection of paths for different images also changes during the course of training. We hypothesize that this lack of correlation and changing intersection may be factor in explaining generalization, because it encourages the model to use different features at different times, and pass the same image through different functions during training. This may improve generalization in ways. 1) By

---

Under review as a conference paper at ICLR 2022
encouraging the model to learn the same image in different ways, and learn different commonalities between images, comparable to model ensembling.
In particular, at time t, image has access to subset of features from image functions fj. and this subset changes over time. Hence, ft may not be too correlated with the end-to-ere function f", k S I on the training set, allowing for generalization.
2) By improving algorithmic stability, as for particular feature, the model not always reliant on the same set of images, so the removal of an image may not adversely affect the loss.
PRIOR WORK
Keskar al. (2099) investigate the effect of sharpness and flatness of minima on generalization, and find that flat minima generalize better. Dinh et al. (2017) 7) find that if the network is re- parametrized, sharp minima can also generalize well. [Smith & Le 2018)) give bayesian perspective on gener- alization and SGD. Dziugaite & Roy (2017) reveal a way compute non-vacuous generalization bounds using PAC-Bayes He Sul (2019) study the local elasticity of neural networks, where the prediction is not significantly perturbed if gradient update is performed another dissimilar point x'. Deng et al. (2021) use local elasticity to compute generalization bound Poggio et al. (220a)) show that neural networks can avoid the curse of dimensionality. Poggio al. (2020b) show that neural networks have complexity control by analyzing the normalized gradient. Veit et al. (2166) find that ResNets behave like ensembles of shallow networks by examining the paths the data travels through.
Algorithmic stability Bousquet & Elisseefi (2000) use McDiarmid's method to show that algorithmic stability implies good generalization error. Hardt et al. (016) analyze the algorithmic stability of SGD.
Studies on the gradient Sankararaman et al. (2220) study the impact of gradient confusion on the speed of neural network optimization concluding that SGD is fast when gradient confusion
is low. YYnn et al. (2018) Jain et al. (2018) study the effect of gradient diversity the speed of optimization and show that greater gradient diversity leads to large mini-batches more effectively speeding up SGD. Fort et al. (2019) investigate the stiffiness of neural networks and measure the cosine similarity between gradients as an approximation of how much one data point reduces loss on another. Chatterjee (2019) connects gradient averaging algorithmic stability and generalization. Jacot et al. (2018) study generalization in the case that the network width reaches infinity, in which case dynamics are governed by the Neural Tangent Kernel. Arora et al. (201)) find that in practice, finite width CNTKs generalize better than infinite width models, indicating there may be some advantage to finite width. Novak al.] (2018) empirically study sensitivity measures such as the input output jacobian, and find that this correlates with generalization. Arpit et al. (2017) study memorization in neural networks. One quantity they consider is the gradient of the loss with respect to particular input sample.
Variance reduction A key idea in traditional machine learning is that of variance reduction. Boosting and bagging are employed as techniques to reduce variance by averaging over different (uncorrelated) models.
SGD as noisy GD Mandt et al. 2016 show that SGD can be interpreted an Ornstein- Uhlenbeck process. Another line of work has investigated the anisotropic noise of SGD as compared GD. Since the SGD update samples a minibatch to calculate the gradient instead of using the whole training set, they view the SGD update as noisy version of the GD update. Zhu et al. (2019) explore the effectiveness of the anisotropic noise in SGD at escaping bad minima of the loss function.
Assessing Layerwise Convergence Raghu al. 2017) Morcos et al. 2018) use canonical correlation analysis to reveal insights into the convergence across different neural networks. They consider each neuron to be vector, with each entry corresponding to the neuron value a training data point, and each layer to be a subspace spanned by its neurons' vectors.
2

---

Under review as a conference paper at ICLR 2022
investigated activation patterns in ReLU networks and found that they use surprisingly few such patterns. [ooswell et al. 2016) im- proved generalization of neural networks by introducing regularizer that de-correlated activations.
Assessing Activation Patterns Hanin & Rolnick
(0019)
Dropout Neural networks may be explicitly regularized using techniques such dropout. Dropout imposes regularization by dropping nodes in way that is typically sampled from Bernoulli distribution. The resulting model can be considered an ensemble. [ao et al. (2019) investigate the effects of separating the forward from the backward pass.
3
ON FUNCTIONS
Mathematically, a function is binary relation between two sets that associates each element of the first set with exactly one of the other set. ReLU functions clearly satisfy this definition, and all images pass through the same ReLU function. Yet, we will argue that each image is more accurately considered to have its own function, because the overall ReLU function is not particularly smooth, and does not, for example, satisfy the nice properties of polynomials use in function analysis. Al- though each image function is caleulated the same weights, find that the varying ReLU paths make them sufficiently different.
GENERALIZATION
At the start of training. both the training data and test data are assumed be drawn from some distribution D. However, during training, input data are sampled uniformly from T, the training set, call this distribution U. The key question behind generalization is why optimizing over Es results in low loss over Ep. In this work, we show that neural networks learn different functions for the same image during training, and that this function diversity may contribute to generalization.
FULLY CONNECTED NETWORK GRADIENTS
We use [N] = (1,2,... N). SGD computes the following update on neural network parameters
W++1 = W. - V/ C
(I)
where V V. L. denotes the gradient of the loss with respect to the weight parameters, and the learn- ing rate. We focus on the case of minibatch SGD, where the gradient update is computed over a batch of data. For particular node evaluated on an image 2, it will have incoming backpropa- gated gradient at time of 2, [f.m) Let N be the number of nodes in the network and L the number of layers. We assume each node has index, in [N]. and furthermore for the sake of notation that all nodes in the first layer are indexed {1....1). that the second layer is indexed {f,, (2) etc. with é N. We will also write {f.1,...,fu { J to denote the coordinates of the input. Let a (((,)))) be the vector of activations for image 2, time from nodes layers j to k. We define a function Pf : [N] + {0, 1} that takes the index of the neural network's s N nodes, and returns if it is on image at time t according to the ReLU activation function and returns otherwise. With some abuse of notation, if we omit the function argument and write př. we define this as the vector of length N containing (n) as each entry. Let B, be the minibatch sampled time f. So the gradient update for particular node in layer Ex, k E [L] at time t is
pt(n) 21 Σ (f.m)
2
(2)
Be
for the weight parameter and EieB, (l.n) for the bias parameter.
Let
((.m)) (x) (f.m) p((n) a. (t,fa (1 x + (f.m)
(3)
Then the overall function fully connected node n computes at time 1 for input x is
I Σ
f(.n)
(2(
(4)
ieB.

---

Under review as a conference paper at ICLR 2022
where (n) denotes the path function for image (not necessarily in the training set), and the layer activations are given by
f( (z) = (x), f(122) f (())))(())
(5)
where the square braces denote concatenation and the layer function for lg may be defined recur- sively.
(x) f(t 145) ((..)(t)))))))))))
(6)
We will define the history of the neural network as
Hr
(LIND)
I
(TTNND))
(7)
IEB
iEBT
The activations (t.)) for an input image 2; are found by projecting the image onto the history of the network, e.g.
a (1.1.11)) ((..1.61))..)
(8)
We will use the notation
(9)
to project the path yf onto the history, i.e. to run the forward pass for F(t,fz) fixing (n) in Equation to be equal to pt for all n. Once enough updates have occurred, Equation 4 contains term f((.m) for all i in the training set. It is therefore not obvious why continuing to update using only the in the training set does not yield function that is very correlated on the training set and only produces low values of the loss on those points (and not on the test set). The gradient of neural networks sometimes treated as a black-box in terms of generalization. However, we can exploit the fact that many neural networks use some form of dot product for each neuron to write the a, terms in Equation i Experimentally, we will measure the normalized dot products of these activations, as well as path overlap.
Evaluating the same point using different functions Let be the current time, 34 E B, and 3 & B₁. At time i, data point x; is evaluated using Equation 6 which contains sf(..)) as in Equation One main observation is that ff(..))) and f (tt-k,a) for 1 < k S t- may have different (n) terms in Equation 4 (i.e. 2% (n) p(=)(n) may not have high normalized dot product). That is say, the same image at different times in training may follow a different evaluation path, i.c. the set of nodes activated for that image may be different.)
This also means that although each f((t.)) may be correlated with the history He of the network, and hence be correlated with the other previous l(t=k,x) for 1 < < L- 1, the ReLU activation causes Pj to be new enough that the network may not already correlated with f(t,,)) the current time i (until I, next updates.) This encourages generalization, as the network, despite having seen Ty before, does not pass through a function too correlated with Øj. This means that although the network has seen image x, before, it learns it using a different function, encouraging generalization. In summary, we show experimentally that Neural networks learn differing functions for the same image during training
Earlier functions are destroyed by subsequent ones In an ensemble, typically an average of func- tions taken to be the final predictor of the form wiy; In the setup described above, although image I, may follow pf at time t, it may subsequently not be assigned that same path again. Hence, it can- not use all of f in future rounds. We believe this also means that having extremely low correlation or independence is not important as it is in ensembles. Rather, there is tradeoff, more simi- larity may mean easier optimization, while somewhat less similarity may lead to more algorithmic stability, due to not re-using the same set of features.
Which functions are optimized We use He to be the function found projecting the path pf onto the history He. Each data point I; at time will have loss ((p) where L is some loss
4

---

Under review as a conference paper at ICLR 2022
function (e.g. the cross entropy loss). Suppose I; is in the updating batch time t, i.. i e Bi, then V)(p)(H)) is directly calculated, and a gradient step is taken to descend on this loss function. Furthermore, since the function pHIH, depends on all previous images z, after the first epoch in training, it is not obvious why while descending on [ip)) the functions C(p) (H4) are not also optimized for images žj in the training set only. We hypothesize that even if the network tries to optimize (H,) in terms of prior Pj [Hg for k < L, this does not correspond to optimizing æj in terms of its current function (H4
From the last time Tj updated, the network cannot re-run the forward pass on Tj, and p will be reassigned according to the ReLU function and the current history H.. So the i E B, do not have access to the end to end back-propagated gradient along the updated pf and instead calculate gradients along their own paths pf for E B₁. Each of these p have some intersection with the previous pý for & < t. typically p p) > O. These paths pf serve to collectively update along P/. but not by simple averaging or by linear combination of the whole paths. The sum of fa Equation 4 do not correspond a single back-propagated function, and are instead separate S stitched together according to the ReLU gate. When re-assigning 2j to its
functions (Equation
new path, the 2; ((..)) in Equation may not correspond the evaluation path ahead in the network that 2y will actually follow (because Př is unknown). The function zj follows at evaluation time is hence a sort of 'rranken-function which is constructed on the fly and is not explicitly optimized. It also does not correspond to simply adding together the fi. as the Pa term in Equation 4 must be applied. Additionally, the previous path that F was following
likely to be destroyed; we find experimentally that the image does not follow pf at a later time t. Also, either all gradient updates to node are applied (if it is on) or none are (if it is off), data points may not selectively pick some functions to follow end-to-end.
Additionally, we find that training images 23 and ny intersect in different places (along different paths) at different times in training, and that x; has highest dot product with different images throughout training. We interpret this to mean that The ReLU gate encourages the use of different features at different times.
We show experimentally that:
For a particular previous f.t k < t, each current updating ff have only partial path intersee- tion with f (They do not have complete access to all previous activations). Furthermore, they intersect on different nodes at different times. (Indicating they may learn from each other in different ways at different times)
The S at different times I. for some fixed image index are different. (Self similarity is not perfect.)
The overall layer function, Equation 6 is not too smooth, supporting the case for consider- ing ft for various as different functions.
For a particular image i, the ranking of which functions f are most path similar, for various times I, changes, indicating that which images are most similar to each other changes. This is contrast, e.g., with K nearest neighbors, where the identity of the nearest neighbor remains fixed for fixed training data.
CNN layers the update for CNN networks may be computed comparably, with Equation modified to contain a sum over patches ej of the original image
(l.m) (x) = Σ (L.m) (n) -7 2the 1) 1) x Σ (l,m)
2g
(10)
5

---

Under review as a conference paper at ICLR 2022
6 EXPERIMENTS BETWEEN DATA POINTS
Equation 1 involves three terms; 2 term, a, term (the activations of the layer before), and path term pt. In our experiments, we first focus on the pt term, as reveals which nodes are on off for partieular function. We use the usual definition of p; and allow the ReLU function the current history M4 to determine if node is on or off. Since the a, term in Equation with æ, which would be the activations of input x at the layer before, we also graph dot products between activations at the same layer. We concatenate all activations into a single vector for a layer. We also conduct experiments on the whole function f applied in isolation in Appendix Section C Experimental details for training the model can be found in Appendix Section
I is dotted
6.1 ALEXNET CROSS PATH ALIGNMENT
We let C denote the elementwise product. We omit the argument for pt and assume that it is (la-1 fx) We measure the quantity ([or)) [[]]or)]l((]] o.) o which determines the overlap between the intersection of paths p and p at the current epoch versus the previous epoch swers the question, does image x; always have access to the same set of activations from xr, We see in Figure ! that the answer is no. We hypothesize that this is the ReLU gate and SGD optimization encourages 2; use different set of previous features from other images Tj. and this this diversity could encourage generalization.
This an-
Ligw
Layar
Nomatosd of opti ge oo
(ptop)) op
Figure 1: Cross path similarity
for AlexNet trained on CIFAR-10 data in the
Kotor 01 on - 1
(atea) (a- (a
left column and the quantity (affa (a 'aa "
in the right column. The top row depicts the 4th
layer of the network and the bottom row the 1rst layer of the network. We find that for majority of training, there is not perfect correlation for either quantity, indicating images intersect each other on different features at different points in training.
6.2 RESNET 18 ON CIFAR 10 CROSS PATH ALGGNEENT
For ResNet-18 trained on the CIFAR 10 dataset, we plot the normalized path overlap as we did for AlexNet. We plot our results in Figure 2 We find that in general, the correlations are higher than those found in AlexNet, but still not perfectly correlated until late in training. ResNet is also significantly deeper than AlexNet, and small difference in each layer cumulatively leads to different overall path. VGG-19 on CIFAR 10 cross path similarity We have similar findings for VGG-19 in Appendix Figure 16

---

Under review as a conference paper at ICLR 2022
Loger
Layer
Normatzed oi on homalas oe a
(r ont-
Figure 2: Cross path similarity
for ResNet-18 trained on CIFAR-10 data. Bot-
(rtor) llt op, [>]
tom row depicts layer 2, top row depicts layer 6.
Lape22
Go 'Oo krtop)il 1)
Figure Cross path similarity
for ResNet-18 trained ImageNet data.
6.3 RESNET 18 ON IMAGENET CROSS PATH ALIGNMENT
For ResNet trained the ImageNet dataset, we plot the normalized distribution of (pfOr))-(P (1lr, J We plot results in Figure 3 We find that this distribution is not perfectly correlated, though becomes more correlated later in training.
6.4 ALEXNET CIFAR 10 PATH DISTANCES
We run experiments to asses how different the S are for different i. We try to asses whether the overlap between path vectors related to the distance between original points in data space. We plot results in Figure g We find that there is not clean linear relationship between distance in original space and overlap in path vectors, supporting the case for considering each image update its own function, and again indicating that there kind of function diversity present in ReLU network training.
Resnet-18 CIFAR-10: We have similar findings for Resnet -18 trained CIFAR-10 in Ap- pendix D
7 EXPERIMENTS ON SELF SIMILARITY
In this section, we seek to asses how different the paths and representations are for a single image throughout the course of training.

---

Under review as a conference paper at ICLR 2022
Figure 4: Path overlap p pý versus distance in original space ||,; xyyle for AlexNet trained on CIFAR 10. Layer shown on the left and Layer 2 shown on the right. We find that there is not clean linear relationship between distance in original space and overlap in path vectors, supporting the case for considering each image update as its own function.
7.1 ALEXNET CIFAR 10
Layer
Noralzzd of
Mornaltzdd of
Figure 5: Self similarity lp! |1I|!T| for AlexNet trained on CIFAR 10 data. Bottom row depicts
layer I, top row depicts layer 4.
For AlexNet trained on the CIFAR 10 dataset, we plot the normalized distribution of (f &x]) p. 2f (f6-11 fx]). We plot our results Figure 5 We find that especially in the beginning of training, of and pt are not perfectly correlated, indicating that the same image &; passing through different nodes different times f.
THE IDENTITY OF THE MOST SIMILAR IMAGE CHANGES
For three images we rank 128 other data points and restrict the resulting plot to the nearest neigh- bors, where rank of means it is the nearest neighbor. We use AlexNet on CIFAR 10 layer 2. We plot our results in Figure Result using activation similarity We present analogous results to above using normalized activation similarity for layer 4 and 2 of AlexNet in Appendix B
TAKEWAY FROM EXPERIMENTS
We find that in all the architectures we examined, the self similarity and the intersection similarity between data points changed throughout (especially the early part) training. These variations may lead to generalization for the similar reasons to ensembling over different models. Unlike in ensembling, we see that f is destroyed at later time f. because v; may not pass through the whole of and instead may only part of it (the part that intersects with pf +.+)). Furthermore, each
8

---

Under review as a conference paper at ICLR 2022
Easo Image
Image
Epeih
Figure 6: Plotting closeness to baseline image 2y for j 10, 15. Closeness defined as path overlap p pt for other image i. AlexNet trained on CIFAR 10 data. We find that images 2, can have different nearest neighbors at different times during training.
other image function fj only correlated with parts of f (on certain nodes.) This may explain why averaging is not necessary for variance reduction.
Mode: 20-
Aluntist lapar CP120
Abaztied layer CF12
Reshae-112 2ayer CIFD
Reshoe 110 layer CIF10
Restoe-18 loyer 2, Imagelet
VEG 19 2, CF10
Figure 7: Early to mid cross path overlap ratio versus generalization.
Ratio early to mid cross path overlap Finally, we can utilize the idea of cross path overlap in an attempt to predict generalization error. We hypothesize that higher (but not perfect) cross path overlap early is good, as typically the learning rate is high and this reflects a large number of shared features in the data On the other hand, high cross path overlap after decay may be bad, as diverse features are no longer being learnt. We plot (averaged over training data) the ratio of early (average over epochs before first decay) mid (averaged over second stage of training) cross path overlap for various models against their generalization error in Figure 7 There is roughly linear relationship. and improving such predictions could be interesting avenue for future work.
CONCLUSION
have studied neural networks from the perspective of considering each image, updating at particular time, as different function. We have highlighted some properties of practical neural networks, namely that they learn varying representations for single image, and that each image intersects each other image in different places during training. We hypothesize that this diversity of representation could lead to better generalization, by encouraging the model learn the same image in different ways (akin model ensembling) and by improving algorithmic stability.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 8141-8150, 2019.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. closer look memorization in deep networks. In International Conference on Machine Learning, pp- 233-242. PMLR, 2017.
Mikhail Belkin, Daniel J Hsu, and Partha Mitra. Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. Advances Neural Information Processing Systems, 11:2300-231,, 2018.
Olivier Bousquet and André Elisseeff. Algorithmic stability and generalization performance. In Proceedings of the 13th International Conference on Neural Information Processing Systems, pp- 178-184, 2000.
Leo Breiman. Bagging predictors. Machine learning, 24(2):123-144 1996.
Satrajit Chatterjee. Coherent gradients: An approach to understanding generalization gradient descent-based optimization. In International Conference on Learning Representations, 2019.
Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra. Reducing over-
fitting in deep networks by decorrelating representations. ICLR, 2016.
Zhun Deng, Hangfeng He, and Weijie Su. Toward better generalization bounds with locally elastic stability. In International Conference on Machine Learning, pp. 2590-2600. PMLR, 2021.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference Machine Learning, pp. 1019-1028. PMLR, 2017.
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv: 1703. 1008, 18, 2017.
Stanislav Fort, Pawet Krzysztof Nowak, Stanislaw Jastrzebski, and Srini Narayanan. Stiffness: A new perspective generalization in neural networks. arXiv e-prints, pp. 1rXi
Hongchang Gao, Jian Pei, and Heng Huang. Demystifying dropout. In International Conference on Machine Learning, pp. 2112-2121. PMLR, 2019.
Boris Hanin and David Rolnick. Deep relu networks have surprisingly few activation patterns. NeuRIPS, 2019.
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning, pp. 1225-1234. PMLR, 2016.
Hangfeng He and Weijie Su. The local elasticity of neural networks. In International Conference on Learning Representations, 2019.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: convergence and gen- eralization in neural networks. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pp. 8580-8589, 2018.
Prateek Jain, Sham Kakade, Rahul Kidambi, Praneeth Netrapalli, and Aaron Sidford. Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification. Journal of Machine Learning Research, 18, 2018.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large- batch training for deep learning: Generalization gap and sharp minima. Journal of Machine Learning Research, 18, 2019.
10

---

Under review as a conference paper at ICLR 2022
Iija Kuzborskij and Christoph Lampert. Data-dependent stability of stochastic gradient descent. In International Conference on Machine Learning, pp. 2815-2824. PMLR, 2018.
Stephan Mandt, Matthew Hoffman, and David Blei. A variational analysis of stochastic gradient
algorithms. In International conference on machine learning, pp. 354-363. PMLR, 2016.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. NeuRIPS, 2018.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of weer-parmmetrization in generalization of neural networks. Interna- fional Conference on Learning Representations (ICLR), 2019.
Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Sensitivity and generalization in neural networks: an empirical study. In International Conference on Learning Representations, 2018.
Matthew Olson, Abraham Wyner, and Richard Berk. Modern neural networks generalize on small data sets. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 3623-3632, 2018.
Tomaso Poggio, Andrzej Banburski, and Qianli Liao. Theoretical issues in deep networks. Proceed- ings of the National Academy of Sciences, (1148))30039-30443 2020a.
Tomaso Poggio, Qianli Liao, and Andrzej Banburski. Complexity control by gradient descent in deep networks. Nature communications, 11(1):1-5, 2020b.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. NeuRIPS, 2017.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Confer- ence on Machine Learning, pp. 5301-5310. PMLR, 2019.
Karthik Abinav Sankararaman. Soham De. Zheng Xu, W Ronny Huang, and Tom Goldstein. The impact of neural network overparameterization on gradient confusion and stochastic gradient de- scent. In International Conference on Machine Learning, pp. 8479. PMLR, 2020.
Robert E Schapire. The boosting approach to machine learning: An overview. Nonlinear estimation and classification, pp. 149-171, 2003.
Samuel L Smith and Quoe V Le. A bayesian perspective on generalization and stochastic gradient descent. Journal of Machine Learning Research, 18, 2018
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199) 1958, 2014.
Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems, 29:550-558, 2016.
Dong Yin, Ashwin Pananjady, Max Lam, Dimitris Papailiopoulos, Kannan Ramchandran, and Peter Bartlett. Gradient diversity: key ingredient for scalable distributed learning. In International Conference on Artificial Intelligence and Statistics, pp. 1998-2007. PMLR, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt. Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
Zhanxing Zhu, Jingfeng Wu. Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochas- tic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In International Conference Machine Learning, pp. 7664-7663. PMLR, 2019.


Reviews:
[{"review_id": "rJZlNSIHag", "reviewer": "Reviewer_ypTe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The primary strength of the paper is novelty in that it could be a contribution of explaining why neural networks generalize despite violating traditional ML theory and rules of thumb.  The idea of \"image functions\" seems different from other approaches to explaining generalization.  Also, although the results are presented for the specific case of classification on images with ReLU networks trained with SGD, it seems that the ideas could be more general in future follow-up work.\n\nHowever, there are some weaknesses with the paper that point to more work and clarity before publication.  The main application shown is in Figure 7 where a candidate statistics namely ratio between early and mid cross path is correlated with generalization performance.  But there are only a few points for comparison which makes it difficult to be confident in the claim that there is a good correlation between the statistic and generalization.  For example both VGG-19 and ResNet-110 perform about the same for CIFAR10, but have significantly different statistics.  For the other data sets such as CIFAR100 there are only two points to establish the correlation.  In general, it would be better to separate the experiments by data set and have more comprehensive results such as showing more statistics in other layers, etc.\n\nAlso, the results show specific statistics for specific layers, but how was the layer chosen?  Do all layers show the same trend or is there a criterion to choose which layer matters?\n\nAnother thing that would help the reader would be to relate the proposed approach to the previous approaches.  Although the contribution is mostly experimental, still it would be useful to know whether the implication is variance reduction such as in ensemble methods or is it some other theoretical idea that the results point to.  Adding some discussion on this in the prior work section and/or elsewhere would help connect the ideas in the paper to what is known in the literature.  Various parts of the paper hint at algorithmic stability, but it is difficult for the reader to see the strong connection here.\n\nFinally there are some minor typos, etc. that can be improved.  The claims made right before Section 2 is somewhat difficult for the reader to interpret at that point as the notation hasn't been explained yet before being used.  Similarly Section 3 is also a bit cryptic before reading the later sections which describe the actual details.  The description of the details in Section 5 may benefit from a diagram and also there may be typos involving the specification of the node/layer indices such as l_1 being repeated for both the last node in layer 1 and the first node in layer 2.  Overall, the explanation of what exactly is an image function and the definition of F could be clarified by perhaps showing more concrete examples, exactly because this idea is somewhat novel.  Figure 4 and 5 captions also seem to have typos for which layer is depicted.", "summary_of_the_paper": "This paper proposes a different look at why neural networks generalize despite optimizing to zero training error, over-parameterization, etc.  The contribution is mostly experimental in the sense of computing various statistics of a model during training and correlating those statistics with generalization performance.  The core idea is to define \"image functions\" which are determined by the training image and the current training iteration.  Correlation statistics on these functions for different and same training images show patterns in the training dynamics, which may indicate generalization for example as training continues towards zero error the statistics still vary.\n\nThe paper focuses on classification with CIFAR10 and ImageNet with AlexNet, VGG, and ResNet models.  In particular the results are specialized for ReLU networks training with SGD.", "summary_of_the_review": "Overall, the paper tackles an important problem, namely explaining generalization via an experimental study and using a novel idea of \"image functions\".  The results indicate promise, however the fact that the experiments do not provide a comprehensive view such as showing the statistics for all or many layers and having more points to establish the correlation in Figure 7 make the paper a little premature.  Also the description of the details is somewhat unclear, and more importantly there isn't a good discussion of how this approach relates to previous work.  It is claimed that the evidence shows algorithmic stability but this isn't justified directly, and the reader also isn't clear whether the results show variance reduction or some other idea that leads to generalization.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ir23inGAm8", "reviewer": "Reviewer_ec11", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an entirely empirical work. The authors look into the activations within a neural network (as either a path or a set of real-valued activations) and examine how such activations change over time during training for the same pair of images. What the authors basically show is that early during training, the activations change over time before they settle when the learning rate is sufficiently small. I do not think this is an insightful finding because this is what would be expected by \"training\" the network: we expect the weights and bias terms to change during training (hence activations would change) until the network is fully trained. \n\nThe authors' arguments about how such findings help understand generalization are heuristic. The authors claim that because the activations change over time, the network learns to use different features for the same image, and that improves generalization! Unfortunately, most of the claims in the paper are of this nature. For example, the authors state that \"less similarity may lead to more algorithmic stability, due to not re-using the same set of features\", where similarity here is the similarity of the path of activations. However, if we consider the simplest possible neural network, which is a logistic regression classifier, similarity will be high throughout training and it is hard to argue that logistic regression is less stable than deep neural networks (e.g. in the sense defined by  Bousquet & Elisseeff 2002). Unfortunately, such statements are the main contributions of the paper and they are neither justified nor precisely stated. \n\nThe paper also needs a lot of improvement in its presentation. Some issues include:\n- The axis labels in the figures are too small to read. \n- The line spacing in some paragraphs is different from the reset of the paper (e.g. Paragraph 2 in Page 1 and the paragraphs in Page 5). \n- The related works section lists papers related to deep learning but I don't see the connection to the present work. In addition, there are lots of research papers that study activations/representations in the neural network that are not cited, which would be more relevant to the present paper. \n- In Equation 4, the same symbol t is used both inside and outside the summation. \n\n\nThe paper contains many typos. Examples:\n- Page 1: \"PAC Bayes based bounds Dziugaite & Roy (2017) approaches exploring\" --> \"PAC Bayes based bounds Dziugaite & Roy (2017) and approaches exploring\"\n- Page 2: \"using PAC-Bayes He & Su (2019)\" --> \"using PAC-Bayes. He & Su (2019)\"\n- Page 2: \"jacobian\" --> \"Jacobian\"\n- Page 4: \"the network may not already correlated\" --> \"the network may not already be correlated\"\n- Page 4: \"that Neural networks\" --> \"that neural networks\"\n- Page 4: \"learn differing functions\" --> \"learn different functions\"\n- Page 9: \"features in the data On the other hand\" --> \"features in the data. On the other hand\"\n", "summary_of_the_paper": "The authors present empirical results about the correlation between the activations in a neural network across time for a fixed pair of images.  They show that for the same pair of images xi and xj, the activations change over time until they settle when the learning rate is sufficiently small. The authors claim that this provides insight into why neural networks generalize, by arguing that the network uses different features at different epochs during training.  ", "summary_of_the_review": "The empirical findings reported in the paper are not insightful and the paper makes claims that are heuristic and not justified. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F8W7ypYUYIH", "reviewer": "Reviewer_fEgL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Weaknesses :\n- Introduction is a bit disorganised, feels a bit too much like a list of independent concepts, and it is hard to extract the relation with the contribution of this paper\n- The paper is poorly written, poorly structured, and the results are extremely difficult to parse, in great part due to imprecise language. Too many sentences are vague and need to be clarified, with words such as “image” used to mean very different things. A few examples : “indicating images intersect each other on different features at different points in training” ; “the ReLU gate and SGD optimization encourages xi to use a different set of previous features from other images xj” ; “supporting the case for considering each image update as its own function, and again indicating that there is a kind of function diversity present in ReLU network training” ; “the same image xi is passing through different nodes at different times t”\n- Due to this lack of rigour, I struggled to understand the main message of the paper. This is not clear either from the experimental section : the fact that activations / paths become more and more correlated from one epoch to the next through training is not surprising,\n\nComments : \n- “The key question behind generalization is why optimizing over E_U results in low loss over E_D” : I disagree with this. In typical studies of generalisation, the train and test data come from the same distribution (when they do not, the problem under study is out-of-distribution generalisation). The question is why optimisation over the empirical distribution (at finite sample size) leads to good generalisation over the full distribution.\n- Fig. 4 : the overlaps need to be normalised on both axes for this plot to be meaningful\n- Fig. 7. : The correlation which appears can hardly be named a “roughly linear relationship”\n\nNits : \n- “Mathematically, a function is a binary relation between two sets that associates each element of the first set with exactly one of the other set” : not all functions are one-to-one mappings…\n- “the second layer is indexed {l_1, ..., l_2}” : shouldn’t it start at l_{1+1} ?\n- “We will also write {l_{-1}, ..., l_0} to denote the coordinates of the input” : is l_{-1} negative ? Shouldn’t l_0 =1 ?\n- Eq. 2 : second dot product should be a simple multiplication, no ?", "summary_of_the_paper": "This paper analyses how the representations of neural networks evolve during training, in particular which ReLU nodes are activated.\nThe authors attempt to show how this leads to a diverse set of functions being learnt.", "summary_of_the_review": "Overall, I found this paper extremely vague and confusing, and struggled to follow the message the authors try to convey. \nIn the current state, I do not think this paper reaches the standards of a venue such as ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Aj1mXxxPmkE", "reviewer": "Reviewer_M7Dz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper analyze the dynamic of “image functions” and claims that analyzing the dynamic of image functions could potentially explain the generalization. I have a few questions and concerns.\n\n1. The major claim of the paper is that neural networks generalize since they learn different functions for the same image during training (page 4). This statement is non-trivial and I cannot see direct link between generalization and the dynamics. When the paper talks about generalization, does it mean the generalization error is small? If this is the case, the paper requires some formal mathematics to bridge the gap between the dynamics of image functions and generalization error.\n\n2. Why call it image functions? It seems that the framework is applicable for any fully-connected networks. The experiment section does not explain this neither. The only thing related to images is CIFAR10 and ImageNet. Based on the naming, I would expect to see more connection to low-level vision function or filters.\n\n3. Another major finding of the paper is that the function for images would be different in different time step. This statement seems to be expected as the network parameter is keep updating via SGD. In particular, the function represented by neural networks is expected to change over time, it would be great if the authors can provide more clarification about this.\n\n4. I found the notation is a bit hard to follow. Section 5 introduces various notions such as path, history, image projection, it would be better to provide a high level idea at the beginning of the section. For me, it seems that section 5 is writing the explicit gradient formulation for fully connected networks.\n\n5. As the major claim is about generalization, it is better to theoretically or empirically provide some evidence related to generalization error. Although figure 7 provide some hints, but it is still elusive why this improves the generalization.\n\n6. The abstract states two way to improve generalization. However, it seems that the claim is not explicitly examined in the experiment section. For instance, how to concretely encourage the model to learn the same image in different ways?\n\nOverall, the paper addresses an important problem. The idea could be potentially interesting, but I found the writing and presentation is a bit confusing.", "summary_of_the_paper": "The paper analyze the dynamic of “image functions” and show that analyzing the dynamic of image functions can explain the generalization. In particular, the authors visualize the cross path similarity and overlap to provide potential insights to generalization of neural networks. ", "summary_of_the_review": "The idea could be potentially interesting, but I found the writing and presentation is a bit confusing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In an attempt to understand generalization, this paper aims at understanding the dynamics of functions presented by the network for different images in the training set. Authors look at activation patterns (whether a ReLU activation is on or off) as a way of characterizing the active paths in the network and approximating the function presented by the network for each image. Authors study different related statics (eg. correlation) and how they evolve during training including.

Pros: 
- Understanding the dynamics of training, how diversity is encouraged by the training procedure and its relationship to generalization is an important problem.
- This paper takes an empirical approach and tries to make interesting empirical observations about the dynamics of the training.

Cons:
- The paper is poorly written in terms of structure, making clear arguments with enough evidence, notation, etc.
- Some empirical trends are shown but their connections to the main claim of the paper about generalization is very weak. The main attempt to connect the observations to generalization is Fig. 7 which shows model accuracy correlated with the ratio of early to mid overlap. This is problematic both because it only has 6 data points and also because a simple correlation analysis is not enough to establish this claim which is more about the cause of generalization.

Reviewers have pointed to various concerns including but not limited to clarity of the paper, lack of rigorous arguments, not providing enough evidence for the arguments, etc. Unfortunately, authors did not participate in the discussion period.

Given the above concerns, I recommend rejecting the paper.
Decision: Reject