Representation Alignment in Neural Networks
Authors: Ehsan Imani
Wei Hu
Martha White
Venue: Accepted by TMLR ()

Abstract: It is now a standard for neural network representations to be trained on large, publicly available datasets, and used for new problems. The reasons for why neural network representations have been so successful for transfer, however, are still not fully understood. In this paper we show that, after training, neural network representations align their top singular vectors to the targets. We investigate this representation alignment phenomenon in a variety of neural network architectures and find that (a) alignment emerges across a variety of different architectures and optimizers, with more alignment arising from depth (b) alignment increases for layers closer to the output and (c) existing high-performance deep CNNs exhibit high levels of alignment. We then highlight why alignment between the top singular vectors and the targets can speed up learning and show in a classic synthetic transfer problem that representation alignment correlates with positive and negative transfer to similar and dissimilar tasks.

Published in Transactions on Machine Learning Research (09/2022)
Representation Alignment in Neural Networks
Ehsan Imani
imani@hallertta.co
University of Alberta
Wei Hu University of Michigan
vol.Oumich. edu
Martha White
whitemm@ulleerta.ccmm
University of Alberta
CIFAR AI Choir
Reviewed on OpenReview: https: // openreview. form? id- FLIWMnZ991
Abstract
is now a standard for neural network representations to be trained on large, publicly available datasels, and used for new problems. The reasons for why neural network represen- tations have been so successful for transfer, however, are still not fully understood. In this paper we show that, after training, neural network representations align their top singular vectors to the targets. We investigate this representation alignment phenomenon in variety of neural network architectures and find that (a) alignment. emerges across variety of different architectures and optimizers, with more aligument arising from depth (b) alignment increases for layers closer to the output and (c) cxisting high-performance deep CNNs cxhibit high levels of aligument. We then highlight why alignment between the top singular vectors and the targets can speed up lcarning and show in classic synthetic transfer problem that representation alignment correlates with positive and negative transfer to similar and dissimilar tasks. A demo is available at https / http//ggtttt/ttteee com www./essaan/reeeaaleeeeeeeooeeee
1 Introduction
A common strategy for transfer learning is first learn a neural network on source (upstream) task with large amount of data, then extract features from an intermediate layer of that network and finally train subsequent model on related target (downstream) task using those extracted features. The premise is that neural networks adapt their intermediate representations- hidden representations to the source task and, due to the commonalities between the two tasks, these learned representations help training target task (Bengio et al., 2013). Availability of large datasets like ImageNet. (Russakovsky et al., 2015) and the News Dataset for Word2Vec (Mikolov et al., 2013) provides suitable source tasks that facilitate using neural networks for feature construction for Computer Vision and Natural Language Processing (NTP) tasks (Kornblith al., 2019; Oquab al., 2014; Devlin al., 2018; Pennington et al., 2014).
the
There is yet much more to understand about when and why transfer is successful. Understanding the properties of the learned hidden representations and their benefits for training similar tasks has remained a longstanding challenge (Touretzky & Pomerlcau, 1989; Zhou et al., 2015; Marcus, 2018). One strategy has been to define properties of a good representation aiinn, and try to either measure or enforce those properties. Disentanglement and invariance are two such properties (Bengio et al., 2013), where the idea is that disentangling the factors that explain the data and are invariant most local changes the input results in representations that generalize and transfer well. Though encoding propertics for transfer is beneficial, it remains important question exactly how to evaluate the representations that do emerge.

---

Published in Transactions on Machine Learning Research (09/2022)
One challenge is that even hidden representations of two neural networks trained identical tasks appear completely different, and studying the representations requires measures that separate recurring properties from irrelevant artifacts (Morcos et al., 2018). One direction has been to analyze what abstractions the network has learned, agnostic to exactly how it is represented. Shwartz-Ziv & Tishby (2017) studied ncural networks through the lens of information theory and found that, during training, the network preserves the information necessary for predicting the output while throwing away unnccessary information successively in its intermediate layers. Using representational similarity matrices, Hermann & Lampinen (2020) found that on synthetic datasets where task-relevance of features can be controlled, learned hidden representations suppress task-irrelevant features and enhance task-relevant features. Neyshabur et al. (2020) showed that neural networks trained from pre-trained weights stay in the same basin in the loss landscape. In reinforcement learning, Zahavy et al. (2016) explained the success of Deep Q-Networks by visualizing how the learned hidden representations break down the input space in a way that respects the temporal structure of the task. Analyses of NLP models have found linguistic information in the hidden representations after training (Belinkov et al., 2017; Shi et al., 2016; Adi et al., 2017; Qian et al., 2016).
Other works have focused individual features in the learned representations. Saliency maps and Layer- Wise Relevance Propagation (Simonyan et al., 2014; Zeiler & Fergus, 2014; Bach et al., 2015) that show the sensitivity of the prediction to each unit in the model are popular in Computer Vision and demonstrate the appearance of useful features like edge or face detectors in neural network. In NLP, Dalvi et al. (2019) studied the relevance of each unit to an external task or the model's own prediction.
In this work we look at singular value decomposition of the learned features and find that after training, neural network hidden representations align their top singular vectors to the task. We discuss this observation in detail in Section 4. Section 5 provides an empirical study using different optimizers, architectures, datasets, and other design choices to support this claim. There are a number of previous theoretical results on optimization and generalization that exploit alignment between the top singular vectors and the task (Arora et al., 2019; Oymak et al., 2019a; Cortes et al., 2012; Canatar et al., 2021). While these results were originally molivated for neural langent. kernels and other kernels, our observation shows that hidden representations of a neural network also satisfy the conditions for these results. In section 6, we discuss one of these previous results, convergence rate developed by Arora et al. (2019) and Oymak et al. (2019a), and describe how our observation makes it applicable for studying feature transfer. Finally, in Section 7 we study feature transfer first in controlled synthetic benchmark and then in pre-trained CNNs and find that positive and negative transfer can be traced to an increase or decrease in alignment between the learned representations and the target task.
2 Problem Formulation and Notation
We focus on the transfer setting. We assune there is a source task where data is abundant and a target task with a small number of available data points. A neural network is learned on the source task, and the final hidden layer used as learned features. These learned features are then given as the representation for the target task, which we learn a linear model. More generally, nonlinear function- another, likely simpler, neural network- could be learned, but we focus on linear models.
More formally, we consider a dataset with n samples and input matrix Xnxm and label vector 9ax1- We assume we have a (learned) feature function Φ : Rm > Rd that maps inputs to features, with representation matrix Bxx where n 2 d. Without loss of generality, we replace the bias unit with a constant feature in the matrix to avoid studying the unit soparately. We learn a lincar model, with weights denoted by Wax1, learned using the squared error in regression and logistic loss in classification. In regression, y E R and the model w is learned using || yll where ||-|| is the lz norm. In classification, y E [-11,+11 +13" and the logistic loss is Σ. -i= log(1 + exp(-<p w y;)) /s))))
A Alignment will be defined in terms of the singular vectors of & and label vector y. The singular value decomposition (SVD) of & is UEVT, where Σnxd is a rectangular diagonal matrix whose main diagonal consists of singular values Lnd that are in descending order and Unxn and Vaxa are orthonormal matrices whose columns Uim and Uidd are the corresponding left and right singular vectors. Implicitly, the singular values Ia are zero, giving this decomposition where E is rectangular. For vector a and ort honormall
2

---

Published in Transactions on Machine Learning Research (09/2022)
basis B, a² denotes B a or the representation of in the basis B. We will use r(-) to denote the rank of matrix.
3 Beyond Expressivity
A set of expressive representations may differ terms of optimization behavior when learning & linear function on those features. In particular, some representations may improve the convergence rate, requiring fewer updates to learn the linear function. There are also theoretical results that relate this improved convergence rate with improvement in generalization ability (Hardt et al., 2016).
Let us clarify this point with an experiment before moving to the transfer learning setting. We sampled 1000 points from the UCT CT Position Dataset (Graf et al., 2011) and created three sets of 1024-dimensional features. The first representation extracted from neural network with one ReLU hidden layer. The second one is obtained with sparse dictionary learning (Mairal et al., 2009). The third one is the representatie alinn learned by a two-layer RBF network. All of these representations are nonlinear and high dimensional and a linear model on each one can perfectly fit this data. We verified this by evaluating the closed form least squares solution on the extracted representations.
Dictionary Learning
RBF Network
Neural Network
0
20
40
60
100
iteration
Figure 1: Performance of a linear model on a ReLU neural network representation (red), RBF network representation (purplc), and dictio- nary learning representation (blue). Dashed and solid curves show train and test mean squared errors. We swept over four different step-size val- ues in {0.01, 0.1, 1, 10) and chose the best performing model.
However, as shown in Figure 1, training a zero initialized linear model with gradient descent these representations shows inter- esting differences between them. First, optimization is faster when the representation is extracted from a neural network. The second difference is in the error of the models trained on these representation when they are evaluated on a separate test set 1000 points. These differences as opposed expressivity, linear separability, or the ability to fit the data are the subject of this study.
In the next section will introduce measure called representation alignment to evaluate the quality of representation in terms of the convergence rate and generalization ability that it provides. Neural networks increase alignment in their hidden representation during the training, more so in hidden layers closer to output. We will then discuss how training with gradient descent on representations with high aligument (such as neural network representations) results in an initial fast phase that reduces most of the loss with small weights.
4 Representation Alignment
Representation aligmment, as defined in this work, is a relationship between label vector and representation matrix. Representation alignment facilitates learning the model w. In this section, we define and discuss alignment, starting first by providing intuition about the reason for the definition.
To understand alignment, consider first the squared loss, rewritten using the SVD
|| P u| | - -UT y1 ||W( - 11"
\| -1
x(0) Σ lawwy 11
Σ
1
(1)
(=1(+)+1
The vector nd is the projection of the entire label vector of n samples onto the basis U. In this rewritten form, it clear that if o, is small and the rotated y = u, y is big, then w. has to become very big order to reduce the loss in this task, which is problematic. The representation & is aligned with the label vector y if uf y large primarily for the large singular values d;.
3

---

Published in Transactions on Machine Learning Research (09/2022)
squared norm projection
Scoone nom of projection
norm projection
equared norm projection
singuer value
sinuuarr volue
singular h
sinuulrr value
(a) Original features True labels
(b) Original features
(c) Hidden representation
(d) Hidden representation
Shuffled labels
Before training
After training
Figure 2: Projection of the label vector different directions. The horizontal axis shows the index of the singular value and is zoomed in to the region of interest. The representations are normalized to length one for proper comparison. Comparing a and shows the aligument in MNIST that is broken when the labels are shuffled. For and d, the network is trained on shuffled labels, and comparing them shows emergence of alignment between the hidden representation and the training labels.
Before giving a formal definition, let us visualize these values. Figure 2 (a) we sampled 10000 points from the first two classes of the MNIST dataset (5000 points from each class) and plotted the singular values of X, the original features in the dataset, and the squared dot product between the label vector y and the corresponding left singular values Ui.. The dot product is noticeably large for the top few singular vectors, and drops once the singular values become small. This need not always be the case. We created the same plot for shuffled labels in Figure 2b. As the association between the features and the labels in MNIST dataset. is lost, the label vector is more or less uniformly aligned with all the singular vectors.
We define alignment relative to a threshold on singular values. We assume ||>,|| S for all i (1...., n), which can be satisfiod by normalization. This constraint ensures singular values are not large simply because frature magniludes large, and so facilitates comparing different representations. Given threshold 2 0, the degree of alignment between the representation & and label vector y is
Alignment(PP V.
Σ
(u, y)²
(2)
(:0.22)
If Alignmen(@e.pg (P. r) is larger for higher т, then the projected label vector is concentrated on the largest singular values.
We visualize hypothetical good and bad alignment in Figure 3, if we were to plot alignment versus the threshold. The green line depicts a hypothetical curve that has higher alignment, and the red bad alignment. The curves start at the same point because at threshold of O all singular values are greater than or equal to the threshold, the sum in equation 2 includes the sum over all terms (u, 2 As the threshold increases, alignment. quickly drops for the red curve, indicating that the projected label vector is not concentrated on the largest singular values. Rather, is spread out across many small singular values, which we later motivate results slow learning. In the green curve, even for a relatively high threshold, the sum still includes most of the magnitude, indicating (u, y)² is small for small singular values.
Good
Bad
threshold
Figure 3: A hypothetical Hignmenn--theshod curve.
When comparing aligument curves in our later experiments, we often see clear difference between the two curves across a wide range of thresholds in Figure 3. In these cases, for brevity, we simply state that the green curve has higher alignment without referencing the thresholds.
4.1 An Illustrative Example
Alignment in high-dimensional representations can be difficult visualize. Before measuring alignment neural network representations, it can be helpful to understand when alignment may be low or high. Figure 4
4

---

Published in Transactions on Machine Learning Research (09/2022)
is an example that illustrates how the relationship between the representation and the label vector affects alignment. A set of 1000 points are normalized to unit length so they fall on circle. 500 points have angles around 7/4 and the angles for the other 500 points are around -37/4. This representation matrix has two right singular vectors (principal components) i1 and 2/2 with singular values 01 = 9.58 and d2 = 2.84. The left and right plots shows two different possible sets of labels, where yellow is for class one and purple for class two.
The labeling the left defines the two classes in the direction of the first principal component, where data is more spread out. As result, the label vector is mostly in the direction of the first singular vector u1. The labeling on the right defines the classes in the direction of so the classes are closer together. The resulting label vector is mostly in the direction of u2 which has a smaller singular value. Both representations are perfectly able to linearly separate the two classes, but the alignment is higher in the leftmost plot for any threshold between d1 and 02.
22
U1
U2
U1
-3.00 -425
-108 -015
Figure 4: The labeling that separates the two classes in the directions where data is more spread out (left) is mostly in the direction of the first singular vector with singular value 9.58. Recall that the weights for the separating hyperplane are orthogonal to the plane; here ei is orthogonal to the plane that separates these two classes. The labeling on the right is mostly in the direction of the second singular vector with singular value 02 = 2.84. It has to use the direction with smaller singular value to fully separate the two classes.
4.2 Emergence of Alignment in Learned Features
Now let return to alignment in the representations learned by neural networks. In first example, in Figures 2 (a) and (b), we used the raw inputs as the features and measured their alignment to the label vector. We could ask if neural network representations improve on this alignment, but because the alignment is already very good, we are unlikely to see notable improvements. Instead, we can synthetically create a much more difficult alignment problem by shuffling the labels. We can ask if alignment can emerge using neural networks even the absence of good aligument between the input, matrix and label vector, i.e., when the labels are shuffled.
We trained a neural network with three hidden layers wide the input layer using Adam (Kingma & Ba, 2014) for 1000 epochs with batches of size 64 on the shuffled labels and extracted the representation from the final hidden layer. Figures 2 (c) and (d) show the squared projection of the training label vector on the singular vectors of the hidden representation matrix before and after training.
200
Neural Network
175-
150
alignment
125
100
RBF Network
The network has learned a representation where the label vector is mostly aligned with the top singular vectors. Recall that this is for shuffled labels. This comparison illustrates emergence of representa- tion aligument in hidden representations, even when there is no such relationship between the label and the input features.
Dictionary Leaming
36.0 125
175 20.
threshold
Figure 5: Alignment of the ReLU NN (red), RBF network (purple), and sparse coding (blue) representations.
One might wonder if this is a property of all representation learning approaches, or particular neural networks. We have already seen that sparse coding with dictionary learning does not appear to have
5

---

Published in Transactions on Machine Learning Research (09/2022)
this property, and in fact when we mcasure the alignment for Figure 1, we find it is much lower than that of the neural network. But sparse coding is an unsupervised procedure. Perhaps whenever we include a supervised component, alignment improves.
The learned RBF network allows us to assess this hypothesis, since it is supervised but has a different architecture than typical neural networks. In particular, its first. layer consists of RBF units and training adjusts the centers of these RBF units. This RBF network could similarly improve alignment, through training, and provides comparison point for whether current NN architectures are particularly able to improve alignment.
We find that alignment for the RBF network is better than sparse coding, but still notably worse than the NN. All the curves start at 200, because at a threshold of 0 all singular values are greater than or equal to the threshold, so the sum in equation 2 includes the sum over all terms (u, y)². As the threshold increases, alignment quickly drops for dictionary Icarning, indicating that the projected label vector is not concentratod on the largest singular values.
Rather, it is spread out across many small singular values, which we later motivate results in slow learning. The RBF network does not drop as quickly, but its curve is more similar to dictionary learning than the NN. The NN has notably better alignment. At threshold of about 8, the sum still includes most of the magnitude, indicating (u, y)² is small for singular values smaller than 8.
This one experiment is by no means definitive. We include it to show that different feature learning methods can have different alignment properties. This work is primarily focused on showing the emergent alignment properties of neural networks, rather than making the much stronger claim that this is unique property to neural networks. But it is motivating that, even in this simple setting, an NN has significantly better alignment; it is a phenomenon worth investigating.
5 Experiments Measuring Alignment in a Variety of Networks
We provide an empirical study of representation alignment with various architectures and optimizers for regression in the UCI CT Position Dataset and for classifying the first two classes of Cifar10 and MNIST, We then investigate the degree of alignment in internal layers of the learned networks.
Training increases alignment in different architectures and optimizers. This section shows one of our main results: neural networks, across depths, widths, activations and training approaches, siguificantly increase aligmment. We test five different settings. In the first, we fix the hidden layer width to 128, the optimizer to Adam and train networks of different depth. In the second, we set the depth to 4, the optimizer to Adam and train networks of different hidden layer width. In the third, we set the depth to 4, the hidden layer width to 128 and train networks with different optimizers. The fourth and fifth settings consider other activations (tanh, PReLU, LeakyReLU, and linear) and batch-sizes (32, 128, and 256). In all of these settings, training increases aligmment in the final hidden representation compared to both the input features and the hidden representation at initialization. The increase in aligmment occurs on both train and test data, Interstingly, the increase in alignment happens even with linear activation where the successive layers cannot. make the data more binary separable. The plots for train data across different architectures and activations are shown in Figure 6 and the rest of the plots are in Appendix A. Note that the networks in this figure do not have convolutional layers, residual connections or batch normalization. Adding depth to these architectures might have no positive effect or even harm performance (Ioffe & Szegody, 2015; He et al., 2016), which could be why we do not find a consistent increase in final alignment as depth increases.
Alignment is higher in layers closer to the output. Conventional wisdom in deep learning is that networks Iearn better representations in the layers closer to output (Goodfellow et al., 2016). Alain & Bengio (2017) trained linear classifiers on the learned hidden representations and showed that linear separability increases along the depth. Another example is in Computer Vision where the first hidden layers of a CNN usually find generic features like edge detectors, and the last hidden layer provides features that are more specialized and allow the final layer to achieve high performance (Krizhevsky et al., 2009; Yosinski al., 2014).
6

---

Published in Transactions on Machine Learning Research (09/2022)
Cifar10
MNIST
CT Position
3000
alignment
alignment
allgnment
origina
Depth
13
Shreshodd
23
threshold
alignment
algnment
alignment
origina
256
Width
128
64
300
20
Bhreshold
3reshold
Brrehhold
LeakyReLU Linear PReLU
200
802
Algimenc
250
origina
Tanh
Activation
80
threshold
threshold
Figure 6: Training increases alignment in hidden representation regardless of architecture. The orange line is the input features, the dashed lines are hidden representations before training, and the solid lines are hidden representations after training. The plots are averaged over runs and the shaded area shows standard errors. The label vector for the regression task was moved to the interval [-1, 1] for easier comparison with classification. The curves for learned representation on MNIST mostly overlap as all models achieve high level of alignment this task.
We test the hypothesis that training a multi-layer network results in hidden representations that are successively better aligned with the labels. For MNIST and CT Position, we pick 10000 random points and train neural network with three hidden layers of width 128 using Adam with batch-size 64 until convergence. Figure shows the alignment for all the hidden representations along with the original features before and after training, averaged over 5 runs.
On both datasets, layers closer the output are monotonically better aligned after training. Comparison with the plots at initialization shows that this pattern is the result of training on the task. In fact, the ordering is completely reversed before training. This is probably due to the loss of information after multiple layers of random transformation in the initialized network. We will also discuss this in later section.
6 Alignment and Convergence Rates
In this section we discuss how aligned representations help speed up learning. We first conceptually explain this phenomenon, by reviewing fine-grained convergence rate. We then provide simple experiment to highlight the connection between alignment and convergence rates.
6.1 Improving Convergence Rates
The role of alignment in optimization and generalization has been recently noted (Jacot et al., 2018; Arora al., 2019; Oymak et al., 2019a; Canatar et al., 2021) in overparameterized networks for the standard training-testinee setting. We leverage these insights for the transfer setting, where neural network- that need not be overparameterized learned to create features for transfer, to new target task. The aim
7

---

Published in Transactions on Machine Learning Research (09/2022)
2003
8030
2
alignment
alignment
2000
6039
2000
4008
309
original 96
10
68
106
11
50 Uhreshold
40
60
60
90
100
threshold
(a) MNIST
(b) CT Position
Figure 7: Aligmment of input and hidden layers before (dashed) and after (solid) training on MNIST and CT
Position. Training increases alignment more for layers closer to the output.
of this paper is to show emergence of alignment in neural network hidden representations and characterize positive and negative transfer in this framework.
understand how alignment can help with convergence rates in the optimization, let us review the following fine- fine-grained convergence rate.
Proposition 1. Given full rank representation matric & and label vector y, let w" = - be the optimal linear regression solution and Oaa be the marimum singular value for 4. The batch gradient descent updales W++1 = w2 -1p (Aw- - y) with slepsize O < TJ < oma and wp = results in
r(
pww- ow"1
Σ (1 n10 22 (u. (u.T))²
(3)
r(0)
| y1/3=
Σ
(1 n (00)) (uTy)²
y) + || y11
==1
Arora et al. (2019) showed this result for an input Gram matrix, similar to T and that the convergence rale exactly dictated by alignment. Their analysis was for infinitely wide ReLU networks. Our goal is not to measure alignment within the network itself, but rather of the representations produced by the neural network. We therefore provide the above proposition for features & given by the neural network; the proof is straightforward and given in Appendix B.
The following proposition uses the result above and relates the convergence rate to representation alignment defined in this paper and highlights the role of threshold. The use of 0.9 in the statement of this proposition is to avoid excessive notation in the main paper and the proof in Appendix B holds for generic ratio by including logarithmic factor.
Under the conditions of the previous proposition, if Aligument( 0 < S mazz then gradient descent needs at most 0(1/(172) iterations to reduce the loss by 0.98.
Proposition 2.
for threshold
Let. us examine why Proposition highlights the role of alignment in convergence rates. Notice that if (u, 2 is large for a larger singular value o,, then (1 - ? is smaller and makes the product in the sum in Equation smaller. Having smaller sum means that u, is closer to w" after iterations. On the other hand, if (u, 2 is large for a small singular value o,, then it is multiplied by larger number (1 - making this sum larger.
8

---

Published in Transactions on Machine Learning Research (09/2022)
fast process
slow process
fast process
slow process
loss
loss
loss
loss
weight
weight
weight
weight
(a) High alignment
(b) Low alignment
The fast and slow processes in a well aligned representation (left) and poorly aligned representation (right), both with rank 2. The singular values
Figure 8:
σ1 = 6 for the fast process and 02 1 for the slow process. The y-axis is the corresponding share of loss and the x-axis is the projection of weights in the corresponding direction. The orange circle shows the state of the process at zero initialization and the green circle show the state of the process al the end of the initial fast phase, i.e., when the last process has mostly reduced its loss. With the well aligned representation in (a), the fast process gets a larger portion of overall training loss, and roduces that loss (from orange to groen). With low alignment in (b), each process gets half of the overall loss, and the fast process reduces less of the loss (less change between orange to green). With high aligmment, there is little remaining error to reduce in the slow phase. With low aligmment, the optimization needs to continue through the slow phase and grow the weights.
We can also consider the optimization process, to understand why convergence is faster. Recall the decompo- sition of the squared error, in Equation 1, but rewritten slightly, and also now consider its gradient.
r())
fww- - v11
Σ
0,
w
o 11?
Ji
i=1
(+)++
r())
V f.w - y/ =
V
1/1
w
9i
Each component of the loss will be roduced by movements of the weights in the direction of right singular vector, seen in the gradient decomposition. Further, the rate of reduction depends on the singular value. We can imagine the weight w. over time as process, where a larger singular value means larger steps or changes in this process. The processes do not interact with each other since the singular vectors are orthogonal to each other and the movement of the weights in one dircction does not depend on its previous movements in other directions. A process with nonzero singular value a, needs to move the weights to v/ /a, to fully reduce its share of training loss. This required movement smaller for processes with larger singular values.
From this, can that gradient descent will exhibit two phases, as has been previously noted (Oymak et al,, 2019a). In initial fast phase, all the processes are reducing their share of the loss. Since the processes with larger singular values are faster, their loss will be reduced in few iterations and with small total movement of the weights. When these processes have essentially stopped, the slower ones with small singular values will keep reducing the loss slowly and growing the weights to large magnitudes. With high alignment gradient descent requires a small magnitude of weights to reduce large portion of the loss the initial fast phase. Early stopping after this amount of loss reduced guarantoes small weights, and thus better generalization, without a need for explicitly regularizing the norm of the weights (Oymak et al., 2019a). Figure 8 illustrates the fast and slow processes and the fast and slow phases for two representations with high and low alignment.
6.2 A Small Experiment Connecting the Alignment Curve and Convergence Rates
To illustrate the impact of alignment on convergence rates, we fix the representation and create two synthetic label vectors with different alignment properties for that representation. The representation is obtained by sampling 1000 points from the UCI CT Position dataset. The first label vector, y1, is set to the 10th singular
9

---

Published in Transactions on Machine Learning Research (09/2022)
vector (approximately magnitude 70). The second label vector, M2, is weighted sum of the 5th and the 50th singular vectors (approxinately magnitude 115 and 30). We visualize their alignment curves in Figure 9.
8.0620
small direction
0.0000
allgnment
06.
W8t small
singuar
1/2
8.00CC
y/
Nost of the Iabel vector In cirection with large singuler
the lebe vector in
intanmadiate
gguial value
60
320
340
180
3000
200C
3000
4000
3000
threshold
Iteration
Figure 9: (left) Aligument curves for the two label vectors. After a threshold 30 the alignment curve drops slightly for y2, representing that small amount of y/2 is aligned with the smaller singular value, and then drops fully at threshold 115, most of y/ aligned with the large magnilude singular value. The alignment. curve for yn drops at 70, since all of y1 is aligned with the singular value of magnitude 70. (right) Learning curves for these two label vectors. The convergence rates are as predicted by the alignment curve.
We trained linear model each of these two label vectors and plotted the learning curves. These two models in fact show different comparative rates of convergence in early and late training. Initially, the model trained on y/2 converges faster as it is reducing the loss corresponding to the 5th singular vector (high magnitude). When this loss mostly reduced, the model keeps slowly reducing the part that corresponds to the small singular value. The model for Y1 consistently decreases error to zero, since it is perfectly aligned with a medium magnitude singular value; it slower than y/2 at first, but then crosses as it maintains the same rale of decrease once yn enters the slow phase corresponding to redueing error for the small singular value.
7 Co-occurrence of Alignment and Transfer
In this section, we discuss why transfer is facilitated when reprosentation learned on a source task is aligned with the labels for a target task. A representation can be said to facilitate transfer if it improves convergence rates and final accuracy (generalization performance). In the last section, we discussed why alignment can improve convergence rates for single task. Now, we evaluate more directly in the transfer setting, where the target task differs from the source task.
7.1 Positive and Negative Transfer in a Controlled Setting
We demonstrate the correlation between alignment, transfer, and task similarity using peaks functions, framework of related and unrelated tasks for neural norks (Caruana, 1997). A peaks function is of the form Px,v.2 = IF X > 0.5 THEN Y ELSE Z where X,Y, Z are variables. There are six possible variables A, B,C, D, E, F defined on [o, 1) and the label of a peaks function depends on three out of these six variables. A total of 120 peaks functions can defined using different permutations of three out of six variables: PA,,CC, PA,B.D..... PF.,D
Each variable is encoded into 10 features instead of being fed directly network as scalar. The features are obtained by evaluating an RBF function centered at points 0.0, 0.1, .... 0.9, with height 0.5 and standard deviation 0.1. Therefore, there are 60 features representing the 6 variables, and scalar label that depends on 30 of these features. Inputs are normalized to a length of one and the labels are normalized to have a mean of zero.
The neural network used the experiments has input units, one hidden layer with 60 neurons and ReLU activation, and one linear output unit. The model is trained by reducing mean squared error using Adam with batch-size 64 and step-size 0.001.
10

---

Published in Transactions on Machine Learning Research (09/2022)
200
alignment
alignment
transfer
alignment
500-
500
300
422-
400
original
400
300
init
20
20
36
40
90
to
10
20
30
45
30
threshold
threshold
threshold
(a) Source task
(b) Similar task
(c) Dissimilar task
Figure 10: Training increases aligmment with the source task and related tasks and decreases aligmment with unrelated tasks. The plots show alignment between the label vectors of the source task, related task, and unrelated task and the original representation (original), the initial hidden representation (init), and the hidden representation after training on the source task (transfer). The results are averaged over 10 runs. Error bars show standard errors. In (a), training on the source task increases the representation alignment. for the source task itself. Aligument with the related task's labels is also increased (b), and the transferred representation is better aligned than the original features. For the unrelated tasks, however, training on the source task reduced the aligument, and the transferred representation is worse than the original features (c).
First we study whether training a neural network on the source task results in a representation that is well aligned with the source tasks and similar tasks. We pick a source task S by randomly choosing 3 variables from A-F and consider two target tasks T1 and T2. TI has the same variables as S but in a different order and T2 uses the three her variables. For example possible setup is: S= PA,B,C, = PB,A,C, T2 = PE.D.r. P- We create 10000 imputs by sampling the variables A-F randomly from [0, 1) and compute the outputs for the source and target tasks to create a dataset of size 10000 for these tasks. Then we train a neural network only on S until convergence and compare the aligmment ween the labels of the source and larget lasks with the hidden representation after training and the hidden representation at initialization.
The hypothesis that training on S increases the aligmment between the hidden representation and the labels of task if the task is similar to S and reduces it if the task is dissimilar. We also include the alignment between the original inputs and the labels. The extracted representations are normalized to length one, to appropriately measure alignment. In Figure 10, we can see that the hypothesis is verified: training the source task increases the alignment for both the source task and the related task, and decreases alignment with the unrelated task.
Next we want to check representations with better aligument also have better generalization when training data scarce. This time we choose 100 out of the 10000 points for the related and unrelated task. and compared the performance of a linear model on the original features, hidden representation initialization, and hidden representation after training on the S. The test error is evaluated on new dataset of size 1000 for evaluating the model's generalization. Figure 11 (a,b) show training and test errors after certain number of iterations for the related and unrelated task. Figure 11 (c,d) show the norm of weights after a certain reduction of training loss.
The relative performance of the representations mirrors their alignment with the labels. Further, the role of alignment is noticeable in the fast and slow phases of the learning curves. When the target task is similar, the initial fast phase of the transferred representation reduces a large ratio of the overall loss with a small generalization gap and without requiring large weights. Comparison with the representation at initialization shows that this benefit comes from training on the source task. In the unrelated task, where training on the source task reduces alignment, optimization with the transferred features enters the slow phase early in training and performs worse than both the original features and the initial hidden representation. This reduction in alignment provides a clear explanation for this negative transfer.
Higher layers can specialize to the source task. We ran the experiment with peaks functions this time with three hidden layers instead of one to let the later layers specialize to the source task. Figure 12 shows
11

---

Published in Transactions on Machine Learning Research (09/2022)
0.31
init
of weights
0:00
original transfer
044
0:0
0.7
0:0
903
180
199
040
150
3019
300
321
538
086
322
043
03
805
terations
Retelloon
loss reduction
(a) Learning curves
(b) Learning curves
(c) Magnitude weights
(d) Magnitude of weights Dissimilar task
Similar task
Dissimilar task
Similar task
Figure 11: Train (dashed lines) and test (solid lines) errors of linear models after a certain number of steps (a,b) and the magnitude of weights after certain reduction of training loss (c,d). The results are averaged over 10 runs after a sweep on three different step-size values. On the related task (a) for the model with the transferred representation that has better alignment the initial fast phase of the training, where the weights and the generalization gap is small, can substantially roduce the training loss (see the initial sharp drop the green curve). This fast phase almost nonexistent in b since the aligmment of the transferred representation is decreased, and from the beginning the model needs grow its weights and the generalization gap to larger magnitudes to reduce the training loss.
layer-by-layer alignment curves of the learned representations on the source task (left) and the related task (right). The monotonous pattern in the main paper persists in the left plot as the representations closer to output adapt to the source lask, and there no such monotonous increase alignment with another task as seen in the right plot. This observation consistent with the experiments by Yosinski et al. (2014) on transfer lcarning with fine-tuning that show layers that closer to output specialize to the source task, often at the cost of performance on the target task. As the depth of the transferred layer increases, the set of possible target tasks that would be deemed similar to the source task becomes smaller, since the learned representation only keeps the information necessary for the source task.
700
700
anngteent
600
alignment
600
500
300
200
300
oiginall
100
threshold
threshoid
(a) Source Task
(b) Target Task
Layers closer to the output specialize the source peaks function (left) and are not better for transfer to the target peaks function (right).
Figure 12:
7.2 Alignment in Pre-trained CNNs and Connection to Transfer
A natural setting to test alignment is in areas where big neural networks pre-trained for easy re-use. One such setting is Computer Vision, where end-to-end training with deep Convolutional Neural Networks (CNNs) has replaced training lincar models on hand-crafted features. Training deep CNN from scratch, however, requires a large amount of labeled data which is not available for many tasks. Instead, a common strategy is to use transfer (Oquab et al., 2014; Goodfellow et al., 2016). CNN is trained on a large corpus of data, presumably related to wide array of target tasks, and made publicly available. The pre-trained model is then used create features for the target lakk-oftnn using the last hidden layer of the pre-trained CNN and simpler model trained these features with the more limited dataset for the target task.
12

---

Published in Transactions on Machine Learning Research (09/2022)
We take CNN first trained on the large ImageNet dataset (Russakovsky et al., 2015), and use features given by the last hidden layer to train linear model on Cifar10 and Cifar100 (Krinhevsky et al., 2009). The benefit. of feature transfer from ImageNet to Cifar10 and Cifar100 is already established in the literature (Kornblith et al., 2019), and we ask whether an increase in representation alignment is behind this gain. The CNN models that are studied are VGG16 (Simonyan & Zisserman, 2014) and the two residual networks ResNet50 and ResNet101 (He et al., 2016).
For each one of target tasks 1000 data points are sampled from two random classes and given #1 labels. We then plot the aligmment for the input features and the hidden representations before and after training ImageNet. Recall alignment from equation 2, E(ea,27) (v, y)² for a given threshold т. Figure 13 (a) shows the results for ResNet101 on Cifar100. The plots for other architectures and Cifar10 in Appendix C show the same trend.
Training on the source task (ImageNet) significantly incrcases alignment with the related tasks (Cifar10 and Cifar100). The aligument stays high and drops off suddenly, indicating that the label vector is largely concentrated on larger singular values. An interesting observation is that the hidden representation at random initialization is extremely poorly aligned. The plots for other architectures in the appendix show that the drop in aligmment at initialization is more extreme in deeper networks. This mirrors what we observed in Figure 7. A possible explanation is that with more depth, successively more information in the inputs is discarded. Previous work has observed that at initialization, deeper layers have little mutual information with the output (Shwartz-Ziv & Tishby, 2017). Such layers are closer to random vectors, in terms of relationship to the label vector, so unlikely to be well aligned.
We then extract 1000 images from two random classos of Cifar10 and Cifar100 and compare the alignment of representation extracted from ResNet101 pre-trained on ImageNet, SIFT (Lowe, 1999) features with dictionary size 2048, and HOG features (Dalal & Triggs, 2005) with 9 orientations, 64 pixels per cell, and 4 cells per block. We also include Radial Basis Function (RBF) representation of the data with bandwidth 1.0. The input to Radial Basis Functions is a flattened image normalized to length one, and the centers are set to all the 1000 normalized flattened training data points to ensure expressivily. We see Figure 13 (b) that, CNN features show considerably higher alignment than all three other representations on Cifar100. The plot for Cifar10 is in Appendix C.
Finally we train models with both squared error and logistic loss on these representations. Figure 13 (c,d) show the learning curves on Cifar100, and Appendix C' provides the plots for Cifarl0. With both loss functions, higher alignment results in faster optimization and better performance on test data. An interesting observation the high training error of RBF despite its expressivity. We computed the closed form least. squares solution for this representation and found that achieves zero squared error and full classification accuracy, which suggests that the high training error of gradient descent is due to extremely slow convergence. Note that alignment is not simple consequence of linear separability, and, as in the case of this RBF representation, linearly separable representation can have low alignment.
7.3 Co-occurrence of Alignment and Task Similarity in Object Recognition
This section studies the relationship between task similarity and alignment in object recognition tasks with fino-tuning. The first experiment is on transfer between different splits of ImageNct and Cifar10. The second experiment is on the Office-31 dataset and focuses negalive transfer.
Saageeee--iffrr00. We study if pre-training a neural network a more similar source task in this setting results in higher alignment between the extracted representations and labels of the target task. Yosinski et al. (2014) proposed splitting ImageNet into 551 classes of artificial objects and 449 classes of natural objects for transfer experiments within ImageNet. We adopt this nothologggy but for transfer experiments from ImageNet to Cifar10. We first split ImageNet into two source tasks of ImageNet-Artinn Artificial with 551 classes and ImageNet-Natural with 449 classes. The target tasks are all binary classification tasks between artificial classes of Cifar10 and all 15 binary classification tasks between natural classes of Cifar10. Details of the splits are in A Appendix D.
13

---

Published in Transactions on Machine Learning Research (09/2022)
ResNet101
aligmment
transter
aligmmmet
HOG
original
RBF
64
threshold
SIFT
treeshold
28
Reretion
69
306
250
(a)
(b)
(c) MSE
(d) Logistic Loss
Figure 13: Measuring Alignment and Convergence rale on Cifar100 using a ResNet101 Pre-Trained on ImageNet. (a) Representation alignment for imput features (original), hidden representation at initialization (init), and hidden representation after training on ImageNet (transfer). Training ImageNet increases the alignment. (b) ResNet. features show higher alignment than handcrafted image features and RBF features. The curves are averaged over 5 runs and the shades show standard errors. (c.d) Train (dashed) and test (solid) learning curves for squared error and logistic loss minimization on the representations in the second plot. CNN representations show the best performance.
The hypothesis is that pre-training on ImageNet- Artiffcial results in higher alignment than pro-training ImageNeet-Ntture if the larget task is classification between artificial objects and lower alignment. if the target task is classification between natural objects. The architectures in this experiment are ResNet18 and Tokens-to-Token Vision Transformer (T2T-ViT) (Yuan et al., 2021). We first pre-train the model on each source task. Then we create a balanced dataset of size 2000 from two natural or two artificial classes Cifar10, feed the input of this dataset the two pre-trained networks, and extract hidden representations from the last four hidden layers. Each of these extracted representations results in an aligmment curve. For each layer, if the target task is classification between natural objects, we subtract the alignment curve of the model pre-trained ImageNet- Artificial from the alignment curve of the model pre- pre-trained on ImageNet-NNttural If pre-training on ImageNet-Natural results in higher aligmment for this target task, the result of this subtraction will be positive. For a larget lask of classification between artificial objects, we subtract the alignment curve of the model pre-trained on ImageNet- Naturaa from the alignment curve the model pre-trained on ImageNet-Artificcan Again, the hypothesis is corroborated if the result of this subtraction is positive.
Figure 14 shows the results of the subtraction averaged over the 21 larget tasks to minimize clutter. Results for individual target tasks are provided in Appendix D. On average over the target tasks, pre-training on similar source task results in higher alignment in all the studied hidden layers and across all values of threshold in ResNet18 and a wide range of thresholds in ViT.VIT. For ResNet18, the difference was more extreme for layers closer to the output, indicating that the hidden layers closer to the output have specialized to the source task.
We also ask if this increase in alignment correlates with transfer performance in more common setup of fine-tuning a multi-class classification network. Similar to ImageNet, split Cifar10 to two target tasks of Cifar Notturll with six classes and Cifar10-Artifich with four classes and fine-tune each pro-trained network each larget (ask. As seen in Table 15, for both larget. tasks and with both architectures pre-training on the similar source task, which has better alignment in the corresponding binary classification tasks, also results in better fine-tuning performance in multi-class classification. Studying the relationship between alignment in hidden layers and dynamics of a fine-tuned multi-class network beyond the scope of this paper and left for future work.
Office-31: We test whether negative transfer correlates with reduction in alignment in experiment inspired by the recent study on negative transfer (Wang et al., 2019). The datasct is Office-31 (Sacnko et al., 2010) which has three domains Amazon, Webcam, and DSLR. The images in these domains are taken with different types of cameras and this results in a mismatch between the imput distributions. The task is to classify classes and the classes are shared among the domains. The neural network ResNet18 which has been pre-trained on ImageNet before the experiment, as otherwise all final accuracies would be low. Each time, one domain is picked the source task and its labels are replaced with uniform random numbers
14

---

Published in Transactions on Machine Learning Research (09/2022)
elightert difference
alighmert difference
100
threshald
Inreshold
Figure 14: Results of subtracting alignment curves for the last four hidden layers of ResNet18 (left) and T2T-ViT (right), averaged over the target tasks. Each curve corresponds to a layer and darker means closer to output. For each layer and each individual binary classification task, we subtracted the alignment curve of the model pre-trained on the less similar source task from the one obtained from pre-training on the more similar source task. Note the y-axis is different in the plots. All the four subtracted curves have large posilive numbers across all thresholds for ResNet18 and wide range of thresholds for
Source (Target Natural
Artificial
Source \Target Natural
Artificial
Natural 94.55
95.70
Natural 84.33
91.00
Artificial 90.49
97.14
Artificial 83.10
92.80
Figure 15: Multi-class classification test accuracy of ResNet18 (left) and T2T-VIT (right) in percent after fine-tuning, Each row is source task from ImageNet and each column is target task from Cifar10. For both larget lasks, pre-training on similar source lask resulted in better performance. This mirrors the relative alignments in corresponding binary classification tasks.
and a different domain is picked as the target task. The difference in input distribution together with label randomization in the source lask results in negative transfer to the target task, that is, a network that is pre-trained on the source task and then fine-tuned on the target task performs worse than a network that is directly trained on the target task. Table verifies negative transfer in all pairs of souree and target task in multi-class classification. Pre-training on Amazon and DSLR resulted in the largest and smallest degree of negative transfer. Further experiment details in Appendix D.
For each pair of source and target domain and for each binary classification task in the target domain we subtract the alignment curve of the model that is pre-trained on the souce task from the alignment curve of the model that is not pre-trained on the source task. The alignment curves are computed from the last hidden layer before fine-tuning and, similar to the previous experiment, the differences are averaged over the binary classification tasks. If the difference is positive it means that, on average over the binary classification tasks, pre-training on the source task has reduced alignment for the target task. The results are provided in Figure 16 where each curve corresponds to pair of source and target task. The curves all have positive values which shows reduction in aligumett in all the six pairs of tasks. Furthermore, pre-training Amazon and DSLR results in largest and smallest reduction in alignment, which mirrors the degree of negative transfer in Table 1.
8 Related Work
Early work by Hansen (1990) studied the closed-form solution of linear regression with & regularization truncated SVD under the Discrete Picard Condition (DPC). DPC states that the magnitude of projections of the label vector on the left singular vectors decays to zero faster than the sequence of singular values. A follow-up paper (Hansen, 1992) studied the closed-form solution of Eerreullrrzeed regression under DPC using L-curves which show the magnitudes of residual and weights at different levels of regularization. The name of these plots comes from their L-shaped appearance which means that certain amount of residual can be reduced with small weights and reducing the rest of the loss requires large weights. The authors attributed this pattern to the projection of the label vector onto the directions of large and small singular values.
15

---

Published in Transactions on Machine Learning Research (09/2022)
115
alighment differencee
Source TTarget
DSLR 88.0
Webcam
Amazon
None
92.5
82.7
DSLR
70.3
65.6
Webcam 30.7
47.3
00
Amazon
4.7
4.6
threshald
Figure 16: Results of subtracting aligument curves for the last hidden layer of Re in the Office-31 dataset, averaged over the binary classification tasks in the target domain. Each curve corresponds to pair of source and target domain. The differences are largely positive which means that pre-training on the source task reduced alignment in all these cases.
Table I: Test accuracy in percent after fine-tuning in the Office-31 dataset. Each row is source task and each column is a target task. The first row, None, shows the performance of the model that was not pre-trained on any of these source tasks. In all pains of source and target tasks, pre-training considerably reduced performance. This mirrors the relative alignments in corresponding binary classification tasks.
Understanding neural networks surprising generalization ability is an active area of research (Jiang et al., 2019). Neural networks generalize well on real-world data despite having the capacity to fit random labels Zhang et al. (2017). This phenomenon, called benign overfilting, has been studied in linear regression and associated with the eigenspectrum of the covariance matrix (Bartlett et al., 2020). Work on the double-descent phenomenon (Belkin et al., 2019; Nakkiran et al., 2021) has also challenged the classic notion of bias-variancs trade-off and provided a framework for studying generalization of models with high capacity.
Jacot et al. (2018) provided the Neural Tangent Kernel (NTK) framework that approximates the behaviour of overparametrized neural networks and can be studied theoretically. They observed faster convergence of neural networks in directions that correspond to the first eigenvectors of the kernel's Gram matrix. As label randomization tests by Zhang et al. (2017) resurrected interest in the relationship between the task and representation, Arora et al. (2019) studied the difference between datasets with true and random labels for gradient descent oplimization and generalization. They provided an analysis for overparameeterized neural networks that requires projecting the label vector on the eigendirections of the Gram matrix and found that true labels are mostly aligned with directions of large singular values. Oymak et al. (2019a) refined the analysis of Arora et al. (2019) by separating the large and small eigenvalues with a threshold and provided better generalization guarantees by early stopping before the slow processes grow the weights large magnitudes.
Recent work has explored adaptation of the network to task (Oymak et al., 2019a;b; Baratin et al., 2020; Kopitkov & Indelman, 2020; Ortiz-Jiménez et al., 2021). They showed that the matrix of kernel associated with the network evolves such that the labels are mostly aligned with the first eigenvectors of the corresponding matrix. Through a similar functional view, Lampinen & Ganguli (2019) studied adaptation of deep linear networks to the task without the kernel approximation and showed the benefit of learning multiple related tasks at the same time.
It is important to distinguish this form of adaptation, which is sometimes called "neural tangent feature alignment" or "neural feature alignment," " and alignment of hidden representations. Neural tangent feature alignment views the whole network as black-box function and studics adaptation of the model in the space of functions. Our work investigates the intermediate layers one by one and shows that each one increases alignment its representation. For transfer learning, neural tangent feature alignment implies that a network is trained on a source task, the whole network can be further trained efficiently on a related target task. Hidden representation alignment means that the features learned in each hidden layer are suitable for training model the related target task.
16

---

Published in Transactions on Machine Learning Research (09/2022)
Finally, Maennel et al. (2020) showed how the weights in each layer of neural network adapt their spectral properties to the input features, which explains the benefit of feature transfer on tasks with similar input. This analysis disregards adaptation to labels and cannot explain why transfer to unrelated tasks on the same input can hurt (like the case negative transfer to dissimilar peaks functions). The authors explain negative transfer through inactivity of ReLU units.
9 Conclusion
In this work, developed the concept of representation alignment in neural networks and showed that it correlates with transfer performance. This alignment emerges from the training process. In contrast, we showed that random neural networks have poor alignment, even if they are sufficiently large to enable accurate approximation. We measured the properties of a variety of noural network representations, showing that it correlates with positive transfer for similar tasks and negative transfer for dissimilar tasks; that widely used CNNs have high levels of aligument; and finally that this property appears to emerge across a variety of architectures, with higher alignment in the later layers.
Understanding how the dynamics of gradient descent leads to the emergence of alignment is an important next step. Recent work on the Neural Collapse (NC) phenomena (Papyan et al., 2020) can help with this direction. The first of these phenomena (NC1) is that at late stages of training, within-class variations of the last hidden layer activations shrink to zero and the activations collapse to the class means. Going back to the example in Figure we can see the relationship to alignment. In the left figure where the points in each class are close to each other alignment high. In the figure on the right, where the points belonging to the same class are separated along the top principal component and are therefore further apart, the label vector is mostly in the direction with smaller singular value and alignment is low. Although this argument is still preliminary and the work on NC focuses on the last hidden layer and late training, still highlight this relationship to provide starting point for future theoretical work.
The authors would like to thank Alireza Fallah, Behnam Neyshabur, members of the RLAI Lab at the University of Alberta, and the anonymous reviewers for their invaluable comments and discussions through this project. This research is funded by the Canada CIFAR Al Chairs program and Alberta Machine Intelligence Institute.
Acknowledgment:
References
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg, Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In International Conference Learning Representations, 2017.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In International Conference on Learning Representations, 2017.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, 2019.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixcl-wise explanations for non-lincar classifier decisions by layer-wise relevance propagation. PloS one, 10(7), 2015.
Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vincent, and Simon Lacoste-Julien. Implicit regularization in deep learning: A view from function space. arXiv preprint arXiv:2008.0090 2020.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 2020.
17

---

Published in Transactions on Machine Learning Research (09/2022)
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. What do neural machine translation models learn about morphology? In Annual Meeting of the Associalion for Compulational Linguistics, 2017.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias variance trade-off. Proceedings of the National Academy of Sciences, 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pullern Analysis and Machine Intelligence, 35(8), 2013.
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and lask-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature commanications,, 12, 2021.
Rich Caruana. Multitask learning. Machine learning, 28(1), 1997.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. A Algorit for learning kernels based on centered alignment. The Journal of Machine Learning Research, 13, 2012.
Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In Conference on Computer Vision and Pattern Recognition, 2005.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. What is grain of sand in the desert? analyzing individual neurons in deep nlp models. In AAAI Conference Artificial Intelligence, 2019.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0488 2018.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
Franz Graf, Hans-Peter Kricgel, Sebastian Pölsterl, Matthias Schubort, and Alexander Cavallaro. Position prediction in et volume scans. 2011.
Per Christian Hansen. The discrete picard condition for discrete ill-posed problems. BIT Numerical Mathematics, 30(4), 1990.
Per Christian Hansen. Analysis of discrete ill-posed problems by means of the --urve. STAM review, 34(4), 1992.
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning, 2016.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Compuler Vision and Pollern Recognition, 2016.
Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems, 2020.
Sergey Tolle and Christian Szegedy. Batch normalisation: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, 2015.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems, 2018
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic gencralization measures and where to find them. 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 141:1112.00000 6980, 2014.
18

---

Published in Transactions on Machine Learning Research (09/2022)
Dmitry Kopitkov and Vadim Indelman. Neural spectrum alignment: Empirical study. In International Conference on Arlificial Neural Networks. Springer, 2020.
Simon Kornblith, Jonathon Shlens, and Quoc V Te. Do better imagenet models transfer better? In Conference on Computer Vision and Pattern Recognition, 2019.
Alex Krizhevsky al. Learning multiple layers of features from tiny images. 2009.
Andrew K. Lampinen and Surya Ganguli. An analytic theory of generalization dynamics and transfer learning in deep linear networks. In International Conference on Learning Representations, 2019.
David G Lowe. Object recognition from local scale-invariant features. In International Conference on Computer Vision, 1999.
Hartmut. Maennel, Tbrahim Alabdulmohsin, Πya Tolstikhin, Robert JN Baldock, Olivier Bousquet, Sylvain Gelly, and Daniel Keysers. What do neural networks learn when trained with random labels? arXiv preprint XrXi::2006. 10455, 2020.
Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online dictionary learning for sparse coding. In International Conference on Machine Learning, 2009.
G Marcus. Deep learning: A critical appraisal. 2018.
Tomas Mikolov, Tlya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, 2013.
Ari S Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. In Advances in Neural Information Processing Systems, 2018.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Tlya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanies: Theory and Experiment, 2021.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What being transferred in transfer learning? In Advances in Neural Information Processing Systems, 2020.
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In Conference on Computer Vision and Pattern Recomnition, 2014.
Guillermo Ortiz-Jiménez, Seyed-Mohsee Moosavi-Dezfooli, and Pascal Frossard. What can linearized neural networks actually say about generalization? 2021.
Samet Oymak, Zalan Fabian, Mingchen Li, and Mahdi Soltanolkotabi. Generalization guarantees for neural networks via harnessing the low-rank structure of the jacobian. arXiv preprint arXiv:1906.085. 05392, 2019a.
Samet Oymak, Zalan Fabian, Mingchen Li, and Mahdi Soltanolkotabi. Gencralization, adaptation and low-rank representation neural networks. In Asilomar Conference on Signals, Systems, and Computers. TEFE, 2019b.
Vardan Papyan, XY Han, and David L, Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 2020.
Jeffrey Pennington, Richard Socher, and Christopher D Manning, Glove: Global vectors for word representa- tion. In Conference on Empirical Methods in Natural Language Processing, 2014.
Peng Qian, Xipeng Qiu, and Xuan-Jing Huang. Investigating language universal and specific properties in word embeddings. In Annual Meeting of the Association for Computational Linguistics, 2016.
19

---

Published in Transactions on Machine Learning Research (09/2022)
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 2015.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, 2010.
Xing Shi, I. Padhi, and Kevin Knight. Docs saring-basce neural mt learn souree syntax? In Conference on Empiricul Methods in Natural Language Processing, 2016.
Ravid Shwartz-7iy and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint trx:::1703.008 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arx::: 1556, 2014.
Karen Simonyan, Andrea. Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In International Conference on Learning Regresenlations, 2014.
David S Touretzky and Dean A Pomerleau. What's hidden in the hidden layers? BYTE, 14(8), 1989.
Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In Proceedings of the JEEF/CVE conference on computer vision and pattern recognition, 2019.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep noural networks? In Advances in Neural Information Processing Systems, 2014.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuichong Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenct. In Proceedings of the IEEE/CVF International Conference Compuler Vision, 2021.
Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. Graying the black box: Understanding dqns. In International Conference Machine Learning, 2016.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Europeon Conference on Computer Vision. Springer, 2014.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.
Bolei Zhou, Aditya Khosla, Àgata Tapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene cnns. In International Conference on Learning Representations, 2015.
A Other Plots for Measuring Alignment in Different Neural Networks
This plots in Figures 17 and show the increase alignment on training data with different optimizers and batch-sizes well as the increase in alignment test data.
20

---

Published in Transactions on Machine Learning Research (09/2022)
Cifar10
MNIST
CT Position
1803
100
alignment
original
Batch Size
128
256
alighment
alignment
allgmment
orig na
RMSProp SGD
Optimizer
Adam
8
Chreshold
Greshold
Bhreshold
Figure 17: Training increases alignment in hidden representations regardless of batch-size and optimizer. The orange line is the input features, the dashed lines are hidden representations before training, and the solid lines are hidden representations after training. The plots averaged over 5 runs and the error bars show standard errors. The label vector for the regression task was moved to the interval [-1, 1] for easier comparison with classification. The curves for learned representation on MNIST mostly overlap as all models can achieve a high level of alignment on this task.
Convergence Rate
For completeness, we include the convergence rate results here.
Proposition 1. Given full rank representation matria & and label vector y, let w" = 4) T be the optimal linear regression solution and dmna be the marimam singular value for tb The batch gradient descent updates W4+1= = w. - -1/ (fw,- - y) with stepsize rj < o maz and wy results in
(())
Σ
2 y)
(4)
r(
y1 2
Σ
no7)2 y) + 1 y/1²
j=1
Proof for Proposition 1, For UVV the thin SVD with Radd let A = -= VAVT and b b= & update
VT The gradient descent update corresponds to the following iterative linear system
W++1 = wi -7/ (dww - 3) = wi - -7(Aw - b)
(I nA) w. + mb
Starting from wo, we get that
= da = Iy(I - b
W2 (1b) nb 7i([1 (A)) (I 7A)}b
Σ - n
W/+1
1=U
21

---

Published in Transactions on Machine Learning Research (09/2022)
Cifar10
MNIST
CT Position
alignment
aligomment
origina
Depth
1000
1800
alignment
803
aligmment
algnmmet
origing
256
Width
128
64
threshold
cresholdd
threshold
LeakyReLU Linear
200
80
PReLL
Algement
Algnmment
Algoment
original
Tanh
Activation
Grrohoold
Ghrodhold
threshold
1800
1300
alignment
elignment
alignment
priging
32
Batch Size
128
256
1000
1800
alignment
alignment
alignmant
prig nd
60
Optimizer
RMSSPrp
SGD
Adam
threshold
Shreshold
threshold
Figure 18: Training increases alignment in hidden representation on test data. The orange line is the input features, the dashed lines hidden representations before training, and the solid lines are hidden representations after training. The plots averaged over 5 rums and the error bars show standard errors. The label vector for the regression task was moved to the interval [-1, 1] for casier comparison with classification.
Further,
I 1- = V - -VVVV =
(I (I-nA) = V(I -
Σ (1--A) nA)' v Σ (1-
==0
22

---

Published in Transactions on Machine Learning Research (09/2022)
If we let A, = Ei=o( nA)' and xy the entry on the diagonal in A, then because X, 0 for all j, each component of this diagonal matrix corresponds to
1 1 (1 (1-1)1)) 1 1- -n Moj Σ n>j}² 1=
1- 1-(1
n
Additionally,
(1- V VEI-NAYVTVE''T
= -V(I-nA)E''">
For An,- (1 n)1))= 9Ay²
where the inverse exists due to conditions on 7, have w == =VV, E y.
Finally we get
|1i+1 - ==1l == > - U =n//V(A, - -T vll
= = wll
Notice that A..j - Reg (1- (1-12,)*+1 and each component of 17(A, Ä
equals (1-aA))*1
We therefore get.
9A3
|10411 w11
I|m(A, - AJΣ U. TTyl' y/l²
r()) (1 (1- (1- (t
oj 4j y
r())
2) 2(-1) (u. v)
Σ j=1
(1 no
which shows the first part of the proposition. The alignment relationship is clear in this result. The term (u] big if (u. v) is big and oj is small. This is further exacerbated by the fact that (1 - nnaa ) 2(++1) shrinks slowly if oy is small. On the other hand, consider the case that v) is notably smaller than dj. even if oj is small. Then this term contributes only small amount to the loss in the beginning, i.c., (N u)² and this small amount is then reduced with each iteration, slowly aj is small and quickly aj is big. The ideal rate is obtained when high magnitude (u, y)² is concentrated on large o5. Now show the rate of convergence of the predictions:
1wr+1 2w0112 || )²1l
IINUEVT V UT m/l
-IIUE(A AJETUT UT yl²
-IIEEA. -AEΣ T y11²
r(0)
Σ [a +1 u,
r(#)
Σ (1 no 2(4+1) (u]
The term (1 - (1-10))(((+1)) appears again in this result and shows the improved convergence rate when (u. y)² is concentrated on directions of large singular values.
Now we can relate convergence in predictions to convergence loss. Gradient descent under the conditions
of the previous proposition converges to the minimum squared error solution. Define b, = || 2ll for
23

---

Published in Transactions on Machine Learning Research (09/2022)
any Rd. If lu- 0 then clearly w = || y/1 = || Dw''l Otherwise tw- is the projection of y the span of Φ, and Ou" is perpendicular to any vector within this subspace, including ow Dww Pythagorean theorem then gives -|o - y/1² = 1 y1l + u Dw"
Under the conditions of the previous proposition, if Alignment(( 1 v.T) 5 for threshold 0 < T S Oman, then gradient descent needs at most - -log(1 w/0)/(217²) iterations to reduce the loss by 0 WEwc
Proposition 2.
Proof for Proposition 2. The amount of reduction in loss after iterations is denoled by w and is equal to
ww Lw = -Ppw 2w21 |1 $w"11
x(0)
r(0)
Σ (u]y)² y)² 2 Σ (1 )03)) (u,Jn²
Σ (u] -(1
(1- - 7 x)²² + Σ
2 - (11-n03))(u33))
0
acas<T
Σ (u] v) -11 n13))
0,2r
8 - (1 - 2
where the second last step follows because each term in the second sum is non-negative and the last step follows from the definition alignment. Now, due to the condition on the step-size and for 0 < t S maa we have 0 < ypp2 and therefore (1 - -1) exp(-2tnrr)) Therefore
2 8(1 exp(-2tmp) = S log(1 w/6)/(217*)
2 1- w/o = 2tt7 2 log(1 - w/
C Other plots for Pre- Trained CNNs
Figure 19 shows alignment plots for different pre-trained CNN architectures. Figure 20 compares alignment curves of CNN features and handcrafted features Cifar10.
D Fine- Tuning Experiment Details
The classes in artificial and natural splits of ImageNet are the ones reported by Yosinski et al. (2014). Cifarl0-Natural consists of classes bird, cat, deer, dog, frog, horse, and Cifarl@-Artifiiall includes airplane, automobile, ship, and truck.
ImageNet- set--ifar11 Splits:
ImageNet-Cifar ResNet18 Hyperparameters: Optimizer: SGD; Momentum: 0.9; Batch-Size: 1024 (distributed over 4 V100 GPUs) for pre-training and 256 (on 1 K80 GPU) for fine-tuning: Initial Learning Rate: 0.1 for pre-training and 0.01 for fine-tuning; Learning Rate Decay Schedule: multiplied by 0.1 after each 30 epochs; Total Number of Epochs: 40 for pre-training and 10 for fine-tuning; Weight Decay: le-4; Image Transformations: resize to 256x256 and random cropping and horizontal flipping augmentation.
0.9; Batch-Size: 256 (distributed over 2 V100 GPUs) for pre-training and 64 (on V100 GPU) for fine-tuning: Initial Learning Rate: 0.01; Total Number of Epochs: 30 for pre-training and 10 for fine-tuning: Weight Decay: 1c-4; Image Transformations: resize to 256x256 and random cropping and horizontal flipping augmentation.
ImageNet-CCfar Cifarr0 12T-VIT Hyperparameters: Optimizer: SGD; Momentum:
24

---

Published in Transactions on Machine Learning Research (09/2022)
transfer
alignment
alignment
alignment
original
Init
threshold
threshold
threshold
(a) VGGI6, Cifar10
(b) ResNet50, Cifar10
(c) ResNet101, Cifar10
300
3996
alignment
transfer
alignment
alignment
original
init
threshold
threshold
threshold
(d) VGG16, Cifar100
(e) ResNet50, Cifar100
(f) ResNet101, Cifar100
Figure 19: Representation aligument for input feat (original), hidden representation at initialization (init), and hidden representation after training on ImageNet (transfer). Training on ImageNet increases the alignment for all three models and on both Cifarl0 and Cifar100.
ResNet101
alginment
HOG
lost
RBF
SIFT
RBF
SIFT
HOG
Res Net101
(a)
(b) MSE
(c) Logistic Loss
Figure 20: Mcasuring Alignment and Convergence rate on Cifar10 using a ResNet101 Pre-Trained on ImageNet.(()) ResNet features show higher alignment than handcrafted image features and RBF features. The curves are averaged over 5 runs and the shades show standard errors. (b,c) Train (dashed) and test (solid) learning curves for squared error and logistic loss mininization different representations. CNN representations show the best performance.
Office-31 Hyperparameters: Optimizer: Adam for pre-training and SGD with momentum 0.9 for fine- tuning; Batch-Size: 256 (on V100 GPU) for pre-training and 256 (on 1 V100 GPU) for fine-tuning; Initial
Learning Rate: 0.001; Learning Rate Decay Schedule: multiplied by 0.1 after each 30 epochs; Total Number of Epochs: 500 for pre-training and 100 for fine-tuning: Weight Decay: 1e-4; Image Transformations: resize to 256x256 and random cropping and horizontal flipping augmentation.
25

---

Published in Transactions on Machine Learning Research (09/2022)
Bresholdd
treashold
algrment ditert-ce
diferernce
algement ditererce
8
Unsedrati
Drewhhok
Figure 21: Individual alignment difference curves for the binary classification target lasks with ResNel18 (above) and T2T-VIT (below). Each plot corresponds one hidden layer. The orange and green curves show individual curves and the averaged curves reported in the main paper.
26


Reviews:
[{"review_id": "qGQTNUUjXS", "reviewer": "Reviewer_4c69", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n(1) The paper is clearly written and explained. Experiments are well described, ad contain enough details. All quantities are well defined and explained. \n(2) The problem of transferability is very important and interesting. In the current context it pertains to generalization to a new classes by retraining top layers,  \n \n\nWeakness: \n(1) The most interesting claim of the paper is that there is a causal relation between representation alignment, and better transfer, as the opening states, that alignment \"plays an important role\" in transfer. The discussion puts forward alignment as a \"reason\" for why NN transfer well.  The paper does not show that causal relation in a convincing case, but instead shows that alignment is correlated with transfer.  This correlation is demonstrated in one synthetic dataset and also using three ConvNets (VGG, ResNet50, ResNet101) on Imagenet. \n\n(2)  The paper does not do a good job in convincing that alignment, and not other properties of the representation is the key factor in transferability. Maybe there are other properties of a representation that could contribute to transfer that are more fundamental than alignment as defined here? \n\n\n\n\n\n\n\n\n(2) The empirical demonstration that representations become more aligned to the labels during training is convincing, but the paper does not put that observation in context of standard analysis of deep learning. It does not explain how that observation is different from previous work.  Clearly, when optimizing a representation layer such that it will be linearly related to the label vector through the classification layer w, that optimization tends to map samples such that projecting them using we will order them in a separable way (positive samples on one side, and negatives on another). This means that after mapping samples to the representation, it is expected to see them \"sorted\" along main axes. \n\n\n\n\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper suggests that representation \"alignment\" with target labels plays a role in providing better transfer.  \nFirst, the paper defines alignment as the quality where the vector of labels (across samples) is correlated with the top eigenvectors of the representation matrix (feature x samples). Second, they show in experiments on 3 small datasets that this alignment grows during training and monotonically increases with the depth where a layer is located in the network. \n\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "To convince that there is a causal relation between alignment and transfer, the paper should show that changing alignment, rather than changing other properties of the representation leads to better transfer.  TO achieve that goal, the paper should separate alignment from other properties of the representation like information content, manipulate only alignment (while maintaining other properties) and measure the effect on transferability.  I suspect that this is very hard to do. \nInstead, the paper can change its main message, and state that it finds that alignment is correlated with transfer", "broader_impact_concerns": "No specific concerns. This paper is about theoretical analysis of how learned representations fit labels in supervised learning. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gvlu5357lr", "reviewer": "Reviewer_VSnv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The overall writing quality is good. The motivation of the paper is clear and the paper is enjoyable to read.\n2. This paper tries to develop and investigate alignment as a potential reason for why neural network representations transfer well. The idea is insightful and convincing.\n3. Empirical observations are insightful and have the potential to inspire future works.\n\nWeaknesses:\n1. The conclusion in Eq(1) is not very rigorious. If $w_i^V$ and $y_i^U$ are independent. The eq(1) can lead to the conclusion that \"it is clear that if $\\sigma_i$ is small and the rotated $y_i^U=u_i^{\\rm T}y$ is big, then $w_i^V$ has to become very big, which is problematic\". However, $U$ and $V$ are all come from the feature matrix $\\Phi$ (singular value decomposition). As a result, $U$ and $V$ are not independent, $w_i^V$ and $y_i^U$ are not independent. It's very possible that $\\sigma_i$ is small and the rotated $y_i^U=u_i^{\\rm T}y$ is big. The present sequence of statements may be misleading. Maybe empirical observations should be put in the front to provide evidences.\n2. In section4.1, where is figure 19? Figure 19 in the supp has green and orange lines while in the text there are only yellow and purple lines. \nFigure 4 is interesting, the authors should replace Figure 19 with Figure 4.\n3. Datasets are not real enough. Most experiments are condducted in simple datasets.\n4. This paper measures alignment and convergence rate on Cifar100 using a ResNet101 pre-Trained on ImageNet. The observations only use frozen feature representations. How does alignment changes during the fine-tuning processes, which can be a more important observation for transfer learning setup.\n5. How this paper can inspire the design of transfer learning algorithm and narrow the distance from the true solution. Further discussions should be provided.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The main contribution of the paper is to develop and investigate alignment as a potential reason for why neural network representations transfer well. Specifically, this paper shows that after training, neural network representations align their top singular vectors to the targets. Further, this paper investigates the above representation alignment phenomenon in a variety of neural network architectures and finds that (1) alignment emerges across a variety of different architectures and optimizers, with more alignment arising from depth (2) alignment increases for layers closer to the output and (3) existing high-performance deep CNNs exhibit high levels of alignment. Finally, this paper highlights why alignment between the top singular vectors and the targets promotes transfer and shows in a classic synthetic transfer problem that representation alignment is the determining factor for positive and negative transfer to similar and dissimilar tasks.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Requested changes are all in weaknesses above:\n1. The conclusion in Eq(1) should be more rigorious.\n2. The picture does not agree with the text in section 4.1.\n3. More real-world datasets should be used in empirical observations.\n4. How does alignment changes during the fine-tuning processes?\n5. How this paper can inspire the design of transfer learning algorithm and narrow the distance from the true solution?", "broader_impact_concerns": "This paper has the potential to inspire future works which focus on transfer learning theories and algorithms. Although the work is not very ready for publication, it can provide insights to the community. My only broader impact concern is that the conclusion in Eq(1) may be misleading to the readers.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8eG6eaO0fB", "reviewer": "Reviewer_HjG1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n* The paper is clearly written, with useful diagrams, and the concept of representation alignment is well explained.\n\n* The paper provides some analytical intuition (proposition 1) as well as empirical evidence for their claims that representation alignment helps training.\n\n* There are a number of carefully controlled synthetic experiments that demonstrate the various points.\n\n* There are some experiments with real data/networks (imagenet->cifar transfer) to test the hypotheses.\n\nWeaknesses\n\n* The real-data experiments are rather limited. The paper studies imagenet->cifar with CNNs (including using custom splits to show that source/task matching is important and increases alignment). However, it is hard to conclude from this limited data that representation alignment is always a good indicator of transfer performance, especially when finetuning, which is more practically useful than training only a linear layer. I think the empirical section would be greatly strengthened if the correlation between representation alignment and better transfer was shown to hold across a variety of different architectures (e.g. CNNs, Transformers) and pre-training strategies (e.g. supervised, self-supervised, auto-encoders).\n\n* The quantification of representation alignment, and showing that this property correlates with improves linear transfer is intuitive. However, various results are presented that are not well explained. For example, in Figure 6 deeper/wider networks do not appear well correlated with improved alignment, although increased depth/width consistently improves performance. It is also observed that random features at initialization are less well aligned than the input features (e.g. Fig 13 (a)). Do these random networks transfer less well than a linear layer on the input? If not, then how does that fit with the rest of the findings?\n\n* The paper claims that representation alignment is the reason for good transfer: \"In this work, we developed and investigated alignment as a potential reason for why neural network representations transfer well.\"  However, I feel this claim is a little strong, the paper shows that aligned representations correlates with good transfer, but does not explain the cause of aligned representations (and hence good transfer).\n\nTypos:\n\n\"training with gradient descent on a representations with high alignment\" -> “on representations”\n\n“help speed learning” -> “speed up” (or accelerate)\n\n\"Training increases alignment in hidden representation\" -> \"representations\"\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces the concept of representation alignment. The representation alignment is the degree to which the label vector is aligned (dot product) with largest (above some threshold \\tau) singular values of the representation matrix (features for each datapoint from the last hidden layer of the model). The paper plots representation alignment curves, sweeping different values of \\tau which provides a measure of how well the representations are aligned with the labels.\n\nThe paper then shows that more aligned representations emerge during the training of neural networks under a variety of conditions. RBF networks/dictionary learning produces less well aligned representations. For linear models (one layer trained on top of hidden representations), the paper provides evidence that more aligned representations result in faster convergence. The paper also shows that better alignment between representations and downstream labels results in better transfer. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "* Expand upon the real-data experiments to show that representation alignment holds as a good predictor of transfer performance on more settings than supervised ImageNet->Cifar with a ResNet.\n\n* Provide further evidence or explanation for the cases where representation alignment does not seem to follow the expected pattern (e.g. larger networks appear not always better aligned).", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "af6mxF6cHX", "reviewer": "Reviewer_7c5a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n+ The empirical study is fairly thorough and covers many aspects of learning deep representations and spans different setups for the architecture, optimization, and tasks.\n+ The results are generally consistent and mostly conclusive for the different setups and tasks that are considered.\n+ Predicting and explaining negative transfer is an open and important direction which seems to be correlated with the used empirical notion.\n\n**Weaknesses**\n\n*Results and Experiments*\n- the paper sometimes implies causal relation between the representation alignment and two notions of generalization and convergence. It should be discussed clearly if it is causality or correlation that the paper is after for each of those two properties and then accordingly discuss if the experiments and observations are sufficient for either of the cases.\n- for generalization, the correlation between generalization and alignment is shown through improved accuracy. One important question here is the case of overfitting. Does the alignment increase during overfitting? If so, does this affect the hypothesis on generalization? How?\n- for convergence, the provided empirical evidence for improved convergence is quite limited. A conclusive observation requires more (well-defined) experiments in different setups and ideally accompanied by experiments on “real-world” datasets that clearly isolate the convergence rate from other factors. Some controlled setup such as Figure 9 are pedagogical and interesting and are important to include as well but even there it needs more in-depth study to establish that the discussed trend in convergence is in fact the result of the design (with the aligned non-zero singular values).\n- the results regarding negative transfer are quite interesting but the task is limited for a general claim w.r.t. alignment. Can this be extended to some other controlled but more realistic learning tasks? \n\n*Related works*\n- the prior empirical and theoretical findings regarding (some defined notions of) representation margin between classes in the intermediate representation space of trained neural networks should have some (potentially direct) relevance to the notion of alignment as used in this work, especially when it comes to correlating it with generalization. This is largely missing from the paper in different sections. The related literature is rich but one possible starting point to find such relevant papers could be the following work, the ones it cites and the ones that cites it: \n“Fantastic Generalization Measures and Where to Find Them” 2019.\n\n*Clarity*\n- the paper commonly states that the deep network’s representation for this study is taken from the last layer. Is the last layer the prediction layer or the last “hidden” layer which is the one before the outputs? If it is the last prediction layer, for the case of classification, is it the logits or after applying the logistic function? How does this affect the discussions around the observations? \n- on the one hand the whole empirical study is based on the comparison of “alignment” in different setups and contrasting high and low alignment, on the other hand, a precise definition of high and low alignment is generally missing. The definition of alignment in equation 2 only considers alignment for a single threshold. The discussions throughout the paper, however, use a qualitative notion of alignment that is never precisely defined based on comparing the plots of the defined alignment over different thresholds. Although the figures suggest a clear qualitative separation in most cases, this can become a source of confusion and subjective analysis and needs to be rectified. That is, hypothetically, not all cases will have one plot completely above or below another; they can intersect in general. \n\n*Minor points*\n- page 2: “Saliency maps and Layerwise Relevance Propagation”: citations missing\n- page 2: “n>=d”: is this always true? Especially for the target task. A discussion is missing on the necessity and imposed limitations of this condition. \n- page 3: when referring to Figure 2a, it does not become clear until later where the representation $\\Phi$ is taken from.\n- page 1-4: generally “alignment” is taken as an accepted notion and phrases such as “alignment is” are generally used. It might be better to use phrases such as “in this work alignment is defined as” or provide references to prior works to establish that this is *the* accepted definition of representation alignment in the literature.\n- page 4: “figure 19” -> probably meant to refer to the main paper’s figure 4\n- page 5: “Equation equation”\n- page 6: “increases alignment” -> increase alignment\n- page 6: “help speed learning”\n- fig 6, caption: “error bars” -> probably “shaded area”\n- fig 6, caption: standard error or standard deviation?\n- fig 6: the orange/red plot for the “original” and the orange/red for the trained alternatives are hard to distinguish and causes confusion since “original” is an important reference.\n- page 7: “the aim of this paper paper is show” -> to show / showing\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- At the highest level, the paper conducts an empirical study based on a notion of representation alignment that is associated with the alignment of a source/target task (in terms of labels) and directions of highest variation in the representation space (as identified by singular vectors of observed data). \n- Using this notion of alignment, it makes a number of empirical observations regarding the learnt representations of neural networks at the final layer of trained deep networks as follows.\n- The representations become more aligned after training a deep network for the source task.\n- The deeper representations of a trained network are increasingly more aligned for the source task.\n- The representation learnt by a deep network is more aligned than those learnt by RBF networks or unsupervised sparse coding optimizations.\n- In the context of transfer learning, poorly-aligned representations can explain negative transfers for a synthetic regression task.\n- In the context of transfer learning, the alignment correlates with similarity of image classification tasks as measured by semantic tree distances (wordnet).\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The main weaknesses I see that need to be addressed are regarding the conclusivity of the experiments, the clarity, and the discussion and comparison with other related works as outlined above.", "broader_impact_concerns": "The reviewer does not see any direct necessity for the inclusion of such section in this specific paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received four detailed reviews with insightful questions and constructive suggestions. After extensive revisions, authors managed to address the majority of comments, in particular through more results. One reviewer still held some reservations towards the paper, such as correlation vs. generalization/negative transfer/convergence rate. AE found these comments very thought-provoking while most of them were responded in an acceptable manner by the authors. The current paper turns out to be a nice study on representation alignment and representation transferability, and provides several useful insights and takeaways for the community of deep learning. It aligns clearly with the TMLR expectations and is recommended for acceptance.
Decision: Accept (poster)