Task structure and nonlinearity jointly determine learned representational geometry
Authors: Matteo Alleman
Jack Lindsey
Stefano Fusi
Venue: ICLR 2024 poster (2024)
Keywords: representational geometry
kernel target alignment
disentanglement
activation function
out-of-distribution generalization
Primary area: visualization or interpretation of learned representations
TL;DR: We show that the geometry of learned representations can be understood in terms of input and target geometries, and that tanh networks, more than relu networks, tend to align with output geometry

Abstract: The utility of a learned neural representation depends on how well its geometry supports performance in downstream tasks. This geometry depends on the structure of the inputs, the structure of the target outputs, and on the architecture of the network.  By studying the learning dynamics of networks with one hidden layer, we discovered that the network's activation function has an unexpectedly strong impact on the representational geometry: Tanh networks tend to learn representations that reflect the structure of the target outputs, while ReLU networks retain more information about the structure of the raw inputs. This difference is consistently observed across a broad class of parameterized tasks in which we modulated the degree of alignment between the geometry of the task inputs and that of the task labels. We analyzed the learning dynamics in weight space and show how the differences between the networks with Tanh and ReLU nonlinearities arise from the asymmetric saturation of ReLU, which leads feature neurons to specialize for different regions of input space. Feature neurons in Tanh networks, by contrast, tend to inherit the task label structure. Consequently, when the target outputs are low dimensional, Tanh networks generate neural representations that are more disentangled than those obtained with a ReLU nonlinearity. Our findings shed light on the interplay between input-output geometry, nonlinearity, and learned representations in neural networks.

Published as a conference paper at ICLR 2024
TASK STRUCTURE AND NONLINEARITY JOINTLY DE- TERMINE LEARNED REPRESENTATIONAL GEOMETRY
Matteo Alleman, Jack Lindsey" & Stefano Fusi Department of Neuroscience, Columbia University ma 38131122 eduu, jackw frekwwindsey@gneee com, sf22378 col umbia.edi
ABSTRACT
The utility of learned neural representation depends on how well its geometry supports performance in downstream tasks. This geometry depends on the struc- ture of the inputs, the structure of the target outputs, and the architecture of the network. By studying the learning dynamics of networks with one hidden layer, we discovered that the network's ' activation function has an unexpectedly strong impact on the representational geometry: Tanh networks tend to learn representa- tions that reflect the structure of the target outputs, while ReLU networks retain more information about the structure of the raw inputs. This difference is consis- tently observed across a broad class of parameterized tasks which we modulated the degree of alignment between the geometry of the task inputs and that of the task labels. We analyzed the learning dynamics weight space and show how the differences between the networks with Tanh and ReLU nonlinearities arise from the asymmetric asymptotic behavior of ReLU, which leads feature neurons to spe- cialize for different regions of input space. By contrast, feature neurons in Tanh networks tend to inherit the task label structure. Consequently, when the target outputs are low dimensional, Tanh networks generate neural representations that are more disentangled than those obtained with a ReLU nonlinearity. Our find- ings shed light on the interplay between input-output geometry, nonlinearity, and learned representations in neural networks.
INTRODUCTION
The geometric structure of representations learned by neural networks sheds light on their internal function and is key to their empirical success. The ability of networks to adapt their representa- tions to capture the structure of training data has been shown to improve their ability to generalize (Atanasov et al. 2021 Yang & Hu 2020; Baratin et al. 2021) and to make effective use of increas- ing dataset sizes (Yyas al. 2022). Moreover, representations learned from data essential to the success of transfer learning between tasks (Neyshabur et al., 2020)).
Representational geometries dynamically evolve during network training and arise from interac- tion between the structure of the inputs provided to the network, the outputs it is trained to produce, and the network architecture. In this study, we conduct in-depth investigation of the impact of input geometry, label geometry, and nonlinearity on learned representations. We employ param- eterized family of classification tasks that allows independently and focus on single- seggeehhiddn--layee layer networks in which we can precisely describe repre- sentation learning dynamics over the course of training.
probe the impact of each of these factors
RELATED WORK
Prior work has observed that hidden layer representations tend to increasingly reflect the geometry of the task labels during training (Atanasov et al. 2021 Fort et al. 2020) and that this phenomenon implicitly regularizes neural network training. Theories have been developed to explain this phe-
nomenon in linear networks (Atanasov et al. 2021 Shan & Bordelon 2021). However, the impact
"quull contribution

---

Published as a conference paper at ICLR 2024
of nonlinearity learned representational geometry is comparatively poorly understood (though see (Sass et al.] (2022); ; Chizat & Bach (2020). Moreover, learned network representations must extract structure besides label structure explain the success of transfer learning (Neyshabur al. 2020).
Many authors have studied the impact of different choices of neural network activation function. Most of the theoretical and empirical work on the subject has focused on the effect of activation function choice network training dynamics and performance (Hayou al. 2019 Ding et al.)
2018 Ramachandran et al. 2017). By contrast, in this work, we focus on how the activation function shapes learned representations tasks where performance is high.
Our work relates to studies on "neural collapse." (Papyan et al. 20200) Kothapalli et al. 2022) a phe- nomenon often observed empirically in which prolonged training causes final-layer representations in deep networks to "collapse" to represent only the label information. Prior theoretical work on the subject shows that neural collapse emerges from gradient descent dynamics an "unconstrained features" parameterization in which the final layer network responses to each datapoint are opti- mized as free parameters (Zhu et al., 2021 Jiang et al. 2023). However, the parameters fit during network training are the weights of the network, and the network architecture, activation function, and input data geometry impose constraints on how network responses can be transformed across layers. Our work sheds light how these constraints impact the propensity for neural collapse.
3 MEASURES OF REPRESENTATIONAL GEOMETRY
In this work, we characterize learned representational geometry in number of ways. The tasks we consider involve mapping inputs, which are generated from small set of binary latent variables, to one (single-output case) or several (multi-output case) binary labels.
First, we track the linear decodability (using an SVM) of different labelings of these clusters from the network representation, including those the network was trained on and those it was not. The discrepancy between decodability of trained vs. untrained labelings measures the extent to which the network preserves rich information about the inputs, or discards all but the label information.
Second, we use kernel alignment metrics, commonly used in assessing the similarity two neural network representations (Cristianini et al.) 2001 Kornblith et al.) 2019 Kriegeskorte et al. 2008). For two mean-cenie centerc representations of a set of d data points X1 E I and X2 E I2 xa we may define corresponding kernel matrices K2 X2 X1, K₂ = X3 X2 Idddd Then, the kernel alignment between these representations defined as
TT(K,22) C(Ki, (T((K, Tr(K2Kg)
Concretely, this corresponds to the entry-wise correlation coefficient between the kernel matrices. For inputs, X, output labels Y, and hidden representations Z, there are two alignment values measure: the 'target alignment' C(Kz Ky), the 'input alignment' C'K2,, Kx). We also vary the 'unput-output alignment' C(Kx, Ky) of our tasks.
Third, we measure the parallelism score (PS) of the target labels in the representation, a measure of disentanglement that indicates which given feature is encoded in the same way regardless of the value of other input features (Bernardi et al. 2020). Concretely, in task with 2 inputs generated by l underlying binary variables //1 ... lk. the parallelism score of representation for variable y: is computed as follows. First, we condition on set of values of the other k - 1 factors. Then, we take the vector that separates the mean value of the representation when y: = +1 and the mean when y: -1. This process is repeated for all 2* possible conditioned values of yj and the average pairwise cosine similarity between the resulting vectors is computed. Parallel coding directions of task-relevant quantities allow for better few-shot generalization and signature of disentangled abstract representations Higgins et al. (2018); Sorscher et al. (2021).
Finally, measure cross-conen coss-connition generalization performance (CCGP) (Bernardi et al. 2020), which measures the extent to which a linear classifier trained to categorize a restricted set of inputs will generalize accurately to categorizing unseen, out-of-distribution inputs. CCGP for quantity
2

---

Published as a conference paper at ICLR 2024
A
B
Decoding
(labels)
Decoding (untrained labelings)
Parallelism
Output
+1
RetU
Hidden layer
Target alignment
Input alignment
CCGP
Input aligneent
Input
0.5
40k
Training steps
Training steps
Training steps
C
Input space
ReLU gradient
Tanh gradient
X
Tanh
ReLU
Inter-class axi
inter-class
ww
Intraediase
Intra
X4
X4
Intra-class axis
Figure 1: A. Schematic binary classification task with unstructured inputs. B. Measures of rep- resentational geometry during training. Error bars indicate standard deviation 20 simulated networks. C. Schematic illustrating the inter-class axis and and intra-class axis (left) and the pro- cedure for computing the expected gradients of the task loss with respect to the input weights, projected along these axes (right two panels). The derivative f' of the activation function is shown by the shading of space, and the vector यī indicates the current value of the input-layer weight being considered. In this example, in the ReLU case, only the 222 data point contributes to the gradient (red arrow). In the Tanh case, the gradient (dashed arrow) receives contributions from all four data points (colored arrows). D. Trajectories of input weights to hidden layer neurons along the inter-class and the intra- axes. Each line segment represents an individual neuron from simulation, and small circles indicate the initial conditions. Vector field indicates the gradient of the task objective.
of interest is high when the representational axes that encode that quantity are relatively unaffected by additional information about the the input, a property which enables few-shot learning [Sorscher et al. (2021); Lindsey & Issa (2023); Johnston & Fusi 2023). Concretely, in a task with binary feature structure as above, compute CCGP for a variable y4+ we fix setting of all values yizi and fit decoder for y/: on this subsampled dataset. Then we evaluate the average performance of other settings of y/jzs This quantity is averaged over the 2 possible choices of the setting of リı used for decoder training.
the decoder on all 2
See Appendix for an example that provides further intuition for the CCGP and PS metrics.
REPRESENTATIONS INDUCED BY BINARY CLASSIFICATION OF UNSTRUCTURED INPUT PATTERNS
To begin, we considered the following simple classification task. Four points Ii, ... 24 N corresponding to prototypical inputs ("cluster centers"),, were sampled randomly from a unit normal distribution. Then, individual samples were drawn from normal distributions centered the x; with variance x² (set to 1.0 throughout the main text, but see Section 5.2). We trained networks with single hidden layer map samples from the first two clusters (x) and x2) to y and from the last two clusters (I3 and st4) to y (Fig- [A],, with cross-entropy loss. For ease of subsequent binary +1 values (all qualitative effects are robust this choice, see Appendix B). This task is designed to assess how much the task structure (gcometry of the outputs) imposes itself on the network's 's hidden layer representation through learning.
analysis, we train only the first-layer weights and freeze the second-layer weights
We found the choice nonlinearity strongly affects the learned geometry. In particular, Tanh net- works learned representations that reflected the low dimensional geometry of the targets (high target alignment, parallelism score, and CCGP), while ReLU networks learned representations that more faithfully preserved the geometry of the input clusters (high input alignment and ability to decode
3

---

Published as a conference paper at ICLR 2024
untrained labelings of the input points) (Fig. [B). Notably, in both kinds of networks, the ability to decode the class labels (the "trained dichotomy") from the network representation was high and increased throughout training. We also measured the ability to decode classes the network was not trained on. Decoding performance for such "untrained dichotomies" was high for both networks, but over the course of training, it increased in the ReL.U networks and decreased in the Tanh networks.
4.1 ANALYSIS OF LEARNING DYNAMICS
4.1.1 METHODS
We next sought to understand our results by analyzing the learning dynamics of the input weights of hidden neurons. To visualize learning dynamics, we made several simplifying assumptions. As mentioned previously, we fixed the output weights of the network throughout training to discrete values, such that learning takes place only in the input weights to the hidden layer. We discretized the distribution of output weights, such that the neurons can be categorized into groups with identical output weights. Furthermore, we make an assumption about the error statisties during training, namely that task performance is approximately equal across items (e.g. the network is just as likely correctly output the labels of input I as it is for input 2). Another equivalent description that the error matrix, Y Y, has constant direction and only changes in magnitude. This assumption is reasonable, given the symmetry of the tasks we consider.
Under these assumptions, we may describe the learning dynamics of the input weights, W. of particular hidden neuron with activation function f and frozen output weights War as follows:
Au A
Σ wolc () I.) Σ Eff 2.)
where i indexes the training examples, Ús is the output of the readout layer with sigmoid nonlin- earity applied) for training example and we have grouped together wo(y - û.) = Note that the evolution of to depends only on own state, and not that of other hidden neurons. Furthermore, each hidden neuron with the same output weights (i.e. those belonging to the same "group") will be subject to the same dynamics. These simplifications allow us to generate a vector field describing network learning dynamics by plotting the Au vector for an arbitrary choice of wo.
We visualize the gradients in a two-dimensional space defined by the inter-class axis and an intra- class axis. The inter-class axis is equal to the covariance between input and output: E, y, In the 4-input, binary classification task under consideration, it corresponds to x1 + H2 23 T4- Intra-class axes are orthogonal to the inter-class axis, and capture differences between inputs of the same label; in this case, the intra-class axes are # - -22 and z3 - I4. The choice which intra-class axis is most useful to visualize depends on the neuron being considered: for neuron with positive output weight, the interesting dynamics take place in the Ø1 22 axis since the neuron maintains approximately zero selectivity for Iy and 3.4. Note that weights in linear network would evolve only along the inter-class axis; dynamics along the intra-class axis are consequence of nonlinearity. See Fig. TC for schematic illustrating these computations.
For a neuron with positive output weight wo the average gradient of the task loss L with respect to that neuron's input weights w is sum of multiple components, each corresponding cluster (Equation]] D The terms of this sum are vectors aligned with the corresponding input Z. weighted by the sign of the label of F,, and the value of f' (((w"r F,) for the given value of ui evaluated at the E. this weighting is indicated by the blue shading in Fig. [C. In the ReLU case, f is when and 7 are positively aligned and zero otherwise. As a result, the gradient tends to push vi further in the direction of inputs H for which it is already selective. In the Tanh case, the gradient pushes towards inputs with the positive label (or away from inputs with the negative label) to which that neuron is neither strongly selective nor anti-selective. This has the effect of dampening strong within-class selectivity for Tanh neurons.
an input
4.1.2 LEARNING DYNAMICS
As a result of the dynamics described above, in Tanh networks, all neurons grew increasingly aligned with the inter-class axis (Fig. D left). ReLU networks instead exhibited heterogeneity across neu-
4

---

Published as a conference paper at ICLR 2024
A
0.5
B
Input alignmens
Target alignment
0.8
Tanh ReLU
Tanh
Inter- Inter-chist closs
0.2
ReLU
tanh
14
Parallalism elism
Tanh
ReLU
Inter class
OC0P
ReLU
0-
ReLU
1.0
0.0
0.5
1.0
Intra-class
Intra- class
Intra- dlass
Separability (8)
Separability (8)
Figure 2: A. Schematic of binary classification task with input structure parameterized by 8, a factor indicating the degree of separability of the two classes (green and magenta clusters). B. Measures of representational geometry following network training as a function of ó. Error bars indicate standard deviation over 20 simulated networks. C. Trajectories of input weights to hidden layer neurons as in Fig. DDD for different values of 6.
rons, with some growing aligned with the inter-class axis and others developing intra-class selectiv- ity (Fig. DD, right). To understand this difference, we looked at the dynamics of gradient descent. In Fig. ID we plot the vector field defined by projecting the expected gradient of the task objective with respect to input weights onto the inter- and intra-class axes, as described above. We find that weights in Tanh networks predominantly evolve direction of increased inter-class selectivity and decreased intra-class selectivity, independent of their initial conditions. Weights in ReLU networks are driven to accentuate the selectivity they possess in their random initial condition.
The different dynamics are explained by the degree of symmetry of the nonlinearity of the activation function: the one-sided saturating behavior of ReLU causes some inputs to be encoded in the weights (those that bring the neuron above threshold) and others to be ignored (below threshold). This breaks the symmetry between the input items that would share the same gradient in the absence of the nonlinearity, leading to specialization and elevated intra-class selectivity. This symmetry breaking is less likely in Tanh neurons, given that Tanh symmetric (both around the origin, which governs dynamics at initialization, and asymptotically). We explore the relative importance of the behavior of the nonlinearities at initialization vs. their asymptotic behavior in Section
EFFECT OF INPUT GEOMETRY ON LEARNED REPRESENTATIONS
5.1 EFFECT OF SEPARABILITY TARGET OUTPUTS
We generalized the previous task by parametrically controlling the geometric arrangement of the cluster centers T1, ..., 2z, rather than sampling them randomly. Specifically, the input geometry was parameterized by a scalar quantity 5 corresponding to the degree which the classes were linearly separable in the input space. The = 1 case corresponds to the previous task, in which the clusters are equidistant, and hence the trained dichotomy is easily decodable (Fig. 2A, right). In the 8 = O case, the clusters were arranged on a two-dimensional square such that Iı and x2 (and Ty and xz4) were positioned on opposite corners of the square (Fig. 2A, left), which is an XOR task and requires a nonlinear transformation of the inputs. Intermediate values of 8 interpolated between these two extremes (Fig. 2A, middle). Note that this construction can be regarded as varying the input-output kernel alignment of the task (low å corresponds to lower alignment).
In the full XOR task (6 = 0), no matter the nonlinearity used by the network, a representation emerged in the hidden layer with low target alignment, parallelism score, and CCGP (Fig. 2 This finding is unsurprising, as in this task, a disentangled representation of the output geometry cannot be obtained by one neural network layer. For intermediate values of 8, however, it is not obvious to what extent the network will leverage the linearly separable component of the inputs for disentanglement. We found that Tanh networks, for any values of 8 greater than small threshold, form representation in which the geometric structure of the inputs was largely discarded in favor of the binary output structure (Fig. 23). For ReL.U networks, the alignment of the representation with
5

---

Published as a conference paper at ICLR 2024
the output structure increased more gradually as å varied from 0 to 1, and strong signatures of the input structure remained in the learned representation regardless of the value of d.
To understand these effects, again analyzed the evolution of the input weights with learning, focusing as before on the inter-class axis and intra-class axes. We found that for d = 0, only intra- class selectivity emerged for all neurons in all networks, which is unsurprising the inter-class axis in this case is degenerate ((x; + 22) (#3 + 244) = 0). For any nonzero values of 6, Tanh neurons almost uniformly developed inter-class selectivity, while ReLU neurons evolved in heterogeneous fashion, with the proportion of intra-class neurons decreasing with ò (Fig 2C).
5.2 EFFECT OF INPUT NOISE
We also assessed the impact of input noise on representation learning (see Fig. 3 To facilitate fair comparison of learned representations across networks, we introduced a distinction between degree of input noise Ctrain and deet used during training and during analysis of learned representations, respectively. The value of test was fixed at 1.0 for all analyses, and the value of Otrain varied. We found that Tanh networks exhibit sharp transition in learned representation, from abstract representations to input of noise at which this transition occurs is related to the separability of the trained dichotomy; for high values of separability, learned abstraction is robust to higher levels of noise during training.
representations, as training noise increases. The level
Deczding accunacy (taired)
Decoding acsunacy (untinines)
CCGP
Parsiletem
Input al aigrment
Target alignment
1,0
Tanh
0.5
0.0
0.5
-1.0
0.0
1.0
1.0
1.0
tolied dicertity
ReLU
0.0
-1.0
0.0
0.5
0.125
2.0 Training noise
0.5
0.125
2.0 8.0
0.125
2.0
2.0 Training noiso
0.5
Training noise
Training nosso
Training nolso
Training noise
Figure 3: Values of various representational metrics for different values of 5 (separability of trained dichotomy) and x² (training noise) in the 5-separable classification task of Section 5
The effect of increased noise in Tanh networks is to induce more "ReLU-like" representations that vary gradually as a function of the input geometry. By comparison, learned representations in ReLU networks are impacted much less by training noise. This behavior makes sense, given the analysis in the previous section revealing that Tanh networks are dramatically affected by the separability of the target outputs in input space, while ReLU networks are less sensitive to the degree of separability. Increasing the degree of input noise has the effect of increasing the degree of separability required for Tanh neurons to reliably tune to the component of input space that correlates with the label. When separability is insufficiently strong given the noise level, Tanh network behavior grows more similar similar to the 8 = O case. Interestingly, we also observe an increase in target alignment and a decrease in input alignment for modest values of input noise in ReLU networks. We explore this phenomenon further in Appendix E.
GENERALIZING ANALYSIS TO MORE COMPLEX TASKS
So far, we studied in depth a set of tasks involving just four input clusters. Now we check the insights derived from these simple tasks hold more broadly. In the 8-separable XOR task, increasing the separability parameter makes the input geometry more similar to the output geometry, resulting in higher target alignment in the hidden layer. We generalize this construction assess more gen- erally how the kernel alignment between input and target patterns determines hidden layer geometry. To do so, we use the following procedure sample tasks with specified input-output alignment.

---

Published as a conference paper at ICLR 2024
We parameterize family of tasks with P randomly placed input clusters and k P binary output targets, in which we control the alignment between input and target kernels (Fig 4A). To do so, we first draw k random but balanced binary target classes, Y. Then, for specified alignment value e, we draw a random input kernel, Xx. from the set of all symmetric positive definite matrices such that C(Kx, Ky) c, where Ky yTy. One element of this set is special, and it is the Kx with the flattest cigenspectrum, i.c. the maximal linear dimensionality. The solid lines in Fig. E use this lakimaal-dimeniional input geometry, while the dots use other random draws with lower dimensionality. With Kx in hand, there are many ways we can generate N-dimensional input patterns X, and we use X OΛLUT, where U and A are the eigenvectors and eiegenvalues, respectively, of Kx, and O is random N P orthonormal matrix. We end up with set of random inputs X, and output labels Y, with a centered kernel alignment of exactly
In Fig 4B, we plot our measures of representational geometry for = 8 and 32 inputs, varying k, the number of targets, from 1 to P - 1. As in the simple case of Section S the target alignment always increases more dramatically for Tanh networks compared to ReLU as input-output alignment increases. Moreover, expected, the parallelism score increases when the target geometry is low- dimensional and decreases when it is high-dimensional, and hence the target geometry itself has low parallelism. Note that the effect of multi-dimennsioal outputs is explored in more depth in case study (A: = 2) in Appendix A. The behavior of CCGP mostly matches that of parallelism, but there are several cases where a large difference in parallelism not reflected by large difference in CCGP, particularly in cases with relatively few outputs, where the decoding task measured by CCGP may be easier.
7 PHENOMENA IN MULTI-LAYER NETWORKS
We wished to see whether our findings on simulated tasks generalize at all to networks with more than one layer. To address this, we chose one of the tasks from Figure 4 (that with P = 32 inputs and = targets) and trained networks with 5 or 10 hidden layers. We focused on tasks in which the out- puts were nonlinearly separable in imput space, real-world tasks require dealing with nonlinearly separable inputs, and such tasks are most likely to expose differences between deep and shallow networks (target-aligned representations cannot be produced in single-layer for nonlinearly sepa- rable targets). We evaluate on two tasks, varying in their difficulty; intuitively, the difficulty is the degree to which the output labels are nonlinearly entangled in the input space (see Appendix F for a more precise explanation). The target and input alignment evolve as expected along the layers of the network, with target alignment increasing and input alignment decreasing (Fig. [5). The effects of non-lineant non-lineariit and task difficulty are also consistent with our results from shallow networks deep tanh networks learn more target-aligned representations than deep relu networks in their final layer, especially for the more difficult task. We also observe interesting phenomena with respect to the progression of target input alignment across layers of the network in the easy vs. hard task, which we leave to future work to investigate in more detail.
CONVOLUTIONAL NETWORK EXPERIMENTS
To assess the applicability of our findings more realistic tasks, we trained convolutional networks image classification task, experimenting with two architectures - a small network with two convolu- tional and two fully connected layers, and the ResNet-18 architecture- and two datasets, CIFAR-10 10 and STL-10. 10. To enable computation of CCGP and parallelism score, we modified the tasks slightly so that the 10 classes in the dataset were assigned to 5 labels, grouping together pairs of classes. This allowed us to treat the input classes analogously to the input clusters in the simulations above for the purpose of computing CCGP and PS. Kernel alignment and test accuracy were also computed. In all cases, we observe the same qualitative dependence of all these metrics on the choice of activation function (though the effects vary magnitude) (Fig
ISOLATING THE ROLE OF ACTIVATION FUNCTION ASYMMETRY
Finally, we sought to identify the source of the different representations learned by Tanh and ReLU networks. We hypothesize two candidate mechanisms: the symmetric saturation of the Tanh func-

---

Published as a conference paper at ICLR 2024
A
Sample targets Y
Sample aligned input kemel
Generate inputs
X X = Kx
Ky
0.2
0.8
KiKy
B
Target geometry
Target alignment
Parallelism
CCGP
P
P
32
8
P
32
8
P 32
1.0
segment
0.4
1anh
0.5
3.0
0.3
0.00
0.15
0.0
0.3
0.00
0.15
0.2
0.00
0.15
k leg;
1.
1.0
eypercobe
0.4
0.0
0.0
0.4
0.0
0.6
0.0
kis P-1
1.0
1.0
0.4
6.0
0.5
Marrass
simplex
0.7
1.9
0.4
6.7
1.0 0.4
1.0
Inout output aligmment
Input output alignment
Input output aligrmant
C
Test accuracy
Target alggmment
Input aligrment
Paraliclism
CCGP
2 layer conv layer FC,
0.50
ReLU
0.2
0.50
0.6
CIFAR-10
0.0
500
500
layer conv +
0.3
0.50
0.7
STL-10
layar FC,
0.28
ReLU
0.2
0.6
ResNet-18
0.85
0.4
CIFAR-10
0.80
0.58
ReLU
0.26
0.2
0.5
0.8
0.00
0.0
500
500
Training spochs
Training apochs
Training opachs
Training spochs
Training spochs
Figure 4: A. Cartoon of the random sampling process, illustrated for P = inputs and k: = outputs. B. Target alignment, PS, and CCGP as functions of input-output alignment in random classification tasks, for different values of P (columns) and (rows). Cartoons schematize the target geometry for each value of (the number of target dimensions). In the plots, solid lines the unique maximum-dimensionn dimensional input geometry for specified alignment, and dots are 12 random samples of other lower-dimensional geometries. All tasks have training noise variance of C. Metrics in the final layer of a convolutional network trained on CIFAR10. Error bars are standard errors over random initializations.
ReLU
Tanh
S-layer
10-layer
S-layer
10-layer
Easy task
Tirpet
Tarptt
Karnel liigmmentt
Hord Taok
Sugert
0.2
Layar
Layer
Layer
Layor
Figure 5: Target and input kernel alignment for the representations at each hidden layer in multi- layer networks. Each network is trained until convergence. The inputs generated in Fig. 4 with the addition of a constraint on the dimensionality the inputs for the 'hard' task (see Section and Appendix F), All tasks have training noise variance of 1.
8

---

Published as a conference paper at ICLR 2024
A
Symmetric saturation
B
Linearity at origin
Tanh
k ==1
k =
k
=
1
k k
5
ReLU
ReLU
ReLU1
1.0
shift shift
1.0
0.6
0.6
0.0
0.1
0.00
0.25
0.0
0.1
0.00
0.25
Figure 6: Activation function perturbation experiment. The target alignment shown, as in Fig. for P = 32 inputs. A. Adding positive saturation to the ReLU function for arguments > 1. B. Shifting the ReLU function negatively (reddish) or positively (greenish) in the argument.
tion, and its linear behavior close to the origin. Our analysis of the gradients in the simple task of Section 4 suggests that the degree of symmetry in the asymptotic behavior of the nonlinearity is im- portant since it prevents individual neurons from developing selectivity for one input over another. However, it is also plausible that the behavior of the nonlinearity locally around the origin matters most if networks learn solutions that do not deviate much from their initialization. To differentiate between these hypotheses, we construct nonlinearities with symmetric asymptotic saturating behav- ior but asymmetric behavior around the origin (Fig. Gh. f(x) = min (ma((x, O), 1)), or asymmetric saturating behavior but linear behavior around the origin (Fig. 6b. f(x) max(x+b, O), b<0).
We find that networks using a nonlinearity with two-sided saturation (Fig. (b)) behave almost iden- tically to Tanh networks despite asymmetry around the origin. Recall from Fig. [ D that in the Tanh case, the two-sided saturation of the nonlinearity prevents neurons from growing overly se- lective or anti-selective for particular inputs with a given label, as when this occurs, the value of j' evaluated at those inputs is low, dampening the magnitude further weight updates aligned with that input direction. Qualitatively, the same phenomenon occurs in the gradients of any nonlinearity that saturates in both the positive and negative directions.
We also tried perturbing the offset of ReLU make it linear and symmetric around the origin with- out modifying its asymptotic behavior. We found this has modest effect on learned representational geometry, leading to more target-aligned representations when the linear region of the ReLU nonlin- earity contained the origin (Fig. op). This makes sense, linear or otherwise symmetric behavior of the activation function derivative I' around the initialized value of the input weights should result in initial evolution of input weights along the inter-class axis with learning, until they reach region of weight space where f' is asymmetric with respect to the inputs Ā.
We conclude that activation function behavior around the origin influences learned solutions, but symmetric asymptotic saturating behavior exerts a powerful influence towards target alignment.
10 CONCLUSIONS AND DISCUSSION
The geometry of learned neural representations combines the structure present in inputs and target outputs, influencing a network's task performance and ability to generalize to new data. Here, introduced framework for modeling the joint structure of task inputs and outputs, and studied how neural representations reflect these structures. Surprisingly, the activation function plays an important role in determining the alignment between the learned representational geometry and the target geometry, with Tanh typically leading to more disentangled representations of the structure of the labels than ReLU. These differences in learned representations trade off different benefits. Disentangled representations are compact Ma et al. (2022). allow for generalization and compositionality, and have been shown to improve adversarial robustness (Willetts et al.) 2019 Yang & Hul 2020 Papyan et al.) 2020). However, they are inefficient representations for storing memories and for binding together information about multiple variables Boyle et al. (2022): Johnston et al. (022)). Moreover, learning disentangled representations of the label structure may impair the ability to transfer learning to tasks with different semantics. Our work sheds light on the aspects of network architecture and task structure factors that are important in navigating this tradeoff.

---

Published as a conference paper at ICLR 2024
11 ACKNOLEDGEMENTS
This work was supported by NSF NeuroNex Award DBI-1707398, The Simons Foundation, The Gatsby Foundation (GAT3708), the Swartz Foundation and the Kavli Foundation. JL was also supported by the DOE CSGF IDE-SCC020347) The authors declare no competing interests.
REFERENCES
Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect. arXiv preprint arXiv:2111.00034 2021.
Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vin- cent, and Simon Jalisne-Jllien. Implicit regularization via neural feature alignment. Interna- tional Conference on Artificial Intelligence and Statistics, pp. 2269-221111 PMLR, 2021.
Silvia Bernardi, Marcus K Benna, Mattia Rigotti, Jérôme Munuera, Stefano Fusi, and C Daniel Salzman. The geometry of abstraction in the hippocampus and prefrontal cortex. Cell, 183(4); 954-967, 2020.
Lara Boyle, Lorenzo Posani, Sarah Irfan, Steven A Siegelbaum, and Stefano Fusi. The geometry of hippocampal ca2 representations enables abstract coding of social familiarity and identity, bioRxiv and in press in Neuron, pp. 2022-01 20,2-11,20 2022.
Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference Learning Theory, pp. 1305-155 1338. PMLR, 2020.
Nello Cristianini, John Shawe-Taylor, Andre Elisseeff, and Jaz Kandola. On kernel-target alignment. Advances in neural information processing systems, 14, 2001.
Bin Ding, Huimin Qian, and Jun Zhou. Activation functions and their characteristics deep neural networks. In 2018 Chinese control and decision conference (CCDC), pp. 1836-1841. IEEE, 2018.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 10-5880--861,,0 2020.
Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the impact of the activation function deep neural networks training. In International conference on machine learning, pp. 2672-2680. PMLR, 2019.
Irina Higgins, David Amos, David Pfau, Sehastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations, arXiv preprint arXiv:1812.02233, 2018.
Jiachen Jiang, Jinxin Zhou, Peng Wang. Qing Qu. Dustin Mixon, Chong You, and Zhihui Zhu. Generalized neural collapse for large number of classes. arXiv preprint arXiv:2310.05351, 2023.
W Jeffrey Johnston and Stefano Fusi. Abstract representations emerge naturally in neural networks trained to perform multiple tasks. Nature Communications, 14(1):104) 2023.
W Jeffrey Johnston, Justin M Fine, Seng Bum Michael Yoo, R Becket Ebitz, and Benjamin Y Hay- den. Semi-orthogonal subspaces for value mediate a tradeoff between binding and generalization. arXiv preprint arXiv:2309.07766, 2023.
Simon Korbllihh, Mohammad Norouzi. Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, 2019.
Vignesh Kothapalli, Ebrahim Rasromani, and Vasudev Awatramani, Neural collapse: review on modelling principles and generalization. arXiv preprint wrXiv:2206.0404/ 2022.
10

---

Published as a conference paper at ICLR 2024
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis- connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:4, 2008.
Jack W Lindsey and Elias B Issa. Factorized visual representations in the primate visual system and deep neural networks. bioRxiv, pp. 2023-04, 2023.
Yi Ma, Doris Tsao, and Heung- Yeung Shum. On the principles of parsimony and seff-consistency for the emergence of intelligence. Frontiers of Information Technology & Electronic Engineering, 23(9): 2022.
Behnam Neyshabur, Hanic Sedghi, and Chiyuan Zhang.
What is being transferred in transfer learn- ing? Advances in neural information processing systems, 33:512-523, 2020.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020.
Prajit Ramachandran, Barret Zoph, and Quoe Le. Searching for activation functions. arXiv preprint arXiv: arXiv:1710.05941, 2017.
Justin Sahs, Ryan Pyle, Aneel Damaraju, Josue Ortega Caro, Onur Tavaslioglu, Andy Lu, Fabio Anselmi, and Ankit B Patel. Shallow univariate relu networks as splines: initialization, loss surface, hessian, and gradient flow dynamics. Frontiers in artificial intelligence, 5:889981, 2022.
Haozhe Shan and Blake Bordelon. A theory of neural tangent kernel alignment and its influence on training, arXiv preprint 1rXiv:2105.144 2021.
Ben Sorscher, Surya Ganguli, and Haim Sompolinsky. The geometry of concept learning. BioRxiv, pp. 2021-03, 2021.
Nikhil Vyas, Yamini Bansal, and Preetum Nakkiran. Limitations of the ntk for understanding gen- eralization in deep learning. arXiv preprint ar arXiv:2206. 100/2, 2022.
Matthew Willetts, Alexander Camuto, Tom Rainforth, Stephen Roberts, and Chris Holmes. Improv- ing vaes' robustness to adversarial attack. arXiv preprint arXiv:1906.002300 2019.
Greg Yang and Edward J Hu. Feature learning in indtnite-width neural networks. arXiv preprint arXiv:2011.14522 2020.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A ge- ometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820 29834, 2021.

---

Published as a conference paper at ICLR 2024
A
EFFECT OF MULTIDIMENSIONAL OUTPUT ON LEARNED REPRESENTATIONS
A
B
Oulpul
0.50
40
-1
Hidden
ReLu
Input
Decoding
(braned)
Decoding
(untrained)
CCGP (rained)
CCGP (uniraned)
lrgut algoment
Target algrmennt
Group +/+
Group +/0
Creup
meer-dlass axis
Inter- class axis
Tanh
Intra-class axis
Intra-olass axis
Inter- -wa--classs 8oi8
ReLU
Intra-class axis
Intra-class axis
Figure 7: A. Schematic of binary classification task with unstructured inputs and two outputs. The outputs trained to classify four input clusters according to different labelings of the inputs. In this case the hidden layer of the network is divided into eight groups of neurons defined by their two output weights (positive: blue, negative: red, zero: no line). B. Measures of representational geometry at network initialization (left of each line segment) and following training (right of each line segment). Error bars indicate standard deviation over 20 simulated networks. D. Alignment (dot product similarity) of imput weights to hidden layer neurons with the four input cluster centers. Shown for four of the eight hidden layer neuron groups. Error bars indicate standard deviation across neurons within single network. E. Trajectories of input weights to hidden layer neurons along the inter-class (Ij + x22 x3 x4) and the intra-class axis (x) x2). Each line sigment represents an individual neuron from simulation, and small circles indicate the beginning of the trajectory (at network initialization). Shown for hidden layer neurons with an output weight of +1. Vector field background indicates the expected average direction that weights will evolve due gradient of the task objective.
Here include in-depth exploration of networks trained with two targets and four items. We modified our tasks by now assigning each of the four input clusters to two different binary labels: (1, 1), (1, 0), (O, 1), and (0. 0). respectively (Fig 7). We modified the network architecture by adding a second output unit; each output unit was tasked with predicting one of the two labels. Thus. in this task, there were two trained dichotomies (21/22 vs. 03/F4, and x 01/23 vs. /1/x44 and one untrained dichotomy (#1/xa vs. F2/D3)). For analysis, we again used frozen readout weights dis- cretized into eight groups (Fig 7A), which we found was sufficient to approximate the behavior of networks trained with randomly initialized output weights. Again, we found a noticeable difference in the learned representations of ReLU and Tanh networks, with Tanh networks reflecting the ge- ometry of the targets more than ReLU networks (Fig 7B). Again, analyzed the source of these phenomena by tracking the alignment input weights to each neuron with the four input cluster centroids. In general, observed that neurons grew positively tuned inputs which matched their
12

---

Published as a conference paper at ICLR 2024
output weights, negatively tuned to imputs mismatched with their output weights. and not tuned all to imputs matched with one output weight and mismatched with the other (Fig. 7C). However, in the ReLU networks, clear asymmetry emerged in the degree of positive tuning to matched imput clusters and the degree of negative tuning to mismatched input clusters. As a result, in the ReI.U networks, roughly half the neurons (those with nonzero outputs both class readout units) devel- oped selectivity primarily for one of the four input clusters. As a result, these ReLU units developed strong selectivity for intra-class differences, while Tanh units did not (Fig. 7D). This behavior was observed to be uniform across the hidden neurons (Fig. (D).
Thus, both the multi-output task produced similar results the similar output task in terms of representational geometry, the source of these effects was somewhat different. In both cases, the gradient of the task objective drives input weights accrue inter-class selectivity Tanh networks. However, for ReLU networks, the picture is different. In the single-output case, the emergence of intra-class selectivity arises due heterogeneity initial conditions of input weights, while in the multi-output case it also arises due to heterogeneity in initial conditions of output weights.
B IMPACT OF CONSTRAINTS ON THE OUTPUT WEIGHTS
Decoding accuracy [trained]
Decoding accuracy (untrained)
Input alignment
Target allgnment
1.00
1.00
1.0
Parallelism
1.0
1.0
sen Hood readocts
0.95
Tost dicretis readects
0.90
T0k 0.90
abk
0.5
40xc
0.0g
40x
0.0%
4bk
Aall yoas madocnt
fleed roddesti
Training steps
Training steps
Training stops
Traninng stops
Training stops
Figure 8: Measures of representational geometry during training for three cases: randomly ini- tialized trainable readouts, randomly intiialized fixed readouts, and discretized (binary +1) fixed readouts.
In experiments used frozen output weights with discretized values. In Fig. we show that the behavior of representation learning is essentially the same when we relax the discretization and allow the readouts to be trained.
C FURTHER DETAILS ON RANDOM ALIGNED KERNEL SAMPLING
Our sampling procedure for tasks with a specified input-output alignment used in Section 5 has some important considerations, which will be further elaborated in this section. Firstly, obviously, the sampled matrices must be symmetric and positive semi-definite (SPSD), and they need to have the required correlation value. Secondly, kernels with the same alignment value can have different dimensionality (i.e. cigenspectra).
It is worth clarifying that assume all kernels are already centered, meaning that the features used to compute them have been mean-subtracted. Or, equivalently, that their nullspace contains the vector 1 of all ones. They should also have unit trace, TrK 1. Both non-restrictive assumptions, since the CKA is invariant to scaling and shifting of features.
The set of matrices with given correlation value, c, to reference matrix Ky is the solution set of a quadratic form while the SPSD matrices (of unit trace) are a compact convex body. We will refer this set as Keg Our procedure is randomly sample SPSD matrix, and project it onto the quadric surface, taking care to remain SPSD. This method certainly does not ensure uniformity, but is simple and empirically shows wide spread of samples even in relatively high dimensions.
An important consideration with this method is the linear dimensionality of the input geometry, intuitively a measure of correlations between input patterns. we define dimensionality in terms of the participation ratio of the cigenvalues, X, of K:
12 A,) p.r.(K) Σ, X2
'C(Kx Kv) = c - &T((K T
13

---

Published as a conference paper at ICLR 2024
Cross-condition generalization performance (CCGP)
Parallelism score (PS)
Fit clasoifier xa semples where xi
Glaseifiar genoralizes samplee where Xi -> high CCGP
10.11
11,11
encoding conditioned on
vector encoding X2
conditioned on
VO,
(1.0)
19, Dj
[1.
and
are parallel -> high parallelism score
01.
Find vector encoding conditioned
(1, 1)
Cleesiler does not generalize somples where
(1,
0)
and
are
Fil classilier for
10, OJ
not paralial >> low parallolism score
1J
Find encoding conditioned on Xi 0
1)
Figure 9: Examples of representational geometries with high and low CCGP and PS, for data gen- erated by two underlying binary variables, i.e. data points of the form (I1, x2) where a; E {0, 1}.
then, per assumption of unit trace, this is just |||] In the special case when 0 the set Ko is convex, and this implies that there is unique kerne matrix Kmax which maximises p.r.(K), i.e. minimises | F. If we draw line between Ky and maxx, get the of input kernels which form the solid lines of Fig. 4B and Fig. 5 of the main text.
D EXPLANATION OF CCGP AND PS METRICS
See Figure 9
E COMPARISON OF WEIGHT DYNAMICS AT DIFFERENT NOISE LEVELS IN RELU NETWORKS
In Section 5.2 observed in ReLU networks interesting increase in the values of target align- ment / parallelism / CCGP, and decrease in the value of input alignnment, for intermediate values of the input noise Here (Fig. [OP we provide an analysis that sheds some light on this phenomenon. For diffeerent noise values, we tracked the relationship between the value of the input weights to network hidden-layer neuron and the corresponding value following training. In the absence of noise, neurons initialized with sufficient intra-class selectivity relative to their initial inter-class se- lectivity are destined to maintain over time (remain in the purple or green regions in Fig. 10 see weight trajectory plots in Fig. ID, 2C for an explanation of why). In the presence of noise, neu- rons with substantial intra-class vs. inter-class selectivity at initialization can nevertheless evolve to develop primarily inter-class selectivity over the course of training. Note that this explanation does not account for why, at extremely high noise values, target alignment drops again (Fig. 3C) we leave in in-depth characterization of this phenomenon to future work, but suspect it arises because the input noise grows substantial enough for the weights to accrue significant projections along axes other than the
visualize here.
14

---

Published as a conference paper at ICLR 2024
Zero input noise
Greater input noise
Inter- class axis
Inter- class axis
Inter-class
Intra-class
Intra-class axis
Intra-class axis
Figure 10: Effect of input noise on training dynamics ReLU networks trained on the tasks in Section 5.2, in the case of maximal separability of the trained dichotomy, varying the level of input noise x² during training. Each point corresponds a hidden-layer neuron the network, and the position of the point indicates the value of the input weights for that neuron initialization. The colors the dots indicate which of the three colored regions the input weights end up in at the end of training.
F EXPLANATION OF TASKS USED IN SYNTHETIC MULTI-LAYER NETWORK EXPERIMENTS
Here we describe the tasks used in Section 7. Formally, the "hard" task is generated by sampling input data of minimal possible dimensionality (5), or equivalently input data kernel of rank 5, subject to the constraint zero input-output kernel alignment. Intuitively. this task involves inputs generated by 5 binary latent variables, and the input-output function requires nonlinear conjunction of all 5 (e.g. whether the number of latent variables set to is even or odd).
The "easy" task involves sampling inputs of maximal possible dimensionality (27 dimensions, i.e. rank-27 input data kernel) subject to the kernel alignment constraint. Intuitively, this task, the targets require nonlinear conjunction of only two of the input dimensions.
15


Reviews:
[{"review_id": "uQUs2evRi2", "reviewer": "Reviewer_MMx3", "summary": "The manuscript analyzes how the geometry of the latent representation of a one layer neural network is influenced by the choice of the activation function, in particular ReLU and Tanh activation functions.\n\nUsing different metrics from the literature, namely kernel alignment of the latent represent, linear decodability, CCPD and sd, it is shown that the latent representation of ReLUnetworks tend to retain more information about the input, while Tanh networks align more with the output(label) representation.\n\nExperiments are performed on multiple synthetic tasks and on Cifar 10.", "questions": "- Can the authors elaborate on the intuition of why Tanh and ReLU behave in this way? And it would be possible to derive theoretical results on this?\n\n- The target alignment  phenomenon of Tanh relates to neural collapse [a] phenomenon : i.e. when training is kept under zero error the representation in the last layer tend to collapse in equidistance clusters aligned to the targets.\nHowever, to the best of my knowledge this phenomenon should be agnostic of the activation function. Can the authors elaborate on this perspective?\n\n[a] Papyan, Vardan, X. Y. Han, and David L. Donoho. \"Prevalence of neural collapse during the terminal phase of deep learning training.\" Proceedings of the National Academy of Sciences 117, no. 40 (2020): 24652-24663.\n\n[b] Zhu, Zhihui, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. \"A geometric analysis of neural collapse with unconstrained features.\" Advances in Neural Information Processing Systems 34 (2021): 29820-29834.\n\n\n- How much the assumptions of fixing weights of the second layer is limiting in terms of measuring the four metrics employed (alignment, decodability, ccpd, sd). Is it needed just in order to approximate the dynamics?\n\n\n- What is the mathematical formulation of the nonlinearity analyzed in section 7 ?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "*Originality*\n\nThe analysis on how the activation function can enforce a different latent representation geometry is to the best of my knowledge, novel and interesting direction to investigate.\n\n*Clarity*\n\nThe paper is overall clear.\n\n*Quality*\n\nThere are multiple experiments on the synthetic setting which  support quite well the claims of the paper. However, the real experiment is not strongly supported( see weaknesses section).\n\n*Significance*\n\nWhile the result on synthetic data are promising, there still some missing buts in order that make somewhat difficult to evaluate the impact of the paper ( see weaknesses section).", "weaknesses": "- While the analysis is interesting and it it wasn't clear to me how much it can impact can the paper have, its current form: (i) it is not clear  how much the analysis extends to real tasks: the experiments of Cifar are somewhat limited (just the alignment metric is reported and it is not clear if this behavior holds for deeper networks: i.e. some ablations on the network depth should be incorporated in the experiment in my opinion)  and the assumptions done on the synthetic tasks are unlikely to hold on larger networks (ii) there is no theory or additional experimental evidence that support why tanh and ReLU behave differently (see question section); \n\n\n- The paper should report a better contextualization in the literature and comparison with similar works (it misses a related work section). I reported some works that should be discussed:\n\n- Hayou, S., Doucet, A.; Rousseau, J.. On the Impact of the Activation function on Deep Neural Networks Training. Proceedings of the 36th International Conference on Machine Learning\n\n- Ramachandran, Prajit, Barret Zoph, and Quoc V. Le. \"Searching for activation functions.\" arXiv preprint arXiv:1710.05941 (2017).\n\n- Ding, Bin, Huimin Qian, and Jun Zhou. \"Activation functions and their characteristics in deep neural networks.\" 2018 Chinese control and decision conference (CCDC). IEEE, 2018.\n\n- Papyan, Vardan, X. Y. Han, and David L. Donoho. \"Prevalence of neural collapse during the terminal phase of deep learning training.\" Proceedings of the National Academy of Sciences 117, no. 40 (2020): 24652-24663.\n\n- Zhu, Zhihui, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. \"A geometric analysis of neural collapse with unconstrained features.\" Advances in Neural Information Processing Systems 34 (2021): 29820-29834.\n\n\n\n- Concerning clarity a better description of the metrics employed (especially SD an CCPD) would be needed, also in terms of mathematical/formal statements,if helpful.\n\n\n\n*Minor*\n\nI spotted a typo:\n\n- section 5 eiegenvalues -> eigenvalues", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZVFWYG6FC9", "reviewer": "Reviewer_eLPi", "summary": "The current paper discussed how ReLu and Tanh activation functions impacted the representation geometry of a single layer feedforward neural network. The authors found that Tanh nonlinearity tend to generate target aligned representation, while RuLe nonlinearity favors input aligned representation. It seems the symmetric saturation of the nonlinearity is the key for target-aligned representation of the Tanh function.", "questions": "1.\tWhy the decoding accuracy is worse in training data with high separability of trained dichotomy for Tanh network (figure 3 upper left)? The separability has non-monotonic effects on input alignment in the Tanh network, why is that?\n2.\tWhy the noise level has non-monotonic effects on the Relu network, consistently observed in all geometric matrix and for all tested separability of trained dichotomy? Author suggested smoothing gradients. What’s the evidence supporting such conclusion?\n3.\tAre the results robust to the input data range, eg. Input data ranges between 0 to 1, vs. input data ranges between -1 to 1.  \n4.\tWhen training data becomes more complicated, as suggested by the noise input analysis and the XOR task, the difference between tanh and ReLu vanish. This suggested that Tanh helps generate a target-aligned representation when the input data is readily separated for clustering. Or in a multi-layer network, using tanh at the final layer seems to be beneficial. This is not a novel conclusion. What’s the new insights learnt from the current study. Provide a discussion on how the results learnt in the current study would have a general impact.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has thoroughly studies the representation geometry using various geometry matrix, which allowed authors to generate insights on whether a network generate input- or target- aligned representation in the hidden layer of a single layer feedforward network. It shows how the representation geometry evolves over the course of learning. In particular the trajectories of input weights to hidden layer neurons for inter-class and intra-class labels is interesting.", "weaknesses": "In general, the results generated by the current study that Tanh nonlinearity helps generate target aligned representation is limited to simple networks and simple input-output mapping. The representation geometry in these simple networks probably are not sufficient for many real-world problems that require capturing intricate patterns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TO4BZaBs6X", "reviewer": "Reviewer_HXSW", "summary": "The authors systematically conducted a series of gradually more intricate experiment to investigate how nonlinearity, label separability, and input data geometry affects the learned representation in the hidden layer of 2 layer MLP, with potential generalization to CNNs. They quantified representation geometry with many statistics, including the CKA with input data, output label; parallelism index; CCGP; classifying ability for unseen label. They found a central unexpected theme that Tanh network seems to keep more geometry about the target label; while relu network seems to align the representation with the input data, and keep more target-unrelated information about the input. Finally, they dissect the source of differences between the nonlinearities ReLU and Tanh, and found that the double sided symmetric saturation of Tanh function probably explain their difference.", "questions": "- The assumptions for making the weight learning animation seems a bit strong, can we alleviate those assumptions? (fixed output weights and discretized output weights)?\n- How to generalize the learning dynamic visualization to non-two-layer deeper networks?\n- Is there a typo of formula in Sec 3.1.1 “*the covariance between input and output: xxx*\" should it be $y_i$?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clear results. Though the general theme is expected, the relevance of activation function still has a little bit surprise.\n- The experiments are systematic and well-designed, which formed an investigation with clear logic. The statistics for quantifying the representation geometry are quite comprehensive and laudable.\n- The way of visualizing the training dynamics in weight space along intra-class vs inter-class axes is illuminating, further this analysis of toy model indeed provides intuition for the phenomena regarding different gradient learning dynamics for different activation functions (at lease in 2-layer networks).\n- Nice controlled experiments to parse out factors explaining difference between relu and tanh, in Sec. 7, showing that the rough two-sided saturation shape is the key. in another perspective, the banded gradient structure of activation function is key.\n- The claim, (if it’s general) will definitely impact how we understand the representation similarity between two systems, e.g. the brain vs CNNs. Namely, if neurons in the visual brain are using a different activation function from the CNNs, even the underlying linear function is the same, the similarity matrix won’t match.", "weaknesses": "- Most of the experiment focused on toy scale examples of binary classification with 2-layer network, even experiments with CNN has only 2 conv-layer networks. I feel it’s within the scope of this paper to show empirical evidence that the observations may generalize to larger scale CNN and larger dataset. (e.g. resnet and ImageNet) Will it be feasible to swap the activation function and show some similar effects?\n- Notation in Eq. (in Sec. 3.1.1), or the method description in Sec. 3.1.1 is a bit confusing. Is it using multi-output setup so $W_O$ is also a vector?\n- There seems to be super interesting intuition going on in Figure 1C, but the text in Sec. 3.1.1 didn’t seem to walk through the logic, leaving the reader to parse the schematics themselves. —— though after working through the math of an example it starts making sense.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uBP149GJW7", "reviewer": "Reviewer_dzyA", "summary": "The paper studies the learning dynamics of 1-hidden-layer neural networks, and focuses on how representation properties are affected by the nonlinearity of the network. Specifically, the paper compares Tanh and ReLU nonlinearity, and show that Tanh networks tend to reflect structure of target outputs while ReLU networks tend to retain more information about the inputs.", "questions": "In most of the work the result focuses on 1-hidden-layer neural networks but in the convolution part there are two FC hidden layers. I'm wondering how depth affect your current observations.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper presents an interesting empirical study on how nonlinear activation functions affect the learned representation in the network, specifically how they align with the target output and the input. The results are evaluated using several previously established metrics, starting from simple toy model dataset, the authors carefully analyzed the effect of input geometry on learned representations, and later extended their results to more complicated tasks.", "weaknesses": "The paper largely focuses on ReLU and Tanh nonlinearity, which are two very specific type of activation functions. It would be nice if the authors can identify what exactly is the property of the nonlinear function that causes the difference in representation, and evaluate further (as they show in Fig. 5 but with more extensive results). \nThe paper also presents mostly empirical evaluations and analysis without theoretical insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)