Training the Untrainable: Introducing Inductive Bias via Representational Alignment

## Abstract

Abstract We demonstrate that architectures which traditionally are considered to be ill-suited for a task can be trained using inductive biases from another architecture. Networks are considered untrainable when they overfit, underfit, or converge to poor results even when tuning their hyperparameters. For example, plain fully connected networks overfit on object recognition while deep convolutional networks without residual connections underfit. The traditional answer is to change the architecture to impose some inductive bias, although what that bias is remains unknown. We introduce guidance, where a guide network guides a target network using a neural distance function. The target is optimized to perform well and to match its internal representations, layer-by-layer, to those of the guide; the guide is unchanged. If the guide is trained, this transfers over part of the architectural prior and knowledge of the guide to the target. If the guide is untrained, this transfers over only part of the architectural prior of the guide. In this manner, we can investigate what kinds of priors different architectures place on untrainable networks such as fully connected networks. We demonstrate that this method overcomes the immediate overfitting of fully connected networks on vision tasks, makes plain CNNs competitive to ResNets, closes much of the gap between plain vanilla RNNs and Transformers, and can even help Transformers learn tasks which RNNs can perform more easily. We also discover evidence that better initializations of fully connected networks likely exist to avoid overfitting. Our method provides a mathematical tool to investigate priors and architectures, and in the long term, may demystify the dark art of architecture creation, even perhaps turning architectures into a continuous optimizable parameter of the network. Code is available at https://github.com/vsubramaniam851/untrainable-networks .

## 1 Introduction

When creating neural networks, we tend as a community to follow recipes that select
among a few architectures known to work for particular tasks
(Ren et al., ; Cong & Zhou, ; Goodfellow et al., ). Architecture is critical. The gains made on
tasks like object recognition are attributable to imposing an inductive bias, i.e., a
prior, on the design of new architectures (Goyal & Bengio, ; Bachmann et al., ). Convolutional networks unlocked many vision problems
(Krizhevsky et al., ; He et al., ) and the recent advent of the
Transformer (Vaswani, ; Devlin, ; Achiam et al., ) did the
same for language. Despite this, finding new architectures and overcoming the
limitations of existing architectures remains a sort of “dark art”. While an architecture
imposes some prior, we often do not fully understand what that prior is. One example of this that remains an open discussion is the role of residual connections in making very deep convolutional networks easier to train (Jastrzebski et al., ).
If we fully understood the priors that architectures imposed, we could
translate between priors and architectures freely – either by specifying the prior
we want and then deriving the appropriate architecture or by dispensing with
architectures and imposing the prior directly.

Recent theorems (Poggio & Fraser, ) state that for each function which is efficiently Turing computable, there exists a deep network that can approximate it well. Furthermore, a graph representing such a function is compositionally sparse; that is, the nodes of the associated Directed Acyclic graph (DAG) represent constituent functions with a small effective dimensionality. A reasonable conjecture is that neural networks with an architecture which is similar to the DAG of the unknown target function are especially successful in learning it, as it is the case for convolutional networks for image recognition and similar tasks. Because we do not understand the relationship between the kinds of priors on the target functions that different architectures impose, even simple questions have no known answer, such as, does there exist an initialization of an FCN that makes it behave like a CNN, though the graphs of the function they represent are fundamentally different?

We introduce a new tool which we term *guidance* to study this problem and
to expand the space of viable networks. Given a *target network*, we guide
it with a *guide network*. In addition to the target’s original loss, the
target attempts to match the representation of its intermediate layers to those
of the guide. We use a measure of representational similarity
(Kornblith et al., ; Cristianini et al., ; Cortes et al., ),
also termed a neural distance function, to compute the distance between
representations of two arbitrary layers. Neural distance functions are often
used in neuroscience to compare activity in networks and brains
(Schrimpf et al., ; Conwell et al., ; Subramaniam et al., ). In light
of recent work that shows that networks of very different architectures have
internal activity that is extremely similar to one another
(Han et al., ; Conwell et al., ; ), we repurpose this
distance function as a means to transfer priors between networks layer by layer.

Figure: Figure 1: Guidance between two networks makes untrainable networks trainable. Given a target which cannot be trained effectively on a task, e.g., a fully connected network which immediately overfits on vision tasks, we guide it with another network. In addition to the target’s cross-entropy loss, we encourage the network to minimize the representational similarity between target and guide activations layer by layer. The guide can be untrained, i.e., randomly initialized. This procedure transfers the inductive biases from the architecture of the guide to the target. The guide is never updated. The target undergoing guidance no longer immediately overfits and can now be trained. Here we show an untrained ResNet guiding a deep fully connected network to perform object classification. The FCN alone overfits, the guided version can now be optimized. It has gone from untrainable to trainable.
Refer to caption: x1.png

We make the following contributions:

- 1.
We develop guidance to transfer priors between networks using representational alignment, investigate one representational alignment method, centered kernel alignment, CKA (Kornblith et al., ), and apply it to several classical problems.
- 2.
Image classification improvements: Deep or wide fully connected networks cease overfitting when guided by a ResNet. No-skip CNNs close much of the gap with ResNets when guided by a ResNet.
- 3.
Sequence modeling improvements: RNNs significantly improve their copy and paste accuracy when guided by a
Transformer. Transformers increase their parity accuracy when guided by an RNN. RNNs close much of the gap to a Transformer when guided by one.
- 4.
Many of these gains exist even when the guide network only transfers over its
priors, not its knowledge, i.e., the guide is untrained and is never updated.
- 5.
By disconnecting the guide early, we provide evidence for a better
initialization regime for fully connected networks that avoids immediate
overfitting.

Guidance can serve as a new tool for investigating the role of architecture and
the relationship between priors and architectures. It can also help discover new
initialization methods, for which no known tool exists. Guidance also has other
potential applications which we leave to future work and only briefly mention in
the conclusion, such as addressing some of the weaknesses of neural architecture
search.

Our work has a number of limitations. We aimed for coverage of many tasks
instead of maximal performance on any one task. This would have required us to
carefully tune the hyperparameters involved. We preferred to show how guidance
works in general rather than in cherry-picked or carefully tuned settings. To
that end, we also did not optimize networks to convergence, nor did we attempt
to experiment with other optimizers. Once we reproduced a well-known problematic training phenomenon,
we showed that it could be overcome. We consider a network trainable enough to overcome a problem when the original problem disappears. For example, successfully training fully
connected networks for object recognition was hopeless because they immediately overfit; using our guidance method they no
longer do so. This does not mean that they are necessarily useful as object
recognizers at present. In the case of fully connected networks, their present
performance with guidance training is too low, but with additional work, we believe their performance could be substantially increased now that their train and test loss are moving in the right direction. In some
cases, by applying guidance, we do see large useful improvements, such as with deep CNNs, RNNs and
Transformers, although much more remains to be exploited in this domain, as well.

## 2 Related Work

Representational Distance: Our method builds on several metrics that measure distance between high-dimensional activations extracted from neural networks or activity in the brain (Klabunde et al., ). Some of these distance metrics make comparisons based on kernel matrices (Kornblith et al., ; Cristianini et al., ; Cortes et al., ) or relative distances (Kriegeskorte et al., ) between sample representations in a set. Others compute linear (Wehbe et al., ; Schrimpf et al., ) or orthogonal projections (Beauducel, ) from one set of representations to another. Others use canonical correlation analysis which finds linear relationships between pairs of vectors (Raghu et al., ; Morcos et al., ). These metrics are designed based on a set of desired invariant properties such as permutation invariance or invariance to linear transformations.

Such approaches have been commonly applied in neuroscience for measuring representational distance of activations from networks and activity in the brain to understand which neural networks are architecturally most similar to the brain (Wehbe et al., ; Conwell et al., ; Subramaniam et al., ; Goldstein et al., ). Under this context, Han et al. () has shown the inability of current representational distance metrics to distinguish representations based on architecture. This provides a basis for our experiments to use these metrics to align representations across different neural networks. This paper gives the foundation for our intuition that networks may have similar representations that allow for transferring inductive biases from one network to another. Moreover, like Han et al. (), we use centered kernel alignment as our main metric of representational alignment. Recent work has explored the relationship between representational and functional distance of networks, discovering transformations between activations of networks and when these make networks functionally equivalent (Klabunde et al., ; Bertalan et al., ).

Untrainable Networks: In this work, we focus on applications of FCNs and plain CNNs for image classification and RNNs for sequence modeling. Prior work has designed such approaches for these tasks, although unsuccessfully relative to the guide networks we use to improve training.

FCNs have been applied for image classification, where small feed-forward networks were trained on object recognition datasets. These networks were designed with 3-5 hidden layers and less than 100 units per layer (Ma & Khorasani, ; Bebis & Papadourakis, ; Khasnobish et al., ; Oh & Suen, ). The performance of these networks was very low, where the goal of the paper was to only maximize training fit rather than generalization performance (Ma & Khorasani, ; Bebis & Papadourakis, ). Other methods were introduced to prevent overfitting in fully connected networks using topological structure (Schittenkopf et al., ) or early stopping (Caruana et al., ), but these methods had poor fits on the training set due to complex architectural design or complex hyperparameter tuning. Deep convolutional networks have been applied to image classification (Krizhevsky et al., ; Bengio et al., ) as well but face problems with vanishing gradients, preventing deep stacking of convolutional networks.

In sequence modeling, RNNs are the classical approach (Schuster & Paliwal, ; Pearlmutter, ; Connor et al., ; Hammer, ) suffering from vanishing and exploding gradients (Hochreiter, ). This limited early RNNs from being applicable to longer sequence lengths which was important for tasks that required memorization (Graves, ). Several additional tricks were developed to improve gradient flow, but ultimately new architectures like long-short term memory networks(Hochreiter, ) and transformers (Vaswani, ) led to stronger improvements. It has also been observed that transformers are untrainable on specific formal language (Bhattamishra et al., ) tasks that require reasoning over a full sequence instead of a sequence-to-sequence task. These tasks are successful with RNNs but not with transformers.

Model Distillation: Guidance shares a resemblance with model distillation (Hinton, ; Gou et al., ; Sanh, ; Hsieh et al., ). Distillation transfers knowledge from a teacher model to a student model by introducing a new component to the loss function that enforces the student model to behave like the teacher model (Kim et al., ; Zhou et al., ). This can consist of penalizing the KL-divergence between the logit predictions of the student and teacher model. Other, more expressive variants have been proposed (Ahn et al., ; Chen et al., ; Huang et al., ; Jin et al., ; Dai et al., ). In these settings, the teacher model is larger, by parameter count, and more performant. Distillation operates on model predictions rather than representations.

Representation-based distillation (Tian et al., ; Chen et al., ; Lin et al., ) and alignment techniques have been proposed to improve alignment between two networks. Certain works have proposed contrastive loss functions on output representations to distill teacher information into the representation space of the student. Other methods introduce correlation congruence or similarity preserving metrics for aligning two networks. Methods have been proposed that use CKA as an alignment approach between representations of two networks or with representations in the brain with notable improvement in network performance (Saha et al., ; Dapello et al., ).

We distinguish guidance from distillation. Guidance can use a smaller untrained guide instead of a larger trained teacher. This is due to guidance operating over intermediate activations of the network rather than the output of the network probabilities or output features, like distillation does. Guidance also operates at many levels at the same time, aligning many layers at once. This helps address the credit assignment problem that gradient descent has when tuning weights early in a network. We also consider many more networks for guidance than is traditional for distillation including networks which have very different architectures like Transformers to RNNs. Distillation is usually carried out between two closely related architectures. We apply guidance to do the opposite.

## 3 Methods

Guidance introduces a term in the loss of a target network, $\mathcal{N}^{T}$, to encourage representational alignment with a guide network, $\mathcal{N}^{G}$. Only the parameters, $\theta^{T}$, of the target are updated — the guide’s parameters, $\theta^{G}$, remain fixed. Per minibatch, representational similarity is computed between the activations of $i^{G}$th layer of the guide, $\mathbf{A}^{G}_{i^{G}}(\theta^{G})$, and activations from a corresponding layer $i^{T}$ of the target, $\mathbf{A}^{T}_{i^{T}}(\theta^{T})$. We refer to the correspondence between layers of the guide $\{i^{G}\}$ and layers of the target $\{i^{T}\}$ as $I$. While this correspondence, $I$, could be complex as any two architectures can form a guide/target pair, here we choose architectures that make the correspondence obvious as is discussed later. For example, the stacked RNNs and Transformers have the same number of layers in our experiments.

The target and guide receive the same input. Per minibatch, we collect activations from intermediate layers of both networks. Layers of guide network are mapped to layers of the target network; see Figure 1. We formulate the loss in terms of minimizing the *representational dissimilarity*, $\bar{\mathcal{M}}$, i.e., the complement of a representational similarity metric, between guide and target activations layer by layer, summing the results. We only consider the centered kernel alignment, CKA, between the activations in this publication. Many other possibilities exist. Any representational similarity function which is differentiable could be used. Efficiency or incremental computation is much more important than it is in traditional applications since this operation happens for every minibatch.

Given $\mathcal{L}_{T}$ as the original loss of the target network, the guide network’s original loss function is irrelevant. The guide need not even have been trained on the same task or the same dataset. It need not even have been trained at all. This latter setting is what allows transferring architectural priors without transferring knowledge from the guide to the target, as there is none in a randomly initialized guide. The final loss we optimize, $\mathcal{L}$ is:

$$ $\mathcal{L}(\theta^{T})=\mathcal{L}_{T}(\theta^{T})+\sum_{i\in I}\bar{\mathcal {M}}({\bm{A}}^{T}_{i^{T}}(\theta^{T}),{\bm{A}}^{G}_{i^{G}}(\theta^{G}))$ (1) $$

This minimizes a task loss while increasing alignment between the target and guide networks given the mapping between them. The mapping may be sparse, not every layer needs to be used. This is important for guidance with transformers or stacked RNNs as will be explained later. We don’t incorporate any weight on the layer-wise similarity component of the loss. Note that the guide’s parameters, $\theta^{G}$, are constants, i.e., the guide is never updated.

### 3.1 Centered Kernel Alignment

To compare representations, we use a representation similarity metric, $\mathcal{M}$, which corresponds to centered kernel alignment (CKA) (Kornblith et al., ; Cortes et al., ; Cristianini et al., ) in our setting. We specifically consider linear CKA.

CKA uses kernel functions on mean-centered representations to compute representational similarity matrices, which are then compared via the Hilbert-Schmidt Independence Criterion (HSIC). More specifically, suppose we have two sets of representations ${\bm{R}}\in\mathbb{R}^{b\times d_{1}}$ and ${\bm{R}}^{\prime}\in\mathbb{R}^{b\times d_{2}}$. We first compute the Gram matrices for each set of representations

$$ ${\bm{K}}={\bm{R}}{\bm{R}}^{T},{\bm{L}}={\bm{R}}^{\prime}{\bm{R}}^{\prime T}$ (2) $$

We center the Gram matrices by introducing a matrix, $H$, where $H={\bm{I}}_{b}-\frac{1}{n}{\bm{1}}{\bm{1}}^{T}$.

$$ $\tilde{{\bm{K}}}={\bm{H}}{\bm{K}}{\bm{H}},\tilde{{\bm{L}}}={\bm{H}}{\bm{L}}{ \bm{H}}$ (3) $$

We compute the HSIC on the Gram matrices.

$$ $HSIC({\bm{K}},{\bm{L}})=\text{tr}(\tilde{{\bm{K}}},\tilde{{\bm{L}}})$ (4) $$

Finally, we define our linear CKA metric as:

$$ $\mathcal{M}({\bm{R}},{\bm{R}}^{\prime}):=\text{CKA}({\bm{K}},{\bm{L}})=\frac{ HSIC({\bm{K}},{\bm{L}})}{\sqrt{HSIC({\bm{K}},{\bm{K}})*HSIC({\bm{L}},{\bm{L}})}}$ (5) $$

In our setting, we consider representational *dissimilarity* and aim to minimize the dissimilarity between representations from our target network and guide network. We define this as:

$$ $\bar{\mathcal{M}}({\bm{R}},{\bm{R}}^{\prime})=1-\mathcal{M}({\bm{R}},{\bm{R}}^ {\prime})$ (6) $$

Linear CKA ranges from 0 0 (identical representations) to $1$ (very different representations). Because of this, we take the complement by subtracting the linear CKA from $1$ to represent dissimilarity.

### 3.2 Layerwise Mapping

We design a simple method for mapping guide layers to target layers as part of providing supervision. The goal of this method is to make guide and target networks architecturally agnostic, i.e. we can supervise any target network with any guide network.

As a simple approach, we evenly spread layer computations of our guide network over our target network. For example, if we consider ResNet-18 and a 50-layer FCN, we would map every convolutional ResNet layer to every second or third linear layer of the FCN. The intuition for this approach follows from the aim of discovering the same function of the guide network using a target network. Through the design of evenly spreading layers of our ResNet-18, we are guiding the FCN to find a function similar to the guide network.

For our mapping, we consider activations from all layers with tunable weights i.e. convolutional, linear, or LSTM/RNN based layers as well as activations from layer normalizations. For multiple stacked RNNs, LSTMs, or transformers, we extract feature representations from intermediate layers in the stack as well. Using all layers is useful for guidance as it provides a strong signal to induce alignment between the target and guide networks during training. We empirically find that more layers leads to stronger results. Skipping layers based on non-linear transformations reduces memory overhead associated with storing representations per batch.

## 4 Experiments

We design several settings with different target and guide networks to thoroughly test our approach. We include a range of image and sequence modeling tasks. In choosing target networks, we consider a broad range of designs for networks that are not traditionally applied (e.g., a FCN in image classification).

To systematically evaluate our approach, we categorize our experiments into the following groups of untrainability that our proposed method improves upon:

- 1.
Untrainable Architectures: Experiments where the target networks are difficult to train due to architectural limitations, irrespective of the task.
- 2.
Untrainable Tasks: Experiments where certain tasks are inherently challenging for specific architectures, making them untrainable without additional supervision.

### 4.1 Tasks

Here, we describe the task settings. For an image-based task, we focus on *image classification* and use the ImageNet-1K dataset (Deng et al., ) for training and testing. We use the splits defined by the dataset. We report accuracy on the validation set for all experiments.

We next consider three sequence modeling tasks that allow for a broader range of architectural settings. We first consider a task called *copy-paste* (Graves, ). In this task, we generate a sequence of numbers in the range of $1-10$. The model is trained to recover the same sequence in the output. In our setting, we consider sequence lengths that range from $20$ to $40$ values total (internal sequence and padding). We generate a copy-paste dataset, sampling sequences containing numbers between $1$ and $10$. We generate a total of $100,000$ examples, training on $80,000$ examples, validating on $10,000$ examples, and testing on $10,000$ examples.

We also include the *parity* task, a binary classification task where a model is fed a bitstring and outputs $1$ when there is an even number of ones in the bitstring and 0 0 otherwise. We generate a series of bitstrings with sequence lengths that range from $2$ to $50$ (as in prior work) (Bhattamishra et al., ).

Finally, we consider a *language modeling* task using the WikiText-103 dataset (Merity et al., ) where models must predict the next token given some context. This uses the train, validation and testing splits defined by the WikiText dataset and for all experiments, we use a context length of $50$. We tokenize the text data using the GPT-2 (Radford et al., ) tokenizer.

### 4.2 Architectures

**Table 1: Guide and target networks across tasks. Our network designs include several untrainable target networks and corresponding trainable guide networks. We focus on FCNs and deep convolutional networks for image classification and recurrent networks for sequence modeling.**
| Tasks | Guide Networks | Target Networks |
| --- | --- | --- |
| Image Classification | ResNet-18 | Deep FCN |
| Wide FCN |  |  |
| ResNet-50 | Deep ConvNet |  |
| Copy-Paste | Transformer | RNN |
| Parity | RNN | Transformer |
| Language Modeling | Transformer | RNN |

For all tasks, we describe our target untrainable architectures for each task separately as well as the guide networks that are employed to make the untrainable network trainable. We give an overview in table 1.

*Image Classification*: We use three target networks: Deep FCN, Wide FCN, and Deep ConvNet. Deep FCN is a fully-connected network with 50 blocks consisting of feedforward layers followed by non-linearities. This network is an untrainable architecture, lacking inductive biases to prevent overfitting and having vanishing gradients. Wide FCN is a fully connected network with 3 blocks with feedforward layers that have 8192 units. This is categorized as an untrainable task due to a saturation in the training performance. Deep ConvNet is the same architecture as ResNet-50 (He et al., ), but without residual connections. This is categorized as an untrainable architecture due to the vanishing gradient problem. We use two guide networks: ResNet-18 and ResNet-50. ResNet-18/50 is a deep convolutional network with 18/50 convolutional blocks and residual connections. We refer to He et al. () for ResNet 18/50 design. We supervise the Deep FCN and Shallow FCN with ResNet-18 and supervise the Deep ConvNet with ResNet-50.

*Sequence Modeling*: For our copy-paste task, we use a vanilla, 4-layer RNN and language modeling tasks, we use a vanilla 5-layer RNN as our target networks. In copy-paste, architectural and algorithmic limitations make RNNs an untrainable architecture. For language modeling, vanishing gradients and limited context incorporation make RNNs an untrainable architecture as the training loss saturates. For the parity task, we use a 1-layer transformer encoder architecture, similar to prior work (Bhattamishra et al., ; Hahn & Rofin, ). For the copy-paste task, we train a guide network, 4-layer transformer decoder model which achieves 96.90% accuracy. Similarly, for language modeling, we train a 5-layer transformer decoder guide network with a context window of 256. Our final perplexity is 34.15. For the parity task, we train a 1-layer vanilla RNN as a guide network which achieves the same 100% accuracy reported by Bhattamishra et al. ().

### 4.3 Training

For each setting, we train the base target network and perform an experiment where both a trained and untrained guide network supervises the base target network. All networks are trained with either cross-entropy loss or binary cross-entropy loss, although other loss functions could be applied as well.

When training networks for object detection using ImageNet-1K, we use the Adam (Kingma, ) optimizer. For all sequence modeling tasks, i.e. copy-paste, parity, and the language modeling task we use AdamW (Loshchilov, ). For language modeling, we also incorporate gradient clipping due to unstable training with long sequences.

To ensure consistency of comparisons across learning curves, we use a consistent batch size of 256. In general, representational similarity metrics are affected by the number of samples in the calculation. The more samples, the better the metric approximates representational distance. We use 256 as a proxy, dependent on GPU memory, although more memory would allow for bigger batch sizes with potentially better results. Due to the large number of training settings, we employ several different learning rates. We tune the learning rate carefully for baseline training to ensure maximal performance of base training. We sweep the parameter across 5 different values and choose the results with the lowest validation loss. This ensures we are choosing the training with the best performance.

After choosing the optimal learning rate, we then train all networks and settings for 100 epochs with 5 random seeds to compute error bars for our training loss, validation loss, and testing accuracy. Our error bars are associated with the standard error across each step across all seeds. We choose the seed-based average test accuracy associated with the epoch with the lowest seed-based average validation loss.

## 5 Results

Here, we report the findings from our use of guidance to train our set of untrainable networks. Results are shown in the following order: image classification task; sequence modeling tasks; language modeling tasks.

**Table 2: Guidance improves performance for image classification. Alignment with a ResNet dramatically improves a deep FCN, particularly with an untrained ResNet. Significant gains are seen with a wide FCN as well. Deep CNNs without residuals gain only with a trained ResNet. Across all settings, guidance can help train architectures that were otherwise considered unsuitable.**
| Experiment | ImageNet Top-5 Validation Accuracy $(\uparrow)$ |
| --- | --- |
| ResNet-18 | 89.24 |
| Untrained ResNet-18 | 0.24 $\pm$ 0.043 |
| ResNet-50 | 92.99 |
| Untrained ResNet-50 | 0.54 $\pm$ 0.029 |
| Deep FCN | 1.65 $\pm$ 0.51 |
| ResNet-18 $\rightarrow$ Deep FCN | 7.50 $\pm$ 1.51 |
| Untrained ResNet-18 $\rightarrow$ Deep FCN | 13.10 $\pm$ 0.72 |
| Wide FCN | 34.09 $\pm$ 1.21 |
| ResNet-18 $\rightarrow$ Wide FCN | 43.01 $\pm$ 0.92 |
| Untrained ResNet-18 $\rightarrow$ Wide FCN | 39.47 $\pm$ 0.31 |
| Deep ConvNet | 70.02 $\pm$ 1.52 |
| ResNet-50 $\rightarrow$ Deep ConvNet | 78.91 $\pm$ 2.16 |
| Untrained ResNet-50 $\rightarrow$ Deep ConvNet | 68.17 $\pm$ 2.54 |

**Table 3: Guidance improves performance for sequence modeling. RNN performance improves dramatically when aligning with the representations of a Transformer for copy and paste, as well as for language modeling. RNNs close most of the gap to Transformers for language modeling. Transformers in turn, improve parity performance when aligning with an RNN. Guidance is able to transfer priors between networks.**
| Experiment | Copy-Paste Accuracy $(\uparrow)$ | Parity Accuracy $(\uparrow)$ | Language Modeling Perplexity $(\downarrow)$ |
| --- | --- | --- | --- |
| RNN | 14.35 $\pm$ 0.01 | 100 | 69.19 $\pm$ 1.89 |
| Untrained RNN | — | 2.32 $\pm$ 0.41 | — |
| Transformer | 96.98 | 71.98 $\pm$ 3.16 | 34.15 |
| Untrained Transformer | 1.04 $\pm$ 0.81 | — | 51948.8 $\pm$ 90.44 |
| RNN $\rightarrow$ Transformer | — | 78.49 $\pm$ 2.16 | — |
| Untrained RNN $\rightarrow$ Transformer | — | 70.38 $\pm$ 4.17 | — |
| Transformer $\rightarrow$ RNN | 23.27 $\pm$ 1.02 | — | 40.01 $\pm$ 1.54 |
| Untrained Transformer $\rightarrow$ RNN | 42.56 $\pm$ 1.51 | — | 59.61 $\pm$ 2.33 |

### 5.1 Image Classification

Figure: Figure 2: Training and validation under guidance for all experiments reported in table 1. For every result in Table 2 and Table 3 we show the training and validation loss with error bars across multiple runs although these are often too small to see. Note that often the best results occur with the untrained guide.
Refer to caption: x2.png

Across all our networks, we observe significant improvement from using a guide network to provide guidance; see table 2. We see significant accuracy gains of 5-10% on validation performance. We also observe significantly better loss curves from a better fit with the training loss and reduced overfitting with the validation loss. Most interestingly, we highlight that using a randomly initialized guide network can perform better than using a trained guide network. For example, the Deep FCN results in the top left of fig. 2 are significantly better with a randomly initialized ResNet-18 as the guide network instead of a trained ResNet-18. This trend also occurs with Wide FCN.

We note that this trend is not entirely consistent as indicated by the Deep ConvNet. One potential explanation is that the architectural prior is the same for both the Deep ConvNet and ResNet-50. This explanation provides an additional interpretation for the role of residual connections and their influence on the representation space. This indicates that residual connections must be trained to have an influence on the representation space, and aligns with prior studies of residual connections (Jastrzebski et al., ; He et al., ).

### 5.2 Sequence Modeling

On the copy-paste task, see fig. 2 and table 3, we see significant improvement when using a transformer as our guide network over an RNN target network. Prior work has observed the difficulty that RNNs have with copy-paste and usually attributes this to a fundamental algorithmic limit on the capability of RNNs to memorize input sequences. Our results show a potential optimization scheme for RNNs that is applicable for sequence memorization. Similar to prior results with fully connected networks, this result improves when using a randomly initialized network. We explore this more thoroughly in further analyses but this is likely because optimization with randomly initialized networks is easier due to the degrees of freedom in CKA.

On the parity task, we similarly see improvement when using an RNN as our guide network over a transformer model as our target network. This is a complementary result to copy-paste and language modeling where the guide network was a transformer and our target network was a RNN. This improves over results from several prior papers that have pointed out fundamental limitations of transformers to perform certain formal language tasks.

Unlike the prior tasks, the performance improves when using a trained RNN as the guide network. This could be due to the wide gap in performance between an untrained RNN and trained RNN on parity. Trained RNNs achieve 100% on the task and could have meaningful positional information to complete the parity task. This information is likely crucial to the transformer which has limited sequence pooling capacity and fewer degrees of freedom unlike an RNN.

On language modeling, similar to copy-paste, we see significant improvements with the RNN target network that is guided by a transformer guide network. While performance generally saturates for the 5-layer RNN, guide network guidance continuously improves the RNN performance by close to 30 points for test perplexity for both trained guide networks and 10 points for randomly initialized guide networks. This implies that information from the transformer can be transferred to the RNN. We see a similar trend with a randomly initialized transformer as the guide network implying that architectural priors in the transformer are driving much of the improvement in guided network performance.

### 5.3 Representational Similarity Loss

Figure: Figure 3: Representational dissimilarity changes across training. We view the representational dissimilarity, measured as the complement of CKA, across training between our target and guide networks. We find that across many settings, target networks reduce dissimilarity more quickly with untrained guide networks, which can explain improved results with untrained guide networks. With language modeling, we find that the RNN reduces dissimilarity with an untrained guide network more quickly despite having a worse performance.
Refer to caption: x8.png

We can view the representational alignment between the guide and target networks during training. This allows us to better understand how this representational alignment influences network performance. We show results in fig. 3.

We notice that across most tasks, reducing representational dissimilarity is easier with activations from randomly initialized networks rather than trained networks. This provides additional evidence of representational alignment for inductive bias transfer. We notice that for certain cases, such as Parity, the randomly initialized guide network has higher representational dissimilarity loss than the trained guide network. This is matched with the Parity result in table 3 and fig. 2.

However, we can also observe more inconsistent results with the Deep ConvNet where the untrained guide network has lower representational dissimilarity loss than the trained guide network, even at the end of training. One possible explanation here is that the inductive bias was more similar for Deep ConvNet and ResNet-50. This means that alignment with an untrained model is uninformative as the only information in an untrained guide network is its own inductive bias which Deep ConvNet matches more closely. So, it is likely that trained features are improving guidance with Deep ConvNet.This result has interesting implications for understanding the role of residual connections. The untrained residual connections have little influence on the inductive biases of the network or the overall representation space. This demonstrates the strength of our method as a way to interpret neural network design choices and how they influence representation and functional aspects of a network. Similarly, for the RNN trained for language modeling, we hypothesize that trained features are more useful for improving RNN performance for a similar reason.

These results are also potentially indicative of architectural properties of RNNs and FCNs which match randomly initialized networks more quickly. For instance, one potential explanation is that RNNs have more degrees of freedom (Bhattamishra et al., ) and therefore, only need inductive guidance rather than trained features. Transformers may require learned features indicating that the bottleneck for transformers on the parity is not algorithmic but feature-based. Future work can use these results as a starting point for the specification of more informative priors in model design (i.e. architectural priors) and training procedures alike.

### 5.4 Error Consistency

Figure: Figure 4: Guidance aligns error consistency. The relationship between the guide networks is mirrored in that of the guided networks even when the target is entirely unlike the guides initially. This is additional evidence that guidance doesn’t just improve performance arbitrarily, the target becomes more like the guide.
Refer to caption: x10.png

Given our guided networks, we can analyze the functional properties of the guided network to confirm whether networks adopt priors from their guide networks. Using Deep FCN as our target model, we guide it with a ResNet-18 or a ViT-B-16 (Dosovitskiy, ). We then measure the error consistency (Geirhos et al., ) between all of the networks which indicates the error overlap between two networks based on the accuracy of the networks, i.e. do the two networks make similar class predictions? The measure first calculates the expected error overlap. Suppose $a_{1}$ is the accuracy of the first guided network and $a_{2}$ is the accuracy of the second. The expected error overlap is given by $c_{\text{exp}}=a_{1}*a_{2}+(1-a_{1})*(1-a_{2})$. Next, we measure the observed error overlap across each sample in the validation set as $c_{\text{obs}}=\text{\# of samples where both models agree}~{}/~{}\text{total trials}$. Finally, we can write $\kappa$ as:

$$ $\kappa=\frac{c_{\text{obs}}-c_{\text{exp}}}{1-c_{\text{exp}}}$ (7) $$

$\kappa$ ranges from $-1$ to $1$, where $1$ is perfect agreement, $-1$ is perfect disagreement and 0 0 is change agreement. When $\kappa>0$, this implies that models make consistent error patterns, $\kappa<0$ implies that models make inverse error patterns, and $\kappa\approx 0$ implies independent error patterns.

See fig. 4. The error consistency between the initial FCNs is entirely unlike the ResNet-18 or ViT-B. Guidance creates two FCNs which have the same relationship to one another. The ResNet-18-guided FCN and ViT-B-guided FCN have the same error consistency with respect to one another as ResNet-18 and ViT-B do. In other words, it’s not just that the FCN gets generically better; rather, it’s that the FCN adopts a prior from the original architecture.

### 5.5 Network Initialization

Figure: Figure 5: Disconnecting the guide early can have long-term effects. We remove the ResNet guide from the loss function of a Deep FCN after an arbitrary 150 steps. Surprisingly, this has little effect. The Deep FCN still does not diverge and the training and validation loss still mirror one another. Very likely, an initialization regime exists for Deep FCNs that prevents overfitting. We now have the tools to investigate this question.
Refer to caption: x11.png

Is guidance needed throughout training, or is the effect of the guide to move the target into a regime where the two are aligned and the target can be optimized further without reference to the guide? The answer to this question can shed light on whether the guide is telling us that better initializations are likely to exist for the target. To answer this question, we disconnect the guide from the target after an arbitrary and nominal number of training steps, 150. We did not investigate what the minimal number of steps needed was.

For FCNs, even this early disconnect avoids overfitting; see fig. 5. Furthermore, while preventing overfitting, we have lower training loss from guidance, indicating a better fit. This implies that there likely exists a better initialization for FCNs. In future work, we intend to investigate what the untrained guide is doing to the FCN weights to prevent validation loss from increasing dramatically.

## 6 Conclusion

We demonstrated that guidance eliminates the failure modes of networks previously considered unsuitable or ineffective for specific tasks. Aligning with another network overcomes these shortcomings by transferring inductive biases—either architectural and knowledge-based, or solely architectural when using an untrained guide. This opens the door to many applications.

For some networks, like fully connected ones, we only overcame the initial obstacle. Further research is needed to refine these into effective vision models, as they avoid immediate overfitting. This may be a matter of scale, optimizers, learning rates, or other aspects of model design and training. Our results suggest practical applications by significantly narrowing the performance gap between vanilla stacked RNNs and Transformers, albeit in small-scale experiments. Given that stacked RNNs are equivalent to single-layer RNNs, most directly to delayed RNNs (Turek et al., ), this implies that complex modifications to RNNs may be unnecessary for language modeling. In future, one might be able to focus on the suitability for inference of different architectures with fewer hard constraints from architecture.

Guidance also proved to be a tool with which to discover the possibility of new initializations. At the moment, no known method exists to find better initializations for networks. In some cases, as with the FCNs for vision, guidance can be disconnected after a nominal number of steps, but still goes on to regularize the target network. This strongly implies that an initialization regime for that target with the same regularization exists. This is all that guidance could do in that case. We now need tools to go backwards, given networks which are correctly initialized and networks which are not, discover what that initialization is. This is a much better place to be in. A systematic sweep of targets and guides to look for better initializations should be carried out.

Our method can be used to study representational and functional design of neural networks in new ways to reinforce or reanalyze prior theory of neural network optimization. For example, we can understand distances between architectural components based on which target networks are easier to guide with a particular guide network. We also refine this notion to include a narrow channel through which guidance can occur, the representational similarity. This can serve as a kind of probe. Different representational similarity methods enforce alignment between different properties of the activations of networks.

Characterizing architectural priors and untrainable networks can be made more precise using our methodology. For example, guidance clearly distinguishes between at least three cases. One where the guide does not help. Another where the architecture of the guide helps, but not the knowledge. And one where both the architecture and the knowledge of the guide help. These are different phenomena whose root causes are not understood at present but that could elucidate the relationship between priors and architectures.

Looking into the long-term future, it may be that architecture can be turned into a prior that we add to the loss of a generic fully-connected network, or any convenient and capable-enough network. Architecture would lose its prominence and the computational substrate could be disconnected from the priors we need for learning and inference. Architecture could potentially even be turned into a parameter to be optimized, rather than a hyperparameter.

#### Acknowledgments

This work was supported by the Center for Brains, Minds, and Machines, NSF STC award CCF-1231216,
the NSF award 2124052,
the MIT CSAIL Machine Learning Applications Initiative,
the MIT-IBM Watson AI Lab,
the CBMM-Siemens Graduate Fellowship,
the DARPA Artificial Social Intelligence for Successful Teams (ASIST) program,
the DARPA Mathematics for the DIscovery of ALgorithms and Architectures (DIAL) program,
the DARPA Knowledge Management at Scale and Speed (KMASS) program,
the DARPA Machine Common Sense (MCS) program,
the United States Air Force Research Laboratory and the Department of the Air Force Artificial Intelligence Accelerator under Cooperative Agreement Number FA8750-19-2-1000,
the Air Force Office of Scientific Research (AFOSR) under award number FA9550-21-1-0399,
and the Office of Naval Research under award number N00014-20-1-2589
and award number N00014-20-1-2643.
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

## Appendix A Appendix Overview

We present additional details of guidance, experiments and analysis, as well as additional results. In Appendix B, we present some limitations of guidance as a method. In Appendix C, we provide additional details for our guidance approach, with a full explanation of centered kernel alignment. In Appendix D, we provide additional details on our architectural designs and network training. In Appendix E, we introduce a new experiment where we feed noise to our guide network rather than real data and compute representational alignment, leading to similarly improved results. This further establishes a transfer of a prior rather than knowledge. In Appendix F, we provide test accuracy metrics over training as a complementary of network performance over training outside of cross-entropy loss. In Appendix G, we include experiments with a new loss function for task loss, mean-squared error, to demonstrate that guidance is agnostic to the task loss function. In Appendix H, we add an additional experiment where we apply guidance with a new metric for representational alignment, representational similarity analysis, rather than CKA.

## Appendix B Methodology Limitations

Our guide network supervision through representational alignment has one primary limitation due to increased memory usage during training. Due to saving activations across several layers of the two networks, GPU memory usage increases dramatically. Moreoever, our methodology works better as batch size increases since this allows for better approximation of representational similarity, increasing memory usage even more. Furthermore, including more layers for supervision leads to improved results.

In this paper, we introduce simple techniques to handle memory constraints such as gradient accumulation and gradient checkpointing (Lin et al., ). In practice, more memory optimization techniques may become necessary to consider larger untrainable networks. Further work could consider using stronger representational alignment strategies to reduce the number of samples necessary to achieve a strong fit.

## Appendix C Methods Overview

We give an overview of guidance in algorithm 1 and highlight crucial changes to base neural network training in either red or blue. We use blue to indicate the collection of network activations and red to indicate the layerwise mapping and representational alignment using a distance metric. This gives an overview of our layer mapping between the target and guide network. Crucially, we find that the simplest layer mapping where we evenly distribute guide network layers across target network layers for supervision obtains strong results.

Figure: Algorithm 1 Guidance: Guide Network Representational Alignment

## Appendix D Architecture and Training Details

### D.1 Architectural Design Details

For all tasks, we describe our target untrainable architectural designs for each task separately as well as the guide networks that are employed to make the untrainable network trainable.

#### D.1.1 Image Classification

Target Networks

Deep FCN: We design a fully-connected network consisting of 50 blocks. Each block contains a feedforward linear layer, a batch normalization, and a ReLU nonlinear activation. The intermediate feedforward linear layers contain 2048 units. This network is untrainable due to vanishing gradients since the network is very deep and due to overfitting.

Shallow FCN: We design a network similar to Deep FCN but only containing 3 blocks where each feedforward linear layer contains 8192 units. This network is considered untrainalbe due to a saturation on the training performance.

Deep ConvNet: We design a deep convolutional network with the same architecture as ResNet-50 (convolutional layers followed by batch normalization) but remove the residual connections. This makes the network untrainable due to the vanishing gradient problem as observed in He et al. (), causing saturation of the loss.

Guide Networks

ResNet-18/50: A deep convolutional network with 18/50 convolutional blocks and residual connections. We refer to He et al. ().

We supervise the Deep FCN and Shallow FCN with ResNet-18 and supervise the Deep ConvNet with ResNet-50.

#### D.1.2 Copy-Paste

Target Networks

RNN: We design a 4-layer RNN with a hidden dimension of 768 units, followed by a fully connected layer. In copy-paste, architectural and algorithmic limitations make RNNs an untrainable architecture fortask. Specifically, RNNs must memorize the input sequence which is difficult, particularly with a padding token. RNNs are generally considered to be unapplicable to the copy-paste task.

Guide Networks

Transformer: We consider a 4 layer transformer decoder architecture with a hidden dimension of 768 units across 12 transformer heads. The transformer is well-suited for copy-paste as the attention mechanism can act as a routing mechanism for the sequence. We train the transformer guide from scratch, as with language modeling and achieve $96.90\%$ accuracy on the task.

#### D.1.3 Parity

Target Networks

Transformer: Similar to Bhattamishra et al. (), we design a 1 layer transformer encoder network with a hidden dimension of 64 units across 4 attention heads. Transformers have lower accuracy on formal language tasks that require reasoning over a sequence in comparison to traditional sequence models (Hahn & Rofin, ). Due to the enormous gap in performance and saturation of performance, we categorize the transformer as untrainable.

Guide Networks

RNN: We include a 1 layer vanilla RNN with a hidden dimension of 64 units. Similar to Bhattamishra et al. (), we achieve 100% accuracy on the task.

#### D.1.4 Language Modeling

Target Networks

RNN: We design a 5 layer vanilla RNN with a hidden dimension of 512 and with a tanh activation function. We train this with on sequences with a context length of 75. This makes the network untrainable due to problems associated with exploding and vanishing gradients during backpropagation through time.

Guide Networks:

Transformer: We design a 4 layer transformer decoder network with 16 attention heads and a hidden dimension of 512. We train the transformer on WikiText with a context length of 256 and achieve a final test perplexity of 34.15.

### D.2 Training

**Table 4: Learning rates for network training. For all networks, we sweep over 5 learning rate values before choosing the learning rate with the lowest validation loss for training. Our training does not use any learning rate scheduling such as a warm-up scheduler although such techniques may improve results.**
| Tasks | Experiment | Learning Rate |
| --- | --- | --- |
| Image Classification | Deep FCN | $1\times 10^{-4}$ |
| Wide FCN | $1\times 10^{-4}$ |  |
| Deep ConvNet | $1\times 10^{-3}$ |  |
| ResNet-18 $\rightarrow$ Deep FCN | $5\times 10^{-5}$ |  |
| Untrained ResNet-18 $\rightarrow$ Deep FCN | $1\times 10^{-4}$ |  |
| ResNet-18 $\rightarrow$ Wide FCN | $1\times 10^{-4}$ |  |
| Untrained ResNet-18 $\rightarrow$ Wide FCN | $1\times 10^{-4}$ |  |
| ResNet-50 $\rightarrow$ Deep ConvNet | $1\times 10^{-3}$ |  |
| Untrained ResNet-50 $\rightarrow$ Deep ConvNet | $1\times 10^{-3}$ |  |
| Copy-Paste | RNN | $1\times 10^{-4}$ |
| Transformer | $1\times 10^{-4}$ |  |
| Transformer $\rightarrow$ RNN | $1\times 10^{-5}$ |  |
| Untrained Transformer $\rightarrow$ RNN | $1\times 10^{-5}$ |  |
| Parity | Transformer | $1\times 10^{-3}$ |
| RNN | $1\times 10^{-2}$ |  |
| RNN $\rightarrow$ Transformer | $1\times 10^{-3}$ |  |
| Untrained RNN $\rightarrow$ Transformer | $1\times 10^{-3}$ |  |
| Language Modeling | RNN | $1\times 10^{-4}$ |
| Transformer | $1\times 10^{-4}$ |  |
| Transformer $\rightarrow$ RNN | $5\times 10^{-5}$ |  |
| Untrained Transformer $\rightarrow$ RNN | $5\times 10^{-5}$ |  |

In Table 4, we show the different learning rate settings we converged to in each experiment. For each experiment, we did a grid search over 5 different learning rate parameters to ensure optimal learning rate setting. We did careful tuning of all training of target networks to ensure maximum performance.

For all image classification tasks, we used the Adam optimizer (Kingma, ), in-line with prior work (He et al., ). For all sequence modeling tasks, we use AdamW (Loshchilov, ), which has been useful in training sequence models like RNNs and Transformers (Radford et al., ).

The training experiments in this paper were completed across 4 H100s and 4 A100 GPUs for 3 weeks in total. GPU optimization techniques were taken such as gradient accumulation and gradient checkpointing and some language modeling experiments used mixed-precision training.

## Appendix E Representational Regularization

Figure: Figure 6: Feeding noise prevents overfitting. We introduce an additional experiment where we feed noise to our guide network rather than the same batch during each training step with guide network guidance. We sample noise from a Gaussian distribution with a mean of 0 and a standard deviation of 1. We find that despite having no information about the images in the batch, the guide network still provides an inductive bias to prevent overfitting. While this noise increase the training loss, this shows a true transfer of an inductive bias that is not driven by pure distillation of similar features.
Refer to caption: x12.png

We also aim to understand the role of the guide network as in guidance. In all experiments, we use trained and untrained guide networks and see consistent improvements for training the target network. The success of untrained networks implies that our training method is not performing distillation but instead truly transferring a prior from the guide network to the target network. To more strictly test this theory, we include an experiment where we feed noise to the guide network instead of the same batch of data fed to the target network as implied by eq. 1.

We apply this experiment to the Deep FCN with an untrained ResNet-18 as the target network. At each training step, we pass a noisy batch which is sampled from a random Gaussian with mean of 0 and standard deviation of 1. We train for 100 epochs and report the learning curve results in fig. 6.

This result confirms our intuition about the role of guide network: as a guide on model priors rather than a pure distillation of information. While the overall cross-entropy loss magnitudes are higher and the overall accuracy is lower when passing noise to the guide network, our results are significantly better than applying vanilla training approaches to the Deep FCN.

## Appendix F Test Accuracy across Training

Figure: Figure 7: Testing accuracy improves across guidance for sequence modeling. We visualize the testing accuracy for sequence modeling as an example to demonstrate that guidance improves accuracy across training and this improvement is significantly better across training. This allows for another intepretation of the method outside of cross-entropy loss.
Refer to caption: x13.png

We plot accuracies over training as a complement to cross-entropy loss in fig. 7 for the sequence modeling experiments. We can use these experiments to de-couple our results from properties of cross-entropy loss that may lead to misleading improvements across training. We find that accuracies improve consitently across training, supporting the loss curve interpretation that guided training improves results.

## Appendix G Mean-Squared Error Loss

Figure: Figure 8: Guidance improves training with MSE loss. We apply guidance to a new loss function, mean-squared error, over the CIFAR-10 dataset. We use the Deep FCN as our target network and an untrained ResNet-18 as our guide network. We find that the performance of the Deep FCN saturates but improves when using a guide. This shows that guidance is not dependent on the particular loss function used.
Refer to caption: x14.png

Throughout this work, we have mainly focused on a classification task and employed the cross-entropy loss function as our main task loss. However, other loss functions can be applied which do not have the same properties as cross-entropy loss for learning classification tasks as well as other kinds of tasks like regression tasks. One such loss is mean-squared error (MSE) loss. We include an additional experiment with guidance where we train the Deep FCN for image classification with a ResNet-18 guide.

We design an experiment where we train Deep FCN on the CIFAR-10 dataset (Krizhevsky et al., ) with an untrained ResNet-18 guide. Our reasoning for using CIFAR-10 instead of ImageNet stems from prior work which demonstrates difficulty with ImageNet training while using a vanilla MSE loss (Weng et al., ). We train the Deep FCN both with and without a guide for 100 epochs with a batch size of 256.

We show results in fig. 8. Similar to fig. 2, we find that guidance signficantly improves performance of the Deep FCN in comparison to base training. Most notably, the Deep FCN saturates on CIFAR-10 during training. However, with ResNet-18 as a guide, the loss continues to decrease across training. This demonstrates that guidance is useful for other loss functions and is unaffected by the task loss.

## Appendix H Guidance with RSA

To further understand the generality of guidance as an approach, we include one experiment where we guide a network using a different metric for representational alignment. We choose representational similarity analysis (RSA) (Kriegeskorte et al., ) as our main approach. Our main criteria for a metric is differentiability and compute time. RSA is differentiable and can be computed quickly per batch. This makes it a natural candidate after CKA. We first describe RSA and use it as part of guidance between ResNet-18 and the Deep FCN.

### H.1 Representational Similarity Analysis

We use the RSA formulation as described in Kriegeskorte et al. (). Specifically, RSA constructs representational dissimilarity matrices (RDMs) for two sets of representations and compares them using an outer similarity function.

Given two sets of representations, ${\bm{R}}\in\mathbb{R}^{b\times d_{1}}$ and ${\bm{R}}^{\prime}\in\mathbb{R}^{b\times d_{2}}$, we first calculate RDMs for each set of representations using a distance function $d$. Formally, we define ${\bm{D}}\in\mathbb{R}^{b\times b}$ as

$$ ${\bm{D}}_{i,j}:=s({\bm{R}}_{i},{\bm{R}}_{j})$ (8) $$

Each row ${\bm{D}}_{i}$ corresponds to the distance between the representations of input $i$ and the representations of all inputs including itself. This is done per-batch, meaning that RSA is sensitive to batch size.

Given two RDMs ${\bm{D}}$ and ${\bm{D}}^{\prime}$ constructed from sets of representations ${\bm{R}}$ and ${\bm{R}}^{\prime}$ respectively, we vectorize the RDM matrices using a function $v$ (since the RDMs are symmetric, we only need to compare the lower triangles), and compute the similarity between the two vectorized RDMs using a similarity function $s$.

$$ $\mathcal{M}({\bm{R}},{\bm{R}}^{\prime})=s(v({\bm{D}}),v({\bm{D}}^{\prime}))$ (9) $$

As with CKA, we use the complement of the similarity to construct $\bar{\mathcal{M}}$. In practice, we define $d$ to be the cosine distance between every pair of inputs and $s$ to be the pearson correlation between the RDMs as done in previous work (Conwell et al., ).

### H.2 Results

Figure: Figure 9: Guidance with RSA as the representational similarity metric maintains similar performance to CKA. We include a furher experiment where we change the metric for representational alignment from CKA to RSA during guidance training. We apply this to the Deep FCN with ResNet-18 as a guide network. We see that, like CKA, RSA alignment also allows for transferring the prior from ResNet-18. However, unlike CKA, the untrained guide network only does marginally better than the trained network, potentially indicating the RSA is better at transferring trained features.
Refer to caption: x15.png

We apply guidance with RSA to Deep FCN as our target network and ResNet-18 as our guide network. Similar to our CKA results, we train for 100 epochs with a batch size of 256, as RSA is sensitive to the number of samples when comparing sets of representations. We see results over the training, validation, and representational dissimilarity loss in fig. 9. The Deep FCN guided by a trained ResNet-18 achieves a top-5 accuracy of *11.02%* and the Deep FCN guided by a randomly initialized ResNet-18 achieves a top-5 accuracy of *11.74%*.

We can first observe that guided training improves over base training as noted in fig. 2 and table 2. This demonstrates the generality of our approach to other metrics. As long as a representational similarity metric is differentiable, we can optimize the metric for alignment between two networks as a method to transfer the prior of one network to another.

We can also observe some minute differences between the results with CKA. Most notably, the trained guide network has similar performance to the untrained guide network. This is likely because less information about trained features are present in the RSA metric. RSA measures relative distance between input instances and imposes a constraint of placing these into relative distances. It could be possible that fewer degrees of freedom are useful for aligning target network with trained guides.

We also include the representational dissimilarity (represented as 1 - RSA) in fig. 9. Similar to prior findings with CKA, the representational dissimilarity with an untrained guide decreases more quickly and stays lower relative to the trained guide. However, unlike CKA, this implies a weaker relationship between task performance the representational similarity. The representational dissimilarity loss is much lower for the untrained guide yet the task performance between the two guided networks is similar. This implies that RSA is worse at capturing task performance dynamics and algorithmic alignment in comparison to CKA.

The results above are useful as a method to interpret similarity metrics. Work in the future could compare representational similarity metrics and their correlation with task performance to study whether representational alignment implies algorithmic/task alignment.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 4
questions: 1. Figure 3 shows the pattern of error consistency, is this any different from what would be obtained from distillation? 

2. Related to fig. 3, are the intermediate representations of these two guide models also similar? A similar plot could be made for the intermediate representational similarities which would be very informative.
soundness: 2
strengths: - The observations are interesting and to some degree novel. 
- A wide range of tasks from several domains are tested empirically. 
-  The paper is well written and methods are explained comprehensively.
summary: This paper proposes guidance, a method where a well-performing guide network directs the layer-wise representations of a target network, transferring inductive biases without modifying the architecture. The technique aims to improve training for architectures traditionally prone to issues like overfitting or underfitting, such as fully connected networks and plain CNNs. Initial results suggest that guidance can improve performance in various settings, but further validation across tasks and more rigorous testing would clarify its robustness and broader applicability.
weaknesses: 1. The main observation of the paper is that the performance of suboptimal architectures can be improved if their intermediate representations are matched to a more optimal guide models. This is an interesting observation but somewhat a natural outcome of what we already know.  
	
	Functionally, representation matching to a guide network allows deeper layers of the target networks to receive useful gradients for learning that they normally do not receive from the task loss. We know that in deep networks w/o skip connections, auxiliary loss functions applied on intermediate layers help with learning [1]. These losses were proposed to mediate the very same issue examined here, that deep CNNs suffer from the vanishing gradient. Relatedly, models with skip connections like ResNets can also be viewed as ensemble of shallow networks as skip connections allow gradient to pass through the deep layers [2]. Suggesting that what skip connections in one way do is to allow many parallel pathways inside the model for letting the gradients flow throughout. I don't see how what is proposed here is fundamentally different from what was show in the prior literature. One way to make this claim more grounded is to show that representation matching outdoes auxiliary losses applied on earlier layers. In general, distillation is a critical missing baseline in all the results. 
	
	[1] Szegedy, Christian, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. "Going deeper with convolutions." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9. 2015.
	
	[2] Veit, Andreas, Michael J. Wilber, and Serge Belongie. "Residual networks behave like ensembles of relatively shallow networks." Advances in neural information processing systems 29 (2016).


2. A second primary observation in the paper is that even when the representation is matched to that of an untrained guide network it leads to substantial improvements in the performance of the target network. While this is an interesting observation, specially in cases where this scenario surpasses the matching of the trained network, the portrayal of the results are inaccurate. The main issue lies in considering the untrained model as incapable of performing the task better than chance. In Table 2, the accuracy of the untrained models are reported as being consistently around chance, which is expected if none of the parameters are trained. However, in the context of this paper which uses network representations as guide, the informative measure would be the accuracy of a linear classifier trained on the penultimate layer features. This value is typically much higher than chance and I expect it to be so here as well which attests to the usefulness of the features at their initial state. 
Apart from this, there are more things to be done in order to at least attempt to offer a plausible explanation of this phenomenon. For example, what are the geometrical differences as a result of matching to the untrained and trained networks? Presumably, since the trained model is better able to distinguish between imagenet classes, matching its representations, if successful, should have been more useful for the target network. But somehow that's not true. 

3. The observed higher utility of the untrained guide network becomes more interesting specially when considering the fact that the matching loss term is included throughout the training. One possible explanation could be that the untrained network geometry is simpler to replicate for the target network while the fully trained model could have a much more clustered representation that is too difficult for the target to learn from. This could be tested by using various guide networks at different stages of training and tracking performance when matching to each, then examining any relationship between geometrical properties of the latent space and performance of the target network.

4. Figure 4 results are interesting in showing that even limited training on the matching loss is still helpful. Would the target network its full accuracy in Table 1 if training is done in two phase where the first phase will fully train the network on the untrained guide representations (weight initialization), and phase 2 would do the task training? I think this experiment is important for supporting the claim about finding a good initial weight in these models using the representation matching idea. 

5. At parts, the writing is very verbose. 6.5 pages of the paper is dedicated to the intro, background, and a methods section that contains only modest novelty. Sections 1 and 2 should be written much more concisely to make room for additional experiments. Some of the details of the experiments could also be moved to the appendix. 

6. There are a few papers that are conceptually very similar to the proposed ideas in this paper but were not cited 

https://arxiv.org/abs/1412.6550

https://arxiv.org/abs/1808.01405

7. The last paragraph of introduction talks about the limitations, would fit better at the end of the paper (e.g. in the discussion)

8. Line 116: "To that end, we also did not optimize networks to convergence," this sounds like a serious limitation that could be avoided. If the networks are not trained optimally some of the conclusions may turn out being incorrect. E.g. the observed differences between matching the trained and untrained models. 

9. Line 257: "We describe the task settings": remove 

Overall, this paper showcases very interesting observations but falls short at providing concrete insights. The paper in its current format reads premature. The methods are not novel, the observations are interesting but no real insight is offered. This is study has many potentials but they are not met yet and so I don't think it is ready for publication.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: I am interested in discussion on both of the weaknesses that I have proposed. Overall I am positive on this paper, but I feel that if the above weaknesses were addressed, it would improve the paper.
soundness: 3
strengths: - Paper is very well written and even enjoyable to read.
- Technique is novel and proposes a new direction in which the field of distillation can go. 
- Technique could revitalise 'dead' neural architectures which failed to take off due to weak performance.
- Experimentation on multiple modalities.
summary: This paper proposes a novel distillation technique called guiding which distils the inductive priors of the 'guide' network into the target network. The paper shows that by using a known task strong network as the guide, a target network which is known to be weak at the specific task can be significantly improved in terms of performance. The paper then proposes that the guiding network can be disconnected from the process at a very early stage so that it simply acts as a initialisation technique for the network.
weaknesses: - I would have liked to see results on smaller datasets. I feel that this could have resulted in a higher number of ablations to determine different configurations of the network e.g. should we be connecting multiple layers to a single layer in the target? Should guide network have the same number of layers as the target? Do all layers in the target need connections to the guide?
- I would have liked to see more in depth discussion of the ViT to CNN/MLP experiments which are hinted at in figure 3.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: * Are the trained guide networks pretrained models that are then fine-tuned during the guidance process?
* Are the untrained guide networks trained in parallel to the target network, or are they frozen during training? If they are not frozen, what is the advantage of training both networks simultaneously rather than focusing solely on training the guide network?
* In Figure 1, when mapping different layers of the guide network to the same layer of the target, it may be important to consider the similarity between these different guide layers.
* How might the results differ if an alternative metric is used to calculate the similarity?
* What impact would guiding only the final layers of the target network have on the results?
* In Table 3, the RNN achieves 100% accuracy and is used as a guide network. What does this imply? Given that the work aims to train untrainable networks using well-performing networks, why use a network that appears to be overfitting as a guide for training the target network?
soundness: 2
strengths: * The paper studies an interesting problem trying to solve the problem of untrainable networks
* The authors analyzed multiple scenarios with different tasks and architectures
* The authors provide the supplementary materials with demo notebooks
summary: This work presents a simple method for training neural networks that are typically considered untrainable. The approach, similar to the teacher-student methodology, involves training a "target" network with the assistance of a "guide" network, which can be either trained or untrained and may have a completely different architecture from the target. The objective is to align the target network's internal representations as closely as possible with the guide ones in terms of CKA similarity.
weaknesses: **Writing and Presentation**: The paper could benefit from improved clarity and readability. Some suggestions to consider: 

* The paper includes expressions that may introduce ambiguity or complexity (e.g. lines 30,42) potentially making it harder for readers to grasp the authors' points fully. Rephrasing these expressions with clearer language would better convey the challenges discussed and enhance accessibility.

* The tables are somewhat difficult to interpret, and it’s unclear if certain entries serve as baselines. For example, Table 2 could be restructured to distinguish between guide and target, rather than using the "experiment" column, and highlight when training is performed without teacher guidance. Making tables and feature descriptions more self-contained and better explained would enhance clarity.

* The Method section could be more clearly organized and written. It is challenging to follow, and restructuring or rephrasing certain parts would likely improve reader comprehension.

* In the Related Work section, the "representation similarity" literature is missing (e.g. 2,4-8). Adding this section could support several claims, such as the one on line 145. 

**Novelty**: The paper’s novelty is limited, as it overlooks relevant related work that addresses similar challenges. For example:

* A similar work exists, such as [1], where authors leveraged relative representations [2] in a teacher-student framework, training the student network to mirror the teacher network’s latent representations.

* The authors’ claim in the conclusion regarding the absence of known methods for improved network initialization is not entirely accurate. For instance, [3] proposed a method for initializing smaller models by selecting subsets of weights from a larger, pretrained model, thereby transferring knowledge from the larger model to smaller architectures.

* Additionally, the experiments lack a comparison with the classic teacher-student setting, which would provide a useful benchmark.

**Contribution**: While the results show some improvement, they also highlight ongoing challenges in making these networks fully trainable. For example:

* In Table 2, the results show an accuracy improvement from 7.5 to 13.10. However, no standard deviation is provided, which limits the interpretation of these results. Moreover, an accuracy of 13.10 is not competitive on this dataset for the image classification task.

* The choice of the networks raises questions, as they do not represent state-of-the-art (SOTA) architectures. It would be interesting to explore whether using a pre-trained network to guide another competitive network could yield further improvements. 

--- 

[1] Ramos, Patrick, Raphael Alampay, and Patricia Abu. "Knowledge Distillation with Relative Representations for Image Representation Learning." International Conference on Computer Recognition Systems. Cham: Springer Nature Switzerland, 2023.
[2] Moschella, Luca, et al. "Relative representations enable zero-shot latent space communication." ICLR 2022.
[3] Xu, Zhiqiu, et al. "Initializing models with larger ones." ICLR 2023.
[4] Huh, Minyoung, et al. "The platonic representation hypothesis." ICML 2024.
[5] Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. arXiv preprint arXiv:2010.15327, 2020.
[6] Shengkun Tang, Yaqing Wang, et al. You need multiple exiting: Dynamic early exiting for accelerating unified vision language model. CVPR 2023.
[7] Zorah Lahner and Michael Moeller. On the direct alignment of latent spaces. In UniReps 2023.
[8] Valentino Maiorca, Luca Moschella, et al. Latent space translation via semantic alignment. NeurIPS 2023.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - It's not crucial to the reception of the paper, but I'm curious if the authors explored the evolution of the CKA similarity across layers comparing early versus later layers throughout guidance akin to [https://proceedings.neurips.cc/paper_files/paper/2014/file/375c71349b295fbe2dcdca9206f20a06-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2014/file/375c71349b295fbe2dcdca9206f20a06-Paper.pdf) 
- I'm also curious whether the authors explored any other similarity metrics besides linear CKA.
- Perhaps I missed it, but where are the experiments with guidance between architectures with a different number of layers described on lines 230-235?
- Did the authors consider weighing the two loss terms proposed in equation 1?  
- Not necessarily a weakness, but I'm curious whether the authors explored transferring equivariant architectures for tasks with known symmetries.
soundness: 3
strengths: - The proposed idea is intuitively presented and I believe could serve as a useful tool for studying network initialization schemes as well as scientifically probing the role of architectural inductive biases.
- I appreciate the authors investigating how to bridge our understanding of the role of architecture by providing a principled mechanism for transferring (some) of the architectural biases of one network architecture to another. 
- Empirically the authors selected a diverse set of tasks with reasonable baselines and choice of target versus guide networks. I appreciate how thorough the authors are disclosing training details, hyperparameters, and the intent of the experiments. I also appreciate the error analysis conducted across architectures and the error bounds for experimental results. 
- Authors draw a distinction between typical distillation approaches and the proposed guidance approach that I find clear and well-motivated.
summary: The authors propose an approach for aligning representations across two different deep learning architectures. One architecture serves as the "teacher" (providing guidance) to the "student" (target network). For example, one architecture could a convolution whereas the other could be a transformer. 

Specifically the authors propose training the target network with an additional loss term penalizing the dissimilarity between the target and fixed guidance network. The authors measure the layer-wise dissimilarity using the complement of linear CKA similarity. 

The authors study four tasks (across vision, language, and arithmetic) to illustrate how their approach can improve performance by guiding an ill-suited architecture for the task with a more suitable architecture for the task using their alignment approach.
weaknesses: - The field has mostly converged on the transformer architecture for most tasks. While this doesn't make the idea presented less interesting scientifically, I'd encourage the authors keep this in mind when motivating the approach as today, practitioners are not frequently choosing across many choices of architectures. I find initialization and the scientific value of understanding the role of architectural bias still valuable and would encourage the authors emphasize these angles instead. 
- The gains for ImageNet are still quite small overall, suggesting we perhaps can't simply transfer over inductive biases or that much much more tuning is needed. Do the authors have any comments on this point? At a minimum, I'd recommend the authors acknowledge and discuss this in the results section.
- Inconsistent gains across trainable versus untrainable without clear intuition as to why this might be happening. I appreciate the author's attempt at clarifying this in the paragraph starting on line 373, but I still feel this section is missing crucial intuition to help readers make sense for the empirical findings across the tables. For example, why would an untrained ResNet-18 provide better guidance relative to a trained ResNet-18 to a Deep FCN as shown in Table 2. This is puzzling to me and am curious if the authors have additional experiments or intuition to better explain this in the context of the rest of the results.
- The authors make several claims about the revolutionary potential of the method that are not well-supported with evidence. While I'm also excited about the proposed method and potential future work that builds on it, I'd suggest the authors downtone what right now appear as spectulations in this draft. For example, lines 85-86 claim the proposed approach "expands the space of viable networks," a claim I believe is not well-supported by this work. Similar claims are made about potential future promises about guidance on lines 108-112 that are not well-supported by the experiments in the work.
	- Section 4 could be improved. You emphasize a distinction between untrained architectures (versus untrained tasks)—defining Untrainable Architectures as target networks difficult to train irrespective of task. In the experiments that follow however,  the focus is very much on whether a given target architecture is ill-suited for a specific task.

Meta-review: (a) summary

This paper investigates how to use a guide network to train a target one traditionally overfitting(FCN) or underfitting(CNN). It performs representation alignment between the guide and target DNN by introducing additional loss terms. The initial results suggest that guidance can improve performance in various settings.

(b) strengths
+ The observation that the performance of a student model can be improved if their intermediate representations are matched to a teacher model, is interesting and is novel to some degree.
+ It investigates the role of network architecture inductive bias for bettering understanding of neural networks.
+ It conducts extensive experiments over a wide range of tasks and analysis.

(c) weaknesses
- The proposed method has limited novelty due to its similarity to teacher-student setting. There are a few papers that are conceptually very similar to the proposed ideas in this paper but were not cited
https://arxiv.org/abs/1412.6550
https://arxiv.org/abs/1808.01405
- It is not easy to verify the correctness of the claims due to some missing baselines (distillation and auxiliary losses tied to intermediate layers).
- It lacks concrete insights into architectural inductive biases based on the proposed approach.
- The presentation is not polished.

(d) decision

Although the paper has potential for better understanding inductive bias of DNN, it is not ready for publication in its current form due to its limited novelty, missing experiments, and unpolished presentation. Please keep the reviewers' comments in mind when preparing a future version of the manuscript.

Decision: Reject