Training the Untrainable: Introducing Inductive Bias via Representational Alignment
Authors: Vighnesh Subramaniam
David Mayo
Colin Conwell
Tomaso Poggio
Boris Katz
Brian Cheung
Andrei Barbu
Venue: NeurIPS 2025 poster ()
Keywords: Representational alignment
neural network optimization
Primary area: deep_learning
TL;DR: We design a method to make untrainable networks trainable using representational alignment.

Abstract: We demonstrate that architectures which traditionally are considered to be ill-suited for a task can be trained using inductive biases from another architecture.  We call a network untrainable when it overfits, underfits, or converges to poor results even when tuning their hyperparameters. For example, fully connected networks overfit on object recognition while deep convolutional networks without residual connections underfit. The traditional answer is to change the architecture to impose some inductive bias, although the nature of that bias is unknown. We introduce guidance, where a guide network steers a target network using a neural distance function. The target minimizes its task loss plus a layerwise representational similarity against the frozen guide. If the guide is trained, this transfers over the architectural prior and knowledge of the guide to the target. If the guide is untrained, this transfers over only part of the architectural prior of the guide. We show that guidance prevents FCN overfitting on ImageNet, narrows the vanilla RNN–Transformer gap, boosts plain CNNs toward ResNet accuracy, and aids Transformers on RNN-favored tasks. We further identify that guidance-driven initialization alone can mitigate FCN overfitting. Our method provides a mathematical tool to investigate priors and architectures, and in the long term, could automate architecture design.

Training the Untrainable: Introducing Inductive Bias via Representational Alignment
Vighnesh Subramaniam's David Mayo', Colin Conwell', Tomaso Poggio', , Boris Katz', Brian Cheung', Andrei Barbu If
MIT CSAIL, CBMM DDpaarment of Cognitive Science, Johns Hopkins University {[vsub851, dmayo2, tp, sprisiss, , cheungb, (abarbu)@mit. cconwel20jhu. edu
Abstract
We demonstrate that architectures which traditionally considered to be ill-suited for a task can be trained using inductive biases from another architecture. We call network untrainable when overfits, underfits, converges to poor results even when tuning their hyperparameters. For example, fully connected networks overfit on object recognition while deep convolutional networks without residual connec- tions underfit. The traditional answer is to change the architecture to impose some inductive bias, although the nature of that bias is unknown. We introduce guidance, where a guide network steers a target network using neural distance function. The target minimizes its task loss plus layerwise representational similarity against the frozen guide. If the guide is trained, this transfers over the architectural prior and knowledge of the guide to the target. If the guide is untrained, this transfers over only part of the architectural prior of the guide. We show that guidance prevents FCN overfitting on ImageNet, narrows the vanilla RNN-Transformer gap, boosts plain CNNs toward ResNet accuracy, and aids Transformers on RNN-favored tasks. We further identify that guidance-driven initialization alone can mitigate FCN overfitting, Our method provides a mathematical tool to investigate priors and architectures, and in the long term, could automate architecture design. Project website at ftttp::s//uttrainaale tee--networ networks. gi github. io
1 Introduction
When creating neural networks, as community, we follow recipes that select among a few archi- tectures that are known to work for particular tasks (63 12 25 Architecture is critical, encoding essential inductive biases i.e. priors that profoundly impact their learning capabilities and perfor- mance across various tasks. Convolutional nets revolutionized vision [4] and Transformers reshaped language 774 20 M. Despite this, architecture design is a dark art because the precise relationship between architectures and the priors they impose is poorly understood. For example, there is discussion about exactly what the role of residual connections is [3)]. This reflects a broader challenge: we rarely understand exactly what inductive biases our architectures encode. Our lack of understanding makes architecture design challenging. Given new application spaces for neural networks with rising compute costs like inference- time scaling [56), this challenge has become more relevant.
Recent theorems (59 state that for each function which is efficiently Turing computable, there exists a deep network that can approximate well. Furthermore, graph representing such a function
Corresponding author.
I Equal senior contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Guidance: Optimization via Representational Alignment
Input Image
0.5
Guide Network
Gorn Layse
Barehhoo
Redd
Go Laysr
Banshhorm
Corn Laysr
Batt NivesNoam
Pooiin
9.0
Valteaiion
8.0
Training
Imput Image
Uncar Layer
Nomm
Liae Layar
Borchtoa
Output
7.0
Target Network
Guided Training 13.10%
Figure 1: Guidance makes untrainable networks trainable via representational similarity. Given target which cannot be trained effectively on task, we train this target with layerwise representationallalignne term against a fixed guide-trained or random-which remains unchanged during task training. This transfers only the guide's 'ss architectural bias, turning a network that would otherwise overfit or underfit into one that learns effectively (e.g.. deep FCN guided random ResNet for image classification).
compositionally sparse, that is the nodes of the associated Directed Acyclic Graph (DAG) represent constituent functions with a small effective dimensionality. A reasonable conjecture that neural networks with an architecture which is similar to the DAG of the unknown target function are especially successful in learning it, as it is the case for convolutional networks for image recognition and similar tasks. However, empirically testing transferring those structural priors remains open challenge. Because we do not understand the relationship between the kinds of priors on the target functions that different architectures impose, even simple questions have no known answer. For example, can an FCN's initialization be tailored to mimic CNN's inductive bias, despite their distinct graphs?
To bridge this gap, we introduce a novel empirical tool, guidance. Given a target network, we guide it with guide network. In addition to the target's original loss, the target attempts to match the representation of its intermediate layers to those the guide. We use a measure of representational similarity
45 16], also termed neural distance function, to compute the distance between representations of two arbitrary layers. Neural distance functions are often used in neuroscience to compare activity in networks and brains (68 14 71]. In light of recent work that shows that networks of very different architectures have internal activity that is extremely similar to one another means transfer priors between networks layer by layer. Surprisingly, even a randomly initialized guide-incapable of solving the task-yields large performance gains, proving architectures alone encode powerful priors. This surprising finding demonstrates that neural architectures alone, independent of parameter training, impose meaningful inductive biases that are useful for downstream tasks.
30 14 15), repurpose this distance function
We make the following contributions:
1. We develop guidance to transfer priors between networks using representational alignment and investigate one representational alignment method, centered kernel alignment, CKA
455
2. We empirically differentiate between architectural and trained inductive biases, showing that architectural priors alone can significantly improve network performance. This underscores the intrinsic structural power of architectures independent learned parameters.
3. We show that RNNs significantly improve their ropy-and-pass aad-ppate accuracy when guided by Transformer. Transformers increase their parity accuracy when guided by an RNN. RNNs close much of the gap a Transformer on language modeling when guided by one.
4. We show that deep or wide fully connected networks stop overfitting when guided by a ResNet. No-skip CNNs close much of the gap with ResNets when guided by ResNet. Fully-connected networks stop overfitting with guidance-only initialization schemes.
Our method provides powerful empirical tool to further theoretical insights in architectural design. Guidance enables systematic investigation of the structural foundations of successful architectures and clarifies the distinction between architectural and trained inductive biases. Minimizing CKA lets

---

you clamp any subset of a target network's 's activations onto those of frozen guide network, sweep that clamp across architectures and priors, and observe what fails or succeeds. Guidance knob you can turn to inject architectural priors at will, something one cannot do with cross-entropy or weight-decay alone.
Our work has a number of limitations. We aimed for coverage of many tasks instead of maximal performance on any one task. This would have required us carefully tune the hyperparameters involved. We preferred to show how guidance works in general rather than in cherry picked and carefully tuned settings. To that end, we also did not optimize networks convergence, nor did we attempt to experiment with other optimizers. Once we reproduced well-known problematic training phenomenon, we showed that it could be overcome. We consider network trainable and a problem to be overcome when the original problem disappears. For example, successfully training fully connected networks for object recognition was hopeless because they immediately overfit; using our guidance method they no longer do so. This does not mean that they are necessarily useful as object recognizers at present. In the case of fully connected networks, their present performance with guidance training is too low, but with additional work we believe their performance could be substantially increased now that their train and test loss are moving in the right direction. In some cases, by applying guidance, we do see large useful improvements, such as with RNNs and Transformers, as well as deep CNNs, although much more remains to be exploited there too.
2 Related Work
Representational Distance: Our method builds on several metrics that measure distance between
nigh-dimensional activations extracted from neural networks or activity in the brain 44 Some of these distance metrics make comparisons based on kernel matrices 45 17 16 or relative distances 46 (55
between sample representations in a set. Others compute linear 75 68 or orthogonal
projections [5] from one set of representations to another. These metrics are designed based on set of desired invariant properties such as permutation invariance or invariance to linear transformations.
Such approaches have been commonly applied in neuroscience for measuring representational distance of activations from networks and activity in the brain to understand which neural networks are architecturally most similar to the brain 775 14 71 (24]. Under this context, Han et al. [30] has shown the inability of current representational distance metrics specifically the metric used here, centered kernel aligmment to distinguish representations based on architecture. This paper provides the foundation for our intuition that networks may have similar representations that allow for transferring inductive biases from one network to another.
This work examines RNNs and transformers for sequence modeling and FCNs and plain CNNs for image classification. Prior work explored similar approaches but performed poorly compared the guide networks we leverage for improved training.
Untrainable Networks:
In sequence modeling, classical RNNs 69
58
13
29] were constrained by vanishing and exploding
gradients [35,, making them unsuitable for long sequence tasks requiring memorization (27]. Gradient flow techniques were developed, but significant progress came from architectures like LSTMs [34] and transformers (74.. Transformers, however, have been found untrainable on formal language tasks requiring full- -el-eeeueecee reasoning. where RNNs succeed [8]
For image classification, small feed-forward networks with 3-5 hidden layers and fewer than 100 57] These efforts prioritized Strategies to reduce overfitting, such as topological structure [67] or early stopping 19. were hindered by complex designs and hyperparameter tuning, leading to poor training fits. Further methods used alignment between thin deep FCNs and wide shallow FCNs to prevent overfitting, an approach similar to our paper (64]. Deep convolutional networks were also applied to image classification 47 but struggled with vanishing gradients, limiting their depth.
units per layer were trained on object recognition datasets [53
training fit over generalization performance and achieved low results 5 6
Model Distillation: Guidance shares resemblance with model distillation 33 26 66
36 Distilla-
tion transfers knowledge from a teacher model to student model by introducing new component to the loss function that enforces the student model to behave like the teacher model 42 76. This usually consists of penalizing the KL-ddivergence between the logit predictions of the student and teacher model.
3

---

Representation-bude based distillation [72 10 50 and alignment techniques have been proposed to im- prove alignment between two networks. Certain works have proposed correlation congruence or similarity preserving metrics 62] for aligning two networks, particularly
way to do architecture search between CNNs 4 Methods have been proposed that use CKA as an alignment approach be- tween representations of two networks or with representations in the brain with notable improvement in network performance [8].
We distinguish guidance from distillation. Guidance can use a smaller untrained guide instead of a larger trained teacher. This due to guidance operating over intermediate activations of the network instead the output of the network probabilities or output features, like distillation does. Guidance also operates at many levels at the same time, aligning many layers at once. This helps address the credit assignment problem that gradient descent has when tuning weights early in a network. We also consider many more networks for guidance than traditional for distillation including networks which have very different architectures like Transformers to RNNs. Distillation is usually carried out between two closely related architectures. We apply guidance to do the opposite.
3 Methods
Guidance introduces term in the loss of target network, .V, to encourage representational alignment with a guide network, A/G. We update only the target's parameters, 0r, while keeping the guide's parameters, 8C, frozen. On each minibatch, we compute similarity metric M (e.g., CKA) between guide-layer activations A and matched target activations A AT. We refer to the correspondence between layers of the guide { and layers of the target {i"} as I. While this correspondence, could be complex as any two architectures can form a guide/target pair, here we choose architectures that make the correspondence obvious as is discussed later. For example, the stacked RNNs and Transformers have the same number of layers in our experiments.
The target and guide receive the same input. Per minibatch, we collect activations from intermediate layers of both networks. Layers of guide network mapped to layers of the target network; see fig. [ We formulate the loss in terms of minimizing the representational dissimilarity, M,, i.e., the complement of a representational similarity metric, between guide and target activations layer by layer, summing the results. Here we use linear CKA, though any differentiable similarity metric could plug into Eq. (I). Efficiency or incremental computation much more important than it is in traditional applications since this operation happens for every minibatch. We discuss details in appendix [
Given Lr as the original loss of the target network, the guide network's original loss function is irrelevant. The guide could be pretrained on another task. In fact, it need not even have been trained at all, only its architecture shapes the target. This latter setting is what allows transferring architectural priors without transferring knowledge from the guide to the target, as there is none in randomly initialized guide. See eq. ① for an overall loss. We discuss details in appendix B
L(B"²) = (r("2)+ + > MMA (0 AC
(1)
iET
Equation (1) minimizes task loss while increasing alignment between the target and guide networks given the mapping between them. The mapping may be sparse; not every layer needs to be used. This important for guidance with transformers stacked RNNs, as will be explained later. Note that the guide's 's parameters, 0² are constants, i.e., the guide is never updated.
Metries like CKA capture and encode inductive biases in neural network computations. For instance, CKA is a measure that depends on second-order statistics, specifically pairwise sample distance matrices. Architectural choices imprint distinct features on those statisties. For example, consider local receptive fields in a convolutional layer. Units that cover neighhoring pixels receive correlated input, and this is reflected in activations. Such correlations matrices, and these can be transferred to distance matrices associated with FCN layers that lack local correlations. Similarly, weight sharing, where the same kernel is applied at every spatial location, will also be reflected in a distance matrix.
reflected in our distance
Layerwise Mapping We design a simple method for mapping guide layers to target layers as part of providing supervision. The goal of this method is to make guide and target networks architecturally agnostic i.c. we can supervise any target network with any guide network.

---

Tasks
Guide Networks
Target Networks
Copy-Paste
Transformer
RNN
Parity
RNN
Transformer
Language Modeling (Small and Large)
Transformer
RNN
Deep FCN
ResNet-18
Image Classification
Wide FCN
ResNet-50
Deep ConvNet
Table 1: Guide and target networks across tasks. Our network designs include several untrainable target networks and corresponding trainable guide networks.
As a simple approach, we evenly spread layer computations of our guide network over our target network. For example, if consider ResNet-18 and a 50-layer FCN, would map every convo- lutional ResNet layer to every second or third linear layer of the FCN. Intuitively, evenly spacing guide-to-target matches encourages the target approximate the guide's compositional functions. Through the design of evenly spreading layers of our ResNet-16 18, we are guiding the FCN to find a function similar to the guide network.
For our mapping, we consider activations from all tunable-weight layers (convolutional, linear, or RNN/LSTM), For multiple stacked RNNs, LSTMs, or transformers, extract feature representations from intermediate layers in the stack as well. Using all layers is useful for guidance as it provides strong signal induce alignment between the target and guide networks during training. We empirically find that more layers leads to stronger results. Skipping layers based on non-linear transformations reduces memory overhead associated with storing representations per batch.
4 Experiments
We design several settings with different target and guide networks to thoroughly test our approach. We include a range of image and sequence modeling tasks. In choosing target networks, we consider broad range of designs for networks that are not traditionally applied (e.g., a FCN in image classification).
To systematically evaluate approach, incorporate two settings. (1) Untrainable Architec- tures: Experiments where the target networks are difficult to train due to architectural limitations, irrespective of the task. For example, memory incorporation in RNNs or overfitting in deep FCNs. (2) Untrainable Tasks: Experiments where certain tasks are inherently challenging for specific architec- tures, making them untrainable without additional supervision. For example, sequence classification with transformers.
Tasks: We describe the task settings. We consider three sequence modeling tasks to allow for broader range of architectural settings. We first consider a task called copy-paste [27]. In this task, generate sequence of numbers in the range of 10. The model is trained to recover the same sequence in the output. In our setting, we consider sequence lengths that range from 20 to 40 values total (internal sequence and padding). We generate a copy-paste dataset, sampling sequences containing numbers between and 10. We generate a total of 100, 000 examples, training on 80, 000 examples, validating 10, 000 examples, and testing 10, 000 examples.
We also include the parity task, a binary classification task where a model is fed a bitstring and outputs when there is an even number of ones in the bitstring and 0 otherwise. We generate series of bitstrings with sequence lengths that range from to 50 as done in prior work (8].
Finally, we consider language modeling task using the WikiText- 103 dataset [54] where models must predict the next token given some context. This uses the train, validation and testing splits defined by the WikiTeett dataset and for all experiments, we use context length of 50. We tokenize the text data using the GPT-2 61 tokenizer.
For an image-based task, we focus on image classification and use the ImageNet- K dataset [19 for training and testing. We use the splits defined by the dataset. We report accuracy the validation set for all experiments.
5

---

Architectures: For all tasks, we describe our target untrainable architectures for each task separately as well as the guide networks that are employed to make the untrainable network trainable. We give an overview in table [
We provide further details in appendix C
Sequence Modeling: For our copy-paste task, we use vanilla, 4-layer RNN our target network. In copy-paste, architectural and algorithmic limitations make RNNs an untrainable architecture. For our language modeling task, we include two settings with a small (4 layer) and large (6 layer, larger hidden dimension) RNN. In this setting, vanishing gradients and limited context incorporation make RNNs an untrainable architecture as the training loss saturates. For the parity task, we use a 1-layer transformer encoder architecture, similar to prior work 88 28). For the copy- task, we train a guide network, 4-layer transformer decoder model which achieves 96.90% accuracy. Similarly, for language modeling, we train 4-layer transformer decoder guide network with a context window of 256. Our final perplexity is 34.15 15 for the small language modeling setting and 33.10 for the large language modeling setting. For the parity task, we train a 1 1-layer vanilla RNN as a guide network which achieves 100% accuracy as reported by [
Image Classification: We use three target networks: Deep FCN, Wide FCN, and Deep ConvNet. Deep FCN a fully-connected network with 50 blocks consisting of feedforward layers followed by non-linearities. This network is an untrainable architecture, lacking inductive biases to prevent overfitting and having vanishing gradients. Wide FCN is fully connected network with 3 blocks with feedforward layers that have 8192 units. This is categorized as an untrainable task due to a saturation in the training performance. Deep ConvNet the same architecture as ResNet-50 50 BI, but without residual connections. This is categorized as an untrainable architecture due to the vanishing gradient problem. We use two guide networks: ResNet- 18 and ResNet-50. ResNet- 18/50 is a deep convolutional network with 18/50 convolutional blocks and residual connections. We refer to He et al.
[31. We supervise the Deep FCN and Wide FCN with ResNet-18 and supervise the Deep ConvNet with ResNet-50.
Training: For each setting, we train the base target network and perform an experiment where both a trained and untrained guide network supervises the base target network. All networks are trained with cross-entropy loss, without loss of generality. For all sequence modeling tasks, i.e. copy-paste, parity, and language modeling we use Adam W [52]. For language modeling, we also incorporate gradient clipping due to unstable training with long sequences. When training networks for image classification using ImageNet-1K, we use the Adam 43 optimizer.
To ensure consistency of comparisons across learning curves, Representational similarity metrics are affected by the number of samples in the calculation, where more samples allows for the metric approximate representational distance better. We use 256 as a proxy, dependent on GPU memory, although more memory would allow for bigger batch sizes with potentially better results. Due to the large number of training settings, we employ several different learning rates. We tune the learning rate carefully for baseline training to ensure maximal performance. We sweep the parameter across 5 different values and choose the results with the lowest validation loss. This ensures we are choosing the training with the best performance.
a consistent batch size of 256.
After choosing the optimal learning rate, we then train all networks and settings for 100 epochs with 5 random seeds to compute error bars. Our error bars are associated with the standard error across each step across all seeds. We choose the seed-based average test accuracy associated with the epoch with the lowest seed-based average validation loss.
5 Results
Sequence Modeling: On the copy-paste task, guiding a 4-layer RNN with a Transformer improves copy-paste accuracy by over 25%. See fig. f and [ studies blamed RNN failures on vanishing gradients and memorization limits. Our results show a potential optimization scheme for RNNs that is applicable for sequence memorization. Remarkably, random (untrained) Transformer guide outperforms a trained one, suggesting pure architectural bias drives gains. We believe this is because optimization with randomly initialized networks is easier due to the degrees of freedom in CKA. See appendix E We plan to explore this more thoroughly in further analyses. These gains persist under our layerwise ablations and metrics; see appendix K and appendix G
On the parity task, a 1 lllyyer Transformer guided by an RNN improves its test accuracy by 7%. This is complementary result to copy- eapyee and language modeling where the guide network was

---

Copy-Paste
Parity Accuracy (t)
Language Modeling (Small) Perplexity())
Language Modeling (Large) Perplexity (1)
Experiment
Accuracy (t)
RNN
14,35 + 0.01
100
69.19 + 1.89
89.13 + 2.00
Untrained RNN
2.32
Transformer
96.98
71.98 + 3.16
34.15 5.19e5 90.44
33.10 5.19e5 + 90.44
Untrained Transformer
1.04 +: 0.81
RNN + Transformer
78.49 + 2.16
Untrained RNN + Transformer
70.38 4.17
Transformer -> RNN
23.27 + 1.02
40.01 1.54
36.91 + 1.04 47.17 1: 2.50
Untrained Transformer + RNN 42.56 1.51
59.61 2.33
Table Guidance improves performance for sequence modeling. RNN performance improves dramatically when aligning with the representations Transformer copy paste, as well as language modeling with small and large RNN architectures. RNNs close most of the gap to Transformers for language modeling and are likely competitive with further scale. Transformers in turn, improve parity performance when aligning with RNN. Guidance is able to transfer priors between networks.
Experiment
ImageNet Top-5 Validation Accuracy (f)
ResNet-18
89.24
Untrained ResNet-18
0.24 + 0.043
ResNet-50
92.99
Untrained ResNet-50
0.54 = 0.029
Deep FCN
1.65 + 1.21
ResNet-18 -> Deep FCN
7.50 1.51
Untrained ResNet-18 + Deep FCN
13.10 0.72
Wide FCN
34.09 0.91
ResNet- 18 Wide FCN
43.01 0.92
Untrained ResNet-18 Wide FCN
39.47 1:2 0.31
Deep ConvNet
70.02 1.52
ResNet-500 - Deep ConvNet
78.91 2.16
Untrained ResNet-50- - Deep ConvNet
68.17 2.54
Table 3: Guidance improves performance
image classification. Alignment with a ResNet dramatically improves a deep FCN, particularly with an untrained ResNet. Significant gains are seen with wide FCN as well, Deep CNNs without residuals gain only with trained ResNet. Across all settings, guidance can help train architectures that otherwise considered unsuitable.
transformer and target network was RNN. This improves results from several prior papers 8 that have pointed out fundamental limitations of transformers to perform formal language tasks.
Unlike copy-paste, the performance improves when using trained RNN the guide network. This could be due to the wide gap in performance between an untrained RNN and trained RNN on parity. Parity uniquely benefits from learned positional encodings in the trained RNN, which the Transformer lacks. This information is likely crucial to the transformer, which has limited sequence pooling capacity and fewer degrees of freedom.
language modeling, similar to copy-paste, guided RNNs halve the perplexity from -70 points to 35 points WikiText- 103, closing in on Transformer baselines. While performance generally saturates for the 4-layer RNN, guidance continuously improves the RNN performance by over 30 points for text perplexity for both trained and randomly initialized guide networks. Scaling up to 6-layer RNN further cuts perplexity by 10 points, indicating guidance scales with model size. This implies that information from the transformer can be transferred to the RNN. We also believe that this has exciting implications for scaling laws with RNNs. We see a similar trend with a randomly initialized transformer as the guide network, implying that architectural priors in the transformer driving improvement guided network performance.
Image Classification: Guidance boosts validation accuracy by 5-10% across our Deep FCN, Wide FCN, and Deep ConvNet; see table[3] We also observe significantly better loss curves from a better fit with the training loss and reduced overfitting with the validation loss. An untrained ResNet- 18 guide
7

---

Yeae Nigs
Traning Meps
Partyy, Transtorme
Language Modsing (Small).
Mode (Largs), RMN
Con
2ing Steze
Thaiing Steps
Thaniing StR5e
Figure 2: Training and validation under guidance for all experiments reported in table 1 For every result
Table Table we show the training and validation loss with error bars across multiple runs, although these are often too small to see. Note that often the best results occur with the untrained guide.
1.0
Base FCN
0.036
0.039
0.0029
0.0013
ResNet-18 18 Guided FCN
0.036
0.49
0.047
0.027
0.6
VIT.B Guided FCN-
0.039
0.49
0.035
0.02
0.4
ResNit-18 0.0029
0.047
0.035
1
0.53
0.2
VITB 0.0013
0,027
0.02
0.53
1
Reateer Guidedd FCN
Basa FCN
VITB Guided PCN
Restee-18 18
vina
Figure 3: Guidance aligns error consistency, The relationship between the guide networks is mirrored in that of the guided networks, even when the target is entirely unlike the guides initially. This is additional evidence that guidance doesn't just improve performance arbitrarily; the target becomes more like the guide.
outperforms its trained counterpart Deep FCN, underscoring pure architectural priors like with copy-paste For example, the Deep FCN results in the top left of fig. are are significantly better with a randomly initialized ResNet- 18 as the guide network instead of trained ResNet- 18. This trend also occurs with Wide FCN. We show results with other neural distance functions in appendix !
The Deep ConvNet (no skips) gains only from trained ResNet-50 smide-implyin residual con- nections require learned weights to shape representation. This explanation provides an additional interpretation for the role of residual connections and their influence on the representation space. This indicates that residual connections must be trained to have an influence on the representation space. This aligns with prior studies of residual connections 39 32
Guided FCNs mirror the ResNet- ViT error overlap, proving they inherit the guide's 's decision patterns. Using Deep FCN as our target model, we guide it with a ResNet- 18 or
Error Consistency:
V6T-B-16 21. We then measure the error consistency [23] between all of the networks; see fig. 3
The error consistency between the initial FCNs is entirely unlike the ResNet- 18 or ViT-B. Guidance creates two FCNs which have the same relationship to one another. The ResNet- 18-guided FCN
and ViT-B-guided FCN have the same error consistency with respect to one another as ResNet-18
8

---

Guidance Initialized Cross-Entropy Loss
9.5
Deep FCN Training Loss
9.0
Guidance Init Training
Deep FCN Validation Loss
8.5
Guidance Init Validation Loss
Cross-Entropy Loss
8.0
7.5
7.0
6.5
6.0
1000
2000
3000
4000
5000
6000
Training Steps
Figure 4: Initializing fully connected nefworks with guidance can overcome overfitting, First, we align Deep FCN to random ResNet-18 18 on noise for 300 steps, then train normally. This two-stage scheme mirrors full guidance, and leads similar performance improvement. This suggests that FCNs have guidance-inspired initializations that avoid overfitting.
and ViT-B do. It's not just that the FCN gets generically better; it adopts prior from the original architecture. We provide further details of the error consistency metric in appendix F
Network Initialization: Is guidance needed throughout training, or is the effect of the guide move the target into a regime where the two are aligned and the target can be optimized further without reference to the guide? The answer to this question can shed light on whether the guide is telling us that better initializations are likely to exist for the target. To answer this question, we minimize the representational dissimilarity between our target and guide network for nominal number of training steps, 300. Then we apply task training on the resulting target network with no guidance. Pre-aligning FCN layers to random guide for 300 steps stops overfitting entirely no ongoing guidance needed; see fig. Furthermore, while preventing overfitting, we have lower training loss from guidance, indicating better fit. This implies that there exists better initialization for FCNs.
We comparison between guidance and knowledge distillation { in fig- 5) We find that guidance improves significantly over distillation, particularly when the teacher network is untrained. This is significant, demonstrating that guidance can exploit untrained networks for transferring inductive biases. This indicates that matching internal representations provides much stronger signal over just matching output behavior. We discuss further appendix H
Guidance Distillation:
Conclusion
We demonstrated that guidance eliminates the failure modes of networks previously considered unsuit- able or ineffective for specific tasks. Aligning with another network overcomes these shortcomings by transferring inductive biases-either architectural and knowledge-based, or solely architectural when using untrained guide. This allows guidance to distinguish tasks and architectures that are dependent on architcctural biases rather than learned biases. We provide further explanations and intuition for guidance in appendix [ There are many potential aspects that may distinguish between the effects architectural and inductive biases in improving performance of these architectures, which we aim to explore in future work.
This also opens the door to many applications. Our method can be used study representational and functional design of neural networks in new ways to reanalyze prior theory of neural network optimization. For example, we can understand distances between architectural components based on which target networks are easier to guide with a particular guide network. We also refine this notion include narrow channel through which guidance can occur, the representational similarity. This can serve kind of probe.

---

Guidance vs Distillation
Hetted
Tuhnod to Untrannd
Imagelet Accurscc
20
Accurscy (1
Figure 5: Guidance outperforms distillation: We include comparison between guidance and distillation for all settings with trained and untrained guide networks/teacher networks. We find that guidance outperforms distillation settings, highlighting that, unlike guidance, distillation fails in settings with an untrained teacher.
We believe guidance also reveals a new conceptual lens inductive biases and architectures. When we use an untrained guide network, we reveal what an architecture by itself brings to the table. Guidance with trained guide reveals how much learned representations These are different phenomena whose root causes are not understood at present but that could elucidate the relationship between priors and architectures.
change performance.
Our results suggest practical applications by significantly narrowing the performance gap between vanilla stacked RNNs and Transformers and showing signs of scaling, albeit in small-scale exper- iments with 150M parameters or fewer. Given that stacked RNNs are equivalent to single-layer RNNs, most directly delayed RNNs (73), this implies that complex modifications to RNNs may be unnecessary language modeling. For other networks, like fully connected ones, we only overcame the initial obstacle. Further research is needed to refine these into effective vision models, they avoid immediate overfitting. In the future, we hope to find methods for making these networks competitive with their guide networks.
Guidance also proved to tool with which to discover the possibility new initializations, At the moment, no known method exists to find better initializations for networks. In some cases as with the FCNs for vision, guidance can be disconnected after nominal number of steps, but still goes on to regularize the target network. This strongly implies that an initialization regime for that target with same regularization exists. This is all that guidance could do that case. We now need tools to go backwards, given networks which are correctly initialized and networks which are not, discover what that initialization is. This is a much better place to in. systematic sweep of targets and guides to look for better initializations should be carried out.
Looking into the long-term future, guidance invites us treat architecture itself trainable prior added directly to generic network's 's loss. Because guidance can rescue models that previously overfit or underfit, we can revisit designs abandoned during neural architecture search. These threads demonstrate the major possibility for future work in this space. Guidance is a tool, not finished and well-understood theory or doctrine. Tools are useful for enabling more discoveries. We believe that these discoveries will allow the community to more easily pursue questions that training difficulties might have prevented in the past, especially with regard understanding the relationships between architectures.
10

---

Acknowledgements
This work was supported by the Center for Brains, Minds, and Machines, NSF STC award CCF- 1231216, the Brains, Minds, and Machines Summer School, the NSF award 2124052, the MIT CSAIL Machine Learning Applications Initiative, the MIT-IBM Watson AI Lab, the CBMM- -iee-nnmmnns Graduate Fellowship, the DARPA Mathematics for the Discovery of ALgorithms and Architectures (DIAL) program, the DARPA Knowledge Management at Scale and Speed (KMASS) program, the DARPA Machine Common Sense (MCS) program, the Department of the Air Force Artificial Intelligence Accelerator under Cooperative Agreement Number FA8750-19-22 Force Office of Scientific Research (AFOSR) under award number FA9550-21-1-0300 and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed implied, of the Department of the Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. V.S. and D.M. are supported by the National Science Foundation Graduate Research Fellowship under Grant No. 214 3141064. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
and the Air The views
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Ehsan Amid, Rohan Anil, Wojciech Kotlowski, and Manfred K Warmuth. Learning from randomly initialized neural network features. arXiv preprint arXiv:2202.064333 2022.
[3] Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vincent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment. In International Conference on Artificial Intelligence and Statistics, pages 2269-2277. PMLR, 2021.
[4] Pouya Bashivan, Mark Tensen, and James J DiCarlo. Teacher guided architecture search. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5320-5329, 2019.
[5] André Beauducel. Recovering wood and mccarthy's s erp-prototypes by means of erp-specific procrustes-rotation. Journal of Neuroscience Methods, 295:20-36, 2018.
[6] George Bebis and George Papadourakis. Object recognition using invariant object boundary representations and neural network models. Pattern Recognition, 25(1):25-44, 1992.
[7] Yoshua Bengio, Yann LeCun, and Donnie Henderson. Globally trained handwritten word recognizer using spatial representation, convolutional neural networks, and hidden markov models, Advances in neural information processing systems, 1993.
[8] Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. On the Ability and Limitations of Transformers to Recognize Formal Languages. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7096-7116, Online, November 2020. Association for Computational Linguistics. doi: wwnlp-main........eemmmmmmm
URL https://
wwanthhooggy.rrr/20000 wttp://000eemmmmmmm.........
[9] Rich Caruana, Steve Lawrence, and Giles. Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. Advances in neural information processing systems, 13, 2000.
[10] Liqun Chen, Dong Wang, Zhe Gan, Jingjing Liu, Ricardo Henao, and Lawrence Carin. Wasser- stein contrastive representation distillation. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pages 16296-16305, 2021.
[11] Emily Cheng. Diego Doimo, Corentin Kervadec, luri Macocco, Jade Yu, Alessandro Laio, and Marco Baroni. Emergence high-dimensional abstraction phase in language transformers. arXiv preprint arXiv:2405.1 2024.

---

[12] Shuang Cong and Yang Zhou. A review of convolutional neural network architectures and their optimizations. Artificial Intelligence Review, 56(3):1905-196,, 2023.
[13] Jerome T Connor, R Douglas Martin, and Les E Atlas. Recurrent neural networks and robust time series prediction. IEEE transactions on neural networks, 5(2):240-254, 1994.
[14] Colin Conwell, David Mayo, Andrei Barbu, Michael Buice, George Alvarez, and Boris Katz. Neural regression. representational similarity, model zoology & neural taskonomy scale in rodent visual cortex. Advances in Neural Information Processing Systems, 34:5590-5607, 2021.
[15] Colin Conwell, Jacob S Prince, George A Alvarez, and Talia Konkle. What can 5.17 billion regression fits tell about artificial models of the human visual system? In SVRHM 2021 @orkshop@ NeurIPS, 2021.
[16] Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research, 13:795-828, 2012.
[17] Nello Cristianini, John Shawe-Tayler Andre Elisseeff, and Jaz Kandola. On kernel-target alignment. Advances in neural information processing systems, 14, 2001.
[18] Joel Dapello, Kohitij Kar, Martin Schrimpf, Robert Geary, Michael Ferguson, David D Cox, and James DiCarlo. Aligning model and macaque inferior temporal cortex representations improves model-to-hune to-human behavioral alignment and adversarial robustness. bioRxiv, pages 2022-07, 2022.
[19] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee, 2009.
[20] Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 2018
[21] Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929,, 2020.
[22] Mingyu Fan, Nannan Gu, Hong Qiao, and Bo Zhang. Intrinsic dimension estimation of data by principal component analysis. arXiv preprint 1002 2010.
[23] Robert Geirhos, Kristof Meding, and Felix A Wichmann. Beyond accuracy: quantifying trial- by-trial behaviour of cnns and humans by measuring error consistency. Advances in Neural Information Processing Systems,
2020.
[24] Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, et al. Thinking ahead: sponta- neous prediction context as a keystone of language humans and machines. BioRxiv, pages 2020-12, 2020.
[25] Ian J Goodfellow, Oriol Vinyals. and Andrew M Saxe. Qualitatively characterizing neural 2014.
network optimization problems. arXiv preprint
[26] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 17866)118--1189, 2021.
[27] Alex Graves. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014.
[28] Michael Hahn and Mark Rofin. Why are sensitive functions hard for transformers? arXiv
preprint arXiv:2402.09963 2024.
[29] Barbara Hammer. On the approximation capability of recurrent neural networks. Neurocomput- ing. 31(1-)):177-123 2000.
[30] Yena Han, Tomaso A Poggio, and Brian Cheung. System identification of neural systems: If we got it right, would we know? In International Conference on Machine Learning, pages 12430-12444. PMLR, 2023.
[31] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[32] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer VisVon-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 11-14, 2016, Proceedings, Part IV 14, pages 630-645. Springer, 2016.
12

---

[33] Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint arXh::1503.02531
2015.
[34]
Hochreiter. Long short-term memory. Neural Computation MIT-Prees, 1997.
[35] Sepp Hochreiter. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Baad Based Systems, 6(02):107-116, 1998.
[36] Cheng-Yu Yu Hsich, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujüi, Alexander Ratner, Ranjay Krishna, Chen- Yu Lee, and Tomas Pfister. Distilling step-by-step! outperform- ing larger language models with less training data and smaller model sizes. arXiv preprint arXiv:23005.22 023011, 2023.
[37] Ehsan Imani, Wei Hu, and Martha White. Representation alignment in neural networks. arXiv preprint arXiv:21 12.07806, 2021.
[38] Francesco Insulla, Shuo Huang, and Lorenzo Rosasco. Towards a learning theory of representa- tion alignment. arXiv preprint arXiv:2502. 2025.
[39] Stanislaw Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. arXiv preprint arXiv: 1e 10.04773, 2017.
[40] William B Johnson, Joram Lindenstrauss, and Gideon Schechtman. Extensions of lipschitz maps into banach spaces. Israel Journal of Mathematics, 54(2):129-138, 1986.
[41] Anwesha Khasnobish, Arindam Jati, Garima Singh, Saugat Bhattacharyya, Amit Konar, DN Tibarewala, Eunjin Kim, and Atulya K Nagar. Object-shape recognition from tactile images using a feed-forward neural network. The 2012 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2012.
[42] Tachyeon Kim, Jachoon Oh, Nak Kim, Sangwook Cho, and Se- Young Yun. Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation. arXiv preprint arXiv:2/05.0891 2021.
[43] Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint
1rXiv=1412.6980, 2014.
[44] Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich. Similarity of neural network models: A survey of functional and representational measures. arXiv preprint
arXiv:2305.0633 2023.
[45] Simon Korblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519-3529 PMLR, 2019.
[46] Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:249, 2008.
[47] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[48] Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. arXiv preprint arXiv: /804. 08838, 2018
[49] Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indrnn): Building longer and deeper rnn. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5457-5466, 2018
[50] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in neural information processing systems, 33: 2351-2363, 2020.
[51] Yujun Lin, Song Han, Huizi Mao. Yu Wang. and William J Dally. Deep gradient com- pression: Reducing the communication bandwidth for distributed training. arXiv preprint arXiv: 1712.0188 2017.
[52] I Loshchilov. Decoupled weight decay regularization. arXiv preprint
2017.
13

---

[53] Liying Ma and Khashayar Khorasani. Facial expression recognition using constructive feed- forward neural networks. IEEE Transactions on Systems, Man, and Cybernetics, Part (Cybernetics), 34(3): 2004.
[54] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.
[55] Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209. 15430, 2022.
[56] Niklas Muennighoff, Zitong Yang. Weijia Shi, Xiang Lisa Li, Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025.
[57] II-Seok Oh and Ching Y Suen. A class-modular feedforward neural network for handwriting recognition. pattern recognition, 35(1):2292244, 2002.
[58]
Barak A Pearlmutter. Dynamic recurrent neural networks. 1990.
[59] Tomaso Poggio and Maia Fraser. Compositional sparsity of learnable functions. Bulletin of the
American Mathematical Society, 2024.
[60] Phillip Pope. Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. arXiv preprint arXiv:2104.08894 2021.
[61] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
[62] Patrick Ramos, Raphael Alampay, and Patrieia Abu. Knowledge distillation with relative representations for image representation learning. In International Conference on Computer Recognition Systems, pages 133-143. Springer, 2023.
[63] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM 2021.
Computing Surveys (CSUR), 54(4):
[64] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv: 1412.6550, 2014.
[65] Aninda Saha, Alina Bialkowski, and Sara Khalifa. Distilling representational similarity using centered kernel alignment (cka). In Proceedings of the the 33rd British Machine Vision Conference (BMVC 2022). British Machine Vision Association, 2022.
[66] Sanh. Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter. arXiv preprint arXiv.1910001108. 2019.
[67] Christian Schittenkopf, Gustavo Deco, and Wilfried Brauer. Two strategies to avoid overfitting in feedforward networks. Neural networks, 10(3)500--516, 1997.
[68] Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, page 407007, 2018.
[69] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(11):2673-261 1997.
[70] Haozhe Shan and Blake Bordelon. A theory of neural tangent kernel alignment and its influence on training. arXiv preprint arXiv:2105. 14301, 2021.
[71] Vighnesh Subramaniam, Colin Conwell, Christopher Wang, Gabriel Kreiman, Boris Katz, Ignacio Cases, and Andrei Barbu. Revealing lision-language integration in the brain using multimodal networks. In International conference on machine learning. PMLR, 2024.
[72] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv: 10699, 2019.
[73] Javier Turek, Shailee Jain, Vy Vo, Mihai Capotă, Alexander Huth, and Theodore Willke. Approximating stacked and bidirectional recurrent architectures with the delayed recurrent neural network. In International Conference on Machine Learning, pages 9658. PMLR, 2020.
14

---

[74] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[75] Leila Wehbe, Ashish Vaswani, Kevin Knight, and Tom Mitchell. Aligning context-based statistical models of language with brain activity during reading. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pages 233-243, 2014.
[76] Wangchunshu Zhou, Canwen Xu, and Julian McAuley. Bert learns to teach: Knowledge distillation with meta learning. arXiv preprint arXiv:2106. 2021.
15

---

A Appendix Overview
We present additional details of guidance, experiments and analysis, as well additional results. In Appendix B we provide additional details for our guidance approach, with a full explanation of we provide additional details on our architectural designs and network training. In Appendix[D D we introduce new experiment where we feed noise to our guide network rather than real data and compute representational alignment, leading to similarly improved results. This further establishes transfer of prior rather than knowledge. In Appendix E we show visualizations of the representational dissimilarity loss over training to give context of dynamics over training and show additional explanations for results with trained and randomly initialized guides. In Appendix [F] we provide further explanation of error consistency as a measure of functional similarity between networks. In Appendix G provide test accuracy metrics over training as complementary of network performance over training outside cross-entropy loss. In Appendix
centered kernel alignment.
Appendix [
provide comparative baseline to guidance, distillation 33 We find that basic distillation performs worse in comparison. In Appendix[]] we provide an interpretation and analysis of guidance to better characterize and understand results based on changes in the internal geometry of network before and after guidance. In Appendix [] we apply guidance with additional neural distance functions including RSA [46) and ridge regression. We find correlation between the success of our approach and degrees of freedom of a particular distance function. In Appendix KK we provide additional ablations over guidance such as guiding a certain number of layers or guiding specific layers of a network.
B Methods Overview
We give an overview of guidance in algorithm|] and highlight crucial changes to base neural network training in either red or blue. We use blue to indicate the collection of network activations and red to indicate the layerwise mapping and representational alignment using a distance metric. This gives an overview of our layer mapping between the target and guide network. Crucially, we find that the simplest layer mapping where we evenly distribute guide network layers across target network layers for supervision obtains strong results.
Algorithm 1 Guidance: Guide Network Representational Alignment
Require: Target network; R with parameters 8" Guide network N Dataset D {(x,, 1.) Repre- sentational Distance Metric Mt Loss function L N do
1: for j
2: # Base training with vanilla loss function
3: outputs + _ (x,)
4: loss +- (o My )
5: # collect layer activations
6:
{A - + activations (2,))
7:
( (A%
11 Ha + activations(A (x,))
8:
# Get step size hetween the number of layers between the two networks for layer mapping.
9:
ifl > then
10:
step (t 1)/(1-1
12:
step +
13:
end
14:
# Map the layers and add up layer-wise representational distance
total +-
for
do
17:
index + (round(u ( step), 1)
18:
rep M(A T AC
19:
total total rep
20: end for
21: loss loss total
end for
16

---

B.1 Centered Kernel Alignment
To compare representations, we use representation similarity metric, M, which corresponds to setting. We specifically consider linear CKA.
centered kernel alignment (CKA) 45 16 17
CKA uses kernel functions on mean- centered representations to compute representational similarity matrices, which are then compared via the fillber--chhiit Independence Criterion (HSIC). More specifically, suppose we have two sets of representations E 1xxdd and R' E 12xda, We first compute the Gram matrices for each set of representations
K = RR L R'R"
(2)
We center the Gram matrices by introducing a matrix, H, where H Ib- -11
K K HLH
(3)
We compute the HSIC the Gram matrices.
HSIC(K, tr(K,t)
(4)
Finally, we define our linear CKA metric as:
HSIC(K, L) M(R :² CK /HSIC(K, K) HSSC L)
(5)
In our setting, we consider representational dissimilarity and aim to minimize the dissimilarity between representations from target network and guide network. We define this
M(R, =1- R),
(6)
Linear CKA ranges from (identical representations) (very different representations). Because of this, we take the complement by subtracting the linear CKA from to represent dissimilarity.
B.2 Methodology Limitations
Our guide network supervision through representational alignment one primary limitation due to increased memory usage during training. Due to saving activations across several layers of the two networks, GPU memory usage increases dramatically. Moreover, our methodology works better batch size increases since this allows for better approximation of representational similarity, increasing memory usage even more. Furthermore, including more layers for supervision leads to improved results.
In this paper, we introduce simple techniques to handle memory constraints such as gradient ac- cumulation and gradient checkpointing [51]. In practice, more memory optimization techniques may become necessary consider larger untrainable networks. Further work could consider using stronger representational alignment strategies to reduce the number of samples necessary to achieve strong fit.
C Architecture and Training Details
C.1 Architectural Design Details
For all tasks, we describe our target untrainable architectural designs for each task separately as well the guide networks that are employed to make the untrainable network trainable.
17

---

C.1.1 Copy-Paste
Target Networks
RNN: We design a 4-layer RNN with a hidden dimension of 768 units, followed by fully connected layer. In copy-paste, architectural and algorithmic limitations make RNNs an untrainable architecture for the task. Specifically, RNNs must memorize the input sequence which difficult, particularly with padding token. RNNs are generally considered to be inapplicable to the copy-paste task.
Guide Networks
Transformer:
We consider a 4 layer transformer decoder architecture with a hidden dimension of 768 units across 12 transformer heads. The transformer is well-suited for copy- copy-paste the attention mechanism can act as a routing mechanism for the sequence. We train the transformer guide from scratch, with language modeling and achieve 96.90% accuracy on the task.
C.1.2 Parity
Target Networks
Transformer: Similar to Bhattamishra et al. [8 we design a layer transformer encoder network with hidden dimension of 64 units across 4 attention heads. Transformers have lower accuracy on formal language tasks that require reasoning over sequence in comparison to traditional sequence models (88 Due to the enormous gap performance and saturation of performance, we categorize the transformer as untrainable.
Guide Networks
RNN: We include a layer vanilla RNN with hidden dimension 64 units. Similar Bhattamishra et al. 8 we achieve 100% accuracy on the task.
C.1.3 Language Modeling
We include two language model settings to test scaling in RNNs. The first setting uses small networks, which we refer to as Small RNN and small Transformer. The other uses a large RNN and large Transformer.
Target Networks
Small RNN: We design 4 layer vanilla RNN with hidden dimension of 512 and with ReLU activation function. We train this on sequences with context length of 75. This makes the network untrainable due to problems associated with exploding and vanishing gradients during backpropagation through time.
We design 6 layer vanilla RNN with a hidden dimension of 1024 and with a ReLU activation function. We train this on sequences with a context length of 128. Prior work has demonstrated that larger RNNs are difficult to train in practice [49].
Large RNN:
Guide Networks:
Small Transformer: We design layer transformer decoder network with 16 attention heads and hidden dimension of 512. We train the transformer on Viki with context length of 256 and achieve a final test perplexity of 34.15.
Large Transformer: We design a 4 layer transformer decoder network with 16 attention heads and hidden dimension of 1024. We train the transformer on WikiText with context length of 256 and achieve final test perplexity of 33.10.
C.1.4 Image Classification
Target Networks
Deep FCN: We design a fully-connected network consisting 50 blocks. Each block contains feedforward linear layer, batch normalization, and ReLU nonlinear activation. The intermediate feedforward linear layers contain 2048 units. This network untrainable due vanishing gradients since the network is very deep and due to overfitting.
18

---

Tasks
Experiment
Learning Rate
RNN
1 x 10-1
Transformer
1 1 10
Copy-Paste
Transformer RNN
1 10
Untrained Transformer - RNN
1x 10
Transformer
1 10-3
RNN
1 10
Parity
RNN + Transformer
1 1 10-3
Untrained RNN + Transformer
1x 10-3
Small RNN
1 1 10
Small Transformer
1 1 10
Small Transformer Small RNN
1 x 10
Untrained Small Transformer -+ Small RNN
1 X 10 44
Language Modeling
Large RNN
1 X 10
Large Transformer
1 X 10
Large Transformer Large RNN
1 10
Untrained Large Transformer + Large RNN
1 10
Deep FCN
1 10-4
Wide FCN
1 X 10
Deep ConvNet
1 1x 10-3
ResNet-18 - Deep FCN
5 10
Image Classification
Untrained ResNet-18 18 + Deep FCN
5 10-5
ResNet-18 - Wide FCN
1 1 4
Untrained ResNet-18 + Wide FCN
1 X 10-4
ResNet-50 + Deep ConvNet
1 X 10-3
Untrained ResNet-50 Deep ConvNet
1 x 10-3
Table 4: Learning rates for network training. For all networks, we sweep over 5 learning rate values before choosing the learning rate with the lowest validation loss for training. Our training does not use learning rate scheduling such warm-up up scheduler although such techniques may improve results.
Wide FCN: We design a network similar to Deep FCN but only containing 3 blocks where each feedforward linear layer contains 8192 units. This network considered untrainable due saturation on the training performance.
We design deep convolutional network with the same architecture as ResNet-50 (convolutional layers followed by batch normalization) but remove the residual connections. This makes the network untrainable due to the vanishing gradient problem as observed in He et al. [1]. causing saturation of the loss.
Deep ConvNer:
Guide Networks
ResNer-18/50: A deep convolutional network with 18/50 convolutional blocks and residual connec- tions. We refer to He et al. 31
We supervise the Deep FCN and Wide FCN with ResNet-18 and supervise the Deep ConvNet with ResNet-50.
C.2 Training
In Table 4 we show the different learning rate settings we converged to in each experiment, For each experiment, we did grid search over 5 different learning rate parameters to ensure optimal learning rate setting. We did careful tuning of all training of target networks to ensure maximum performance.
19

---

Training Cross-Entropy
Validation Cross-Entropy
7.0
Training
RosNet
9.0
6.8
8.5
Cross Entroyy Loss
6.6
8.0
7.5
7.0
6.2
1000
3000
20
Traniing Stops
Training Epochs
Figure 6: Feeding noise prevents overfitting. We introduce an additional experiment where we feed noise guide network rather than the same batch during each training step with guide network guidance. We sample noise from a Gaussian distribution with mean of and a standard deviation having no information about the images in the batch, the guide network still provides an inductive bias to prevent overfitting. While this noise increase the training loss, this shows true transfer of an inductive bias that is not driven by pure distillation of similar features.
We find that despite
For all image classification tasks, used the Adam optimizer [K], For all sequence modeling tasks, we use A [52] which has been useful in training sequence models like RNNs and Transformers 61].
in-line with prior work [1I.
The training experiments in this paper were completed across H100s and 4 A100 GPUs for weeks in total. GPU optimization techniques taken such as gradient accumulation and gradient checkpointing and some language modeling experiments used mixed-precision training.
D Representational Regularization: Guidance with Noise
We also aim understand the role of the guide network as in guidance. In all experiments, we use trained and untrained guide networks and see consistent improvements for training the target network. The success of untrained networks implies that our training method is not performing distillation but instead truly transferring a prior from the guide network to the target network. To more strictly test this theory, we include an experiment where we feed noise to the guide network instead of the same batch of data fed to the target network implied by eq. (I).
We apply this experiment to the Deep FCN with an untrained ResNet-18 the target network. At each training step, we pass noisy batch which is sampled from random Gaussian with mean of 0 and standard deviation of 1. We train for 100 epochs and report the learning curve results in fig. 6
This result confirms our intuition about the role of guide network: as a guide on model priors rather than pure distillation of information. While the overall cross-entropy loss magnitudes are higher and the overall accuracy is lower when passing noise to the guide network, our results are significantly better than applying vanilla training approaches to the Deep FCN.
E Representational Similarity Loss
We can view the representational alignment between the guide and target networks during train- ing. This allows us to better understand how this representational alignment influences network performance. We show sequence modeling results in fig. and image classification results in fig.
20

---

Copy Fasto, RMN
Parity, Transformer
Language Modzling, RNN
4.0
Thired Transfomerr Gutte
naaned Guise
Taired Thanffomer Guide
Trveffimer
Transfarmer
1.0
14-
0.-
10-
13
0.0-
0.-
10-
0.6-
230
150 2000 1250 1500
230
150 2000
1250 2300
15000 20020 20100
Trasing
Figure 7: CKA representational similarity loss for sequence modeling tasks. We visualize the total CKA dissimilarity loss across all layers across training for all three sequence modeling tasks. The CKA dissimilarity loss represents the representational alignment between our guide network and target network. We can observe that for the copy-paste task and language modeling task, the target network aligns with a randomly initialized network more quickly. This could be because of special properties of RNNs.
Deep FCN
Wide FCN
Deep ConvWet
Traned Beshet-30 Gulde
Untrained Ro Guile
Tramed Reduee-11 Guite
Untroined Mo Guale
20-
Untroined RosMllla Guide
Trannd Restue--0 Gotte
12
2.0
15-
10-
10-
2002 2002 5000
4000 3000
8000
2000 2000 2000 4000 2000 6200
2002 2000 2000 4000 3000 8200
Traniig
Figure 8: CKA representational similarity loss for image classification. We visualize the total CKA dissimilarity loss across all layers across training for image classification. The CKA dissimilarity loss represents the representational alignment hetween our guide network and target network. We can observe that for Deep FCN and Wide FCN, the target network aligns with randomly initialized network more quickly. This corresponds with results where randomly initialized guide networks had superior performance to trained guide networks.
We notice that across most tasks, reducing representational dissimilarity is easier with activations from randomly initialized networks rather than trained networks. This provides additional evidence of representational alignment for inductive bias transfer. We notice that for certain cases, such as Parity, the randomly initialized guide network has higher representational dissimilarity loss than the trained guide network. This is matched with the Parity result in table 2 and fig. 2
However, we can also observe more inconsistent results with the Deep ConvNet where the untrained guide network has lower representational dissimilarity loss than the trained guide network, even at the end of training. One possible explanation that the inductive bias was more similar for Deep ConvNet and ResNet-50. This means that trained features are more important for better Deep ConvNet results and representational alignment with a trained network is important. This result has interesting implications for understanding the role of residual connections. Since untrained ResNet-50 is easier to align with than trained ResNet-50, this demonstrates that residual connections influence representation spaces during training. The untrained residual connections have little influence on the inductive biases of the network or the overall representation space. This demonstrates the strength of our method as a way to interpret neural network design choices and how they influence representation and functional aspects of a network.
These results are also potentially indicative of architectural properties of RNNs and FCNs which match randomly initialized networks more quickly. For instance, one potential explanation is that RNNs have more degrees of freedom [8] and therefore, only need inductive guidance rather than trained features. Transformers may require learned features indicating that the bottleneck for transformers on the parity is not algorithmic but feature-based. Much of the future work can use these results to design better networks with more informed designs.
21

---

Early Deep FCN Layer Simil arity
Deep Layer Similarity
0.76
0.74
Mornh w
0.72
My mym 1
0.73
0.66
0.70
0.62
0.69
2,000 3000
3000
Training Steps
Thairing Steps
Figure 9: CKA dissimilarity decreases more in earlier network layers than later layers. When separate the CKA dissimilarity across layers of target network, we find that earlier layers optimize more and earlier. We take two layers from a Deep FCN during guidance with a randomly initialized ResNet-18. The early layer comes from the 15th FCN block. The later layer comes from the 43rd FCN block. We see that both layers are eventually optimized but the later layer receives less supervision and has higher CKA at the end of training.
E.1 Layerwise Analysis
We provide a deeper analysis of patterns of the representational dissimilarity across different layers during guidance in fig. [ We find that earlier layers generally have higher CKA similarities with their corresponding layer from the guide network and later layers have lower CKA similarities. Furthermore, these later layers optimize later in the training process.
F
Error Consistency
We measure error consistency (x) 23 between the guided target networks which indicates the enror overlap between two networks based on the accuracy of the networks, i.e. the two networks make similar class predictions? The measure first calculates the expected eиror overlap. Suppose is the accuracy of the first guided network and 02 the accuracy of the second. The expected error overlap is given by Coxp a₁ a2 (1 a1) (1 a2). Next, measure the observed error overlap across each sample in the validation set as Cabs of samples where both models agree total trials. Finally, write
Cobs
Cexp
K
Cesp
ranges from -1 to 1, where is perfect agreement, -1 is perfect disagreement and 0 is change agreement. When к > this implies that models make consistent error patterns, implies that models make inverse error patterns, and R implies independent error patterns.
F1 Guide Network Representation Comparison
We contextualize the findings in emor consistency by comparing the representations of the guide networks, in this case ResNet-18 and ViT-B-16. We apply a layer mapping between layers ResNet- 18 and ViT-B-16 and compute the representational similarity over 1000 input images. Results are shown in fig. 10
Our findings are useful for error consistency. If models are inheriting inductive biases from their guide network, then the models would have similar methods to process low-level image features are indicated by stronger CKA in earlier layers between the ResNet- 18 and ViT-B. This means that errors will be consistent for low level features but inconsistent for high level features collected in later layers.
22

---

CKA Similarity between ResNet-18 Layers and VITB Layers
0.8
ClA Similanty
0.6
0.4
02
Layer Index
Figure 10: Earlier layers ResNet-18 and ViT-B are more similar. We analyze the representational similarity between activations from layers ResNet-18 18 and ViT-B-16 via CKA. We find that carlier layers are more similar while later layers have divergent representations. We see that this manifests in distinct error consistency patterns when ResNet-18 and ViT-B are used guide networks.
CopysPwete
Parily
snguage Modeing
0.75
Tained Trnnfoee
Urtainndd Turnfomar Galee
0.70
Yesting Acuracy
Todlig Replosty
0.30
0.25
Ustrained Tonefmmer Guide
1.3
0.03
0.20
0.15
Thining
0.58
Untoaiied ROU Coide
53-
60
20
Touning Fpocs
Toasing Fpech
Trasiing Spoces
Figure 11: Testing accuracy improves across guidance for sequence modeling. We visualize the testing accuracy for sequence modeling example demonstrate that guidance improves accuracy across training this improvement is significantly better across training. This allows for another interpretation method outside of cross-entropy loss.
G Test Accuracy across Training
We plot accuracies over training as a complement cross-entropy loss fig. [1 for the sequence modeling experiments. We can use these experiments to de-couple our results from properties of cross-entropy loss that may lead to misleading improvements across training. We find that accuracies improve consistently across training, supporting the loss curve interpretation that guided training improves results.
H
Basic Distillation Comparison
To show the effectiveness of guidance, we compare it with distillation from 33 Distillation involves transferring knowledge from a performant teacher network to less performant student network via maximizing the alignment of the output logits. This encourages the student to have similar predictions as the teacher network. This occurs via the following loss function. Assume Q is the logits extracted from the target (student) network and the P is the logits extracted from the guide (teacher) network.
(8) where y is the ground truth labels, T is the temperature to soften the logits, and the weighting factor between the distillation loss and cross-entropy loss. In this case, KI refers to the Kullback- I Liebler divergence and a corresponds with the softmax function. practice, we set a 0.5 and T to 2. We continue to track the full cross-entropy loss across training well.
Cassill = 7 * KL(o(Q/TT)l((//T))) + +1 (1-a) CcE(
23

---

Training Loss
Base Deep FCN
Triine Diaillation
RN Distilaaioo
8.5
Cross Entroy uss
7.5
6.5
1000 2000
6000
20
40
100
Training Steps
Traiiing Tpochs
Figure 12: Distillation does not prevent overfitting. We compare basic distillation 33 to see if we can prevent overfitting. We use Deep FCN as student network. We find that distillation with trained ResNet-18 teacher network leads to small improvement in performance but still has some patterns overfitting. Distillation with an untrained ResNet-18 18 teacher network hurts performance.
1.0
Base FON
0.051
0.025
0.0029
0.0013
0.8
Reetve-18 Biettied FON
0.051
0.26
0.6
VTB Distteed PCN
0.4
ls
0.0013 0.0019
Heatle Distitdd distted
FGN
Distlld FCN
Rester
viaa
Figure Distillation lowers error consistency. general, we find that distillation results emror consistency patterns that are less consistent than what is reported with guidance.
We show accuracy-bast bbased results in fig. 5|and loss curve results in fig. 12 Distillation with a trained network can improve performance but much less than guidance. Distillation with an untrained network reduces performance on average, although not by significant amount.
H.1 Error Consistency
We show error consistency performance over distilled networks rather than guided networks in fig.
I Guided Network Analysis and Interpretation
The results from guidance open many questions in order to explain why untrained guide networks can be better at improving target network performance. We provide an intuitive explanation as well geometric analysis of guided networks to see if there is a stronger interpretation.

---

Layerwise comperison of intrinsic di dimension ecross base and guided Deep FCN
Incindl Oimension
Rondom
steps
Deeg FCN luntrainec guidance): 200 steas
Deep FCK: End
FCK luntrained guidance)i End
itrained guidencel: steps
Deep (trained guidencel: End
Layers
Figure 14: Guidance preserves intrinsic dimensionality, avoiding over-regularization. We measure the PCA-based intrinsic dimensionality of representations from each layer of both guided and unguided Deep FCN initialization i.e. Random FCN, 300 steps of training. and the end training. We find that guidance with an untrained guide network better preserves intrinsic dimensionality in comparison to base training or using trained guide. All networks collapse to the same intrinsic dimensions. This establishes that guidance does change the dynamics of training based on geometric features of the target network.
L1 Interpreting Guidance
We believe several prior works support our findings and interpretations in this paper. We cover them here. First, 38 is a recent paper that considers task-aware representational alignment. Their theory provides generalization bound via kernel alignment. They show that when "stitcher" maps representations or a source network to a target output, the excess risk of the stitched model is upper- bounded by the CKA alignment between them. This provides sarming-theoreti guarantee that the CKA term in guidance reduces the hypothesis class possibilities seen by an optimizer. Overfitting or underfitting becomes harder.
Shan and Bordelon 770 investigates how network's neural tangent kernel (NTK) aligns with a target output during training. The paper shows that NTK alignment accelerates convergence and lowers generalization error deep linear networks. This aligned kernel condition is inserted by hand in guidance. Similarly, ( uses Rademacher complexity tools to show that alignment of tangent-kernel features onto a small set of task-relevant directions compresses formalizes the notion of guidance as automatic regularizer, where task directions are replaced by guide network settings. Finally, (37] demonstrates that after training, the top singular vectors of network's hidden activations align with the task target vectors. This empirically supports the layerwise CKA choice in guidance. We believe that CKA bounds the risk or complexity in terms of kernel alignment. The NTK and Rademacher analyses show that alignment shrinks the effective hypothesis space and improves conditioning. This aligns with findings based on singular vectors. We could sharpen this theory by changing the alignment used in guidance e.g. moving from aligning kernels to aligning on singular vectors or eigenvectors instead. A full PAC-style proof specialized to guidance has not been shown in paper but we leave this to future work.
effective model class. This
1.2 Geometric Analysis via Intrinsic Dimensionality
We aim to understand how guidance with randomly initialized guide network differs from a trained guide network. To do so, we compare the representation space of a target network guided by a trained guide network and randomly initialized guide network using intrinsic dimensionality.
Intrinsic dimensionality (ID) refers to minimum number of dimensions required to capture the structure variability in the input data. It represents the true complexity of the data manifold, ignoring noise or redundant dimensions. Previous work has found that neural networks have low ID, capturing data in low- fow-dimensional manifolds [48 60). Following (22), for given threshold 3, the
25

---

intrinsic dimension is the d E N such that the ratio of explained variance for d dimensions of a N dimensional PCA is above 3:
E var(vi) 2ز=1 var(y,)
> B
(9)
First, we measure the ID of both unguided and guided Deep FCNs across all layers at different points of training; see fig. 14 Crucially, at 300 steps of training, we notice that guidance with an untrained guide network preserves the initial II) found in the randomly initialized Deep FCN representations. The trained guide network and base training achieve low ID values, which consistent with findings in prior work [I]] At the end of training, all networks reach the same intrinsic dimension. This finding indicates that the dynamics of training, as shown by ID, change with or without guidance. The Deep FCN without guidance achieves low ID too early in training, and this is likely similar for the guidance with a trained guide network.
One interpretation of these results is that ID has an effect on overfitting in the Deep FCN. When ID is too low, the Deep FCN overfits. Therefore, we can discover a new regularization scheme for training the Deep FCN based on ID). In this scheme we introduce new loss function, which designs differentiable version of PCA-ID) based on a specific ID threshold and forces the of the representations to be above particular ID. In particular, given the activations from specific layer of the target network, A a variance threshold T, and a target ID we first find the SVD of the activations,
A. UEVT
(10)
We extract the cigenvalues using the singular values, Ag = 23 and find the total variance using the cigenvalues.
T= Σ 29 + j=1
(11)
where is the total number of nonzero singular values and is a small constant for numerical stability. Afterwards, we find the explained ratios and cumulative sums of the eigenvalues:
A Pj T
Ck
Σ j=1
(12)
We use the cumulative sum find the loss for being below the target ID in eq.
(13
beloow Σ Ca(3(c-- T))
(13)
where a sigmoid function and 3 controls the sharpness of the loss. Similarly, eq. (4p gives the loss for being too far above the target ID.
L L o(B(T ct))
(14)
Our total loss given by L Labove + feelow. Using this loss control ID, we tune each layer in our Deep FCN and see a final validation loss given in fig. [5] We find that using our new loss function to control the ID of every linear layer our Deep FCN leads to improved validation performance and an accuracy of 14.65%. This aligns with guidance, indicating that guidance may be controlling the ID of the target network. Furthermore, we have used guidance find a new regularization scheme based on intrinsic dimensionality.
26

---

9.5
Base MLP
9.0
ID-Guided FCN
8.5
Validation Loss
8.0
7.5
7.0
6.5
6.0
0
20
40
60
80
100
Epochs of training
Figure Controlling intrinsic dimensionality Deep FCN representations improves image classification performance. introduce novel loss function regularize the ID Deep FCN representations in each linear layer of the network during training. We find that this leads to improved validation accuracy, similar to guidance. This shows that guidance correlates with geometric modifications to target networks and can find new regularizations.
Linear decoding accuracy of untrained ResNet-18 layers
0.8
0,7
Accuracy (%)
0.6
0.
034
2
10 Layer Index
13
17
20
21
Figure 16: ImageNet classes are barely decodable from a randomly initialized ResNet-18, In order to assess the performance of our randomly initialized networks, we design linear decoder to decode ImageNet classes from all layers of the network. Chance accuracy 0.1% 1% (1/1000) the graph above. We find that, while can decode ImageNet classes an accuracy above chance, the accuracy still very low.
27

---

1.3 Linear Decoding
We assess whether object classes are decodable from internal representations of randomly initialized ResNet- 18. A potential explanation for improvements in the target network with an untrained guide is the ability to linearly decode the ImageNet classes with significant accuracy at certain layers of the untrained guide network.
We train linear decoder with 4000 ImageNet images from the train set and test on 1000 images from the validation set for each layer. We show results in fig. [6 We find that the linear decoder never achieves any accuracy greater than 0.8% for any of the layers. Furthermore, later layers contain little information that is useful to linearly decode ImageNet classes. This means that linearly decodable information isn't ' present in the guide network and this aspect isn't driving improvements in target networks. We note that this matches findings in Amid et al. [2], which reports that linear decodability from a ResNet-Ie 18 achieves 3.4% top-1 accuracy. The increase in performance is likely due to using a much larger dataset.
J Guidance with RSA and Ridge Regression
J.1 Representational Similarity Analysis
We use the RSA formulation described in Kriegeskorte et al. 46) Specifically, RSA constructs representational dissimilarity matrices (RDMs) for two sets of representations and compares them using an outer similarity function.
Given two sets of representations, E Rbxd, and R' E Rbxd2, we first calculate RDMs for each set of representations using a distance function d. Formally, we define D E IR as
Da. = sR,, R2)
(15)
Each row D, corresponds to the distance between the representations of input and the representations of all inputs including itself. This is done per-batch, meaning that RSA is sensitive to batch size.
Given two RDMs and D' constructed from sets of representations R and respectively, we vectorize the RDM matrices using function (since the RDMs are symmetric, we only need to compare the lower triangles), and compute the similarity between the two vectorized RDMs using a similarity function
=
(16)
As with CKA, we use the complement of the similarity to construct M. In practice, we define d to be the cosine distance between every pair of inputs and be the pearson correlation between the RDMs as done in previous work [14].
We apply guidance with RSA Deep FCN our target network and ResNet-18 our guide network. Similar to our CKA results, we train for 100 epochs with a batch size of 256, as RSA is sensitive to the number of samples when comparing sets of representations.
J.2 Ridge Regression
We used similar ridge regression formulation [14 71 without cross- validation.
Given two sets of representations, R E ggxd; and R' E gxxda, we first apply a sparse random projection the representations. Since the dimensionality of the representations is prohibitively large, the projection makes the ridge regression feasible to compute. We refer to the resulting representations P and P' which correspond to the projected representations R and R' respectively. P and P have dimension d, where d is fixed using the Johnson-Linden Lindenstrauss lemma 40).
Afterwards, we mean-center the representations and apply ridge regressions using the original least- squared solution as follows. Our goal to predict the representations P' using regressors over P. We first split our representations into training set i.c. Purin P' and testing set Press P where
28

---

Training Cross-Entropy Loss
Validation Cross-Entropy Loss
RSA Dissimilarity Loss
7.0
10-
Uniraired Rechee-18 Guide
Trained
9.0
14-
Cress-Entopy Loss
6.0
12
Entropy
1-RSA Loss
6.6
7.5
6.
6.5
0 1000 2000
3000
4000 5000
6000
20
40
80
200
2000 2000 3000
4000 5000 6000
Training Steps
Training Epachs
Training
Figure 17: Guidance with RSA the representational similarity metric maintains similar performance to CKA. We include further experiment where we change the metric for representational alignment from CKA RSA during guidance training. We apply this to Deep FCN with ResNet-18 18 as guide network. We see that, like CKA, RSA alignment also allows for transferring the prior from ResNet-18 18. However, unlike CKA, the untrained guide network only does marginally better than the trained network, potentially indicating the RSA is better transferring trained features.
the training set contain half the representations and the testing set contains the other half. We first a set of regressors 3 as follows:
B ((P ) ' Pmain + AIa) (Prrin) '
(17)
train
where À is the ridge penalty, which is hyperparameter. The coefficients 3 are then used to predict the held out data where:
P
3
(18)
We measure the cosine similarity between the predicted representations Pics and actual representations Piest-
M(R, R') cosine (Paine(Pi Piest-
Pi
(19)
We apply guidance with RSA Deep FCN as our target network and ResNet- as our guide network. Similar to our CKA results, we train for 100 epochs with a batch size of 256, ridge regression is sensitive to the number of samples when comparing sets of representations. We manually tune the A hyperparameter, finding that X = 10.0 is optimal for the trained guide network and À = 100.0 is optimal for the untrained guide network.
JJ3 Results
RSA: We see results over the training, validation, and representational dissimilarity loss in fig. 17 The Deep FCN guided by trained ResNet-18 achieves an accuracy of 11.02% and the Deep FCN guided by randomly initialized ResNet-18 achieves an accuracy of 11.74%
We first observe that guided training improves over base training noted in fig. 2 and table This demonstrates the generality of our approach to other metrics. As long as a representational similarity metric is differentiable. we can optimize the metric for alignment between two networks as a method to transfer the prior of one network to another.
We can also observe some minute differences between the results with CKA. Most notably, the trained guide network has similar performance to the untrained guide network. This is likely because less information about trained features present in the RSA metric. RSA measures relative distance between input instances and imposes a constraint placing these into relative distances. It could be possible that fewer degrees of freedom are useful for aligning target network with trained guides.
Ridge: We see results over the training, validation and representational dissimilarity loss in fig. The Deep FCN guided by a trained ResNet-18 achieves an accuracy of 9.46% and the Deep FCN
29

---

Training Cross-Entropy Loss
Validation Cross-Entropy Loss
Ridge Dissimilarity Loss
7.0
Trained
9.0
6.8
Uniraired Reshee-18 Guide
Cress-Entropy Loss
8.5
16-
6.6
Cross Entropy
1-Ridge Less
6.4
2.5
2.0
6.2
12-
6.5
0
1000 2000 3000
4000 5000
6000
20
40
80
200
2000 2000 3000 4000 5000 6000 Training
Training Steps
Training Epachs
Figure 18: Guidance with ridge regression as the representational similarity metric improves performance CKA. We change the metric for representational alignment from CKA to ridge regression during guidance training. apply this to the Deep FCN with ResNet-18 as guide network. We that, CKA, ridge regression alignment also allows for transferring the prior from ResNet-18 18, We find that this improves over CKA significantly
Experiment
CIFAR-10 Test Accuracy (f)
Deep FCN
60.58
All layer Guidance
70.15
Last layer Guidance
67.18
Last two layers Guidance
68.03
Last five layers Guidance
67.50
Last ten layers
69.31
First layers
79.22
First five layers
79.58
First two layers Guidance
73.34
First layer Guidance
65.11
Multiple Guide Layers
68.14
Table Guiding earlier layers deep networks leads to better results. apply ablation experiment to identify which layers lead to stronger improvement when guided. We use Deep FCN, guided by a randomly initialized ResNet-18 18 on CIFAR-10. We find that guiding earlier layers leads to strong improvement, even over guiding all layers. Guiding any layer leads to improvement of performance.
guided by a randomly initialized ResNet-18 achieves an accuracy of 15.69%. Similar to RSA and CKA, the guided training improves over base training.
Similar to CKA, we observe that randomly initialized guide networks outperform trained guide networks. Furthermore, performance with ridge regression is better than with CKA. This finding intuitive. Ridge regression generally has more degrees of freedom than other similarity metrics because of fewer invariances imposed on the metric. This means that the solution search space is larger, leading to better results. We believe this provides a promising path forward for making target networks have better performance.
Furthermore, ridge regression has other desirable properties such as potential explainability via probing predicted representations to measure similarity. We can use probing analyses on predicted representations to see what information the target has inherited from the guide network. This opens up many avenues for studying guidance in the future.
30

---

Experiment
Copy-Paste Accuracy (1)
RNN
14.35
All layer Guidance
42.56
Last layer Guidance
38.19
Last three layers Guidance
42.33
Last two layers Guidance
41.55
First layer Guidance
27.59
First two layers Guidance
28.15
First three layers Guidance
33.93
One guide layer
36.11
Table Guidance of later layers improves RNN performance. We apply ablation experiment RNNs trained for copy-paste to see whether guiding certain layers lead improved performance. We find that guiding later layers leads to stronger performance overall. Furthermore, RNN layers are guided by several guide network layers in the transformer such as the linear layer and layer-normalization in the transformer decoder. Including both of these leads to better results.
K Ablation Experiments
We analyze the current design of our layer mapping for guidance by experimenting with the number of layers used in guidance and whether more complex mappings exist like mapping several layers of the guide network single target network.
We run layer-wise ablation experiments the Deep FCN guided by an untrained ResNet-18 over CIFAR-10 10. Similarly, experiment with RNNs guided by untrained transformers over the copy- paste task.
In table 5 we first show the effect of guiding over subset of layers in the Deep FCN, evaluated over CIFAR-10. We find that earlier layers are much more impactful for deep network. Intuitively, this could be due to guidance providing aiding with the credit assignment problem in deep networks: gradients don't propagate properly to earlier layers. However, shoows that later layers in the RNN are more useful to apply guidance to when improving copy-paste performance. In general, we find that guiding any layer leads to improvements in results generally, showing the general applicability.
Furthermore, in table table we consider new methods to map guide network layers to target network layers. When guiding Deep FCN with ResNet-18 18, we only apply 1-1 layer mapping supervision i.e. each Deep FCN layer is guided by only one ResNet-18 layer. From table 5 introducing multiple sources of supervision from guide network layers by allowing one-to-many mapping decreases performance. However, with the RNN, we guide with representations from the linear layer and layer normalization in the transformer decoder. One could consider that linear layers are redundant with layer normalization for guidance, so we remove its representations as potential supervisory target. We find that this hurts performance, showing that RNNs benefit from multiple levels of supervision from its guide.
Understanding the dynamics of guidance supervision is interesting and could allow for understanding training dynamics of neural networks or allow us to form cross-architectrall relationships.
31

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], , [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We introduce guidance as a tool for taking neural networks that are considered ill-suited for specific tasks and making them trainable by aligning their activations with a trainable network like a transformer. We specifically design guidance for a suite of networks like fully-connected networks and recurrent neural networks and show systematic improvements. We state these findings in the abstract and introduction and use these to make claims about deeper understanding neural networks and potential for designing new architectures.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long clear that these goals not attained by the paper.
32

---

2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The paper addresses all limitations at the end of the introduction as well in the appendix. We take care to make point out assumptions of our results but also establishes how we identified failures architectures.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only lested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper is empirical by nature. We do not rely proofs. We do formulate loss functions with particular properties. These loss functions borrow from prior literature such as prior work that use neural distance functions so there are few theoretical derivations necessary.
Guidelines:
The answer NA means that the paper does include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
33

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We upload code associated with all experiments in this paper. Furthermore, we also describe all architectures in detail, include all hyperparameters, used in the paper such as batch size, learning rate, and optimizer, and cover number of training steps. See our appendices or section where we cover our training setting. We run all experiments with open-source datasets that are widely available or can be easily generated.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
new model (e.g., large language model), then there should
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: See previous answer. We upload code and also only work with open datasets or generated datasets. All data is available at this time.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not
34

---

including code, unless this is central to the contribution (e.g., for new open open-souree benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We either datasets with standard train/test splits deseribe in detail how we generate the rraniingvaliidaiionnesingg data for generated datasets like parity or copy-paste.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We train with multiple seeds for all runs of base training and guidance and average over seeds to find the average training performance and standard error. We make all plots with this average performance and standard error for comparison of statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
35

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe the compute (4 H100s) used for this experiment as well as the amount of time needed to get the experiments to work with our code set up.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https::/t/:/tt http:///warrr...cccc/bccceeeeeee
Answer: [Yes]
Justification: We conform with all aspects of the Code of Ethics; our data does not involve human subjects or have privacy concerns. We also do believe this work will have harmful impacts and believe this work will have positive impacts on neural architecture design.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our conclusion is dedicated potential impacts of our paper, with socictal impacts as well.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate
36

---

to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Our paper does not require any safeguards since not in high risk category.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We provide proper citation for the creators of ImageNet and Wikitext- 103. We provide proper citation for the inspiration to use other tasks such as parity and copy-paste.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's 's creators.
37

---

13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce/release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not crowdsource or conduct research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We do not any crowdsourcing nor research with any human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
38

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
We do not use LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
39


Reviews:
[{"review_id": "lfqwfcASpK", "reviewer": "Reviewer_CY31", "summary": "This paper introduces guidance, a novel method to train architectures typically considered untrainable for certain tasks by transferring inductive biases from a guide network. The key idea is to steer a target network using representational alignment (e.g., CKA) with the guide. The target network is trained to minimize both the task loss and a representational similarity loss relative to the guide. The authors demonstrate improvements in training performance across architectures and tasks, for example, improving RNNs using Transformers, and mitigating overfitting in FCNs using ResNets. The paper includes empirical analysis to differentiate architectural and learned inductive biases, highlights cases where guidance helps (or doesn’t), and proposes practical applications such as new initialization strategies and architecture search.", "questions": "- Can the authors provide theoretical evidence on why representational alignment to a guide helps avoid overfitting/underfitting, beyond anecdotal examples?\n\n- What are the memory and runtime costs of this method compared to training a standard network or distillation-based approaches?\n\n- Why not use the guide network directly for inference, especially if it already performs well on the task?\n\n- How sensitive is the method to the choice of guide network? Is there a risk that a poor guide could hurt performance?", "limitations": "Yes!", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "*Strengths*\n\n- The guidance mechanism presents a novel and creative way to inject inductive bias into architectures that struggle to train successfully on specific tasks.\n\n- The authors explore a range of architectures (RNNs, Transformers, FCNs, CNNs), tasks (e.g., copy-paste, image classification), and guide/target combinations to support their claims.\n\n- The method could be influential in improving training robustness and re-enabling the use of previously discarded architectures.\n\n*Weaknesses*\n\n- My main concern of this paper is the lack of theoretical justification of the method. The paper does not provide a clear theoretical grounding for why or how guidance helps prevent overfitting or underfitting in the target network. The mechanism by which representational similarity to guide network results in better generalization is not well understood or analyzed.\n\n- Another concern, that may prevent the method to be useful in practice, is the high memory and computation cost of the method. The method involves keeping the guide network (often a large model) in memory during training, and performing additional gradient computations for the representational loss. This raises concerns about scalability, especially for large guide networks.\n\n- The paper does not justify why the target network is trained to mimic the guide rather than simply using the guide network itself for the task. In some cases, using the guide directly may be simpler and more efficient, especially if the guide is already trained.", "was_revised": true, "final_justification": "My concerns are almost resolved, except a rigorous theoretical justification for the proposed method.", "rebuttal": "We thank the reviewer for their thoughtful review. We are glad the reviewer found the paper novel and interesting, appreciated the wide variety of settings, and considered guidance to be potentially influential. We hope to address some concerns here.\n\n> W1: theoretical justification of the method. The paper does not provide a clear theoretical grounding for why or how guidance helps prevent overfitting or underfitting in the target network\n\nThank you for this question. While we don’t have theoretical justification in the form of a proof, we do have potential explanations that reference prior work. We will update the draft with this discussion and will add these references as related work. \n\nFirst, [1] is a new paper that considers task-aware representational alignment. Their theory provides a generalization bound via kernel alignment. They show that when a “stitcher” maps representations or a source network to a target output, the excess risk of the stitched model is upper-bounded by the CKA alignment between them. This provides a learning-theoretic guarantee that the CKA term in guidance reduces the hypothesis class possibilities seen by an optimizer. Overfitting or underfitting becomes harder. \n\n[2] investigates how a network’s neural tangent kernel (NTK) aligns with a target output during training. The paper shows that NTK alignment accelerates convergence and lowers generalization error in deep linear networks. This aligned kernel condition is inserted by hand in guidance. Similarly, [3] uses Rademacher complexity tools to show that alignment of tangent-kernel features onto a small set of task-relevant directions compresses the effective model class. This formalizes the notion of guidance as an automatic regularizer, where task directions are replaced by the guide network settings. \n\nFinally, [4] demonstrates that after training, the top singular vectors of a network’s hidden activations align with the task target vectors. This empirically supports the layerwise CKA choice in guidance. \n\nA full PAC-style proof specialized to guidance has not been shown in our paper. But we hope that the above covers a theoretical gap. We believe that CKA bounds the risk or complexity in terms of kernel alignment. The NTK and Rademacher analyses show that alignment shrinks the effective hypothesis space and improves conditioning. This aligns with findings based on singular vectors. We could sharpen this theory by changing the alignment used in guidance e.g. moving from aligning on kernels to aligning on singular vectors or eigenvectors instead. \n\n> W2: the high memory and computation cost of the method\n\nThis is a fair point. However, one of the exciting findings in our paper is that guide networks need not be too large. In fact, we find that guide networks can be an order of magnitude smaller in terms of the number of parameters and still achieve reasonable results. For example, we tried further experiments where we designed an RNN with 345M parameters (equivalent to GPT-2 Medium) and demonstrated that competitive scaling using an untrained transformer guide that had 50M parameters. We are experimenting with even smaller transformer guides and expect to see similar findings. The smaller the guide, the less memory the model and activations take.\n\nWe would also like to point out that in the current compute paradigms e.g. test-time-compute, inference costs are much more expensive than training costs. Having networks with better inference time efficiency is important. Guidance may come with a higher training cost, but the result is a network that has much lower inference cost. This is important given that the current finding with test-time-compute is that generating more tokens leads to better performance.\n\nGuidance can also work naturally with data parallelism, tensor parallelism, or model parallelism setups in most deep learning libraries. We have used some of these techniques for scaling up RNNs. The main footprint memory-wise is storing activations, but there are many techniques one could use to avoid taking GPU memory, such as storing activations to disk or using half-floats instead of full floats. The main speed footprint comes from CKA calculations, but there are parallelism optimizations that can be applied. \n\n> W3: The paper does not justify why the target network is trained to mimic the guide rather than simply using the guide network itself for the task\n\nWe apologize, we should have made this clearer. As stated above, one main setting our paper focuses on is when inference costs are much larger than training costs e.g. with test-time-compute. Models like transformers, are slow at inference time, with quadratic time attention. This severely impedes the deployment efficiency of current sets of models. We expect the trend of inference costs dominating training costs to continue. This motivates the need for novel, efficient architectures with more reasonable inference costs. Our paper aims to expand this potential space of architectural selection choices. There are many efforts that explore alternative approaches, such as SSMs/Mamba, and we believe guidance can aid this effort to create more efficient architectures. With cases like RNNs or FCNs, we can’t guarantee that guidance will find a target network architecture with better efficiency; efficiency is not always about making FLOPs parallelizable as done in training but ensuring that more FLOPs can be used more quickly, as seen in test-time-compute.\n\nFurthermore, the findings in our paper could be very beneficial for expanding possible network selection for practitioners when searching for network architectures as an alternative to neural architecture search. One can now design architectures with better guarantees on training and can compare architectures more fairly without the need for evaluating conditions on trainability, as done in NAS. In general, a larger number of potential architectures leads to more settings where networks can be applied. \n\n> Q1: What are the memory and runtime costs of this method compared to training a standard network or distillation-based approaches?\n\nWe present our runtime and memory costs for two settings. For the Deep FCN setting, our base memory usage is 8GB (batch size of 256) during training and our wallclock runtime is about 17 minutes per epoch on an H100 with our current implementation. With an untrained ResNet-18 guide, our memory usage is 10GB of GPU memory and our runtime is about 28 minutes per epoch on an H100 with our current implementation. Similarly, for our RNN with copy-paste setting, our base memory usage is 2GB of memory during training and our wallclock time is about 3 minutes an epoch on an H100 with our current implementation. With an untrained transformer guide, our memory usage is 3GB an epoch and our wall clock time is also about 3 minutes an epoch. We believe this shows that the memory and runtime cost of guidance is not much more expensive than training a standard network. As stated previously, this training cost can come with benefits for the larger inference cost.\n\n> Q2: How sensitive is the method to the choice of guide network? Is there a risk that a poor guide could hurt performance?\n\nThe method is sensitive to the choice of guide network from running a few sanity checks like guiding a Deep FCN with another Deep FCN. In this case, we find that the Deep FCN overfits and does not improve performance. We also find that using larger guide networks may not necessarily improve the performance of the target network as much due to capacity issues in the target network. The target network may not be expressive enough to learn the representation in the guide network. We will add these discussions to the paper.\n\n[1] Insulla et. al. “Towards a Learning Theory of Representation Alignment”. Preprint, 2025.\n\n[2] Shan and Bordelon. “A Theory of Neural Tangent Kernel Alignment and Its Influence on Training”, Preprint. 2022.\n\n[3] Baratin, George, et. al. “Implicit Regularization via Neural Feature Alignment”. AISTATS, 2021.\n\n[4] Imani, et. al. “Representation Alignment in Neural Networks”, TMLR 2023."}, {"review_id": "FiRYiPEQPy", "reviewer": "Reviewer_iSbz", "summary": "The paper proposes a new approach to boost the performance of neural networks with \"wrong\" inductive biases. The authors show that training a neural network (target network) while forcing its internal layers to behave (in terms of CKA alignment) like the layers of another neural network (the guide) with \"proper\" inductive biases. They show this brings about improvements even when the guide network is untrained.", "questions": "- Do initialization experiments give the same results as the distillation experiments (i.e., prior transfer term for first 300 step vs when the prior transfer term prolongs throughout entire training)? Is it true across language modeling too?\n\n- What if you take the covariance matrices of the guide network's states rather than a more \"complicated\" loss term? Does it imply a better init for RNN?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The work is very creative! It shows that inductive biases can be copied to networks from the weights only, bypassing the rigidities of the architecture. \n- It has a wide variety of experiments with a decent experimental design, including both image and language modeling. Showing good results across all of them.\n- This opens the door for exciting approaches to initialization, potentially allowing the transfer of inductive tendencies of one hard-to-train architecture to a fast-to-train architecture.\n- Remarkably, the approach does not require putting manual work into designing such initializations.\n\n\nWeaknesses (minor):\n- While, as I said, the approach is very novel as far as I can tell, the authors do not make the distinction between THEIR contribution (the transfer of inductive prior only) and ESTABLISHED approaches (teacher-student) and consider them together. I would make the distinction clearer. Having said that, it is mostly a presentation matter, as it is clear that the method works in the important case (i.e., when the guide is untrained)\n\n- Intiuition: I would have liked it if the authors could explain the mechanisms as far as they can (beyond the abstract interpretation of transfer of prior). For example, I think it is worthwhile to mention in passing that in the case of FCN -> ResNet, the inductive bias is probably a simple one (probably making the initialization matrix sampled from around the identity matrix). Other things I am curious about:\n* Intuition about how CKA can encode complicated information such as the inductive bias of a network (eg, how would the CKA of an untrained residual network differ from not untrained FCN, is it a matter of covariance matrices?)\n\n\n- Finally, I think that the initialization interpretation is somewhat more satisfying and less unsettling than that that requires training with the inductive bias throughout the entire training process.", "was_revised": true, "final_justification": "This work helps to understand, from a clean point of view, novel aspects of teacher-student distillation. Especially, the role of just the network's bias on network distillation, at some points, just the prior gives superior results over trained models. This is exciting as a tool as well, as it allows to teach a model a bias unnatural to it. The authors have also discussed in the rebuttal that they were able to disprove the fact that initialization alone is enough, which is again a further contribution. \n\nI like this paper for both its potential in applications as well as its novel approach to first-principles analysis of teacher-student dynamics. \nMy main concern is their presentation, conflating their contribution and distillation, and showing it from a slightly less principled angle than I'd wish them to. \nA second concern is that there might be other explanations for the phenomenon, which is a main concern with any paper with such bold claims, but they have done more than enough for a single paper to avoid this criticism. It remains to be explored in future work.", "rebuttal": "Thank you for your thoughtful review. We are glad that the reviewer found the paper creative, appreciated the wide variety of experiments, and was excited by the initialization experiments. We hope to address concerns here: \n\n> W1: the authors do not make the distinction between THEIR contribution (the transfer of inductive prior only) and ESTABLISHED approaches (teacher-student) and consider them together\n\nThanks. We will make this clearer in the paper. We agree that methodologically, guidance makes a contribution with transferring the inductive prior only. However, we also want to point out that we can use guidance to differentiate between transferring knowledge and transferring a prior. All experiments with an untrained guide transfer an architectural prior such as translational equivariance or receptive fields in a convolutional layer. When we transfer knowledge, we use a trained guide similar to teacher/student settings. Guidance can identify when the architectural prior is useful for improving task performance, a finding that has useful scientific implications. Knowledge distillation cannot do this -- when we use an untrained network as a teacher, we find performance decreases, even in the initialization case (see below). This is intuitive; we are matching outputs of a randomly initialized network on noise. But, beyond the simple methodological difference, this is the scientific setting that guidance can distinguish. We believe such distinctions can help us understand relations between architectures i.e. which networks can take advantage of a structural prior to improve task performance rather than requiring the same structural prior along with knowledge. \n\n> W2: if the authors could explain the mechanisms as far as they can (beyond the abstract interpretation of transfer of prior). For example, I think it is worthwhile to mention in passing that in the case of FCN -> ResNet, the inductive bias is probably a simple one (probably making the initialization matrix sampled from around the identity matrix).\n\nThank you. Yes, we agree that we would like to go deeper into understanding guidance. The extra CKA term reshapes the optimization landscape by pulling the FCN’s activations toward the manifold realized by a frozen ResNet. Because CKA depends only on gram matrices, the FCN does not need to learn literal identity weight matrices; instead, it must recover the ResNet’s locality and weight-sharing statistics.\n\nThere are many additional results that support this possibility. Appendix I shows that, after guidance, the FCN’s layerwise ID curve is similar to the ResNet’s. High ID delays collapse that normally indicates memorization, acting as an implicit regularizer. This is also supported by initialization experiments. Furthermore, Appendix K demonstrates that guiding early ResNet layers delivers the largest performance boost. Those layers encode exactly the low-level locality and translation equivariance biases that an FCN lacks. This is supported by the error consistency metrics. Overall, this connects with the ID findings: the convolutional geometry injected by those layers maintains high ID and eases generalization.\n\nThe ResNet’s skip connection does cause its first few activations to resemble the input, so matching them encourages near-identity activations in the FCN. But our intuition is that the performance jump comes from the richer structure supplied by convolution and weight sharing, not from setting FCN weight matrices to the identity. Guidance works because it steers the target network into a representational regime that combines statistically useful activations (high ID, initialization) and functional geometric priors (locality, equivariance). \n\n> W3: Intuition about how CKA can encode complicated information such as the inductive bias of a network (eg, how would the CKA of an untrained residual network differ from not untrained FCN, is it a matter of covariance matrices?)\n\nThis is a great question. CKA is a measure that depends on second-order statistics, specifically pairwise sample distance matrices i.e. the gram matrices formed by taking a distance between every pair of samples with shape [num_samples x num_samples]. Architectural choices imprint distinct features on those statistics. For example, consider local receptive fields in a convolutional layer. Units that cover neighbouring pixels receive correlated input, and this is reflected in our activations. Such correlations are reflected in our distance matrices, and these can be transferred to distance matrices associated with FCN layers that lack local correlations. Similarly, weight sharing, where the same kernel is applied at every spatial location, will also be reflected in a distance matrix, even if weights are random. \n\n> Q1: Do initialization experiments give the same results as the distillation experiments (i.e., prior transfer term for first 300 step vs when the prior transfer term prolongs throughout entire training)? Is it true across language modeling too?\n\nWhile we cannot share external links to images of plots, we have run an experiment where we recreate our initialization with a distillation loss function rather than guidance. Our finding is that distillation does not recreate the finding with initialization. This is expected: we are matching outputs of a randomly initialized network on noise. With distillation, our initialization experiment gets an accuracy of 1.75%, which is close to running the Deep FCN with standard training.\n\nWe find that initializing an RNN with guidance leads to little improvement in language modeling or copy-paste. This is useful for confirming the specific failures of the RNN: initialization does not solve problems with repeated matrix multiplications that must learn to remember, mix, or forget inputs from prior time steps. We believe this is unlikely to have been solved using better initialization since layer errors compound in a deep RNN. We believe the initialization experiments are better reserved for the FCN and are excited to try extending them to transformers as well. \n\n> Q2: What if you take the covariance matrices of the guide network's states rather than a more \"complicated\" loss term? Does it imply a better init for RNN?\n\nTo ensure our own understanding, it seems the reviewer is asking whether we could replace CKA with a loss function that computes the L1 or L2 distance between the guide network and target network covariance matrices for a layer pair. It’s likely that this could transfer some architectural bias. But, our intuition is that it would be harder to optimize as it gives up certain properties of CKA such as scale- and rotation-invariance. Scale invariance is important because if one layer’s variance spikes, this would dominate the basic covariance loss. Similarly, matching covariance matrices would force an exact orientation match due to a lack of rotation invariance. And, CKA implements an implicit layer normalization while covariance matching does not, meaning that deeper layers with a tiny variance would contribute little to the covariance matching loss. However, we hope to investigate many possible alignment functions, from ones with more degrees of freedom to ones with less degrees of freedom to better understand how these properties influence how information is shared across networks. We also note that the memory usage from the covariance matrices (num_features x num_features) would be significant. \n\nAs discussed above, we don’t believe that guidance would be a very good initialization for the vanilla RNN due to the specific failures of RNNs themselves.\n\n> Q3: Line 176, what goes on there?\n\nThank you for catching this. Apologies, this was typo that was probably from shortening the paper. The sentence should say \"To systematically evaluate our approach, we divide our experiments into settings with untrainable architectures and untrainable tasks.\""}, {"review_id": "2uFhOXCWlD", "reviewer": "Reviewer_Tqwd", "summary": "This paper proposes to apply a guidance network to train a target network, the intermediate representations between the two networks are aligned. Thus, the target network could be less overfitted.", "questions": "Please see the weaknesses above.", "limitations": "I would suggest to better clarify the purpose of ths draft, if it tries to better regularize the model, then introduce the method and compare against other regularization methods comprehensively, the pros and cons, the performance and cost. If this draft intends to propose a better disllation process, then rethink the target task accordingly. I would encourage a major modification before resubmitting.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 1, "significance": 2, "originality": 1, "strengths_and_weaknesses": "The clarity of this draft needs improvement, some points are not clear enough to me.\n\n1. Line 2 “We call a network untrainable when they overfit, underfit..” This definition seems very vague to me. A model could be under- or over-fitted due to numerous reasons, but they are all trainable. And model regularization is a long-standing topic, maybe this study falls into this bucket.\n\n2.Line 10 “If the guide is trained, … If the guide is untrained,...” There exist different model architectures, some well-designed models can learn the inductive bias better. If the model is “untrained”, personally I don’t believe the model contains “the” inductive bias we want for down-stream models or tasks. We can discuss a little why the untrained may also slightly work later in this thread.\n\n3. Line 16 “Our method provides a mathematical tool…”, I can only find some empirical findings in the experiment based on the widely used loss and accuracy, is this the new tool provided in this draft? Please correct me if I am wrong.\n\n4. I can see the terms “mitigate”, “stop” overfitting occurred many times, personally, I highly doubted if the distillation-like solution is necessary for this purpose. There are different ways to prevent overfitting on network, simple dropout or l2 regularization. However, I didn’t see any mathematical discussion nor empirical comparison on different regularization. Which may deliver “cheaper” solutions for overfitting.\n\n5. Is it novel that the finding of using a guidance network can prevent overfitting? This is not very surprising. Given the guidance network is not overfitted, then aligning the internal representation between the two models will definitely “compromise” the guidance model. And this is core assumption behind distillation.\n\n6. Line 121, “We distinguish guidance from distillation…”, distillation can be applied to either the output or the intermediate features, I didn’t see any difference between this draft and other distillation studies based on Eq1, though the purpose could be slightly different. I won’t be surprised if a large trained teacher can do a better job for this purpose, given the better teacher learns a better representation which can better generalize on the validation data, then “compromising” on it can also deliver a better validation loss(less overfitted).\n\n7. Following the question in Q2, is the use of “untrained” model for a guidance novel? Some models maybe initialized as a Gaussian filter, which could be roughly used as a Gabor filter back to the era of hand-crafted features. Again, comprising on this un-fitted initialization can lead to a less over-fitted curve, but this procedure is meaningless. Aligning with unlearned filters will also lead to an under-fitted target network, which makes the use of untrained models unnecessary, the target model can not properly fit on any thing, tuning the loss weight is trivial in this case.\n\nTo sum up, applying distillation (I still believe the trick used in this draft is the same as the common distillation) singly for over-fitting seems not necessary, I belive other solutions can do the same job more efficiently. Distillation can be used to train a better target model for sure, and that superiority of the target model also includes less-overfitting.", "was_revised": true, "final_justification": "After the discussion, I still believe the proposed method is utilizing knowledge distillation with hand-crafted features, though new names are assigned to those un-trained networks as architectual biases. I changed my rating, but this is still below acceptance to me at this moment.", "rebuttal": "We thank the reviewer for their thoughtful review. We hope to address concerns on the clarity of our draft and the aim of our paper. \n\n> W1: A model could be under- or over-fitted due to numerous reasons... And model regularization is a long-standing topic...\n\nWe apologize if our definition seemed vague. We specifically called architectures trainable when their performance was near-chance. For our networks, this near-chance performance is driven by overfitting, underfitting, or some issue with incorporating state or memory. This is also based on discussions from prior work, see [1, 2, 3, 4]  which connect overfitting and underfitting to trainability. [1] states that “Deeper neural networks are more difficult to train”. In [1], SGD cannot find a low-error solution even though one exists by construction. That is the very definition of an untrainable architecture. [2] treat overfitting as a core obstacle that must be fixed for a model to be usable. [3], [4] and many other papers talk explicitly about improving trainability by fixing either optimization instability (underfitting) or excessive capacity (overfitting). Our goal was to take such architectures that have near-chance performance and show that we could achieve performance significantly above chance, even obtaining an architecture that was potentially usable. \n\nOur paper aims to be more general than focusing entirely on model regularization. Our current focus is on failures driven by architecture that cannot be solved by current optimization or regularization techniques. Guidance with untrained networks could discover new regularization strategies that could not be uncovered before. However, our overall goal is to discover when neural networks failures are due to difficulties with knowledge priors, structural priors, or initialization. \n\n> W2: … personally I don’t believe the model contains “the” inductive bias we want for down-stream models or tasks.\n\nWe respectfully disagree. Prior work and our paper show that untrained networks contain a great deal of information that is useful for a downstream task. For example, [5] states that the structure of a CNN generator network is sufficient to capture a great deal of low-level image statistics prior to any learning and yields competitive denoising and inpainting performance with untrained networks. Convolutions impose local receptive fields, weight sharing, and translation equivariance, all of which shape our distance matrices as part of CKA. Residual connections preserve gradient flow. Those structural properties show up immediately in the gram matrices that CKA compares, which is why an untrained ResNet guides an FCN so effectively. Self-attention also imposes inductive biases useful for modular arithmetic [6]. Our paper provides further empirical proof of this. \n\nTrained models certainly embed both architectural and learned biases that may capture a task’s structure more cleanly. But that does not negate the fact that architecture alone already encodes a nontrivial prior in most cases. One major point of guidance is to isolate architectural bias from learned bias, a distinction methods like distillation cannot make. Results with untrained guide networks would not work if an untrained network truly lacked inductive bias that was relevant for downstream models or tasks. \n\n> W3: Line 16 “Our method provides a mathematical tool…”, is this the new tool provided in this draft? \n\nWe will revise the language to characterize our claim better. We believe guidance really is a mathematical tool rather than just an accuracy trick, unlike distillation. CKA is a differentiable kernel-alignment metric that is scale invariant and agnostic to labels. Minimizing CKA lets you clamp any subset of a target network’s activations onto those of a frozen guide network and sweep that clamp across architectures and priors and observe what fails or succeeds. Guidance is a knob you can turn to inject architectural priors at will, something one cannot do with cross-entropy or weight-decay alone. We use this in concrete ways such as in Appendix K where we investigate the distinction between early vs late priors to show that early ResNet layers matter most for FCNs or later transformer layers matter most for RNNs. In Table 2 and 3 as well as Appendix E, we show a distinction between architectural and learned priors to isolate pure architectural bias. We gave a formal method for discovering initializations of networks with this alignment, showing that the guidance “clamp” created a better weight basin. Initialization is extremely relevant given the random nature of discovering initialization methods overall. And, as shown in responses to other reviewers (CY31), kernel-alignment theory shows that forcing two models to share the same hidden-feature kernel tightens generalization bounds and improves conditioning. Guidance, to us, is a controllable, differentiable kernel-alignment feature clamp that lets researchers dissect architectural priors and is a regularizer to investigate what priors are helpful. \n\n> W4: There are different ways to prevent overfitting on network, simple dropout or l2 regularization.\n\nThank you. We include baselines comparing our Deep FCN optimized with L2 regularization and AdamW to guidance. We report the results below:\n\n|Setting |Accuracy |\n|---|---|\n| DFCN + L2  \t| 6.4 |\n| DFCN + AdamW   | 7.2 |\n| Guided DFCN\t| 13.1 |\n\nWe cannot report any loss curves here. We can confirm that L2 regularization prevents some overfitting but not as well as guidance. \n\n> W5: Is it novel that the finding of using a guidance network can prevent overfitting? This is not very surprising.\n\nWe hope to make this more evident. First, the focus of our paper is not only on overfitting. We consider networks that struggle to incorporate memory like RNNs or underfit like Deep ConvNets or Deep FCNs. We believe having a general purpose method in all of these cases is novel. We would also like to point out that across all of settings, we see major improvements. We take non-functional architectures like FCNs or RNNs and turn them into functional architectures by transferring structural priors. This is not reproduced by logit-based distillation either. \n\nWith regards to regularization, we also see guidance and our paper as novel. Most regularizers prevent overfitting by shrinking weights, adding noise, or transferring task targets. Here, architectural patterns alone restore generalization. Deep FCN, deep plain CNNs, RNNs, are all known to overfit or underfit despite tuning, identity/orthogonal inits, dropout, weight decay, label-smoothing, and distillation. In our paper, guidance is the first single technique shown to rescue all three classes of pathological failures without changing architecture or adding train time augmentations.\n\n> W6: I won’t be surprised if a large trained teacher can do a better job for this purpose...\n\nIn our paper, many of the guide networks are smaller in terms of parameter count and layer count. We find that larger guide networks do not necessarily do a better job. The target network does not have the capacity to learn useful information from a larger guide network like knowledge and structural priors. If we attempt to guide a 4-layer RNN with a trained 6-layer transformer for language modeling, our final perplexity is actually 52.6 which is worse than the score reported. The RNN does not have the capacity to match the guide network. This shows that what we can learn from guidance goes beyond distillation. We have now found out a relationship between the parameter count of an RNN and how much is sufficient to learn what a specific transformer can learn. \n\n> W7: Following the question in Q2, is the use of “untrained” model for a guidance novel...?\n\nThis is not what we empirically find in our paper. If alignment with unlearned filters merely “compromised” the target, we would expect little change in training loss and lower test accuracy. This paper observes the opposite: training loss improves while the generalization gap closes. For instance, 4- and 6-layer RNNs improve with an untrained transformer, achieving LM perplexities that are competitive with LSTMs. This shows proper fitting. Underfitting is not theoretically guaranteed either. [7] and [8] prove that matching the feature kernels reduces complexity while preserving optimal risk. In our setting, this provides evidence that we are shrinking the function class just enough to avoid memorization while ensuring a solution is reachable. \n\nWe also argue that guidance is distinct from Gaussian/Gabor-like filters. A single Gaussian filter is a hand-crafted prior that is applied to the first layer. A randomly initialized guide such as ResNet contains dozens of layers of convolutions, residuals and batchnorm stats. This stack must impose priors like locality and receptive fields, residual spectra, etc. CKA is sensitive to this structure, not just first layer edge filters. \n\nWe believe the novelty of our results agrees with findings in [5] where authors show that random filters carry signal. We are the first, to the best of our knowledge, to use these filters as an external tool to fix a different architecture. Our comparison against distillation showed that distillation needed a trained teacher. \n\n[1] He et. al. “Deep Residual Learning for Image Recognition”, CVPR 2016.\n[2] Srivastava et al. “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”. JMLR 2014.\n[3] Ioffe & Szegedy. “Batch Normalization”, ICML 2015\n[4] Wang and Fu. “Trainability Preserved Neural Pruning.” ICLR 2023.\n[5] Ulyanov et. al. “Deep Image Prior”. CVPR 2018. \n[6] Zhong and Andreas. “Algorithmic Capabilities of Random Transformers”. NeurIPS 2024.\n[7] Canatar et. al. “Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks.” Nature Communication, 2021. \n[8] Bietti and Mairal. “On the Inductive Bias of Neural Tangent Kernels”, NeurIPS 2019."}, {"review_id": "Amdr8Gesht", "reviewer": "Reviewer_xZWb", "summary": "The paper investigates whether models traditionally deemed unsuitable for certain tasks can perform well when guided by inductive biases from more suitable architectures. The proposed method employs a neural distance function and layerwise representational similarity to guide the \"unsuitable\" model. The authors conduct experiments across several architectural pairs, including FCN vs. CNN, RNN vs. Transformer, and No-skip-CNN vs. ResNet. Interestingly, they observe that even randomly initialized (untrained) models can serve as effective guides, leading to performance improvements in the target model.", "questions": "1. The authors claim that \"Guidance can use a smaller untrained guide instead of a larger trained teacher.\" Would using a larger trained guide network further enhance the performance of the target network compared to using a smaller one?\n2. Have the authors considered aligning feature statistics (e.g., mean and variance) [1] during representation matching?\n\nRef.\n\n[1] Mirza et al., \"ActMAD: Activation Matching to Align Distributions for Test-Time-Training\", CVPR 2023", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n1. The paper introduces the concept of an inductive bias gap between modern neural network architectures and proposes bridging this gap through representational alignment. The method is simple, yet the findings are surprising.\n2. The experiments are extensive and systematic, validating the effectiveness of the proposed approach.\n3. The method presents a useful framework for studying architectural priors, with potential applications in neural network design and initialization.\n\n\nCons:\n1. Some claims in the paper should be toned down. For example, the statement about \"providing theoretical insights\"  (line 62) of the introduction seems overstated given the mainly empirical nature of the paper.\n2. Some of the tables should be improved. In Table 1, consider reorganizing the layout to help readers easily identify which experiments involve \"untrainable\" architectures versus \"untrainable\" tasks. In Table 2, highlight the target networks using different background colors for better readability. In Table 5, include the Deep FCN results from Table 2 to improve readability and allow for easier comparison.\n3. Given that distillation is a natural and well-established baseline, why was it not thoroughly compared and included in Table 2?\n4. The explanation regarding why untrained guide networks sometimes yield better performance in Table 2 is not fully convincing. In Appendix E, the Wide FCN paired with untrained ResNet shows lower similarity loss, yet paired with the trained ResNet achieves higher accuracy.", "was_revised": true, "final_justification": "I appreciate the authors’ efforts in the rebuttal, and my overall concerns have been addressed. However, since some revisions are still needed, my score remains unchanged.", "rebuttal": "We thank the reviewer for their thoughtful review. We are glad the reviewer found the ideas in the paper novel, the method simple, and the results surprising. We hope to address some concerns below.\n\n> W1: Some claims in the paper should be toned down. For example, the statement about \"providing theoretical insights\" (line 62) of the introduction seems overstated given the mainly empirical nature of the paper.\n\nThis is fair. We will aim to tone some of the claims in the paper, specifically noting guidance’s ability to differentiate between structural and knowledge-based priors. \n\n> W2: Some of the tables should be improved. In Table 1, consider reorganizing the layout to help readers easily identify which experiments involve \"untrainable\" architectures versus \"untrainable\" tasks. In Table 2, highlight the target networks using different background colors for better readability. In Table 5, include the Deep FCN results from Table 2 to improve readability and allow for easier comparison.\n\nThank you. These are great suggestions. We will absolutely fix the layout of Table 1 and 2 to better indicate the understanding of untrainable architectures and tasks as well as have a better coloring scheme to indicate performance. Such changes will also go to Table 3. We really appreciate the help on readability. \n\n> W3: Given that distillation is a natural and well-established baseline, why was it not thoroughly compared and included in Table 2?\n\nThis is fair. We will include the numbers from the appendix into the main paper upon acceptance. This will be an additional set of rows, copied here for simplicity. We will modify the table to match your suggestions on formatting as well.\n\n| Experiment                                | Copy-Paste Accuracy | Parity Accuracy | Language Modeling Perplexity |\n|-------------------------------------------|---------------------|-----------------|------------------------------|\n| RNN Teacher/Transformer Student           | N/A                 | 75.11           | N/A                          |\n| Untrained RNN Teacher/Transformer Student | N/A                 | 68.98           | N/A                          |\n| Transformer Teacher/RNN Student           | 19.61               | N/A             | 52.00                        |\n| Untrained Transformer Teacher/RNN Teacher | 14.33               | N/A             | 73.16                        | \n\n> W4: The explanation regarding why untrained guide networks sometimes yield better performance in Table 2 is not fully convincing. In Appendix E, the Wide FCN paired with untrained ResNet shows lower similarity loss, yet paired with the trained ResNet achieves higher accuracy.\n\nThis is a fair point, the reviewer is right that our initial explanation in the paper was completely thorough. We will add some discussion here and update our paper to address these points. We believe the inconsistency with the Wide FCN is likely due to properties of wide neural networks as well as the systemic failure of the Wide FCN. For the Wide FCN, an untrained ResNet-18 is easier to mimic. This is likely due to the simplicity of the initialization of an untrained ResNet-18 as well as properties of very wide linear layers as universal function approximators. The lower CKA distance with the untrained guide network indicates strong architectural transfer. But the lower accuracy indicates that the feature transfer is not task aligned i.e. the learned filters are more relevant for the Wide FCN. Intuitively, the Wide FCN already generalizes reasonably, and wide linear layers can approximate convolutional layers [1]. So the remaining gap is feature quality, not architectural gaps. Therefore, a trained guide adds class selective filters that a random guide cannot provide. \n\nOne other explanation for our performance with untrained guides is that target networks that overfit can be aided with a randomly initialized guide network. Targets that lack discriminative features like the Wide FCN may be aided by a trained guide that adds semantic bias. We will aim to make this more clear in the paper.\n\n\n> Q1: Would using a larger trained guide network further enhance the performance of the target network compared to using a smaller one?\n\nThis is a great question. We actually find that a larger, trained guide network does not perform as well as a smaller guide most of the time. Our intuitive explanation for this is that the target network does not have the capacity to learn useful information from a larger guide network. We find that having a sufficient number of parameters is important for transferring knowledge and priors from the guide network. For example, if we attempt to guide a 4-layer RNN with a trained 6-layer transformer for language modeling, our final perplexity is actually 52.55 which is worse than the score reported. The RNN does not have sufficient capacity to learn the same function as the guide network. We hope this shows that what we can learn from guidance goes beyond distillation. We have now found out a relationship between the parameter count of an RNN and how much is sufficient to learn a function that a specific transformer can learn. \n\n> Q2: Have the authors considered aligning feature statistics (e.g., mean and variance) [1] during representation matching?\n\nWe did not consider this, thank you for the suggestion. This could validate whether our intuitions about CKA are correct. CKA is a measure that depends on second-order statistics, specifically distance matrices across samples. Architectural choices imprint distinct features on those statistics. For example, consider local receptive fields in a convolutional layer. Units that cover neighbouring pixels receive correlated inputs, and this is reflected in our activations. Such correlations are reflected in our distance matrices, and these can be transferred to distance matrices associated with FCN layers that lack local correlations. It’s unclear whether the same is captured by simple mean and variance statistics. One concern is that the mean does not capture correlated inputs well enough, and this could lead to potential failure. However, it would be exciting if even the mean and variance capture enough information to influence how an FCN or RNN layer is activated, and this could have strong implications for understanding alignment and inductive biases of architectures in the future. Due to the time constraints of this rebuttal, we could not include this experiment but will go deeper into this direction in the future.\n\n[1] Malach and Shalev-Shwartz Computational Separation Between Convolutional and Fully-Connected Networks, ICLR 2021."}]

Author rebuttal:

Meta-review: This paper introduces guidance, a method for training networks that are not considered suitable for certain tasks by utilizing a guide network. The method introduces a novel way to incorporate inductive bias into the training process. There were concerns with the memory and computational costs of the method as well as the lack of theory behind some of the claims. Overall, the reviews were positive and I believe the paper should be accepted. The authors should consider removing theoretical claims as the paper is mostly empirical.
Decision: Accept (poster)