Training the Untrainable: Introducing Inductive Bias via Representational Alignment

Authors: Vighnesh Subramaniam, David Mayo, Colin Conwell, Tomaso A Poggio, Boris Katz, Brian Cheung, Andrei Barbu
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Representational alignment, neural network optimization

Abstract:
We demonstrate that architectures which traditionally are considered to be ill-suited for a task can be trained using inductive biases from another architecture.  Networks are considered untrainable when they overfit, underfit, or converge to poor results even when tuning their hyperparameters. For example, plain fully connected networks overfit on object recognition while deep convolutional networks without residual connections underfit. The traditional answer is to change the architecture to impose some inductive bias, although what that bias is, is unknown. We introduce guidance, where a guide network guides a target network using a neural distance function. The target is optimized to perform well and to match its internal representations, layer-by-layer, to those of the guide; the guide is unchanged. If the guide is trained, this transfers over part of the architectural prior and knowledge of the guide to the target. If the guide is untrained, this transfers over only part of the architectural prior of the guide. In this manner, we can investigate what kinds of priors different architectures place on a fully connected network. We demonstrate that this method overcomes the immediate overfitting of fully connected networks on vision tasks, makes plain CNNs competitive to ResNets, closes much of the gap between plain vanilla RNNs and Transformers, and can even help Transformers learn tasks which RNNs can perform more easily. We also discover evidence that better initializations of fully connected networks likely exist to avoid overfitting. Our method provides a mathematical tool to investigate priors and architectures, and in the long term, may demystify the dark art of architecture creation, even perhaps turning architectures into a continuous optimizable parameter of the network.

Decision: reject
Reviewer scores: 3, 8, 5, 8

Reviews and discussion:
<thread>
creation date:
1729644780411

thread_type:
official_review

<thread_object>
creator: reviewer_sep7

<content>
summary:
The authors propose an approach for aligning representations across two different deep learning architectures. One architecture serves as the "teacher" (providing guidance) to the "student" (target network). For example, one architecture could a convolution whereas the other could be a transformer. 

Specifically the authors propose training the target network with an additional loss term penalizing the dissimilarity between the target and fixed guidance network. The authors measure the layer-wise dissimilarity using the complement of linear CKA similarity. 

The authors study four tasks (across vision, language, and arithmetic) to illustrate how their approach can improve performance by guiding an ill-suited architecture for the task with a more suitable architecture for the task using their alignment approach.

strengths:
- The proposed idea is intuitively presented and I believe could serve as a useful tool for studying network initialization schemes as well as scientifically probing the role of architectural inductive biases.
- I appreciate the authors investigating how to bridge our understanding of the role of architecture by providing a principled mechanism for transferring (some) of the architectural biases of one network architecture to another. 
- Empirically the authors selected a diverse set of tasks with reasonable baselines and choice of target versus guide networks. I appreciate how thorough the authors are disclosing training details, hyperparameters, and the intent of the experiments. I also appreciate the error analysis conducted across architectures and the error bounds for experimental results. 
- Authors draw a distinction between typical distillation approaches and the proposed guidance approach that I find clear and well-motivated.

weaknesses:
- The field has mostly converged on the transformer architecture for most tasks. While this doesn't make the idea presented less interesting scientifically, I'd encourage the authors keep this in mind when motivating the approach as today, practitioners are not frequently choosing across many choices of architectures. I find initialization and the scientific value of understanding the role of architectural bias still valuable and would encourage the authors emphasize these angles instead. 
- The gains for ImageNet are still quite small overall, suggesting we perhaps can't simply transfer over inductive biases or that much much more tuning is needed. Do the authors have any comments on this point? At a minimum, I'd recommend the authors acknowledge and discuss this in the results section.
- Inconsistent gains across trainable versus untrainable without clear intuition as to why this might be happening. I appreciate the author's attempt at clarifying this in the paragraph starting on line 373, but I still feel this section is missing crucial intuition to help readers make sense for the empirical findings across the tables. For example, why would an untrained ResNet-18 provide better guidance relative to a trained ResNet-18 to a Deep FCN as shown in Table 2. This is puzzling to me and am curious if the authors have additional experiments or intuition to better explain this in the context of the rest of the results.
- The authors make several claims about the revolutionary potential of the method that are not well-supported with evidence. While I'm also excited about the proposed method and potential future work that builds on it, I'd suggest the authors downtone what right now appear as spectulations in this draft. For example, lines 85-86 claim the proposed approach "expands the space of viable networks," a claim I believe is not well-supported by this work. Similar claims are made about potential future promises about guidance on lines 108-112 that are not well-supported by the experiments in the work.
	- Section 4 could be improved. You emphasize a distinction between untrained architectures (versus untrained tasks)—defining Untrainable Architectures as target networks difficult to train irrespective of task. In the experiments that follow however,  the focus is very much on whether a given target architecture is ill-suited for a specific task.

questions:
- It's not crucial to the reception of the paper, but I'm curious if the authors explored the evolution of the CKA similarity across layers comparing early versus later layers throughout guidance akin to [https://proceedings.neurips.cc/paper_files/paper/2014/file/375c71349b295fbe2dcdca9206f20a06-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2014/file/375c71349b295fbe2dcdca9206f20a06-Paper.pdf) 
- I'm also curious whether the authors explored any other similarity metrics besides linear CKA.
- Perhaps I missed it, but where are the experiments with guidance between architectures with a different number of layers described on lines 230-235?
- Did the authors consider weighing the two loss terms proposed in equation 1?  
- Not necessarily a weakness, but I'm curious whether the authors explored transferring equivariant architectures for tasks with known symmetries.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730636155422

thread_type:
official_review

<thread_object>
creator: reviewer_kh2h

<content>
summary:
This work presents a simple method for training neural networks that are typically considered untrainable. The approach, similar to the teacher-student methodology, involves training a "target" network with the assistance of a "guide" network, which can be either trained or untrained and may have a completely different architecture from the target. The objective is to align the target network's internal representations as closely as possible with the guide ones in terms of CKA similarity.

strengths:
* The paper studies an interesting problem trying to solve the problem of untrainable networks
* The authors analyzed multiple scenarios with different tasks and architectures
* The authors provide the supplementary materials with demo notebooks

weaknesses:
**Writing and Presentation**: The paper could benefit from improved clarity and readability. Some suggestions to consider: 

* The paper includes expressions that may introduce ambiguity or complexity (e.g. lines 30,42) potentially making it harder for readers to grasp the authors' points fully. Rephrasing these expressions with clearer language would better convey the challenges discussed and enhance accessibility.

* The tables are somewhat difficult to interpret, and it’s unclear if certain entries serve as baselines. For example, Table 2 could be restructured to distinguish between guide and target, rather than using the "experiment" column, and highlight when training is performed without teacher guidance. Making tables and feature descriptions more self-contained and better explained would enhance clarity.

* The Method section could be more clearly organized and written. It is challenging to follow, and restructuring or rephrasing certain parts would likely improve reader comprehension.

* In the Related Work section, the "representation similarity" literature is missing (e.g. 2,4-8). Adding this section could support several claims, such as the one on line 145. 

**Novelty**: The paper’s novelty is limited, as it overlooks relevant related work that addresses similar challenges. For example:

* A similar work exists, such as [1], where authors leveraged relative representations [2] in a teacher-student framework, training the student network to mirror the teacher network’s latent representations.

* The authors’ claim in the conclusion regarding the absence of known methods for improved network initialization is not entirely accurate. For instance, [3] proposed a method for initializing smaller models by selecting subsets of weights from a larger, pretrained model, thereby transferring knowledge from the larger model to smaller architectures.

* Additionally, the experiments lack a comparison with the classic teacher-student setting, which would provide a useful benchmark.

**Contribution**: While the results show some improvement, they also highlight ongoing challenges in making these networks fully trainable. For example:

* In Table 2, the results show an accuracy improvement from 7.5 to 13.10. However, no standard deviation is provided, which limits the interpretation of these results. Moreover, an accuracy of 13.10 is not competitive on this dataset for the image classification task.

* The choice of the networks raises questions, as they do not represent state-of-the-art (SOTA) architectures. It would be interesting to explore whether using a pre-trained network to guide another competitive network could yield further improvements. 

--- 

[1] Ramos, Patrick, Raphael Alampay, and Patricia Abu. "Knowledge Distillation with Relative Representations for Image Representation Learning." International Conference on Computer Recognition Systems. Cham: Springer Nature Switzerland, 2023.
[2] Moschella, Luca, et al. "Relative representations enable zero-shot latent space communication." ICLR 2022.
[3] Xu, Zhiqiu, et al. "Initializing models with larger ones." ICLR 2023.
[4] Huh, Minyoung, et al. "The platonic representation hypothesis." ICML 2024.
[5] Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. arXiv preprint arXiv:2010.15327, 2020.
[6] Shengkun Tang, Yaqing Wang, et al. You need multiple exiting: Dynamic early exiting for accelerating unified vision language model. CVPR 2023.
[7] Zorah Lahner and Michael Moeller. On the direct alignment of latent spaces. In UniReps 2023.
[8] Valentino Maiorca, Luca Moschella, et al. Latent space translation via semantic alignment. NeurIPS 2023.

questions:
* Are the trained guide networks pretrained models that are then fine-tuned during the guidance process?
* Are the untrained guide networks trained in parallel to the target network, or are they frozen during training? If they are not frozen, what is the advantage of training both networks simultaneously rather than focusing solely on training the guide network?
* In Figure 1, when mapping different layers of the guide network to the same layer of the target, it may be important to consider the similarity between these different guide layers.
* How might the results differ if an alternative metric is used to calculate the similarity?
* What impact would guiding only the final layers of the target network have on the results?
* In Table 3, the RNN achieves 100% accuracy and is used as a guide network. What does this imply? Given that the work aims to train untrainable networks using well-performing networks, why use a network that appears to be overfitting as a guide for training the target network?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660671135

thread_type:
official_review

<thread_object>
creator: reviewer_vzfs

<content>
summary:
This paper proposes a novel distillation technique called guiding which distils the inductive priors of the 'guide' network into the target network. The paper shows that by using a known task strong network as the guide, a target network which is known to be weak at the specific task can be significantly improved in terms of performance. The paper then proposes that the guiding network can be disconnected from the process at a very early stage so that it simply acts as a initialisation technique for the network.

strengths:
- Paper is very well written and even enjoyable to read.
- Technique is novel and proposes a new direction in which the field of distillation can go. 
- Technique could revitalise 'dead' neural architectures which failed to take off due to weak performance.
- Experimentation on multiple modalities.

weaknesses:
- I would have liked to see results on smaller datasets. I feel that this could have resulted in a higher number of ablations to determine different configurations of the network e.g. should we be connecting multiple layers to a single layer in the target? Should guide network have the same number of layers as the target? Do all layers in the target need connections to the guide?
- I would have liked to see more in depth discussion of the ViT to CNN/MLP experiments which are hinted at in figure 3.

questions:
I am interested in discussion on both of the weaknesses that I have proposed. Overall I am positive on this paper, but I feel that if the above weaknesses were addressed, it would improve the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730823154836

thread_type:
official_review

<thread_object>
creator: reviewer_rthw

<content>
summary:
This paper proposes guidance, a method where a well-performing guide network directs the layer-wise representations of a target network, transferring inductive biases without modifying the architecture. The technique aims to improve training for architectures traditionally prone to issues like overfitting or underfitting, such as fully connected networks and plain CNNs. Initial results suggest that guidance can improve performance in various settings, but further validation across tasks and more rigorous testing would clarify its robustness and broader applicability.

strengths:
- The observations are interesting and to some degree novel. 
- A wide range of tasks from several domains are tested empirically. 
-  The paper is well written and methods are explained comprehensively.

weaknesses:
1. The main observation of the paper is that the performance of suboptimal architectures can be improved if their intermediate representations are matched to a more optimal guide models. This is an interesting observation but somewhat a natural outcome of what we already know.  
	
	Functionally, representation matching to a guide network allows deeper layers of the target networks to receive useful gradients for learning that they normally do not receive from the task loss. We know that in deep networks w/o skip connections, auxiliary loss functions applied on intermediate layers help with learning [1]. These losses were proposed to mediate the very same issue examined here, that deep CNNs suffer from the vanishing gradient. Relatedly, models with skip connections like ResNets can also be viewed as ensemble of shallow networks as skip connections allow gradient to pass through the deep layers [2]. Suggesting that what skip connections in one way do is to allow many parallel pathways inside the model for letting the gradients flow throughout. I don't see how what is proposed here is fundamentally different from what was show in the prior literature. One way to make this claim more grounded is to show that representation matching outdoes auxiliary losses applied on earlier layers. In general, distillation is a critical missing baseline in all the results. 
	
	[1] Szegedy, Christian, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. "Going deeper with convolutions." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9. 2015.
	
	[2] Veit, Andreas, Michael J. Wilber, and Serge Belongie. "Residual networks behave like ensembles of relatively shallow networks." Advances in neural information processing systems 29 (2016).


2. A second primary observation in the paper is that even when the representation is matched to that of an untrained guide network it leads to substantial improvements in the performance of the target network. While this is an interesting observation, specially in cases where this scenario surpasses the matching of the trained network, the portrayal of the results are inaccurate. The main issue lies in considering the untrained model as incapable of performing the task better than chance. In Table 2, the accuracy of the untrained models are reported as being consistently around chance, which is expected if none of the parameters are trained. However, in the context of this paper which uses network representations as guide, the informative measure would be the accuracy of a linear classifier trained on the penultimate layer features. This value is typically much higher than chance and I expect it to be so here as well which attests to the usefulness of the features at their initial state. 
Apart from this, there are more things to be done in order to at least attempt to offer a plausible explanation of this phenomenon. For example, what are the geometrical differences as a result of matching to the untrained and trained networks? Presumably, since the trained model is better able to distinguish between imagenet classes, matching its representations, if successful, should have been more useful for the target network. But somehow that's not true. 

3. The observed higher utility of the untrained guide network becomes more interesting specially when considering the fact that the matching loss term is included throughout the training. One possible explanation could be that the untrained network geometry is simpler to replicate for the target network while the fully trained model could have a much more clustered representation that is too difficult for the target to learn from. This could be tested by using various guide networks at different stages of training and tracking performance when matching to each, then examining any relationship between geometrical properties of the latent space and performance of the target network.

4. Figure 4 results are interesting in showing that even limited training on the matching loss is still helpful. Would the target network its full accuracy in Table 1 if training is done in two phase where the first phase will fully train the network on the untrained guide representations (weight initialization), and phase 2 would do the task training? I think this experiment is important for supporting the claim about finding a good initial weight in these models using the representation matching idea. 

5. At parts, the writing is very verbose. 6.5 pages of the paper is dedicated to the intro, background, and a methods section that contains only modest novelty. Sections 1 and 2 should be written much more concisely to make room for additional experiments. Some of the details of the experiments could also be moved to the appendix. 

6. There are a few papers that are conceptually very similar to the proposed ideas in this paper but were not cited 

https://arxiv.org/abs/1412.6550

https://arxiv.org/abs/1808.01405

7. The last paragraph of introduction talks about the limitations, would fit better at the end of the paper (e.g. in the discussion)

8. Line 116: "To that end, we also did not optimize networks to convergence," this sounds like a serious limitation that could be avoided. If the networks are not trained optimally some of the conclusions may turn out being incorrect. E.g. the observed differences between matching the trained and untrained models. 

9. Line 257: "We describe the task settings": remove 

Overall, this paper showcases very interesting observations but falls short at providing concrete insights. The paper in its current format reads premature. The methods are not novel, the observations are interesting but no real insight is offered. This is study has many potentials but they are not met yet and so I don't think it is ready for publication.

questions:
1. Figure 3 shows the pattern of error consistency, is this any different from what would be obtained from distillation? 

2. Related to fig. 3, are the intermediate representations of these two guide models also similar? A similar plot could be made for the intermediate representational similarities which would be very informative.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732280379734

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (Part 1)

comment:
We thank the reviewers for their constructive reviews. We are glad reviewers found the problem and observations in the paper interesting (rthw, vZFS), the potential for guidance to be useful for studying architecture (seP7, vZFS), and appreciated the multiple scenarios and experiments through which guidance was applied (Kh2H, seP7, vZFS). We address some of the common feedback here as well in the individual responses.

* Novelty and Positioning (rthw, Kh2H, seP7): We list a number of novel observations that have not been reported in the literature before. We will make these clear in our manuscript as well.
1. For the first time, **we transfer over the architectural prior of one network architecture to another network architecture**; not the knowledge of one network to another, independent of any data. An untrained (randomly initialized) CNN that has never seen an image, that never learns because it is never updated, in a setting where no images are shown to either network, only randomly (uniformly) sampled matrices of the correct format, can give its architectural prior to a another network of another architecture. That other architecture, which was otherwise poorly regularized and doomed to overfit, now does not. 
2. For the time first, **this shows that fully connected networks can be systematically initialized in a novel way guided by another network architecture to make them avoid overfitting**. No changes to the architecture needed. We provide this initialization method, although, we leave explaining it for future work.
3. This casts doubt on one of the most common ideas of what architecture does in ML. It is not the fact that CNNs or Transformers add an architectural prior by way of convolutions, hierarchy, and/or attention, that regularizes them and allows them to work tasks like for object recognition. This story is pervasive in the current ML literature, appearing in some of the most eminent publications like LeCun, Y., Bengio, Y., & Hinton, G. (2015). "Deep learning." Nature, 521(7553), 436-444. The consequence is that it is assumed that certain architectures are vastly superior to others. For example, Transformers are, with minor exceptions, superior to RNNs. We demonstrate that this relationship isn’t so clear cut: RNNs can help Transformers and vice versa.
4. We can change how a significant subset of ML is done. Usually we hope to find better initialization methods, regularization methods, or optimizers, without any idea if one exists, what it might look like, what a feasible trajectory looks like, or what the final trained network is. We turn this walk into the darkness into a much more systematic search. We provide networks that have initializations that work which we don’t understand, that have regularizations that work, that have training trajectories or final variants that perform well, but that we do not understand. We turn a stab in the dark into a systematic reverse engineering effort.
5. Guidance is a new form of probing. We can now for the first time systematically ask what is the relationship between two architectures? Is what relates them the statistics of the eigenvalues of their activations? Sparsity? Receptive fields? Etc. Any question can be trivially turned into a new similarity metric, plugged into guidance, and systematically tested. We can now systematically ask, what is the relationship between the prior a CNN imposes compared to that of a Transformer? For example, we can repeat our Deep FCN results with a Transformer rather than a CNN. Then ask how are the two resulting networks different from one another, both in their activation patterns and their behavior? There are countless publications claiming to reproduce Transformer results with CNNs and vice versa. We can help establish the relationship between the two.
Some reviewers asked questions related to 5. For example, is the relationship between FCNs with a trained and untrained guide that one has a different effective rank than another? This is an interesting question, and would provide an easy explanation as well as an easy method to regularize FCNs. In appendix J we show that unfortunately things are not so simple. Effective rank and intrinsic dimensionality does not explain the results and is not effective.
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) summary

This paper investigates how to use a guide network to train a target one traditionally overfitting(FCN) or underfitting(CNN). It performs representation alignment between the guide and target DNN by introducing additional loss terms. The initial results suggest that guidance can improve performance in various settings.

(b) strengths
+ The observation that the performance of a student model can be improved if their intermediate representations are matched to a teacher model, is interesting and is novel to some degree.
+ It investigates the role of network architecture inductive bias for bettering understanding of neural networks.
+ It conducts extensive experiments over a wide range of tasks and analysis.

(c) weaknesses
- The proposed method has limited novelty due to its similarity to teacher-student setting. There are a few papers that are conceptually very similar to the proposed ideas in this paper but were not cited
https://arxiv.org/abs/1412.6550
https://arxiv.org/abs/1808.01405
- It is not easy to verify the correctness of the claims due to some missing baselines (distillation and auxiliary losses tied to intermediate layers).
- It lacks concrete insights into architectural inductive biases based on the proposed approach.
- The presentation is not polished.

(d) decision

Although the paper has potential for better understanding inductive bias of DNN, it is not ready for publication in its current form due to its limited novelty, missing experiments, and unpolished presentation. Please keep the reviewers' comments in mind when preparing a future version of the manuscript.