Title: Training the Untrainable: Introducing Inductive Bias via Representational Alignment
Venue: NeurIPS 2025
Accepted: yes
Keywords: Representational alignment

neural network optimization
Abstract: We demonstrate that architectures which traditionally are considered to be ill-suited for a task can be trained using inductive biases from another architecture.  We call a network untrainable when it overfits, underfits, or converges to poor results even when tuning their hyperparameters. For example, fully connected networks overfit on object recognition while deep convolutional networks without residual connections underfit. The traditional answer is to change the architecture to impose some inductive bias, although the nature of that bias is unknown. We introduce guidance, where a guide network steers a target network using a neural distance function. The target minimizes its task loss plus a layerwise representational similarity against the frozen guide. If the guide is trained, this transfers over the architectural prior and knowledge of the guide to the target. If the guide is untrained, this transfers over only part of the architectural prior of the guide. We show that guidance prevents FCN overfitting on ImageNet, narrows the vanilla RNN–Transformer gap, boosts plain CNNs toward ResNet accuracy, and aids Transformers on RNN-favored tasks. We further identify that guidance-driven initialization alone can mitigate FCN overfitting. Our method provides a mathematical tool to investigate priors and architectures, and in the long term, could automate architecture design.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates whether models traditionally deemed unsuitable for certain tasks can perform well when guided by inductive biases from more suitable architectures. The proposed method employs a neural distance function and layerwise representational similarity to guide the \"unsuitable\" model. The authors conduct experiments across several architectural pairs, including FCN vs. CNN, RNN vs. Transformer, and No-skip-CNN vs. ResNet. Interestingly, they observe that even randomly initialized (untrained) models can serve as effective guides, leading to performance improvements in the target model.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n1. The paper introduces the concept of an inductive bias gap between modern neural network architectures and proposes bridging this gap through representational alignment. The method is simple, yet the findings are surprising.\n2. The experiments are extensive and systematic, validating the effectiveness of the proposed approach.\n3. The method presents a useful framework for studying architectural priors, with potential applications in neural network design and initialization.\n\n\nCons:\n1. Some claims in the paper should be toned down. For example, the statement about \"providing theoretical insights\"  (line 62) of the introduction seems overstated given the mainly empirical nature of the paper.\n2. Some of the tables should be improved. In Table 1, consider reorganizing the layout to help readers easily identify which experiments involve \"untrainable\" architectures versus \"untrainable\" tasks. In Table 2, highlight the target networks using different background colors for better readability. In Table 5, include the Deep FCN results from Table 2 to improve readability and allow for easier comparison.\n3. Given that distillation is a natural and well-established baseline, why was it not thoroughly compared and included in Table 2?\n4. The explanation regarding why untrained guide networks sometimes yield better performance in Table 2 is not fully convincing. In Appendix E, the Wide FCN paired with untrained ResNet shows lower similarity loss, yet paired with the trained ResNet achieves higher accuracy.\n\nQUESTIONS:\n1. The authors claim that \"Guidance can use a smaller untrained guide instead of a larger trained teacher.\" Would using a larger trained guide network further enhance the performance of the target network compared to using a smaller one?\n2. Have the authors considered aligning feature statistics (e.g., mean and variance) [1] during representation matching?\n\nRef.\n\n[1] Mirza et al., \"ActMAD: Activation Matching to Align Distributions for Test-Time-Training\", CVPR 2023\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to apply a guidance network to train a target network, the intermediate representations between the two networks are aligned. Thus, the target network could be less overfitted.\n\nSTRENGTHS AND WEAKNESSES:\nThe clarity of this draft needs improvement, some points are not clear enough to me.\n\n1. Line 2 “We call a network untrainable when they overfit, underfit..” This definition seems very vague to me. A model could be under- or over-fitted due to numerous reasons, but they are all trainable. And model regularization is a long-standing topic, maybe this study falls into this bucket.\n\n2.Line 10 “If the guide is trained, … If the guide is untrained,...” There exist different model architectures, some well-designed models can learn the inductive bias better. If the model is “untrained”, personally I don’t believe the model contains “the” inductive bias we want for down-stream models or tasks. We can discuss a little why the untrained may also slightly work later in this thread.\n\n3. Line 16 “Our method provides a mathematical tool…”, I can only find some empirical findings in the experiment based on the widely used loss and accuracy, is this the new tool provided in this draft? Please correct me if I am wrong.\n\n4. I can see the terms “mitigate”, “stop” overfitting occurred many times, personally, I highly doubted if the distillation-like solution is necessary for this purpose. There are different ways to prevent overfitting on network, simple dropout or l2 regularization. However, I didn’t see any mathematical discussion nor empirical comparison on different regularization. Which may deliver “cheaper” solutions for overfitting.\n\n5. Is it novel that the finding of using a guidance network can prevent overfitting? This is not very surprising. Given the guidance network is not overfitted, then aligning the internal representation between the two models will definitely “compromise” the guidance model. And this is core assumption behind distillation.\n\n6. Line 121, “We distinguish guidance from distillation…”, distillation can be applied to either the output or the intermediate features, I didn’t see any difference between this draft and other distillation studies based on Eq1, though the purpose could be slightly different. I won’t be surprised if a large trained teacher can do a better job for this purpose, given the better teacher learns a better representation which can better generalize on the validation data, then “compromising” on it can also deliver a better validation loss(less overfitted).\n\n7. Following the question in Q2, is the use of “untrained” model for a guidance novel? Some models maybe initialized as a Gaussian filter, which could be roughly used as a Gabor filter back to the era of hand-crafted features. Again, comprising on this un-fitted initialization can lead to a less over-fitted curve, but this procedure is meaningless. Aligning with unlearned filters will also lead to an under-fitted target network, which makes the use of untrained models unnecessary, the target model can not properly fit on any thing, tuning the loss weight is trivial in this case.\n\nTo sum up, applying distillation (I still believe the trick used in this draft is the same as the common distillation) singly for over-fitting seems not necessary, I belive other solutions can do the same job more efficiently. Distillation can be used to train a better target model for sure, and that superiority of the target model also includes less-overfitting.\n\nQUESTIONS:\nPlease see the weaknesses above.\n\nLIMITATIONS:\nI would suggest to better clarify the purpose of ths draft, if it tries to better regularize the model, then introduce the method and compare against other regularization methods comprehensively, the pros and cons, the performance and cost. If this draft intends to propose a better disllation process, then rethink the target task accordingly. I would encourage a major modification before resubmitting.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new approach to boost the performance of neural networks with \"wrong\" inductive biases. The authors show that training a neural network (target network) while forcing its internal layers to behave (in terms of CKA alignment) like the layers of another neural network (the guide) with \"proper\" inductive biases. They show this brings about improvements even when the guide network is untrained.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The work is very creative! It shows that inductive biases can be copied to networks from the weights only, bypassing the rigidities of the architecture. \n- It has a wide variety of experiments with a decent experimental design, including both image and language modeling. Showing good results across all of them.\n- This opens the door for exciting approaches to initialization, potentially allowing the transfer of inductive tendencies of one hard-to-train architecture to a fast-to-train architecture.\n- Remarkably, the approach does not require putting manual work into designing such initializations.\n\n\nWeaknesses (minor):\n- While, as I said, the approach is very novel as far as I can tell, the authors do not make the distinction between THEIR contribution (the transfer of inductive prior only) and ESTABLISHED approaches (teacher-student) and consider them together. I would make the distinction clearer. Having said that, it is mostly a presentation matter, as it is clear that the method works in the important case (i.e., when the guide is untrained)\n\n- Intiuition: I would have liked it if the authors could explain the mechanisms as far as they can (beyond the abstract interpretation of transfer of prior). For example, I think it is worthwhile to mention in passing that in the case of FCN -> ResNet, the inductive bias is probably a simple one (probably making the initialization matrix sampled from around the identity matrix). Other things I am curious about:\n* Intuition about how CKA can encode complicated information such as the inductive bias of a network (eg, how would the CKA of an untrained residual network differ from not untrained FCN, is it a matter of covariance matrices?)\n\n\n- Finally, I think that the initialization interpretation is somewhat more satisfying and less unsettling than that that requires training with the inductive bias throughout the entire training process.\n\nQUESTIONS:\n- Do initialization experiments give the same results as the distillation experiments (i.e., prior transfer term for first 300 step vs when the prior transfer term prolongs throughout entire training)? Is it true across language modeling too?\n\n- What if you take the covariance matrices of the guide network's states rather than a more \"complicated\" loss term? Does it imply a better init for RNN?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces guidance, a novel method to train architectures typically considered untrainable for certain tasks by transferring inductive biases from a guide network. The key idea is to steer a target network using representational alignment (e.g., CKA) with the guide. The target network is trained to minimize both the task loss and a representational similarity loss relative to the guide. The authors demonstrate improvements in training performance across architectures and tasks, for example, improving RNNs using Transformers, and mitigating overfitting in FCNs using ResNets. The paper includes empirical analysis to differentiate architectural and learned inductive biases, highlights cases where guidance helps (or doesn’t), and proposes practical applications such as new initialization strategies and architecture search.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n\n- The guidance mechanism presents a novel and creative way to inject inductive bias into architectures that struggle to train successfully on specific tasks.\n\n- The authors explore a range of architectures (RNNs, Transformers, FCNs, CNNs), tasks (e.g., copy-paste, image classification), and guide/target combinations to support their claims.\n\n- The method could be influential in improving training robustness and re-enabling the use of previously discarded architectures.\n\n*Weaknesses*\n\n- My main concern of this paper is the lack of theoretical justification of the method. The paper does not provide a clear theoretical grounding for why or how guidance helps prevent overfitting or underfitting in the target network. The mechanism by which representational similarity to guide network results in better generalization is not well understood or analyzed.\n\n- Another concern, that may prevent the method to be useful in practice, is the high memory and computation cost of the method. The method involves keeping the guide network (often a large model) in memory during training, and performing additional gradient computations for the representational loss. This raises concerns about scalability, especially for large guide networks.\n\n- The paper does not justify why the target network is trained to mimic the guide rather than simply using the guide network itself for the task. In some cases, using the guide directly may be simpler and more efficient, especially if the guide is already trained.\n\nQUESTIONS:\n- Can the authors provide theoretical evidence on why representational alignment to a guide helps avoid overfitting/underfitting, beyond anecdotal examples?\n\n- What are the memory and runtime costs of this method compared to training a standard network or distillation-based approaches?\n\n- Why not use the guide network directly for inference, especially if it already performs well on the task?\n\n- How sensitive is the method to the choice of guide network? Is there a risk that a poor guide could hurt performance?\n\nLIMITATIONS:\nYes!", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}