Workshop Submission: Towards Making Untrainable Networks Trainable

Authors: vighnesh  subramaniam
tomaso  poggio
boris  katz
brian  cheung
andrei  barbu
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-24 05:14:57

Abstract: We demonstrate that architectures which traditionally are considered to be ill-suited for a task can be trained using inductive biases from another architecture.  Networks are considered untrainable when they overfit, underfit, or converge to poor results even when tuning their hyperparameters. For example, plain fully connected networks overfit on object recognition while deep convolutional networks without residual connections underfit. The traditional answer is to change the architecture to impose some inductive bias, although what that bias is remains unknown. We introduce guidance, where a guide network guides a target network using a neural distance function. The target is optimized to perform well and to match its internal representations, layer-by-layer, to those of the guide; the guide is unchanged. If the guide is trained, this transfers over part of the architectural prior and knowledge of the guide to the target. If the guide is untrained, this transfers over only part of the architectural prior of the guide. In this manner, we can investigate what kinds of priors different architectures place on untrainable networks such as fully connected networks. We demonstrate that this method overcomes the immediate overfitting of fully connected networks on vision tasks and makes plain CNNs competitive to ResNets.

Main paper: 
 introduction: 
 When creating neural networks, we tend as a community to follow recipes that select among a few architectures known to work for particular tasks [39, 9, 17] . Architecture is critical. The gains made on tasks like object recognition are attributable to imposing an inductive bias, i.e., a prior, on the design of new architectures [19, 2] . Convolutional networks unlocked many vision problems [31, 21] and the recent advent of the Transformer [46, 15, 1] did the same for language. Despite this, finding new architectures and overcoming the limitations of existing architectures remains a sort of "dark art". While an architecture imposes some prior, we often do not fully understand what that prior is. One example of this that remains an open discussion is the role of residual connections in making very deep convolutional networks easier to train [24] . If we fully understood the priors that architectures imposed, we could translate between priors and architectures freely -either by specifying the prior we want and then deriving the appropriate architecture or by dispensing with architectures and imposing the prior directly.Recent theorems [37] state that for each function which is efficiently Turing computable, there exists a deep network that can approximate it well. Furthermore, a graph representing such a function is compositionally sparse; that is, the nodes of the associated Directed Acyclic graph (DAG) represent constituent functions with a small effective dimensionality. A reasonable conjecture is that neural networks with an architecture which is similar to the DAG of the unknown target function are Training Loss Guidance between two networks makes untrainable networks trainable. Given a target which cannot be trained effectively on a task, e.g., a fully connected network which immediately overfits on vision tasks, we guide it with another network. In addition to the target's cross-entropy loss, we encourage the network to minimize the representational similarity between target and guide activations layer by layer. The guide can be untrained, i.e., randomly initialized. This procedure transfers the inductive biases from the architecture of the guide to the target. The guide is never updated. The target undergoing guidance no longer immediately overfits and can now be trained. Here we show an untrained ResNet guiding a deep fully connected network to perform object classification. The FCN alone overfits, the guided version can now be optimized. It has gone from untrainable to trainable.especially successful in learning it, as it is the case for convolutional networks for image recognition and similar tasks. Because we do not understand the relationship between the kinds of priors on the target functions that different architectures impose, even simple questions have no known answer, such as, does there exist an initialization of an FCN that makes it behave like a CNN, though the graphs of the function they represent are fundamentally different?To study these problems, we design guidance, which is a method to make untrainable networks such as FCNs trainable by transferring the prior of a trainable architecture to the representations of the untrainable architecture. Guidance applies a representational similarity comparison between a target untrainable network and trainable guide network across the activations of several layers of the two networks. We optimize both the task loss of the target network and the representational similarity between the two networks. We find strong improvement of the trainability and performance of our target network on the ImageNet task, indicating an ability to transfer a prior for spatial processing from a convolutional architecture to a fully-connected network. 
related work: 
 Representational Distance: Our method builds on several metrics that measure distance between high-dimensional activations extracted from neural networks or activity in the brain [28] . Some of these distance metrics make comparisons based on kernel matrices [29, 12, 11] or relative distances [30] between sample representations in a set. Others compute linear [47, 43] or orthogonal projections [3] from one set of representations to another. Others use canonical correlation analysis which finds linear relationships between pairs of vectors [38, 35] . These metrics are designed based on a set of desired invariant properties such as permutation invariance or invariance to linear transformations.Such approaches have been commonly applied in neuroscience for measuring representational distance of activations from networks and activity in the brain to understand which neural networks are architecturally most similar to the brain [47, 10, 44, 16] . Under this context, [20] has shown the inability of current representational distance metrics to distinguish representations based on architecture. This provides a basis for our experiments to use these metrics to align representations across different neural networks. This paper gives the foundation for our intuition that networks may have similar representations that allow for transferring inductive biases from one network to another. Recent work has explored the relationship between representational and functional distance of networks, discovering transformations between activations of networks and when these make networks functionally equivalent [28, 6] .Untrainable Networks: In this work, we focus on applications of FCNs and plain CNNs for image classification. FCNs have been applied for image classification, where small feed-forward networks were trained on object recognition datasets. These networks were designed with 3-5 hidden layers and less than 100 units per layer [34, 4, 25, 36] . The performance of these networks was very low, where the goal of the paper was to only maximize training fit rather than generalization performance [34, 4] . Other methods were introduced to prevent overfitting in fully connected networks using topological structure [42] or early stopping [7] , but these methods had poor fits on the training set due to complex architectural design or complex hyperparameter tuning. Deep convolutional networks have been applied to image classification [31, 5] as well but face problems with vanishing gradients, preventing deep stacking of convolutional networks.Model Distillation: Guidance shares a resemblance with model distillation [22, 18, 41, 23] . Distillation transfers knowledge from a teacher model to a student model by introducing a new component to the loss function that enforces the student model to behave like the teacher model [26, 48] . This can consist of penalizing the KL-divergence between the logit predictions of the student and teacher model.Representation-based distillation [45, 8, 32] and alignment techniques have been proposed to improve alignment between two networks. Certain works have proposed contrastive loss functions on output representations to distill teacher information into the representation space of the student. Other methods introduce correlation congruence or similarity preserving metrics for aligning two networks.Methods have been proposed that use CKA as an alignment approach between representations of two networks or with representations in the brain with notable improvement in network performance [40, 13] .We distinguish guidance from distillation. Guidance can use a smaller untrained guide instead of a larger trained teacher. This is due to guidance operating over intermediate activations of the network rather than the output of the network probabilities or output features, like distillation does. Guidance also operates at many levels at the same time, aligning many layers at once. This helps address the credit assignment problem that gradient descent has when tuning weights early in a network. We also consider many more networks for guidance than is traditional for distillation including networks which have very different architectures like Transformers to RNNs. Distillation is usually carried out between two closely related architectures. We apply guidance to do the opposite. 
methods: 
 Guidance introduces a term in the loss of a target network, N T , to encourage representational alignment with a guide network, N G . Only the parameters, θ T , of the target are updated -the guide's parameters, θ G , remain fixed. Per minibatch, representational similarity is computed between the activations of i G th layer of the guide, A G i G (θ G ), and activations from a corresponding layer i T of the target, A T i T (θ T ). We refer to the correspondence between layers of the guide {i G } and layers of the target {i T } as I. While this correspondence, I, could be complex as any two architectures can form a guide/target pair, here we choose architectures that make the correspondence obvious as is discussed later. For example, the stacked RNNs and Transformers have the same number of layers in our experiments.The target and guide receive the same input. Per minibatch, we collect activations from intermediate layers of both networks. Layers of guide network are mapped to layers of the target network. We formulate the loss in terms of minimizing the representational dissimilarity, M, i.e., the complement of a representational similarity metric, between guide and target activations layer by layer, summing the results. We only consider the centered kernel alignment, CKA, between the activations in this publication. Many other possibilities exist. Any representational similarity function which is differentiable could be used. Efficiency or incremental computation is much more important than it is in traditional applications since this operation happens for every minibatch.Given L T as the original loss of the target network, the guide network's original loss function is irrelevant. The guide need not even have been trained on the same task or the same dataset. It need not even have been trained at all. This latter setting is what allows transferring architectural priors without transferring knowledge from the guide to the target, as there is none in a randomly initialized guide. The final loss we optimize, L is:EQUATIONThis minimizes a task loss while increasing alignment between the target and guide networks given the mapping between them. The mapping may be sparse, not every layer needs to be used. This is important for guidance with transformers or stacked RNNs as will be explained later. We don't incorporate any weight on the layer-wise similarity component of the loss. Note that the guide's parameters, θ G , are constants, i.e., the guide is never updated. 
experiments: 
 We design several settings with different target and guide networks to thoroughly test our approach. We include image-based settings and sequence modeling-based settings as well. In choosing target networks, we consider a broad range of designs for networks that are not traditionally applied (e.g., an MLP in image classification). We consider these networks as "untrainable" for a wide range of reasons such as difficulties with getting a good fit on the training set, overfitting on the validation set, and/or poor test accuracy regardless of fit on both sets. In most settings, these issues are driven by algorithmic constraints of the target network, which becomes the inspiration for our design.We focus on image classification and use the ImageNet-1K [14] dataset for training and testing. We use the splits defined by the dataset. We report accuracy on the validation set for all experiments.We use three target networks: Deep FCN, Wide FCN, and Deep ConvNet. Deep FCN is a fullyconnected network with 50 blocks consisting of feedforward layers followed by non-linearities. This network is an untrainable architecture, lacking inductive biases to prevent overfitting and having vanishing gradients. Wide FCN is a fully connected network with 3 blocks with feedforward layers that have 8192 units. This is categorized as an untrainable task due to a saturation in the training performance. Deep ConvNet is the same architecture as ResNet-50 [21] , but without residual connections. This is categorized as an untrainable architecture due to the vanishing gradient problem. We use two guide networks: ResNet-18 and ResNet-50. ResNet-18/50 is a deep convolutional network with 18/50 convolutional blocks and residual connections. We refer to (author?) [21] for ResNet 18/50 design. We supervise the Deep FCN and Shallow FCN with ResNet-18 and supervise the Deep ConvNet with ResNet-50.For each setting, we train the base target network and an experiment where a guide network supervises the base target network. All networks are trained with cross-entropy loss. We use the Adam [27] optimizer with a learning rate of 1e-4. 
results: 
 We show image classification results with our three networks in Figure 2 and Table 1 .Across all our networks, we observe significant improvement from using a guide network to provide representational guidance. We see significant accuracy gains of 5-10% on the test performance. We also observe significantly better loss curves from a better fit with the training loss and less overfitting with the validation loss. Most interestingly, we highlight that using a randomly initialized guide network can perform better than using a trained guide network. For example, the Deep FCN results in Figure 2 is significantly better with a randomly initialized ResNet-18 as the guide network instead of a trained ResNet-18. This trend also occurs with Wide FCN.We note that this trend is not entirely consistent as indicated by the Deep ConvNet. One potentially explanation is that the architectural prior is the same for both the Deep ConvNet and ResNet-50. This explanation provides an additional interpretation for the role of residual connections and their influence on the representation space. Specifically, the power of residual connections are not just architectural, but training-based.Overall, these experiments establish that these three networks which are usually not applicable to image classification can be operationalized under our training approach, especially with randomly initialized networks. Training Epochs 6.0 Validation Cross-Entropy LossFigure 2 : Untrainable Networks for Object Detection. We train three networks on ImageNet both without access to a guide network and with access to a trained and untrained guide network. Base training results are shown in red, our approach with a trained guide network is shown in blue and our approach with an untrained guide network is shown in green. 
conclusion: 
 We introduced guidance, a new method for neural network training to make untrainable networks trainable. We applied our method to fully-connected networks and deep convolutional networks trained on ImageNet, guided by ResNet-18 and ResNet-50. Our method is simple and scalable, applicable to any pair of guide and target networks across many tasks. This method allows for studying unknown properties of neural networks related to initialization or optimization and their relation to the prior inductive biases of a network. Characterizing architectural priors and untrainable networks can be made more precise using our methodology.[47] Leila Wehbe, Ashish Vaswani, Kevin Knight, and Tom Mitchell. Aligning context-based statistical models of language with brain activity during reading. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 233-243, 2014.[48] Wangchunshu Zhou, Canwen Xu, and Julian McAuley. Bert learns to teach: Knowledge distillation with meta learning. arXiv preprint arXiv:2106.04570, 2021.A Guide-Target Network Mapping Methods 
 
 Appendix: To compare representations, we use a representation similarity metric, M, which corresponds to centered kernel alignment (CKA) [29] in our setting. We specifically consider linear CKA.CKA uses kernel functions on mean-centered representations to compute representational similarity matrices, which are then compared via the Hilbert-Schmidt Independence Criterion (HSIC). More specifically, suppose we have two sets of representations R ∈ R b×d1 and R ′ ∈ R b×d2 . We first compute the Gram matrices for each set of representationsEQUATIONWe center the Gram matrices by introducing a matrix, H whereEQUATIONWe compute the HSIC on the Gram matrices.EQUATIONFinally, we define our linear CKA metric as:EQUATIONIn our setting, we consider representational dissimilarity and aim to minimize the dissimilarity between representations from our target network and guide network. We define this as:EQUATIONLinear CKA ranges from 0 (identical representations) to 1 (very different representations). Because of this, we take the complement by subtracting the linear CKA from 1 to represent dissimilarity.We design a simple method for mapping guide layers to target layers as part of providing supervision. The goal of this method is to make guide and target networks architecturally agnostic i.e. we can supervise any target network with any guide network.As a simple approach, we evenly spread layer computations of our guide network over our target network. For example, if we consider ResNet-18 and a 50-layer FCN, we would map every ResNet layer to every second or third linear layer of the MLP. The intuition for this approach follows from the aim of discovering the same function of the guide network using a target network. Through the design of evenly spreading layers of our ResNet-18, we are guiding the MLP to find learn a function similar to guide network.For our mapping, we consider activations from layers with tunable weights i.e. convolutional, linear, or LSTM/RNN based layers. For multiple stacked RNNs, LSTMs, or transformers, we extract feature representations from intermediate layers in the stack as well. Skipping layers based on non-linear transformations reduces memory overhead associated with storing representations per batch.We provide an algorithmic description of guidance in Algorithm 1.Algorithm 1 Guidance: Guide Network Representational Alignment # collect layer activations 6:{A T i T } t i T =1 ← activations(N T (xj)) 7: {A G i G } l i G =1 ← activations(N G (xj)) 8:# Get step size between the number of layers between the two networks for layer mapping. 9:if l > 1 then 10:step ← (t -1)/(l -1) 11: else 12:step ← 1 13:end if 14:# Map the layers and add up layer-wise representational distance 15:total ← 0 16:for i = 1 → l do 17: index ← min(round(i × step), l -1) 18: rep ← M(A T index , A G index ) 19:total ← total + rep 20:end for 21:loss ← loss + total 22: end forOur work has a number of limitations. We aimed for initial improvements of trainability instead of maximal performance on any one task. This would have required us to carefully tune the hyperparameters involved. We preferred to show how guidance works in general rather than in cherrypicked or carefully tuned settings. To that end, we also did not optimize networks to convergence, nor did we attempt to experiment with other optimizers. Once we reproduced a well-known problematic training phenomenon, we showed that it could be overcome. We consider a network trainable enough to overcome a problem when the original problem disappears. For example, successfully training fully connected networks for object recognition was hopeless because they immediately overfit; using our guidance method they no longer do so. This does not mean that they are necessarily useful as object recognizers at present. In the case of fully connected networks, their present performance with guidance training is too low, but with additional work, we believe their performance could be substantially increased now that their train and test loss are moving in the right direction. In some cases, by applying guidance, we do see large useful improvements.Our guide network supervision through representational alignment has one primary limitation due to increased memory usage during training. Due to saving activations across several layers of the two networks, GPU memory usage increases dramatically. Moreoever, our methodology works better as batch size increases since this allows for better approximation of representational similarity, increasing memory usage even more. Furthermore, including more layers for supervision leads to improved results.In this paper, we introduce simple techniques to handle memory constraints such as gradient accumulation and gradient checkpointing [33] . In practice, more memory optimization techniques may become necessary to consider larger untrainable networks. Further work could consider using 


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Developed an interesting approach to improve the performance of otherwise underperforming neural networks. These underperforming neural networks are Deep FCN, Shallow FCN and Deep ConvNet.
main_review: main_review: - Authors have taken inspiration from neuroscience to develop the approach of making untrainable networks trainable. They have defined a method that introduces a source network (one that acts as a guide) and a target network (one that is untrainable). They have optimized both task loss and representational similarity between the source and target network. The task is clearly defined. The results of the experiments suggest an improvement. The approach provides a novel way of understanding these untrainable networks.
- The task has been performed on a limited number of target networks. It would have been interesting to see how one can scale this approach to various other untrainable networks.
title: title: The authors propose an effective method to train typically hard-to-train networks (targets) by aligning their features with those of a more suitable architecture (source). While the approach is novel, its practical utility is unclear. The method requires training the target while utilizing the intermediate features from the source network. This raises questions about the end-product's value compared to simply using the source network.
main_review: main_review: Clarification needed on training with untrained source network setting:

1. Are both source and target networks trained in parallel, with an additional alignment loss for the target?  
2. Or are features from a randomly initialized source network used to align the target's features?

If it's the latter, effectiveness seems unlikely as randomly initialized CNN features, while structured, would be unrelated to the downstream task. Please clarify the exact procedure and its rationale.

**Motivation concerns:**   
The paper's motivation for training disadvantaged architectures (e.g., MLPs) is unclear. The method uses a pretrained source network (e.g., CNN) on the same dataset to guide the target network. This raises two key issues:

- The source network likely outperforms the target network.  
- It's unclear why one would use the target network over the already trained source network.

**Experimental comments**

- Aligning the intermediate features using alternative similarity metrics (e.g. cosine similarity, mse) could be interesting.  
- Effect of aligning only the penultimate layer features instead can be used as a lightweight comparison.
title: title: Interesting narrative, but need proper references for better context
main_review: main_review: The paper introduces a method to make "untrainable" neural networks trainable by using a source network as a guide. Essentially, this closely aligns with the traditional teacher-student framework, where the source (teacher) network supervises the target (student) network through representational alignment.

While the paper presents an interesting perspective, particularly in applying these concepts to traditionally difficult-to-train architectures, its novelty is more in its framing than in the core approach.

The paper makes some notable contributions, including the interesting finding that randomly initialized source networks sometimes outperform trained ones - a phenomenon that distinguishes this work from typical teacher-student approaches. However, the paper does not adequately address the connection between its method and the well-established teacher-student framework, which is a significant oversight.

The related work in the introduction is notably lacking in terms of references to existing work on knowledge distillation, model transfer, and other teacher-student approaches, which would have provided clearer context and differentiation. Furthermore, while the experiments demonstrate meaningful improvements across several different types of traditionally difficult-to-train architectures, they are not sufficiently benchmarked against existing teacher-student methods, making it difficult to assess the true contribution relative to the SOTA. Without these comparisons, the claim of significant performance improvements remains isolated and lacks grounding in the broader literature.

Suggestions to the authors to improve their paper:
- Add related work references to address the connections with existing teacher-student frameworks.
- Include comparisons to SOTA methods in the teacher-student paradigm.

In conclusion, while the work presents an interesting narrative and shows promising results for improving traditionally difficult architectures, it needs stronger positioning and benchmarking against relevant, existing frameworks to better highlight its contributions.