Title: Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Residual Connection

Orthogonalization

Orthogonal Residual Updates

Representation Learning

Deep Learning
Abstract: Residual connections are pivotal for deep neural networks, enabling greater depth by mitigating vanishing gradients. However, in standard residual updates, the module’s output is directly added to the input stream. This can lead to updates that predominantly reinforce or modulate the existing stream direction, potentially underutilizing the module’s capacity for learning entirely novel features. In this work, we introduce _Orthogonal Residual Update_: we decompose the module’s output relative to the input stream and add only the component orthogonal to this stream. This design aims to guide modules to contribute primarily new representa-tional directions, fostering richer feature learning while promoting more efficient training. We demonstrate that our orthogonal update strategy improves generalization accuracy and training stability across diverse architectures (ResNetV2, Vision Transformers) and datasets (CIFARs, TinyImageNet, ImageNet-1k), achieving, for instance, a +3.78 pp Acc@1 gain for ViT-B on ImageNet-1k. Code and models are available at https://github.com/BootsofLagrangian/ortho-residual.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors revisit a classic component present in most modern network architectures: residual connections. Using the Gram-Schmidt method, one can decompose the output of the residual branch in a ResBlock into the sum of two components: a component parallel to its input (and thus the main branch) and an orthogonal one. The authors show that the parallel component can point in the opposite direction (\"anti-aligned\") wrt. the main branch, which could annihilate the residual signal; they further conjecture that this could consequently hinder efficient information propagation in the network. \n\nUsing this decomposition, the authors create a new network architecture that simply subtracts the parallel component from the residual branch in every residual block, thus only keeping the orthogonal one. They evaluate this variation on residual connections on two different architectures (ResNets and ViTs) and multiple datasets, and perform ablation studies to show their effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n- The motivation behind the proposed modified residual connection is simple, clear and intuitive. \n- The additional computational overhead is clearly described (in variables). \n- The idea seems to be novel to the best of our knowledge. \n- Code is provided by the authors.\n- Experimental evidence comprises multiple architectures and datasets; The runs for the main results are repeated 3-5 times, and standard deviation is reported.\n- Experimentally (ref. Fig 4), the authors show a continuous transition between ordinary residual connections and the orthogonal residual connections with a monotonous response.\n- The experiment in Figure 3 provides experimental evidence for anti-alignment.\n\nWeaknesses:\n- The experiment in Figure 3 is moderately insightful, as 150 Epochs is already enough to reach an almost full training performance on Cifar10/100.\n- The reported baseline (Table 2) for ViT-B on ImageNet-1K seems suspiciously low with 71.09%. Google reports 85.49% performance for ViT-B/16 on ImageNet.\n- A rundown of the added overhead (e.g. time difference in ms on the same hardware) in the specific experimental scenario used would be nice, as less operations do not always translate to faster runtime in CUDA pipelines.\n\nOverall, I think the idea is novel, well motivated and executed; however the experimental evaluation raises some serious doubts (particularly that this method changes the layer-wise gradient norms, paired with an evaluation a single LR) that need to be cleared out before the paper can be considered for acceptance.\n\nQUESTIONS:\n- Can you explain the low baseline performance for the ViT models on ImageNet? The reported 71.09% are vastly under the usual baselines reported by Google or PyTorch which amount to 80-85%. This makes me doubt the significance of the results, especially as the ResNet runs are not reported on ImageNet. Generalization performance is highly dependent on learning rates, which could effectively differ between the methods, as the layer-wise gradient norms differ between your method and the baseline. A possibility of ruling out the learning rate dependence could be a quick logarithmic sweep across learning rates (maybe on a cheap dataset, if compute is an issue). Another would be to repeat the experiment, but constrain the layer-wise gradient norms to be equal between the runs.\n\n- I am confused by Table 7: results show that the orthogonal residual update performs worse than the regular residual connections for all 3 initializations. Why is that? What initialization is used in Table 2?\n\n- Can you provide a simple number on how much the runtime actually increased for the specific benchmarks used (e.g. ResNet or ViT-B on ImageNet)?\n\n- [1] Show a frequency dependency in the average correlation between the main and residual branch in ResNets (ref. Figure 7). The authors argue that this induces a \"Frequency-dependent signal-averaging\" which further dampens high frequencies in the loss surface, which they conjecture is one of the benefits of residual connections and more generally multi-path architectures. As your method renders the signal from the main and residual branch uncorrelated for all frequency bins, do you think this could negatively affect trainability?\n\nWe are happy to revise our score if our reservations about the experimental results are cleared.\n\n[1] Ringing ReLUs: Harmonic Distortion Analysis of Nonlinear Feedforward Networks (ICLR 2021)\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-06-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an orthogonal update strategy for residual connections, which only add the orthogonal component to the input stream. The authors provide some experiments of image classification for this strategy and demonstrate that this strategy can gain some improvement.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- The issue of residual connections which this paper focus on is important. The proposed orthogonal residual update strategy enhances the performance, especially on ViT models like Table 2 shows. \n\nWeaknesses:\n- The motivation is not clear. Why the scaling operations caused by the parallel component is not necessary? The magnitude is also an important factor of updating. What is the rationale behind performing orthogonal decomposition on the model's output stream in this work? Are there any cited studies that support this viewpoint?\n- The organization of the paper is not well-structured. For example, the Methods section lacks an analysis of the issue concerning the original residual connection.\n- The output space of the model is not necessarily a Euclidean space, which imposes significant limitations on the proposed orthogonal scheme.\n\nQUESTIONS:\n- The proposed orthogonal update rule can be rewritten as x_{n+1}=f(\\sigma(x_n))+(1-s_n)x_n, and this rule is similar with the skip connections of scaling in [1]. What causes the results in this paper to be opposite to those in [1]? Table 1 in [1] shows that this update rule is worse.\n- Why does the proposed method perform worse than the original residual connection under various initialization schemes in Table 7? Which initialization method was chosen in the main paper? \n- How does the proposed orthogonal residual update perform when applied to other commonly used modules and optimizers?\n[1] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Identity mappings in deep residual networks. In ECCV 2016.\n\nLIMITATIONS:\n- Lack of the issue concerning the original residual connection.\n- The Experiments section only includes image classification tasks, without addressing other problems or networks that employ residual connection structures.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the residual update applied in ViTs and ResNet. Instead of directly using the linear residual, the authors propose to only use the orthogonal components of the output layer while discarding the parallel components. The authors also designed two different ways to do the orthogonal residual update: feature-wise and global-wise, where the feature-wise design performs universally better than the global-wise orthogonal residual update. With this simple design, the proposed method shows potential for faster convergence and effective performance improvement when applied to smaller models on small-scale datasets. However, the effectiveness of larger models with larger-scale datasets remains to be further investigated. To analyze the proposed method, the authors have provided extensive ablation studies and theoretical analysis to highlight the effectiveness of the proposed orthogonal residual update.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This paper introduces an interesting idea to just apply the orthogonal residuals, which is argued in the paper to be the effective components in the residual update. Aligning with previous research, the idea is reasonable and has achieved relatively good performance at least on small models shown in the paper.\n- The authors have shown extensive experiments to analyze and explain why the orthogonal residuals are effective and dominant in the network training, and how they affect the performance of the model. \n- The overall writing of the paper is clear, and the authors have provided extensive ablation studies to analyze the proposed method.\n\nWeaknesses:\n- The proposed method seemed to be less effective on the larger model with larger-scale datasets or deeper networks. Can the authors provide more insights? Because residual update is designed for deeper and more complex networks, it would be good to show the effectiveness of the proposed method when applied to larger models.\n- Although the proposed method is interesting, and the theoretical analysis could validate the effectiveness of using the orthogonal residual update, the experiments did not provide strong evidence to clearly support the argument that the orthogonal residual update is more effective in the network training.\n- The reported results for ViTs on different datasets are relatively low, and this might be because different implementations of ViTs have been used in the paper. However, will the hyperparameter choices affect a lot in the performance of the proposed method? What is the robustness of applying the proposed method to different ViT variants?\n- Could the authors provide more theoretical or intuitive explanations to Figure 3 (b) and (d) regarding the cosine similarity between $x_n$ and $f(\\sigma(x_n))$? Because the relationship of the learned $f(\\sigma(x_n))$ and $x_n$ is very important in understanding whether the orthogonal component is effective than just using linear residual.\n- The authors have mentioned faster convergence of getting rid of the parallel residuals, but only update with the orthogonal residuals. Could the authors provide more efficiency comparisons in the paper?\n\nQUESTIONS:\n- In Figure 3 (b) and (d), the authors have shown an interesting plot where the cosine similarity between $x_n$ and $f(\\sigma(x_n))$ is increasing after the transition point for the proposed method. Does it imply that the proposed orthogonal residual is trying to learn the linear residual during training? I wondered at different training stages of the proposed method, how does the performance change? Will stopping training at early steps before the increase of the cosine similarity reaching a certain threshold actually help with improving the performance?\n- Why does the stability constant have a relatively large effect on the performance of the proposed method? Can the authors provide more insights?\n- How does the normalization affect the proposed method when applied in ViTs or ResNet? Based on the results shown in the paper, the norm of the proposed residual update is very distinct compared to the linear residuals.\n\nLIMITATIONS:\nThe authors have provided the discussion of the limitation of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper reconsiders the essential function of Residual Connection and proposes a new approach, Orthogonal Residual Update. This approach removes the components parallel to the residual stream that are assumed to be redundant and are added to the residual stream. This approach has been reported to improve image classification performance in CIFAR, TinyImageNet, and ImageNet-1k.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1 The motivation and idea of the proposed method are very simple, clear, and easy to understand.\n\n2 The idea is novel and valuable in the sense that it encourages reconsideration of the basic elements of the shortcut path from a new perspective.\n\n3 The proposed method improves the convergence of training, and in some cases, the reduction of training time is actually confirmed.\n\nWeaknesses\n\n1 It is not clear whether the main claim of this study is to “propose a practical method” or to \"deepen the basic understanding of shortcut paths\". If the former is the subject matter, then more careful performance evaluation and examination of various architectures, downstream tasks, and non-image domains, as pointed out in Weakness 3, would be necessary. On the other hand, if the latter is the goal, quantitative and qualitative analysis showing how the visual representation is actually changed is required, as long as the proposal is based on the understanding that richer features are acquired by removing redundant representations. In the current manuscript, the steps taken in both directions are somewhat insufficient and give the impression of being half-hearted.\n\n2 The discussion of the experimental results in Figure 3 alone is not convincing enough to strongly support the validity of the proposed method. For example, in the ordinary Shortcut Path, the cosine similarity tends to become negative as the learning progresses, which suggests that the parallel component of f(\\sigma(x)) is in the opposite direction of x. In other words, there may be a function inherent in the normal Shortcut Path that reduces the parallel components by learning, e.g., to suppress norm divergence. On the other hand, the proposed method tends to converge to large positive values of cosine similarity and to reduce the vertical component. These observations call into question the assertion made in this paper's premise that the parallel component is redundant and that rich features can be obtained by adding only the vertical component.\n\n3 Regarding the performance evaluation shown in Table 2, it is possible that the Top-1 Accuracy is low, especially in the Linear settings of ViT-S and ViT-B, and that the training has not converged sufficiently. For example, according to the original paper of Vision Transformer and [1], the Top-1 Accuracy is generally expected to be in the high 70% range. If there is a possibility of insufficient training, it is difficult to judge the effectiveness of the method based on these results, and I believe that more careful setup and validation is needed.\n\n[1] L. Yuan et al, Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet, 2021.\n\nQUESTIONS:\nWith regard to the three Weaknesses listed above, please respond to the following points.\n\n1 Is the main contribution of this research a practical method proposal? Or is it a fundamental study that contributes to the understanding of shortcut paths? Depending on your position, please clarify the main point of this paper.\n\n2 Regarding the change in cosine similarity seen in Figure 3, it appears as if the regular Shortcut Path behaves as if it removes redundant parallel components by learning. Do the extraction of the vertical components added by the proposed method really provide “richer features”? Please provide any additional discussion in this regard, including any additional analysis of visualization or representation quality.\n\n3 Are the experimental settings for ViT-S and ViT-B in Table 2 a comparison with sufficient training convergence?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}