Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networks
Authors: Giyeong Oh
Woohyun Cho
Siyeol Kim
Suhwan Choi
Youngjae Yu
Venue: NeurIPS 2025 poster ()
Keywords: Residual Connection
Orthogonalization
Orthogonal Residual Updates
Representation Learning
Deep Learning
Primary area: deep_learning
TL;DR: Our Orthogonal Residual Update improves deep networks by adding only novel, stream-orthogonal module outputs, boosting generalization, stability, and efficiency.

Abstract: Residual connections are pivotal for deep neural networks, enabling greater depth by mitigating vanishing gradients. However, in standard residual updates, the module’s output is directly added to the input stream. This can lead to updates that predominantly reinforce or modulate the existing stream direction, potentially underutilizing the module’s capacity for learning entirely novel features. In this work, we introduce _Orthogonal Residual Update_: we decompose the module’s output relative to the input stream and add only the component orthogonal to this stream. This design aims to guide modules to contribute primarily new representa-tional directions, fostering richer feature learning while promoting more efficient training. We demonstrate that our orthogonal update strategy improves generalization accuracy and training stability across diverse architectures (ResNetV2, Vision Transformers) and datasets (CIFARs, TinyImageNet, ImageNet-1k), achieving, for instance, a +3.78 pp Acc@1 gain for ViT-B on ImageNet-1k. Code and models are available at https://github.com/BootsofLagrangian/ortho-residual.

Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networks
Giyeong Oh
Woohyun Cho
Siyeol Kim
Suhwan Choi
Youngjae Yu
Yonsei University
Maum.Al o
(hard2251, k106419, cykiim052)@@onneeii ac. kr
claude@maum ai
Seoul National University" youngjaeyu@sni ac kr
Abstract
Residual connections are pivotal for deep neural networks, enabling greater depth by mitigating vanishing gradients. However, in standard residual updates, the module's ' output is directly added to the input stream. This can lead to updates that predominantly reinforce modulate the existing stream direction, potentially underutilizing the module's capacity for learning entirely novel features. In this work, introduce Orthogonal Residual Update: we decompose the module's 's output relative to the input stream and add only the component orthogonal to this stream. This design aims to guide modules to contribute primarily new representa- tional directions, fostering richer feature learning while promoting more efficient training. We demonstrate that our orthogonal update strategy improves generaliza- tion accuracy and training stability across diverse architectures (ResNetV2, Vision Transformers) and datasets (CIFARs, TinylmageNet, ImageNet-1kk 1k), achieving, for instance, +3.78 pp Acc@ @ gain for VIT-B ImageNet-lk. Code and models are available at https: : // gi thub. com/ Bttt//oooooooaaaggooee haam//othie ortho-aa resi dual.
Introduction
Residual connections [1] have been a cornerstone in deep learning, fundamentally enabling the training of substantially deeper neural networks by mitigating vanishing gradients. The original ResNet architecture [1] updated internal state I, via Ix+1 a
[(x,)), where the non-linear activation Jact was applied after the summation, meaning In did not propagate purely linearly. Subsequent work, notably ResNetV2 [2]. introduced full linear mappings of the form In+1 x, + (x, This design, where pre (e.g., normalization and activation) precedes the residual function f and no transformation follows the addition, allows the unmodified 2,, to serve as a linear residual stream [3] that passes representation directly across layers. This principle of linear residual stream now prevalent feature many modern bigh-capacity architectures, including contemporary Transformers [4, 5] and large language models [6-12] that predominantly employ pre-layer normalization [2, 13].
In such architectures, complex modules f (e.g., attention or MLP modules) operate on a (or nor- malized version of this, (2x)) linear residual stream In, and their output f(((2x)) additively combined with En. Conceptually, this module output f(a(),)) can be decomposed with respect the input stream Fn into two components: fl, parallel to Fne and fi, orthogonal to T,,. The
"Coreesponiigg author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Linear Residual Update
Orthogonal Residual Update
f(x (
f(xn)
f f(xn)= fii
((n)
fL
Module
Module
Cr,
De m+1
C m-1
Figure 1: Intuition behind our orthogonal residual update. Left: The standard residual update adds the full output of module f() to the input stream Zn. Right: Our proposed update first decomposes the module output fxx,) into a component parallel to In (ff) and a component orthogonal to En (f_). We then discard fil and add only the orthogonal component fi to the stream.
component parallel to In can effectively rescale the stream, prompting the question whether updates should prioritize directions not already present in hn. These considerations raise whether module capacity is best allocated to novel directions rather than further modulation of the existing stream. Throughout, we write fl Sn Zn and f- f(o((z)) Sn Zn. where Sn denotes the scalar projection of (zx)) onto Zn.
Such scaling need not use the full expressivity of f: isolating f1 emphasizes contributions in novel directions. The orthogonal component fi, in contrast, inherently introduces new directional components into the representation space, distinct from the current stream In- We hypothesize that by explicitly isolating and utilizing only the orthogonal component fl for updating the residual stream, modules are able focus on contributing novel aspects to the representation.
As proof of concept, we study the Orthogonal Residual Update, replacing #n+1 = xn + f'aaaaaaa )) with 1x++1 Ir + fu (..).. As conceptually illustrated in Fig. 1, instead of linear additive update In + fa (z.) )). our approach employs In + fu (Ix), where fi (xx)) is the component of the modulc's output f(r (x))) explicitly made orthogonal the input stream Tn. We evaluate this approach on ResNetV2 [2] and the Vision Transformer (ViT) [5] across standard image classification benchmarks, including the CIFAR datasets [14], TinyImageNe [15], and ImageNet- 1k [16]. Across ResNetV2 and ViT on standard benchmarks, we observe improvements in generalization and distinct training dynamics, as shown in Figs. 2, 3. Our main contributions are:
We re-examine additive updates in networks with linear residual streams, noting that the component parallel to the input stream may rescale-and and at times become anti-aligned with-the stream, potentially impeding information propagation.
We propose simple, principled modification, Orthogonal Residual Update, which isolates and uses only the orthogonal component fi to encourage novel directions and mitigate interference from f.
We empirically show improvements in generalization, training stability, and overall efficiency across ViT and ResNetV2 on standard image classification datasets.
2 Related Works
Modifying the Stream Itself Beyond the standard linear skip connection [2], prior work has modified the skip path 1x, to control propagation encode inductive biases by replacing / with fixed linear transforms P or I (e.g., norm- womm-preerrvngg orthogonal P [17] or structure-inducing entangled I' [18]). These approaches fundamentally alter the skip (i.e., I++ P/T) and analyze how changed spectral/sparsity properties affect learning; in contrast, we preserve the linear skip I and refine only the additive update í (x(x) combined with it.
Orthogonality in Deep Learning Orthogonality is a recurring principle in deep learning, valued for promoting stable signal propagation and beneficial representation properties. Orthogonal weight initialization [19, 20] and training-time enforcement via regularization or manifold-constrained optimization (e.g., Stiefel) [21-24] are common tools; more recently, Orthogonal Finetuning (OFT) adapts pre-trained models via orthogonal weight transforms [25]. At the optimizer level, Muon [26]
orthogonalizes updates via Kewton-Schull iterations.

---

Linear Residual Update
Orthhg Update
70%
6.00
60%
Train Loss
50%
5.00
40%
30%
4.00
20%
3.00
Orthogonal Residuel Update Linear Update
60k
Training Step
Relative Wall Clock (Linear 100%)
(a) Training loss training iterations.
(b) Validation accuracy vs. relative wall-clock.
Figure 2: Orthogonal update accelerates convergence and enhances generalization efficiency com- pared to the standard linear update baseline (ViT-B ImageNet- 1k results shown). (a) Faster Convergence: Orthogonal update (blue) achieves significantly lower training loss in fewer iterations. (b) Improved Time-to-Are Time-to-Accuracy: Acuracy: Orthogonal update (blue) attains higher validation Ace @ consis- tently outperforms linear update with minimal overhead.
Orthogonal Residual Updates
3.1 Preliminary
For any non-zero vector 2in Rd module output (((x)) be uniquely expressed the of two distinet components: one component that is directly related to the current stream In. and another component that is independent of Fn. We can write this abstractly as:
f(o(zz)) = fi + fi.
(1)
where fli is the component parallel to Int and fi the component orthogonal to Ine Since fi must lie along the direction of Zn, it necessarily takes the form fl = azn, This projection step. analogous to the fundamental operation the Gram-Schmidt orthogonalization process, yields a as follows:
(Rn, ((o(x)) and fi f(o(zw)) ful
A.In, where
(2)
(In,
Here (-.-) denotes the dot product. Consequently, the orthogonal component fi. capturing the part of (x(,)) linearly independent from Fn. obtained by subtracting this parallel component. By construction, this component satisfies (In, fi) = O. Interpreting these components, fi signifies the portion of the module's output that merely scales the direction already present in the stream En- Conversely, fi indicates the 'novel' representation contributed by the current module relative Tn.
3.2 Orthogonal-only Update Rule
Building upon the decomposition introduced in the previous subsection, now propose update mechanism. Instead of adding the full module output (((())) to the stream as in standard residual updates, we advocate for using only the component fi (x..) derived from f(a ((r))), as illustrated in Fig. 1. This component explicitly defined as:
(
fa (=,) ((((())) -,,
where sn || +
Note that we now explicitly include the small constant > 0 in the denominator for numerical stability during computation, detail omitted in the ideal decomposition in Sec. 3.1. Throughout our experiments, we set = 10 sensitivity is reported in Sec. 4.7.
Our proposed orthogonal-only update rule is thus simply:
Fn+1 = Fr fi (2.)).
This formulation is motivated by two key properties. First, from a geometric perspective, it acts as an efficient approximation of an exponential map on the representation manifold, leveraging the local Fuclidean structure of the tangent space at x,, Second, from practical optimization standpoint, it preserves the identity gradient path essential for stable training deep networks. The full derivation is deferred to Appendix B.
3

---

3.3 Implementation and Computational Overhead
Algorithm 1 Orthogonal Residual Update
Input: stream module oulput f. dimension set D (non-batch dims to reduce over), stability >> 0.
1: 8m < sum(un @ f. D, keepdim=True)
2: < sum C Im: D, keepdim=True)
3: Sn Sm/ (sa +)
fi - 8n Zn
return In f'
Alg. subsumes both variants via the choice of the reduction set D: feature-wise uses D {idx_ff }. i.c. reducing over the feature axis (Transformer: d; CNN: channel dimension C²), whereas the global variant sets to all non-batch dimensions (i.e., fatten-then-reduce)) Setting D to all non-batch dimensions is also possible; however, it ignores structural priors of attention heads and convolutional kernels. Since global projection can interfere with attention, we only evaluate the global variant on CNNs in this paper.
(a) Approximate FLOPs per Transformer block. Thop d desdla: FFN assumes a Ad expansion. Our feature-wise orthogonal connection introduces only ((s)) FLOPs on top of the block.
(b) Training throughput (img/s) and over- head (%) of Ortho-F relative to the linear residual baseline.
Module
Connection
Total FLOPS
Arch.
Linear
Ortho-F
Overhead
Lincar
ns Sad2 + 442d ad
ResNetV2-34
1737.2
1634.0
5.94%
Attention
Orthogonal
ss Sad2 4 4s2d ad 6sd 2s
ResNerV2250
1002.8
876.7
12.58%
Linear
16sd + sd
VIT-S
3476.1
3466.3
0.28%
MLP (FFN)
Orthogonal
16ed + ad + 6sd + 2s
VIT-B
1270.1
1246.2
1.88%
Table I: Computation practice. Orthogonal projection adds O(sd) FLOPs per block (bold in (a)); throughput in (b) measured under identical conditions.
We instantiate the update either feature-wise (default) global, both captured by Alg. 1 with different reduction sets D. Feature-wise vectorization-friendll and requires only ((sd) FLOPs per block (ss 6sd for the projection plus 2s for normalization), negligible relative attention/FFN (Tab. la). Empirically, the throughput overhead of Ortho-F (Orthogonal feature-wise) is modest (Tab. lb): < 2% on ViT-S/B and ~3-13% on ResNetV2 under identical batch sizes and hardware; detailed PyTorch code and AMP/compilation notes are in Appendix G.
3.4 Observed Internal Dynamics of Orthogonal Updates
Before turning to large-scale results, we first examine the in-block dynamics of VIT-S on Tinylm- ageNet (5 seeds). We track two diagnostics that probe the mechanism: (i) the stream norm ||>,,]| and (ii) the magnitude of the parallel component of the module output, Il (z.,)l recalling that (xx))) 8,,n, and fi (xr,) (()() 8n with 8, defined in Eq. (3). Summary curves P/Attentiion,, blocks 0-5) are shown Fig. 3; cosine similarity and additional per-block traces are deferred to Appendix D.
We consistently observe an early, layer-dependent juncture where trends diverge (indicated by upward arrows in each panel): for the linear pathway, the parallel contribution diminishes and ||=,,11 typically exhibits a peak followed by decrease; for the orthogonal pathway, ||,,, stabilizes while the orthogonal component is maintained. Sign-sensitive trajectories (e.g.. directional alignment) and the Jacobian analysis are deferred to Appendix D and B.4.
The Transition Point.
Key observation I: Linear induces parallel suppression. Across layers, the linear residual update exhibits a reduction in the contribution of the parallel component of the module output. Concretely, in Fig. 3 (b,d) the carallel-component energy I (Ixx)]] for the linear baseline (red) typically declines after the Transition Point and often settles near a low plateau most clearly in deeper MLP and Attention Blocks-indican that the model progressively suppresses fi during training. In contrast, orthogonal variant (blue) maintains or grows | as the module learns its full output while

---

MIP Blcek 0
MEP Wloc
F Block
MLP Slock 0
MLP Block 1
15-
NLP Block
300
[
t0
500
100
200
Transhen
364
1ik
26k
66
1000
1200
1500
550
1000
40D
10-
16k
24t
26
10s
24h
De.
304 20x
81
18k
24h
26
1ik
Steps
Taining Steps
Tainiin
Trining
Training
(a) Stream norm, ML.P blocks.
(b) Parallel component norm, MLP blocks.
Aerention
Attertions Birck
Attertion Bleck
Attention Albertion
Attention Sorkk
200
100
16k
23x
24k
16k
164
24k
164
2
8k
16k
24h
Attention Block
Attention Dlock
Attention Block
Attention Diock
Attention Block
Altention Block
750
900-
3300
[xw))
002
1000
250
500
Training
Thaining
Ortigeesa Insidcal
dual
Libear
(c) Stream norm, Attention blocks.
(d) Parallel component norm, Attention blocks.
Figure 3: Internal dynamics (ViT-S,
5 seeds). Each subfigure shows blocks 0-5 (MLP top, Attention bottom). Ours denotes orthogonal updates: Linear denotes the standard residual. (a,c) After the Transition Point, orthogonal updates stabilize the stream norm ||=,,]1. whereas linear updates typically exhibit post-transition decrease. (b,d) The parallel component energy [1 )l( follows distinct layer-wise profiles for linear vs. orthogonal updates. Signed parallel coefficients and orthogonal-componen traces are analyzed in the Appendix D.
the update remains orthogonal. This confirms that parallel suppression occurs under the linear residual pathway.
Key observation I:: Orthogonal stabilizes the stream norm. After the Transition Point (Fig. 3 (a), (c)), the orthogonal update #n+1 = In + (xx) stabilizes the stream norm ||>,,| and the updates act predominantly as rotations on near-constant-norm state. Importantly, we do not constrain the module f(() itself to be orthogonal; we only add the orthogonal component of its output to the stream. Thus f()) may still contain a parallel part, and since (In, fi (Ix)) O (finite e and precision), parallel alignment can re-enter the module output; see Appendix B for analysis. Although the module output (Ixx) can still accumulate parallel component during learning (via gradients on s,,) and small numerical bias may remain due to the stability constant e, the update path discards the parallel term by construction. Hence, there is no direct positive-feddbar loop via the residual addition that would amplify or suppress the parallel component; instead, orthogonal energy is preserved and steers representation change directionally.
3.5 Stream Scaling vs. Geometric Projection
Having observed parallel suppression under the linear pathway, we ask whether simply learning stream gain can reproduce this behavior. To test whether granting the model explicit freedom to rescale the stream leads it to suppress the parallel component, we consider learned stream-scaling variant
Tn+1 = Fnn + 0r Fn + [(x)) (equivalently #n = (1 + +z))a + f(x)))
(5)
where ag is learnable per-layer scalar initialized at O. Fig. visualizes the trajectories ay (step) by block. Granting this freedom yields systematic suppression of the parallel pathway in MLP blocks (negative drift of ax) and small, near-zero or mildly positive values in Attention. Thus, learned rescaling primarily adjusts layerwise magnitude, whereas our method computes stream- dependent projection 8, (Ir) to remove the parallel component and update with fa (x,).. However, ag is input-invariant: there is no single ag that makes ael x 2| + (f(o xx,) = 0 for all z,, so
5

---

Altettinn
Atteriion Macc?
MIP
M Block
1.5
3.5
d 5.0
Steza Anerddnn
Slega oct
21139
-0.
Staps
-22
Strps Bock
6.5
00.00
t oo
d
8 -25
1.80
-92
Stppa
Stags
Steps
(a) Attention blocks: ar(step).
(b) MLP blocks: ai (stet))).
Figure 4: Learned stream-scaling variant: per-block trajectories ar (median across seeds; band interquartile range). Values below zero correspond stream attenuation. This diagnostic contrasts a layerwise, input-invariant scalar with our input-dependent projection.
simple scaling cannot guarantee an orthogonal update. In contrast, 8n (mn) depends on the current stream state and removes the parallel term. Beyond this learned rescaling, Appendix E introduces unified residual family parameterized by (pe. 0t) that mixes parallel/orthogonal contributions, i.c.,
Tn+1 2n + pe(sin de fl (z+) + cos 0g (Ix)).
4 Experiments
4.1 Experimental Setting
Datasets. We evaluate our orthogonal update mechanism against standard linear connections [2] by training models from scratch on several image classification benchmarks: CIFAR-IO, CIFAR- 100 [14] (both 32 x 32, 10/100 classes, 50k train10kk val. images), TinylmageNet [15] (64 x 64, 200 classes, 100k train/10k val.), and ImageNet-Ik [16] (224 224, 1000 classes, æ1 w1.28M train/50k val.).
Standard data augmentation strategies appropriate for each dataset and architecture were employed (full details in Appendix A.
Architectures. Our experiments utilize two primary architecture families: ResNetV2 [2] (employ- ing standard configurations like -18, -34, -50, -101) and Vision Transformers (ViT) [5] (VTT-S: 384 hidden dim, layers, 6 heads; ViT-B: 768 hidden dim, 12 layers, 12 heads). For ViTs, input images are processed into patch embeddings with [CLS] token and 1D positional embeddings. and the [CLS] token's 's output representation is used for classification. Due to computational constraints, our ImageNet- Ik evaluations focused on VIT models. All models use a final linear classification layer.
Training Setup. We follow standard protocols for each architecture family. ResNerV2 models trained for 200 epochs with SGD [27] (batch size 128); ViT's for 300 epochs with Adam W [28] (batch sizes 1024 for smaller datasets and 4096 for ImageNet-1k 1k). Full hyperparameters (e.g., learning- rate schedules and warmup, weight decay/momentum or Adam betas, label smoothing, gradient clipping). augmentations, patch sizes, and compute details provided in Appendix A.2. We use stability constant e 10-6 by default; sensitivity and mixed-precision notes are deferred to Sec. 4.7 and Appendix G.
We emphasize that our VIT experiments follow the challenging academic paradigm of training from scratch on ImageNet- Ik, rather than leveraging pre-training on large-scale proprietary datasets. This widely adopted setting requires strong regularization (e.g.. MixUp. CutMix) to ensure robust convergence. Consequently,
results should be interpreted within the context of rigorous paired comparison, designed to isolate the precise impact the residual update mechanism.
4.2 Image Classification
We evaluated our orthogonal update mechanism by training various ResNetV2 and Vision Trans- former (ViT) architectures from scratch on standard image classification datasets using consistent augmentation strategies per architecture family (see Appendix A for full settings). Due to computa- tional constraints, our ImageNet-I1 k experiments focused on VIT models. All performance metrics presented in Tab. 2.

---

Table Mean std. Val Acc@1 (%) on runs. Results are averaged over the 5 best validation epochs from each run. Performance of standard Linear updates is compared against our orthogonal updates: Orthogonal-F (Feature-wise) and Orthogonal-G (Global). Due to computational constraints, ImageNet- Ik experiments focused on ViTs.
Architecture Connection
Dataset (Acc@1 1 % mean std.)
CIFAR-10
CIFAR-100
TinyImageNe ImageNet-1k
Linear 89.82=0034
71.92:00:4
51.30.2040
70.76
ViT-S
Orthogonal-F F
90.61
73.86
52.57:071
72.53
Linear
87.28=0441
n8
55.29
73.27
VIT-B
Orthogonal-F
91.93
75.07
57.87
77.05=0021
Linear
95.06=0.15
20.67
62.04.:0.9
ResNetV2-18
Orthogonal-E
95.26::0112
77.87
62.65-10.14
Orthogonal-C
95.25=0.11
77.53
62,32
Linear
95.49
78.92:10.1
64.61
ResNetV2-34
Orthogonal-F F
95.75=0.13 78.97
65.46-10.30
Orthogonal-C 95.53=0012
78.71:2024
65.38
Linear
94.75
17.90
63.74.1..8
ResNetV2-50
Orthogonal-F
94.71
77.43
64.22
Orthogonal-6 G
94.75=0.10
77.56:2034
64, 14..40:0336
Linear
94,86
77.72:t0.3
63.77:10:22
ResNetV2-101
Orthogonal-F F
94.80=0.13
78.50:20.26
65.78:10.22
Orthogonal-G G
94.75=0.13
78.37
65.87:1023
Orthogonal updates demonstrated particularly strong benefits in VIT models. For instance, ViT-B with our method achieved notable +3.78 pp increase in Acc@ on ImageNet-Ik over the baseline. In contrast, while generally positive, the performance gains from orthogonal updates on ResNetV2 architectures appeared more modest relative those in ViTs. We posit that this difference in efficacy may, in part, be related to the Diimeniionalityto-dde families. See Appendix for the definition and estimates of ?.
ratio", characteristics of these architecture
Table 3: VIT-S LR sweep CIFAR-10/100: Val Acc@
1 (mean over 3 runs).
CIFAR-10
CIFAR-100
LR
Linear
Orthogonal
Linear
Orthogonal
5 10-4
90.41
90.56.1032
71.46=0.59
72.48
10-4
90.45
90.91
7..13
72.99
1 x 10
89.95-0.36
90.36
70.59_0.0
73.11
2 x 10
84.85
87.38
62.17=1.21
69.71 1099
10
66.09
72.20
42.61 22882
48.24
Learning-rate robustness. We sweep the initial learning 5 x 10-3) while keeping all other hyperparameters identical to the main recipe (optimizer, schedule, regularization, and augmentation). As summarized in Tab. the orthogonal update consistently outperforms the linear baseline.
logarithmic grid (5 10-1 to
4.3 Representational Metries
Relative to the linear-residual baseline, the orthogonal update (i) increases Effective Rank and Spectral Entropy (broader, more uniform spectra), (ii) markedly reduces feature standard deviation (more stable activations), and (iii) yields nontrivial linear CKA [29], suggesting structural differences.
The metrics are defined as follows:
Effective Rank is defined as exp( ()), where H == - Σ3 log the spectral entropy, and p, := >x/ E, X are the normalized eigenvalues of the feature covariance matrix.

---

Table 4:
Representational metrics on ViT (ImageNet-Ik) 1k).
Metric
Linear
Orthogonal
Effective Rank
$72.9
599,9
+4.7%
Spectral Entropy
6.512
6.539
+0.41
CKA (linear)
0,546
Feature Std. Dev.
0.407
0.193
-52.5%
CKA (linear) [29]: A similarity metric for two representations Y,Y defined as X T
[X IXTX|Ia TYT
Feature Std. Dev.: The average per-feature standard deviation, 1 Std(Fx).
Table 5: Connection-type transfer ViT-S (with reset). Each trains 300-300 epochs on the same dataset. Optimizer/LR scheduler are re-initialized at the switch. Optimizer states are cleared at switch points isolate connection effects.
Dataset
Start Arch.
End Arch.
Ace@ (%)
Acc@5 (%)
Linear
Linear
92.78+0m6
99.74=a08
Linear
Orthogonal
92.88+0.14
99.72=a08
CIFAR-I0
Orthogonal
Linear
93.89
99.75=004
Orthogonal
Orthogonal
94.10
99.73
Linear
Linear
74.22.10.3
92.261.0.33
Linear
Orthogonal
74.02.10.4
91.9610.17
CIFAR-100
Orthogonal
Linear
75.63
92.91
Orthogonal
Orthogonal
75.38
92.20
Linear Linear
53,24.0.13
75.251021
Linear
Orthogonal
14
74.20=a20
Orthogonal + Linear
54.58=t0.10
76.45=024
Orthogonal
Orthogonal
53.88:10.9
75.34
Table 6: Continuous training (without reset). VIT-S trained for 150 - 150 epochs without mid-
training re-initialization.
Dataset
Start Arch.
End Arch.
Ace@ (%)
Acc@5 (%)
Linear
Linear
89.82+034
99.65=a08
Linear
Orthogonal
91,00+014
99.66=am
CIFAR-10
Orthogonal
Linear
18.18:005
99.72=001
Orthogonal
Orthogonal
90.61+021
99.69=a08
Linear
+ Linear
11.92
92. 1:0.18
Linear
Orthogonal
71.64100.6
91.961-024
CIFAR-100
Orthogonal
+ Linear
74.14
92.69::0.19
Orthogonal
+ Orthogonal
73.86+4031
92.23
Linear Linear
51.30.1.10
75.1910066
Linear
Orthogonal
50.78:t0.22
73.91
TinyImageNet
Orthogonal Linear
53.3310022
76.06:044
Orthogonal
Orthogonal
52.57:071
75.33
4.4 Ablation on Adapting Connection Types
We study how training adapts when switching residual-conmection types in VIT-S on CIFAR-10/100 and TinylmageNet. evaluate two regimes that disentangle potential confounds:
with optimizer/scheduule reset at the switch to isolate the pure effect of changing the connection.
without re-initialization, changing only the resietal-connection type while preserving the optimizer state and learning-rate scheduler (continuous training).

---

4.4.1 Switching with optimizer/scheddller reset
As summarized in Tab. 5, O consistently surpasses L-L. Moreover, O-L often matches or exceeds 0->0, whereas -~ shows no consistent gains over L-L.. This pattern suggests that orthogonal updates are most beneficial early in training, while later linear updates can refine the learned representations.
4.4.2 Switching without optimizer re-initialization
Across datasets in Tab. 6, the same trend holds: 2-002--L and O-L remains competitive or best despite preserving optimizer/scheduler states, while L-0 does not reliably improve on L-LL Hence, the gains are attributable to the update geometry rather than optimizer resets.
Joint interpretation. The regimes yield consistent conclusions. Orthogonal updates confer benefits early (cf. O-0 2 L-)), and O-L is competitive or best even without re-initialization. indicating that the gains stem from the update geometry rather than optimizer resets. In contrast, L-0 shows no systematic improvement over L-LL
4.5 Orthogonal Connection Probability
To investigate the impact of stochastic orthogonal connections, we conducted ablation study ViT-S using TinyImageNe (N=3 runs). In this setup, each residual connection employs an orthogonal update with probability æ, otherwise defaulting to the standard linear connection. We varied from 0.0 (all linear) to 1.0 (all orthogonal).
The effect of varying is visualized in Fig. 5. A clear and statistically positive trend is evident for both Acc@ and Acc@5 accuracy (statistical p-value < 0.05, based on Pearson correlation shown in Fig. 5). This strongly suggests that consistent application of orthogonal updates (i.e., higher T) is beneficial for ViT-S performance on this task.
Apogt
53
val S508
75
= 385
30
5
9
0
0
Probobility Orthagonal Update (m)
Probability Orthogonal Update
Figure 5: Effect of orthogonal update probability VIT-S performance on TinylmageNet (N=3). Error bars represent +1 standard deviation. The Pearson correlation coefficient (R) and p-value between ı and accuracy are displayed in each subplot, indicating positive correlation.
4.6 Orthogonal Layer Pattern Ablation
This ablation study investigates the impact of the placement and number of orthogonal connections within VIT-S model on the Tiny dataset. We compare various patterns of applying the orthogonal update to specific layer indices (or groups thereof) against a baseline model ("None") with linear residual updates and a model where all layers employ orthogonal residual updates ("All (0-5)"). Tab. 7 shows that applying orthogonal connections to greater number of layers tends to yield improved performance.
Table 7: Performance on ViT-S layer patterns for orthogonal updates on Tiny ImagemeeeNet (N=3 runs). Layer indices indicate where orthogonal connections were applied. "None" the linear update, "All" applies orthogonal update to all 6 layers. Metrics are Acc@) I and Aco@5 (%) with mean + std.
Applied Orthogonal Connection Layer Indices
Metrie (%)
None
0,1
2,3
4,5
0,1,2,3
0,1,4,5
2,3,4,5
All (0-5)
Acc
51.88-042
51.45:1016
52.20 74.79_029
51.32-063
52.44_038 51.55:030
52.142015 19.09_009
52.98-045
Acc
25.20
25.0
75.622841
25.05
17.
58

---

535
Sest kp Acc1%)
2.250
Best Vol uo3s
$5.0
Lass
2.225
52.5
52.0
Best
2.175
51.5
51.0
2.125
Figure 6: Effect of stability constant on VIT-S best validation performance. The x-axis represents é on a logarithmic scale. Error bars indicate +1 standard deviation across runs.
4.7 Stability Constant
We examine the sensitivity of our method to the stability constant introduced in Eq. 3, which prevents division by zero during the orthogonal projection. We varied across several orders of magnitude, E E -0-0, 10 1010- 10 100 1,103,, using VIT-S on TinylmageNet over five runs.
Fig. 6 visualizes the best Acc@I and validation loss achieved for each value. We observed that the c= 10 setting yielded results with the lowest standard deviation across the runs indicating greater stability and reproducibility. Prioritizing this stability, we adopted 10 for the default constant throughout this paper, noting that its average performance remains competitive.
Conclusion
We revisited additive residual connections and showed that module outputs can contain sizable com- ponent parallel to the residual stream. We introduced the Orthogonal Residual Update, which discards the parallel part and updates using the orthogonal component fi. Across Vision Transformers and R2 on standard caage-classification benchmarks, we observed improvements in generalization and distinct training dynamics with modest computational overhead. Given the ubiquity of residual connections, we hope this study encourages the community to probe residual-stream geometry at larger scales and in broader modalities.
Limitations Our experiments are limited by compute: models up to ViT-B and datasets up to ImageNet-Ik, with non non-exhaustive hyperparameter sweeps. We did not evaluate substantially deeper ResNetV2 variants on ImageNet-Ik, web-scale datasets, or large language models, and our analysis focused on image classification. deeper theoretical account of when/why orthogonal updates help, and their morizooizon stability, remains open.
Future Work Promising directions include (i) module-aware strategies (attention vs. MLP), (ii) schedules that mix early orthogonal updates with later linear ones, (iii scaling studies to test the role of the width-to-depth ratio 7, and (iv) applications beyond classification (e.g., diffusion models and sequence modeling). We anticipate that community-scale evaluations will be particularly valuable given how widely residual connections are deployed.
Acknowledgement
This work was partly supported by an Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korean Government (MSIT) (No. RS-2020- 11201361, Artificial Intelligence Graduate School Program (Yonsei University), No. RS-2024- 00353131 and No. RS-2021-1121
Artificial Intelligence Graduate School Program (Scoul National University)). the National Research Foundation of Korca(NRF) grant funded by the Korea (AMrmmeen(MSTTT(S--200-000
and KAIT GPU project.
References
[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016. 1, 25
10

---

[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer CioonEECVV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 11-14 2016, Proceedings, Part IV 14, pages 630-645. Springer, 2016. 1,
2, 6, 19
[3] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodke Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https:/transformee-crraatt putis022 //ffamework/iddee.nnee html.
[4] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020. 1, 2, 6, 25
[6] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint Stt.:3307.09888 2023. 2,
[7] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
[8] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 1xX4422422.19437. 2024.
[9] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL attps://arxiv. org/ 2abp/2310.00e
[10] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 260:09 2023.
[11] Sungjun Han, Juyoung Suk, Suyeong An, Hyungguk Kim, Kyuseok Kim, Wonsuk Yang, Seungtaek Choi, and Jamin Shin. Trillion 7b technical report. arXiv preprint arXiv:2504. 15431, 2025.
[12] LG Research, Soyoung An, Kyunghoon Bac, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Seokhee Hong, Junwon Hwang, Hyojin Jeon, Gerrard Jeongwon Jo, et al. Exaone 3.5: Series of large language models for real-world use cases. arXiv preprint arXiv:2412.04862, 2024.
[13] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng. Shuxin Zheng. Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture In International conference on machine learning, pages 10524-10533. PMLR, 2020.
[14] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 2, 6
[15] Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 23IN, 7(7)33,
2,
[16] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3)22111555 2015. doi: 10.10077/11123--15-000

---

[17] Jingdong Wang, Yajie Xing, Kexin Zhang, and Cha Zhang. Orthogonal and idempotent transformations for learning deep neural networks. arXiv preprint arXiv:1707.05974. 2017. 2
[18] Mathias Lechner, Ramin Hasani, Zahra Babaice, Radu Grosu, Daniela Rus, Thomas A Hen- zinger, and Sepp Hochreiter. Entangled residual mappings. arXiv preprint arXiv:2206.01261, 2022. 2
[19] A Saxe. J McClelland, and S Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Proceedings of the International Conference on Learning Represenatations 2014. International Conference on Learning Represenatations 2014, 2014. 2,
19
[20] Jeffree Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. Advances in neural information processing systems, 30, 2017.
[21] P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds. In Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2009.
2
[22] Jun Li, Fuxin Li, and Sinisa Todorovic. Efficient riemannian optimization on the stiefel manifold via the cayley transform. In International Conference Learning Representations, 2020. URL https:// /pwp:////epeerei.e..mmmom wwwpeeiwwwweeeeeeeeeeeeeeeeeeeee htnt/fooummiddHHan ANKDH.
[23] Lei Huang, Li Liu, Fan Zhu, Diwen Wan, Zehuan Yuan, Bo Li, and Ling Shao. Controllable orthogonalization in training dnns. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 6429 6438, 2020.
[24] Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pages 854-863, PMLR, 06-11 Aug 2017. URL https: http:////ooceees.....o mlr. press/ arrees/v0//ae cisee//s htmml. 2
[25] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36:79320-79362, 2023. 2
[26] Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL
attps://kelleejjoddan. gi thub. tttpppsssmooo/////p/o//p
[27] Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint
arXIv:1609.04747,2000 2016. 6
[28] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Conference on Learning Representations, 2019, URL https: /ottps://openrevv/ http:/iwwwwwwee.rreeeeeeeeee net/forum? id-Bkg6RiCqy7. 6
[29] Simon Korbllihh, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519-3529. PMLR, 2019. 7,
[30] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018. 16
[31] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 249-256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL btttps://proceee
mlr press/v9/ pttpaer//////looo1100... gloott10a....m html. 19
12

---

[32] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 1026-1034. IEEE Computer Society, 2015. doi: N0.109//CCCV2205.1223 URL. https: //doi.org/ 10110000000 1109/ CICP//CCC..... 2015. 123. 19
[33] Maciej Skorski. Alessandro Temperoni, and Martin Theobald. Revisiting weight initialization of deep neural networks. Vineeth N. Balasubramanian and Ivor Tsang, editors, Proceedings of The 13th Asian Conference Machine Learning, volume 157 of Proceedings of Machine Learning Research, pages 1192-1207. PMLR, 17-19 Nov 2021. URL
mlr. wwt.press/vile vv1p// akorski22a. html. 19
[34] Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the impact of the activation function on deep neural networks training. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2672 2680. PMLR, 2019. URL http:tttp///ptttt http::/proceeeingg..... .mlr. prwss/essvvv....... v97 h97/hayou19.... html.
[35] Pieter-Jan Hoedt and Günter Klambauer. Principled weight initialisation for input-convex neural networks. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http:tt/ppperrsse nips fileeeerfeeeeeeeeeeessss//eeeeee www.fffilllppppaee paper/ 2023 00233/ash//90
html. 19
[36] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. Advances in Neural Information Processing Systems, 2016. URL httpps//pppeneerrerreeeettteeee BJLa_ZC9. 21
[37] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 18)):9 2019. 25
[38] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020. 25
[39] William Peebles and Saining Xie. Scalable diffusion models with transformers. Proceedings of the IEFF/CVF international conference computer vision, pages 4195-4205, 2023. 25
13

---

A Hyperparameters and Hardware
This section details the training details and data augmentation settings used for the Vision Transformer (ViT-S/-B) and ResNetV2 experiments presented in the main paper. Specific configurations for each model across the different datasets are provided in Tab. 8 (ViT-S/-B) and Tab. (ResNetV2).
A.1 Architecture Specific Hyperparameters
Table 8: Full training hyper-parameters of ViT-S/-B and data augmentations (mean/std. results in Tab. are based on five independent runs with these settings).
Hyperparameter CIFAR-10
CIFAR-100 Imag-lmageNet
ImageNet-Ik
Epochs
300
300
300
300
Batch size
1024
1024
1024
4096
Base LR
1x10
1x10
5x100
5x10
Min LR
0
5x10-a
5x10-5
Optimizer
AdamW, (8),8)) = (0.9, 0.999), weight-decay 1 x 10
LR scheduler
Cosine, 10-epoch linear warm-up
Resolution (px)
32
32
64
224
scale (0.8, 1.0)
scale (0.08, 1.0)
Random crop
pad 4
pad 4
ratio (0.75, 1.33)
ratio (0.75, 1.33)
Patch size
4x4
4x4
8x8
16x16 16
Color jitter
brighmness/comtrrss/stttannn
0.4, hue 0.1
Horizontal flip
0=5
MixUp a prob
a= 0.8, prob = 1.0
CutMix
1.0
Label smoothing
0.1
Random erase
p 0.25, scale (0.02, 0.33), ratio (0.3, 3.3) (9,5)
RandAug (N, M)
5))
(9, 5)
(9,9)
Normalization
datasct-spocific mean/std
Table 9: Full training hyper-panne parameters and data augmentations for ResNetV2 models (e.g.. ResNetV2-18, 18, -34, -50,-101 -101) CIFAR and ImageNet datasets. Mean/std. results in Tab. 2 are based on five independent runs with these settings.
Hyperparameter CIFAR-10 CIFAR-100
Tiny-ImageNet
Epochs Batch size
200
128
Base LR
x 10-1
Optimizer
SGD, momentum 0.9
Weight decay
10
LR scheduler
MultiStep, decay by 0.2 at epochs 80, 120
Warm-up epochs
0 (Not used)
Resolution (px)
32
32
64
Pad 4px,
Random 64 crop, scale (0.08-1.0), ratio (0.75-1.33)
Random crop
random 32x32 crop
Horizontal flip
p= 0.5
Color jitter
(Not used)
MixUp a prob
(Not used)
CutMix a
(Not used)
Label smoothing
(Not used)
Random erase
(Not used)
RandAug (N, M)
(Not used)
Gradient Clipping
(Not used)
Normalization
Dataset-specific mean/std
14

---

A.2
Computational Resources and Training Times
The experiments were conducted using variety of NVIDIA GPUs. The following provides an overview of the typical hardware configurations and approximate training times for key models and datasets. Actual training times could vary based on the specific GPU models available and system load during experimentation.
ImageNet-Ik Training (Vision Transformers):
ViT-B: Trained on system with 8x NVIDIA 440S GPUs for approximately 42 hours. ViT-S: Trained on system with 8x NVIDIA L40S GPUs for approximately 15 hours.
CIFAR-10/100 and Tiny ImageNet Training (Vision Transformers):
ViT-B:
* CIFAR-10/100: Typically trained on 2x NVIDIA H100 GPUs. Approximate training time was hours per run.
Tiny ImageNet: Typically trained on 2x NVIDIA H100 GPUs. Approximate training time was roughly double that of the CIFAR experiments, around hours per run.
VIT-S:
* CIFAR-10/100: Typically trained on 8x NVIDIA RTX 3090 GPUs for approxi- mately hour per run.
Tiny ImageNet: Typically trained on 8x NVIDIA RTX 3090 GPUs for approxi- mately hours per run.
CIFAR-10/100 and Tiny ImageNet Training (ResNetV2 Models):
All ResNetV2 models (ResNetV2-18, -34, -00,-00) were trained systems equipped with 4x NVIDIA RTX 3090 GPUs.
As representative example, full training run for ResNerV2-34 on CIFAR-10 took approximately hours.
Training times for other ResNetV2 model variants on Tiny ImageNet, and for all R2 variants the CIFAR-10/100 datasets, scaled accordingly with model depth and dataset size.
All training times are approximate and reflect the end-to-end duration for the specified number of epochs.
15

---

B Formal Derivation and Theoretical Guarantees
B.1 Geometric Interpretation as an Exponential Map Approximation
Our update rule can be rigorously interpreted from the perspective of differential geometry. We consider the learned feature space decomposition ((r(xx) = fiI + fi occurs in the tangent space Tr M at the point 2n on this manifold, which a vector space where Euclidean geometry locally applies.
high-dimensional, curved manifold M. The orthogonal
The geometrically formal tool for moving from point 2n, along a direction specified by a tangent vector Ta M is the exponential map, exps,, (w) Te,, M - .M. Our proposed update rule, En+1 = 2n + fi (Zn). can be understood as a first-order Taylor approximation of this map:
2n+1 expx, (fi(z))
This linear approximation is highly accurate when the tangent vector v = (xxx is small relative to the local curvature of the manifold. Our empirical results provide strong support for this condition. As shown in our paper's internal dynamics analysis (e.g., Figs. 10 and I1), the norm our update vector, (x,,), I consistently remains orders of magnitude smaller than the norm of the feature stream itself, ||=x,]|| ||- This ensures the update constitutes a small, local step on the manifold where the Euclidean approximation is valid. Therefore, our method is not naive application of Euclidean geometry but a computationally efficient mechanism that leverages the local structure of the learned manifold to guide feature exploration.
B.2 Neural ODE Perspective: Norm Dynamics under Orthogonal Flow
can also interpret our recursion as an explicit Euler discretization of continuous-time (Neural ODE [30]) flow. Let ar) evolve according to
= f(oxx) - s(x) s(x) = (x)).
a(t) = (z)(t,,
| +
Then the squared norm satisfies the exact identity
d
dt |=(t)]|
2(x(t())))))))))))(())) f(a -()) 2(x((t,
(1-
|a(()) [=(()]2 +e
f(((())
1x(1)]² + (6)
In the idealized case e = 0 (exact orthogonal projection), Eq. (6) reduces to क [==(1)] 0, i.e., the flow remains on the sphere ||=[(()] const. For 0, the radial component is suppressed by the factor e/(]|=(t)|12 + e), which is small whenever [xx()]]² >
Moreover, the discrete update Fn+1 nn + (zn) yields
|| || 12a 2( fi (zn)) + [lf
so the first-order norm drift term vanishes when e (since (2 fi (2,)) 0), and for it remains attenuated according to Eq. (6). Empirically, we observe stable feature norms and Ilf (x.))l
<< | throughout depth (see Figs. 10 and 11).
B.3
Identity gradient path preservation
Let (I,)) f(o(xx)) Sn with sn [(((z))).))) and update In+1 In + fi (xx). Then 1| te
0rnt1 (1 sn)1 xx (e) TT)T @f(a(rn))
Oxn
Oxn
@f((((z)) dzn
(Sn / + af((zn)
=1
(8)
Oxn
Or,
Since the Jacobian equals the identity plus residual term, the crucial identity path / enabling unhindered gradient flow across depth preserved, the same sense as standard residual networks. The stability constant guarantees differentiabilityy even when ||x,,]]]]] || is small.
16

---

B.4 Analysis of the Derivative Term
(a #n 8a
a(s
Fnf
sal Isotropic scaling: affects all components, incl. purallel to xn.
dzn
Outpul along Fai magnitude from f" propagates f'j and fa effects
In (J 2nn
2 2 f) b2
Un 22
Output along dh via Jacobian J interaction
Rank-1 updato along atri
modulates a.. -aaraallell strength
where b b= 1222211
Deconstructing these terms reveals how information related to the I, parallel component of (o (denoted fi) and its 2, -rtthooggn component (TT, to distinguish from the update (Tn)) can propagate through the gradients:
The first term, s, isotropically scales all components of an input perturbation, including any component parallel to In. Since (Ir,, J)/b, its magnitude is directly influenced by the alignment of f with In.
The second term, Znf always maps an input perturbation output in the direction of zn.
If we decompose = fil + STTe this term becomes sn =n(/)+/2"* (sn.f)/(Imn] Fn (fT") The first part explicitly carries the fi influence via rank-1 update in the xn direction. The second part shows how even the true orthogonal component f contributes to gradient term that effectively "leaks" into the 2n direction.
b
The third term, #n( [JFa) similarly produces outputs only along In, with its magnitude depending on complex interactions involving the module's Jacobian Jy. This term can mix influences from both parallel and orthogonal components of f as captured by Jy-
The fourth term, 2(xx.f) 2,, is a rank-1 update that directly adjusts components in the ba
Tn direction, scaled by the alignment (I,, f). It often acts as form of negative feedback or dampening for the -paralll contributions.
Thus, derivative (z) @f(e(z.)) a(s.an) de., does not imply that the x,, -aaaaalllll aspects of (x((),) are ignored during backpropagation. Instead, their influence intricately incorporated into the learning dynamies. The presence of in also means that 8, (and consequently fi (x,,) itself, as shown in Eq. (10)) does not perfectly nullify the parallel component, allowing for controlled modulation of information along the x, stream, which be beneficial for norm stability (further discussed in Sec. B.5).
dz..
den
(In: fu (xxw) (5 (((())
(10)
| 1|2 +
B.5 Proof of the Non- Vanishing of the Parallel Component
Suppose that & 0 but, not exactly zero.
(snT.) 51 (sn (In,
an)
2(xn, 8) |1r12)
(SnIn)
(11)
Oxn
b
62
(Knn (fnd) xnn)
28, +
Sntn
20
(12)
(J (Jyzn; Fmr)
In
(13)
0
(i.e., f(a(xn)) - x.,)
The expression is zero if
(J
[J = 0
(Jacobian action is orthogonal) (zero input)
Fn
17

---

1. Condition 1:
Experimentally this almost never occurs. To have Sn 0 requires
(f(o(zn)), xn) - 0,
i.e. the output vector of f at (Ix) must be exactly orthogonal to the basis direction In practice-diu due to continuous nature of f, model noise, and Moating-point effects-there virtually always some nonzero component along any chosen direction, so the inner product is never exactly zero.
2. Condition 2:
This orthogonality condition holds precisely when the Jacobian J, of the mapping f is orthogonal to the basis direction 2n. In practice, whenever perturbations along 2n do not influence the differential behaviour f, the gradient and In are orthogonal, and the condition is met.
3. Condition 3
This the trivial "zero input" case, which simply never shows up in real data. Requiring the entire input vector to be identically zero before the nonlinearity a is degenerate scenario-ane one that your network or dataset will practically never produce-so you won't 't observe this condition in experiments.
18

---

C Ablation Studies
C.1 Learning Curves of ResNetV2 and Ours-G Differing the Initialization Method
We evaluated the robustness of our Orthogonal Residual Update to various initialization methods. Specifically, we implemented three initialization approaches: Xavier [31], Kaiming He [32], and Or- thogonal [19] initialization for the network weights. For all experiments, the biases of the convolution layers and batch normalization layers were initialized zero, while the batch normalization weights were set to one. Previous research has established that initialization methods significantly impact model performance and convergence properties [33-35]. Therefore, we conducted comparative analysis of the variation of the top-5 validation accuracy (reported in Fig. 7) and final accuracy values (reported Tab. 10) between the original identity connection and our proposed approach. Our experimental configuration followed the hyperparameter settings from ResNetV2 [2], with training conducted on the CIFAR- 100 dataset for 60,000 steps.
Table 10: Mean # std oftop-1 (Aco@) I) and top-5 (Acc@5) accuracy from 5 independent runs. runs.We used ResNetV2-50-- for OURS.
Initialization
Connection
Acc@ 1(%)
Acc@5
ResNetV2
77.40=
93.00
Orthogonal[19) 19]
Ours-G
77.22+ 0.24
93.28 0.15
ResNetV2
77.43 0.37
93.07 0.26
Xavier[31]
Ours-G
77.09
93.07 0.09
ResNetV2
77.78
92.82 0.20
Kaiming He[32]
Ours-G
77.54 0.20
93.37 0.12
Table 11: Hyperparameter settings of the experiment varying the initialization method. We used identical hyperparameter setting across all experiments.
Hyperparameter
Values
Epochs
200
Batch size
128
Base LR
10
Optimizer
SGD, momentum 0.9
Weight decay
x 10
LR scheduler
MultiStep, decay by 0.1 at epochs 80, 120
Warm-up up epochs
0 (Not used)
Resolution (px)
32
Pad 4px,
Random crop
random 32 crop
Horizontal flip
p p= 0.5
Color jitter
(Not used)
MixUp a / prob
(Not used)
CutMix
(Not used)
Label smoothing
(Not used)
Random erase
(Not used)
RandAug (N, M)
(Not used)
Gradient Clipping
1.0
Normalization
Dataset-specific mean/std
19

---

0.8
Acouracy
12
0.7
0.0
0.5
Orthegonal updabe updaro
Oetheggnnll
Lnear restbual update
15000
20000
50203
62000
20000
30003
40002
(a) Kaiming Top 5 Validation Accuracy
(b) Xavier Top Validation Accuracy
8.9
5.0:
Acouray
0.7
Orthogomal residur updaite
Unner reciuual update
30000
20000
30003
40000
50030
50500
(c) Orthogonal Top Validation Accuracy
Figure 7: Mean value of validation accuracy over training steps from 5 independent runs for different initialization methods: (a) Kaiming Top-5, (b) Xavier Top-5, (c) Orthogonal Top-5. The shaded regions show the intervals one standard deviation above and below the mean.
C.2 Norm Exploding in R2 with Orthogonal Updates
We observed a notable phenomenon ResNetV2 architectures during later training stages: the L2 norm of the output of the final convolutional layer module, f*x,), tended to increase dramatically, particularly with linear residual updates. Interestingly, this explosion in [I was not always accompanied by similarly large input stream various ResNetV2 configurations, we focus here on ResNerV2-34 trained on Tiny ImageNet, using the same experimental settings detailed in Tab. 9.
| 11. While this behavior noted across
Fig. illustrates this trend, comparing the evolution of the module output norm (zx() its component parallel to the input stream (1,f((=,))13)))) and its component orthogonal to the input stream (Ifx() 3) for both linear and orthogonal residual updates in the final block. For linear updates, significant increase, especially in [I, and often its parallel component (x,)) is evident at later steps.
Connection Type
Connection Type
12 NNrm Squared Volue
5000
Orthogonal-F
Orthagonal-ci
5000
Orthogonal
4202
4000
3002
2202
2000
1000
20000
40000
50003 Training Steps
90000 200000 320000 260000 100000
20000 40000 50003
90000 200000 120000 243020 180000
Training Steps
(a) Orthogonal component
([
(b) Parallel component norm (I I1?)
Figure 8: Component norms of the final convolutional module's output in ResNetV2-34 on Tiny ImageNet without a final LayerNorm. Both plots compare Linear, Orthogonal-F (feature-wise), and Orthogonal-G (global) updates. (X-axis: Training Steps, Y-axis: L.2 Norm Squared). The explosion or significant growth of component norms visible, particularly for the parallel component under certain updates.
20

---

This phenomenon is suspected to be related to the accumulation of the parallel component fii (zx) over layers, as discussed in Sec. 3.4, which can influence the overall module output (zn)) when the module itself explicitly regularized for orthogonality. To mitigate this, we implemented simple solution: adding LayerNorm [36] (LN) layer immediately before the final classifier head, applied to the output of the last residual stream (i.e., after the final global average pooling in ResNetV2). This LayerNorm effectively normalizes the activations passed to the classifier.
As shown in Fig. 9, the introduction of this final LayerNorn significantly stabilized the norms of (Ix,) and its components for the final convolutional module, especially for the linear residual updates, preventing the previously observed explosion.
1750
1750
Connection Type
Connection Type
Uneer
Norm Squered Velue
1300
Orthogonai
Orthogonai-G
Orthogona--G
Norm Squared
1200
1000
/50
750
500
502
0
20000
40000 60000
80000 100000 Training Steps
180000 103000
160000
20000
60000
62000
300000 120000 163050 160000 Traning Steps
80000
(a) Orthogonal component norm (lIfal
(b) Parallel component norm (llf 11?).
Figure 9: Component norms of the final convolutional module's output in ResNetV2-34 on Tiny ImageNet with final LayerNorm applied before the classifier head. (X-axis: Training Steps, Y-axis: 1,2 Norm Squared). The norm explosion is mitigated, and component norms remain more stable throughout training.
With this LayerNorm fix applied to ResNetV2-34 Tiny ImageNet, the model with linear residual updates achieved top-1 accuracy Updates (Feature-wise) achieved 65, 5..8%%%%%33% Tab. 12 summarizes these indicative results. While the norm explosion was primarily an issue for linear updates, applying LayerNorm a common practice and ensures fair comparison.
1..22% while model with our Orthogonal Residual
Table 12: Indicative top-1 accuracy (%) of ResNetV2-34 on TinylmageNet with and without the final LayerNorm (LN) before the classifier. Values are mean + std. from 5 runs. denote metrics from Tab.
Configuration
Connection
Top-1 Acc. (%)
Linear
64.61
W/O final LN;
Orthogonal-F -
65,46
Orthogonal-
65.38+0.35
Linear
65.88.10.22
W/ final LN
Orthogonal-F
65.83-10.38
Orthogonal-6
65.66:20.15
This investigation suggests that while orthogonal updates inherently promote more stable norm dynamics within the residual blocks (as discussed in Sec. 3.4), careful consideration of normalization the network's s extremity, partieularly before the classifier, can be beneficial for all types of residual connections in deep CNNs to prevent potential instabilities arising from the final layers.
21

---

MIP Blcek 0
MEP Slor
P Block
MLP Slock D
MLP Block 1
15-
NLP Block 2
300
[
t0
500
100
200
Trenshen
36k
16k
26k
66
1000
1200
1500
550
1000
40D
10-
16k
24t
26
10s
24h
De.
304 20x
91
18k
24h
26
1ik
Taining Steps
Taining Steps
Taaniing
Trining
Training
(a) Stream norm, MI. blocks.
(b) Parallel component norm, MLP blocks.
Actention Block
Attention Black
Altention Block
Attention Black
Albercion Bleck
Altention Block
300
502-
150-
(x,)))
a2D-
500
100
200-
1000
Actention Slock
1200
Attention Diock
3500
Attention Block
Attention Black
Albention Bleck
Attention Block
1000
If [mm)
5.
500
25
302-
16K
24x
24
10x
24k
64
16k
20c
21
164
24x
16k
20x
8r
10k 241
Twintig Staps
Thiiiig Steps
Taininn Steps
Twining Steps
Thaiting Steps
Twiriing Strpp
Orthogenal basidua Updets
Linear dua Upcare
Ontrogonal Fanicaal updion
(c) Stream norm, Attention blocks.
(d) Parallel component norm, Attention blocks.
Each subfigure shows blocks 0-5 (MLP top, Attention bottom). Ours denotes orthogonal updates; Linear denotes the standard residual. (a,c) After the Transition Point, orthogonal updates stabilize the stream norm || whereas linear updates typically exhibit a post-transition decrease. (b,d) The parallel component energy (xxx)) follows distinct layer-wise profiles for linear orthogonal updates.
Figure 10: Internal dynamies (ViT-S, TinylmageNet, 5 seeds).
D Additional Details on Internal Norm Dynamies
This appendix consolidates the block internal dynamics of ViT-S (Tiny ImageNet, 5 seeds). We present two complementary views: (i) stream and alignment signals in Fig. 10 (stream norm ||=,,,1? and cosine cos(In, J' (ar(()) and (ii) component energies in Fig. 11, where we plot the squared 1.2 norm of the orthogonal component [I (a (2?))) with the cosine panels shown alongside for context. Placed together, these figures provide consolidated, self-contained view of the dynamics discussed in Sec. 3.4.
Across layers, the orthogonal update (blue) maintains a substantial orthogo- nal-component energy [I (a (xx)) throughout training (Fig. 11b, 1ld), whereas the linear residual (red) exhibits a progressive reduction, most prominently in deeper blocks. This trend aligns with the post-Transition-Poinn stream norm and a divergence in cosine, while the orthogonal pathway stabilizes the stream norm and avoids collapse of I| Early-layer nuances exist (e.g.. MLP block O under linear may hold [ 2| longer before declining), but the overall pattern is consistent: orthogonal updates preserve novel (orthogonal) directions, whereas linear updates tend to diminish them over training.
General observations.
behavior in Fig. 10: the linear pathway shows peak-and- decay of the
Parallel component dynamics. While explicit panels for || (axx))) are omitted, its evolution can be inferred from the joint trends of (lf (Fig. 11) and cosine (Fig. 10). Under orthogonal updates, [[ remains substantial and the cosine increases in later training, which is consistent with the full module output gradually accumulating a parallel part even though the update step f1 is strictly orthogonal. This follows from the decomposition ((r(x)) = 8nIn + f- (a (o(xx)) ) gradients through the S, term allow the parallel component to grow, whereas the update path discards it. By contrast, the linear pathway shows a sustained reduction of f| together with peak-then-decay pattern in ||r,,]| (Fig. 10), indicating a shift of capacity toward stream-aligned directions. A Jacobian-based derivation is provided in Appendix B.4.
The Transition Point and cosine dynamics. The Transition Point Fig. 10 (the inflection of ||r,,,1? with the onset of divergence in cos( ((x))))) coincides with a redistribution of component
22

---

NLP Block
MLP Block
HLP Block
NLP Block
MIL Block
RLP Block
0.2
1f (o(x))) [f
120
0.0
0.0
D44
9:25
200
300-
20
50-
-0.4
-244
HLP Block
0.30
0.25
2.53
8 B
75
320
0.00
2
16x
20%
Bc
18h 24k
Bc 108 201
1t
24t
8C:
30k
2ik
L0c 24c
Training Staps
Training Staps
Fairing Stops
Traning Staps
Wairinn Stops
Training Staps
(a) Cosine similarity (MLP blocks).
(b) Orthogonal component norm (MLP blocks).
cattu (etta.))
6id
LElotrail IP IIf (otsw)))
50
4.5
5.00
20
28x Altention Dlock
10x
20x
8k
164
24k
16k
Atbertion Block
Altention Block
Actention Block
Attention Block
Attertion Block
2.50
2.50
R.25
6:25
30-
5.00
66 L6t
16c
24t
66 16c
10x
20c
1.64
01.
16k
heiring Steps
Traming Slapps
The Tiining Steps
Briin Ri Staps
Thaining Stacs
Traiiing Stepss
Orteponal fas dur Upeate
Uneer Fecis Lociin
.... Inaal Ornegoratty (1
Cothagons hesidial Lpdrrs
Liber Fas dua Uppare
(c) Cosine similarity (Attention blocks).
(d) Orthogonal component norm (Attention blocks).
Figure 11: Cosine similarity and orthogonal norm across layers (VIT-S, Tiny ImageNet, seeds). Each subfigure aggregates blocks 0-5 for MLP (top) and Attention (bottom). Ours (orthogonal updates) vs. Linear (standard residual). Orthogonal updates preserve the orthogonal component energy. while cosine trajectories diverge around the Transition Point. For stream norms and broader view of alignment, see Fig. 10.
energies in Fig. 11. After this point, the linear pathway typically exhibits peak deeay of (.
| together with a sustained reduction of Il (o ))12 most clearly in deeper MLP/Attention blocks, indicating drift toward creeamalligeed updates and diminished novelty. In contrast, the orthogonal pathway stabilizes ||=,I|l while preserving [| across depth; the subsequent rise in cos(En, (xx))) is explained by the full module output gradually accumulating parallel portion Sn even though the update step remains strictly orthogonal. Empirically, earlier blocks (e.g., 0-1) often maintain or mildly increase both || l| and alignment, producing gentle cosine uptick, whereas later blocks (e.g., 3-5) show a dominant parallel share with non-negligible ||/ 112. sustaining directional diversity without collapse. This reconciles the seemingly counterintuitive growth of cosine under orthogonal updates with the preserved orthogonal energy, consistent with the mechanism in Sec. 3.4.
Implications for layer-wise application of orthogonality. The depth-dependent trends of (axx))) and llf (a.))) suggest that the effect of orthogonality is not uniform across layers. If one considers targeted application (e.g., early-only or late-only see Sec. 4.6), these compo- nent dynamics provide guideline: early blocks often sustain larger Il.f 2| and can seed diversified features that propagate forward, whereas later blocks may benefit from preserving non-negligible orthogonal energy while alignment grows. That said, our main configuration applies orthogonality throughout the network (Tab. 2), and the observed persistence of f| across many layers, together with the ablation results in Tab. 7, indicates that full-depth application a robust default; selective patterns can recover substantial portion of the gains but generally do not surpass the all-layers setting.
23

---

E Extended Comparisons: Unified Residual Family with Fixed Start Conditions
We consider the unified residual family
-n-1 = + Pe(sin de (1(( + cos cos (zw)),
20 0e 2.
(14)
and track. for each block the phase-plane coordinates (pr sin 0e, Pt cos 0e), which correspond to the parallel and orthogonal shares, respectively. Two canonical start conditions are used:
(pe, 0²) - (v2, 4) (Linear start; gives psin c = 1);
(pe, 0z) (1,0)
(Orthogonal start; gives psin A 0, cos 1).
We then train under Eq. (14) and visualize trajectories over steps.
Attercian Block
Attantion Black
Attention Black
NLP Block
MLP Block 1
MLP Block 2
1.5
1.5
0.5
0.5
-0.5
sin
sin
p
-0.5
sin MUP Block
Attoncion Block 3
Attontion Block 4
Attontion Block 5
2.0
HLP Block
Block
0.5
-0.5
0.0
-1.5
0.0
-1.
0.0
15
-15
0.0
1.5
sin 8
s- 8
p p.snn
Sin
9700
19400
29100
5700
19400
29100
Linear Residual Update
+ Orthagonal Residual Update
Linear Residual Update
Orthogonal Residual Update
(a) Orthogonal start-
Attention blocks.
(b) Orthogonal start-
MLP blocks.
Block
HLP Brock
MIF
1.5
2.0
0.0
vin
n- vin
-0.5
Actention Block
Attention
1.5
1.0
0.0
-95
-1.5
5.0
+3.3
0.0
-1.
0.9
1.5
p. S-.inn8
p--en
0--8 9
9703
19400
29100
9700
19400
29100
step
step
Linear Residual Update
Orthogonal Residual Updete
Linear Residual Update
Orthogonal Residual Update
(c) Linear start
Attention blocks.
(d) Linear start
MLP blocks.
Figure 12: Unified-residual phase plane with fixed start conditions. Axes are psin 0 (parallel share, I) and cos 0 (orthogonal share, y); color encodes training steps. Crosshairs indicate the canonical start points: (0, 1) for the orthogonal start and (1, 1) for the linear start. Trajectories reveal how each block balances paraalel/orthogonal contributions over training.
Observations. For clarity, the r-axis encodes the parallel share psin 0 and the y y-axis the or- thogonal share coss 8. Orthogonal-stann start runs begin at (0, 1) on the y-axis and retain sizable orthogonal share across depth; linear-start runs begin (1, and typically drift toward lower y (reduced orthogonal share), especially in deeper blocks. Attention and MLP trace different paths, but both show the same trend: when updates remove the parallel component, the orthogonal share does not collapse, consistent with the
analysis in Sec. 3.4.
24

---

T/ Architectural Ratio of Width to Depth
Definition n. For Vision Transformers and other Transformer- architectures (e.g., GPT- 2 [37], LLaMA [6], T5 [38], DiT (39]), we define
dmodel
Transformer
Libocks
For ResNets, we define
Davg
E Exestuges Bu Ck
ResNet
Davg
B
Biotl
where Bg is the number of residual blocks in stage k, CE is that stage's output-channel dimensional- ity (i.e., the dimension of the feature map to which the residual ((xa) added), and Biotal Σx Bk- Equivalently,
Ekestages Bg Ck
ResNet
B
lotal
On the ratio Before analyzing inter-family differences, we first consider the ratio 7, which heuristically characterizes representational width per sequential processing block; formal definition is provided immediately below. For models of comparable size, ViT-S (22.0M) yields = 384/6 / s 64, whereas ResNetV2-34 (21 (M.8M) yields 7 3776 162 z 14.8. The lower y of ResNetV2. 34 suggests more compact per- per-block representational space, potentially leaving less directional redundancy for standard updates to exploit and partly explaining the more modest gains observed.
Comparative 7 Values. Tab. 13 lists the architectural parameters and calculated y values for models discussed in this paper and several external reference models.
Table 13: Comparison of values across various model architectures. dmodet denotes hidden dimension; L denotes number of layers/blocks. For ResNets, B& blocks per stage, Cgat is output channels per stage, Biotat is total residual blocks, and Davg B&Cou Brotat- The for ResNets is Davg/ Byotai.
Family Model
dmode! or Davg
L or Brotel
Notes
Models from this paper's experiments
ResNetV2
ResNet-18 18
240.00 (Daeg)
8
30.00
Basic blocks
ResNet-34
236.00 (D (avg)
16
14.75
Basic blocks
ResNet-50
944.00 Dang)
16
59.00
Bottleneck blocks
ResNet-101
985.21 (D (Daeg)
33
29.85
Bottleneck blocks
VIT
VIT-S (ours)
384 (dmodet)
64.00
Used in paper
ViT-B (ours)
768 (dmodel)
12
64.00
Used in paper
External Reference Models
ResNet
ResNet-1001
37.33 (Dang)
168
0.22
[1]
VIT
VIT-S (std.)
384 (dmodet)
12
32.00
[5]
VIT-L
1024 (dmodei)
24
42.67
ViT-H
1280 (dmodet)
32
40.00
Language Models
GPT-2 XL.
1600 (dmadei)
48
33.33
1.5B params [37]
LLaMA-7B
4096 (dmodet)
32
128.00
[6]
T5-XL
2048 (dmodei)
24
85,33
Encoder/Decoder [38]
Diffusion DIT-XL/2
1152 (dmodet)
28
41.14
[39]
Discussion. The 7 values presented in Tab. 13 span wide range. reflecting diverse architectural philosophies. As noted in the main paper, the ViT models used in our experiments (ViT-S 7 = 64, ViT-B = 64) have substantially higher values compared to ResNerV2-34 (7 14.75) and are comparable or higher than ResNerV2-18 (r 30.00) and ResNetV2-101 (2 s 29.85). Interestingly, ResNetV2-50 (? 59.00) also exhibits a high y value, approaching that of our VIT
25

---

models. This is primarily due to its bottleneck architecture: our definition for ResNets calculates D Dang based on the expanded output channel dimensions of the blocks (i.e., the dimensionality of the stream where the residual [xx,, is added) and Brotal as the count of these residual blocks.
Since bottleneck blocks in ResNetV2-50 feature significantly wider output dimensions (resulting in Dang 944) than the basic blocks in models like ResNetV2-34 (D 236) for the same number of residual blocks (Brotat = 16), its calculated y is consequently higher.
It is important to acknowledge a nuance in this definition when applied to bottleneck ResNets versus Transformers or basic-blos ResNets. While our y calculation consistently uses the dimensionality of the stream where the residual sum occurs, single bottleneck block internally performs multiple distinct convolutional operations with varying channel dimensions (e.g., a 1x1 channel reduction, a 3x3 convolution in the narrower space, and 1xl channel expansion). If one were to define an " "effective 7 that accounts for this internal sequential processing or averages across these varying internal channel widths (rather than primarily considering the wide output where the addition happens), the y value for bottleneck architectures like ResNetV2-50 could be interpreted as being substantially lower. Thus, while our current 7 definition highlights the width of the feature space available for the residual addition, it may not fully capture the operational "compactness" imposed by the narrower processing stages within each bottleneck block, especially when comparing against Transformer layers that typically maintain a more uniform processing width throughout.
Despite this definitional consideration, the consistently calculated high for ResNet indicates a large dimensional space for its residual updates. The precise interplay between this y metric, specific architectural inductive biases (e.g., local receptive fields in CNNs vs. global attention in Transformers), and the observed efficacy the Orthogonal Residual Update warrants further nuanced investigation. The diverse y values seen in other reference models such the deep and narrow ResNet-1001 (CIFAR config) versus the very high 7 values for large language models like LLaMA-7B and T5-XL-furthe emphasize that 7 is one of several factors influencing how different residual update mechanisms might perform across various architectural paradigms.
such as the extremely low y for
26

---

G PyTorch Implementation
This section provides example Py implementations for the channel-wise and global orthogonal- ization functions used compute the orthogonal component (rx) from module output f (xn) and an input stream Zn,- These functions encapsulate the core logic of Eq. 3 from the main paper.
def caatoogona__channeeeee (x: torch. Tensor, f_x: torch. Tensor, dim: int, eps:
torch. Tensor) -> torch, Tensor: и*.
Orthogonal residual connection (channel-wise).
x residual stream tensor
f_x module output tensor (e.g., from Attention, MLP, or Conv if channel-vise) dim dimension along which to compute orthogonality (e.g., channel dimension) eps small epsilon tensor for numerical stability ***
# Ensure eps is on the same device as if it's tensor eps eps. to device)
dot_product (x f_x). (dim, keepdim=True) nora_x_squared (x x) sum (dim, keepdin<True) :floa(() () + eps
Ensure scale is cast back to original dtype if float16/bfloat16 scale_factor (dot.product norm_x_squared) to (dtypee=x.dyyp))
projection_ cojjection_ontoxx
onto_x x = scale_factor *
f_orthogonal f_x orojection_onto_x
return f_orthogonal
Listing PyTorch function for channel-wise orthogonalization.
def .orthogonal_glolee laaplllglllllll (x: torch. Tensor, f_x: torch. Tensor, dim: int, eps: torch. Tensor) -> torch, Tensor:
Orthogonal residual connection (global). x residual stream tensor f_x module output tensor (e.g., from a convolutional block)
dim starting dimension for flattening (all subsequent dims will be flattened)
eps small epsilon tensor for numerical stability
original_shape x.shape
# Convert negative dim to positive for consistent unsqueezing later
positive_dim_ide dim if dim > 0 else len(original_shagg ssappssspee + dim
eps eps. (x.
flattened = x. www.ltteestttrt.diineiie. wwt.:ddstiiiieeediinmmmomm # [B, H, W] -> [B, C*H*W] dim=1 (for NCHW)
f flattened
aaaatensssarrt.diirpsttttte 2D from start_dim
or [B, D] if already
Sum over the flattened dimensions (which is now dim=1 if start_dim was 1, or the last dim)
For clarity, explicitly use dim=-1 for sum over the last (flattened) dimension dot dot_product = flatttteeddd f_x_flattened) sum sum(di==-1, keepdin=True) norm_a_squared (x_flattened x_flattened) sum (dim=-1, keepdin=True). .flaat float eps
scato_factor (dot_product norm_ xes__e_suaaed) (oddtypexx.dtype)
Reshape
allow broadcasting with original shape
27

---

It needs to have trailing dimensions of size 1 to match x's rank post-flattening start_din
num_dims dildddiim__uuateee sou_nsqueeze len (enorgiinal_shae because dot_product keeps one dim for in range (aum_dims________ to_ unsqueeze): scale_factor - scale_factoo unsqueeze
(positive_ idx
projection_ontee x scale_ factor x # Broadcasting happens here __orthogonal f_x projection_onto.x
return f_orthogonal
Listing 2: PyTorch function for global orthogonalization.
The dim argument in orthhgonal,, channel typically refers to the channel dimension (e.g., dim=1 for NCHW tensors) where orthogonality is computed independently for each spatial location token. For www.ohoootllgg....ooo _global, dim specifies the starting dimension from which the tensor is flattened before computing single global projection scale per batch element; for instance, for an NCHW tensor, dim=1 would flatten C, H, and W dimensions together. The choice between these depends on the layer type and desired granularity of orthogonalization discussed in Sec. 3.3 of the main paper.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly claim our method in Abstract, Introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We include the limitation section and explain what the limitation of our paper.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
We formulate our methods based on Linear Algebra.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all information needed to reproduce the main experimental results, including training details and hyperparameters (in the main paper and appendix) and an implementation link.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
30

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide detailed algorithm section, and the core codes which can reproduce all the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
iemiiissssooooooo oeeioonooooloooo oolooliioo icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/ wwcp/bllc///eeeeccceeeceecceelee http//uudee//eeeee CCtees//ooSSubmmeecc cessonnoolcyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We explain hyperparameter settings and training details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We include standard deviation and error bar in the figures we attached.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
31

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we include detailed information on experiments which hardware used for.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpp/://///uip////eeeeee cc//pbblcc/eeie csGuidel de ines?
Answer: [Yes]
Justification: All authors carefully read the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
This work is a general methodological contribution and does not introduce application-specific systems or new datasets. Therefore, we do not foresee direct societal impacts from the work itself.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
32

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper did not contain any potential harmful contents.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use publicly available datasets (CIFAR-10/100,, ImageNet- Ik, and Tiny- ImageNet-200) and properly cite the original sources. CIFAR-10/100 request citation on the official dataset page, and ImageNet (and ImageNet an ImageNet subset) used under ImageNet's 's terms of access for non-commercial research/educational use; we do not redistribute the images.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
33

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce method with clear algorithm, well-documented table, and PyTorch implementation in the main and the Appendix.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: did not conduct any crowdsourcing or research involving human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not include any research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
34

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We did adopt LLM as the component our core method.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp///eee..pp/eee cc/Connferences/20255// /LLM) for what should or should be described.
35


Reviews:
[{"review_id": "Iy3IBDmVUj", "reviewer": "Reviewer_2TRn", "summary": "This paper reconsiders the essential function of Residual Connection and proposes a new approach, Orthogonal Residual Update. This approach removes the components parallel to the residual stream that are assumed to be redundant and are added to the residual stream. This approach has been reported to improve image classification performance in CIFAR, TinyImageNet, and ImageNet-1k.", "questions": "With regard to the three Weaknesses listed above, please respond to the following points.\n\n1 Is the main contribution of this research a practical method proposal? Or is it a fundamental study that contributes to the understanding of shortcut paths? Depending on your position, please clarify the main point of this paper.\n\n2 Regarding the change in cosine similarity seen in Figure 3, it appears as if the regular Shortcut Path behaves as if it removes redundant parallel components by learning. Do the extraction of the vertical components added by the proposed method really provide “richer features”? Please provide any additional discussion in this regard, including any additional analysis of visualization or representation quality.\n\n3 Are the experimental settings for ViT-S and ViT-B in Table 2 a comparison with sufficient training convergence?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1 The motivation and idea of the proposed method are very simple, clear, and easy to understand.\n\n2 The idea is novel and valuable in the sense that it encourages reconsideration of the basic elements of the shortcut path from a new perspective.\n\n3 The proposed method improves the convergence of training, and in some cases, the reduction of training time is actually confirmed.\n\nWeaknesses\n\n1 It is not clear whether the main claim of this study is to “propose a practical method” or to \"deepen the basic understanding of shortcut paths\". If the former is the subject matter, then more careful performance evaluation and examination of various architectures, downstream tasks, and non-image domains, as pointed out in Weakness 3, would be necessary. On the other hand, if the latter is the goal, quantitative and qualitative analysis showing how the visual representation is actually changed is required, as long as the proposal is based on the understanding that richer features are acquired by removing redundant representations. In the current manuscript, the steps taken in both directions are somewhat insufficient and give the impression of being half-hearted.\n\n2 The discussion of the experimental results in Figure 3 alone is not convincing enough to strongly support the validity of the proposed method. For example, in the ordinary Shortcut Path, the cosine similarity tends to become negative as the learning progresses, which suggests that the parallel component of f(\\sigma(x)) is in the opposite direction of x. In other words, there may be a function inherent in the normal Shortcut Path that reduces the parallel components by learning, e.g., to suppress norm divergence. On the other hand, the proposed method tends to converge to large positive values of cosine similarity and to reduce the vertical component. These observations call into question the assertion made in this paper's premise that the parallel component is redundant and that rich features can be obtained by adding only the vertical component.\n\n3 Regarding the performance evaluation shown in Table 2, it is possible that the Top-1 Accuracy is low, especially in the Linear settings of ViT-S and ViT-B, and that the training has not converged sufficiently. For example, according to the original paper of Vision Transformer and [1], the Top-1 Accuracy is generally expected to be in the high 70% range. If there is a possibility of insufficient training, it is difficult to judge the effectiveness of the method based on these results, and I believe that more careful setup and validation is needed.\n\n[1] L. Yuan et al, Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet, 2021.", "was_revised": true, "final_justification": "The additional experiments revealed interesting implication regarding the orthogonal component of the features, providing limited yet positive evidence in support of the paper’s hypothesis. Taking into account the discussions between the other reviewers and the author as well, I have decided to maintain my initial score, which leans toward acceptance.", "rebuttal": "We sincerely thank the reviewer for their valuable feedback and insightful questions. The comments help us clarify the core contributions of our work and the context of our experimental results. We are happy to provide detailed responses and additional analysis below.\n\n---\n\n### On the Paper's Contribution and Positioning\n> \"It is not clear whether the main claim of this study is to “propose a practical method” or to \"deepen the basic understanding of shortcut paths\"... please clarify the main point of this paper.\"\n\n\nWe thank the reviewer for this excellent question, as it allows us to clarify the core positioning of our work. Our primary contribution is a fundamental study that provides a new understanding of residual connections, with our proposed method serving as a practical proof-of-concept that validates our hypothesis.\n\nOur fundamental hypothesis is that the parallel component of the standard residual update ($f_\\parallel$​) can be redundant or even detrimental, and that encouraging novel feature directions via _Orthogonal Residual Update_ leads to more efficient representation learning. A deeper analysis, supported by Appendix Figure 10, reveals that the standard _Linear Residual Update_ consistently learns to suppress the parallel component, with its norm approaching zero during training. This strongly supports our hypothesis that this component may be unnecessary.\n\nIn contrast, the dynamics of our _Orthogonal Residual Update_ are more complex and intriguing than a simple removal of a redundant component. The increasing cosine similarity seen in Figure 3 is not a monolithic phenomenon. Instead, it results from at least three different observed behaviors across layers:\n- Both orthogonal and parallel components of the full module output increase (e.g., MLP block 0).\n- The orthogonal component stabilizes or decreases, while the parallel component is maintained or grows (e.g., Attention blocks 2, 3, 4).\n- Both components decrease, but the orthogonal component does so more slowly (e.g., Attention block 5).\n\nThis complex behavior does not contradict our hypothesis. Instead, it reveals that our method, by acting as a strong inductive bias, does not entirely eliminate the parallel component from the module's full output, but rather re-purposes its role within the optimization dynamics. We believe this non-trivial, counter-intuitive result about how networks adapt to geometric constraints is a key contribution to the fundamental understanding of shortcut paths.\n\nFinally, the practical aspect of our work serves as the direct validation of this hypothesis. Our strictly controlled experiments (Table 2, Figure 2) show that this fundamental change leads to consistent improvements in performance and stability. Crucially, as our new quantitative analysis confirms, this performance gain stems from the model learning demonstrably richer and more diverse feature representations. This provides the essential empirical proof for our primary, more fundamental contribution. We will clarify this positioning in the revised manuscript.\n\n---\n\n### On Feature Richness and Cosine Similarity Dynamics\n> \"The discussion of the experimental results in Figure 3 alone is not convincing enough... Do the extraction of the vertical components added by the proposed method really provide “richer features”?\"\n\nWe agree that our claim of \"richer features\" requires direct, quantitative evidence. As attaching new visualizations is not possible during the rebuttal period, we have conducted a comprehensive supplementary analysis on the feature representations learned by both the Linear and our Orthogonal ViT-B models on the full ImageNet-1K validation set. We extracted the 768-dimensional CLS token features from the final transformer block (before LayerNorm) of our fully trained models, using the 288-epoch checkpoint from the complete 300-epoch training run. This ensures our analysis reflects the final, converged state of the representations.\n\nThe results, summarized below, consistently demonstrate that our Orthogonal Residual update produces richer and more diverse representations.\n\n|Metric | Linear Residual Update | Orthogonal Residual Update |  Change | Interpretation |\n|:-:|:-:|:-:|:-:|:-:|\n|  Effective Rank | 572.9 | 599.9 | +4.7% | ↑ Higher global diversity |\n| Spectral Entropy | 6.512 | 6.539 | +0.41% | ↑ More uniform spectrum |\n| CKA Similarity | - | - | 0.546 | Structurally different representations |\n| Feature Std. Dev. | 0.407 | 0.193 | -52.5% |↑ More stable numerical properties |\n\n$\\text{Effective Rank (Participation Ratio)} = (\\Sigma_{i} p_{i}^2​)^{−1} $ where $p_i$​ are the normalized singular values.\n\n$ \\text{Spectral Entropy} = - \\Sigma p_{i} * log(p_{i}) $\n\n**Key Findings**:\n1. Higher Global Diversity: We measured the Effective Rank of the feature covariance matrix. A higher rank indicates that features span a higher-dimensional space and are less collapsed. Our method achieves a 4.7% higher effective rank, providing strong evidence that it encourages the model to learn a richer global representation.\n2. Fundamentally Different Structure: We used Centered Kernel Alignment (CKA) to measure structural similarity. A score of 1.0 would imply identical representations. The resulting score of 0.546 demonstrates that the features learned by our method are structurally different from those of the standard update, supporting our claim that the orthogonal update is a meaningful inductive bias.\n3. Enhanced Diversity with Superior Stability: Critically, our method achieves greater representational diversity while simultaneously enhancing numerical stability. The feature standard deviation is 52.5% lower, indicating that the features are more tightly clustered and well-behaved. This is not a trade-off, but a synergistic improvement: our orthogonal updates encourage the model to explore a wider range of feature directions without resorting to large-magnitude, potentially unstable feature values that can hinder optimization.\n\n\nThese complementary metrics robustly support our conclusion that the Orthogonal Residual update leads to richer, more effective feature representations. We will include this detailed analysis in the appendix of the revised paper.\n\n---\n\n### On ViT Baseline Performance and Training Convergence\n> \"it is possible that the Top-1 Accuracy is low, especially in the Linear settings of ViT-S and ViT-B, and that the training has not converged sufficiently.\"\n\nWe thank the reviewer for this critical question. The baseline performance is a direct and expected consequence of the specific, challenging training paradigm we chose to ensure a fair comparison.\n- The \"ImageNet from Scratch\" Paradigm: The original ViT paper demonstrated that without massive pre-training (e.g., on JFT-300M), ViTs underperform CNNs on ImageNet-1k. To address this, influential follow-up works like DeiT and T2T-ViT established a new training recipe for this from scratch setting, which requires strong regularization (e.g., MixUp, CutMix) to prevent overfitting. Our experimental setup adheres to this modern, robust recipe.\n- A Fair Comparison at Practical Convergence: Our 300-epoch training schedule ensures a robust comparison at a point of practical convergence. As shown in Figure 2(b), the performance advantage of our method is not a short-term artifact. The gap is established early and is consistently maintained or widened throughout the vast majority of the training process. The rate of improvement for both models slows significantly in the final 100 epochs, indicating that while absolute performance might marginally increase, the stable, long-term relative gap between the methods is not in doubt. This makes our comparison a valid and reliable measure of the orthogonal update's efficacy.\n\nTherefore, we are confident that our results represent a scientifically valid comparison between two methods at a point of performance saturation, and the reported improvements robustly reflect our method's efficacy.\n\n\n[1] Yuan, L., et al. Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet. ICCV 2021.\n\n---\n\n\nThank you again for your thoughtful engagement with our work. This discussion has been invaluable in helping us refine our arguments. We hope that our detailed response clarifies our contribution and resolves any remaining questions."}, {"review_id": "RTNyeJ52aC", "reviewer": "Reviewer_b9E3", "summary": "This paper focuses on the residual update applied in ViTs and ResNet. Instead of directly using the linear residual, the authors propose to only use the orthogonal components of the output layer while discarding the parallel components. The authors also designed two different ways to do the orthogonal residual update: feature-wise and global-wise, where the feature-wise design performs universally better than the global-wise orthogonal residual update. With this simple design, the proposed method shows potential for faster convergence and effective performance improvement when applied to smaller models on small-scale datasets. However, the effectiveness of larger models with larger-scale datasets remains to be further investigated. To analyze the proposed method, the authors have provided extensive ablation studies and theoretical analysis to highlight the effectiveness of the proposed orthogonal residual update.", "questions": "- In Figure 3 (b) and (d), the authors have shown an interesting plot where the cosine similarity between $x_n$ and $f(\\sigma(x_n))$ is increasing after the transition point for the proposed method. Does it imply that the proposed orthogonal residual is trying to learn the linear residual during training? I wondered at different training stages of the proposed method, how does the performance change? Will stopping training at early steps before the increase of the cosine similarity reaching a certain threshold actually help with improving the performance?\n- Why does the stability constant have a relatively large effect on the performance of the proposed method? Can the authors provide more insights?\n- How does the normalization affect the proposed method when applied in ViTs or ResNet? Based on the results shown in the paper, the norm of the proposed residual update is very distinct compared to the linear residuals.", "limitations": "The authors have provided the discussion of the limitation of the proposed method.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- This paper introduces an interesting idea to just apply the orthogonal residuals, which is argued in the paper to be the effective components in the residual update. Aligning with previous research, the idea is reasonable and has achieved relatively good performance at least on small models shown in the paper.\n- The authors have shown extensive experiments to analyze and explain why the orthogonal residuals are effective and dominant in the network training, and how they affect the performance of the model. \n- The overall writing of the paper is clear, and the authors have provided extensive ablation studies to analyze the proposed method.\n\nWeaknesses:\n- The proposed method seemed to be less effective on the larger model with larger-scale datasets or deeper networks. Can the authors provide more insights? Because residual update is designed for deeper and more complex networks, it would be good to show the effectiveness of the proposed method when applied to larger models.\n- Although the proposed method is interesting, and the theoretical analysis could validate the effectiveness of using the orthogonal residual update, the experiments did not provide strong evidence to clearly support the argument that the orthogonal residual update is more effective in the network training.\n- The reported results for ViTs on different datasets are relatively low, and this might be because different implementations of ViTs have been used in the paper. However, will the hyperparameter choices affect a lot in the performance of the proposed method? What is the robustness of applying the proposed method to different ViT variants?\n- Could the authors provide more theoretical or intuitive explanations to Figure 3 (b) and (d) regarding the cosine similarity between $x_n$ and $f(\\sigma(x_n))$? Because the relationship of the learned $f(\\sigma(x_n))$ and $x_n$ is very important in understanding whether the orthogonal component is effective than just using linear residual.\n- The authors have mentioned faster convergence of getting rid of the parallel residuals, but only update with the orthogonal residuals. Could the authors provide more efficiency comparisons in the paper?", "was_revised": true, "final_justification": "The authors have resolved most of my concerns. I encourage the authors to incorporate reviewers' suggestions to add efficiency comparison, clearer norm visualization (Fig. 10), and more intuitive explanation of the cosine similarity figures to the final version to further strengthen the paper.", "rebuttal": "We sincerely thank the reviewer for their detailed feedback and insightful questions. We appreciate the opportunity to clarify these important aspects of our work. To be concise and address all points within the character limit, we will group our responses thematically.\n\n---\n\n### On Effectiveness, Evidence, and Baselines (Weaknesses 1, 2, 3)\n> \"The proposed method seemed to be less effective on the larger model... experiments did not provide strong evidence... reported results for ViTs on different datasets are relatively low...\"\n\nWe thank the reviewer for these critical questions about our method's efficacy. We will address them cohesively.\n\n- **On Scalability and Effectiveness (W1)**: Our results demonstrate clear effectiveness on large-scale models. The +4.3%p absolute improvement for ViT-B on ImageNet-1k is a significant gain on a challenging benchmark, directly showing that our method's benefits scale effectively.\n- **On Evidence and Baselines (W2, W3)**: We agree the baseline performance requires context. Our primary goal was a rigorous 'paired comparison' to isolate the effect of the orthogonal update, not to set a new SOTA record. Our experiments follow the demanding \"ImageNet-1k from scratch\" paradigm, which, as established by works like DeiT, requires strong regularization (e.g., MixUp, CutMix) to prevent overfitting in Transformers. This explains why our baseline differs from models pre-trained on massive private datasets.\n- **On the Strength of Evidence**: The evidence for our method's superiority is robust. As shown in Figure 2b, the performance gap between our method and the linear baseline is established early and\n consistently maintained or widened over 300 epochs. This stable, long-term advantage proves our method's efficacy is a genuine architectural effect, not an artifact of training duration. The consistency of these gains across diverse architectures (ResNetV2, ViT) and datasets provides a strong and cohesive body of evidence for our claims.\n\n### On Model Dynamics and Efficiency (Weaknesses 4, 5 & Questions 1, 2, 3)\n> \"Could the authors provide more theoretical or intuitive explanations to Figure 3... Does it imply that the proposed orthogonal residual is trying to learn the linear residual...?\" (W4, Q1)\n\nThis is an excellent question that probes the core dynamics of our method. The increasing cosine similarity does not imply our method is simply \"learning to be linear.\" Instead, it reveals a fundamentally different and more controlled update mechanism compared to the standard linear update, a distinction best understood by examining Appendix Figure 10.\n\n- Linear Update's Dynamic: \"Explosive\" Reinforcement. In a standard Linear Update, the module can directly reinforce the existing feature stream $x_n$. As observed in Figure 10, this can lead to moments of \"explosive\" updates around the transition point, where the norm of the parallel component $||f_\\parallel||^2$ grows significantly. This suggests the standard method sometimes aggressively scales existing representations.\n- Orthogonal Update's Dynamic: Controlled \"Compensation\" via Leakage. In contrast, our Orthogonal Update structurally prevents this direct, potentially unstable reinforcement. The module learns to produce a large parallel component in its full output, but this is explicitly discarded from the update vector. We hypothesize that the module then relies on the small, controlled $\\epsilon$-leakage (mathematically defined in Eq. 9) to perform the necessary, subtle scaling adjustments to the stream. This leakage acts as a controlled \"compensation\" mechanism, allowing for stable, fine-tuned modulation rather than the direct, sometimes explosive, reinforcement seen in the linear case.\n\nTherefore, the increasing cosine similarity is a sign of a more sophisticated adaptation. The module learns to handle the main information flow (via the parallel component of its full output), while our method ensures the actual update remains a controlled injection of novel information, using leakage for fine-tuning. This is a key finding about how networks adapt to and leverage geometric constraints for more stable learning.\n\n---\n\n> \"Why does the stability constant have a relatively large effect on the performance of the proposed method?\" (Q2)\n\nThe constant ϵ ensures numerical stability but also plays a subtle optimization role. As shown in Appendix B.1 (Eq. 9), a non-zero $\\epsilon$ allows a small, controlled amount of the parallel component's gradient information to \"leak\" through. This can be beneficial as it prevents the optimization from stalling in regions where a purely orthogonal update might have near-zero gradient with respect to certain loss components, thus aiding learning stability. Our ablation in Figure 5 shows performance is robust across several orders of magnitude, with $\\epsilon=10^{-6}$ providing the best stability in our experiments.\n\n---\n\n> \"How does the normalization affect the proposed method... the norm of the proposed residual update is very distinct...\" (Q3)\n\nThe reviewer's observation is correct and points to a key benefit: our method provides intrinsic geometric stability. The distinct norm behavior is a direct result of our geometric constraint. When the update norm is much smaller than the stream norm ($ || f_\\perp (x_n) ||  \\ll  ||x_n || $ ), which is empirically validated in our figures , the update $x_{n+1} = x_n + f_\\perp (x_n) $ acts as an approximate rotation. This dynamic causes the representation to effectively \"walk on a hypersphere,\" preserving the stream's magnitude ( $ || x_{n+1} || ^ 2 \\approx || x_n || ^2 $ ) while changing its direction. This provides an inherent mechanism for norm control, which is particularly beneficial for architectures like Transformer that rely heavily on explicit normalization layers for stable training.\n\n---\n\n> \"The authors have mentioned faster convergence... Could the authors provide more efficiency comparisons in the paper?\" (W5)\n\nThe paper provides two clear efficiency comparisons.\n1. **Computational Cost**: Section 3.4 and Table 1 present a detailed FLOPs analysis, demonstrating that the theoretical overhead of our method is marginal compared to the main module's complexity (e.g., Attention or MLP blocks).\n2. **Practical Efficiency**: More importantly, Figure 2b provides a direct wall-clock time-to-accuracy comparison. It shows that our method not only achieves a higher final accuracy but also reaches any given accuracy target in significantly less time. This demonstrates superior overall training efficiency, where the small computational cost per step is heavily outweighed by faster convergence.\n\n--- \n### Formatting Concerns\n> \"... some figure legends could be made bigger and clearer.\"\n\nThank you for the suggestion. We will ensure all figure legends are improved for clarity and readability in the camera-ready version.\n\n---\n\n\nThank you again for your thoughtful engagement with our work. This discussion has been invaluable in helping us refine our arguments. We hope that our detailed response clarifies our contribution and resolves any remaining questions."}, {"review_id": "hXp1wNgjJy", "reviewer": "Reviewer_SmmT", "summary": "This paper introduces an orthogonal update strategy for residual connections, which only add the orthogonal component to the input stream. The authors provide some experiments of image classification for this strategy and demonstrate that this strategy can gain some improvement.", "questions": "- The proposed orthogonal update rule can be rewritten as x_{n+1}=f(\\sigma(x_n))+(1-s_n)x_n, and this rule is similar with the skip connections of scaling in [1]. What causes the results in this paper to be opposite to those in [1]? Table 1 in [1] shows that this update rule is worse.\n- Why does the proposed method perform worse than the original residual connection under various initialization schemes in Table 7? Which initialization method was chosen in the main paper? \n- How does the proposed orthogonal residual update perform when applied to other commonly used modules and optimizers?\n[1] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Identity mappings in deep residual networks. In ECCV 2016.", "limitations": "- Lack of the issue concerning the original residual connection.\n- The Experiments section only includes image classification tasks, without addressing other problems or networks that employ residual connection structures.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- The issue of residual connections which this paper focus on is important. The proposed orthogonal residual update strategy enhances the performance, especially on ViT models like Table 2 shows. \n\nWeaknesses:\n- The motivation is not clear. Why the scaling operations caused by the parallel component is not necessary? The magnitude is also an important factor of updating. What is the rationale behind performing orthogonal decomposition on the model's output stream in this work? Are there any cited studies that support this viewpoint?\n- The organization of the paper is not well-structured. For example, the Methods section lacks an analysis of the issue concerning the original residual connection.\n- The output space of the model is not necessarily a Euclidean space, which imposes significant limitations on the proposed orthogonal scheme.", "was_revised": true, "final_justification": "After the rebuttal, I tend to increase my score to 4.", "rebuttal": "We sincerely thank the reviewer for their critical and detailed feedback. We believe some of the concerns may stem from a lack of clarity in our initial presentation, particularly regarding our motivation and the distinction from prior work. We are grateful for the opportunity to address these points.\n\n--- \n### On Motivation, Magnitude, and Novelty\n> \"The motivation is not clear. Why the scaling operations caused by the parallel component is not necessary? The magnitude is also an important factor of updating... Are there any cited studies that support this viewpoint?\"\n\nWe thank the reviewer for these insightful questions, which allow us to clarify the core principles of our work.\n- **Motivation and Magnitude**: Regarding the lack of analysis on the original connection, our analysis in Appendix Figure 10, which shows the standard network learns to suppress the parallel component, directly addresses this by revealing a previously un-discussed inefficiency in the original design. Our core hypothesis is that high-capacity modules (e.g., attention, MLPs) are sub-optimally utilized if their capacity is spent on simple scaling operations (the parallel component, $f_\\parallel$). Our method encourages the module to learn novel feature directions instead. We clarify that our method does not discard magnitude information from the stream $x_n$; it discards the part of the update that only scales the existing stream. In fact, Figure 3 (a, c) shows our method leads to a more stable stream norm, suggesting that removing the potentially conflicting parallel update aids in controlling signal magnitude.\n- **Empirical Justification**: Our hypothesis is strongly supported by the learning dynamics of standard networks. As shown in Appendix Figure 10, when using a standard Linear Residual Update, the network itself learns to suppress the parallel component ($||f_\\parallel||$ trends towards zero). Our method simply enforces this empirically discovered optimal behavior as a structural inductive bias from the start.\n- **Novelty**: Our contribution is a novel approach to residual connections. Prior work on orthogonality focused on either\n  - (a) regularizing network weights  [2, 3]  or\n  - (b) modifying the skip-path itself with fixed transformations [4, 5].\n\nOur approach is distinct: we preserve the standard identity skip-path and instead impose a dynamic, input-dependent geometric constraint on the additive update signal. The consistent empirical gains across diverse settings serve as strong validation for this new perspective.\n\n\n--- \n\n### Regarding the concern that the model's output space is not necessarily a Euclidean space\n> \"The output space of the model is not necessarily a Euclidean space.”\n\nWe thank the reviewer for this sharp and important question. We completely agree that the feature space is best understood as a high-dimensional, curved manifold. This perspective is central to our method's motivation, and we are grateful for the opportunity to clarify this connection.\n\nAs the reviewer astutely notes, a key property of smooth manifolds is that they are _locally Euclidean_. Our method leverages precisely this property. The orthogonal decomposition $f(\\sigma(x_n​))=f_\\parallel ​+f_\\perp$​ occurs in the tangent space $T_{x_n}​​M$ at the point $x_n$​ on the manifold, which is a vector space where Euclidean geometry locally applies.\n\nOur update rule, $x_{n+1}​=x_n ​+f_{\\perp}​(x_n​)$, can be rigorously interpreted as a first-order Taylor approximation of the exponential map,$exp_{x_n}$ $(f_\\perp ​(x_{n}))$.\nThe exponential map is the geometrically formal tool for moving along a curved manifold from a point $x_n$​ in a direction specified by a tangent vector $f_\\perp​ (x_n​)$. The approximation is given by:\n$\\text{exp}_{x_n}​​(v) \\approx x_n ​+v$.\n\nThis linear approximation is highly accurate when the tangent vector $v=f_\\perp​(x_n​)$ is small relative to the local curvature of the manifold. Our empirical results provide strong support for this condition. As shown in our paper's Figure 3 and Appendix Figure 10, the norm of our update vector, $||f_\\perp​(x_n​)||$, consistently remains orders of magnitude smaller than the norm of the feature stream itself, $||x_n||$ (e.g., $\\sim 10^1$ vs. $\\sim 10^3$). This ensures the update is a small, local step on the manifold where the Euclidean approximation is valid.\n\nTherefore, our method is not a naive application of Euclidean geometry to a global non-Euclidean space. Rather, it is a computationally efficient mechanism that leverages the local Euclidean nature of the learned manifold to guide feature exploration. This provides a novel inductive bias for navigating the feature space, and the strong empirical gains in our work validate this geometrically-motivated approach. We will add this clarifying discussion to the revised paper.\n\n--- \n### Relation to Prior Work\n> ”The proposed orthogonal update rule can be rewritten as... similar with the skip connections of scaling in [1]... Table 1 in [1] shows that this update rule is worse.\"\n\nThis is an excellent point that highlights a crucial distinction. While the rewritten update rule is structurally reminiscent of scaling in [1], their purpose and effect are opposite, which explains the different outcomes. The distinction is fundamental:\n- **He et al. [1]** : Uses a fixed hyperparameter (e.g., $\\lambda_n$) for signal modulation. As they correctly showed, this can impede the identity gradient flow and harm performance.\n- **Our Method**: Uses a dynamic function , $s_n(x_n, f(x_n)) $, to compute a geometric projection. Its purpose is to enforce orthogonality, not to modulate the signal. Crucially, as shown in Eq. 5, this operation is explicitly designed to preserve the identity gradient path, which aligns with the core principle for stable training established by [1].\n\nTherefore, our method is not a form of the detrimental scaling explored in [1], but a new geometric constraint compatible with the principles of deep residual learning.\n\n--- \n### On Experimental Scope and Paper Organization\n> \"The organization of the paper is not well-structured... How does the proposed orthogonal residual update perform when applied to other commonly used modules and optimizers?\"\n\nWe thank the reviewer for this feedback, and we will improve the paper's structure by adding a more explicit motivation in the Methods section.\nRegarding the experimental scope, our focus was strategic. To establish a **general and foundational principle**, we tested our hypothesis on the most fundamental and diverse building blocks of modern deep learning: the convolutional blocks of ResNetV2 and the Self-Attention/MLP modules of Vision Transformer. By demonstrating consistent gains with standard optimizers (SGD, AdamW) across these canonical architectures, our goal was to provide strong evidence for the universal applicability of the orthogonal update principle. We believe this focused study provides a clearer and stronger foundation for the community than a broader but less controlled survey of specialized modules.\n\n--- \n[1] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Identity mappings in deep residual networks. In ECCV 2016.\n\n[2]  Huang et al., Controllable orthogonalization in training dnns. CVPR 2020.\n\n[3] Cisse et al., Parseval networks: Improving robustness to adversarial examples. PMLR 2017.\n\n[4] Wang et al., Orthogonal and idempotent transformations for learning deep neural networks. arXiv:1707.05974.\n\n[5] Lechner et al., Entangled residual mappings. arXiv:2206.01261.\n\n\n---\n\nThank you again for your thoughtful engagement with our work. This discussion has been invaluable in helping us refine our arguments. We hope that our detailed response clarifies our contribution and resolves any remaining questions."}, {"review_id": "Ofu2iuV5yU", "reviewer": "Reviewer_rEbg", "summary": "The authors revisit a classic component present in most modern network architectures: residual connections. Using the Gram-Schmidt method, one can decompose the output of the residual branch in a ResBlock into the sum of two components: a component parallel to its input (and thus the main branch) and an orthogonal one. The authors show that the parallel component can point in the opposite direction (\"anti-aligned\") wrt. the main branch, which could annihilate the residual signal; they further conjecture that this could consequently hinder efficient information propagation in the network. \n\nUsing this decomposition, the authors create a new network architecture that simply subtracts the parallel component from the residual branch in every residual block, thus only keeping the orthogonal one. They evaluate this variation on residual connections on two different architectures (ResNets and ViTs) and multiple datasets, and perform ablation studies to show their effectiveness.", "questions": "- Can you explain the low baseline performance for the ViT models on ImageNet? The reported 71.09% are vastly under the usual baselines reported by Google or PyTorch which amount to 80-85%. This makes me doubt the significance of the results, especially as the ResNet runs are not reported on ImageNet. Generalization performance is highly dependent on learning rates, which could effectively differ between the methods, as the layer-wise gradient norms differ between your method and the baseline. A possibility of ruling out the learning rate dependence could be a quick logarithmic sweep across learning rates (maybe on a cheap dataset, if compute is an issue). Another would be to repeat the experiment, but constrain the layer-wise gradient norms to be equal between the runs.\n\n- I am confused by Table 7: results show that the orthogonal residual update performs worse than the regular residual connections for all 3 initializations. Why is that? What initialization is used in Table 2?\n\n- Can you provide a simple number on how much the runtime actually increased for the specific benchmarks used (e.g. ResNet or ViT-B on ImageNet)?\n\n- [1] Show a frequency dependency in the average correlation between the main and residual branch in ResNets (ref. Figure 7). The authors argue that this induces a \"Frequency-dependent signal-averaging\" which further dampens high frequencies in the loss surface, which they conjecture is one of the benefits of residual connections and more generally multi-path architectures. As your method renders the signal from the main and residual branch uncorrelated for all frequency bins, do you think this could negatively affect trainability?\n\nWe are happy to revise our score if our reservations about the experimental results are cleared.\n\n[1] Ringing ReLUs: Harmonic Distortion Analysis of Nonlinear Feedforward Networks (ICLR 2021)", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 4, "strengths_and_weaknesses": "Strengths: \n\n- The motivation behind the proposed modified residual connection is simple, clear and intuitive. \n- The additional computational overhead is clearly described (in variables). \n- The idea seems to be novel to the best of our knowledge. \n- Code is provided by the authors.\n- Experimental evidence comprises multiple architectures and datasets; The runs for the main results are repeated 3-5 times, and standard deviation is reported.\n- Experimentally (ref. Fig 4), the authors show a continuous transition between ordinary residual connections and the orthogonal residual connections with a monotonous response.\n- The experiment in Figure 3 provides experimental evidence for anti-alignment.\n\nWeaknesses:\n- The experiment in Figure 3 is moderately insightful, as 150 Epochs is already enough to reach an almost full training performance on Cifar10/100.\n- The reported baseline (Table 2) for ViT-B on ImageNet-1K seems suspiciously low with 71.09%. Google reports 85.49% performance for ViT-B/16 on ImageNet.\n- A rundown of the added overhead (e.g. time difference in ms on the same hardware) in the specific experimental scenario used would be nice, as less operations do not always translate to faster runtime in CUDA pipelines.\n\nOverall, I think the idea is novel, well motivated and executed; however the experimental evaluation raises some serious doubts (particularly that this method changes the layer-wise gradient norms, paired with an evaluation a single LR) that need to be cleared out before the paper can be considered for acceptance.", "was_revised": true, "final_justification": "The authors have cleared out all of our objections in their rebuttal.", "rebuttal": "We appreciate the reviewer’s thoughtful feedback. Below, we address each concern, grouped by topic.\n\n---\n\n### On ViT Baseline Performance and Robustness to Learning Rate\n> \"The reported baseline (Table 2) for ViT-B on ImageNet-1K seems suspiciously low... Generalization performance is highly dependent on learning rates... A possibility of ruling out the learning rate dependence could be a quick logarithmic sweep...\"\n\nWe thank the reviewer for this critical question. It touches upon two key aspects: the baseline's absolute performance and its robustness to learning rates.\n- **On the Baseline Performance and Training Paradigm**: You are correct that our ViT-B baseline of 71.09% is lower than the ~85% reported by Google. This difference stems from a fundamental choice in training paradigm. The original ViT model's high performance relies on pre-training on the massive, non-public JFT-300M dataset. Our work, however, follows the more challenging and widely adopted academic paradigm of training from scratch on ImageNet-1k. As established by influential follow-up works like DeiT and T2T-ViT [1], this setting requires strong regularization (e.g., MixUp, CutMix, RandAugment) to prevent overfitting. Our experimental setup adheres to this modern, robust training recipe. Therefore, our baseline is a valid and fully converged result within this specific paradigm, not an undertrained model. This is further evidenced by the clear performance saturation shown in our accuracy curves.\n- **On Robustness to Learning Rate**: We agree that the change in layer-wise gradient norms makes robustness to LR a crucial concern. To address this directly, we conducted an LR sweep on CIFAR–10, -100 for our ViT-S model. The results below demonstrate that our method's superiority is not an artifact of a single learning rate.\n\n_CIFAR-10 LR sweep results of Validation Accuracy@1 on ViT-S_\n\n|LR|linear|orthogonal|\n|:-:|:-:|:-:|\n|5e-4|0.9041 ± 0.0015|0.9056 ± 0.0032|\n|6e-4|0.9070 ± 0.0012|0.9073 ± 0.0015|\n|7e-4|0.9054 ± 0.0023|0.9101 ± 0.0022|\n|8e-4|0.9045 ± 0.0016|0.9091 ± 0.0033|\n|9e-4|0.9014 ± 0.0023|0.9057 ± 0.0027|\n|1e-3|0.8995 ± 0.0036|0.9036 ± 0.0015|\n|2e-3|0.8485 ± 0.0107|0.8738 ± 0.0042|\n|5e-3|0.6609 ± 0.0129|0.7220 ± 0.0266 |\n\n\n_CIFAR-100 LR sweep results of Validation Accuracy@1 on ViT-S_\n\n|LR|linear|orthogonal|\n|:-:|:-:|:-:|\n|5e-4|0.7146 ± 0.0059|0.7248 ± 0.0037|\n|6e-4|0.7139 ± 0.0037|0.7295 ± 0.0014|\n|7e-4|0.7155 ± 0.0039|0.7283 ± 0.0036|\n|8e-4|0.7113 ± 0.0092|0.7299 ± 0.0029|\n|9e-4|0.7099 ± 0.0050|0.7260 ± 0.0053|\n|1e-3|0.7059 ± 0.0080|0.7311 ± 0.0039|\n|2e-3|0.6217 ± 0.0121|0.6971 ± 0.0091|\n|5e-3|0.4261 ± 0.0282|0.4824 ± 0.0250|\n\nThese results provide conclusive evidence that our method's superiority is not an artifact of a single, fortunate learning rate. The consistent performance advantage across a wide range of learning rates demonstrates the fundamental robustness of the orthogonal update mechanism. We will add this comprehensive sweep to the Appendix.\n\n---\n### On the Interpretation of Table 7 and ResNetV2-50 Performance\n> \"I am confused by Table 7: results show that the orthogonal residual update performs worse than the regular residual connections for all 3 initializations. Why is that? What initialization is used in Table 2?\"\n\nWe thank the reviewer for pointing out this potential source of confusion. We apologize for not making the distinction clearer.\n- **Different Experimental Setups**: The results in Table 7 are from a specific ablation study in Appendix C.1 designed solely to test initialization robustness. This experiment used ResNetV2-50 with our Global (Orthogonal-G) update and was run for a limited duration (60,000 steps, approximately 150 epochs) on CIFAR-100. In contrast, our main results in Table 2 use the superior Feature-wise (Orthogonal-F) update and report performance for fully trained models (200 epochs). As shown in Table 2, our primary Orthogonal-F method demonstrates clear performance gains on ResNetV2-18, -34, and -101.\n- **On ResNetV2-50 Performance**: We acknowledge the performance gains on ResNetV2-50 are more modest. As we hypothesize in Section 4.2, we believe this is connected to architectural properties captured by our γ-ratio. The bottleneck design of ResNetV2-50 gives it a much higher γ-ratio (59.0) than ResNetV2-34 (14.75) . This suggests a nuanced interplay between representational width, architectural design, and our update's efficacy, which, as we state, \"warrants further dedicated investigation.\"\n- **Initialization Method**: For our main experiments in Table 2, we used standard initialization schemes appropriate for each architecture: Kaiming Uniform initialization for both ResNetV2 and ViTs, with batch normalization parameters initialized as described in Appendix C.1.\n\n---\n### On Practical Runtime Overhead\n> \"Can you provide a simple number on how much the runtime actually increased for the specific benchmarks?\"\n\nWe thank the reviewer for this practical and important question. We agree that theoretical FLOPs do not always translate directly to runtime. To provide a direct answer, we have measured the practical training throughput (in images per second) for several of our key architectures on the hardware used in our experiments.\n\n_Throughput (sample images / sec) comparison table. higher is better._\n\n|Architecture | Linear | Orthogonal-F|Overhead (vs. Linear)|Orthogonal-G| Overhead (vs. Linear)|Hardware|Dataset|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|ResNetV2-18|3,053.61|2,975.72|2.55%|2,613.48|14.41%|2x rtx 4090|tiny imagenet|\n|ResNetV2-34|1,737.15|1,633.98|5.94%|1,620.78|6.70%|2x rtx 4090|tiny imagenet|\n|ResNetV2-50|1,002.79|876.69\t| 12.58%|869.23\t|13.32%|2x rtx 4090|tiny imagenet|\t\n|ViT-S|3,476.13|3,466.28|0.28%| - | -|4x A100 80GB\t|imagenet 1k|\t\n|ViT-B|1,270.10|1,246.21|1.88%| - | - |4x A100 80GB|\timagenet 1k|\n\nAs the results show, the practical throughput overhead of our primary Orthogonal-F update is modest. For ResNetV2 models, it ranges from ~3-13%, and for the high-capacity ViT-S and ViT-B model under `torch.compile`, the overhead is even lower at less than 2%. We believe this minor cost is a very favorable trade-off for the substantial gains in final accuracy and training stability that our method provides.\n\nFurthermore, this small per-step overhead is often outweighed by improved convergence speed. As shown in Figure 2b, our method exhibits a significant time-to-accuracy advantage on large-scale tasks. For ViT-B on ImageNet-1k, our method reaches the baseline's final accuracy in ~40% less wall-clock time, demonstrating superior overall training efficiency.\n\nWe will add this detailed throughput analysis to the appendix.\n\n---\n### Understanding the Distinction: Statistical vs. Geometric Orthogonalization\n> \" ... As your method renders the signal from the main and residual branch uncorrelated for all frequency bins, do you think this could negatively affect trainability?\"\n\nThe key distinction lies in how orthogonality is achieved. In [1], Frequency-Dependent Signal-Averaging (FDSA) emerges as a statistical byproduct - residual branches become less correlated at higher frequencies due to independent weight distributions and harmonic generation. Our method, however, implements geometric orthogonalization at each layer, ensuring $f_\\perp(x_n)$ is explicitly perpendicular to $x_n$.\n\nWhy This Enhances Rather Than Impairs Trainability:\n\n- **Amplified Smoothing Effect**: While [1] shows that residual connections provide \"passive\" FDSA benefits, our orthogonal updates actively maximize this effect. By explicitly removing the parallel component $f_\\parallel$, we eliminate redundant signal modulation that could interfere with gradient flow, potentially achieving stronger blueshift mitigation than standard residual connections.\n- **Preserved Expressivity**: Unlike the activation function modifications in [1] (e.g., Leaky ReLU slope adjustment) that reduce model expressivity to improve trainability, our method maintains the full computational capacity of each module. We only restructure how the module output is integrated into the residual stream, not the module's internal complexity.\n- **Deterministic Stability**: Rather than relying on statistical decorrelation across frequencies, geometric orthogonalization provides deterministic norm preservation and directional diversity. Our experiments show this leads to more stable stream norms ($||x_n||^2$) and consistent rotational updates that guide representations toward new feature directions without magnitude explosion.\n- **Empirical Evidence**: Our results demonstrate that this \"complete orthogonalization\" across frequency bins actually improves trainability, particularly in high-capacity architectures like Vision Transformers. The substantial performance gains (+4.3%p on ImageNet-1k) suggest that maximizing the FDSA principle through geometric means enhances rather than hinders the beneficial effects identified in [1].\n\nIn essence, our method can be viewed as taking the FDSA insight from [1] to its logical conclusion - if frequency-dependent decorrelation helps training, then explicit orthogonalization should help even more, which our empirical results confirm.\n\n[1] Ringing ReLUs: Harmonic Distortion Analysis of Nonlinear Feedforward Networks (ICLR 2021)\n\n\n---\nThank you again for your thoughtful engagement with our work. This discussion has been invaluable in helping us refine our arguments. We hope that our detailed response clarifies our contribution and resolves any remaining questions."}]

Author rebuttal:

Meta-review: Tl;dr: Based on the reviews, rebuttal and ensuing discussion I recommend accept.

### Paper Summary

Paper proposes a simple modification to residual connections in deep neural networks, where the component parallel to the input to a block is discarded before the residual is added back to input. Empirical results are provided on several standard benchmarks using ResNetV2 and ViT architectures.

### Strengths and weaknesses

Strength: 1) Simple and novel idea, that is easy to implement, 2) Good empirical validation on several standard benchmarks and models, 3) Reasonable analysis: backed by empirical observations. Author rebuttal to the reviewer questions further strengthened the paper with additional experiments and analysis.

Weaknesses: 1) Motivation: Initial concerns from reviewers. Initial manuscript did not clearly motivate why the parallel component of the residual update may be detrimental. 2) ViT baseline: concerns about low baseline performance taint reported gains. 3) Effectiveness on very deep models needs further investigation as per reviewers.

### Decision justification

The paper introduces a simple and novel idea that can significantly impact the design of deep neural networks. Empirical results are compelling. The authors did a great job in their rebuttal, addressing the concerns with additional experiments and analysis. Additional analysis and clarifications from the rebuttal and discussion should be incorporated into the final version of the paper. While there are lingering minor concers, the paper merits acceptance in my view. All reviewers were leaning positive as well.
Decision: Accept (poster)