Cutting the Skip: Training Residual-Free Transformers
Authors: Yiping Ji
James Martens
Jianqiao Zheng
Ziqin Zhou
Peyman Moghadam
Xinyu Zhang
Hemanth Saratchandran
Simon Lucey
Venue: ICLR 2026 Poster (2026)
Keywords: Vision Transformers
Skip Connections
Network Conditionin
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Transformers have achieved remarkable success across a wide range of applications, a feat often attributed to their scalability. Yet training them without residual (skip) connections remains notoriously difficult. While skips stabilize optimization, they also disrupt the hierarchical structure of representations, raising the long-standing question of whether transformers can be trained efficiently without them. In this work, we address this problem by analyzing the Jacobian of a skipless transformer block, showing why residuals improve conditioning and revealing that their stabilization benefits can be recovered through a principled initialization strategy. Building on this insight, we introduce the first method that enables stable and efficient training of skipless transformers without altering the standard architecture. We validate our approach on Vision Transformers (ViTs) in both supervised and self-supervised settings, demonstrating that skipless ViTs trained with our initialization overcome the usual optimization barriers, learn richer hierarchical representations, and outperform strong residual baselines on dense prediction benchmarks. These results show that skip connections are not a fundamental requirement for training ViTs and open new avenues for hierarchical representation learning in vision models.

Published as a conference paper at ICLR 2026
CUTTING THE SKIP: TRAINING RESIDUAL-FREE TRANSFORMERS
Yiping Ji
James Martens
Jianqiao Zheng' Ziqin Zhou'
Peyman Moghadam2
Xinyu Zhang'
Hemanth Saratchandran' Simon Lucey'
Australiaa Institute for Machine Learning, Adelaide University 2AAT661, CSIRO
University of Auckland
ABSTRACT
Transformers have achieved remarkable success across a wide range of applica- tions, feat often attributed their scalability. Yet training them without residual (skip) connections remains notoriously difficult. While skips stabilize optimiza- tion, they also disrupt the hierarchical structure of representations, raising the long-standing question of whether transformers can be trained efficiently without them. In this work, we address this problem by analyzing the Jacobian of a skip- less transformer block, showing why residuals improve conditioning and revealing that their stabilization benefits can be recovered through a principled initialization strategy. Building this insight, we introduce the first method that enables stable and efficient training of skipless transformers without altering the standard archi- tecture. We validate approach on Vision Transformers (ViTs) in both super- vised and self-suue settings, demonstrating that skipless ViTs trained with our initialization overcome the usual optimization barriers, learn richer hierarchi- cal representations, and outperform strong residual baselines on dense prediction benchmarks. These results show that skip connections are not fundamental re- quirement for training ViTs and open new avenues for hierarchical representation learning vision models.
INTRODUCTION
Over the past decade, large transformer-basee models have achieved remarkable success, demon- strating strong zero-shot and generalization capabilities across tasks and domains through a single, reusable model (Caron et al., 2021; Comanici et al., 2025; Wang et al., 2025). Their ability to be trained at great depth relies heavily on skip connections (He et al., 2016), which have become a cornerstone of modern deep learning models. This unprecedented scalability in depth is widely regarded as a key factor behind the astonishing performance of transformer-batt architectures.
However, the reliance on skip connections raises an important question: do such networks truly operate at the depth implied by their architecture? Prior work (Veit et al., 2016; Gromov et al., 2025) suggests that residual connections make networks behave if they much shallower than their nominal depth. An earlier study (He et al., 2023) was the first to investigate "skipless trans- formers". introducing modified self-attention block preserve well-behaved forward kernels. Al- though this modification improved trainability, the resulting models still converged significantly more slowly than their residual counterparts. This paper addresses this gap by introducing a theo- retically grounded initialization scheme that does not require architectural changes. Combined with a second-orde optimizer (Vyas et al., 2025), our approach enables skipless Vision Transformers to achieve training speeds comparable to residual-bar based models.
The concept skip connections dates back the 1960s: Rosenblatt al. (1962) described three- layer multilayer perceptron referred to as a cross-coupled system, where skip-like couplings were
"Corresoonding c-maillyiping wwdlaide..du...

---

Published as a conference paper at ICLR 2026
already present. Decades later, skip connections were popularized in ResNets (He al., 2016) and subsequently adopted in transformers (Vaswani et al., 2017), and they are now considered crucial for training very deep networks. One proposed explanation for their effectiivneess is that they improve the conditioning of the network Jacobian, thereby facilitating gradient flow and enabling faster, more stable convergence (Ji et al., 2025b). Empirical evidence also suggests that self-attention tends to be disproportionately ill-condiionnddaalin as an optimization bottleneck- other components such feed-forward networks, underscoring the stabilizing role that residual connections can play within transformers.
to
While skip connections are vital for optimizing modemn neural networks, they also change how architectural depth is functionally expressed. Deep networks are intended to form compositional hi- erarchies in which representations become progressively more abstract layer by layer (LeCun al., 2015). However, skip connections disrupt this hierarchy by continually reintroducing information from earlier layers into later ones. This shorteutting interrupts the intended progression of abstrac- tion (Zhang al., 2024) and can limit the network's ability to learn rich, deeply composed features. As a result, networks with skips often behave as they are effectively shallower than their nominal depth suggests. Prior studies have shown that in ResNets, skip connections reduce the role of deep compositions, making networks behave like ensembles of shallower subnetworks (Veit et al., 2016). In modern transformers, this effect ers contribute so little to the final prediction that they can be pruned with minimal loss (Gromov al., 2025). Together, these findings suggest that while skip connections are indispensable for op- timization, they may obscure the true representational benefits of depth - motivating our goal of designing transformers without shortcuts.
more pronounced: after convergence, many deeper lay-
To the best of our knowledge, the only prior work to train skipless transformers is that of He et al. (2023), who modified the self-attention block to maintain well-behaved forward kernels and pre- vent the kernel matrix from collapsing toward rank 1. Although their method successfully removes residual connections, does so by altering the standard transformer architecture, and the modified attention blocks are not compatible with widely used optimizations such as Flash Attention (Dao, 2024). In contrast, our approach requires no architectural changes: we retain the standard trans- former block design and achieve stable training of skipless transformers solely through principled initialization strategy.
Guided by the first principle of gradient-basee optimization good network conditioning (see Sec- tion 4.1) (.l))wee we analyze the Jacobian of transformers and use this insight to design a principled ini- tialization strategy that enables stable training of skipless models. Our main contributions are:
Jacobian analysis: We provide a theoretical study of the transformer Jacobian and show that skip connections stabilize optimization by improving its conditioning.
Initialization without architectural changes: Guided by this analysis, we introduce simple, theoretically grounded initialization scheme that requires no changes to the trans- former block, remains fully compatible with FlashAttention, and enables stable end-to-end training of skipless transformers.
On image classification benchmarks, skip- less models trained with second-order optimizer converge quickly standard residual transformers and achieve comparable accuracy.
Supervised training at parity with residuals:
Improved self-supervised representations: In supervised learning, skipless models outperform residual transformers in dense prediction tasks, while being pfrfmeter-effciient, training faster and producing more semantically coherent representations.
Enabling depth studies: Our approach makes it possible- for the first time- to system- atically study truly deep (skipless) Vision Transformers, offering new insights into hierar- chical representation learning in vision.
2 TRANSFORMERS: TERMINOLOGY AND NOTATION
A standard transformer begins with token embedding Xo E Raxd, where n is the number of tokens and is the embedding dimension. This embedding is then passed through a stack of L transformer blocks. Each block consists of two main components: Self-Attention Block (SAB)
2

---

Published as a conference paper at ICLR 2026
and a Fred-Forward Network (FFN), defined in Eqs. 1 and 2, respectively. The SAB applies Self-Attention (SA) together with a residual (skip) connection, while the FFN applies multilayer perceptron (MLP), also with residual connection. We denote by X₁ the output embedding after the f-th transformer block. In summary, we have
X; = X + SA(Xa-1) and
(1)
X, = X; + MLP(X;);
(2)
Self-attention allows the network to selectively attend to relevant parts of the input and is core component of modern transformers.
Omitting & for clarity, the self-attention operation is defined
SA(X) AVW".
(3)
where Q = XW K = XW v = XWV, and the attention matrix is A = ((QK))
Here, Q K, V the queries, keys, and values, respectively. The parameter matrices W WK WV, W Igdxd are learnable, and n7-) is typically the softmax function.
In practice multi-head atlention is used. The projection matrices are divided across h heads, such that
1o. WKK W. Rdxdd
dA
For head we compute
A, = (Q,K)) and the final output is obtained by concatenating across heads:
SA(X) Concat A V1 A A
(4)
3 RELATED WORK ON SKIPLESS ARCHITECTURES
Many works have successfully removed skip connections in CNN architectures, overcoming op- timization challenges and achieving competitive performance (Zhang al., 2022: Zagoruyko & Komodakis, 2017: Martens et al., 2021). In contrast, in the transformer domain, to the best of our knowledge, only one paper has investigated training skipless language transformers (He et al., 2023) by modifying the Self-Attentin Block. Based on the observation that skipless transformers are suf- fering from rank collapse (Noci et al., 2022), where the kernel matrix converges depth to have rank 1, they modified the self-attention block to maintain rell-behaved kernels initialization. Our work differs from this previous attempt in that we focus on the conditioning of the network Jacobian instead of the properties of the kernel, our modifications are purely to the initialization of the weight matrices, and our experiments consider vision models instead of text.
4 NETWORK JACOBIAN ANALYSIS
4.1 PRELIMINARIES
Throughout this paper, when analysing the network Jacobian, we denote the transformer network as fxx 0) E "Rd where x = vec[X]] is the vectorized token embedding, is the number of tokens, is the feature dimension, and 0 denotes all learnable network parameters such that p dim Importantly, fxx; 0) deliberately omits the token embedding and output-head so that we can focus on the internal interactions of the transformer blocks; for this reason the network output of size rd..
For batch of m input examples, we define the stacked output
F(0)= = [f (x1: 0): ... f(xm: 8)] E R mad
(5)
The network Jacobian is then J of the network's 's training dynamics. We define the condition number as the ratio of the largest to smallest singular value xx[ smax min'
OF E m and its conditioning provides a key indicator
3

---

Published as a conference paper at ICLR 2026
Prior research has shown that improved transformer conditioning leads to more stable training and stronger results. For example, Ji et al. (2025a) improved the conditioning of low-rank matrices using sinusoidal activations, thereby enhancing low-rank learning without additional parameters. Similarly, Saratchandran & Lucey (2025) introduced conditioned embedded tokens, strengthening conditioning with minimal overhead. More recently, Ji al. (2025b) argued that the primary role of skip snnrectioon-particlllee within the self-attention block-is to improve conditioning, and demonstrated that transformers fail train their absence.
A central hypothesis of this work is that residual (skip) connections, while essential for optimiza- tion, violate the hierarchical principle of deep networks by continually injecting shallow features into deeper layers. Removing these shortcuts makes training challenging because the Jacobian of skipless transformers is poorly conditioned at random initialization (Ji al., 2025b). Building on theoretical analysis of the network Jacobian, we propose principled initialization strategy that directly improves conditioning. This enables training completely skipless transformers speeds comparable to standard residual models while learning richer, more semantically coherent internal representations.
4.2
DECOMPOSITION OF THE NETWORK JACOBIAN
The Jacobian of the transformer network F can be decomposed into block columns:
0F
J
JJ.JJ.
Jr. Jz).
00
where
J: OF 20(C)
j. OF
E R mmndxpp
E IR mndxpr
ag(z)
Here, Jr and Jg are the Jacobians of the final output with respect to the parameters the 6-th SAB and FFN sub-blocks, respectively, and Σt= (pe + pe) = dfm =
Following Ji et al. (2025b), adopt the simplifying assumption that the conditioning of the full Jacobian is controlled by its worst-conditioned sub-blocks. In particular we write
k(J) max (I((J), k().)).
That the spectral condition number of the entire network Jacobian is assumed to no larger than the worst condition number among all SAB and FFN sub-block Jacobians.
This assumption does not hold universally, but we provide justification for under mild block- incoherence conditions with balanced blocks (see Section A.4.4). Ji et al. demonstrated both theo- retically and empirically that SAB sub-block Jacobians are significantly less well-conditioned than their FFN counterparts. For this reason our focus on this paper is around the condition of the SAB sub-block Jacobian Jg-
4.3 DIVING INTO SUB-BLOCKS: SKIP CONNECTIONS IMPROVE CONDITIONING
With skip connections, the vectorized SAB and FFN sub-block updates at layer E are x( (0) fSA (x (1, 0(0) ) + (f-1) E Rnd
(0) fMMP(( (2) ((0) + (() Rdd
Denoting the derivative of the SA and MLP output with respect to the corresponding inputs by
KP 0 (x
K, 0f 0(0)
padxnd
E R ndxnd
(x-1
Ox
have the derivative of the network output with respect to the SA parameters at layer l is:
0x(x;0) 2
f+1
II (K. Ind) (K; Ind) (K, Ind
(x(1 0(0)
nddxpr
(8)
20(0)
If skip connections are not present, we have:
0f(x;))
00(1) = II K,K,) K. 01((()(1111; 00(0)
6+1
E R nnxxe
(9)
1=b
4

---

Published as a conference paper at ICLR 2026
From Eq. (5), the Jacobian Jg Runddp: is the concatenation of 0f(x:0)
2000 Rndxpe for
ples. By assumption i), we have x(Je) is bounded by the largest n( (o1e) for m samples. Thus, comparing Eqs. (8) and (9), we can see clearly how the skip connections (Ind term) help network conditioning. As we stated before assumption ii), compared to the conditioning of the MLP Kr, the conditioning of the SA K2 is much worse (explored in Section 5.1 under default truncated nor- mal initialization with Proposition 1). The addition of the identity matrix Ind in Eq. (8) shifts the spectrum of K2 from zero, regularizing the smallest singular values.
(f(x00)
These observations invite the question: is there an alternative way to maintain the good conditioning of K2 such that x² (Kr) as <<Ke + Ind) in skipless transformer?
5 A NEW INITIALIZATION TO ENABLE SKIPLESS TRANSFORMERS
Based on the previous analysis, goal is improve the conditioning of n: k(K)). To this end, first give an expression K, at layer f:
K = (Xc-1 WWW & L.) A (WXW?)T & A..
(10)
where A, E RR the attention matrix, and A E R xnd is the derivative of the attention matrix with respect to the input (((-1) = vec(X.-1) (vectorized when forming the derivative matrix) !
Using this expression we will proceed to derive a principled initialization for the weight matrices w WK. wV and, W to improve the conditioning of Kg- W!WD appears in both term and w appears in the As and A-
5.1 INITIALIZATION FOR W. WO
A key observation (see Eq. 10) is that the product w/ W appears in both terms of the Jacobian. For training to be stable, this product must be well-conditioned in order to improve the condition of K,. The best-case scenario when it is a (scaled) orthonormal matrix, because that case all of its singular values are equal, so that k(W) W 1. To achieve this, we first initialize random square matrix Q d with zero-mean, unit-variance entries. Then we perform an SVD decomposition such that Q USV and we assign w c. U and w c V where is scaling constant. This ensures the matrix W/WO is scaled orthonormal.
5.2 INITIALIZATION FOR W WOWKT
Recall the attention A, softmax(Mc), where M. = - w/w? T X The conditioning of A critically depends on the structure of its logits M,.
Proposition 1 (Softmax conditioning: diagonal dominance vs. diffuse rows). Let S, (M,) E Raxx denote the row-wise softmax with lemperature
O.
(Diffuse rows). If each row of M, has a small range (difference between maximum and minimum) ATT then Sz (Mg) is close the uniform matrix L11 which has rank In this case x(S)(M,) Z I. and the conditioning worsens as n grows.
(Diagonal dominance). Mf diagonal dominant, i.e. Mù maxx41 1604sMMM Mij 7 0, then Sy (Me) is close to an identity matrix, and
1+t(1/T) x(S((M)) I I-5(7/7)²
with as Hence S, (M,) is well-conditioned when diagonal logits are dominant.
E is defined head index in the previous section but in the following sections redefine the f. as the
block index
5

---

Published as a conference paper at ICLR 2026
An illustration of this proposition is in Section A.4.1. This proposition highlights the key insight: at random initialization, the logits M are "diffuse", hence, the attention matrix A is close to the uniform matrix and thus ill-conditioned (see Section A.4.1), which is the main cause of the ill- conditioned ()
To address this, we initialize the query and key projections w and W such that
W w aZ + SI.
(11)
where the entries of Z are sampled Zu ~ M(., 2). is the weight dimension, is the identity ma- trix, and a, 3 are scalar constants. This scheme -osesimometsss called mimetic initialization (Trockman & Kolter, 2023) has been shown empirically improve both convergence and final performance in transformers.
Our contribution is provide theoretical motivation: the identity term 3I encourages diago- nal dominance in w which in turn helps ensure that the initial attention operator is well- conditioned at the start of training. However, we emphasize that diagonal dominance w does not automatically imply that the transformed matrix X WoW wo X is also diagonally domi- nant. In Section A.4.1 we detail the conditions under which this property carries over after projection by the token embeddings X.
A scaled orthonormal w w and diagonal dominant attention map A guarantees that the second term of Kg, namely (W W @ As, is well-conditioned. The remaining question is whether this also ensures well-conditioned K₃ overall.
Proposition 2. (Conditioning of K.) Let K2 = B, + E,, where E, = W'wo & I.)) (the "perturbation term") and B, W wY so A (the "dominant term"). With above ini- tialization (which ensures diagonal dominance of w? K₁K, well-conditioned.
The detailed proof is provided Section A.4.3. The intuition behind this proposition is that the largest singular value of perturbation term Eg is smaller than the smallest singular value of the dominant term By, then <(())) <(:)).
Takeaway
By initializing W. Wo be scaled orthonormal and W to be a diagonally dominant structure, we improve the conditioning of the network Jacobian, addressing the main barrier that has historically prevented the training of completely skipless transformers.
EXPERIMENTS
evaluate our methods in supervised learning and self-supervised learning settings. All of our experiments will be Vision Transformers (ViTs) (Dosovitskiy et al., 2020), which have emerged as powerful models in the field of computer vision, demonstrating remarkable performance across various tasks.
6.1 SUPERVISED LEARNING WITH SKIPLESS VIT
We first validate our skipless VITs on supervised learning image classification tasks. The model in this subsection is ViT-Base (12 layers, 12 heads, head dimension 64, token dimension 768). The skip models are standard ViT-Base, while in the skipless models we remove all skip connections (from both the SABs and FFNs), and use the proposed initialization for the SA weights (choosing = 3 = 0.6 and = 3)2. The scaled-corrected uniform orthogonal initialization (Martens et al., 2021) is used for the MLP parameters. Our implementation follows the setup in (Xu et al., 2024), except that for a fair comparison we disable the drop path, which is not applicable in skipelss models. All experiments are conducted on the ImageNet-Ie (Russakovsky et al., 2015) dataset. We further compare the performance when using Adam W (Loshchilov & Hutter, 2019) and SOAP (Vyas et al., 2025) optimizers.
We observed that initialization hyperparameters (o,, 8, are not highly sensitive.

---

Published as a conference paper at ICLR 2026
AdamW optimizer
SOAP optimizer
7
VIT-B Skip
VIT-B + Skip
Training Loss
VIT-B Skipless
6-
VIT-B + Skipless
VIT-B Skipless Init
VIT-B Skipless Init
5
4-
3
50
100
250
300
50
100
150
200
250
300
Epoch
Epoch
Figure 1: Supervised training loss of VTT- Base using AdamW (Left) and SOAP (Right) optimizers.
Table Validation accuracy Base on ImageNet- Ik using Adam and SOAP optimizers.
Model
Accuracy
ViT-Base + AdamW
80.3%
skip
ViT-Base SOAP
80.1%
ViT-Base Adam
61.4%
skipless
ViT-Base SOAP
77.0%
ViT-Base AdamW ViT-Base + SOAP
78.1%
skipless + our init
80.8%
Results and Analysis. As shown in Table 1, the removal of skip connections severally hampers the convergence ViT-Base when trained with Adam W. This evident both in the substantial accuracy drop (61.4% vs. 80.3%), and the high training loss with slow convergence illustrated in Fig. (left). Using SOAP can partially alleviate this issue, enabling skipless models to converge more reliably and recover much of the lost performance, while they still underperform standard ViTs with skip connections. Incorporating our proposed initialization significantly mitigates these issues. When trained with AdamW, skipless ViT-Base recovers most of the lost performance. Moreover, when combined with SOAP, skipless models can converge as fast as vanilla ViT-based at the standard 300 epochs and achieve 80.8% accuracy, surpassing the skip-based ViT-Base baseline by 0.5%. These results demonstrate that the proposed initialization is essential for enabling competitive training of skipless ViTs across optimizers.
6.2 SELF-SUPERVISED LEARNING WITH SKIPLESS VIT
We further evaluate our skipless VIT model in the self-supervised setting. Specifically, we adopt DINO (Caron et al., 2021), widely used self-supervised framework based on self-distillation with- out annotations. Here we use ViT-Small (12 layers, heads, head dimension 64, token dimension 384), and model recipe to the previous subsection. We compare results under both Adam and SOAP opti- mizers. For quantitative evaluation, extract representations from individual or multiple blocks of frozen pre-trained models, and assess them on two downstream tasks: dense linear probing seg- mentation (in Section 6.2.1) and object discovery (in Section 6.2.2). For qualitative evaluation (in Section 6.3), we use Principle Component Analysis (PCA) (Abdi & Williams, 2010) to project the learned representations into 3-channel feature maps, visualized RGB images.
1.8, 3 1,
for the initialization parameters, and otherwise follow similar
6.2.1 DENSE LINEAR PROBING SEGMENTATION
We evaluate linear probing on dense features for the semantic segmentation task. linear classifier trained top of the representation, with performance measured by mean intersection-over-uiiie (mloU) on PASCAL VOC2012 (Everingham et al., 2015), ADE20K (Zhou et al., 2019), and COCO- Stuff (Caesar et al., 2018) datasets. We sweep over learning rates and train for 30 epochs. For ADE20k and COCO-Stuff, we randomly sample 3,000 training images due to resource constraints.
Results and Analysis. As shown in Table 2, our skipless DINO ViT-Small models trained with the Adam V optimizer achieve higher performance than their skip-based counterparts on the VOC2012 and COCOStuff benchmarks when evaluated with the representation extracted from the single layer.

---

Published as a conference paper at ICLR 2026
Table 2: Pretrained DINO ViT-Small models for 300 epochs. We also evaluate the checkpoint at 200 epochs for skipless models. Performance on linear probing segmentation tasks on different datasets.
VOC2012
COCOStuff
ADE20K
Epochs
300
300
200
300
300 skipless
200
300 skip
300 skipless
200 skipless
skip
skipless
skipless
skip
skipless
single feature
AdamW
56.3 51.3
62.3
62.1
24.6
24.9
28.3
23.7
22.5
22.8
SOAP
57.6
63.4
21.3
23.5
27.6
20.5
21.3
22.5
multiscale AdamW
61.6
65.4
65.0
26.7
28.0
28.2
26.0
26.3
27.0
SOAP
61.3
59.5
64.8
25.9
25.1
27.6
25.3
23.7
25.6
AdamW
ADE20K
AdamW - COCO-Stuff
Adam VOC2012
27.5
mloU (%)
25
25.0
20
skip skipless
22.5
skip skipless
skip skipless
20.0
50-
SOAP ADE20K
SOAP COCO-Stuff
SOAP VOC2012
27.5
mloU (96)
25
60
20
skip
22.5
skip
skip
skipless
skipless
50-
skipless
20.0
20
60
100
140
180 Epoch
220
260
300
20
60
100
140
180 Epoch
220
260
300
20
60
100
140
220 260 300 Epoch
180
Figure 2: Performance of dense linear probing segmentation results using skip and skipless DINO ViT-Small models with AdamW and SOAP optimizers throughout the pretraining. The range of y-axis is the same for per column.
In contrast, these models show reduced accuracy the ADE20K dataset under the same single-layer setting. We attribute this to the greater scene complexity in ADE20K, where multi-scale information is critical. The skip-based models can implicitly mix representations across layers, providing form of multi-scale context. However, our skipless models enforce a stricter hierarchical structure, yielding abstract features at each layer. Based on this, when multiscale layer features explicitly aggregated at evaluation time, skipless models once again surpass their skip-connected counterparts. While training with SOAP, overall we observe the performance drops for both skip and skipless models and we conjecture that is due to the inductive bias of optimizers (Pascanu et al., 2025). Further, we demonstrate the depth analysis in Table 3. We train our models using the Adam W optimizer with different depths for 300 epochs and evaluate using the checkpoint at 200 epochs. Our models with 10 blocks perform comparably with skip models.
6.2.2 OBJECT DISCOVERY
Detecting salient objects is a fundamental problem in computer vision with applications in real- world vision systems. Traditional methods rely on supervised learning using large-scale high-quality annotated data, which is expensive and time-consuming to obtain these annotations (Loshchilov & Hutter, 2019). To address this challenge, recent works (Siméoni et al., 2021; Wang et al., 2023) have explored self-supervised pre-traiee pre-trained models, which produce high-quality and abstraction feature representations without requiring manual labels. In this subsection, we validate our pretrained DINO models using TokenCuT (Wang et al., 2023), a graph-based algorithm that leverages self-supervised transformer features for salient object detection. Following prior observations (Amir et al., 2022) that positional information gradually diminishes across layers, we compare representations from different transformer blocks and report the best-performing results. We use VOC2012 (Everingham et al., 2015) and COCO20k (Lin et al., 2014) as the evaluation datasets.
Results and Analysis. As shown in Fig. 3, skipless models consistently outperform their skip- connected counterparts by substantial margin on both the VOC2012 and COCO20K datasets under both Adam W and SOAP optimization, indicating that the representations from skipless models
8

---

Published as a conference paper at ICLR 2026
Figure 3: Pretrained DINO ViT-Small models for 300 epochs. For skipless models, also evalu- ated checkpoint at 200 epochs. Performance on object discovery tasks using TokenCut on VOC2012 and COCO20K datasets.
VOC2012
COCO20K
Epoch
300
300
200
300
300 skipless
200
Optimizer
skip
skipless
skipless
skip
skipless
AdamW
32.3
53.5
54.0
21.2
36.5
38.5
SOAP
49.4
63.2
68.1
27.5
46.7
54.1
Original
Skip
Skipless
Figure 4: Visualize learned representations from pretrained DINO models without cherry-picking
abstract and high-quality. Furthermore, in Table 4, we evaluate end-to-end trained models of varying depths and find that skipless VITs with only 9 layers surpass skip-based 12 layer VITs both datasets, highlighting the efficiency of the skipless design.
6.3 QUALITATIVE EVALUATION
To deeply analyze effectiveness of our skipless ViTs, we visualize representations of pre-trained models. Here, we choose the features from 11-th blocks. As shown in Fig. 4, we select the first 40 images from the COCO validation set without therry-picking Ten examples are shown in the main paper, and the rest are shown in Fig. 7. PCA is applied project the representations into three channels and render them as RGB images. The figure clearly demonstrates that models with skip connections, the features appear patchy and noisy, shallow information is repeatedly injected into deeper layers, hindering the learning of high-level semantic representations. In contrast, skipless models yield clearer object boundaries between different semantic regions with more consistent colors within the same object. These results suggest that skipless ViTs capture more abstract and semantically coherent features.
DISCUSSION AND LIMITATION
Our experiments focus on Vision Transformers, as they offer a well-understood and widely used backbone for analysis, visualization, and controlled experimentation. The inherently compositional structure of vision tasks also makes this domain particularly suitable for examining how skipless models form hierarchical representations. Due to computational constraints, our evaluations are limited to ViT models in the parammerr range. We expect that extending these insights to larger-scale architectures, such as billion-parameen parameter models, will provide further understanding of skipless training at scale and represents an exciting direction for future work.
Although our conditioning analysis relies on mild block incoherence assumption that we do not explicitly verify. The strong and consistent empirical performance of our initialization across dif- ferent model depths and dense prediction tasks suggests that this theoretical simplification remains practically meaningful.
8 CONCLUSION
In this paper, we present a theoretical analysis of the transformer Jacobian and, building on this first principle, propose theoretically grounded initialization scheme that requires architectural mod-

---

Published as a conference paper at ICLR 2026
ifications. This scheme enables efficient training of skipless Vision Transformers. Furthermore, skipless models outperform their residual-based counterparts on dense prediction tasks, suggesting that they learn more abstract and higher-quality internal representations. We hope our work provides new insights into hierarchical representation learning in vision.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work uses only publicly available benchmark datasets and does not involve human subjects, personally identifiable information, sensitive data. Our methods intended for advancing fun- damental research in machine learning.
REPRODUCIBILITY STATEMENT
We have taken care to ensure the reproducibility of all results presented in this paper. Where external code was used, explicit references are provided. We will release the code upon publication.
REFERENCES
Hervé Abdi and Lynne Williams. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4):433-459 2010.
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features dense visual de- scriptors. ECCVW What is Motion For?, 2022.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1209- 1218, 2018.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
Tri Dao. FlashAttention 2: Faster attention with better parallelism and work partitioning. In Inter- national Conference on Learning Representations (ICIRR) 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2000. 1929, 2020.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 11 111(1):98-11) -13,,,933666211 2015.
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Dan Roberts. The unreasonable ineffectiveness of the deeper layers. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https: /tthttss///ppeeerrrewwww htwpte/www.e//ooummmer forum?id = ngmEcEer8a.
appenreview.net
Bobby He, James Martens, Guodong Zhang. Aleksandar Botev, Andrew Brock, Samuel I. Smith, and Yee Whye Teh. Deep transformers without shortcuts: Modifying self-attention for faithful signal propagation. In The Eleventh International Conference on Learning Representations, 2023.
URL https:://// http//ppeneveiieeeeeeeeeeeeeeeee net forum? id=Nd rSQMMjs
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Yiping Ji, Hemanth Saratchandran, Cameron Gordon, Zeyu Zhang, and Simon Lucey. Effi cient learning with sine-activated low-rank matrices. In The Thirteenth International Confer- ence on Learning Representations, 2025a. URL https:/ /owpnrevieeeeeeeee/ee/ee net forum?id- cWGCkd7mCp.
/
11

---

Published as a conference paper at ICLR 2026
Yiping Ji, Hemanth Saratchandran, Peyman Moghadam, and Simon Lucey. arXiv preprint wrXiv:2505.019% 2025b.
Always skip attention.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553))436 444, 2015.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- Learning Representations, 2019. URL https::/openreview... htt wtt.://iwww.eeee/tmmii.ieee.
Bkg6RicqY7.
James Martens, Andy Ballard, Guillaume Desjardins, Grzegorz Swirszez, Valentin Dalibard, Jascha Sohl-Dickstein, and Samuel S Schoenholz. Rapid training of deep neural networks without skip connections or normalization layers using deep kernel shaping. arXiv preprint arX00:2110001765, 2021.
Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems,
2022.
Razvan Pascanu, Clare Lyle, Ionut- -lad Modoranu, Naima Elosegui Borras, Dan Alistarh, Petar Velickovic, Sarath Chandar. Soham De, and James Martens. Optimizers qualitatively alter solu- tions and we should leverage this. arXiv preprint arXiv:2507. 12224, 2025.
Frank Rosenblatt et al. Principles of neurodynamics: Perceptrons and the theory of brain mecha- nisms, volume 55, Spartan books Washington, DC, 1962.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjecv Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJC)), 1 15(3):211-22) 2015. doi: 10. 1007/s 1263-015-0816-y.
Hemanth Saratchandran and Simon Lucey. Enhancing transformers through conditioned embedded tokens. arXiv preprint arXiv:2505. 12789, 2025.
Oriane Siméoni, Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with supervied transformers and 10t.:22999.4279 2021.
no labels. arXiv preprint
Asher Trockman and J Zico Kolter. Mimetic initialization of self-attention layers. In International Conference on Machine Learning, pp. 34456-34468. PMLR, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems, 29, 2016.
Nikhil Vyas, Depen Morwani. Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham M. Kakade. SOAP: Improving and stabilizing shampoo using adam for language modeling. In The Thirteenth International Conference on Learning Representations, 2025. URL https: opwpreevee....eo wowtt//foommmi-IIIIXte...... I=Dxxxxxppx...
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 5294-5306, 2025.
Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, James Crowley, and Dominique Vaufreydaz. Tokencut: Segmenting objects in images and videos with self-supervised transformer and normalized cut. IEEE transactions on pattern analysis and machine intelligence, 45(12):15790-15800 2023.
12

---

Published as a conference paper at ICLR 2026
Zhiqiu Xu, Yanjie Chen, Kirill Vishniakov, Yida Yin, Zhiqiang Shen, Trevor Darrell, Lingji Liu, and Zhuang Liu. Initializing models with larger ones. In The Tweifth International Confer- Learning Representations, 2024. URL https: http:ss///penreeew.netee net wtw.rrrvwwww..et/oormiii...o forum?id- dyrGMhi i cMw.
Songlin Yang and Yu Zhang. Fla: A triton-based library for hardware-efficient implementations of linear attention mechanism, January 2024. URL #lah--irear-atteniio
Sergey Zagonuyko and Nikos Komodakis. Diracnets: Training very deep neural networks without
skip-connections. arXiv preprint arXiv:1706.0000 1706.00388, 2017.
Guodong Zhang, Aleksandar Botev, and James Martens. Deep learning without shortcuts: Shaping the kernel with tailored rectifiers. In International Conference Learning Representations, 2022.
URL https::////. http///ppeeeeweweeeeeee net / forum?
Xiao Zhang, Ruoxi Jiang, William Gao, Rebecca Willett, and Michael Maire. Residual connections harm generative representation learning. arXiv preprint arXiv:2404. 10947, 2024.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):02-331,, 2019.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 USE OF LLMS
Large language models (LLMs) were used to assist with proofreading, formatting, and improving the clarity of writing. All technical contributions, experiments, and analyses were designed and conducted by the authors
A.2 END-TO-END TRAINING WITH LESS DEPTH
In this section, train end-to-end skipless DINO ViT-Small models for 300 epochs with varied depth using Adam W optimizer and evaluate on linear probing semantic segmentation and object discovery tasks. We use the checkpoint at 200 epochs.
Table End-to-end training performance on dense linear probing segmentation on our models with varied depth (AdamW).
depth
VOC2012
COCOStuff
ADE20K
skip
12
61.6
26.7
26.0
12
65.0
28.2
27.0
11
66.2
28.0
26.7
skipless
10
64.1
27.1
25.2
9
61.1
26.0
24.4
Table 4: End-to-end training performance on object discovery models with varied depth (AdamW).
depth
VOC2012
COCO20k
skip
12
32.3
21.2
12
53.5
36.5
11
47.4
31.9
skipless
10
43.9
25.4
9
34.8
24.0
A.3 JACOBIAN
In this section, we provide the derivation of Kg-
The derivative of the SA output with respect to the input is
Ovec (SAr(Xe-1)) Ovec (
(
08(1-1)
Ovee (V,)
= T LLJA + (L. 8 Ar)
(12)
1x
=(VeT 8 LL)A + (L. & (W)
② Inxa)
= =VV, 8 ILLA,, W. @ Ar
The Jacobian of the attention matrix to the input is
14

---

Published as a conference paper at ICLR 2026
K (x 0(0))
E BRndxad
Ox(e-13
Ovec (SAr (X+-1) ()
0x((-1)
Ovec (SA, (Xe-1) Ovec (X,
Ovec (SAr(X.-))
2x
(13)
= =(W? & &L,) Ovec (SA,
0E(1-1)
-(W? @ (IV (TAA + WE @A.)
=(WW? Ve @ ( + W W. AAI)
(X--WW T,
A (W/W?) &
A.4 PROOF
A.4.1 SOFTMAX CONDITIONING
In this section, provide the empirical demonstration of the Proposition 1. We conduct simple simulation experiment and the result is shown Fig. 5. We choose a square matrix M E 10 for the "diffuse" case. Empirically,
and set 0.1, 8 for the "peak" case and 0.1, we can see that when choosing large 3 (ensuring diagonal dominance), the softmax produces a near identity matrix with 1.1. However, M is truncated normal initialized, each row the softmax output is near uniform and the output is ill-conditioned with к ~ 730.1.
Distribution of XX and XZX
Given E Raxd (O, I), we have the mean and varicance of A XXT as follows,
Diagonal entries (i =)::
A
Xae
E[Aa] d,
Var(A.) = 2d.
(14)
where X is Wishart distribution.
Off-ddiggonal entries (i + j):
E[A 0,
Var(Aug) d,
(15)
Then given Z ~~ NO, (T), we have the mean and variance of B XZX as follows,
Softmax of Peaked vs Diffuse Matrices
Peaked rows (K 1.1)
Diffuse rows (K 730.1)
0.104
0.8
0.102
0.100
0.4
0.098
0.2
0.096
Figure 5: Left: We choose
0.1, 3 = to ensure diagonal dominance. Right: We choose
a == 0.1, 0.
15

---

Published as a conference paper at ICLR 2026
Diagonal entries (i =j):
E| = O,
Var(Ba) 2 d-2 2.
(16)
Off-Of diagonal entries (i + j):
B[ 0,
Var(B:;) 1.
(17)
For the combined matrix (attention map) C aB + 3A, we have Diagonal entries = j):
E|C,] 3d,
Var(C.: = a² 2) (2d).
(18)
Off-diagonal entries (i + j):
E|C = 0. Var(Ci;) = a² + t.dd
(19)
The default initialization is equivalent to 3 = 0 (no diagonal dominance in weight initialization) and OO)) (usually around 0.04). All the values in the attention map mean with variance much smaller than 1, which satisfy the diffuse condition.
When B > 0, the difference between the diagonal elements and off off-diagonal element y C₁ Cij follows
(V( +) (3
(20)
(The covariance between Ca and Cij is close to 0 when is large.), which satisfy the diagonal dominent condition with proper B.
To justify assumption that the token embeddings X 1xdd (O, I). show that their empirical distribution closely matches zero-mean, approximately isotropic Gaussian (shown in Fig. 6).
A.4.2 PROOF OF PROPOSED INITIALIZATION
Lemma 1. (Jacobian of the softmax function) Let A = softmax (M), where M = XWO XT. With the proposed initialization (see Eq. (11)), we can show that the 2-norm of the derivative (vec(A) scales, (X)
0 (ae-F)
Proof. See the derivation in Eq. (12), have the bound:
Ovec( (A)
@vec(A)
-
(21)
@vec(X) (X) 12 @vec(M))
Since softmax a row normalization, the Jacobian JA = matrix. For each block i, we have:
(tvea(M) E pn²xnn is block diagonal
@vec( A)
(JA.)jR = Avj(ōsx Ark),
(22)
where Aij the i-th row and j-th column entry of A.
Obviously, using our proposed initialization (laerger 3 leads to more diagonally dominant), we have:
Ag = 1 - 0 and Aij 09)
(23)
Based on this, we analyze the order of magnitude of the elements J..:). the case of j = l:
(JA.)a = Aa(1 Aa) O(ae -) -()=-
(24)
(JAs)a= Ax(1 [6a1-aa) O -okee-)),, )
(25)
16

---

Published as a conference paper at ICLR 2026
Data
0.4
Standard Normal (0,
0.3
0.3
Dersity
Density
Density
0.2
0.2
0.2-
0.1
0.1
0.1
0.0
-5.0
-2.5
0.0
2.5
0.0
5.00
-2.5
0.0
2.5
0.0
-2
Value
Value
Value
block 3
block 4
block 5
0.4
0.4
0.3
0.3
0.3
Density
Density
Density
0.2
0.2
0.1
-4
-2
4
-4
-2
0
4
-2
D
2
4
Value
Value
Value
black
block
block 8
0.4
0.3
0.3
0.3-
Density
Density
Density
0.2
0.2
0.1
0.1
0.1
0.0
0.0
0.0
-2
Value
Value
Value block 11
block_9
block_10
0.4
0.4
0.4
0.3
0.3
Density
Density
Density
0.2
4
-2
0.0
-2
0.0
-4
-2
Value
Value
Value
Figure 6: Distribution of token embeddings after the pre-layer norm throughout the blocks using proposed initialization..
17

---

Published as a conference paper at ICLR 2026
In the case of j + k:
(JA.))a = -ijAiik = Olae -0
(26)
Then we have the bound for |JA.:]::
d (JA,,))a)² d- -
JJA,IlE
(27)
J.km1
Therefore, the 2-norm is:
JA.ile & |JA,i = O
(28)
Then we have:
Ovec(A) Ōvec X)
-max||JA.Il] = O
(29)
A.4.3 CONDITIONING K₁
In this section we provide the proof for Proposition 2
Proof. We show that, with proposed initialization, A is well conditioned such that k(A)
Next step, show that the term E is small perturbation of Jacobian J.
Bound | using norm submultiplicatiitty:
BII2 < | 12 W/ ((veo(Ax(X 1)))
(30)
0x
Since || W/ 12 < [ || = and |||||2 is bounded, using Lemma 1, we have:
www 2| s ((ae
(31)
Combine the bound [B]|2 & and
E 2 < (a) and there exists a and 3 such that | < B 22 Therefore E is small perturbation of B and J is well-conditioned k(J) = 1.
A.4.4 CONDITION OF MATRIX CONCATENATION
Let M [A BJ (d)
Denote the spectral norms and minimal singular values by
max max maxx (A), max (B)} Smin min main (A), miin
and the mutual coherence parameter measuring alignment between A and B and balanced condi- tion 7 measuring the norm difference the matrices:
A B 2
(32)
max(]|A|
2.
B 12)
max Tmaax (A), Fmax [B)] min { @maxx (A), Tmax (B))
=
(33)
min
A
2,
B
B
18

---

Published as a conference paper at ICLR 2026
For any nonzero vectors
Rd
y
R
have
M
1
M
M
y
Y
||Ax|2 + |By| + 2(Ax, By)
(34)
|
lyl?
||x|| + 1
X
1|
Hence the largest singular value of M is
| +
BBy 2| + 2(Ax, By)
O max (M)²
x.y
max
|x|
|
max (A) | o max (B)² || 2 2pl x
max x.y
||X|
|y|
(35)
2p x
max max (A)? max (B)
x.yşU
max
| + 1y
max + p.
The smallest singular value of M is
|2x|2 + |Byl + 2(Ax, By)
O min (M)
min
x,y#0
| 2 y
min (A) |||| 12 Jmin (B) || ||y||
min xyy0
||x|| + ||
(36)
-2pl|x |||y||
min (amin (A)? mmnn (B)2] min
x.y
|| 1 y
mmin p.
Therefore
<(M)
3 max (M)
2 + P
mase
Smax
P
Kmaxi
(37)
minn (M)
x2 min
8min
min
where Kmax is the largest condition number of the component matrices.
Under a mild incc-inncoeeeecce condition (i.e., + 0), and balanced blocks (r + 1), the concate- nated condition number is controlled by the worst block condition number Kmax-
A.5 VISUALIZATION
As shown in Fig. 7, the skipless DINO ViT-Small produces representations that form much more semantically coherent clusters. Moreover, Fig. reveals a clear hierarchical progression: earlier layers capture meaningful subparts and mid-level structures, while deeper layers focus on complete objects. Such hierarchical patterns are weaker and diffuse in the residual basclines.
A.6 DINO PRETRAINING LOss
In Fig. we demonstrate the DINO pretraining loss for both skip DINO ViT-Small and skipless DINO ViT-Small with proposed method.
A.7 JACOBIAN CONDITIONING DURING TRAINING
We tracked the condition numbers the full Jacobian (R(JJTT) as well as the per-layer Jacobian kernels during DINO VIT-Small training. As shown in Table 5, the standard skipless baseline di- verges early in training. resulting in numerical overflow. In contrast, our skipless init variant main- tains condition number of the same order of magnitude the residual (skip) baseline throughout
19

---

Published as a conference paper at ICLR 2026
Original
LPSharan
Skip
Skipless
Original
Skip
Skipless
Original
Skip
Skipless
Original
Skip
Skipless
Figure PCA visualization representation
20

---

Published as a conference paper at ICLR 2026
Skipl
Skipless
Layer
Layer 8
Layer
Layer
Layer
Figure 8: PCA visualization of representation from multiple layers
AdamW optimizer
SOAP optimizer
10
VIT-S Skip
10
VIT-S + Skip
VIT-S Skipless Init
ViT-S + Skipless Init
Training Loss
6-
150 Epoch
200
250
50
100
150 Epoch
250
300
Figure 9: DINO ViT-Small Pretraining Loss
21

---

Published as a conference paper at ICLR 2026
Table 5: Condition number of the full Jacobian x(JJ during DINO ViT-S training.
Epoch 20
Epoch 60
Epoch 120
Epoch 180
Epoch 240
Epoch 300
skip
127
72
82
93
113
144
skipless
00
0C
00
00
00
skipless + init
195
340
261
160
176
207
the entire optimization trajectory. This demonstrates that our initialization successfully mitigates the optimization pathologies that typically arise in skipless architectures.
We also analyzed the conditioning of the layer-wise Jacobian kernels, computing <(K(K) for skipless models and <<(K, + () I) for residual models. Table 6 shows results across depth and training epochs.
Table 6: Layer-wise Jacobian condition numbers across layers. We report s(K,K)) for skipless, and <((K, (); I) for skip.
Epoch 20
LO
L4
L7
LI1
skip
1.05
1.49
1.35
1.04
skipless
00
skipless init
10
2.7
7.5
12.5
Epoch 100
L.
L44
L7
L11
skip
1.06
1.54
1.25
1.25
skipless
00
00
skipless init
6.25
3.7
8.54
19.2
Epoch 200
LO
L4
L7
LI1
skip
1.03
1.57
1.21
1.37
skipless
00
skipless init
6.25
3.84
11.1
21.7
Epoch 300
LO
L4
L7
L11
skip
1.02
1.38
1.28
1.51
skipless
0
0
skipless init
3.44
3.57
14.3
20.2
A.8 LANGUAGE MODELING
We extended our evaluation to Language Transformers. We pretrained 10M parameter model on the C4 dataset for 20k steps (using AdamW) and evaluated zero-shot performance on five common- sense reasoning tasks following (He et al., 2023). We use a batch size of 256k tokens and ablate We use 0, 0.5 for initialization. The zero-shot per- formance on 5 downstream tasks are shown in Table 7. We use the flash-ll Sichhlineee linear at tention codebase (Yang & Zhang, 2024).
learning rate of (1 x
22

---

Published as a conference paper at ICLR 2026
Table 7: Zero-shot performance on downstream tasks.
Architecture
BoolQ
HellaSwag
Winogrande
PIQA
SIQA
Average
skip
57.3
29.7
50.9
63.9
36.5
47.7
skipless
skipless + a=0 3-0.5
61.2
28.2
52.2
62.7
36.5
48.2
Without our initialization, the standard skipless Transformer diverges immediately. With our method, it achieves comparable performance with the residual baseline. Crucially, while prior work (He et al., 2023) can train skipless Transformers, it requires 5 times more training steps to match residual performance. In contrast, our method achieves this the same training speed (Ix steps) as the baseline, demonstrating significantly superior efficiency.
23


Reviews:
[{"review_id": "lzr60nWY5o", "reviewer": "Reviewer_WDvd", "summary": "This paper reformulates the problem of stabilizing training in residual-free transformers as minimizing the condition number of the network Jacobian. It proposes a principled parameter initialization method that improves the conditioning of the self-attention layer. The analysis shows that the Jacobian’s conditioning is dominated by two key matrix products: $W^Q W^{K\\top}$ and $W^V W^O$ and that initializing $W^V W^O$ to be scaled-orthonormal and enforcing diagonal dominance in $W^Q W^{K\\top}$ can significantly improve conditioning of self-attention layer Jacobian and thus enhance training stability.\nExperiments on both supervised learning (ImageNet classification with ViT-Base) and self-supervised learning (DINO feature pretraining with ViT-Small) demonstrate that this initialization enables stable optimization of skipless transformers. The method is compatible well with SOAP optimizer and achieves comparable or better performance than residual-based counterparts. Visualization of the final-layer features  shows that skipless models trained with the proposed initialization yield clearer class boundaries.", "questions": "1. **Optimizer inconsistency: SOAP > AdamW in supervised, but Adam > SOAP in self-supervised**. In the supervised ImageNet setting with ViT-Base, skipless+init trained with a second-order method (SOAP) reaches parity/slight gains vs residual baselines , whereas in DINO pretraining with ViT-Small the best results appear under Adam rather than SOAP (Fig. 2 in the paper). Please analyze why SOAP’s advantage does not carry over to DINO.\n2. **Limited theoretical robustness beyond initialization.** Could you track $\\kappa(J)$, per-layer $\\kappa(K_\\ell)$ over epochs for both skipless+init and residual baselines? \n3. **More evidence for hierarchical/compositional representations**. Author shows clearer class boundaries from late-layer features in skipless+init models and gains on dense probing/object discovery  , but direct, layer-wise evidence of hierarchy/compositionality is limited. Please provide more evidence for hierarchy/compositionality.\n4. **More model scale diversity**. Could you provide the results of ViT-Large and ViT-huge to evaluate scalability of proposed method. (Consider demands on computing resources, it's listed as optional. )\n5. **Is minimizing the Jacobian condition number sufficient for stability?** Skipless+init converges to accuracy similar to residual models, but residual training shows smoother loss curves(Figure 1), suggesting potential landscape differences not captured by condition-number alone. If so, will such difference cause instability? (Although this analysis would be valuable and necessary for complete understanding, it falls beyond the scope of this paper. Therefore, it is listed as an optional question.)\n\n Authors’ rebuttal adequately addressing the first three points is necessary for me to raise the final score.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper has clear writing and is easy to follow.\n2. This paper introduces a novel principled initialization method that is effective on alleviating  instability in residual-free transformer training.\n3. This paper conducted experiments on multiple tasks including image-classification and DINO feature pertaining.", "weaknesses": "1. **Limited theoretical robustness beyond initialization**. While the principled initialization helps improve conditioning at the beginning of training, but the paper provides no guarantee that good conditioning persists as parameters evolve during training. In contrast, skip connections offer a structural and long-term stabilizer independent of parameter drift.\n2. **Simplified input distribution assumption**. The theoretical derivation(Appendix A4.1) assumes token embeddings are drawn from  a normal distribution with mean 0 and identity covariance, which deviates from the empirical distribution after LayerNorm or embedding layers where the tokens are correlated across dimensions and not centered at 0.\n3. **Indirect evidence for hierarchical representation**. While visualizations suggest clearer feature boundaries and clean feature maps, no direct qualitative and quantitative evidence is provided to show skipless ViT yields more hierarchical or compositional features than residual ones. Given it’s considered as one strong benefit of skipless ViT. More evidence is supposed to be provided.\n4. **Limited model scale diversity**. The experimental validation involves only ViT-Base (supervised) and ViT-Small (self-supervised). Without testing larger models (e.g., ViT-Large or ViT-Huge) and different model size on the same task, it remains unclear whether the initialization scales effectively with increasing parameter count.\n5. **Material Missing**. In 5.2, Appendix E is mentioned but is missing in provided materials.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H31haQrpte", "reviewer": "Reviewer_RZmz", "summary": "This paper investigates training ViTs without residual connections. The authors provide a Jacobian-based theoretical analysis showing that skip connections improve network conditioning, and propose a principled initialization strategy that enables stable training of skipless transformers without architectural modifications. The key contributions include: 1. theoretical analysis of transformer Jacobian conditioning; 2. a novel initialization scheme; and 3. empirical validation showing skipless ViTs can match residual models in supervised learning and outperform them in dense prediction tasks.", "questions": "1. Can you provide more experiments on LLM?\n2. Can you ablate the specific properties of SOAP that enable skipless training?\n3. Do the benefits hold at larger scales?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Principled Theoretical Foundation: The authors provide some mathematical insights on the Jacobian analysis, which provides clear intuition for why skip connections help optimization.\n2. This approach requires no architectural modifications.\n3. The authors provide thoughtful experiments.", "weaknesses": "1. The restriction to Vision Transformers is a significant limitation.\n2. The improvements are minor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NGcO15w15s", "reviewer": "Reviewer_rJvD", "summary": "This paper proposes an initialization to train a transformer model without skip connection. The initialization strategy is developed based on a theoretical analysis of the transformer Jacobian, which reveals that skip connections stabilize optimization by improving Jacobian conditioning. To replicate this stabilization effect without skip connections, the paper initializes the product of the projection matrices to ensure the good conditioning of attention matrix.  Results on ViT show that, when combined with SOAP optimizer, skipless models achieve comparable convergence speed and accuracy to residual-based models in supervised image classification task, and outperform residual-based models in self-supervised dense prediction tasks.", "questions": "- Why does an abrupt inflection point appear in Figure 1? The loss is obviously not smooth.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The overall paper is well-written and easy to follow.\n- The proposed initialization scheme requires no architectural modifications.\n- The experimental validation is comprehensive, covering both supervised and self-supervised learning settings on ViTs. And the result aligns with the analysis and explanation.", "weaknesses": "- How to demonstrate \"learning richer, more semantically coherent internal representations\" is unclear. Additionally, the significance of training skipless transformers remains ambiguous.\n- Regarding experiments across different scales, I am curious whether this method is only effective for Vision Transformers (ViTs) or applicable to transformer models with general architectures. This is because the analysis in the paper has little relevance to vision-specific characteristics.\n- The paper only explores which type of initialization can lead to better conditioning of the network Jacobian, but it does not essentially guarantee better model performance.\n\nOverall, this work represents a valuable initial attempt. However, for different transformer architectures, including those with various subsequently proposed activation functions, normalization layers, and MoE structures, the applicability and generalizability of this initialization method still require further experimental validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)