Analyzing the Training Dynamics of Image Restoration Transformers: A Revisit to Layer Normalization
Authors: MinKyu Lee
Sangeek Hyun
Woojin Jun
Hyunjun Kim
Jiwoo Chung
Jae-Pil Heo
Venue: ICLR 2026 Poster (2026)
Keywords: Low-level vision
image restoration
network architecture
normalization
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: This work reveals and analyzes extreme feature statistics in image restoration transformers raised by LayerNorm, and provides a simple drop-in replacement.

Abstract: This work analyzes the training dynamics of Image Restoration (IR) Transformers and uncovers a critical yet overlooked issue: conventional LayerNorm (LN) drives feature magnitudes to diverge to a million scale and collapses channel-wise entropy. We analyze this in the perspective of networks attempting to bypass LayerNorm’s constraints, which conflict with IR tasks. Accordingly, we address two misalignments: 1) per-token normalization that disrupts spatial correlations, and 2) input-independent scaling that discards input-specific statistics. To address this, we propose Image Restoration Transformer Tailored Layer Normalization (i-LN), a simple drop-in replacement that normalizes features holistically and adaptively rescales them per input. We provide theoretical insights and empirical evidence that this design effectively captures important spatial correlations and better preserves input-specific statistics throughout the network. Experimental results verify that the proposed i-LN consistently outperforms vanilla LN on various IR tasks.

Published as a conference paper at ICLR 2026
ANALYZING THE TRAINING DYNAMICS OF IMAGE RESTORATION TRANSFORMERS:
A REVISIT TO LAYER NORMALIZATION
MinKyu Lee, Sangeek Hyun, Woojin Jun, Hyunjun Kim, Jiwoo Chung, Jac-Pil Heo' Sungkyunkwan University (2minkyulee, hse1032, junwoojinjin, arithu3, jiwoo.gg. jaepilheo) @ gmail.com http://giiub..oommmmiikkylllllil
ABSTRACT
This work analyzes the training dynamics of Image Restoration (IR) Transform- ers and uncovers a critical yet overlooked issue: conventional LayerNorm (LN) drives feature magnitudes to diverge to million scale and collapses channel-wise entropy. We analyze this in the perspective networks attempting to bypass LN's constraints that conflict with IR tasks. Accordingly, we address two misalignments between LN and IR: 1) per-token normalization disrupts spatial correlations, and scaling discards input-specine statistics. To address this, we propose Image Restoration Transformer Tailored Layer Normalization (i-LN), a simple drop-in replacement that normalizes features holistically and adaptively rescales them per input. We provide theoretical insights and empirical evidence that this simple design effectively leads to both improved training dynamics and thereby improved performance, validated by extensive experiments.
2) poput-independeed
INTRODUCTION
Image restoration (IR) aims to reconstruct high-quality images from degraded inputs. With the success of Vision Transformers (Dosovitskiy et al.) (2020), Transformer-bart based architectures coupled with LayerNorm (LN) have been actively adopted for IR tasks and are now a common standard (Liang al. 202T Chen et al., 2023a Hsu et al. 2024). However, despite recent architectural advances, the underlying training dynamics of IR Transformers remain underexplored.
This inspires to take closer look their internal behavior, leading us uncover a critical yet overlooked phenomenon: feature magnitudes diverge dramatically, reaching scales up to a million, while channel-wis wise feature entropy drops sharply (Fig Tp. Interestingly, this phenomenon aligns with previous studies (Karras et al., [2020] Wang et al.) (022a)) which similarly observed visual artifacts and abnormal features when coupled with specific normalization layers. However, discussions specific to the unique requirements of IR tasks and IR Transformers were not made.
Building on these insights, we hypothesize that the observed feature divergence in IR Transformers arises from networks attempting to circumvent LN, due to constraints of LN that do not align with the unique requirements of IR tasks. Accordingly, we identify two key mismatches between LayerNorm and IR tasks; supported by both theoretical insights and extensive empirical analysis.
First, LayerNorm operates a per-token manner, without considering inter-pixel (token) relationships. This disrupts the spatial correlations in input features, an aspect crucial for high-fidelity image restoration. Second, it maps intermediate features into a unified normalized space, limiting the range flexibility of internal representations. This thereby disregards the input-dependent statistical variability (Lim et al.) (0177) that is inherent in IR tasks. Together, these mismatches significantly hinder IR Transformer's ability to accurately preserve low-level features throughout the network, which is necessary for faithful image restoration.
While one intuitive solution could be the complete removal of normalization layers prior works have done Lim et al. 2017b] Wang et al. 2018 Karras et al. 2020 2024), our experimental observations
Coorrsponning Author.

---

Published as a conference paper at ICLR 2026
1=7
Lay LayerNorm
Feature Scale
467
3e7
Durs
Entroyy
Entropy
LayarNorm
167
0e0
300
Layorkormy
200
200
Layer
100
Layer
100
Iterations (KI
15 20
(a) Feature Magnitudes
(b) Channel-wise Entropy
(c) Entropy close-up
Figure 1: Visualization of feature magnitudes and channel-wise entropy during training of an Image Restoration (IR) Transformer using conventional LayerNorm (LN) and i-LN (Ours). (a) Evolution of feature magnitudes across layers and training iterations, highlighting the dramatic divergence (to million-scall scale) under conventional LN. (b-e) Channel-wiss entropy with LN drops sharply at the very early stage of training, indicating the emergence of acute peaks hidden in specific channels. Ours i-LN exhibits well-distributed activation across channels and significant stability.
highlight significant training instability when normalization is entirely omitted from IR Transformers [ the network fails to converge.
In this work, we show that these issues can be addressed in surprisingly simple manner; leading to significant stability and substantial performance gain. We propose the Image Restoration Transformer Tailored Layer Normalization (i-L.N), which acts as drop-in replacement to conventional (vanilla) LayerNorm by better aligning with the unique requirements of IR tasks.
Instead of normalizing each token independently, we propose to apply normalization across the entire spatio-channel dimension within IR Transformers 3), effectively preserving spatial correlation- ships among tokens, contrary to vanilla per-token LayerNorm. Furthermore, We rescale features with the normalization parameters after each attention and feed-forward layer, explicitly enabling range flexibility and accounting for input-dependent variations in internal feature statistics. Together, these modifications effectively preserve low-level feature statistics throughout the network, better aligning with the requirements of IR tasks.
Extensive experiments show that i-LN leads both stable training dynamics with improved perfor- mance across various IR benchmarks. Additionally, we observe cues suggesting robustness under anduced-precision configurations, RealWorld IR tasks, and improved spatial correlation modeling.
REVISITING LAYER NORMALIZATION IN IR TRANSFORMERS
Observation (Abnormal Feature Statistics). Our initial analysis focuses tracking the trajectory of internal features during the training of IR Transformers. We visualize the squared mean of
intermediate features at each basic building block of the network, following (Karras al. 2024). We
select the x4 SR task using the HAT (Chnn et al.) 2023a) model as the representative IR task (Fig [
The analysis reveals that feature statistics diverge dramatically, reaching values up to million scale. To pinpoint the origins of this feature divergence, we analyze the feature entropy across the channel-axis. Analysis demonstrates sharp decrease in feature entropy, which indicates the presence of channels with extreme values that dominate the statistics. Since these extreme values are unusual, this motivates us further investigate. Accordingly, we analyze the training dynamics across configurations by varying the network scale (Fig 22 2b). varying the IR tasks (Fig 2c). and varying the normalization scheme (Fig 4); and observe that this phenomenon occurs across all configurations utilizing standard IR Transformers. While this type of hidden abnormal behavior aligns with the observations in prior studies (Karras et al. 22200) Wang et al. 2018 2022a). further discussion did not gain much attention, especially regarding the unique properties and requirements of IR Transformers.
In the following, we provide further insights into this phenomenon by examining the characteristics of LayerNorm (LN), the de facto normalization in IR Transformers. We start by defining the spatial relationship between pixels (i.e.. inter miter- structure), and further show that conventional LayerNorm cannot preserve this. For simplicity, we neglect the affine parameters for theoretical analysis.
2

---

Published as a conference paper at ICLR 2026
Max: 5.Be+06
Max: 7.6e+07
10"
10
10'
Basellne (LN)
DR-Swin
10"
DN--wwnn
CAR-Hat
10
10
DR-Hat
CAR-Swin
10
10
10"
10
10
Ours (i-LN)
DN-Swin
DR-Swin
CAR-Hat
10
10"
10.
DR-Hat
CAR-Swin
10'
2 blocks blocks
1 block
blocks blocks
10²
24
48
96
120
10"
blocks
0 0K
10'
100K
200K
300K
100K
200K
300K
0K
100K
200K
300K
Training Iterations
Training Iterations
Training Iterations
(a) Network Depth
(b) Network Width
(c) Image Restoration Task
Figure 2: Feature magnitude evolution in IR Transformers across different settings. (a-b) Feature divergence signifies the network scales. (c) Feature divergence appears across various Transformer backbones and IR tasks: super-resolution (SR), denoising (DN), deraining (DR), JPEG compression artifact removal (CAR), demonstrating that this phenomenon widespread. It can be effectively mitigated by simply replacing conventional LayerNorm with the proposed i-LN.
Definition 1 (Inter-pixe Structure and Preservation). Let x E RLLCC be a featuue map with L tokens. We write the f-th token as Ze B.C and the e-th element of as Bt,e E TR. The niter-pixel structure of feature map is given by the set of relative differences Ar == {2e 24 S f, L).
Definition 2 (Structure Preserving Transformation). A transformation T is said to preserve inter-pixel structure up to scale if there exists a homothety H (x) a. + with a > and IRC, succh that
T(x;) - T(xx) = H(x; - xx) a( xx) for all f, k.
Such maps preserve all angles and pairwise distance ratios, and correspond to a single global shift and uniform scaling across all tokens. For = 1, T is said to preserve structure absolutely.
Intuitively, consider x as a point cloud in RC, where each point represents a token. A structure- preserving transformation may only uniformly scale and shift the entire cloud. That is, the overall shape of the point cloud should be preserved up to a single global scaling factor and translation.
Vanilla Per-token LayerNorm (Baseline). Conventional Transformer architectures utilize the per- token LayerNorm (LN) as the de facto normalization scheme which operates as follows:
LN(x)) = Vo?+e (x2 (xe pe) B, Jce [x(r,e,),
of R. ne)
(1)
where Ea[-] is taken over the channel dimension c, and 7, B E IR are each affine parameters applied after the normalization step, and LN operates for each token de given the entire input feature
Proposition 1. (Vanilla LayerNorm fails to preserve structure). Let TLn be the normalization in vanilla per-token LN. Then, in general, there do not exist a. > O and an orthogonal Q such that
Tix(xz) (=)(
Q (xe F&) for all xe; Tk.
Thus TIn is not even conformal on the token set. Since homotheties are strict subclasses of conformal maps, Tin is not a homothety and therefore does not preserve inter-pixel structure in general.
The exception arises in degenerate cases where all tokens share identical per-token mean and variance, in which case similarity map can exist (i.e., inter-pixel structure is preserved), Such cases are extremely rare in practice. Our intuition is that since LN cannot naturally preserve inter- pixel structure, networks learn to generate large magnitude features regardless of the input, thereby, manipulate the overall feature statistic to behave similarly to this exceptional degenerate scenario.
Remark.
Inspired by prior observations, hypothesize that feature divergence arises from fundamental mismatch between the requirements of IR tasks and the constraints imposed by LayerNorm, leading us propose tailored normalization scheme that aligns with the unique requirements of IR tasks.
3

---

Published as a conference paper at ICLR 2026
Per-token normalization stats.
Spatially holistic normalization stats.
Input-adaptive scaling
11
(1...
Axis [c]
Atin FFN
Axis [L,c]
Ath FFN
Affine
Affine
(a) Vanilla Image Restoration Transformer (LN)
(b) Our Image Restoration Transformer (i-LN)
Figure 3: Comparison between IR Transformer blocks using conventional per-token LayerNorm (LN) and our proposed i-LN. Contrary to conventional LN, which normalizes each token independently. our i-LN applies holistic normalization across the entire spatio-channel dimension, preserving essential spatial correlations between tokens. Additionally, i-LN auput-adaptively rescales features after the attention (Att)) and feedforward (FFN) layers, thereby better preserving input statistics and allowing feature range flexibilityy. These together enable IR Transformers to preserve low-level characteristics of input throughout the network, aligning with the unique requirements
METHOD: TAILORING LAYERNORM FOR IR TRANSFORMERS
Spatial Holisticness in Normalization (LN*). We propose simple variant of LN that improves in preserving inter-token spatial relationships of input features, which we refer to as LN*, Instead of normalizing each token individually as LN, we derive normalization statisties from the entire spatio-channel dimension of the input feature as follows:
LN (x) =7 Vo² +c
(x (-- +
= Ee,erre,
=² Eeo -1)))
(2)
where the expectation Er,e[-] is taken over both spatial (f) and channel dimensions (c). This straightfor- ward modification effectively mitigates the issue raised by the per-token operation in vanilla per-token LayerNorm. While normalization methods in CNNs already inherently work in spatially holistic manner, the implications of such holisticness in normalization and the corresponding spatial structure corruption without it have received little attention, particularly in the context of IR Transformers. With this point, the following section aims to provide further intuition and establish connections between holisticness and spatial structure (i.e., inter-pixel structure) preservation.
Proposition 2. (LN* preserves structure). Let Tex be the normalization defined by LN with global mean y and std. >> O computed over all tokens and channels. Then for any two tokens Iı, Tk.
TIx- (zz) Tix- (Ix) (1/a)(zz (z))
Thus, Tix is a homothety, and accordingly, preserves spatial structure up to a global scale.
Remark. In short, LN is pruucture-preservie up to one missing scalar (i.e., the global scale). We handle this loss of information by explicitly reintroducing it later, described below.
Preserving Input Dependent Statistics. We further tailor the normalization operator to better suit the requirements of IR tasks. Specifically, we address the issue of input-blind normalization. While IR tasks require the preservation of input-dependent feature statistics for faithful reconstruction, both conventional LayerNorm and even the holistic LN* overlooks this aspect by mapping features into unified normalized space. Although normalization improves training stability, it also causes the model lose critical input-dependent information (i.e., the missing global scale term of inter-pixel structure) by restricting the range flexibility of internal representations (Lim et al.) 20176).
Accordingly, we propose a simple input- adaptive rescaling strategy that effectively tackles this issue. We rescale the output of Attention and FFN by their standard deviation computed in the preceding normalization process as the yellow line Fig 3p, which we refer to as i-LN. Accordingly, a typical Attention or FEN block B could be further improved by coupling with i-LN as follows:
B(x (,LLN o +c. (LN"(())
(3)
4

---

Published as a conference paper at ICLR 2026
Table Comparison between var- ious normalization schemes. i in- dicates that BatchNorm is evaluated in train-mode. SH indicates the spa- tial holisticness of the normaliza- tion scheme, including the setting without any normalization (None). Experiments are performed for x4 .1866
S4
BSD100
Urbum100 100
Mangal08
Method
PSNK
SSIM
PSNR
SSSIM
PSNR
[SSTM]
PSNK
SSIM
LayerNorm
x
28.79
.7876
27.68
7711
26.55
.8015
31.01
.9150
LayerScale
28.89
.7887
27.76
.7426
26.75
.8058
31.37
.9178
RMSNorm
x
28.88
.7879
27.74
77117
26.67
.8037
31..4
.9165
ReZero
28.81
.7861
26.41
.7964
31.05
.147
None
InstaneeNorm
28.98
.7077
27.80
.4455
27.02
.8136
31.46
.199
BatchNorm' !
28.95
.7901
27.80
7442
26.70
.8123
31.39
SR with HAT, The best result for each setting is highlighted in bold.
(-LN (Ours)
29.01
.7918
27.84
.7456
27.17
.8167
31.82
.9228
where I' is either the according Attention FFN operation of block B. Overall, this reintroduces the original feature statistic lost due to normalization. This simple strategy enables IR Transformers to better preserve the per-instance statistics throughout the network and allows range flexibility to intermediate features. We later show that this leads to an order of magnitude more stable feature distribution (i.e., higher entropy) and overall improved IR performance.
Remark. This simple input-adaptive rescaling strategy explicitly reintroduces the missing global scaling term that LN* could not preserve (i.e.. the restricted range flexibility issue).
4 EXPERIMENTS
Training Settings. Since recent works have discrepancies in their detailed training settings (Chen et al., 2024), reimplement baseline methods and our method under identical settings for fair comparison. Networks for deraining (DR) were trained on Rain 3K (Jiang et al., 2020), while DF2K (DIV2K (Agustsson & Timofte 2017) Flickr2K (Lim et al. 2017a)) was used for other tasks. Only basic augmentations (random flips, rotations, crops) were applied, without mixing augmentations, progressive patch sizing, warm-start. SwinIR (Liang al. 202T), HAT (Chen al., 2023a), and DRCT (Hsu et al., (024) were used as backbones. Configurations for each backbone model (c.g.. model size, batch size, and patch size) are adjusted meet computational constraints and are denoted by subscripts (e.g.. HAT, 2AT2). The full specifications are provided in Appendix 5
Evaluation Settings. Standard benchmarks are employed including: Set5 (Bevilacqua et al. 2012). Set14 (Zeyde et al. 2010), BSD100 (Martin et al. 2001), Urban (Huang et al., 2015).
Mangal 10 (Matsui et al. (017) for SR: CBSD68 (Martin et al.) (001), Kodak (Franzen, 1999).
McMaster (Zhang et al. 1201)). Urban 100 for DN; LIVEI (Shiikh 2005). Classic5 (Foi et al.) 2007
Urban 100 (Huang al. (2115) for CAR; Test100 (Zhagg et al. 2019) and Rain1001 (Yang et al.
2017) for DR. We crop Urban100 into non-overlapping 256x256 patches due to memory limits for
CAR and DN. We report PSNR and SSIM indices. Experiments were performed on NVIDIA A6000s.
4.1 NORMALIZATION SCHEME VARIATION
We analyze the effects of various normalization techniques, in- cluding representative normalization schemes as vanilla Layer- Norm (LN) (Ba et al.) 20166) per-token RMSNorm (RMS) Zhang
10"
None Fall
10
& Sennrich 2019). InstanceNorm (IN) (Ulyanov et al. 2016). Batch-
10
Norm (BN) (loff & Szegedy 2015). and our proposed i-LN, Con-
IN
None
10
sidering previous studies where completely removing normalization from SR networks (Lim et al.) (2017) Wang et al.) 2018) led to perfor- mance improvements, we additionally tested a similar configuration indicated as None, where normalizations are entirely removed.
10
1
H-N (ours)
OK
100K
200K
300K
Further, we investigate the empirical impacts of recent methods de- signed to stabilize Transformer training: ReZero (RZ) (Bachlechner et al. 2 and LayerScale (LS) (Touvron et al. 1021). ReZero removes LayerNorm from Transformer blocks and multiplies learn- able arro-initialized scalar to the residual path. Similarly, LayerScale multiplies near-zen coan-zero-initaalee initialized learnable diagonal matrix to the residual path but reintroduces LayerNorm. Since both methods ini- tially multiply (near) zero-scale factor to the network output, we consider them as potential solutions
Training Iterations
Figure 4: Feature divergence across various normalizations.
5

---

Published as a conference paper at ICLR 2026
Table 2: Quantitative comparison between the conventional LayerNorm (LN) and our proposed i-LN across diverse IR tasks. The best result for each setting is highlighted in bold.
Backbone
Scale
PSNR
SSIM
PSNR
Set14
ISSIM
PSNR
BSDIOO
[SSIM
PSNR
Urban100
SSIM
Mangal09 PSNR
SSIM
Backbone
Testset
PSNR
Metric
SSIM
HAT,
38.14
.610
33.78
.9196
.9000
52.16
.9297
.9788
HAT, LN
R0in
34.35
.971
HAT + 1-NN
x2
38.37
.6619
34.08
.9218
32,42
.9028
33.32
.9385
39.69
.9744
HAT, + +1IN
36.20
.6411
DRCT, +LN
X2
38.19
9613
33.28
.9197
32.28
.9010
32.60
.9323
39.23
.9785
SwinlR; + LN
Rain
100L
33.00
.9334
DRCT; +
38.23
.9614
33.86
.9206
32.31
.9014
32.79
.9344
39.40
.9788
SwinlR; LLLN
34.43
.9527
1AT, + +LN
x4
32.51
.8992
28.79
.7876
27.68
.7411
26.55
8015
31.01
.9150
[IXT, + +LN
Test100
29.52
.8905
HAT +ILN
32.72
.9019
.7915
.7456
27.17
.8167
31.82
.9288
HAT, +-LN
30.14
.9022
DRCT1 LN
x4
32,50
8989
28,
85
77871
27.73
7414
26.63
8021
31.24
.9199
SwinlR, LN
Test100
27.45
8766
DRCT, + 5-LN
x4
32.57
.9977
28.91
.7887
27.76
.7426
26.79
.8063
31.41
.9188
SwinlR; S-LN
29.87
.8982
(b) Image deraining (DR)
(a) Single image super-resolution (SR)
Urban100 PSNR
CBSD68 PSNK
Kodak24 PSNR
McMaster
Urban100
LIVEI PSNR
Classic5
Backbone
PSNR
Backbone
4
PSNR
SSIM
SSIM
PSNR
SSIM
HAT +LN
15
35,489
34,285
35.347
35,440
HAT, + LN
10
28.45
8514
27.89
8808
29.94
8167
HAT, i-IN
15
35,558
34,296
35,366
35,477
HAT + --NN
10
28.52
.8530
27.90
.8057
29.96
.8178
SwinlR: +TN
15
35,077
34. 164
35.
38.
SwiniR: +TN
10
27.86
.400
27.65
.7995
29.72
.8111
SwinlR; + +-N
15
35,138
34,181
35.177
35,223
SwinlR: i-LN
10
27.92
.8410
27.62
.7933
29.72
8111
HAT +LN
25
33,296
31.622
32,864
33,105
HATT + LN
40
35,26
93002
32.63
9158
34.34
9060
HAT: i-LN
25
33.384
31,632
32.887
33.139
HAT +-LN
40
33.36
.9312
32,67
.9162
34.39
.9666
SwinlR. + LN
23
32,753
31.480
32.643
32.829
SwinlR; LN
40
32.62
9245
32,34
9127
34.11
9036
SwinIR, + s-LN
25
32.803
31.489
32,660
32,848
SwinlR; +LLN
32,68
.9252
32.35
.9229
34.12
(c) Color image denoising (DN)
(d) Image JPEG compression artifact removal (CAR)
to resolve the feature increasing issue in IR tasks. Notably, these methods also align with prior stud- ies (Lim et al. 2017b Wang et al. 2018), where multiplying small scale factor to the residual path components helped the network to converge. Overall, this study aims to explore I) the feature divergence tendency of per-token and holistic normalizations and 2) determine which normalization method yields the best performance.
Feature Divergence Behavior. Fig 4 illustrates that feature divergence always emerges when using per-token normalizations: vanilla LN, RMSNorm, and LayerScale. In contrast, spatially consistent normalizations as our i-LN or BN, IN, ReZero do not exhibit the divergence trend. For the configura- tion without any normalization, we observe failure to converge due to unstable training. However, the feature magnitudes are schemes without the per-token operation. This observation also aligns with our hypothesis that the feature divergence phenomena is closely related to the per-token operation, and also reveals that any spatially holistic normalization could potentially reduce this effect.
before this failure occurs, aligning with other normalization
Performance Comparisons. We further analyze the empirical performance for each normalization scheme in Ta where vanilla LN performs worst since it neglects inter-token relationships and maps features into a unified normalized space, neglecting input-dependent low-level statistics.
Per-token variants (LS, RMS) improve over vanilla LN, but underperform methods that use spatially consistent normalization. Without normaliza- tion (None). the network fails converge, likely due unstable gradients caused by the absence of normalization. RZ, which also omits normal- ization, similarly yields limited performance.
25
(Gs) eiiii aivisd
None
The spatially holistic variants (IN, BN) outperform those with per-token schemes (LN, LS, RMS) or normalization omitted schemes (None, RZ). However, BN leads to a significant performance drop in eval-mode, de- spite being healthy in train-mode; indicating the necessity of per-image statistic- normalization in IR tasks. Mcanwhile, IN performs better than vanilla LN but worse than ours. This is since IN (and also BN) discard crucial channel-wise information necessary for representing deep features, resulting in limited performance.
10
100K
200K
300K
Training llerations
Figure 5: Eval-mode BN and removing all normal- ization (None) fails.
Overall, our i-LN achieves the best performance among all examined methods, demonstratigg its effectiveness in preserving important inter-token spatial relationships and internal statistics, and ultimately the input low-level features throughout the network.

---

Published as a conference paper at ICLR 2026
IAN
HAT
DRCT
SwinlR
HQ
HQ crop
Beseline(LN)
Ours ((-LN)
HQ
HQ crop
Baseline( .N)
Ours (I-LN)
(a) Qualitative results super-resolution.
(b) Qualitative results for image deraining.
HAT
HAT
SwinTR
SwinIR
HQ
HQ crop
Baseline(LN)
((-LN)
crop
Baseline(LN)
Ours (I-LN)
(c) Qualitative results for denoising.
(d) Qualitative results for JPEG artifact removal.
Figure 6: Qualitative comparison across four representative image restoration tasks.
4.2 ANALYSIS UNDER TASK VARIATION
Feature Divergence Behavior: Fig. 2c illustrates the evolution of feature magnitudes across various Image Restoration (IR) tasks, including Image Repputtisooltion (SR), Image Denoising (DN), Image Deraining (DR), and JPEG Compression Artifact Removal (CAR). The figure clearly demonstrates that feature divergence consistently occurs across all restoration tasks under conventional LayerNorm. In contrast, integrating our i-LN effectively resolves this issue, maintaining stable and well-bounded feature scales throughout training. This consistent stabilization of internal feature magnitudes confirms the general applicability and robustness of our proposed method across diverse IR scenarios.
Benchmark: Image SusortResolution (SR). Tab and Fig 6 illustrate quantitative and qualitative results for SR. Compared to vanilla LayerNorm, we achieve significant improvements across bench- marks. Notably, SR benefits greatly from our method due the inherent nature of SR: the input is entirely reliable, since it exactly aligns with the low-frequency information in the ground truth. By precisely preserving these input features, our method substantially enhances restored image quality. We additionally provide comparison against the official public models under computationally extensive settings in Appendix B.I
Image Deraining (DR). Similarly, Tab and Fig. 6p demonstrate substantial improve- ments of our method in image deraining compared to conventional LayerNorm. This improvement particularly pronounced because our method effectively preserves reliable input regions, specifically the local areas unaffected by rain streaks. By explicitly maintaining these local correspondences with the ground truth, our i-LN method achieves improved restoration accuracy.
Benchmark:
Benchmark: Image Denoising (DN) Tab. 2 and Fig. O demonstrate that our method consistently outperforms conventional LayerNorm in image denoising tasks. However, the observed performance improvements are smaller compared to SR and Deraining. This relatively reduced benefit arises because denoising involves uniformly distributed corruptions across the entire image, limiting the advantage gained from explicitly preserving particular input features. Despite this, visual examples confirm meaningful improvements in recovering sharp edges.
Benchmark: JPEG compression artifact removal (CAR). Similarly, Tab. 21 and Fig. 6 demon- strate consistent improvements of our method over LayerNorm for JPEG compression artifact removal. However, these performance gains remain sma ller than those achieved in SR and Deraining. Similar denoising, JPEG artifacts affect images globally and irregularly, limiting the advantage of explicitly preserving specific input details. Still, visual examples illustrate consistent improvement in accurate artifact reductions, highlighting our method's 's broad effectiveness across various IR tasks.
Real-world Image Restoration. To further validate the effectiveness and robustness of the proposed method, we conduct further experiments under the challenging real-world degradation configurations. Here, we choose the representative Real-ESRGAA GAN Wang al. (2021) degradation pipeline and synthesize both the train and test images accordingly. Experiments are performed under the x4 SR task with the HAT; model. In Fig, and Tab. 13, provide qualitative and quantitative results, respectively. As demonstrated, our i- i-LN shows significant improvements even under the complex real-world degradation settings, successfully reconstructing fine-details and sharp edges.
Benchmark:
7

---

Published as a conference paper at ICLR 2026
4.3 ABLATION STUDY
To analyze the contribution of each component in i-LN, we conduct an ablation study by selectively removing spatial holisticness and rescaling. Compared Tabī] we increase the network capacity and training iterations (denoted as HAT2) to ensure that the observed benefits are not simply due to faster convergence. and [2 we provide a quantitative and qualitative analysis results under the X SR task. Removing either the rescaling strategy (Rs) or the spatial holisticness (SH) consistently reduces restoration quality, confirming their complementary roles in improving IR performance.
10'
FUN
Entropy
10
LN"
LN
10
Tab. 3 Fig 13
-2 10
i-LN
LN Configuration
LN
Figure 7: Channel entropy collapses exponentially as we remove each compo- nent (spatially holisticness and rescaling) of our i-LN; falling back to vanilla LN.
We then examine feature statistics in terms of channel entropy (Fig. 7 Starting from our full method, we remove the rescaling method (falling back LN*) and subsequently also remove the spatial holistic scheme (falling back to vanilla per-token LN). Here, we observe that chan- nel entropy collapses exponentially, indicating that each component contributes to maintaining well-distributed activations across channels.
Overall, using both components together achieves the best results in terms of both restoration quality and stable feature statistics. Spatial holisticness (LN*) effectively preserves inter-token relationships, while the rescaling strategy further restores the missing global scale that LN* alone cannot maintain.
4.4 INTRIGUING PROPERTIES UNDER VANILLA LN
4.4.1 HOW NETWORK SCALE IMPACTS FEATURE DIVERGENCE
We further investigate how the overall network size affects feature divergence by varying the depth and width of the IR Transformer individually. As shown in Fig. 2a- 2b
larger models consistently diverge faster and to higher magnitudes. In particular, the emergence of an extreme valued feature appears be a cumulative process: in order for a newly generated outlier channel dominate the statistics, it must surpass the already abnormal activations propagated through the residual path, resulting in increasingly extreme values as the network scales. Taken together, our analysis reveals a potential vulnerability unique to low-level restoration at scale, where enlarging capacity does not merely amplify representational power but also exacerbates pathological feature growth.
4.4.2 CHANNEL IMBALANCE AND BIAS ALIGNMENT
Le4
SR
le5
JPEG
Earlier, we observed extreme feature norms and imbalances in channel entropy, indicating highly peaky feature distribu- tions concentrated in specific channels. Interestingly, despite these severe imbalances, baseline IR Transformers manage to converge and produce outputs with well-bounded magnitudes. To gain insights to this paradox. we take a closer look at the final normalization layer (LN). In Fig. we visualize the un- normalized feature magnitudes along the channel dimension before the final normalization layer and compare them with the learned affine bias parameter (3) across various IR tasks.
Mean
-5
le-1
Affine Bias
-5
Channel Index
Channel Index
Figure 8: Alignment of affine bias pa- rameters in the last LN and channel- wise magnitude of input feature; showing a compensatory mechanism.
We observe sharp peaks in the bias parameters precisely align- ing with the channels exhibiting high magnitudes. This exact alignment reveals a compensatory mech- anism where the learnable affine terms (7, 8)) of LayerNorm counteract abnormal channel activations, allowing baselines to yield normal images. Additionally, this also indicates that the normalization operation (μt, or) itself is incapable of directly removing these extreme peaks.
Moreover, although the observed bias-feature alignment allows baseline IR Transformers to maintain reasonable outputs, this mechanism should be regarded as a compensatory shortcut rather than a fundamental fix. The fact that networks must rely on such peaky biases to counteract extreme channel activations leaves the model fragile and prone to failures, including potential training instability and failure in practical scenarios such reduced- inference discussed in See 4.5.2
8

---

Published as a conference paper at ICLR 2026
S00K 100K
100K
(a) Baseline (I.N)
(b) Ours (i-LN)
(a) HR
(b) LN
(c) i-LN
Figure 9: Visualization of Relative Position Embeddings (RPE) per head, for training iteration 100K and 500K. Ours exhibit well-structured RPEs, indicating the superiority in understanding the spatial relationship between pixels.
Figure 10: Half-precision inference sults for x4 SR. LN leads to artifacts while i-LN achieves near-zero fi- delity loss compared to full- pull-precision.
1dx
Backbone
SH
Rs
BSD100
Urban100
Manga 109
Idx
Backbone
Quantization
Urban 100
Manga109
1
HAT2 (LN)
27,7897
26,8779
31,3444
HAI +
W
ints
26.8711
31.5266
HAll
27,8615
27.3373
31.8888
2
HATz i-N
W
int8
27.5818
32.1657
3
HAT
27.9034
27.5335
32,0837
3
HAT: + LN
W
in14
25.0242
28.0831
4
HAT, (6-LN)
<
27.9206 27.5849
32.1694
HAT +
5
HAI LN
W4F
1p16
7,4640
5,0736
Table 3: Ablation study. SH indicates introduc- ing spatial holisticness (identical LN*) and Rs indicates our rescaling strategy. Idx I and 4 each identical to vanilla LN and our i-LN, respectively. Experiments conducted for x4 SR.
HAT, + +1IN
W4F
fp16
27.5849
32.1693
Table 4: Quantitative results under low -w-prrcision inference. W indicates weight-only quantization, W+F indicates weight and feature quantization.
4.5
INTRIGUING PROPERTIES UNDER i-LN
4.5.1
EVIDENCE OF ENIIANCED SPATIAL CORRELATION VIA STRUCTURED RPE
Relative Position Embeddings (RPE) explicitly encodes relative spatial positions between tokens in an input-agnostic manner, similar to the convolution operation that inherently captures the spatial locality through their structured kernel patterns. Accordingly, we can consider well-structured RPEs strong indicator of enhanced spatial correlation understanding. In Fig [ proposed normalization method influences spatial relationship modeling by visualizing the learned RPE of both the baseline IR Transformer and proposed method. The baseline Transformer exhibits noisy, unstructured embedding patterns, suggesting a limited capability to effectively model spatial correlations. Conversely, our method produces RPEs that resemble well-structured convolutional filter patterns, clearly indicating superior capture of spatial relationships. This structured embed- ding aligns with our hypothesis that our spatially holistic normalization better preserves intrinsic spatial correlations, helping the network to learn spatial relations more effectively. In Fig. 12 we provide further visual examples of RPEs between LN, i-LN and also the ablated variants L and LN+Rescaling, which shows aligning results with the discussion above.
we analyze how
4.5.2 ROBUSTNESS IN Low PRECISION INFERENCE
Image restoration networks often require deployment on lightweight edge devices, creating significant demand for efficient inference in IR Transformers. A common approach enhance inference efficiency is reducing precision during model deployment. Consequently, we conducted experiments under seduced-precision inference conditions to empirically evaluate the effects of i-LN. Initially, we applied linear weight quantization to the model weights. As shown in Tab vanilla LayerNorm resulted in substantial performance degradation, while i-LN demonstrated remarkable stability. We further conducted half-precisiod inference experiments, casting both internal feature values and weights to malf-precision floating-pon point numbers. Fig [o]ilustateteett that vanilla LayerNorm generated extensive regions of black dots, indicating network-generate infinity values due extreme internal feature magnitudes inadequate for low-precision conditions. Notably, no substantial performance degradation was observed in regions where the network maintained finite feature values. This highlights the necessity of well-bounded feature values achieved by i-LN, emphasizing its critical role in enabling efficient inference for IR Transformers.
4.5.3 EMPIRICAL EFFECTS OF IMPROVED STABILITY
To further probe training stability, conduct extensive multi-run experiments across diverse random seeds (Fig [T]. Here, we use HAT, with a smaller batch size of Vanilla LN exhibits inconsistent optimization trajectories, with large fluctuations in feature statistic evolution patterns and substantial discrepancy in final PSNR results for each run. In contrast, our i-LN produces significantly lower variance between multiple runs, in terms of both training statisties and the final reconstruction

---

Published as a conference paper at ICLR 2026
....
LN
.... LN
....
LN
2e9
HLN (Ours)
4e3
J-LN (Ours)
26.0
HLN (Ours)
2e9
263
25.5
2e9
1e9
0
25,0
5e8
24.5
0
0K
100K
200K
0K
100K
200K
100K
200K
Training steps
Training steps
Training steps
(a) Feature Variance
(b) Feature Mean
(c) PSNR
Figure 11: Feature statistics and PSNR across multiple runs with different random seeds. Trajectories show significant fluctuation under vanilla LN, while our i-LN maintains well-bounded and consistent results across all seeds ( 4 SR, Urban100)
performance. These results demonstrate that i-LN provides more reliable optimization landscape, reducing susceptibility to randomness in initialization or data ordering, which is an important practical advantage for training IR networks. Also refer to Appendix E. for analysis over multiple batch sizes.
RELATED WORK
Image Restoration Transformers. Recent advances in Image Restoration (IR) transformers (Chen
al. 2024| Zamir et al. [022 Wang al. 2022c Zhang et al. 2022) show superior performance over CNNs (Dong et al. 2015 Kim et al. [016 Zhang et al. (2088) by leveraging attention mechanisms to effectively model rangrranee context. pioneering work, SwinIR (Liang et al., (021). adopted an efficient Triin-Tranffrmer (Liu et al. 2021) based architecture in IR tasks, balancing computational
cost and restoration quality. notable method is HAT, which originated as a super-resolution model (Chen et al.) 2023b) but expanded to general image restoration tasks (Chen al. 2023a). By unifying spatial and channel attention within a hybrid attention framework, HAT surpasses existing IR Transformers in both restoration fidelity and robustness across various IR tasks.
Abnormal Feature Behaviors. Normalization a key element in enhancing stability and performance in deep networks, but also lead to unintended feature behavior. EDSR (Lim al., 201 7b), which is a foundational work in super-resolution pointed out that BatchNorm removes range flexibility of intermediate features, leading to a performance drop. Accordingly, normalization layers are removed in the most recent CNN-based SR architectures. Meanwhile, ESRGAN (Wang et al. 2018) and StyleGAN2 (Karras et al. 2020) observe that InstanceNorm and BatchNorm, respectively, cause water droplet-like artifacts. They suggest that the generator might learn to deceive feature statistics by sneaking abnormal values in internal features to reduce the effects of normalization. EDM2 (Karras) et al. 2024) identifies feature magnitude divergence in diffusion models. Accordingly. they redesign the network architecture to preserve the magnitude based on statistical assumptions, leading to overall performance enhancement. DRCT (Hsu al. 2024) notes that feature map intensities drop sharply at the end of SR networks, leading to information bottlenecks, and shows that dense residuals help.
CONCLUSION
We analyzed the training dynamics of Image Restoration (IR) Transformers and highlighted an overlooked phenomenon: divergence of feature magnitudes accompanied by collapses in channel- wise entropy. We interpret this networks attempting to bypass the constraints of conventional LayerNorm, whose per-token normalization and input-independent scaling disrupt spatial correlations and restrict the flexibility needed for accurate restoration. To address this, we introduced Image Restoration Transformer Tailored Layer Normalization (i-LN), simple drop-in replacement for LayerNorm. designed to better align with the unique characteristics of IR tasks and preserve important low-level features of the input throughout the network. i-LN normalizes jointly across spatial and channel dimensions and incorporates neputddeenndent rescaling, aligning normalization more closely with the demands of IR tasks. Extensive experiments show that é-LN prevents feature divergence, stabilizes channel entropy, improves robustness under low-precision inference, and significantly enhances IR performance across diverse tasks.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLED
This work was supported in part by MSIT/ITTP (No. RRS-2022-11220680, RS-2020-12001811 RS-2019- I1190421, RS-2024-00459618, RS-20244000302227, RS-2024-00437633, RS-2024-00437102, RS- 2025-25442569, MSIT/NRF (No. RS-2024-003377229, and KNPA/KIPOT (No. RS- 202--3292233...
REPRODUCIBILITY STATEMENT
Experimental settings for both training and evaluation are described in Sec Detailed hyperparameter settings and network configurations for each model variant are described in Appendix B.I and Tab Detailed algorithm to calculate the channel entropy in Appendix A.2 We plan to release the code for further reproducibility.
REFERENCES
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 126-135, 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXhv:1607.06450 2016.
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley, Rezero is all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence, pp. 1352-1361. PMLR, 2021.
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low-complexity single-image super-resolution based nonnegative neighbor embedding. 2012.
Yochai Blau and Tomer Michaeli. Rethinking lossy compression: The iree-distortion-pereeppii tradeoff. In International Conference on Machine Learning, pp. 675 685.6.5.200 PMLR, 2019.
Rob Brekelmans, Daniel Moyer, Aram Galstyan, and Greg Ver Steeg. Exact rate-distortion in autoencoders via echo noise. Advances in neural information processing systems, 32, 2019,
Xiangyu Chen, Xintao Wang, Wenlong Zhang, Xiangtao Kong, Yu Qiao, Jiantao Zhou, and Chao Dong. Hat: Hybrid attention transformer for image restoration. arXiv preprint arXiv:230900233, 2023a.
Xiangyu Chen, Xintao Wang, Jiantao Zhou, Yu Qiao, and Chao Dong. Activating more pixels in image super-resolution transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22367-22377, 2023b.
Xiangyu Chen, Zheyuan Li, Yuandong Pu, Yihao Liu, Jiantao Zhou, Yu Qiao, and Chao Dong. A comparative study of image restoration networks for general backbone network design. In European Conference on Computer Vision, pp. 74 91. Springer, 2024.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang, Image super-resolution using deep convolutional networks. JEEE transactions on pattern analysis and machine intelligence, 38(2): 295-307, 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929 2020.
Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Pointwise ahape-adaptive det for high- quality denoising and deblocking of grayscale and color images. IEEE transactions image processing, 16(5):1395-144 1311- 2007.
R. Franzen. Kodak lossless true color image Volume 4,
us http://xO...a//zzhhat///ccaaaa 1999.
11

---

Published as a conference paper at ICLR 2026
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference learning representations, 2017.
Chih-Chung Hsu, Chia-Ming Lee, and Yi-Shiuan Chou. Dret: Saving image super-resolution away from information bottleneck. arXiv preprint arXiv:2404.00722, 2024.
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5197-2265 2015.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. pmlr, 2015.
Kui Jiang, Zhongyuan Wang. Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, and Junjun Jiang. Multi-scale progressive fusion network for single image deraining. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 8346-8355, 2020.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8110-8111 2020.
Tero Karras, Miika Aittala, Jaakko Lchtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24174-24184, 2024.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1656-1654, 2016.
Jingyun Liang, Jiczhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Im- age restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1844,184, 2021.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017a.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference computer vision and pattern recognition workshops, pp. 136-144, 2017b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pp. 416-423. IEEE, 2001.
Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using mangal dataset. Multimedia Tools and Applications, 76:218 81 1-28888, 2017.
H Sheikh. Live image quality assessment database release 2. edddresearechquuliiyy 2005.
http://tive. ece. utexas.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 32-42, 2021.
12

---

Published as a conference paper at ICLR 2026
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv: 2016.
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy, Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, pp. 0-0, 2018.
Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super- resolution with pure synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 1905-1914, 2021.
Xintao Wang, Chao Dong, and Ying Shan. Repsr: Training efficient vgg-style super-resolution networks with structural re- parmmeterization and batch normalization. In Proceedings of the 30th acm international conference on multimedia, pp- 2556-2564, 2022a.
Xintao Wang, Liangbin Xie, Ke Yu, Kelvin C.K. Chan, Chen Change Loy, and Chao Dong. BasicSR: Open source image and video restoration toolbox. https:///itthu. com/ XP ixeelroupp BasicsR 2022b.
Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 17683-17693, 2022c.
Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1357-1366, 2017.
Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming optimization dilemma latent diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 15703-15712, 2025.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming- Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5728-5739 2022.
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In International conference on curves and surfaces, pp. 711-730. Springer, 2010.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
Dafeng Zhang, Feiyu Huang, Shizhuo Liu, Xiaobing Wang, and Zhezhu Jin. Swinfir: Revisiting the swinir with fast fourier convolution and improved training for image super-resolution. arXiv preprint arXiv:2208. 11247, 2022.
He Zhang, Vishwanath Sindagi, and Vishal M Patel. Image de-raining using a conditional generative adversarial network. IEEE transactions on circuits and systems for video technology, 30(1 I): 3943 3956, 2019.
Lci Zhang, Xiaolin Wu, Antoni Buades, and Xin Li. Color demosaicking by local directional interpolation and nonlocal adaptive thresholding, Journal of Electronic imaging, 20(21:0 023016, 2011.
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European conference on computer vision (ECCV), pp. 286-301, 2018.
Yupeng Zhou, Zhen Li, Chun-Le Guo, Song Bai, Ming-Ming Cheng, and Qibin Hou. Srformer: Permuted self-attention for single image super-resolution. In Proceedings of the IEEF/CVF international conference computer vision, pp. 177801 2023.
13

---

Published as a conference paper at ICLR 2026
Table 5: Overview of model variants. indicates the group (type) of each model variant: from lightweight to computationally extensive settings. The according placements of the experiments are specified as Main (i.e., the main article) and Appendix. The placements of the detailed network hyperparameters and training configurations for each variant are highlighted in bold.
Variant
Description
Typel (Lightweight Computational Contiguration)
These configurations used for most analyses. All models trained from scratch without the Warm-start strategy (i.e., the x4 SR models are not finetuned from the x SR weights), Mixing Augmentations, Progressive Patch Sizing.
SwinlR
Main
Details are provided Tab. 10 network architecture the official SwinlR-light model imple- mentation.
This variant shares the same
DRCT, Maan
Details are provided in Tab. 11 This variant is a lightweight variant of the DRCT [Hsu al. (2024) model implementation, The embedding dimension is reduced in order align the network architecture with the HAT, model.
HAT Main & Appendix
Details are provided in Tab. 13 This is a variant a lighter version of the HAT-S Chen al. (2023b) model, modified with a slightly reduced embedding dimension. This change was made since the standard HAT-S, despite being denoted as small, requires more Mult-Adds than the full-sized SwinIR model.
SRFormer - Appendixx
Details are provided in Tab. 12 This variant is a lightweight variant of the SRFormer Zhou et al. (2023) model. The overall capacity reduced to align with the networks specified above.
Type2 (Moderate Computational Configuration)
These configurations are used for the ablation study and low-precision inference analysis.
HAT Main
Details provided in Tab. 14 This variant shares the same network capacity as the official HAT-S implementation, which is slightly heavier than HAT, However, the training budget is reduced for computational efficiency compared to HAT-S (the public model); the patch size and the batch size each. Aligning with Type1 configurations, all models were trained from scratch without the warm-start strategy for 300K.
halved
Type3 (Extensive Computational Configuration)
These configurations used when comparing with official public models and validating the scalability of our method.
HAT Appendix
Details are provided in Tab. 15 This variant shares the same network architecture as the official full-sized HAT implemen- tation and precisely follows the training configuration of the public model. Quantitative results from this variant are copied from the original paper(Chen
Chen et al. (023b).
14

---

Published as a conference paper at ICLR 2026
Table Quantitative results for classical image super-resolution under computationally extensive setting. i indicates that we have precisely followed the architecture and training settings of the official public model, as specified Tab 15 HAT requires 40 GPU days for x2 SR (500K train iterations) and additional 20 GPU days for x4 SR (250K finetuning iterations). The best results for each setting are highlighted in bold, respectively.
Set5
Set14
BSDI00
Urban 100
Manga 109
Backbone
Scale
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
HAT + +LN
x2
38.63
9630
34.86
.9244
32.62
.9033
34.45
.9666
40.26
.8099
HAT + i-LN (Ours)
x2
38.65
.6311
34.92
.8276
32.63
.9053
34.60
.9476
40.38
.9811
HAT + LN
x4
33.04
9056
29.23
.7973
28.00
.7517
27.97
.8368
32,48
.9922
HAT + i-LN (Ours)
x4
33.12
.9064
29.26
.7981
28.00
.7520
28.04
.8388
32.56
.9299
A EXPERIMENTAL DETAILS
A.1 MODEL IMPLEMENTATION DETAILS
Since this work provides extensive analysis for more than 60 configurations, analyses throughout this work are performed on various settings due to computational efficiency. We provide implementation details in terms of both network architectural hyperparameters and training configuration for each model variant in Tab
Typel (Lightweight Sefting): These models are the lightweight variants of the original implementations. These configurations are used in Tab[aand Tab2 where effects of different normalization schemes and task variations are analyzed.
Type2 (Moderate Setting): These model variants indicate moderate computational budget settings. They are used for the ablation study and also for the analysis in low-precision settings (Tab Tab
Fig I0
Type3 (Computationally Extensive Setting): These model variants indicate computation- ally extensive settings. This configuration is used to validate the scalability of our method (T which aligns with the official implementation of the public models.
A.2 CHANNEL ENTROPY
Algorithm D represents a simple pseudocode to calculate the channel-axis entropy used our analysis. A sharp drop in channel-axis entropy indicates that feature activations are becoming concentrated in a few specific channels. Analysis throughout this work shows that this entropy collapse is intrinsically linked the feature divergence problem that arises from conventional LayerNorm in Image Restoration (IR) Transformers.
Algorithm Channel Entropy Calculation
Require: Activation tensor x of shape (C, H, W), small constant e for numerical stability.
Ensure: A single scalar entropy value.
Step 1: Average the total activation magnitude over spatial-dim.
Step 2: Convert to a probability distribution.
Step 3: Compute channel entropy.
1: function CNARNNLEE
2: Tavg <- mcan (abs(r), dims = (H, W))
> Step
3: p f- softmax(Fawy
Step
4: entropy 6- sum(p log(p + c))
Step
5: return entropy
6: end function
15

---

Published as a conference paper at ICLR 2026
Table Quantitative results for x super-resolution on the SRFormer (Zhou al. 2023 network architecture. The network capacity and the training budget are adjusted as in Tab 12 w which aligns with experimental settings T The best results for each setting are highlighted in bold, respectively.
Sct5
Sct14
BSDI00
UrbanI00
Manga109
Backbone
Scale
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR SSIM
PSNR SSIM
SRFormer: LN
x4
32.41
.8972
28.77
.7533
27.68
.7398
26.43
.7957
30.98
.141I
SRFormer; i-LN (Ours)
x4
32.45
.8799
28.81
.7862
27.70
.7407
26.49 .7997
31.10
.9522
Table 8: Quantitative results for x4 super-resolution with additional regularization methods. GC denotes Gradient Clipping. and KLD denotes an auxiliary KI--Divergence loss. Neither proved effective addressing the instability caused by LayerNorm. GC slightly improves stability but still allows extreme feature magnitudes (5.6 10%), comparable to the vanilla baseline (5.8 x 10%). KLD regularization enforces smoother statisties but leads notable performance drop. In contrast, proposed i-LN yields magnitudes close to W(0, 1) (around 1.2) while consistently outperforming all alternatives. The best results for each setting highlighted in bold.
SerS
Set14
BSD100
Urban 100
Mangal 009
Backbone
Scale
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
HAT LN HAT + LL + GC HAT; + LL + KLD
x4
32.51
.992
28.79
.7876
27.68
.711
26.55
8015
31.01
.1500
x4
32,55
.996
28.87
.7882
27.74
.7417
26.70
.8077
31.31
.9169
x4
32.36
8974
28.65
.7853
27.64
.7402
26.34
.7972
30.41
.1055
HAT i-LN (Ours)
x4
32.72
.9019
29.01
.7915
27.84
.7456
27.17
.8167
31.82
.9228
B ADDITIONAL BENCHMARK RESULTS
B. SCALING MODELS AND COMAPRISON AGAINST PUBLIC MODELS
In Tablo we validate the scalability of the proposed i-LN under computationally extensive settings. Specifically, we train our models on top of the full-sized HAT architecture variant, with the exact training configurations of the public model as specified in Tab 15 The models are indicated as HAT where means that we have precisely followed the exact network architecture hyperparameters and training configurations for fair comparison. HATt for x2 SR and x4 SR variants requires 40 GPU days and 20 GPU days wall-clock time each, with NVIDIA RTX A6000s under the representative RSR WWan et al. 2022b framework.
Benchmark. In Tab we validate that replacing the conventional I. LayerNorm with the proposed i-LN leads to significant performance gain also the computationally extensive setting where the networks have significantly larger capacity Accordingly, both 1) lightweight settings, as shown in our main article and 2) also in computationally extensive settings in Ta showing the scalability of our i-LN.
conclude that the proposed i-LN is effective in
Training Details. Scores are from the original paper for the baselines. Here, we follow the original training scheme where x4 SR models are trained under warm-start configuration (i.e., finetuned from x2 SR model weight).
B.2 ADAPTATION TO EFFICIENT SR NETWORK
In Tab 7 we further validate the effectiveness on top of the ERFormer[Zhou et al. (2023) architecture, representative efficient SR network. Similar other Typel model variants, the training configurations are adjusted. Refer to Tab 12 for the detailed configurations.
Discussion. SRFormer utilizes Permuted Self-Attention (PSA) mechanism. Accordingly, features across multiple pixels are reshaped into single feature-pixe (pixelshuffle-styee) Thus, the per- token vanilla LN implicitly takes normalization parameters across multi-pixels. While the effect of permuted self-attention in the perspectives of normalization was not discussed in the original work, our work suggests insights that PSA induces (partially) spatial holisticness normalization, a potential factor for the performance gain of SRFormer (i.e., potentially reducing the performance gap against ours). Seeking further improvements regarding the relationship between the reshaping operation and normalization may be valuable direction for future work.
16

---

Published as a conference paper at ICLR 2026
C OTHER REGULARIZATION TECHNIQUES FOR TRAINING STABILITY
Beyond our proposed i-LN, one may ask whether simpler regularization methods could mitigate the training instabilities of IR Transformers. We therefore examined common strategies such as gradient clipping (GC) and KL divergence (KLD) regularization in Tab
While GC is widely used to bound exploding gradients, our experiments confirmed that it does not prevent the emergence of extreme feature magnitudes in IR Transformers. The maximum feature magnitude observed during training with GC was 5.6 x 10 As a reference, the maximum feature magnitude for the vanilla HAT was 5.8 10 In contrary, our HAT shows 1.2, very closely aligning with the expected magnitude of random noise sampled from the normal distribution (0 1), which vanilla model, it consistently underperforms compared to i-LN.
Additionally, while GC leads to slight performance improvement against the
Likewise, KLD regularization can stabilize feature statistics, but at the cost of substantial reconstruc- tion performance degradation. Specifically, observed that although KLD encourages well-behaved distributions, the resulting models suffered PSNR drops even below the baseline with vanilla LN. This is consistent with prior findings in rate distortoo theory (Brekelmans et al. 2019 Blau & Michaeli 2019), and also to VAE literature (Higgins et al. 2017 penalties reduce reconstruction fidelity.
Yao et al. (025), where strong regularization
Overall, these results highlight that although dies, they are either ineffective (GC) or detrimental reconstruction quality (KLD). This further emphasizes the necessity of normalization methods tailored to the unique requirements of IR Trans- formers.
regularization may offer partial reme-
D FURTHER ABLATION STUDY
Here, we further compare our full method i-LN against the baseline (vanilla) LN, and two variants: LN* and LN+Rescaling, which each are ablations of out method without rescaling (LN*) and without the spatial holistic type of normalization (LN+Rescaling). Below, we provide additional visualization of relative position embeddings (RPEs) and also further visual comparison of the according x4 SR result. Quantitative comparison can be seen in Tab of the main article. Fach of these experimental setups is directly aligned with the ablation analysis presented in Tab. oof the main article. Accordingly, all experiments were performed with the HAT₂ configuration for the synthetic (bicubic) x4 SR task.
D.1 ABLATION STUDY: RELATIVE POSITION EMBEDDINGS (RPE) VISUALIZATION
In Fig. 12, provide comprehensive visual comparison of the learned Relative Position Embed- dings (RPE) across different normalization strategies. The figure compares vanilla Layer Normaliza- tion (LN), our proposed i-LN, well as two ablated variants, LN* and LN+Rescaling.
To obtain deeper insight into the dynamics and stability of RPE formation, we visualize both early- stage training (100K iterations) and fully converged models (500K iterations), and further examine representations from shallow block (RHAG. 0) and deep block (RHAG.5).
Across all settings, vanilla LN exhibits highly noisy RPE structures throughout the entire training process. Even after convergence, its embeddings fail to organize into meaningful spatial patterns, suggesting limited ability to encode coherent spatial correlations. The LN* variant, which adopts spatially holistic normalization, occasionally reveals global structures, but these patterns remain weak and are overshadowed by considerable noise. The LN+Rescaling variant shows improved structure in deeper layers, yet its shallow-layer embeddings remain unstable and inconsistent. This indicating that rescaling alone is insufficient to guide early-layer RPE formation, reflected in low reconstruciton scores in Tab B
In contrast, our proposed i-LN consistently produces substantially clearer and more structured RPE maps, with significantly reduced noise across both shallow and deep layers and across all training stages. The strong spatial coherence visible in the embeddings aligns with the quantitative improvements reported in Tab. 3 where i-LN achieves the highest performance among all evaluated variants. These visual results confirm that i-LN facilitates stable and meaningful spatial relational modeling throughout the entire network depth and training trajectory.
17

---

Published as a conference paper at ICLR 2026
LN
ELN (Ours)
LN
LN*Rescale
Sballow Layer
Layer
(a) 100K training iterations
Shallow
Deep Layer
(b) 500K training iterations
Visual comparison of Relative Position Embeddings (RPH) across attention heads. We show the RPEs from the last three attention layers of each shallow (REAG. 0) and deep (RHAG. 5) building block of HAT Chen et al. (2023a) (i.e., the RHAG Block), as well as early training (100K iterations) versus fully converged models (500K iterations). Each corresponds an experimental setting aligned with Tab. where vanilla LN and our i-LN are the primary comparison, and LN and LN+Rescale represent ablation variants obtained by selectively removing components our method. Our full method (i-LN) yields cleaner and more stable RPE patterns, with substantially reduced noise across variations in training iteration and layer depth.
Figure 12:
(a) HQ
(b) HQ crop
(c) LN
(d) i-LN (Ours)
(N)
(f) LN+Rescaling
Figure 13: Visual comparison between LN and i-LN, along with the ablated variants LN and LN+Rescaling with the HAT, configuration. Experimental settings follow the ablation study in the main article (Tab. 3}. The proposed i-LN more faithfully reconstructs fine details, producing sharper edges and clearer complex patterns than the LN baseline.
D.2 ABLATION STUDY: x4 SR VISUALIZATION
In Fig 1 provide additional visual comparisons between the SR outputs obtained by networks each employing LN, LN* LN+Rescale and our i-LN.
Across all cases, the model equipped with i-LN produces the sharpest and most faithful reconstruction of fine- ine-grained structures, including thin edges, repetitive patterns, and high-frequency textures. The restored images exhibit not only improved clarity but also enhanced local contrast and reduced artificial smoothing, indicating that i-LN effectively preserves low-level feature statistics throughout the network.
18

---

Published as a conference paper at ICLR 2026
(a) HQ
(b) HQ crop
(c) Input
(d) i-LN (Ours) (c) LN
Figure 15: Visual comparison between LN and i-LN for the real-world x4 super-resolution task with HAT, Both the training images and the test images were synthesized following the Real-
ESRGAN (Wang al. 2021 degradation pipeline.
By contrast, the baseline vanilla LN often yields blurry and overly smoothed outputs, where crucial high-frequency details are lost. This degradation consistent with the unstable and noisy RPE behavior observed earlier, suggesting that vanilla LN struggles to maintain coherent spatial relations required for accurate detail reconstruction, The ablated variants LN (spatial holistic normal- ization without rescaling) and LN+Rescaling (rescaling without spatial holisticness) show partial improvements over LN but still fall short of i-LN.
Overall, the qualitative comparisons provide visual evidence that both components of i-LN (spatial holisticness and input-adaptive rescaling) are essential.
E ADDITIONAL EXPERIMENTS ON THE STABILITY AND ROBUSTNESS
In this section, we provide additional experiments that further validate the stability, robustness, and general applicability of the proposed i-LN across a wide range of practical and challenging training scenarios. Specifically, we evaluate: 1) real-world super-resolution settings, robustness under multiple training batch sizes.
Across all settings, i-LN consistently outperforms and exhibits more stable behavior than the con- ventional per-token LayerNorm (LN). Unless otherwise specified, the base configuration follows the HAT, setting for the x4 SR task. all experiments. we compare against the vanilla LN baseline under identical training configurations.
F. ROBUSTNESS ACROSS VARYING BATCH SIZES
To evaluate the robustness of i-LN, we train HAT; models but with varying batch size from 2 to 8; while keeping all other hyper- parameters fixed (Fig. 14. This experimental design is chosen in order to mimic unstable training configurations without heavy hy- perparameter search. For each configuration, we report PSNR and SSIM for the x SR task on Urban 100 100 throughout training. As shown in Fig. models with i-LN consistently achieve higher PSNR/SSIM than the baseline across all batch sizes. Notably, the performance gap remains stable as the batch size decreases. These results demonstrate that the benefit of i-LN is not tied a particular training setup, and that its stability advantages per- sist even under extremely small-batch training (e.g., batch size 2). This property is especially valuable for memorr--onsttained environments where large batches are infeasible.
27.0
HAT, LN
HAT: + FLN (Ours)
26.5
26.0
25.5
24.5
0K
100K
200K
300K
Training steps
Figure 14: PSNR for x SR with the HAT, model, but with batch size 2, 4, 8 (Urban100).
E.2 WOALD SUPER-RESOLUTION
To evaluate the practical effectiveness of i-LN, we adopt a real- world degradation setup following the RealESRGAN Wang al. (021) pipeline. These analyses
19

---

Published as a conference paper at ICLR 2026
Sct5
Set
BSIDTO0
Urban100
Manga109
Run
Backbone
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
RealHAT, + LN
26.27
.7601
24.43
.6272
24.39
.881
22.01
.6044
23.65
.7466
RealHAT, + i-LN
26.38
.7656
24.58
.6322
24.47
.5918
22.19
.6146
23.96
.7557
RealHAT' + INN
26.77
.7739
24.27
.6201
24.29
.5839
22.00
.6056
23.61
.7503
RealHAT + i-LN
26.90
.7777
24.46
.6272
24.37
.5876
22.19
.6141
23.84
.7587
RealHAT + LN
24.39
.6927
24.77
.6325
24.32
.5823
21.96
.5973
23.71
.7522
RealHAT]] 1-I.N
24.58
.6993
24.94
.6377
24.40
.5862
22.16
.6055
23.99
.7602
RealH[AT + LN
25.86
.7472
24.72
.6389
24.35
.5878
21.84
.5954
23.22
.7365
RealHAT + 1-LN
26.05
.7531
24.93
24.43 .6477
.5913
22.04
.6046
23.48
.7450
RealHAT + I.N
25.08
.7086
24.46
.6355
24.38
.5899
21.93
.6033
23.46
.7458
RealHAT + i-I.N
25.26
.7120
24.63
.6411
24.46
.5933
22.10
.6115
23.73
.7554
RealHAT, + LN
25.68
.7365
24.53
.6308
24.35
.8644
21.95
.6016
23.53
.7463
Avg.
RealHAl1 + i-NN
25.83
.7415
24.71
.6366
24.43
.3900
22.14
.6101
23.80
.7550
Table 9: Quantitative results for real-world x4 super-resolution across five random seeds. Both the training images and the test images were synthesized following the Real-ESRGAN (Wang et al. degradation pipeline. The best results for each setting highlighted in bold.
2021
complement the main text by demonstrating that the advantages of i-LN are not restricted to controlled usage and unstable training regimes.
laboratory settings, but generalize to
Training is performed on synthetic DF2K pairs, and testing is conducted on synthetically degraded versions of standard SR benchmarks (Set5, Set14, BSD1000, Urban100, Manga109), Degradation syn- thesis strictly follows the RealESRGAN procedures. To assess robustness, we repeat each experiment across five different random seeds and report the average score the resulting PSNR/SSIM scores.
As shown in Tab and Fig 15 i-LN leads to consistent and significant improvements over vanilla LN across all benchmarks, despite the increased complexity the real-world degradation pipeline. These results highlight that i-LN is not only theoretically grounded but also practically beneficial in more challenging real-world restoration scenarios.
F ADDITIONAL QUALITATIVE COMPARISON
We provide additional visual comparisons between IR Transformers with our proposed i-LN against their counterparts using vanilla Layer Normalization (LN). As shown in the following figures, i-LN
consistently produces sharper structures, cleaner textures, and fewer artifacts across a range of low-
level vision tasks. Qualitative results are provided for: (i) super-resolution (Figs. [6 and 17] (ii) image denoising (Figs. [8 and 19), (iii) JPEG compression artifact removal (Figs. 23 and 22 and
(iv) image deraining (Figs. 20 and 21
20

---

Published as a conference paper at ICLR 2026
(a) HQ
(b) HQ crop
(c) Input
(d) i-LN (Ours)
(c) LN
Figure 16: Visual comparison between LN and i-LN for the super-resolution task with HAT;,
(a) HQ
(b) HQ crop
(c) Input
(d) i-LN (Ours)
(e) LN
Figure 17: Visual comparison between LN and i-LN for the super-resolution task with DRCT,
(a) HQ
(b) HQ crop
(c) Input
(d) i-LN (Ours)
(e) LN
Figure 18: Visual comparison between LN and i-LN for the denoising task with HAT,
(a) HQ
(b) HQ crop
(c) Input
(d) i-LN (Ours)
(c) LN
Figure 19: Visual comparison between LN and i-LN for the denoising task with SwinIR,,
21

---

Published as a conference paper at ICLR 2026
(a) HQ
(b) HQ crop
(c) Input
(d) i-LN (Ours)
(c) LN
Figure 20: Visual comparison between LN and i-LN for the deraining task with HAT
(a) HQ
(b) HQ crop
(c) Input
(d) i-LN (Ours)
(e) LN
Figure 21: Visual comparison between LN and i-LN for the deraining task with SwinlR
(a) HQ
(b) HQ crop
(c) Input
(d) i-LN (Ours)
(e) LN
Figure 22: Visual comparison between LN and i-LN for the JPEG artifact removal task with HAT
(a) HQ
(b) HQ crop
(c) Input
(d) i-LN (Ours)
(c) LN
Figure 23: Visual comparison between LN and i-LN for the JPEG artifact removal task with SwinIR
22

---

Published as a conference paper at ICLR 2026
G DETAILED DERIVATIONS FOR STRUCTURE PRESERVATION
G.1 NOTATION AND PRELIMINARIES
Let X E R be the feature matrix with tokens xą E RC (row-vectors). Define the inter-pixel (inter-token) structure by the set of pairwise displacements
AX {कt 24 1, S L}.
map T RC + RC preserves inter-pixel structure up scale if there exists homothety ((x) = az + with a > 0, E E RC such that
T(xe) - T(x)) = a(x: 2&) for all e, k.
Equivalently, all angles and pairwise distance ratios are preserved.
We analyze the pure normalization maps (i.c., the normalization before the affine (7, 3) is applied); any global translation by / does not affect AX, and scalar post-scale can be absorbed into the homothety factor
G.2 PROPOSITION (VANILLA LAYERNORM FAILS TO PRESERVE STRUCTURE)
Let TLn denote the transformation defined by the normalization in vanilla per-token LayerNorm. In general there do not exist a and an orthogonal Q such that for all tokens Zz, Zk.
(a+( (wx) a Q - xx).
Hence TLN not conformal on the token set. By the nested class relation Homothety C Similarity C Conformal, it follows that TLN is neither similarity nor homothety, and thus does not preserve inter-pixel structure in general.
Proof.
Write per-token means and standard deviations
C
= ( Σ
C
Wige -m)²) 1/2
Σ ft..s
Fe =
of
The pure LN map (no 3) acts componentwise as
3e 14
TIN(xe)
0e
so for two tokens é, k,
Atk = Tin (ix( - TiNN(1))
It
Zk
Ht
1.
Ok
Assume for contradiction there exist a > and orthogonal Q such that A = a Q(x - xx) for all f, k. Compare the coefficients of and Ia both sides of (1). Because the equality must hold for arbitrary token values, we must have
1 T= aQ and T= aQ, σk
ai
hence σ for all f, l and Q must be proportional to the identity. With = a. the bias term in (1) reduces ((= 1, which must vanish for all thus μi = 14 for all Therefore the assumed similarity can hold only in the degenerate case where all tokens share identical per-token mean and variance.
For real features, (ne,o) ) are not constant across tokens, so the assumption leads to contradiction. Hence no single similarity map exists; TlN is not conformal and does not preserve spatial structure.
Remark. The degenerate equal-statisties case is precisely the exception noted in the main text.
23

---

Published as a conference paper at ICLR 2026
G.3 PROPOSITION 2 (LN* PRESERVES STRUCTURE)
Let Ti denote the transformation defined by the normalization in spatially holistic LayerNorm (LN*) with global mean and standard deviation
1 LC
Σ Dt,es
a=
=1o E(rec 12)) 1/2 0.
Then for any tokens dig; Tk.
TLN- Tin- (zx)
so Tin- is homothety and preserves inter-pixel structure up global scale.
Proof.
TIn- (without y, 8)
Tx) T. ml
Hence
2t u1
26 11
1
Tin- (are) TIN (Ix)
(xx 5x )
This is exactly homothety with scale factor =o therefore angles and pairwise distance ratios of A.X are preserved and the spatial configuration is rigid up to uniform scale.
24

---

Published as a conference paper at ICLR 2026
Network Architecture Hyperparameters
Network Architecture Hyperparameters
Embedding Dimension
60
Layer Depths
[6. 6, 6, 6]
Embedding Dimension
96
Attention Heads
[6. 6, 6, 6]
Layer Depths
[6, 6, 6, 6, 6, 6]
Window Size
x 8
Attention Heads
[6, 6, 6,
Ratio
Window Size
16
Residual Connection
'Icon''
MLP Ratio
Residual Connection
'Iconv'
Dataset Configuration
Dataset Configuration
Training Datasct
DIV2K Flickr2K
PatchSize BatchSize
Training Dataset
DIV2K + Flickr2K
Denoising
64 64 16
PatchSize BatchSize
Deraining
128 x 128 8
x Super Resolution
64 x 64 16
JPEG Artifact Removal
64 64 16
X4 Super Resolution
128 128 16
Noise Degradation
torch. randn OpenCV
Degradation
MATLAB
JPEG Degradation
Optimizing Configuration
Optimizing Configuration
Total Iterations
300K Adam
Total Iterations
300K Adam
Optimizer
Optimizer
Learning Rate (LR)
2 x 10-4
Learning Rate (IRR)
x 10
Adam Betas
(0.9, 0.99)
Adam Betas
(0.9, 0.99)
Weight Decay
Weight Decay
Scheduler (7 0.5)
StepLR
Scheduler (7 = 0.5)
StepLR
Milestones (K)
250
Milestones (K)
250
Loss Function
LI Loss
Loss Function
L1 Loss
Table 11: Hyperparameters and training con- figurations for the model variant DRCT, Network architectural terminology is based on terminologies either in the official implemen- tation of each work or the implementation in Bas i cSR Wang et al. (002b)).
Table 10: Hyperparameters and training con- figurations for the model variant SwinlR Network architectural terminology is based on terminologies either in the official implemen- tation each work or the implementation Bas i cSR Wang al. (2022b).
25

---

Published as a conference paper at ICLR 2026
Network Architecture Hyperparameters
Embedding Dimension
96
Layer Depths
[6, 6, 6, 6, 6, 6]
Attention Heads
[6. 6,
6, 6, 6, 6]
Window Size
16 x 16
Network Architecture Hyperparameters
MLP Ratio
2
Embedding Dimension
60
Compress Ratio
24
Layer Depths
[6, 6, 6, 6]
Squeeze Factor
24
Attention Heads
[6. 6. 6, 6]
Overlap Ratio
0.5
Window Size
16 x 16
Conv Scale
0.01
MI.P Ratio
2
Residual Connection
"connv
Residual Connection
cconv
Dataset Configuration
Dataset Configuration
Training Dataset
DIV2K Flickr2K
Training Dataset
DIV2K Flickr2K
PatchSize BatchSize
PatchSize BatchSize
Denoising
64 16
x4 Super Resolution
128 128 | 16
Deraining
128 x 128 |8
SR Degradation
MATLAB
JPEG Artifact Removal
64 x 64 16
x2 Super Resolution
64 x 64 16
Optimizing Configuration
x Super Resolution
128 128 16
Total Iterations
300K
Noise Degradation
torch.randn
Optimizer
Adam
JPEG Degradation
OpenCV
Learning Rate (LR)
2 10m4
SR Degradation
MATLAB
Adam Betas
(0.9, 0.99)
Optimizing Configuration
Weight Decay
0
Scheduler (y = 0.5)
StepI.R
Total Iterations
300K
Milestones (K)
250
Optimizer
Adam
Loss Function
1.1 Loss
Learming Rate (I.R)
2 10-4
Adam Betas
(0.9, 0.99)
Table 12: Hyperparameters and training con- figurations for the model variant SRFormer Network architectural terminology is based on terminologies either in the official implemen- tation of each work or the implementation in Basi BasicS3 Wang et al. (20226).
Weight Decay
0
Scheduler (ry = 0.5)
StepLR
Milestones (K)
250
Loss Function
LI Loss
Table 13: Hyperparameters and training con- figurations for the model variant HAT, Net- work architectural terminology is based on ter- minologies either the official implementa- tion of each work or the implementation in BasicSR Wang et al. (202bb).
26

---

Published as a conference paper at ICLR 2026
Network Architecture Hyperparameters
Embedding Dimension
Layer Depths
[6, 6, 6, 6, 6, 6]
Attention Heads
[6. 6, 6.
6]
Window Size
16 x 16
MI.P Ratio
2
Compress Ratio
24
Squeeze Factor
24
Overlap Ratio
0.5
Conv Scale
0.01
Residual Connection
'Iconv'
Dataset Configuration
Training Dataset
DIV2K Flickr2K
PatchSize BatchSize
Super Resolution
128 128 16
SR Degradation
MATIAB
Optimizing Configuration
Total Iterations
500K
Optimizer
Adam
Learning Rate (LR)
2 10 10-4
Adam Betas
(0.9, 0.99)
Weight Decay
Scheduler (7 = 0.5)
MultiStepLR
Milestones (K)
[250, 400, 450, 475]
Loss Function
LI Loss
Table 14: Hyperparameters and training con- figurations for HAT,, This variant uses the HAT-S architecture but is trained with a reduced budget. Differences from HAT, are highlighted in bold. Network architectural terminology is based terminologies either in the official im- plementation of each work or the implementation BasicSR Wang al. (2022b).
27

---

Published as a conference paper at ICLR 2026
Network Architecture Hyperparameters
Embedding Dimension
180
Layer Depths
[6. 6]
Attention Heads
[6.
6, 6, 6]
Window Size
16 x 16
MLP Ratio
2
Compress Ratio
Squeeze Factor
30
Overlap Ratio
0.5
Cony Scale
0.01
Residual Connection
'Iconv'
Dataset Configuration
Training Dataset
DIV2K + Flickr2K
PatchSize BatchSize
x2 Super Resolution
128 x 128 32
Super Resolution
256 x 256
32
SR Degradation
MATLAB
Optimizing Configuration
Optimizer
Adam (0.9, 0.99)
Adam Betas
Weight Decay
Scheduler (7 = 0.5)
MultiStepl.R I.1 Loss
Loss Function
x2 Super Resolution
Total Iterations
500K
Learning Rate (LR)
2 x 10-4
Scheduler Milestones
[250, 400, 450, 475]
x4 Super Resolution
Total Iterations
250K
Learning Rate (LR)
1 10
Scheduler Milestones (K)
[125, 200, 225, 240]
Pretrained
finetune from x2 SR weight
Table 15; Hyperparameters and training configurations for HATI, This variant uses the full-sized HAT architecture and precisely follows the training settings of the public model. Network architectural terminology based on terminologies either in the official implementation of each work the implementation in BasicSh, Wang et al. (2022b).
28

---

Published as a conference paper at ICLR 2026
H THE USE OF LARGE LANGUAGE MODELS (LLMS)
In this study, LLMs were used for text editing, grammar correction, and coding assistance for graph visualization.
29


Reviews:
[{"review_id": "CrQ8EE7w2K", "reviewer": "Reviewer_Msrh", "summary": "The paper address the image reconstruction task in vision domain. \nThe authors found that image reconstruction using transformers with vanilla layer norm lead to feature magnitudes divergence and  channel entropy collapses.\nThe authors further analyze the patterns by relaxing the constraints of layer norm, i.e., per-token normalization and input-independent scaling. Specifically, vanilla layer norm breaks spatial correlation due to the per-token normalization and discard input-specific statistics due to the input-independent scaling, which are critical for image restoration tasks. To handle these issues, the authors propose i-LN, which normalizes holistically across space and channels and re-scales each block by the input’s own standard deviation to recover global scale. Experimental results qualitatively and quantitatively show that i-LN obtain better PSNR and SSIM on super-resolution, deraining, denoising, and JPEG artifact removal than vanilla LN on various benchmarks.", "questions": "* Why were three type of model implementations necessary?\n* I recommend to reorganize the location of Table 5 in appendix. There would be better position than the top of the first page of Appendix", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Easy to read\n* Simple but intuitive claims\n* Extensive experiments", "weaknesses": "* Major concern\n    * While the authors demonstrated the strengths of the proposed i-LN over vanilla layer norm (LN) and spatially holistic layer norm (LN*), the experiments only show the comparisons with vanilla LN and other normalization approaches. If the provided claims are valid, I guess the superiority among LN, LN*, and i-LN on the image restoration problem should be LN < LN* < i-LN. Could you further validate this superiority, not only over specific benchmarks but over various benchmarks?\n    * While the authors provided channel entropy collapse comparison among LN, LN*, and i-LN in Fig. 8, it seems not sufficient since the volume for this comparative analysis is too light than the overall volume of the experiments. And still there's no qualitative or quantitative performance comparison among LN, LN* and i-LN, while both are necessary.\n    * Suggestion\n        * I recommend to supplement these experimental results. Since the results for LN and i-LN are already secured, only results for LN* are necessary for the following comparisons:\n            * relative position embedding comparison (Fig. 9) among LN, LN* and i-LN\n            * Qualitative results comparisons (Fig. 6) among LN, LN* and i-LN\n            * Quantitative comparisons (Table 1, 2) for at least two datasets (e.g., Set14 and BSD100) among LN, LN* and i-LN\n    * I'll raise my rating if this concern is resolved\n* Explanation for the derivation of feature divergence in Figure 4 and other experimental results in some place (e.g., the paragraph in lines 260-267) would be helpful", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PABqIKoWIJ", "reviewer": "Reviewer_EZi7", "summary": "This paper studies the effect of Layer Normalization (LN) in Transformer-based image restoration networks, focusing on the observed instability and feature magnitude divergence during training. The authors analyze how the per-token LN operation may disturb spatial correlations and feature statistics, and propose a variant called i-LN, which normalizes across both spatial and channel dimensions with an input-dependent scaling term. Experiments across several image restoration tasks (SR, DN, DR, CAR) show modest but consistent improvements in stability and quantitative performance. Overall, the work offers a careful empirical examination of normalization behavior in low-level Transformer models, though the methodological change is relatively minor.", "questions": "1. Could the authors provide quantitative evidence of efficiency, such as runtime, FLOPs, or memory cost, to support the claim that the proposed holistic normalization incurs negligible computational overhead?\n\n2. Since i-LN aims to stabilize feature statistics, have the authors considered evaluating it under an All-in-One or real-world degradation setting to demonstrate robustness across diverse degradation types?\n\n3. The paper focuses on Transformer-based IR models — do the authors expect i-LN to generalize to non-Transformer or hybrid CNN–Transformer architectures? If so, could preliminary results or insights be shared?\n\n4. The analysis emphasizes improved stability and entropy preservation, but could the authors quantify this improvement (e.g., through variance reduction, entropy scores, or training convergence curves) to better support these claims?\n\n5. Given that the proposed method is relatively simple, could the authors discuss whether i-LN interacts with other normalization techniques (e.g., InstanceNorm, GroupNorm) or whether similar stability effects could be achieved by alternative normalization formulations?\n\nPlease also refer to the **Weaknesses**.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper focuses on a clear and well-defined problem — the instability of Layer Normalization in image restoration Transformers — and provides intuitive analysis to support it.\n\n2. The proposed i-LN module is conceptually simple and easy to implement, making it practical for existing Transformer-based IR frameworks.\n\n3. Experiments across several restoration tasks show consistent but moderate improvements, indicating the method’s general usefulness.\n\n4. The paper is clearly written and well-organized, making the motivation and results easy to follow.", "weaknesses": "### Main Concerns:\n1. Since the proposed i-LN is designed to stabilize feature statistics across varying inputs, it would be informative to evaluate its effectiveness in an All-in-One restoration setting, where a single model must generalize across multiple degradation types. This could further validate the claimed robustness and statistical consistency advantages.\n\n2. While the proposed i-LN improves feature stability by normalizing across spatial and channel dimensions, this inevitably increases the computational scope. The paper claims negligible overhead but provides no quantitative analysis to support this; it would be helpful to report the actual runtime or FLOPs to assess the trade-off between stability and efficiency.\n\n3. For a low-level vision paper, the visual comparisons are too limited and small in scale. Given that image restoration quality is best evaluated through perceptual inspection, more visual comparison samples are expected to be included in at least the appendix or supplementary materials, but this is missing.\n\n4. While the proposed i-LN is claimed to improve feature stability and robustness, the paper does not include any evaluation on real-world degradations, which are common and practically relevant in image restoration. Since the method is designed to better preserve low-level statistics, testing it on real-world datasets would be important to demonstrate its practical effectiveness and generalization beyond synthetic benchmarks.\n\n5. Despite the solid analysis, the performance gain is modest, which may limit perceived impact.\n\n6. The method is incremental in implementation—essentially a normalization variant—though well justified.\n\n7. Computational overhead or efficiency impact of holistic normalization is not deeply discussed.\n\n8. Could include more discussion on generalization beyond image restoration tasks (e.g., classification, segmentation)?.\n\n### Minor Concerns:\n1. The computation resources are expected, for example, the authors only mentioned in Line 236 via A6000 GPUs, but how many for each experiment?\n\n2. The code of the proposed method is expected to be released.\n\n3. The term “IR Transformer” is not clearly defined — since the architecture is largely identical to standard vision Transformers, the authors should clarify whether the analysis truly targets image restoration–specific behavior or applies more broadly to general Transformer models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zefYP0BTgz", "reviewer": "Reviewer_SRbU", "summary": "This paper investigates why Image Restoration (IR) Transformers often show unstable training behavior when using standard Layer Normalization (LN). The authors find that LN causes feature magnitudes to diverge and channel entropy to collapse, indicating a fundamental mismatch between LN and dense prediction tasks.\nTo address this, they propose i-LN, a simple drop-in replacement that applies spatially holistic normalization and input-adaptive rescaling. i-LN stabilizes training and improves performance across multiple IR tasks (super-resolution, denoising, deblurring, and artifact removal) and Transformer architectures.", "questions": "see the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Novel insight into a fundamental issue – The paper is the first to thoroughly diagnose the instability and abnormal training dynamics of IR Transformers caused by LN. The observations of feature divergence and entropy collapse are well-supported and enlightening.\n\n2. Strong theoretical grounding – The authors formalize the notion of structure-preserving transformations and mathematically show that LN fails to maintain spatial isomorphism, whereas the proposed i-LN satisfies these properties.\n\n3. Simple yet impactful method – i-LN is conceptually straightforward, easily implemented, and compatible with existing architectures without any retraining overhead.", "weaknesses": "1. The theoretical analysis is insightful but somewhat idealized — it assumes a simplified setting without the learnable affine parameters (β, γ) and does not account for the nonlinear activations that follow normalization. While this abstraction helps clarify the core argument, it may limit the generality of the conclusions in practical Transformer architectures.\n2. Qualitative evaluation could be expanded – The visual examples are convincing, but adding perceptual metrics (e.g., LPIPS, FID) or user studies would make the improvements more tangible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jXYmhBRM9k", "reviewer": "Reviewer_U1uB", "summary": "This paper analyzes the training dynamics of the image restoration Transformer, identifies the limitations of layer normalization, and proposes a global version of layer normalization that normalizes features holistically. Through theoretical analysis and experimental verification, its effectiveness in various tasks of image restoration is demonstrated.", "questions": "1. Could you provide the pseudo-code of the i-LN you proposed? This would be helpful for us to quickly get started with the implementation and accurately reproduce the results.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and the theoretical analysis is solid.\n2. The method is very simple and has been verified for its effectiveness in various IR tasks. I highly appreciate its motivation of specifically designing methods for low-level visual tasks. This is more impressive compared to simply transplanting methods from other fields directly.\n3.This method is plug-and-play and may become the standard approach for designing network structures in the field of low-level vision.", "weaknesses": "No obvious weaknesses are identified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)