Trust but Verify: Adaptive Conditioning for Reference-Based Diffusion Super-Resolution via Implicit Reference Correlation Modeling
Authors: Yuan Wang
Yuhao Wan
Siming Zheng
Bo Li
Qibin Hou
Peng-Tao Jiang
Venue: ICLR 2026 Poster (2026)
Keywords: diffusion_models
deep_learning
reference-based super-resolution
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper introduces a correlation-aware gating mechanism that adaptively balances fidelity and reference guidance in single-step diffusion super-resolution, improving robustness under varying reference quality.

Abstract: Recent works have explored reference-based super-resolution (RefSR) to mitigate hallucinations in diffusion-based image restoration. A key challenge is that real-world degradations make correspondences between low-quality (LQ) inputs and reference (Ref) images unreliable, requiring adaptive control of reference usage. Existing methods either ignore LQ–Ref correlations or rely on brittle explicit matching, leading to over-reliance on misleading references or under-utilization of valuable cues. To address this, we propose Ada-RefSR, a single-step diffusion framework guided by a "Trust but Verify " principle: reference information is leveraged when reliable and suppressed otherwise. Its core component, Adaptive Implicit Correlation Gating (AICG), employs learnable summary tokens to distill dominant reference patterns and capture implicit correlations with LQ features. Integrated into the attention backbone, AICG provides lightweight, adaptive regulation of reference guidance, serving as a built-in safeguard against erroneous fusion. Experiments on multiple datasets demonstrate that Ada-RefSR achieves a strong balance of fidelity, naturalness, and efficiency, while remaining robust under varying reference alignment. Code and models are available at https://github.com/vivoCameraResearch/AdaRefSR.

Published as a conference paper at ICLR 2026
TRUST BUT VERIFY: ADAPTIVE CONDITIONING FOR REFERENCE-BASED DIFFUSION VIA IMPLICIT REFERENCE CORRELATION MODELING
Yuan Wang Yuhao Wan'
Siming Zheng Bo Li Qibin Hou*
Peng- Tag Jiang
Nankai University vvvvo Bluelmage Lab, vivo Mobile Communication Co., Ltd.
"Coreesponding author: pt. siang@vivo...
ABSTRACT
Recent works have explored reference-based super-resolution (RefSR) to miti- gate hallucinations in diffusion-basee image restoration. A key challenge is that real-world degradations make correspondences between low-quality (LQ) inputs and reference (Ref) images unreliable, requiring adaptive control of reference us- age. Existing methods either ignore I-QRRef correlations or rely on brittle explicit matching. leading to over-reliance on misleading references or under-utilization of valuable cues. To address this, we propose Ada-RelSR, single-step diffu- sion framework guided by a "Trust but Verify" principle: reference information is leveraged when reliable and suppressed otherwise. Its core component, Adaptive Implicit Correlation Gating (AICG), employs learnable summary tokens to dis- till dominant reference patterns and capture implicit correlations with LQ features. Integrated into the attention backbone, AICG provides lightweight, adaptive reg- ulation of reference guidance, serving fusion. Experiments on multiple datasets demonstrate that Ada-ReiSR achieves strong balance of fidelity, naturalness, and efficiency, while remaining robust under varying reference alignment. Code and models are available at this url.
built-in safeguard against erroncous
INTRODUCTION
Diffusion-basee single-image super-resolution (SISR) methods (Wang et al., 2024b; Wu et al., 2024b) have recently attracted significant attention, owing to the strong generative priors they in- herit from large-scale text-to-image models. While capable of producing visually pleasing results, these methods often suffer from hallucinations and -ittr---istributionn artifacts (Aithal al., 2024), fabricating or omitting details due to the inherently ill-posed nature of SISR.
To address these challenges, researchers have explored reference-based super-resolution (RefSR) (Jiang et al., 2021; Zhang et al., 2023a), which incorporates an auxiliary reference image (Ref) alongside the LQ input to provide complementary high-frequency details for guided reconstruction. The effectiveness of RefSR depends on establishing meaningful cross-modal correspondences be- tween IQ and Ref features so that useful information can be transferred. Yet, when LQ images are heavily degraded, matching becomes unreliable, rendering it critical to regulate how much the model relies on references versus its intrinsic SR capability.
To balance this reliance, some strategies have been proposed, most of which can be interpreted as weighting mechanisms, differing mainly in how the weights are determined. PFStorer (Varanka et al., 2024), designed for face SR, employs global learnable vector that uniformly controls the reference branch across inputs, but fails to adapt to different alignment qualities (c.g., well-aligned poorly matched pairs). ReFIR (Guo al.. 2024), on the other hand, generates spatial gates from token-wise similarity maps, but these explicit correlation metrics are easily disrupted by noise and long-tail distributions (where majority identical tokens dominate the computation over minority critical ones). In practice, such weighting-based approaches often suffer from two issues (cf. Fig. 1): (1) The weighting mechanism may result in excessive insufficient reliance on references-over injecting reference cues (Fig. top row, green-boxed column) or under-utilizing them (Fig. top
Work done when interning vivo.

---

Published as a conference paper at ICLR 2026
Reference
HQ
S3Diff
SUPIR+ReFIR
SeeSR+ReFIR
Ours
Figure 1: Visual comparisons: S3Diff is single-imag imgge generation network, and ReFIR is the cur-
rent SOTA for reference based restoration. Our method not only outperforms ReFIR leveraging reference details but also shows stronger robustness against degradations than S3Diff.
row, red-boxed columm; middle row, red-boxed regions); (2) Semantic mismatches caused by noisy explicit correspondences (e.g., duplicated bird eyes in Fig. 1 bottom row, purple-hoxed region, due to ReFIR's erroneous matching regions to bird eyes), which introduce unnatural regions. Fig. 2(a) and Fig. 2(b) illustrate these behaviors: PFStorer's global gating fails to adapt to alignment quality, while ReFIR's explicit correlation maps are disrupted by noise, leading to unreliable gating.
Gating
(b) Explicit Correlation
Gating
(c) Implicit
Gating (Ours)
ref-aligned tokens
erc tokerss
SrC Toloons
correlation
output tokens
correlation
output
output
ref-aligned
giobal
Tusion tokens
leamsatle vector
ref-eligned fusion tokens
summary
TONAITS
learnable takers
Figure 2: Comparison of gating strategies. Prior methods either use non- ion-interactive global gating (PFStorer) depend on explicit token correlations that are easily disturbed by noise (ReFIR). In contrast, we introduce implicit, correlation-driven gating that selectively activates useful reference cues. "sre tokens" denote main-branch features, "ref tokens" are raw reference features, and "ref- aligned fusion tokens" reference features aligned to the source space through feature fusion. Here, G denotes the per-token gating values generated by our AICG, while represents the full token-to-token similarity map used in explicit correlation-based gating methods like ReFIR.
Our key insight addresses the challenges outlined above. Reference fusion should maximize the use of reliable regions while suppressing misleading ones to prevent unnatural artifacts. Following a "Trust bul Verify" paradigm, the model first trusts reference regions likely aligned with the LQ input to enhance detail recovery, and then verifies and attenuates unreliable regions to reduce dis- tortions. This approach naturally addresses both the over/under-rellance on references and semantic mismatches weighting-basc based methods.
Motivated by this principle, propose Ada-RefSR, single-step reference-based diffusion frame- work. Its core component, Adaptive Implicit Correlation Gating (AICG) (Fig. 2(c)), employs leamable summary tokens to implicitly model latent correspondences between LQ and reference features. These tokens function as a compact intermediary, extracting salient reference cues while filtering out semantically inconsistent information. When applied after reference injection, AICG
2

---

Published as a conference paper at ICLR 2026
adaptively modulates reference contributions, providing on-the-fly correction of erroneous feature integration. This mechanism preserves the fidelity and naturalness of the generated content, while enhancing robustness in ettreval-augmeete scenarios characterized by variable reference quality. Our contributions can be summarized as follows:
AICG: Adaptive Implicit Correlation Gating, We propose AICG, lightweight implicit correlation gating module that directly addresses a key challenge in RefSR: how reli- ably use reference information to restore LQ inputs degraded by real-world artifacts. By reusing existing projections in the attention module and introducing only a few learnable summary tokens, AICG implicitly models LQ-Ref correlations while adding negligible computational overhead.
Ada-RefSHe Strong generalization, robustness, and speed. Built upon AICG, Ada- RefSR achieves stable reference-based enhancement across diverse tasks and degradation scenarios. Its single-step diffusion design provides over 30x speedup compared multi- step RefSR baselines, enabling fast and robust SR in both aligned and mismatched refer- ence conditions.
RELATED WORK
2.1 DIFFUSION MODEL FOR IMAGE RESTORATION
Diffusion--bai based super-resolution has rapidly advanced in recent years, with variety of models ploring different ways to exploit text-to-image generative priors. StableSR (Wang et al., 2024b) in- troduced time-aware encoder balance fidelity and naturalness, while DiffBIR (Lin et al., 2024) first restores base results before refining details via diffusion. PASD (Yang et al., 2024) incorpo- rated degradation removal module with pixel-aware cross-attention, and SeeSR (Wu al., 2024b) enhanced semantic understanding through degradation-awwree prompt extractor. Most of these ap- proaches rely on multi-step denoising, which is computationally expensive. To improve efficiency, recent works turn to single-step paradigms: SinSR (Wang et al., 2024c) distills one-step model from a multi-step teacher, OSEDiff (Wu al., 2024a) simplifies ControlNet with KL regularzzation, S3Diff (Zhang et al., 2024) adopts degradation- -unded LoRA, and PISA-SR (Sun et al., 2025) de- couples pixel and semantic-level objectives. Despite these advances, diffusion-baar SR still suffers from intrinsic hallucination, where spurious structures emerge due to insufficient constraints.
2.2 REFERENCE BASED IMAGE SUPER RESOLUTION
RefSR enhances SISR by leveraging auxiliary reference images. Early works such as SRNTT (Zhang et al., 2019) treat as neural texture transfer, while TTSR (Yang et al., 2020) introduces cross-scale attention for texture alignment. To preserve domain-specific features, Hu et al. (2020) utilized 3D facial priors as reference guidance. MASA-SR (Lu al., 2021) adapts spatial features to handle distribution discrepancies, C2-Matching (Jiang et al., 2021) exploits contrastive correspon- dence with knowledge distillation, and DATSR (Cao et al., 2022) applies deformable attention for multi-scale transfer. LMR (Zhang et al., 2023a) further extends the paradigm to multi-reference settings, and PFStore (Varanka et al., 2024) employs Icarnable vectors to balance reference influ- ence. Despite these advances, most methods assume manually specified registration between LQ and reference images, limiting real-world applicability. ReFIR (Guo et al., 2024) moves toward sttrivall-uggmene RefSR, but its training-free design and fixed reference weighting restrict adapt- ability in practice.
3 METHODOLOGY
In this section, we introduce our reference-basee super-resolution framework named Ada-RelSR, which aims recover high-quality images from low-quality (I.Q) observations X1, by adaptively leveraging an external reference image Xref. either retrieved from a large-scale database or directly provided by the user. Our method consists of two key components: (1) reference attention module for injecting fine-grained details, and (2) a correlation-aware adaptive gating mechanism to balance reference utilization and input consistency. Figure illustrates the overall pipeline.
3

---

Published as a conference paper at ICLR 2026
(4) Reference based Image Restoration Backhone (Ada-RefSR)
(b) Adaptive Implicit Correlation Gating (AICG)
ReforceeNeie
SA CK
diastep
Surmniyy
Dve (itt
radiled by
Xap
AICE
Figure 3: Overview of our framework. It comprises two components: (a) a reference based restora- tion backbone, and (b) a correlation-aware adaptive gating mechanism.
3.1 TRUST: DIRECT REFERENCE FEATURE INJECTION
Maximizing Reference Utilization in the Trust Phase. The "trust" phase aims to maximize the utilization of reference features, ensuring that all potential LQ- reference matches are captured for detail enhancement. The intention is to ensure that no potentially useful correspondence is dis- carded at this stage, even at the risk of introducing mismatches. We build on single-step diffusion SR backbone that already provides strong fidelity for Xıg. and augment it with reference-guide detail extraction without upfront filtering. Existing methods are insufficient: IP-Adapte--style ap- proaches (Ye et al., 2023) transfer only global identity, missing local matches, while ControlNet- style methods (Zhang et al., 2023b) discard valid features under spatial misalignment (Hu et al., 2023). To overcome these limitations, we employ ReferenceNet (Hu et al., 2023) instantiated with SD-Turbo (Sauer et al., 2023) and fix its timestep to I, enabling full-stage, fine-grained reference feature extraction.
Injecting Reference Features via Attention. To leverage these extracted features, we employ a Reference Attention (RA) module (Fig. 3(a)). RA is initialized by copying the backbone's self-attention weights, with backbone features as queries and ReferenceNet features as key-value soinss-allwiinn every LQ query to selectively draw from all reference cues and directly realize the "trust" objective. Let the backbone feature at given layer be Hare
R lare xd and the correspond- ing ReferenceNet feature be Hnef E Rhres xd, We define the attention projections as:
Q Hore WQ, K = WK,
V V=Hre Wy.
where Wo, WK, Wv E BRdxd are learnable projection matrices. The vanilla RA is then computed as:
Hout ZeroLinear (Softmax
OK Vd
Harcy
where ZeroLincar stabilizes training in early stages and prevents disruption of the original Stable Diffusion features (Rombach al., 2021). The above method extracts detail-rich, spatially mean- ingful features via ReferenceNet and fuses them with LQ backbone features through RA module without requiring spatial alignment. Residual connections further preserve the prior knowledge of the original super-resolution network.
While vanilla RA effectively injects reference details, its indiscriminate fusion often causes local semantic inconsistencies and unnatural textures (Fig. 4). Attention map visualizations reveal the root cause: the "trust" phase inevitably allows LQ regions to pull features from semantically mismatched areas in the reference (e.g., non-eye regions borrowing eye features). These issues underscore the need for a complementary verification phase that selectively regulates reference influence.
3.2 VERIFY: ADAPTIVE IMPLICIT CORRELATION GATING
An effective way to realize this verification is through gating mechanism, mediating between baseline restoration fideelt and setrrence-guider enhancement. However, existing gating strategies remain limited. PFStore employs a global learnable vector, disregarding LQ-Ref feature correlations and thus failing to adapt to varying alignment quality. ReFIR constructs spatial gates via token- wise similarity but suffers from two inherent drawbacks: (1) explicit correspondences are highly
4

---

Published as a conference paper at ICLR 2026
(4) lmage
Spe result (c)
T.a-BA rwsite
(r) Vant RE-
Vanilla
Figure 4: Vanilla RA and ReFIR both introduce artifacts by overusing irrelevant reference regions (left: duplicate eye; right: cluttered background). Our gating mechanism suppresses these activa- tions, leading to more natural results. Zoom in for better visualization.
susceptible to noise (Bolya et al., 2023) and long-tail biases; (2) reliance fixed user-specified weight (0.5) prevents adaptation to varying alignment quality. These weaknesses manifest clearly in comparative experiments (Tab. 2).
To address these limitations and fulfill the "Verify" phase of our "Trust but Verify" paradigm, we propose an Adaptive Implicit Correlation Gating (AICG) mechanism (Fig. 3(b),, which implicitly models LQ-Ref correlation to adaptively regulate reference guidance. While inspired by the high- level idea of learnable tokens DETR (Carion et al., 2020), AICG differs fundamentally: Instead of serving decoding queries, our tokens summarize reference features to provide implicit, token- wise reliability estimates for gating. This design suppresses noise and mismatches while retaining essential structures. Importantly, AICG remains lightweight, introducing only a few learnable tokens and reusing existing projections and intermediate variables within the RA module.
Reference Summarization via Learnable Tokens. We introduce set of learnable summary to- kens Ts E Ad that compactly summarize the reference features. First, we map them into key space using the reference-attention projection:
Ts KG
(3)
where W K is the key projection matrix in Eq. 1. We then compute the correlation between summary tokens and reference keys:
K Kswm Softmax
SK' Va
K KERMA Mxd
where K R is the reference key sequence in Eq. 1. Intuitively, K Keum captures the most
informative textures and foreground structures in a compact M-token form (M < Lref), incurring negligible additional cost.
Implicit Reference Correlation based Gating, Next, we compute the attention between LQ
queries and the summarized reference:
QK Vd
Smap Sofumax
E ERAMM
(5)
where Q are the LQ query sequence in Eq. 1. We aggregate the logits across M summary tokens and attention heads to form a scalar gating value for each LQ token:
A
G = M 1=1
>
Smaplis
E 1R4x11
(6)
where a--) denotes the sigmoid function. Finally, the vanilla RA output adaptively modulated as:
Hot ZeroLinear G RA(H Hrcr)) + H
(7)
where denotes element-wise scaling along the token dimension.
5

---

Published as a conference paper at ICLR 2026
Discussion. Our adaptive gating mechanism embodies the "Trust but Verify" principle in reference-basee based diffusion SR. Instead of indiscriminately injecting reference information, it adap- tively regulates contributions--truse
semantically aligned regions while suppressing mismatches (see Fig.4, Fig.6)-thereby mitigating hallucinations and artifacts. In addition, learnable summary tokens provide compact interface between large-scale reference features and I.Q queries, reinforc- ing robustness and making implicit correlation modeling both adaptive and reliable (see Fig. 7),
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Training Datasets and Implementation Details. We adopt S3Diff (Zhang al., 2024)
baseline, fine-tuning only the new reference attention module while freezing other components preserve pre-trained super-resolution priors. Training data include: (1) synthetic SR datasets (DIV2K, DIVSK, Flickr2K), with 1024 x 1024 regions cropped into two 512 x 512 sub-regions (HQ and Ref). Ref images augmented via random rotation (details in appendix A.1); (2) a face reference matching dataset (Li et al., 2022) with predefined HQ-Ref triplets. LQ images for all datasets use the RealSRGAN degradation pipeline (Wang et al., 2021). To improve robustness, 20% of HQ-Ref pairs in synthetic datasets are randomly replaced with irrelevant samples. Training uses two NVIDIA A40 GPUs, Adam optimizer (Kingma Ba, 2014), initial learning rate x 10-5, batch size 16, and 11K iterations for convergence. The training objective follows the original de- sign of S3Diff, combining an L2 reconstruction loss with perceptual and GAN losses, with detailed formulation provided in appendix B.
Evaluation datasets and Metrics. To evaluate our method's reference effectiveness in dense matching, use datasets CUFEDS (Zhang et al., 2019) and WRSR (Jiang al., 2021), pro- cessed following ReFIR's protocol (Guo et al., 2024). Following Varanka et al. (2024), we test 40 human identities: 5 non-overlapping high-quality (HQ) pairs per identity, yielding 162 test pairs (all 512x512). To verify category-based retrieval enhancement, we construct a self-collected bird dataset ( 8,460 HQ images; 66 as HQ images, others as retrieval database, all 512x512; details in appendix A.2). Consistent with prior works, we use four reference- based metrics (PSNR, SSIM (Wang et al., 2004), LPIPS (Zhang et al., 2018), FID (Heusel et al., 2017)) and three no-reference metrics (NIQE (Zhang et al., 2015), CLIPIQA (Wang et al., 2023), MUSIQ (Ke et al., 2021)).
Compared Methods. To evaluate our method in mitigating hallucinations in diffusion based mod- els, we compare against ReaISRGAN (Wang et al., 2021), PISA-SR (Sun et al., 2025), S3Diff (Zhang et al., 2024), OSEDiff (Wu et al., 2024a), and large-model reference SR approaches like ReeSR+ReFIR and SUPIR+ReFIR (Guo et al., 2024) on general scene datasets. For dense-scene reference datasets, we add C2Matching (Jiang et al., 2021) and DATSR (Cao et al., 2022) to eval- uate reference generation. For face datasets, we further compare with DMDNet (Li et al., 2022), CodeFormer (Zhou et al., 2022), FaceMe (Liu et al., 2025) and InstantRestore (Zhang et al., 2025) validate face-specific reference SR performance. To ensure fairness, all use a single reference image, with the number of reference images set to for all methods.
4.2 COMPARATIVE RESULTS
Quantitative Comparison. We present quantitative results on four datasets with varying reference matching degrees, as shown in the table. Several observations can be made:
Overall performance. Our method achieves the best results across most reference-based metries (PSNR, SSIM, LPIPS, FID), consistently surpassing the baseline S3Diff and other diffusion based model. On no-reference metrics (NIQE, MUSIQ, CLIPQA), it attains com- parable scores, showing that reference cues are effectively exploited without compromising perceptual naturalness.
Comparison with reference-based methods. Compared with ReFIR, our approach deliy- ers substantially higher fidelity, confirming its advantage in structural and textural restora- tion. Notably, SUPIR+ReFIR underperforms our method, suggesting that ReFIR lacks general adaptability across different SR backbones. Moreover, method outperforms

---

Published as a conference paper at ICLR 2026
Table 1: Algorithm performance comparison across multiple datasets. The best performance is highlighted in red bold, the second- best in blue bold, and rows corresponding to "Ours" are shaded with light background. Notation: S-x denotes inference with time steps; indicates methods that use reference images.
Dataset
Method
PSNR
SSIM
LPIPS
4
FID 4
NIQE 4
MUSIQ 1 CLIPQA f
C2-Matching*
21.0199
0.5303
0.8040
292.5921
9.5425
16.8868
0.1227
DATSR*
21.0196
0,5306
0.8071
292.7726
9.6305
16.8736
0.1269
Real-ESRGAN-SI
20,3084
0.5543
0.3697
175.9116
3.8861
64.7123
0,4214
CUFED5
S3Diff-S1
20.4594
0,5234
0.3544
160.1337
3.7096
63.8519
0,5238
OSEDifI-SI
18.8748
0.4935
0.3669
150.7504
4.0005
65.2446
0,5077
PISA-SR-SI
20.3371
0.5286
0.3707
164.6927
4.1992
64.2840
0,5043
SUPIR-ReFIRRSSS0
18 18.9994
0.4711
0,4359
151.2700
4.2745
62.7384
0,4235
SecSR+ReFIR -550
20.2169
0,5255
0,3452
137.3693
3.7426
70,5375
0,5538
Ours-S1
20.4843
0,5461
0,2894
127.8945
3.8701
67.9070
0,5174
C2-Matching*
22.8766
0.5784
0.7471
1606
9.0121
19.9772
0.1527
DATSR*
22.9017
0,5820
0.7583
153,5259
9.3999
20.1124
0.1613
Real-ESRGAN-SI
22.1491
0.5974
0.3631
97.9142
3.6976
64.5176
0.6076
WRSR
S3Diff-S1
21.9122
0.5620
0.3542
63.8237
4,4171
63.8237
0,6149
OSEDifI-SI
21.3263
0.5603
0.3261
71.5148
3.7364
71.1935
0,6899
PISA-SR-S1
21.8813
0.5682
0.3383
62.9958
4.0359
69.5187
0.6407
SUPIR+ROFIR
21.0134
0.5378
0.3949
76.4993
3.6407
70.1242
0.6219
BASR+RRFFIR-SS55
21.8334
0.5673
0.3435
61.9597
3.8919
72.1520
0.7120
Ours-S1#
21.9722
0.5777
0.3061
53,2811
3.7429
69.8608
0.6612
Real-ESRGAN-SI
25.0809
0.7044
0.5306
178.8849
8,8496
29.3109
0.3571
S3Dift-S1
24.8422
0.7106
0.2903
60,9452
5.3195
73.1125
0.8285
OSEDifI-SI
23.7861
0.7086
0.2949
49.1911
4.9796
74.7567
0,8451
Bird
PISA-SR-S1
24.4891
0.7003
0,2816
67.9959
4.8513
74.3633
0.8373
SUPIR-REFIR*-SSO
23.8344
0.6974
0.3811
94.9298
5.4422
63.6740
0.6478
SeeSR+ReFIR
23.7211
0.6985
0,2945
52.4038
5.6182
75.1056
0.8617
Ours-S1#
25.2998
0.7292
0.2536
36.4249
5.1771
72.0039
0.8303
Code-former
26.2677
0.7082
0.3798
99.6599
5,4105
63.9886
0,5192
DMDNet*
26,8993
0.7472
0.2316
56.6262
4.6030
72.9495
0.6395
Real-ESRGAN-SI
26.1678
0.7142
0.5113
162.4299
8.9504
26.4788
0.3630
Face
S3DifT-S1
26.6881
0.7344
0.2189
51.1219
5.0599
74.1095
0.6963
OSEDifI-SI
25.5302
0.7434
0.2263
55,0428
4.9550
71.7721
0,6459
PISA-SR-SI
26.2433
0.7254
0.2126
48.9154
4.5423
75.9855
0.7191
SUPIR-ReFIR*SST
25.7231
0.7062
0.2623
57.1432
3.9686
73.0854
0,6260
Sce5R+ReFIR*-SS -550
26.3191
0.7375
0.2293
57.2910
5.2885
75.4230
0,7236
FaceMe*-S50
26.6591
0.7245
0.2500
54.3278
4.7295
72.4144
0,6483
InstantRestore*-S)
26.2245
0.7350
0,2070
51.2283
5.9473
69.7209
0.5726
Ours-S1#
27.1271
0.7523
0.1749
42.7042
4.4880
74.3828
0.6453
DMDNet and InstantRestore across all metrics, highlighting its effectiveness in face refer- ence super-resolution.
Task-specific observations. In scene-level reference SR, traditional approaches (e.g.. C2- Matching, DATSR) yield higher PSNR/SSIM, while diffusion-basse methods emphasize perceptual realism. On the WRSR benchmark, method also demonstrates strong fi- delity-peroeption balance under real-world degradations. On bird and face benchmarks, method consistently outperforms existing approaches in reference- only minor variations in no-reference metrics due to imperfect LQ-Ref Ref alignment. Notably, in face reference SR, method surpasses recent SOTA approaches (Faceme, InstanceRe- store) by leveraging the baseline SR prior via reference attention, while AICG implicitly filters irrelevant reference regions prevent misleading feature injection.
metrics, with
Overall, method shows clear advantages in seference-based metrics while maintaining strong competitiveness in no-reference evaluations, demonstrating its effectiveness in utilizing reference information without compromising perceptual quality.
7

---

Published as a conference paper at ICLR 2026
HQ
LQ
C2-Matching
DATSR
SUPIR+ReFIR
SeeSR+ReFIN IR
Reference
RealSRGAN
PISA-SR
OSEDifF
S3DiFF
Ours
Figure 5: A Case study on WRSR dataset: To highlight differences, we cropped key regions from HQ images and their References, showing comparisons across all methods.
Qualitative Comparison. In Fig. 5, we compare representative approaches for scene reference generation. Traditional methods (e.g.. C2-Matching, DATSR) fail under strong degradations, while single-step SR models (PISA-SR. S3Diff, OSEDiff) often produce distorted details or block arti- facts. Multi-step SR with ReFIR methods alleviate some issues but still suffer from severe hal- lucinations-ffe for instance, SeeSR+ReFIR reconstructs the wrong bird species and SUPIR+ReFIR generates incorrect background-indiee indicatinn limited effectiveness in reference utilization. In con- trast, our method achieves a more reliable balance between fidelity and naturalness.
4.3 ABLATION STUDY
Table Ablation study on WRSR Dataset and Bird Dataset, with the best performance highlighted
in bold.
WRSR dataset
Face dataset
Method
PSNR f
SSIM 1
MUSIQ
CLIPIQA f
PSNR
SSIM t
MUSIQ
CLIPIQA f
Vanilla
21.9508
0.5737
69.7028
0.6471
27.0795
0.7495
74.3385
0.6444
Global
21.6267
0.5610
69.0841
0.6281
27.0613
0.7498
73.1474
0.6308
ReFIR
21.7753
0.5668
68.9322
0.6356
26.9430
0.7473
74.3762
0.6483
AICG
21.9722
0.5777
69.8608
0.6612
27.1271
0.7523
74.3828
0.6453
Full Comparison of AICG with Existing Gating Mechanisms, To further verify the efficacy of our AICG in balancing intrinsic SR priors and reference guidance, we conduct additional ablation studies on the WRSR and Face datasets, with four method variants: (i) Vanilla (Vanilla RA): re- moves the gating term to directly use reference features: (ii) Global (Global Iearnable weight+RA):
adopts the global learnable weight from (Varanka et al., 2024); (iii) ReFIR (ReFIR+RA): integrates ReFIR's s spatial gating (Guo et al., 2024) into vanilla reference attention; (iv) AICG (AICG+RA): our full proposed method. All variants are trained for 11K steps under identical settings.
The results in Table 2 show that AICG consistently outperforms the Vanilla baseline across datasets, confirming the effectiveness of our design. Unlike ReFIR, which shows some gains on the Face dataset mainly in no-reference metrics but fails dense scenarios like WRSR, AICG maintains stable improvements across both structured cases (e.g., faces with clear local correspondences) and dense scene-level references. Guided by our "Trust but Verify" principle, it exploits reliable while suppressing mismatches, achieving robust gains under diverse I.Q-Ref alignment conditions.
Visualization of AICG. Fig. shows AICG filters out misleading reference cues: irrelevant back- grounds suppressed, and even if key structures are absent, generated regions are down-weed weightee This shows AICG mitigates artifacts and regulates local details, aligning with "Trust but Verify" design philosophy.
8

---

Published as a conference paper at ICLR 2026
AICG Mask shown on GT image
Masked
Retrieved Ref
Figure 6: Visualization of our AICG mechanism (threshold 0.5). The gate suppresses irrelevant background regions while adaptively masking missing details (e.g., the bird's head), enabling fine- grained control of reference utilization.
Figure 7: Visualization of learnable summary tokens. Each token captures distinct semantic regions, such as human parts, sky, grass, or bird-specific features, while others represent background context.
Interpretability of Learnable Summary Tokens. In our network, learnable queries act as clus- tering centers that aggregate reference regions into compact semantic tokens. Fig. 7 visualizes sev- cral samples, showing that tokens specialize distinct patterns-e.g. human parts, sky, grass, or bird-specific cues like beaks and feathers-whis others capture background context. These results confirm that our token design effectively condenses high-dimensional reference features into inter- pretable, semantically meaningful representations.
4.4 PERFORMANCE ANALYSIS: ROBUSTNESS AND EFFICIENCY
Cura
DNON
SI0WT Dees lis)
SURR ttane line
Table 3: GPU Memory and In-
PSHR (dB)
SSIM
LPIPS
ference Time at Different Resolu- tions.
3150
30.3
5120
512 x 512
Mem (GB)
Time (s)
3330
32
26
02
04
08
S3Diff
5.82
0.30
Ada-RefSR
12.66
0.41
0.57
SecSR+ReFIR
11.94
9.03
1024 x 1024 Mem (GB)
Time (s)
2.8 0. 6.4
S3Diff
7.18
0.74
Masching
Mooching
Figure 8: Comparison of different methods on the face dataset under varying reference matching ratios. Our method main- tains superior reference-based metrics and naturalness even with mismatched references.
Ada-RefSR
15.54
1.35
SecSR+ReFIR
18.95
40.23
Robustness of our method. As shown in Fig. 8, both our method and SUPIR+ReFIR benefit from higher LQ-Ref alignment. However, SUPIR+ReFIR falls below its baseline (SUPIR) when the ratio < 0.7. showing sensitivity to noisy references. In contrast, our method consistently sur- passes its baseline (S3Diff) even ratio=0 and achieves no-reference scores close HQ images,

---

Published as a conference paper at ICLR 2026
demonstrating stronger robustness. Notably, DMDNet even degrades in PSNR/SSIM, highlighting the difficulty of maintaining fidelity under misaligned references.
Efficiency and Model Size of Ada-ReiSR. Table 3 provides a comprehensive analysis of our model's performance footprint. Our proposed Ada-ReISR is nearly 30x faster than the optimization-heay SeeSR+ReFIR model when generating 1024 x 1024 images, underscoring its superior efficiency. Our core model contains 2678.88 89 M parameters about twice that of S3Diff (1326.77 77 M). Among them, only 61.98 M parameters from reference attention and 0. 0.20 M from learnable summary tokens are trainable, enabling effective fine-tuning, Despite this larger capacity, the inference time compared to S3Diff only roughly doubles. This an expected trade-off for the necessary reference processing and the enhanced personalization capability. Future work will ex- plore lightweight injection strategies, such as token pruning or sparse attention, to further minimize computational cost.
Efficiency of AICG. A key design objective of AICG is to maintain minimal overhead relative to vanilla Reference Attention (RA). While ReFIR incurs a substantial cost increase of 16.0% due its explicit Lice x Leef correlation matrix, AICG replaces this operation with an implicit estimation using only M-16 summary tokens. confirmed by the numerical analysis in Appendix (Table 6), this design adds merely 0.13% overhead on top of RA and eliminates the need to materialize large similarity matrices, offering both computational and memory efficiency.
CONCLUSIONS
We proposed Ada-RefSR, a single-step reference-based super-resolution (RefSR) framework grounded in the "Trust but Verify" principle, addressing a key challenge in RefSR- lever- aging reference information under real-world degradations, where LQ-Ref Ref mismatches often cause existing methods to overuse misinterpret reference cues. Central to design is the Adap- tive Implicit Correlation Gating (AICG) mechanism, which not only implicitly models LQ-Ref correlations to generate confidence-aweer gates but is also lightweight, introducing minimal com- putational overhead. This allows Ada-RefSR to exploit reliable reference cues effectively while suppressing misleading ones without sacrificing efficiency. Extensive experiments across diverse datasets confirm that Ada-RefSR achieves a strong balance between fidelity and perceptual qual- ity, consistently outperforming state-of-the-as RefSR methods. Future directions include exploring patch-wise reference guidance for finer control and developing alternative injection mechanisms to further improve flexibility and robustness under diverse reference conditions.
REPRODUCIBILITY STATEMENT
We provide comprehensive details ensure reproducibility of our work. The construction of our synthetic training dataset and the bird retrieval benchmark are described in Sec. A. and Sec. A.2, respectively. Our training objectives, including reconstruction, perceptual, and adversarial losses, are formally defined in Sec. B. Implementation details such as network architecture, optimization settings, and training schedules are reported in Sec. section 4. To further support reproducibil- ity, submit in the supplementary materials the complete bird retrieval test dataset well as the face test images used in our experiments. Other public datasets (DIV2K, DIV8K. Flickr2K, WRSR, CUFED5) are available online and can be accessed following the protocols described in the appendix.
USE OF LARGE LANGUAGE MODELS (LLMS)
We used large language models (LLMs) solely for writing assistance and literature understanding (e.g., g. polishing style, improving clarity, summarizing related works). All ideas, methodology. periments, and analyses were conducted by the authors, who take full responsibility for the content of this paper.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Sumukh K Aithal, Pratyush Maini, Zachary Lipton, and J Zico Kolter. Understanding hallucinations in diffusion models through mode interpolation. Advances in Neural Information Processing 2024.
Systems,
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your VIT but faster. In International Conference on Learning Representations, 2023.
Jiezhang Cao, Jingyun Liang, Kai Zhang, Yawei Li, Yulun Zhang, Wenguan Wang. and Luc Van Gool. Reference-based image super-resolution with deformable attention transformer. In European conference on computer vision, 2022.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Hang Guo, Tao Dai, Zhihao Ouyang, Taolin Zhang, Yaohua Zha, Bin Chen, and Shu-tao Xia. Re- fir: Grounding large restoration models with retrieval augmentation. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. (66593-46621 Curran Associates, Inc., 2024. doi: 10.52202/ 079017-1478. URL https: /proceed ngs neurips wctpaper_ppeceeeeeeeeee fwwee//aaeaepeeeeeereeeeeeeeeeee 2024/finH 5285552cd90
6c 60 PoapeeaaapPpper--offeeeeeeeo
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by two time-scale update rule converge to local nash equilibrium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gameett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL http:p/:/p..ccopops..o.. neurips cwtpsccrpppppee////eeeeee http://///./eeaae11100//eeeeeeee
Paper.pdf.
Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. arXiv preprint arXiv:2311.17117, 2023.
Xiaobin Hu, Wenqi Ren, John LaMaster, Xiaochun Cao, Xiaoming Li, Zechao Li, Bjoern Menze, and Wei Liu. Face super-resolution guided by 3d facial priors. In European Conference Computer Vision, pp. 763-780. Springer, 2020.
Yuming Jiang, Kelvin CK Chan, Xintao Wang, Chen Change Loy. and Ziwei Liu. Robust reference-basee super-resolution via c2- 22-matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2103-2112, 2021.
Junjie Ke, Qifei Wang, Yilin Wang. Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5128-1188588 5137, 2021. doi: 10.1 0..1099CCCV49922.22000000
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
Xiaoming Li, Shiguang Zhang, Shangchen Zhou, Lei Zhang, and Wangmeng Zuo. Learning dual memory dictionaries for blind face restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Bo Dai, Fanghua Yu, Wanli Ouyang, Yu Qiao, and Chao Dong. Diffbir: Towards blind image restoration with generative diffusion prior, 2024.
11

---

Published as a conference paper at ICLR 2026
Siyu Liu, Zheng-Peng Duan, Jia OuYang, Jiayi Fu, Hyunhee Park, Zikun Liu, Chunle Guo, and Chongyi Li. Faceme: Robust blind face restoration with personal identification. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
Liying Lu, Wenbo Li, Xin Tao, Jiangbo Lu, and Jiaya Jia. Masa-sr: Matching acceleration and spatial adaptation for reference-based image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6368-6377, 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.0711 193, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL
https: org/ wwwp////////ooooo//ooooo/ea.aeaa 221.. 000000000.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis- tillation, 2023. URL
org/ abs abss/231... 1///2111.100...
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 140v:1140.15566 2014.
Lingchen Sun, Rongyuan Wu, Zhiyuan Ma, Shuaizheng Liu, Qiaosi Yi, and Lei Zhang. Pixel- level and semantic-level adjustable super-resolution: A dual-lora approach. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2025.
Tuomas Varanka, Tapani Toivonen, Soumya Tripathy, Guoying Zhao, and Erman Acar. Pistorer:
Personalized face restoration and super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2372-2381, 2024.
Ao Wang, Hui Chen, Lihao Liu, Kai Chen, Zijia Lin, Jungong Han, and Guiguang Ding. Yolov10 Real-time end-to-end object detection. arXiv preprint arXiv:2405.144: 2024a.
Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In AAAI, 2023.
Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. International Journal of Computer Vision, (22(12):52995944 2024b.
Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pp. 1905-1914, 2021. doi: 10.109//CCVW54120.2222 00217.
Yufei Wang, Wenhan Yang, Xinyuan Chen, Yaohui Wang, Lanqing Guo, Lap-Pui Chau, Ziwei Liu, Yu Qiao, Alex C Kot, and Bihan Wen. Sinsr: diffusion-based image super-resolution in a single step. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 25796 25796-25805, 2024c.
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions Image Processing, 13(4):600-612, 2004. doi: 10. 10.1100//PPP.003.300
Rongyuan Wu, Lingchen Sun. Zhiyuan Ma. and Lei Zhang. One-step effective diffusion network for real-world image super-resolution. arXiv preprint arXiv:2406.088 00V:22006.08177,, 2024a.
Rongyuan WU, Tao Yang. Lingchen Sun, Zhengqiang Zhang, Shuai Li, and Lei Zhang. Seesr: Towards semantics-aware real-world image super-resolution. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 25456-25461 2024b.
12

---

Published as a conference paper at ICLR 2026
Fuzhi Yang. Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. Learning texture transformer In CVPR, June 2020.
network for image
Tao Yang, Rongyuan Wu, Peiran Ren, Xuansong Xie, and Lei Zhang. Pixel-aware stable diffu- sion for realistic image super-resolution and personalized stylization. In European conference computer vision, pp. 74-91. Springer, 2024.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models, 2023. URL //apivi org. 2b30/ 06721.
Aiping Zhang, Zongsheng Yue, Renjing Pei, Wenqi Ren, and Xiaochun Cao. Degradation-guided one-step image super-resolution with diffusion priors. arXiv preprint MrXiv24499.17558 2024.
Howard Zhang, Yuval Alaluf, Sizhuo Ma, Achuta Kadambi, Jian Wang. and Kfir Aberman. Instantrestore: Single-step personalized face restoration with shared-image attention. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers, pp. 1-10, 2025.
Lin Zhang, Lei Zhang, and Alan C. Bovik. feature-enriched completely blind image quality evaluator. IEEE Transactions Image Processing, 24(8):2579-253) 2015. doi: 10. 109/TIP. 2015.2426416.
Lin Zhang, Xin Li, Dongliang He, Fu Li, Errui Ding, and Zhaoxiang Zhang. Lmr: A large-scale multi-reference dataset for reference-based super-resolution. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 13072-13081, 2023a. doi: 10.1000/0
2023.01206.
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In IEEE International Conference on Computer Vision (ICCV), 2023b.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 586-595, 2018. doi: 10. 20.1 800068..
Zhifei Zhang, Zhaowen Wang, Zhe Lin, and Hairong Qi. Image super-resolution by neural texture transfer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Shangchen Zhou, Kelvin C.K. Chan, Chongyi Li, and Chen Change Loy. Towards robust blind face restoration with codebook lookup transformer. In NeurIPS, 2022.
13

---

Published as a conference paper at ICLR 2026
APPENDIX OVERVIEW
The appendix is organized as follows:
Section A: Construction of the training dataset and the bird retrieval test dataset.
Section B: Complete loss functions used for training Ada-RefSR.
Section C: Spectral visualization of restoration backbone features before and after Refer- ence Attention on CUFEDS.
Section D: Ablation study on the number of summary tokens, showing that 16 tokens yield the best performance.
Section E: Computational complexity analysis comparing AICG (implicit correlation gat- ing) with explicit correlation gating.
Section F: Robustness tests on the face dataset under different reference degradations (blur and affine transformations).
Section G: Visual comparisons on artificial textures.
Section H: Additional visual results on bird retrieval, face, and dense-scene reference datasets (WRSR).
A MORE DETAILS FOR DATASET
A.1 GENERAL REFERENCE DATASET FOR TRAINING
Since no high-quality scene reference dataset exists online most available datasets are of too low resolution support diffusion-basse SR we construct synthetic scene reference dataset. Specifi- cally, we use DIV2K, DIVSK, and Flickr2K sources, from which 1024 1024 regions are cropped with sliding window. Each region further split into two 5 512x512 sub-regions, one serving as the HQ image and the other the Ref. To enhance robustness against distortions and better simulate real-world reference variations, the Ref images are additionally augmented via random rotations. Finally, LQ images generated from the HQ images using the RealSRGAN degradation pipeline (Wang et al., 2021), resulting in paired LQ-HQ-Ret Ref triplets for training.
A.2 BIRD RETRIEVAL DATASET
To construct high-quality retrieval dataset for evaluating reference enhancement within the same category, we collected a bird retrieval dataset consisting of 8,460 high-resolution bird images sourced from the web. We first used BLIP filter out irrelevant images without birds, and then applied YOLOv10 (Wang et al., 2024a) to detect birds and extract the largest square crops contain- ing them. Among these images, were randomly selected as the test set, while the remaining images served the high-quality reference database. Low-quality (LQ) images from the test set using the degradation pipeline of ReaISRGAN to form LQ-HQ pairs.
generated
For retrieval, given an LQ image Xıg. searched for semantically relevant reference X,ef from the database ID via cosine similarity in the feature space:
X = arg max sim (xx,))))))))))))))())) of (x(()))
(8)
where (i.) is an image feature encoder and sim(...) denotes cosine similarity. We adopt CLIP (Radford et al., 2021) and DINOv2 (Oquab et al., 2023) as feature extractors, as they provide robust semantic representations that suppress noise unrelated to image content.
Representative samples are shown in Fig. 9, where each sample includes the low-quality (LQ) in- put, its high-quality ground truth (GT), and the retrieved reference (Ret)-with similarity scores (computed via DINOv2) anmotated to quantify retrieval relevance. These scores directly reflect the matching degree between LQ and Ref.
14

---

Published as a conference paper at ICLR 2026
Mone
Moseling
Figure 9: Representative samples from the bird retrieval dataset, including the Low quality (I.Q) input, its Ground Truth image (HQ), and the retrieved reference (Ref).
15

---

Published as a conference paper at ICLR 2026
B LOss FUNCTIONS
Our reference based SR network is trained with a weighted combination reconstruction, percep- tual, and adversarial losses:
Littal X1 Xrec + X2h + Xahandv
The reconstruction loss is defined as
Cree G[og Xres) X
(10)
the perceptual loss is defined as
C We Xres)) Vi xx
Lper
(11)
where (.)+) denotes the i-th feature map from a pretrained VGG backbone (Simonyan & Zisserman, 2014). The adversarial loss adopts the standard GAN formulation:
Ladv = Ex,, pea [lo Dw (Xtt))
(12)
Ex X pape [log Dw (Go(Xig Xrer)))]
where Go is our Ada-RefSR model and Dw is the discriminator (Caron et al., 2021).
C VALIDATION FOR REFERENCE ATTENTION BASED FEATURE INJECT
C.1 RELATIVE POWER SPECTRUM VISUALIZATION
Power Spectrum: With Without Sederencc Altentionn
Figure 10: Visualization of the Relative Power Spectrum for Intermediate Feature Maps.
We visualize the power spectrum response curves the CUFEDS dataset to validate the effec- tiveness of our reference attention in injecting reference information from a frequency- -omaann per- spective. Specifically, compare the output results of att (first attention module) across all
16

---

Published as a conference paper at ICLR 2026
Transformer Blocks-wil with total of 16 layers available for comparison. Here, "Without" denotes the intermediate- layer outputs of the original baseline model, while "With" represents the frequency responses after integrating
reference attention mechanism.
As shown in the Fig. 10, after introducing reference attention, the frequency responses are improved across most layers, with notable enhancements in the top layers (Layer 1, 2, 13). This indicates that reference attention enables the model to more completely and balancedly retain and enhance the local detail information of features. It further confirms that the introduction of reference images primarily achieves the transfer of image details and textures, ultimately leading to clearer visual details and stronger realism in the generated results.
D
How MANY SUMMARY TOKENS ARE NEEDED?
Table 4: Comparison of token numbers on WRSR and Bird datasets across multiple metrics.
Dataset
Token
PSNR
SSIM
LPIPS
FID
NIQE
MUSIQ
CLIPIQA
21.8914
0.5716
0.3063
53.1421
3.6916
69.4745
0.6439
WRSR
16
21.9722
0.5777
0.3061
53.2811
3.7429
69.8608
0.6612
32
21.9127
0.5764
0.3055
52.5586
3.7712
69.7996
0.6569
8
25.2595
0.7263
0.2556
36.5373
5.0629
71.3751
0.8216
Bird
25.2998
0.7292
0.2536
36.4249
5.1771
72.0039
0.8303
32
25,2286
0.7273
0.2538
38.2489
5.1859
72.1367
0.8234
Herein, we further conduct experiments on the required number of learnable tokens. We set the number learnable tokens to 8, 16, and 32 respectively, with detailed results reported Table
As shown in Table 4, when the number of learnable tokens is set to 16, the better performance is achieved on both the CUFEDS and Bird datasets. This indicates that learnable tokens do not need to be excessive, and small number of tokens already exert their effectiveness.
E COMPUTATIONAL COMPLEXITY ANALYSIS
In this section, we provide a theoretical comparison between the computational costs of Ex- plicit Correlation- Gating in ReFIR (Method 1, denoted as m2) and our proposed Implicit Correlation-based Gating (Method 2, denoted as m2).
Notation. We analyze the additional computation introduced by each gating mechanism relative to the shared geference-AAttenionn (RA) module. We define the following dimensions:
Hyre E R.berredd xd; Source feature sequence.
. Href E RR xd, Reference feature sequence.
Ts Mxd, Learnable summary tokens (specific Method 2).
E.1 COST DERIVATION
Reference Attention Baseline. Both methods share the underlying RA computation. This in- volves five linear projections (to_g, to-k, toLv, zero.linear, to.out). The base computa- tional cost is:
Cruse (3I.rr 22rref dd + AIssrccireed.
(13)
Method 1: Explicit Correlation. Method calculates the full ooken-to-token cosine similarity matrix between Hyre and Href- This introduces an explicit interaction cost of sre The total cost is:
C(m1) Crase + #2LrrcLredd
(3T + 2 2T do + G..rrebragd.
(14)
17

---

Published as a conference paper at ICLR 2026
Method 2: Implicit Correlation (Ours). two lightweight steps:
Method avoids full pair-wise similarity by introducing
1. Reference Summarization: Summary tokens attend to reference features (reusing RA projections), costing Md + r Md.
2. Implicit Estimation: Source queries interact only with the summary tokens Tş, costing Md.
Summing these with the baseline, the total cost is:
C(m²) = Cruse + Md + (2Lref + 2L Md
= (3 + 2 ) + Aberobmeed + Md + (2Lref + 2barr)Md
(15)
E.2 ASYMPTOTIC COMPARISON
To clearly visualize the efficiency gain, we consider the typical setting where sequence lengths dominate the summary token count and hidden dimension:
Lare
Lref L, L L> M, L L>d.
Under these conditions, the dominant complexity terms simplify as shown in Table 5.
Table 5: Complexity comparison of gating mechanisms. Method 2 significantly reduces the coeffi- cient of the quadratic term.
Method
Dominant Term
Interaction Type Complexity
Explicit (m²)
6L2d
All-to-All Summarized
O
Implicit (m²)
4L'd
O(L3)
The relative computational load is approximately:
C(m2)
4L2d
0.67.
(16)
C(m1)
6L2d
Thus, the proposed implicit correlation gating reduces the attention complexity burden by roughly 33% while maintaining robust adaptivity.
E.3 NUMERICAL EVALUATION
To quantify the practical overhead introduced by each method, we substitute typical high-resolution settings used in experiments. We assume input image size of 512 512, resulting in sequence length L = 4096 (64 64 latent space after 8x VAE downsampling). along with a hidden dimension 1024, and M 16 summary tokens.
Overhead Analysis. Based on Eq. equation 13, the baseline Reference Attention (RA) requires approximately 2.15 10a1 FLOPs.
Method (Explicit): Calculating the full L x L similarity matrix introduces substantial overhead of z 3.44 x 10 operations, increasing the total computational cost by 16.0%.
Method (Implicit, Ours): By leveraging M 16 summary tokens, the additional cost is drastically reduced to x 2.69 10 operations. This represents a negligible overhead of only 0.13% over the baseline.
As shown in Table 6, while Explicit Gating imposes heavy computational burden, our Implicit Gating achieves adaptivity with virtually zero additional cost (~ 127x more efficient in terms of added overhead).
18

---

Published as a conference paper at ICLR 2026
Table 6: Numerical complexity comparison 4096, d 1024, M
16). Our Method
introduces negligible overhead compared to the baseline, whereas Method adds significant cost.
Module
Total FLOPs
Added Cost
Relative Overhead
Baseline RA
2.15 x 10"
Explicit Gating (m²)
2.50 x 10
3.44 x 10
+16.00%
Implicit Gating (m2)
2.15 x 10
2.69 10
+0.13%
F ROBUSTNESS TO VARIOUS REFERENCE QUALITY AND MISALIGNMENT
We conducted controlled experiments on the face reference super-resolution benchmark proposed in paper to systematically investigate the robustness of Ada-RefSR. The study considered varied reference conditions, including degraded reference quality and potential misalignment. Since all faces in this dataset frontal, implemented types of synthetic degradations applied the reference images:
1. Gaussian Blur Degradation: Reference images were convolved with Gaussian kernels of progressively increasing sizes (3, 5, 7, 9, and 11 pixels) to simulate varying levels of blur.
2. Affine Degradation: Graded affine transformations were applied simulate potential mis- alignment. This included simultancous variations in rotation angle, scale deviation, transla- tion ratio, and corner offsets. Five intensity levels were defined, with each level increasing all transformation parameters relative to the previous, thereby quantifying the severity of misalignment.
Teansform:
Affine
Affine level
Affine
Uhe degrddatio:: level
Ihh degrodation: level
Bhur degradarion: level
Blu degradation level
Uhr degridation : level
Figure 11: Examples Gaussian blur and affine degradations on reference images at different intensity levels: affine transformations (Ist row) and Gaussian blur with increasing kernel sizes (2nd row).
Figure 11 visualizes the impact of each degradation type at different intensity levels. We evaluated all reference-based face SR methods under these controlled degradations and summarized perfor- mance metrics in line charts (Fig. 12).
Observations. Under affine degradation, Ada-RefSR demonstrates strong robustness: metrics such as FID and LPIPS remain largely stable even as misalignment increases, whereas alternative methods exhibit notable performance drops, highlighting their sensitivity to spatial inconsistencies. Under Gaussian blur degradation, our method maintains competitive performance relative to face- specific SR approaches such as InstantRestore, DMDNet, and FaceMe. Across nearly all metries, Rda-RefSR consistently outperforms these baselines, even at higher degradation levels.
19

---

Published as a conference paper at ICLR 2026
PARK 1AN0
PARK
182
MUXIQ
CUPIDA
CLUPAQA
Kamal
(3) Reference images with Ganssia Dlur Induced degradation
(b) Reference Images with Affine Transformaiion Induced Degradation
Figure 12: Performance curves for various face RefSR methods Degradation level of the ref- erence image is varied. Results are shown for two degradation types: (a) Gaussian Ilur-Induced degradation and (b) Affine Transformation-Indeee degradation.
images with Blur-Induced degradation
(b) Images Transfonnstixnn--nee Degradation
Figure 13: Average AICG gating activations across layers of the reference attention module under varying reference quality. Lower-quality references lead reduced gating activations, illustrating adaptive down-weighting of unreliable regions.
These results indicate that Ada-RefSR can effectively leverage reference information while remain- ing resilient to variations in quality and alignment. This robustness is primarily attributed to the Adaptive Implicit Correlation Gating (AICG) mechanism.
Interpretability Analysis. To provide further insights, examined the internal gating activations of AICG across layers of the reference attention module. As shown in Fig. 13, the average gating activations decrease progressively as reference quality declines. This demonstrates that AICG adap- tively attenuates contributions from unreliable reference regions, thereby maximizing the utility of informative reference features while mitigating the adverse effects of poor-quality or misaligned references.
Visualization Examples. Figure provides additional qualitative examples under different refer- degradations. When the reference undergoes affine transformations, the restored faces remain highly consistent with those obtained using the clean reference, indicating strong robustness to mis- alignment and viewpoint inconsistencies. Under Gaussian blur, method still preserves identity while only losing fine-grained details that are absent from the degraded reference. When an unre- lated reference image provided, our results naturally revert to those of the single-image baseline (S3Diff), confirming that the proposed AICG mechanism effectively suppresses irrelevant reference cues and prevents quality degradation. These observations further validate the "trust but verify" behavior of Ada-RefSR across diverse reference conditions.
20

---

Published as a conference paper at ICLR 2026
Grand Trah
Degial Reaah
Win Affize Torsfom
Win Birr Degratiii
Eazskm
S30df
L
M L
EL
IVI
L
IVI LL
IVI
L
R
R
R
R
R
R
M
LL
R
Goand
Origiaal Revalt
Aftine adation
Randon
SSDII
IC
IC
Organol
Degrodation
Degradation
Figure 14: Generated results under different reference degradations (first row) with corresponding LQ and reference images (second row). The last column shows the S3Diff baseline. Our method remains effective under affine and blur degradations, and defaults the baseline when using random references.
21

---

Published as a conference paper at ICLR 2026
G
MORE RESULTS ON ARTIFICIAL TEXTURES
CUFED5 26
HQ
LQ
C2-Matching
DATSR
SUPIR+ReF IR
SeeSR+ReF IR
Reference
Rea1SRGAN
PISA-SR
OSEDiFF
S3Diff
Ours
CUFEDS 27
HQ
LQ
C2-Matching
DATSR
SUPIR+ReFIR
SeeSR+ReFIR
Reference
Rea15RGAN
PISA-SR
OSEDiFF
S3Diff
Ours
CUFED5 __9
HQ
LQ
C2-Matching
DATSR
SUPIR+ReFIR
SeeSR+ReFIR
Reference
Rea15RGAN
PISA-SR
OSEDiff
S3DLFF
CUFEDS 71
HQ
LQ
C2- Matching
DATSR
SUPIR+ReFIR
Ree5R+ReFIR
Reference
Rea1SRGAN
PISA-SR
OSEDiff
S3DIff
Ours
Figure 15:
Visual comparisons on complex artificial textures from the CUFEDS dataset. Our method better restores contours and material textures, providing richer details.
To further demonstrate the generalization capability of our model beyond natural scenes, we provide additional visual examples involving complex artificial textures in Fig. 15 and Fig. 16. These cases include variety of man-made patterns such logos, wall tiles, door frames, and fabric materials.
22

---

Published as a conference paper at ICLR 2026
CUFEDS 81
C2-Matching
DATSR
SeeSR+ReFIR
Reference
Rea1SRGAN
PISA-SR
OSEDiff
S3Diff
Ours
CUFEDS 87
C2-Matching
DATSR
SUPIR+ReFIR
Reference
Rea1SRGAN
PISA-SR
OSEDiff
S3Diff
Ours
CUFEDS 98
HQ
LQ
C2-Matching
DATSR
SUPIR+ReFIR
ReeSR+ReFIR
Reference
BRGAN
PISA-SR
OSEDiff
S3Diff
Ours
CUFEDS 122
C2-Matching
DATSR
SUPIR+ReFIR
ReeSR+ReFIR
Reference
Rea1SRGAN
PISA-SR SR
OSEDiff
S3Diff
Ours
Figure 16: Visual comparisons on complex artificial textures from the CUFED5 dataset. Our method better restores contours and material textures, providing richer details.
23

---

Published as a conference paper at ICLR 2026
For example, in the CUFED5_20 26 case (Fig. 15), both C2-Matching and DATSR fail restore the degraded input, as they rely on the assumption of synthetic x4 downsampling and lack the ability to handle real-world degradations. Single-step diffusion SR baselines (PISA-SR, OSEDiff, S3Diff) either retain noticeable noise or produce overly smoothed textures. Although SUPIR+ReFIR and SeeSR+ReFIR able to generate complete WiFi logo, they exhibit notable structural inconsis- tencies: SUPIR+ReFIR produces unnaturally sharp boundaries and over-bright colors inconsistent with the reference, while SeeSR+ReFIR introduces sharp and unrealistic transition near the top of the symbol. In contrast, our method not only removes the degradation effectively but also pre- serves both the soft boundary and the original color distribution of the logo, faithfully matching the reference.
Beyond logos, our approach also successfully recovers fine-grained man-made textures with high geometric fidelity. In CUFED5_59 (Fig. 15), our method accurately reconstructs the layout, bound- ary structure, and material appearance of wall tiles. CUFED5_6 (Fig. 16), Ada-RefSR restores the fabric texture and the thin black frame in the upper region, which competing methods fail to preserve. Similarly, in CUFED5 98 (Fig. 16), our model effectively captures the curtain material and restores its structural regularity.
These examples collectively demonstrate that our model generalizes well to structured, artificial patterns. The combination of (1) a strong single-image SR prior and (2) trust-but-verify reference integration mechanism allows Ada-RefSR to selectively leverage reference cues while maintaining geometric accuracy and material consistency across wide range of real-world textures.
H MORE VISUAL RESULTS
In this section, provide additional visual comparisons across the three evaluated datasets fur- ther demonstrate the effectiveness of our proposed method. Specifically, the visual results for the bird retrieval, face, and WRSR datasets are illustrated Fig. 17, Fig. 18, and Fig. 19, respectively.
24

---

Published as a conference paper at ICLR 2026
HQ
LQ
RealSRGAN
SUPIR+ReFIR*
SeeSR+ReFIR
Reference
PISA-SR
OSEDiff
S3Diff
Ours
HQ
LQ
RealSRGAN
SUPIR+ReFIR*
See5R+ReFIR*
Reference
PISA-SR
OSEDiff
S3Diff
Ours
HQ
LQ
RealSRGAN
SUPIR+ReFIR#
SeeSR+ReFIR
Reference
PISA-SR
OSEDiFF
S3Diff
Ours
Figure 17: Visual comparison results on the bird retrieval dataset.
25

---

Published as a conference paper at ICLR 2026
HQ
LQ
DMDNet
SeeSR+ReFIR*
FaceNe*
InstantRestoret
Reference
RealSRGAN
OSEDiff
S3D1ff
HQ
LQ
DMDNet
See5R-ReFIR*
FaceMe*
InstantRestoret
Reference
RealSRGAN
PISA-SR
OSEDIff
S3D1ff
Ours
DMDNet*
SeeSR+ReFIR-
FaceMe*
InstantRestore#
Reference
Rea1SRGAN
PISA-SR
OSEDiff
S3DIff
Ours*
Figure 18: Visual comparison results on the face dataset.
26

---

Published as a conference paper at ICLR 2026
HQ
LQ
C2-Matching
DATSR
SUPIR+ReFIR
SeeSR+ReFIR
Reference
Rea1SRGAN
PISA-SR
OSEDiff
S3Diff
Ours
LQ
C2-Matching
SUPIR+ReFIR
SeeSR+ReFIR
Re Reference
Rea1SRGAN
PISA-SR
OSEDiFF
S3Diff
Ours
HQ
11
C2-Matching
DATSR
SUPIR+ReFIR
SeeSR+ReFIR
Reference
Rea1SRGAN
PISA-SR
OSEDiff
S3Diff
Ours
HQ
LQ
C2-Matching
DATSR
SUPIR+ReFIR
SeeSR+ReFIR
Reference
RealSRGAN
PISA-SR
OSEDLfF
S301ff
Ours
Figure 19: Visual comparison results on the WRSR dataset.
27


Reviews:
[{"review_id": "xEi1U1Kbli", "reviewer": "Reviewer_cRgB", "summary": "The paper proposes Ada-RefSR, a single-step reference-based diffusion framework for super-resolution. The method addresses the challenge of establishing reliable correspondences between low-quality (LQ) inputs and reference images, which is particularly difficult under severe degradations. To mitigate issues of over- or under-reliance on references seen in existing weighting-based fusion strategies, the authors adopt a “Trust but Verify” paradigm: the model leverages references when they align and suppresses them otherwise. The core component, Adaptive Implicit Correlation Gating (AICG), uses learnable summary tokens to capture dominant reference patterns and implicit correlations with LQ features. Integrated into the attention backbone, AICG adaptively regulates reference guidance and prevents erroneous feature fusion. Experiments on multiple datasets show that Ada-RefSR achieves a good balance of fidelity, naturalness, and efficiency, while maintaining robustness under varying reference alignment.", "questions": "No", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed “Trust but Verify” perspective is an interesting and promising approach. \n\n2. The authors conduct extensive experiments to validate the effectiveness of their method.", "weaknesses": "1. The key innovation of this paper is the Adaptive Implicit Correlation Gating (AICG) mechanism. Currently, the introduction lacks a critical figure illustrating the authors’ approach and the core differences compared with prior methods, which is highly important.\n\n2. The authors need to explain why their method does not perform well on the WRSR dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QlhIK5d5ZM", "reviewer": "Reviewer_c7Cn", "summary": "This paper presents Ada-RefSR, a single-step diffusion framework for reference-based super-resolution that follows a “Trust but Verify” strategy. The method first injects reference features through a reference-attention module and then uses an Adaptive Implicit Correlation Gating (AICG) mechanism to adaptively filter mismatched reference cues. Experimental results on multiple datasets show improved fidelity, perceptual quality, and efficiency compared with recent diffusion and RefSR methods.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper clearly defines the problem of over- or under-reliance on reference images in diffusion-based SR and provides a coherent conceptual framework to address it. \n\n- The proposed AICG module is lightweight and easily pluggable into existing backbones, enabling adaptive control of reference guidance without additional supervision. \n\n- This paper provides clear visual evidence, such as attention maps, gating masks, and token visualizations, that help interpret the mechanism’s behavior.", "weaknesses": "- The main limitation lies in the novelty boundary of AICG. Its design using learnable tokens for implicit correlation modeling is conceptually close to DETR-style query or prototype aggregation, and the paper does not clearly explain how it fundamentally differs from those approaches. \n\n- The paper also lacks comparisons with strong face-specific RefSR methods, which would help validate the generalization of Ada-RefSR in specialized domains. \n\n- There is no detailed analysis of different reference scenarios, such as varying reference quality, domain gaps, or alignment errors, which would strengthen the claim of robustness under diverse reference conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4RSuwhc4p2", "reviewer": "Reviewer_Fy7H", "summary": "This paper proposes Ada-RefSR, a diffusion-based model for Reference-based image super-resolution. The core idea of this paper lies in two terms: the fidelity and generalization. It proposes AICG to adaptively balance intrinsic SR fidelity with reference-guided enhancement, mitigating hallucinations. AICG enables Ada-RefSR to generalize RefSR beyond narrow domains to diverse scenarios, maintaining robustness under varying reference alignment. Further, built on one-step diffusion model, its inference is efficient compared to other multi-step diffusion model.", "questions": "Refering to the weaknesses. Clarify the contributions specfically for Ref-SR. The visualization of some complex textures are also important to reveal the generalization of the proposed method.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper proposes AICG designed for RefSR task. Ada-RefSR injects LQ features directly using a residual connection besides the reference feature selection. It allows the model to preserve the prior knowledge. Also, a gating mechanism is applied in the reference feature attention components to adaptively select useful information from the reference image.\n2.\tThe model achieves SOTA performance with efficient one-step diffusion model.\n3.\tThe idea is straightforward and easy to follow.", "weaknesses": "1.\tThe novelty is limited, since most of the model design and idea like LQ feature residual connection is straightforward and has been proposed in classic non-diffusion methods. The paper should discuss the novelty specifically for Ref-SR task.\n2.\tThough claiming efficiency as one of the contributions, the paper provides insufficient discussions and experiments on efficiency and model size.\n3.\tThe visualizations are mostly derived from animal images, are there results and visual comparisons on more complex textures like text or some artificial objects to better show the generalization of this model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)