AnyUp: Universal Feature Upsampling
Authors: Thomas Wimmer
Prune Truong
Marie-Julie Rakotosaona
Michael Oechsle
Federico Tombari
Bernt Schiele
Jan Eric Lenssen
Venue: ICLR 2026 Oral (2026)
Keywords: feature upsampling
representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: A universal feature upsampling model that can be used to upsample any feature from any to any resolution and generalizes to features unseen during training.

Abstract: We introduce AnyUp, a method for feature upsampling that can be applied to any vision feature at any resolution, without encoder-specific training. Existing learning-based upsamplers for features like DINO or CLIP need to be re-trained for every feature extractor and thus do not generalize to different feature types at inference time. In this work, we propose an *inference-time* feature-agnostic upsampling architecture to alleviate this limitation and improve upsampling quality. In our experiments, AnyUp sets a new state of the art for upsampled features, generalizes to different feature types, and  preserves feature semantics while being efficient and easy to apply to a wide range of downstream tasks.

Published as a conference paper at ICLR 2026
ANYUP: UNIVERSAL FEATURE UPSAMPLING
Thomas Wimmerl2, Prune Truong', Marie-Julie Rakotosaona', Michael Oechsle² Federico Tombari Bernt Schiele', Jan Eric Lenssen'
MMa Planck Institute for Informaties, SIC, 2ETH Zurich, *Google, *TU Munich
https: /ttp://wwimmtttmm///mmmmmm/m mmerth github. 1o/anyup
SigLIP 2
CLIP
DINOV2
AnyUp
ResNet
A universal feature upsampling model
DINOv3
Figure 1: AnyUp a universal feature upsampling model that can upsample any feature from any to any resolution, generalizing unseen features while achieving state-of-the-an performance.
ABSTRACT
We introduce AnyUp, method for feature upsampling that can be applied any vision feature at any resolution, without encoder-specifie training. Existing learning-based upsamplers
features like DINO or CLIP need to be re-trained for every feature extractor and thus do not generalize to different feature types at inference time. this work, we propose an inference-ting -imeeneeeemeeeeeee feature-as aaguosii up- sampling architecture to alleviate this limitation and improve upsampling quality. In our experiments, AnyUp sets a new state the art for upsampled features, generalizes different feature types, and preserves feature semantics while being efficient and easy to apply to wide range of downstream tasks.
INTRODUCTION
General image feature extractors, such as DINO (Caron et al. 2021 Oquab et al. 2023 Xue et al,,
2025), CLIP (Radford al. (2021). SigLIP (Tschannen al. 2025 or MAE (He et al. (022). have become fundamental building blocks in modern day computer vision, providing, e.g., semantics or language-alignment wide range of down-stream applications, such depth estimation or 3D reconstruction (Feng et al. 2024 Wang et al., 2025). open-vocabulary semantic segmentation (En- gelmann al. 2024; Wysoczańska al. 2024), or benchmarking generative models (Asim al.] 2025). An important limitation of such traitreeed models, which are usually transformer-based, is that their output feature map resolution limited the number of transformer tokens, preventing the prediction of pixel-level features. Therefore, several recent works, such FeatUp (Fu et al. (2024), Lo (Huang et al., 2025), or JAFAR (Couairon et al. 2025) propose learned feature upsampling methods.
While such feature upsampling methods perform well when paired with the vision encoders with which they were trained, they generally not encoder-agnostic inference time and need to be retrained to be usable with different feature extractor. This can be costly or, in the case of the latest large vision models (Xue al. 2025). even infeasible with limited computing resources,

---

Published as a conference paper at ICLR 2026
Table Categorization of feature upsampling methods. AnyUp is the first learnable method that generalizes to any input feature at inference time, while being able to upsample from any to any resolution and being aask-agnostic.
any any encoder
any task
trainable
resolution
Bilinear Upsampling
Nearest-Neighbor Upsampling
Large Input Image
x
Bilateral Filtering (Tomasi & Manduchi 1998
X
X
Guided Filtering (He et al. (2012)
X
Change of ViT Patchification Stride
SAPA [Lu et al.
2022
CARAFE Wang et al.
2019)
DySample (Luu al.
2023)
ReSFU (Zhou et al.
2025)
Resize Convolution (Odena
2016
LiFT Suri et al.
2024)
UpatUp (Fu al.
(0044)
FeatSharp (Ranzinger et al.
2025
LIIF (Chen et al. (021))
X
LoftUp (Huang et al. (2025)
JAFAR (Couarron et al. 2025
AnyUp (Ours)
as, during training of the upsampler, the vision encoder must queried multiple times for every training sample, often also on a higher-resolution image. In this work, we alleviate this limitation and propose AnyUp, a learned feature upsampling method that generalizes to features any size provided in any resolution while achieving state-of-the-an performance across various downstream tasks. As shown Tab. D AnyUp is the first of its kind.
Feature upsamplers extract information from low-resolution feature map and high-resolution RGB guidance image, to infer which pixel in the high-resolution image should receive which fea- ture (Couairon al. 2025). The key limitation of existing work is that their way of processing the low-resolution feature map is specific to the dimensionality and type of the used feature. In contrast, Un employs feature-agnostic layer that allows to process any feature and can also generalize to novel feature types. In addition to the feature-agnostie layer, we introduce a window attention procedure and crop-based training strategy, which further improve feature upsampling quality.
In our experiments, AnyUp establishes new state of the art for feature upsampling. Its key ad- vantages include universal applicability it can be trained and applied across all feature types and resolutions and robust generalization to feature types it was not trained on. Moreover, Any Up ensures high fidelity by minimizing the distortion of original feature semantics. Through ablations, we analyze different design choices and draw general insights for the feature upsampling task.
In summary, our contributions are:
We introduce AnyUp, an upsampling model that is feature-agnostic inference time, i.e., that needs to be trained just once and can be used to upsample features from any source at any resolution and dimensionality.
We propose feature- agnoosic layer that captures information from features varying type and dimensionality.
We introduce window attention-baan based upsampling architecture that can be trained effec- tively using an image part-based loss and retains the input feature space through consistency regularization.
We show that AnyUp outperforms prior upsampling methods while being able generalize to any vision encoder at test time.
We make our code and pre- trained weights publicly available at giving access light-weight, training-free and easy-to-use feature upsampler.
2

---

Published as a conference paper at ICLR 2026
RGB Image
Low Resolution
AnyUp (Ours)
JAFAR
LoftUp
FealUp
Guided Filter
Bilinear
Figure 2: Visual comparison against other methods. RGB channels correspond to the first three principal components computed over all features. Previous methods result excessive smoothing contain other artifacts: See, e-g., in the first row, the smoothed-out cloud features in LoftUp the feature distribution shift for the mountains in JAFAR, as well as the oversmoothing and halo- artifacts of FeatUp and Guided Filter in the third row, AnyUp (Ours) produces sharp output feature maps while preserving the input feature quality.
RELATED WORK
Prior works on feature upsampling can be categorized into several groups, based on their capabilities and reliance on data, as summarized in Tab.
Training frre upsampling methods can usually be used with any input features, for any task, and can upscale to any resolutions, while being relatively light-weight. This makes them, i.., bilinear upsampling, de-facto standard choice many applications where high-resolution features are required. Methods like Guided Filtering (He al. (012)) also work surprisingly well on favorable data samples, see Fig. 2 but degrade on, e.g., low-contrast or more complex images, requiring insufficient blur (see the last row in Fig. 2). Changing the stride the patchification in the vision transformer (Dosovitskiy et al. 202T), simply increasing the resolution of the input image results in computationally heavy inference and risks moving the features out-of-distribution (Ranzinger et al.
per-sample tuning of hyperparameters and often still causing excessive
2025).
Learnable upsampling methods (Suri al. 2024 Fu et al.
Couairon al., 2025) try to improve the quality and robustness over learning-free approaches. The parameteriza- tion. however, always results in the loss of feature independence inference time, i.e., upsampling models are fitted to a specific feature encoder (and input dimensionality). In addition, some meth-
2024 Huang at. 2025
ods are fixed to a specific downstream task or model (Wang et 2019 Lu et al. 2022] Liu al.,
2023
Zhou et al. 2025) or cannot be used with arbitrary scaling factors OOdena et 20. Wang
al.
2019 Lu et al. 2022; Liu et al.) 2023) Suri et al. 2024 Fu et al. 2024] Ranzinger et al.)
The current of-te-tf approaches, JAFAR (Couairon et al. 2025) and LofuUp (Huang 2025). both concurrent works, formulate feature upsampling as a single or stacked high-res to low-res attention, respectively. We adopt this formulation as it offers the advantage of being nat- urally resolution-an -agluttion-anosticc, i.e., features can be upsampled from any to any resolution. However, in AnyUp, aim to make this architecture encoder-agnostic, i.e., the upsampling model should be trained once and subsequently be applicable to features from any vision encoder, potentially of different feature dimensionality. More details on selected related work can be found in App. A
2025
ctt al.
TASK FORMULATION
Given input RGB image Inr Hx the goal of feature upsampling methods f is to upsam- ple low-resolution input feature maps E glxxxxx := e(Irr) to high-resolution output feature maps q : (p, I ) E R ffxxxx The necessity of this task is grounded in the use of large pre-trained vision encoders e(-) as feature extractors and powerful semantic priors recent years. Almost all such models, be convolutional neural networks (Fukushimal 1969 LeCun al. 1989 vi-
3

---

Published as a conference paper at ICLR 2026
f(p. 1)
Layer
er'
Care.
f(p,1
p
e(1)
Window Attention
Training Strategy
(b) Architecture
Figure 3: Method Overview. Up performs window attention-based upsampling (4.2) Input features are processed with a feature-agnostic layer (4.1). During training, features computed for randomly sampled image parts are used as reference for the respective part of the upsampled feature map (4.)..
sion transformers (Dosovitskiy et al. 2021), perform learned downsampling latent resolutions h H x W, which is necessary for efficient image processing using these large neural net- works. However, when exploiting the features of these powerful models for downstream tasks, is often necessary work with high-resolution features, e.g., when performing dense, pixel-wise pre- dictions, or aggregating information from multiple views 3D (Engelmann et al. 2024 Ye al.,
2024 Asim al. 2025)
LEARNING AGNOSTIC FEATURE UPSAMPLING
We are interested lightweight and low- wwwpaaameterizedd model to prevent memory and compute bottlenecks. For this, we base our model on an attention- architecture, similar to that in pre- vious works (Couairon al.) 2025 Huang et al. 2025) (see Fig. 3). More specifically, we adopt the architecture of JAFAR (Couairon et al. 2025), which we briefly describe next. The input image and low-rcsolution feature map p are both first passed through convolution blocks with residual connections. Next, positional encodings are applied to the image features. Queries for the final attention layer are computed directly from these pixel features, while information from both the downsampled image and the low-resolution feature map is used to compute the keys. The values in the attention layer are simply the unprocessed patch features from the input feature map.
In our method, aim to fix two of the main limitations of this architecture. First, replace the initial feature processing layer, which acts only on fixed-dimensional inputs and needs to be learned per vision backbone with our proposed feature-agnostic convolution layer (Sec. [4.1). Second, simplify the task of the upsampler by restricting the source features to local windows in the attention computation (Sec. 4.2
We further improve the training pipeline for our feature upsampling method by using an image part-based strategy that is explained in Sec. 4.3.1] as well as additional consistency regularization improving the robustness to noise and input feature space preservation (Sec. 4.3.2).
4.1 FEATURE-AGNOSTIC UPSAMPLING BY DESIGN
In our feature-agnostic layer, we aim to represent feature maps from any source model with any dimensionality as feature maps with canonical dimensionality by convolving them with learned filter basis. We hypothesize that an attention- combinations of the input features for every pixel, mostly needs to understand overall local structure changes in the input feature map. To capture this structural information while staying agnostic to the input features (and their dimensionality). we design a convolutional layer that captures structure independently for all input channels and aggregates this information later. We illustrate the structure of this layer in Fig.
upsampling model, where the outputs are linear

---

Published as a conference paper at ICLR 2026
In proposed feature- agnostic layer. each imput channel Pr is convolved with a learned kernel basis (1/ B j=t........ fol- lowed by a softmax operation on the activations along the basis filter dimension. The resulting filter-wise contri- butions are then averaged over all input channels. More formally, the output feature fy of the feature-agnostic convolution layer can be computed as
softrae
Figure Feature-agnostic layer. Input channels processed independently and contributions to basis filters are averaged over all channels leading to outputs invariant to input dimensionality.
exp(p: (b3)) iE(I....,N) (I.,n.... explp: * vby))
Σ
J=N N
where N is the (varying) number of input channels and M the canonical number of output channels. Note that we leave out the spatial structure of the convolution operation to simplify notation in Eq
4.2 LOCAL WINDOW ATTENTION
Analyzing the attention patterns of JAFAR (Couairon et al. 2025), find that in the global attention mechanism, where a pixel query can attend to any feature patch in the input feature map, sometimes vastly unrelated and distant image areas are used references for upsampling. We argue that can avoid such patterns and simplify the upsampling problem by restricting the attention computation local windows around the query point (Ramachandran et al.) 2019). As high-resolution feature is now a linear combination of much smaller set of coarse features compared with the global attention, the optimization objective for the upsampler gets easier, as well as efficiency improved. We refer the reader to App. F for visualization of the mentioned attention outliers.
4.3 TRAINING PIPELINE
Computing high-resolution features, ie., with a very high-resolution input image, is infeasible for compare at training time. Besides being com-
obtaining reference "ground-truth" that putationally infeasible at scale, extreme high-resolution inputs also effectively move most models diitr--dittiibtiinn. as noted by |Ranzinger al. (2024).
To circumvent this, prior work proposed different techniques to leverage low-resolution features. FeatUp (Fu al., (024) proposes upsampler training as "multi-view reconstruction". where equiv- ariance to small image-space perturbations is optimized for, which requires carefully designing these augmentations without shifting images out-of-distribution for the vision backbone. In LoftUp (Huang et al.) (025), segmentation masks are used as high-resolution guidance signal at the cost of having to query large segmentation model every training step. the simplest case, in JAFAR Couairon et al. (2025), training is only performed at low resolutions, i.e., learning to upsample 16x to 32x32 feature map, where both are computed from respectively scaled input images (Suri et al. 2024 Couairon al. 2025). While Couairon al. (2025) argue that this low- resolution training is sufficient for training an attention- based upsampling mechanism, we propose an improved training strategy based local image crops that lets us train more powerful feature upsampler, while being faster and more memory-efficient.
4.3.1 DATA SAMPLING
We make use the fact that we do not necessarily need to supervise the upsampling on the full feature map. Instead, choose supervise method only smaller local crops. Our training pipeline is illustrated Fig. 3 More specifically, we take a high-resolution image E R "W and randomly sample smaller local crop thereof J' R We then downsample 7 to the same
5

---

Published as a conference paper at ICLR 2026
Image
Biinaar
Festlp
Lottip
Anyllp (Durs)
Ground
Ney
talin
dnchspactess bes Galoe leiing
Mew
the
Sher
Pick
JAPAK
Ground
Figure 5: Qualitative Probing Results. Visualization of linear probing results for monocular depth estimation NYUv2 and semantic segmentation ADE20K. More visualizations given in App.
resolution h X w. Using the computed features e(T) and q = e(I'), we upsample p to a high- resolution feature map q - f(I.p) E I gg xw', where h w' are chosen in such way that the cropped part o' corresponding to I' matches the resolution of ý. Note that while LoftUp (Huang al. 2025 Sec. 4.2) proposes similar strategy in their second training stage, they compute the loss against an EMA of their method instead of ground-truth features and at much higher resolution, resulting in much more compute-heavy training. On the other hand, our sampling method is more light- than JAFAR's simple strategy [Couairon et (2025) described Sec. 4.3 as we do need to compute reference features at higher resolution, e.g., 448x448.
4.3.2
OBJECTIVE FUNCTION
We follow prior work in feature denoising and feature upsampling (Yang et al. 2024 Couairon et al. 2025 and minimize the distance between predicted features q' and target features y:
Loses (q's 1ccolsg .) + ++2L2 Q))
In addition, add self-consistency regularization [=gdf--onsiitenncy which detail in App. C11 as well as an input-consisenncy regularization Linput-conssiteeccy where we simply compute Leos-mse with the input features p and the accordingly downsampled predicted output features q. We do this to further improve the locality of the upsampled features, which is important for sub-object-level tasks like surface normal estimation, as well as maintaining the input feature space.
EXPERIMENTS
We train An on the ImageNet dataset (Krizhevsky et al. 2017) and provide further implemen- tation details in App. C Comparisons against previous works provided in Sec. [.1] where evaluate probing quality, feature space preservation, and behaviour with different resolution changes. Sec. 5.2 provides more results on the generalization of Any to other features test time. Sec. 5.3 wraps the experiments with an ablation study.
5.1 COMPARISON TO PRIOR ART
We mainly compare our method to FeatUp Fu et al. 2024), as well as the concurrent LoftUp (Huang al. 20255 and JAFAR (Couairon al. 2025). which both are general, attention-based feature upsampling methods that can upsample from and to any resolution]) However, we note that both models have the significant disadvantage of being agnostic the input features at test time and need to be retrained per vision encoder. Our Any does not suffer from this problem and is thus a more general method that can be applied out-of-the-bor to any new features. not mentioned otherwise, the DINOv2 ViT-S model (Oquab et al. 2023) is used as feature extractor for training and testing the experiments, as other works also provide pre-trained weights for this model.
While both, LoftUp and JAFAR, considered concurrent according to ICLR guidelines, choose to compare AnyUp mainly against these methods and FeatUp, they the strongest available competitors.

---

Published as a conference paper at ICLR 2026
Qualitative Analysis. We visualize PCA projections of the upsampled features in Fig. which shows a visual comparison against prior methods. We observe that AnyUp produces sharp output feature maps while preserving the input feature quality. Prior works, the other hand, often still result in oversmoothed feature maps or feature distribution shifts as indicated by the red circles.
Semantic Segmentation. We follow the evaluation scheme proposed by prior works (Couairon al., 2025] Sec. 4.3.1), where linear probe, i.e., a 1x1 convolution layer, trained to predict semantic segmentation labels from the high-resolution feature maps. The input images are resized to 448x448 resolution and features are thus of resolution 28x28 or 32x32 depending on the patch size of the vision transformer used for feature extraction. Upsampling is generally performed back to the input image resolution, i.e., equaling 14x or 16x upsampling, not mentioned otherwise. We evaluate performance the COCO-Stufi (Lin et al. 2014), ADE20k (Zhou et al. 2018) and PASCAL VOC (Everingham et al. 2014) datasets.
Results as mean Intersection- over-Union (mloU) and pixel-wise accuracy. As observed in Tab. - AnyUp provides state-of-the-an upsampling performance for downstream semantic
reported
Table 2: Semantic Segmentation. Highlights for best, second and third best scores.
COCO
PASCAL-VOC
ADE20k
mloU (1)
Acc. (t)
mloU (t)
Acc. (1)
mloU (t)
Acc. (1)
Bilinear FeatUp
59.48
79.32
81.43
95.38
40.54
74.12
61.95
81.14
83.37
96.04
42.19
75.57
LoftUp
62.15
81.32
83,69
96.11
42.02
75.72
JAFAR
61.82
81.07
84.36
96.22
42.06
75.48
segmentation.
AnyUp
62.16
81.37
84.00
96.19
42.43
75.85
Depth and Normal Estimation. We follow Probe3D (Banani et al. 2024 for evaluating the downstream performance depth and surface normal estimation. Notably, resize input images, target depth and surface normal maps to 224x224 resolution, if not mentioned otherwise. Results are reported as the root mean square еттог (RMSE). For normal estimation, we further report the ac- curacy of predictions for specified angular thresholds. For depth estimation, we additionally report the amount of pixels for which the ratio of prediction to ground-truth is less than 1.25. For more details on the benchmarking, as well as the employed metrics, we refer the reader to Banani et al. (2024, App. A.3).
the score, corresponding
As shown in Tab. Un outperforms all competitors on these tasks, showcasing its strength in preserving the locality of upsampled features. In contrast, LoftUp smoothens the features per-object too much due to its training objective using object segmentation masks and therefore performs sub- optimal on tasks like surface normal estimation.
Table 3: Surface Normal and Monocular Depth Estimation. AnyUp outperforms previous up- samplers for geometric tasks. Evaluation on the NYUv2 dataset (Silberman et al. 2012).
Surface Normals
Depth (Absolute)
Depth (Relative)
RMSE (L)
11.25 (1)
22.5% (1)
30 (1)
RMSE (L)
81 (f)
RMSE (L)
61 (1)
Bilinear
32.70
0,26
0,53
0.66
0.4925
0.8081
0.3582
0.9112
FeatUp
32.69
0.25
0,53
0,66
0.4816
0.8156
0,3413
0.9193
LoftUp
33.94
9.26
0.51
0.64
0,4847
0,8127
0,3478
0,9166
JAFAR
31.54
0.28
0.56
0.68
0.4906
0.8052
0.3481
0,9180
AnyUp
31.17
0.29
0.57
0.69
0.4755
0.8216
0.3378
0.9233
Upsampling from Any to Any Resolution. We follow the same evaluation procedure as described before but now vary the input feature resolution and target upsampling size, where the latter also corresponds to the size of the guidance image given to the upsampler methods. The semantic seg- mentation results are obtained on the COCO dataset. As FeatUp (Fu et al. 2024) does not support varying upsampling ratios, always upsample by a factor of 16x and subsequently perform bilin- ear downsampling to the desired output resolution. Results are shown in Tab, 4 AnyUp outperforms its competitors across most resolution changes while only slightly performing worse than the best competing method when upsampling from 16 - 112 pixels.

---

Published as a conference paper at ICLR 2026
Table Upsampling from any to any resolution. Linear probing results for Semantic Segmenta- tion (COCO) and depth estimation when varying the feature map and output resolutions.
Semantic Segmentation
Depth Estimation
16 112
32 224
32 112
16
112
32
224
32
112
mloU (t)
Acc. (t)
mloU (T)
Acc. (1)
mioU (f)
Acc. (1)
RMSE
RMSE
RMSE
RMSE
RMSE
RMSB
(460111
(Mb111
(1b6
(w011)
(9001)
(Wb(C)
Bilinear
56.38
77.17
59.42
79.28
59.40
79.27
0.4927
0.3586
0.4606
0.3274
9,4600
0,3223
PeaUUp
79.15
$1.10
0.3145
0.4513
0.3160
Up
18.92
29.32
61.68
81.06
61.20
80.69
0.4896
0.3533
0.5991
0.3264
0.4636
0.3296
JATAR
59.79
79.87
$1.14
61.66
80,94
0,4871
0.3458
0.4825
0.3489
0.4812
0.3498
ApyUp
59.63
79.75
62.25
81.41
62.07
81.26
0.4746
0.3364
0.4441
0.3079
0.4455
0.3073
Feature Space Preservation. Ideally, the up- sampled features should stay in the same space and distribution as the original, low-resolution features. If that is the case, a linear probe trained on the original feature space should di- rectly transfer to the high-resolution features outputted by a given upsampler for the same input feature extractor, without any finetuning. We test how much different feature upsampling methods preserve the feature distribution, by using linear probes pre-trained on the origi- nal DINOv2 (ViT-S) feature extractor (Oquab et al. 2023 2 In T
Table 5: Feature Space Preservation. Seman- tic Segmentation (ADE20k) and Depth Estima- tion (NYUv2) with linear probes pre-trained on low-resolution DINOv2 features. AnyUp retains the input feature distribution while improving up- sampling quality. LoftUp does not retain the input feature distribution, hence its results are heavily degraded. Guided Filtering requires tuning of hy- perparameters per sample.
Semantic Sogmentttiio
Depth Estimation
mloUi (t)
Acc. (1)
6g
5] we evaluate this for two different tasks, depth estimation on the NYUv2 dataset (Silberman et al., 2012) and seman- tic segmentation on the ADE20k dataset (Zhou
Bilineur
39.73
73.32
0.505
0,816
Guided Filter
37.54
72.25
0.518
0.813
FenUUp
40.19
74.05
0.504
0.818
LafUp
4.27
46.58
0.765
0.802
JAFAR
39.06
J3.75
0.503
0.815
AnyUp
40.83
74.94
0.498
0.822
et al. 2018). Qualitative results are in App.
Up preserves the input features the best, while improving the prediction quality at higher res-
olution through improved upsampling, when compared to learnable and heuristic methods. On the other end of the spectrum, predictions using LoftUp are heavily degraded, which can explained by the affinity matrix loss employed in its training. aligning intra-image feature similarities instead of directly supervising the predictions with the target features (Huang et al. 2025 Sec. 4.2). We include the implicit feature upsampling version of FeatUp (Fu et al. 2024), whose computational cost also prohibits upsempler-specífic probe training, in a similar experiment on images with lower resolution in App. B.2
Test model
Somantic Segmentation
WT-S It
VIT-R
I
VTEL
Depth Estimation
RMSE (abs.) (1)
0.4
RMSE (rel.) (
0.3
mloU (1)
Acc. (1)
0.3
0.2
0.2
0.1
0.1
VITSS 0
VITSS VIT-B
VIT:S
VIT.S
Train model
Train model
Train model
Train model
Figure 6: Generalization to other model sizes. We vary the model used during upsampler training and the model used for linear probing (test model) for AnyUp. We observe: (1) the general trend viT-L > VIT-B 2 VIT-S in linear probing holds no matter the model used in training and (2) there no significant degradation in upsampling quality when training smaller, less powerful VIT.
We note that pre-trained linear prohes and other feature upsampling methods were not released for the recent DINOv3 (Xue 2025 method, hence we stick to DINOv2.
8

---

Published as a conference paper at ICLR 2026
Computational Performance. The proposed window-bused attention used for the main upsam- pling operation can be implemented efficiently, reducing the runtime and memory requirements by more than 50% compared prior works JAFAR and LoftUp, while being slightly less efficient than FeatUp. More details on parameter count, FLOPPs and runtime, as well as memory usage in forward and backward passes be found in App. B.T
5.2
AGNOSTIC UPSAMPLING
In Fig. we visualize PCA pro- jections of features, where the same feature upsampling model instance, i.e., a model that was trained only on DINOv2 ViT- S features, is applied on fea- tures extracted from image en-
Table 6: Generalization to other models. AnyUp matches or
surpasses the performance of prior upsampling methods while being trained on a fundamentally different feature extractor.
When trained with the same feature extractor used during test time, further small improvements are possible.
Train Model
Test Model
mloU(AAc
RMSE (abs /rel)
coders ranging from ResNet (He al. 2016) to DINOv3 (Xue al.
AnyUpp
DINOv2 (VI--S)
Sigl.P 21o0pp
$1.68 (73.35)
0.59
AnyUp
SigLIP 2 20
SigLIP 2top
54,45 (75.49)
0.57 0.44
Loftlp
SigI.IP 2
SigLIP glanpp
40.73 (64.87)
0.72 0.60
2025). In Tab. 6 we
AnyUp
DINOv2 (VIT-S)
SigLIP 2ARRR
58,51 (78.36)
0.91 /0.58
showcase the remarkable gener- alization capabilities of AnyUp, where a model trained on DI- NOv2 and tested SigLIP 2 outperforms or comes close to the performance of models directly trained on SigLIP 2. Also, a DINOv2-trained AnyUp model generalizes well to DINOv3 features. Finally, as detailed in Fig. 6 An generalizes well across different DINOv2 architectures.
AnyUpp
SigLIP 24AN
SiglIP 2AA
60.32 (79.57)
0.90/ /0.57
JAFAR
SigLIP 2 AARA
SigLIP 2ARA
60.10 (79.40)
0.93 / 0.60
ApyUp
DINOV2 (VIT-S)
DINOv3 (VIT-S
0.53/
AnyUp
DINOV3 (VIT-S")
DINOV3 (WI--S )
62.99 (81.84)
0.51
Training on Multiple Feature Extractors. Our feature-agnostic upsampler architecture allows for training on multiple vision backbones. We make of this property in experiment test whether such training can increase generalization performance to completely unseen features at test time. In this experiment, we train Un on features from the following models: DINOv2 (ViT-S), CLIP (ViT-B). Lig (ViT-B), DINOV2 w/ registers (ViT-S), and ViT-B trained on ImageNet clas- sification. In Tab. 7 we demonstrate that such training strategy can further increase generalization to unseen test features (i.e., DeiT (ViT-B)), increasing overall performance across test feature ex- tractors. We note that this strategy comes at the price of small degradation of quality on DINOv2 features when compared against the encoder-specific upsampler, which we attribute to overfitting on the encoder-speciffe positional encoding artifacts of the latter.
Training. Training on multiple feature extractors achieves compara- ble performance on seen backbones while improving generalization to unseen ones, outperforming encoder-specific training in cross-backbone evaluation.
Table 7:
DENO+2
SigT.IP
DcTT
Avcrage:
An mained
mlot:
(19
079
(sho ())
(el)) (1)
mlnU
471
(1)
(16) (1)
(L))
mleU
(13
(7)
(sho
(ell)
(72
(1)
(shd
(ee))
DENOO2
62.16
$1.37
0,4755
0.3578
58.51
78.36
0.9111
0.5980
53,93
75
116667
0.897
78.54
0.6778
0,4709
Multi Buckboos
62.04
81.30
0.4767
0.3392
59.53
79.07
0,9088
0.5776
54.33
76.56
IU6418
0.4886
58.90
78.95
0,6758
0,4635
5.3 ABLATION STUDY
In Tab. g we find that the performance of UnyUp surpasses all its ablations. is also evidenced that all proposed components lead to notable impact. Note that for the ablation of the data sampling, replace ours by the simpler training approach of JAFAR Couairon al. 2025)).
Table 8: Ablations. Effects of removing specific model or training components.
mloU (Acc.) (1)
RMSE (abs rel) ())
Any
62.16 (81.37)
0.4755/ / 0.3378
window alln.
62.12 (5L.34)
0.34449
wro our data samping
4.3 1.3.1
62.03 (81 28)
0.4773 0.3387
Luttereeeery
43.2
52,02 (81,33
0.4763 0.3363
w/o regularization
4.2
61.90 (81.25)
0.4786 0.3401
Remarkably, we observe that when remov- ing the information flow from the input
path key computation
61.97 (81.23)
0

---

Published as a conference paper at ICLR 2026
feature map to the key computation in the upsampling model (see Fig. results that match the performance prior feature upsampling methods, see, e.g., RMSE for abso- lute depth estimation in Tab. This indicates that an upsampling strategy based only position and color matching between a high-resolution image and its downsampled equivalent can already provide relatively strong prior for feature upsampling when trained with pipeline.
still able obtain
To further explore this property and quan- tify the effectiveness of our proposed feature-agnostic layer (Sec. 4.I. we vary the numbers of basis filters an addi- tional experiment. As can be seen in Tab. [ upsampling performance increases with larger filter basis size in the feature- agnostic layer, thus validating our hypoth- esis that successful upsampler also needs to be able to see the low resolution fea- ture map to distinguish between image ar- eas that are of similar color but of different semantics. We decide
Table 9: Impact of Filter Basis Size Performance. Scores highlighted from best (gray) to worst (white).
We show the previous state-of-the-arr results in sepa- rated line for comparison.
M
mloU (t)
Acc (?)
RMSE (abs) ())
RMSE (rel) (4)
256
62.19
81.38
0.4765
0.3371
128
62.16
81.37
0,4755
0.3378
64
62.09
81.34
0,4759
0.3380
32
62.07
81.31
0.4779
0,3402
62.09
81.32
0.4773
0,3404
8
62.00
81,27
0.4792
0.3415
61.97
81,23
0.4791
0,3441
filter basis size of M = 128 in our implementation.
SOTA
62.15
81,32
0,4816
0.3413
LIMITATIONS
FeatSharp (Ranzinger et al. 2025) combines feature upsampling with explicitly de-noising the input features from positional encoding artifacts (Yang et al. 22024). We note that this learned por-model denoising step on the input features can be easily prepended to our pipeline but as the learned denoising weights have not been made public the time of submission, analysis in our experiments. Further, through our training strategy on image parts, our model is effectively also trained to mostly ignore such positional encoding artifacts.
do not include this
In addition, our upsampling approach based on single image-to- tot-tto-faatuur-ttttttioo is computa- tionally efficient and outperforms previous works while heing more general. However, relies on the simplifying assumption that upsampled features can be computed as linear combinations of low-resolution input features: patch feature almost certainly encodes sub-patch-level spatial in- formation in its high-dimensional channels, see, e.g., works trying to reconstruct RGB images from patch features (Bordes et al.) 2022.). Such information could likely be extracted and used feature upsampling when using larger and more complex upsampling model.
CONCLUSION
We introduced AnyUp, trst-offitss-knnd method for feature upsampling from any resolution to any resolution, which generalizes to feature representations that it was not trained on. Key technical novelties include feature-agnostic layer, windowed attention, and training strategy, which work together to achieve state-of ofte--of-the-ar upsampling quality. We make our code and models publicly available at http:///gghuubbcooo//mmmeeettti
Acknowledgements This work was partially funded by the Saarbrücken Research Center for Visual Computing, Interaction, and Artificial Intelligence (VIA). Thomas Wimmer is supported by the Max Planck ETH Center for Learning Systems. Jan Eric Lenssen is supported by the German Research Foundation (DFG) 556415750 (Emmy Noether Programme, project: Spatial Modeling and Reasoning).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Mohammad Asim, Christopher Wewer, Thomas Wimmer, Bernt Schiele, and Jan Eric Lenssen. Met3r: Measuring multi-view consistency in generated images. In 2025 IEEE/CVF Confer- Computer Vision and Pattern Recognition (CVPR), pp. 6034 6044. IEEE, 6 2025. doi: URL http: //dx doi http:://A...ad....coo//...0///pp
10. N0.1199/cDD52734.2225.000
2025. 00566
Mohamed El Banani, Amit Raj. Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Ru- binstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3d IEEF/CVF Conference
Computer Vision and Pattern Recognition (CVPR), pp. 21795-21806 IEEE, 6 2024, doi: 10.1109//eo 10.1009/vp:52733.20000 02059, URL http: //x doi org. 1109 http9//vv555222222 2024. 02059
awareness of visual foundation models. In 2024
Florian Bordes, Randall Balestriero, and Pascal Vincent. High fidelity visualization of what your self-supervised representation knows about. Transactions on Machine Learning Research, 2022,.
Mathilde Caron, Hugo Touvron, Ishan Misra, Herve Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9630-9640, IEEE, 10 2021. doi: 10. (000/fccc4889222222.005555 Utthttp: http:/xx.ioooo.oo doi http://.ooo.r/////////o
00951
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with lo- implicit image function. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8624 8634. IEEE, 6 2021. doi: 10. 11110///or444437.2211.0000 http: doi. wwt/ /10 1wt.: 6437 2021 00852
URL
Paul Couairon, Loick Chambon, Louis Serrano, Jean -am-meelu Haugeard, Matthieu Cord, and Nicolas Thome. JAFAR: Jack any feature at any resolution. arXiv preprint arXiv:2506. 1115506611166.1 11136, 2025.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- ence on Learning Reprexentations, 2021.
Francis Engelmann, Fabian Manhardt, Michael Niemeyer, Keisuke Tateno, and Federico Tombari. OpenNeRF: Open set 3d neural scene segmentation with pixel-wise features and rendered novel views. In The Twelfth International Conference on Learning Representations, 2024.
Mark Everingham, S. M. Ali Eslami, Lue Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. Interna- tional Journal of Computer Vision, (11(1):9--366, 6 2014. ISSN 1573-1405. doi: 10.1007/ doi www.:oo.....ooooooooo / 014 014-0733----01 0733-5
$11263-014-0333-5. URL BtLLhtt::///
Jiashi Feng, Zilong Huang, Bingyi Kang, Xiaogang Xu, Lihe Yang. Hengshuang Zhao, and Zhen Zhao. Depth anything v2. In Advances in Neural Information Processing Systems 37, NeurlPS 2024, pp. 21875-21911. Neural Information Processing Systems Foundation, Inc (NeurIPS), 2024. doi: 10.52202/099017-06888 URL http: //tx. doi. org/ www...........22222221.. 52202/
079017-0688
Stephanic Fu, Mark Hamilton, Laura E. Brandt, Axel Feldmann, Zhoutong Zhang, and William T. Freeman. FeatUp: A model-agnostie framework for features at any resolution. In The Twelfth International Conference on Learning Representations, 2024.
Kunihiko Fukushima. Visual feature extraction by a multilayered network of analog threshold el- ements. IEEE Transactions Systems Science and Cybernetics, 5(4):322-333, 1969. ISSN 0536-1567 doi: 10. 199.09/sss..99.....00.... URL http:/ /ttp:://x..ce doi www.o/ooog/1.... 1969 300225

---

Published as a conference paper at ICLR 2026
Ali Hassani, Fengzhe Zhou, Aditya Kane, Jiannan Huang, Chich- Yun Chen, Min Shi, Steven Wal- ton, Markus Hochnerbach, Vijay Thakkar, Michael Isaev, et al. Generalized neighborhood atten- tion: Multi ciltessiimniioall sparse attention at the speed of light. arXiv preprint arXiv:2504.1622, 2025..
Kaiming He, Jian Sun, and Xiaoou Tang. Guided image filtering. IEEE transactions on pattern analysis and machine intelligence, 35(6):1377-1409, 2012.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 778-778. IEEE, 6 2016. doi: 10. 10.11099cv::2206.990 URLhttpp /tctp://dx.c doi. 1tt.:i.orrg1111100000000.. cvpr. 2016.016990 90.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked autoencoders are scalable vision learners. In 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 15979-15988, IEEE, 6 2022. doi: 10.1109/cv9:22688. 2022.01553. URL http:h/t:/ http:/xxxcccc.coo org 1119: 2022
Haiwen Huang, Anpei Chen, Volodymyr Havrylov, Andreas Geiger, and Dan Zhang. LoftUp: Learning a coordinate-based feature upsampler for vision foundation models. arXiv preprint arXiv:2504. 14032, 2025.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In 2023 IEEF/CVF International Conference on Computer Vision (ICCV), pp. 3992 4003. IEEE, 10 2023. doi: 10. 10.10000ic55500702223..........1 URL http:/ttp//c.. //tp org 10. www.:110//cccco 111111101///111111
2023.22220 00371
Alex Krizhevsky, Iya Sutskever, and Geoffrey E. Hinton. ImageNet classification classifciaao lutional neural networks. Communications of the ACM, 60(6):84 0008884900. 2017.
with deep convo-
Y. LeCun, Boser, Denker, D. Henderson, R. Howard, W. Hubbard, and L. D. Jackel. Back- propagation applied to handwritten zip code recognition. Neural Computation, 1(4):541-551, 12 1989. ISSN 1530-888X doi: 10.1 10.116///co..9888
URL
orq/10.
11 neco. 199999..9 ...44.511 541
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft COCO: Common Objects in Context, pp. 740-755. Springer International Publishing, 2014. ISBN 9783319106021.
Wenze Liu, Hao Lu, Hongtao Fu, and Zhiguo Cao. Learning to upsample by learning to sample. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6004-6014, IEEE, 10 2023. doi:
URL http: //tp: doi http://tt......./.....o/o/o.. wttp:o...100...01...1...........
1cev51070, 2023. 00554
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Hao Lu, Wenze Liu, Zixuan Ye, Hongtao Fu, Yuliang Liu, and Zhiguo Cao. SAPA: Similarity-aware point affiliation for feature upsampling. Advances in Neural Information Processing Systems, 35: 20889-20901, 2022.
Augustus Odena, Vincent Dumoulin, and Chris Olah. Deconvolution and checkerboard artifacts. Distill, 1(10), 10 2016. doi: (0.23915/distill.00003 URL Sttt:://xx.di..ozg/// www..doi..ooo www.:o...........
23915/distin 10003)
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fermandez, Daniel HAZIZA, Francisco Massa. Alaaeldin El-Nouby, et al. Dinov2: Leam- ing robust visual features without supervision. Transactions on Machine Learning Research, 2023.
12

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PmLR, 2021.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. Advances in neural information processing systems, 32, 2019.
Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. AM-RADIO: Agglomerative vi- sion foundation model reduce all domains into one. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1P.1124 12490-12500. IEEE, 6 2024.
Mike Ranzinger, Greg Heinrich, Pavlo Molchanov, Bryan Catanzaro, and Andrew Tao. FeatSharp: Your vision model features, sharper. In Forty-second International Conference Machine Learn- ing, 2025.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor Segmentation and Support Inference from RGBD Images, pp. 746-760. Springer Berlin Heidelberg, 2012. ISBN 9783642337154.
Saksham Suri, Matthew Walmer, Kamal Gupta, and Abhinav Shrivastava. Lift: A surprisingly sim- ple lightweight feature transform for dense vit descriptors. In European Conference on Computer Vision, pp. 110-128. Springer, 2024.
C. Tomasi and R. Manduchi. Bilateral filtering gray and color images. Sixth International Conference on Computer Vision (IEEE Cat. No. No.998H33271), ICCV-98, pp. 839-846. Narosa Publishing House, 1998. doi: 10.1109/iccv. 19981111....... URI. http: doi org./orr/.....
1109/iceve 1998. 710815
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Nacem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14788. 2025.
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. VGGT: Visual geometry grounded transformer. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5294-53066 IEEE, 6 2025. doi: 10. 1a.110//vp5527342225..040
URL http:
2025. 00499
Jiaqi Wang, Kai Chen, Rui Xu, Ziwei Liu, Chen Change Loy, and Dahua Lin. CARAFE: Content- aware ReAssembly of FEatures. In 2019 IEEE/CVF International Conference Computer Vision (ICCV), pp. 3007-3016 IEEE, 10 2019. doi: 10.1109//cw22219900031 doi. ooi.org/1e 10. 1109/icev 2019 0011900010
URL http: //dx
Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Recovering realistic texture in image super-resolution by deep spatial feature transform. In 2018 IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 606-615. IEEE, 6 2018. doi: 10.1 10.1109/cpp:22188.0007 URL
Sttp::/dde doi org 110111109///0 2018 00070
Monika Wysoczańska, Oriane Siméoni, Michaël Ramamonjisoa, Andrei Bursuc, Tomasz Trzciński, and Patrick Pérez. CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentation. In European Conference on Computer Vision, pp. 320-337. Springer, 2024.
Tongjun Xue, Zhenqian He, and Guanwei Wei. Dinov3-UNet: A pretrained vision transformer enhanced u-net for surface defect segmentation. In 2025 5th International Symposium on Artificial Intelligence and Intelligent Manufacturing (AlIM), pp. 39 42. IEEE, 9 2025. doi: 10. 00.11099imm667112255..22222
http//xx.o.o.ooooo..oooooo BRLhttt::// www..oo..ggo....o 10.
2025. 2025.11233022
13

---

Published as a conference paper at ICLR 2026
Jiawei Yang. Katie Z Luo, Jiefeng Li, Congyue Deng, Leonidas Guibas, Dilip Krishnan, Kilian Q Weinberger, Yonglong Tian, and Yue Wang. Denoising vision transformers. In European Confer- on Computer Vision, pp. 453-469, Springer, 2024.
Chongjie Ye. Lingteng Qiu, Xiaodong Gu, Qi Zuo, Yushuang Wu, Zilong Dong, Liefeng Bo, Yuliang Xiu, and Xiaoguang Han. StableNormal: Reducing diffusion variance for stable and sharp normal. ACM Transactions on Graphies, 43(6);1-18, 11 2024. ISSN 1557-7368. doi: 10.1145/3677971. URL htt.pp//oo.......rr//..112666663
doi
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):302-33) 321, 12 2018. ISSN 1573- 1405. doi: 10.1007//11263-0013 1140-0. URL
doi http:///kk.....crrg0000111///111 111100//111212-0---111------1111
Minghao Zhou, Hong Wang, Yefeng Zheng, and Deyu Meng. A refreshed similarity-based up- sampler for direct high-ratio feature upsampling. IEEE Transactions Neural Networks and Learning Systems, pp. 1-15, 2025, ISSN 2162-2388. doi: 10.1 10.11099mms URL
/ttp://dx: /ttp////////////oooo///////////.
14

---

Published as a conference paper at ICLR 2026
A ADDITIONAL DETAILS ON RELATED WORK
FeatUp (Fu et al. 2024):
Fu al. 2024 propose training the upsampler in "multi-vier recon-
struction" setting, where the feature computation and learned upsampling should be equivariant to small image augmentations, using learned feature downsampling operation. FeatUp consists of multiple stacked joint bilateral upsampling (JBU) layers making upsampling only possible for inte- ger factors.
LoftUp (Huang et al. 2025): Huang et al. 2025) propose stacked high-res to low-res attention.
Their training strategy consists of two stages: First, SAM (Kirillov al. 2023 is used for object mask generation. Low-resolution features are then blended per-mask using bicubic upsampling. In their second stage, self-distillation with student and teacher models is performed. Instead of using a cosine-similarte loss to compare their upsampled predictions with the ground truth, [Hung et al. 2025) propose an affinity matrix loss, where the intra-image pixel feature similarity matrices are matched. This loss is computed on high-resolution images.
JAFAR (Couairon et al. 2025): Couairon et al. (2025) propose feature upsampling as a single high- res to low-res attention. Training is performed low resolutions and small scale changes (up to 4x). While a spatial semantic feature modulation (Wang et al. 2018) is used to merge information from the downsampled image and the input feature map JAFAR, find that this is giving no- ticeable improvements and replace with simple concatenation of features followed by a standard ResNet block.
FeatSharp [Ranzinger et al. 2025): : FeatSharp builds upon FeatUp (Fu et al. 2024), where tiling of
the input image and multiple forward passes of the encoder model used get higleonne outputs. De-biasing from positional encoding artifacts learned. The weights for the upsampler are openly available, hence we do not compare against this method.
B ADDITIONAL EXPERIMENTS
B. PERFORMANCE ANALYSIS
We summarize performance in terms of runtime and peak memory usage in Tab. TO (measured single NVIDIA A100 GPU). Global attention requires / x W x h x computations, while window attention, using windows sized (relative to the input size), significantly reduces the requirement to only H x W oh x ow computations. In practice, we base our implementation on the WATTEN library (Hassani et al. 2025.), which implements CUDA kernels for several architectures that make use of the sparsity in window attention and set a = 0.2. We further extend our code to enable trade-off between memory and speed through chunked computation of the attention that enables even larger input images and feature maps.
Table 10: Performance Analysis. Up benefits from the sparsity of the window attention.
(a) Parameter count and inference time (ms) FLOPs comparison (GFLOPs) by resolution.
Memory usage (GB) by resolution for forward backward passes.
Model
Params (M)
1122
224
448
Mudel
112²
22
448
Time (ms)
FLOPS
Time (mn))
FLOPS
Time (m))
FLOPs
Fwd
Bwd
Fwd
Bwd
Fwd
Bwd
ReaUUp
8.5
2.6
10.4
73.7
LofUp
201.2
Loftip
JAFAR
4.3
5.5
113.9
22.3
AnyUp
08
2.7
5.2
12.8
20.6
92.4
82.4
AnyUp
08
0.8
3.3
12.9
B.2 COMPARISON AGAINST FEATUP'S IMPLICIT UPSAMPLING
Due to the computational burden of FeatUp's proposed implicit upsampling, where high-resolution features are optimized in a NeRF-like manner at inference time, it is infeasible to train linear probes for semantic segmentation
depth/normal estimation, done in Sec. Sec.p.1 Instead, performance when used with pre-trained linear probes from DINOv2, as in Tab. 3 We show the results in Tab. 11 Note, that evaluate images of 224px instead of 448px used in Tab. 5 as
analyze its
15

---

Published as a conference paper at ICLR 2026
Table 11: Comparison against FeatUp (implicit), Feature space preservation comparison with pre-trained linear probes for DINOv2 ViT-S features when upsampling to 224px.
Seminntic Segmentation
Depth Estimation
mloU (1)
Acc.
RMSE
6g (1)
Bilineur
35.54
70.20
0.510
0,818
Guided Filler
32.17
68.25
0.537
0.802
FeatUp
35.60
71.40
0.510
0.816
- implicit
36.70
70.95
0.513
0,818
LoftUp
3.82
44.35
0.534
0.803
JAFAR
37.52
72.64
0.505
0.822
AnyUp
37.91
72.78
0.502
0,822
the inference-time optimization of FeatUp is too costly to run on higher resolutions. As in Tab. AnyUp outperforms all prior works when using pre-trained probes.
C IMPLEMENTATION DETAILS
We train our method on the ImageNet dataset (Krizhevsky et al. 2017) for a total of 100,000 training steps. We choose batch size of 4 with 4 random local crops per training image. Training takes around hours on a single NVIDIA-H100 GPU. We use the AdamW optimizer (Loshchilov & Hutter 2019) with learning rate of 2e-4 and batch size of 4.
We note that the results for semantic segmentation linear probing reported in Sec. 5.1 deviate from the results reported by Couairon et al. (2025). This is caused by fixing bug in the probing training compared to their impiementation. We ran all experiments with the official published weights for concurrent works.
C.I SELF-CONSISTENCY REGULARIZATION AND DATA AUGMENTATIONS
Figure 7: Randomly sampled augmentations applied on a test image (top left).
We add self-consistency regularization Lyelf consistenyy also proposed in prior work (Couairon al. 2025). where we increase the level and diversity of data augmentations to increase the robustness of our upsampling method. We show batch of randomly sampled augmented images in Fig. The self-consistency regularization is computed as
Lielf consisteey
deas. mse (f(p. Inr). ). f(p, I,,))
(3)
where I/ is the noised/augmented version of the high-resolution image. The self-consistency regularization does not rely on ground-truth features & and is thus computed at higher resolution, Le., 224x224.
16

---

Published as a conference paper at ICLR 2026
D COMPULSORY NOTE ON LLM USAGE
No large language models were used in writing or ideation this paper except as aid for styling Fig.
E ADDITIONAL VISUALIZATIONS
Input Image
Input Feature Map
Attention w/o window
Attention w/ window
Pixel Query
Figure 8: Visualization of attention artifacts and removal thereof through local window attention. Unconstrained, global attention leads to upsampled features relying on information from far-away, non-related objects. We provide a simple fix to this which also simplifies the upsampling problem for the model by restricting attention only to local windows that are computed relative to the feature map size.
17

---

Published as a conference paper at ICLR 2026
Figure 9: Linear Probing results semantic segmentation using a pre-trained DINOv2 (VIT-S) probe. AnyUp preserves the input feature space while upsampling and sometimes even outperform- ing the ground-truth, e.g., the tower segmentation in the fifth row. While FeatUp's predictions are generally consistent with the low-resolution predictions, smoothens object boundaries too much and is inable to get sharp segmentations. LoftUp shifts the feature distribution while upsampling, resulting in erroneous predictions with the pre-trained probe. Predictions using JAFAR suffer from reduced details for thin objects, see, e.g., the lights row 2 or the fence in row 4.
18

---

Published as a conference paper at ICLR 2026
M0L
177
BITH BY
Figure 10: Linear Probing results for depth estimation using a pre-trained DINOv2 (ViT-S) probe. Up stands out by preserving sharp edges from the guidance image while preserving the locality of features needed for smooth depth map prediction and preserving full objects, as, e.g., the white board in the fourth row. Note that the feature distribution shift of LoftUp is less pronounced for depth estimation, which can be partly attributed to the scale-shift alignment performed after prediction.
19

---

Published as a conference paper at ICLR 2026
Brrpp
pM
V/0
Dalin
Naiing
Figure 11: Additional linear probing results for semantic segmentation using probes trained on upsampled DINOv2 (ViT-S) features. AnyUp outperforms prior upsampling methods by also being able to segment fine-details and giving cleaner segmentation outputs.
20

---

Published as a conference paper at ICLR 2026
Figure 12: Additional linear probing results for depth estimation using probes trained on upsampled DINOv2 (ViT-S) features. AnyUp consistently gives sharp object boundaries while matching the ground truth the best.
21


Reviews:
[{"review_id": "kgsKbABSUg", "reviewer": "Reviewer_kTsf", "summary": "Previous feature upsampling methods cannot generalize among different feature encoders, and need to be re-trained for different encoders. This paper proposes AnyUp, which addresses the limitation of existing upsamplers, which is feature-agnostic at inference time. Experiments validate the effectiveness of AnyUp.", "questions": "- Have you tried training AnyUp on a mixture of features from different encoders (e.g., DINOv2, CLIP, MAE) simultaneously? Would this \"universal\" training further improve its generalization to completely unseen encoders compared to training on just one?\n- You motivate window attention by noting that global attention can pull in \"vastly unrelated and distant\" features. Can you quantify the performance gain and efficiency (FLOPs/latency) improvement from using windowed attention versus the global attention used in the JAFAR baseline?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The feature-agnostic inference is meaningful for feature upsampling\n- The paper is well-written, and the method is simple and easy to follow\n- The experiments are comprehensive demostrating the effectiveness. The visualizations are sufficient to explain the design and show the good visual appreance of the generated feature.", "weaknesses": "- The techniques used in the method is not very new, for example, window attention-based upsampling has already been used in SAPA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bTKMiyGcLP", "reviewer": "Reviewer_MPXT", "summary": "This work introduces AnyUp, an algorithm for upsampling deep vision network representations while retaining their semantics. Most importantly, AnyUp expands beyond other works in the literature by introducing a layer that once-trained will work on any deep network's features. They achieve this with the introduction of a new windowed guided upsampling layer that operates on features of any dimensionality. They justify their work with a suite of experiments that test the ability of features to be used in downstream dense prediction tasks, the ability to be used as a drop-in replacement for existing tasks, and the ability to generalize across networks and resolutions.", "questions": "- How does AnyUp handle tokens that are non-spatial like those in DinoV2 before the paper Vision Transformers need registers\n- During training AnyUp uses crops of the original image to help it learn to reconstruct the intricacies of high resolution features, to what extent does this reliance on image parts change its semantics (i.e the features for an object can change when that object is small or large relative to the field of view)\n- What does AnyUp learn that enables such promising transfer? What do these filters learn that enable them to upsample? Perhaps adding a section in the appendix might help the curious get a sense for what the parameters you add learn.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Simple and elegant idea that is of interest to the community\n- Good comparisons to prior work and nice panoply of experiments\n- Nice ablations and thorough investigations of transfer to other backbones and resultions\n- Good writing and presentation", "weaknesses": "- Nit: line 53: some prior methods like FeatUp dont require evaluating the backbone on higher resolution images but rather small pixel jitters\n- Your explanation of the feature agnostic upsampler is a little hard to understand from the small paragraph provided, consider expanding this to provide a bit more intuition behind the math here. Also see questions section for another possible way to provide intuition behind this core contributionm\n- FeatUp's implicit version, which performs better than its JBU version does not seem to be compared against in this work, this method is also allows for any resolution inference, though at a time cost to perform the implicit optimization\n- Consider adding flop / peak mem comparisons\n- Some prior works evaluate in the joint training context to demonstrate that upsamplers can be used as a replacement for resize convs in unets. One might consider that experimental setting here as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VeOq0V7Sz7", "reviewer": "Reviewer_9SrJ", "summary": "This paper presents AnyUp, a universal upsampling framework that can upsample features from any encoder to any resolution. This paper contributes i) a feature-agnostic convolutional layer independent of feature dimensions, ii) a local window attention mechanism to avoid irrelevant matches introduced by global attention in space, improving the quality and efficiency of upsampling, and iii) a lightweight training strategy based on local image cropping to get rid of generating high-resolution features as supervision. Experimental results show that AnyUp achieves state-of-the-art performance on several segmentation, surface normal estimation and depth estimation benchmarks. Crucially, AnyUp can generalize effectively to vision encoders unseen during training.", "questions": "As the paper's most central technical novelty, the Feature-Agnostic Layer (Section 4.1) is clearly explained in principle, but lacks a deep experimental analysis of its internal mechanism.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  Clear Motivation and Significant Value: The \"encoder-specific\" problem is a real pain point when applying recent model-agnostic upsamplers. A \"train once, use anywhere\" upsampler is of high practical value for the dense prediction community.\n2.  Comprehensive Experimental Validation: The paper conducts extensive comparisons against strong baselines on multiple benchmarks (COCO, PASCAL-VOC, ADE20k, NYUv2), consistently achieving SOTA or highly competitive performance. Generalization experiments also demonstrate the model’s transferable ability on different feature extractors.\n3.  Insightful Analysis of Feature Space Preservation: The feature space preservation experiment in Table 5 is an interesting supplementary analysis. Beyond performance gains, it reveals why AnyUp succeeds—by maximally preserving the semantic distribution of the original features during upsampling, which offers more insights than simple metrics.", "weaknesses": "1.  Insufficient Analysis of the Feature-Agnostic Layer: How sensitive is the model to the hyperparameter M (the number of basis filters)? What kind of visual patterns do the learned basis filters (ψj) actually capture? How much performance improvement can feature-agnostic layers itself bring? What is the performance difference between the proposed feature-agnostic layer and the standard convolution? Experiments on these factors are expected for a comprehensive understanding.\n2.  Parameters and inference time should be analyzed. As an inherited version from JAFAR, the number of parameters of AnyUp and the inference time should be discussed, just like JAFAR did.\n3.  A key design choice is that the Value vector (V) in the attention mechanism directly uses \"unprocessed input features\" (line 197), justified as a way to preserve the feature space. This assertion lacks direct experimental evidence (e.g. What impact will inserting a convolution layer in the value-branch could have on performance?). \n4.  Ambiguous terms used in the motivation table: what is the meaning of “any task”? If I interpret correctly, most upsamplers (e.g., Dysample) can be used for different tasks. It is just a matter of improved or decreased performance. It seems that the paper would like to say whether an upsampler should be retrained on a new task. This is not what ‘any task’ means.\n5.\tTo strengthen the comparison of the encoder-agnostic property of AnyUp in Table 6, it is better to include results from other upsampling models when they operate on features from different encoders. Specifically, these models could be generalized to different encoders in a training-free manner (e.g., via channel-wise linear interpolation) for a supplementary comparison.\n6.\tSome typographical and formatting errors: line-090, “it is was not trained on”, line-242, “the query point (Ramachandran et al., 2019)., As a high” (This point is not considered for rating)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WyqBJ0biVP", "reviewer": "Reviewer_spdT", "summary": "This paper proposes a feature upsampling method applicable to any visual feature at any resolution without encoder-specific training. The Feature-Agnostic Layer achieves applicability to feature dimensions. The results show that it achieves state-of-the-art performance on multiple datasets.", "questions": "1. This LOCAL WINDOW ATTENTION cannot capture long-range dependencies like GLOBAL ATTENTION can. Will this lead to performance degradation?\n2. Are these artifacts introduced by the GLOBAL ATTENTION itself? Please analyze the reasons.\n3. What is the significance of sampling the image and then supervising the local part? Is it merely for being more lightweight and efficient?\n4. Table 7 shows that the model with data sampling has better performance. Logically, the unsampled supervision information should be more comprehensive and should yield better results. Could you explain why data sampling has better performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. This method makes a great contribution to the community. It solves the problem that previous methods require re-training for specific encoders and achieves more universal upsampling.\n2. The article is well-written and easy to understand.\n3. Feature-Agnostic Layer is a novel, concise and effective module.\n4. It can be seen from the experimental results that AnyUp has achieved SOTA performance.", "weaknesses": "Some details of the methods were not explained clearly. Please refer to the section Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)