FeatSharp: Your Vision Model Features, Sharper
Authors: Mike Ranzinger
Greg Heinrich
Pavlo Molchanov
Bryan Catanzaro
Andrew Tao
Venue: ICML 2025 poster (2025)
Keywords: computer vision
perception
upsampling
Primary area: applications->computer_vision
TL;DR: We provide a method of upsampling vision model features by jointly leveraging the low resolution buffer and a mosaic of higher-resolution tiles.

Abstract: The feature maps of vision encoders are fundamental to myriad modern AI tasks, ranging from core perception algorithms (e.g. semantic segmentation, object detection, depth perception, etc.) to modern multimodal understanding in vision-language models (VLMs). Currently, in computer vision, the frontier of general purpose vision backbones is Vision Transformers (ViT), typically trained using contrastive loss (e.g. CLIP). A key problem with most off-the-shelf ViTs, particularly CLIP, is that these models are inflexibly low resolution. Most run at $224 \times 224$px, while the "high-resolution" versions are around $378-448$px, but still inflexible. We introduce a novel method to coherently and cheaply upsample the feature maps of low-resolution vision encoders while picking up on fine-grained details that would otherwise be lost due to resolution. We demonstrate the effectiveness of this approach on core perception tasks as well as within agglomerative model training using RADIO as a way of providing richer targets for distillation. Code available at https://github.com/NVlabs/FeatSharp

FeatSharp: Your Vision Model Features, Sharper
Mike Ranzinger' Greg Heinrich 1*
Pavlo Molchanov
Bryan Catanzaro
Andrew Tao
Abstract
FeatUp FeatSharp
The feature maps of vision encoders funda- mental to myriad modern Al tasks, ranging from core perception algorithms (e.g. semantic segmen- tation, object detection, depth perception, etc.) to modern multimodal understanding in vision- language models (VLMs). Currently, in com- puter vision, the frontier of general purpose vi- sion backbones is Vision Transformers (ViT), typ- ically trained using contrastive loss (e.g. CLIP). A key problem with most off-the-shelf ViTs, par- ticularly CLIP, is that these models are inflexi- bly low resolution. Most run at 224 X 224px. while the "high-resolution" versions are around 378 448px, but still inflexible. We introduce a novel method to coherently and cheaply upsam- ple the feature maps of low-resolution vision en- coders while picking up on fine-grained details that would otherwise be lost due to resolution. We demonstrate the effectiveness of this approach on core perception tasks as well as within agglom- erative model training using RADIO as way of providing richer targets for distillation. Code available at http://giitubbcoo///aab//////i
VIT
DFN CLIP
SigLIP
PalíGermms
DINOv2-L-reg
1. Introduction
SAM
Vision foundation models (VFM) (Awais et al., 2023) have seen widespread use since the beginning of the modern era of computer vision using deep learning (Krizhevsky et al., 2012), primarily used perform transfer learning (Plested & Gedeon, 2022) (e.g. finetuning VFM on downstream task), information retrieval (Babenko et al., 2014; Zhang & Liu, 2024), and most recently, to power visual capabilities for vision-language models (VLM) (Alayrac et al., 2022; et al., 2024: Liu et al., 2023; Lin et al., 2023). The re- cent shift toward using Transformers (Vaswani et al., 2017) for computer vision (ViT (Dosovitskiy al., 2021)) has tremendously moved the field forward, but has generally
RADIO12.5L
Figure I. PCA visualizations of features from basketball scene. Column 1: Raw features produced by the model at normal reso- lution (e.g. 14x downsample for DFN CLIP. SigLIP, PaliGemma.. and DINOv2, 16x downsample for SAM and RADIOv2.5-L. Col- umn 2: Raw features the 4x upsample resolution (we interpolate the position embeddings for those models that don't natively sup- port resolution changes). Column 3: FeatUp-JBU 4x upsampling (prior work). Column 4: FeatSharp 4x upsampling. NOTE:
'quaal contribution 'NVIDIA. Correspondence to: Mike Ranzinger <mmranzngerr@iiidicco...
"Real 4x" technically only makes sense for models with strong scale equivariance, such as DINOv2, RADIO, and SAM.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

FeatSharp: Your Vision Model Features, Sharper
JBU Upsample
Prediction
Low- Cow-Resolution Features
FestUp
JBU
Featurizer
Tiled Features
FeatSharp
Full-Res Reference
Do-Blas
Repeat with Tiles
Figure 2. Upsampling architecture diagram. We combine the upsampled features coming from FeatUp (Fu et al., 2024) with the tiled features and mix them with FeatSharp to produce a feature map with higher fidelity. The tiled features have more detail, but also representation issues such as the difference in upper and lower body at the tile boundary. "Full-Re Res Reference" is for display purposes, as for model that docsn't 't exhibit stable resolution scaling (e.g. DFN CLIP, SigLIP, etc.) don't have access a target hi-res feature map. Learned modules have a fire icon, and frozen modules a snowflake.
left the use of VFMs in a tricky spot: Transformers are com- putationally demanding and have poor algorithmic scaling properties (O(n²) for 1D sequences, or O((w- h)² for 2D inputs), leaving the majority of models to be relatively low- resolution. For example, perhaps the most popular family of VFMs to date, CLIP (Radford et al., 2021), typically runs at 224 or 336px input resolutions, and produces spatial features at a 14x downsample (e.g. 2242 - 162). Owing to the nature of learned position embeddings, ViTs also tend to be relatively inflexible to changes of input resolution, allowing for changes, but requiring finetuning (Dosovitskiy et al., 2021).
image-specific implicit network. While they demonstrate particularly compelling results with their implicit network, their results using the stack of JBU filters lack refined details (shown in figure 17 in the appendix). Along with the lack of granular refinement, it's impossible for this approach to capture fine-griined details that are too small for the vision backbone to detect at its native resolution. To this end, we take inspiration from both FeatUp's 's JBU approach, as well as the recent trend in VLMs such as LLaVA 1.6 (Liu et al., 2024), Intern (Chen al., 2024), NVLM (Dai et al., 2024b) and Eagle (Shi al., 2024b) to tile an image, aggre- gating local features from soeed-low-resolttoon model, to build an upsampler that simultancously leverages the raw pixel guidance. low-res feature guidance, and regional tile guidance, resulting in substantially more detailed feature maps which are also capable of capturing details too small for the original resolution. Specifically, we:
It is possible that the strict dependence on the training res- olution is an artifact of the algorithm used for training, as DINOy2 (Oquab et al., 2023; Darcet et al., 2023) is quite ro- bust to interpolating its position embeddings, producing sta- ble features at various resolutions (Ranzinger et al., 2024b), ignoring for the moment that DINOv2, being transformer, is expensive use at high-resolution. A recent technique called AM-RADIC (Ranzinger et al., 2024a), borrowing ideas from ViTDet (Li et al., 2022), Fic (Beyer et al., 2023), and RO-ViT (Kim et al., 2023), has attempted to cre- ate resolution-flexibe ViT, however it is still dependent on low-resolution ViTs as distills from other seminal VFMs which are low-res only: DEN CLIP (Fang et al., 2023) and SigLIP (Zhai et al., 2023).
Build on top of FeatUp's JBU algorithm (Fu al., 2024) by adding de-biasing and tiling fusion modules to incorporate detailed tile features, resulting in signif- icantly higher levels of detail, with extensive experi- ments demonstrating effectiveness.
Study the relationship between FeatUp's U feature consis- tency and ViT-Denoiser's (Yang et al., 2024a) approach to cleaning the features of a ViT at its native resolution.
Recently, FeatUp (Fu et al., 2024) aims to directly ad- dress the problem of low-resolution vision features by us- ing one of two learned upsampling algorithms: A model- specific generalized upsampler using Joint Bilateral Up- sampling (JBU) (Kopf et al., 2007), or -peel-s-e
Introduce an improved training setting for AM-RADIO (Ranzinger et al., 2024a) demonstrating a +0.6% im- provement across the entire benchmark suite, and better teacher adapter features.
2

---

FeatSharp: Your Vision Model Features, Sharper
pling, we can also use small, fixed budget of inferences (similar in spirit to their implicit model), and use mosaic of tiles guidance at the higher resolution. This choice gives us a richer, and semantically relevant, feature space to merge from. Additionally, it allows us to incorporate information from regions that were too small for the low-res view, but become visible within a tile. Small details are limitation of every approach that doesn't acquire extra samples from the base model, as they rely on all relevant information already being encoded by the initial model evaluation.
Slice First Half
SWiGLU MLP
Feature Denoising Related to multi-view consistency, ViT-Denoiser (Yang et al., 2024a) noticed that ViT fea- tures are generally very noisy (although some are much cleaner than others), and also propose a multi-view con- sistency formulation to learn how to separate fixed noise, conditional noise, and semantic content. We notice the deep ties between ViT-Denoiser and FeatUp, that multi-view consistency provides a way to eradicate fixed-patter noise from the feature buffer. Drawing inspiration from this, we add a learnable bias buffer (similar to learned position em- beddings) at the output of the base model. This simple change works because fixed patterns will degrade multi- view consistency, as the pattern is always local to the view, and lacks global coherence.
Local MHSA
Channel Concatenate
JBU Upsample
Tiled Mosaic
Figure 3. Diagram of the FeatSharp module. We first concatenate the JBU upsampled and tiled mosaic feature maps along the chan- nel dimension. We then apply transformer block with sliding window attention followed by MLP (in this case. SwiGLU), and then slice off the first half of the channels, corresponding the bilinear upsampled huffer. The role of FeatSharp thus is to refine the JBU buffer by leveraging the tile buffer.
VLMs The use of tiling to increase information is cur- rently very prominent in VLMs (Liu et al., 2024; Chen et al., 2024: Dai et al., 2024a), albeit an alternative ap- proach is to instead leverage the models at hi-res themselves (Beyer et al., 2024; Wang al., 2024). We also see RA- DHHei.c(teii et al., 2024) being primarily useful at high-resolution within VLMs. In the increasingly VLM- centric approach to computer vision, we turn our focus to RADIOv2.5, as it has a training procedure that relies on matching a high-resolution student against low-resolution teacher, an application area that is perfect for studying fea- ture upsampling, it would provide richer guidance to the distillation.
2. Related Work
Feature Upsampling The most obvious baseline for fea- ture upsampling is to use traditional filtering approaches such as bilinear or bicubic upsampling. The alternative to evaluate the network at higher resolution, however it comes with the dual drawback that computational cost in- creases (quadratically in the case of Vision Transformers), and also that many models (ViTs in particular) have trou- ble extrapolating from their trained resolution (Beyer et al., 2023; Dehghani et al., 2023). If we expand our view to include parametric approaches, then deconvolution (Noh et al., 2015; Shi et al., 2016; Dumoulin & Visin, 2016) and resize-conv (Odena et al., 2016) are popular choices. There are also adxel-adaptive approaches such CARAFE (Wang et al., 2019). SAPA (Lu et al., 2022), and FeatUp (Fu et al., 2024).
Agglomerative Models In the agglomerative model space, there are currently three major approaches: RADIO (Ranzinger et al., 2024b;a; Heinrich et al., 2024). Theia (Shang et al., 2024), and UNIC (Sariyildiz et al., 2024). We focus our attention on RADIO because is the only approach that directly tries to tackle resolution flexibility as well as high high-resolution.
3. Method
We leverage FeatUp's 's training algorithm of treating the up- sampling problem as that of multi-vieen consistency between the upsampled and then downsampled features and different low-res views of the same image.
We adopt FeatUp's formulation of multi-view consistency as a way to train an upsampler, however, we notice that instead of solely relying on raw RGB pixels as guidance for upsam-
3

---

FeatSharp: Your Vision Model Features, Sharper
Encoder
Figure Visualization of the tiling process. An input image (left) is split into tiles, each of which is resized to match the input resolution of the encoder, fed through the encoder independently. and then stitched back into a higher resolution feature map. Feature maps shown are from DFN CLIP, and they are resized to be larger than actual for demonstration purposes.
3.1. Review FeatUp: Joint Bilateral Upsampling (JBU)
factorize 2 to get a set of upsample factors, using a JBU,, for each prime factor. This decomposes to an identical operation before when log2 2 Z+. but allows for an easy guide for any other integer, e.g. for 14x upsam- ple corresponding to a patch-size- 14 backbone, we'd use (JBU7x o JBUxxx (f()), stack.
Given a high-resolution signal G (e.g. the raw pixels) as guidance, and a low-resolution signal Fir that we'd like to upsample, and let Ω be neighborhood of each pixel in the guidance. Let -) be similarity kernel that measures how close two vectors are. Then
As is typical with bilateral upsampling, this method is very sensitive to strong edges in the guidance buffer, however, it also tends to over-smooth features in regions of lower contrast. Particularly, struggles with feature patterns such as SAM (figure 1) where there are interior edges in feature space, but not pixel space. This results in the features being blurred inside of objects.
1
Σ
(irla,
(a.
(1)
Krange (G G(a, 1 Kspatial (li. j], (a, bl))
We don't 't make any changes to their downsampler, instead opting to just use their Attention Downsampler without modification. We then focus on two changes, one to output normalization, and the other how upsampling guidance is computed.
with Z being a normalization to make the kernel sum to 1. Kepatial is a Gaussian kernel with learnable a Eapatial defined as
| y112 20 spatial
Eapatial v) exp
3.2. Feature Normalization
FeatUp supports either leaving the features coming from the backbone as-is (e.g. no normalization), using Lay- erNorm to better condition the outputs for feature learning, For a similar motivation as PHI-S (Ranzinger et al., 2024a), we want to avoid using the raw featurre as they have very distinct distributions, and we' d also like to avoid using Lay- erNorm as it makes the features incompatible with the orig- inal feature space. Naïvely learning the raw feature space across the suite of teachers without normalization often led to convergence issues, particularly given the wide variance of activations. Thus, we adopt PHI-S as way to standardize the backbone features without distortion and to retain the ability to model the original distribution. We compute the distribution statistics over 100k samples from the training dataset.
and kronge as
krange (F. v)
softmax (a.))e
h(G(,,)) h(G(a,b))
orange
with h(x) being learned MLP projector. They define
Fur (JBU(.,z) o (BU(, o..)) (f(x),)
(4)
as stack of 2x upsamplers, thus enabling pfwer-of-2 up- sample factors. With being the original input image and f(z) being the low-resolution feature map. We note that 2x is not a necessary part of the architecture, and that their implementation supported arbitrary factors, so we simply propose to take given upsample factor z E Z4 and prime

---

FeatSharp: Your Vision Model Features, Sharper
JBU upsampled buffer with the tiled feature map and feed it to the block. After the block is computed, we slice the first C' dimensions of the output, with C' being the model feature dimension, and treat that as the refined features. The slic- ing strategy takes advantage of the fact that a transformer block has a residual pathway, and thus a no-op from the transformer would be equivalent to returning the bilinear upsampled features. Through the attention mechanism, the model is able to consider the local neighborhood and refine its features to achieve better multi-view consistency. To this end, we train our model identically to FeatUp's multi-view consistency algorithm. We do not employ any special loss functions beyond the MSE loss on multi-view consistency, contrary to FeatUp's use of Total Variation and Conditional Random Field losses. We provide ablations wrt architecture choice in appendix B.2.
Figure 5. Visualization of 2 upsampling using bilinear (leff) ver-
sus tiling (right), using the DFN CLIP encoder.
3.3. Tile-Guided Attentional Refinement
Joint-Bilateral Upsampling is able to retain object bound- aries primarily in instances when there are noticeable changes in intensity in the RGB input image. This results in sharp contours, but within a region, we end up with vague and blurry feature representations. Owing to the reliance on raw pixel intensities, object contours that are less discrimina- tive in color space often get blurred with the neighborhood. Finally, because the upsampling operation is only truly op- erating on the low resolution feature maps of the model, it's impossible for JBU to introduce new details into the feature map that are visible/encodable at higher input reso- lutions. FeatUp's implicit upsampler doesn't 't have this same limitation because it's constructing global view from nu- merous local views of the original image, enabling detailed refinements. We propose an intermediary method between JBU which leverages single featurizer inference, and the implicit model, which relies on numerous inferences and is thus cost prohibitive'.
3.4. Denoising
Drawing inspiration from (Yang et al., 2024a), we notice that the problem formulation has very similar solution to FeatUp (and ours), owing to the fact that all methods are using multi-view consistency and thus learn to eliminate position-based artifacts. From their formulation:
VTT()) f(x) + ((Epos) + h(z, Epos) ((Yang et al., 2024a), Eq 5)
We add a learnable g buffer, such that
Inspired by the use of tiling in Vision-Language Models (VLMs) (Liu al., 2024; Shi et al., 2024a; Dai al., 2024b), we develop an attentional refinement block that is able to integrate the information between a JBU upsampled feature map, as well as a feature map composed of tiles. We show an overview of the algorithm in figures 2, 3 and 4. The diagram shows actual results using RADIOv2.5-L. which the most scale equivariant foundation model (Heinrich et al., 2024), and generally the strongest visual foundation model (Lu et al., 2024; Drozdova et al., 2024; Guo al., 2024). Because the model has strong resolution scaling, it provides us with good way to compare the results of the upsampling process against the feature maps of the same resolution attained by increasing the resolution of the input image. We also observe that even just 4x tiling, there are major discontinuities in the tiled feature map, which the FeatSharp module must overcome produce unified higher-resolution image.
f(x) = f(x) +
(5)
with f(x) being the frozen vision encoder. The learnable I allows our model to learn and negate the fixed position artifacts that the encoder produces. Notably, given that we are also using the base model for the tiles, this learned buffer is applied to all of the generated tiles as well. We visualize these biases in figure 11. It's entirely possible for FeatSharp to remove the biases itself, but we found that having this learnable bias buffer consistently improves multi-view consistency, which we show in table 7 in the appendix.
3.5. Complexity
An important point about this method is that because of the tiling, it requires more evaluations of the base vision model to construct the high-resolution feature map. How- ever, due to the scaling properties of global self-attention, our proposed method always has better scaling properties than running the original model at higher resolution (assum- ing the model is capable of doing this in the first place). Specifically, let fx) be the relative cost of computing Feat- Sharp, and g (zx) the relative cost of running the base model
For the FeatSharp module, we leverage a single Atten- tion+SwiGLUU transformer block. In order to prevent the quadratic cost of global attention, we instead use 2D local attention (Ramachandran et al., 2019). We concatenate the
hhaps///gihuhh.cm/mmaaiiii 22/Faaaaa////aaaaa/usssse
5

---

FeatSharp: Your Vision Model Features, Sharper
on the hi-res input, with e being the number of tiles per dimension, and c being the cost of processing single
Shil view Consstency Tidellty by Model and Upowmpler
tile:
2.5
f(z) S c EE
i=1
(6)
g(z) = c c2²))²
= cr
[(x) S g(x) Vx > 1
Figure 6. Fidelity plot for different models and upsampling meth- ods. Higher values are better. We don't ton't show SAM 4x because of OOM issues training these models.
We show the empirical scaling cost in figure 14 in the ap- pendix, and prove equation 6 in appendix E.2. We also note that experiments for FeatSharp only use the global view, plus the final level of tiles, thus fx) simplifies to f(x) = e (1 + n²), however we prove the general case, as progressive upsampling may be beneficial in future work.
(DFN CLIP (Fang et al., 2023), SigLIP (Zhai et al., 2023)), Self-super (DINOv2-L-reg (Darcet et al., 2023)), Segmentation (SAM (Kirillov et al., 2023)), VLM (PaliGemma (Beyer et al., 2024)), and Agglomerative (RADIOv2.5-L (Ranzinger et al., 2024a)). Results are in figure 1. The original feature maps run the spectrum from ex- tremely noisy (SigL.P)) to very clean (R.DIOv..5LLL SAM), which allows us demonstrate the effectiveness of the ap- proach diverse set of models. Taking SAM for an example, the way in which it has thick edge outlines cannot be reproduced in the shape interior by FeatUp, primarily be- cause the bilateral upsampler is operating on the raw pixels, where the interior edge doesn't exist in the real image. For all of the featurizers, FeatSharp able to achieve more legi- ble representations. In particular it is more able to closely match the real hi-res features in the second column.
4. Upsampling Results
We consider upsampling to be important in cases where one is given fixed pretrained model, and the goal is to extract more information out of it, for a given image. We study our method in relation to UpatUp from core multi- view consistency standpoint in this section, from a semantic segmentation linear probe standpoint, and also for training a new RADIO-like model with hi-res teacher targets.
4.1. Fidelity
Multi-View Consistency Following (Ranzinger et al., 2024a), we use their definition of fidelity (equation 51) for multi-view consistency, where higher fidelity value means that the seaampled-rransfommdddddowwwww resentations are closer to the raw transformed predictions from the model.
4.3. Semantic Segmentation
Semantic segmentation has the potential to benefit from in- creased resolution, it allows for label contours to be more precise, and potentially for regions to be recovered that are otherwise too small. The first setting we evaluate on is we train both FeatUp and FeatSharp at 2 x and 4x upsampling, both using PHI-S. We resize the input size to be the featur- izer's native input resolution, which we call "Ix x Input Size", and we also consider "2x Input Size". where we double the input size, and feed directly to the featurizer in the case of "Baseline", or we allow the upsampler to have higher resolution guidance while keeping the featurizer input fixed at 1 x resolution. We show these results in figure 7. In most cases, both upsampling algorithms produce higher quality segmentations than the baseline, however, FeatUp ip is worse than the "Baseline 2x" " method for RADIOv2.5-L and VIT. In all cases, FeatSharp is superior to both FeatUp and also the baselines by significant margins. We even improve upon SOTA RADIO's published result of 51.47 with 2x upsam- pling combined with 2x input size, producing model that attains 53. mloU, a +1.66 mloU improvement. RADIO itself improves with the 2x input size, but not to the same degree as with FeatSharp, with FeatSharp being 57% faster.
rep-
f(X,)) MSE(Y, ,UY) MSE(X, Y)
with X being the warped predictions and Y the targets. This serves as a proxy measure for how well the upsam- pler is working, as arbitrarily warping and downsampling it results in representations closer to the real prediction at low resolution. We show these results in figure 6, where we observe that FeatSharp consistently achieves the highest fidelities, substantially so with the "cleaner" models such as DINOv2-L, 5ADIO and SAM-H.
4.2. Qualitative
We run this upsampling method on seven different foun- dation models coming from diverse domains such as su- pervised (VTT, (Dosovitskiy et al., 2021)), contrastive
6

---

FeatSharp: Your Vision Model Features, Sharper
We notice that 3x upsampling is generally slightly worse than 2x 4x for both upsamplers, but leave investi- gation into why future work. This figure also provides insight into the general ability of these foundation mod- els to operate at resolutions that deviate from their native resolution. The CLIP family models (DFN CLIP, SigLIP, PaliGemma) are unable to benefit from this increased reso- lution at all, or in the case of PaliGemma, degrade with it, while the fiss-class-dense models like DINOv2 and RADIO natively benefit from increased resolution. Surprisingly, even though ViT is solely trained as classification model, it also benefits from native resolution increases.
RADIOv2.5-L
Upsample
AP
Upsampler
Factor
Sm
Md
Lg
Baseline
1
51.38
28.73
56,56
73.72
Bilinear
51.61
28.43
56.98
74.14
SAPA
2
41.44
15.92
45.08
69.77
ReSFU
2
49.81
26.22
55.37
73.55
FeaUUp
2
46.71
21.77
52.01
72.25
FeatSharp
2
54.83
34.72
59.40
74.40
SBLLIP2-S040M--51
Baseline
52.66
30.31
57.94
74.31
Bilinear
2
52.69
30.19
57.84
74.16
SAPA
ReSFU
2
50.84
28.45
56.18
73.69
4.4. Object Detection
FeatUp
2
47.42
22.87
53.17
72.80
FeatSharp
55.93
36.85
61.00
74.62
We integrate our method, FeatUp-JBU, the baselines, as well as SAPA (Lu et al., 2022) and the preprint ReSFU (Zhou et al., 2025) into Detectron2 using the Edge² code- base, and probe on COCO 2017 (Lin et al., 2014). We use a [(frozen) featurizer] + [(frozen) upsampler] VYT- Det (Li et al., 2022) + DINO (Zhang et al., 2023)² (DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection) pipeline. We evaluate these methods on both RADIOv2.5-L (Heinrich et al., 2024) and the recently (Tschannen et al., 2025) models. We show the results in table 1, where FeatSharp clearly best able to improve object detection results over baseline and comparison methods, particularly for small objects, benefitting from the additional tile guidance. We also note that we were unable to use SAPA with SigL.P? due to a CUDA configuration error in their backprop kernel.
Table I. COCO 2017 object detection results using Detectron2 and various upsampling methods for both RADIOv2.5-L. and SigLII2-S0400M. 'SAPA was unable to process this model's s input size/dimension, producing CUDA configuration error.
we consider two techniques which are popular in the VLM literature: Tiling (Liu et al., 2024), and S2 (Shi et al., 2024a). Both of these rely on tiling, but S2 also considers the low- res version. Because we need the feature space to remain the same as the low-res partition of RADIO, we opt to upsample the low-res feature map, and then interpolate the upsampled-low-rre against the tiled version, using y = B low-res + (1 5)- - high-res. We set / = 0.5 as it's unclear what an optimal balance might be, and it's expensive to search this space. As final baseline, we include FeatUp's JBU variant, as the implicit version would be prohibitive to use within a training loop".
proposed SigLIP2-S0440ME
4.5. Agglomerative Models
We build upon RADIOV2.5-L. (Heinrich et al., 2024) as it learns directly from the spatial features of teacher models. In particular, we consider whether we can improve upon their multi-resolution training strategy by using FeatSharp to convert the low-res teachers into hi-res teachers. We convert the teachers in the bottom left quadrant "Low-Res Teacher / High-Res Student" in their Figure 6 into "High-Res Teacher / High-Res Student" by using the upsampler. We consider a few different comparative baselines in order to prove the efficacy of the technique. For our baseline, we make one change to the recipe in (Heinrich et al., 2024), which is to bilinearly upsample the teacher to match the student, as opposed downsampling the student. We stress that this produces strong baseline, it scores even better than RADIO on average. The reason make this change is so that we're across the board comparing upsampling methods, with bilinear being the simplest technique. Then,
In figure 8 we qualitatively visualize the DFN CLIP adap- tor features learned by the radio model. We can see that each upsampling method has substantial impact on the resulting feature maps. The baseline method exhibits strong high-firquency artifacting starting at 768px. This is likely when RADIO "mode switches" to high- is something that (Heinrich et al., 2024) addressed for the backbone features, but apparently still exhibit for the adap- tor features. We observe that Tiling and S2 exhibit not only high-freyuency noise patterns like the baseline, but also obvious grid patterns, arising from the use of tiles. More troublesome,
which
how the student learned to mimic representation switches within tiles for both Tiling and S2, where the mulch in one cell gets a different feature repre- sentation (thus color) than another, based on whether any of the dog is present in the tile view. FeatUp appears to mode switch starting at 768px into smooth, but low-detail feature space. FeatSharp remains smooth and highly de-
http://dgenz.gitte wwwpir.gtthhhhho.. hatphbuhhooeegggpp///aaaapiiiiie html 3 Noo to be confused with the DINO/DINOv2 foundation mod-
els.
*-5 minutes per image
7

---

FeatSharp: Your Vision Model Features, Sharper
DFN CLIP
DINOv2-L
PaliGemma
52
Baseline Inpt-1x
Baseline Inpt-2x
45
45.0
mloU
FeatSharp Inpt-1x
50
FeatSharp Inpt-2x
42.5
FeatUp Inpt-1x
40
48
FealUp Inpl-2x
2x
4x
2x
3x
RADIOV2.5LL
SigLIP
ViT
30
45
miou
28
40
51
2x
3x
4x 2x
3x
4x 2x
3x
4x
Upsample Amount
Upsample Amount
Upsample Amount
Figure 7. ADE20k (Zhou et al., 2017) Semantic segmentation results for different featurizers and upsamplers. We also vary the input size between Inpt-1x and Inpt-2x the featurizer's 's native resolution. x Resolutions: DFN CLIP= = 378px, DINOv2-L 448px, PaliGemma 448px, RADIOv2.5-L= 512px. SigLIP 378px, ViT = 224px. The dark line represents the mean of runs, with shaded areas showing the standard deviation. Because the x-axis is the upsample amount, the baselines should technically be single points on "Ix" x-coord, but instead draw line to make easier see the change in the upsamplers across the upsample amounts. E.g. for "RADIO, Baseline Inpt-2x", , we can see that it's better than FeatUp 2x upsampling, but worse than FeatSharp x upsampling.
tailed resolution increases, however, visually, it's still possible that the features are mode switching. We show another comparison in appendix F with the SigLIP adaptor head.
version of RADIO with FeatSharp teachers generally does better than RADIOv2.5-L (Heinrich et al., 2024), which is the current state of the art, where we improve over it on everything except for the VILA task. We report all of the raw per-task benchmark scores in tables 3, 4, 5 and the appendix.
Along with improvements in the adaptors, we also study the effects on the backbone features for the RADIO model. Following (Maninis et al., 2019; Lu et al., 2024) report the MTL Gain (Am) across suite of tasks. Unlike the prior works, instead of leveraging single-task baseline, we opt to report the change relative to the baseline training run.
5. Conclusion
We have presented novel feature upsampling technique named FeatSharp that achieves higher multi-view fidelity than the current best method, FeatUp. We achieve this by joining FeatUp's JBU upsampler with a mosaic of tiles, and then process with a single local attention block. We demonstrate its effectiveness ADE20K semantic segmen- tation linear probing, where the use of FeatSharp improves over both baseline and FeatUp, even with the strongest segmenter, RADIO, which itself can handle hi-res inputs robustly. We also demonstrate our effectiveness in object detection with frozen backhone and upsampler, and see AP benefits in particular for small objects, but also medium and large. We then demonstrate the effectiveness of FeatSharp by employing it directly within RADIO training, enabling hi-res distillation targets for low-res-only teacher models. In doing so, our SanSSharrp-ADIOO largely improves on dense vision task benchmarks, and yields an overall improvement over our reproduction baseline, which itself improves over RADIOv2.5-L. the current state of the art. We believe this work be useful both drop-in extension of existing vision systems which rely on pretrained vision encoders, as well as the newly trained FeatSharp- RADIO model with hi-res teachers, which can emulate these same models. Ow-
Let
8m 100 (-1) M, -M
MB,
T
Am
pm
where M. is the metric for the current model on task i, and MB. is the metric for the bascline model. le is O when higher task values are better, and when lower is better.
We show the MTL Gain results in table 2. Given that the results are relative our baseline run, S2 and FeatSharp are the only two methods to improve, however, only FeatSharp was categorically better, leading to +0.39% improvement across all benchmarks the only two that incorporate both low-res and hi-res fea- tures, with $2 perhaps being considered a baseline to Feat- Sharp, so their improvements suggest that this extra detail indeed useful for RADIO training. We also see that our
average. These two methods are
8

---

FeatSharp: Your Vision Model Features, Sharper
Baseline DFN CLIP
256px
512px
768px
1024px
p048px
Tile DFN CLIP
S2 DFN CLIP
FeatUp DFN CLIP
FeatSharp DFN CLIP
Figure 8. Visualization of our trained RADIO's DFN CLIP adaptor when the high-res partition used various teacher upsample schemes.
Upsampler
Classification
Dense
Prohe 3D
Retrieval
Pascal Context
NYUDv2
VII.A
Am %
RADIOv2.5-I.
-0.47
-0.09
-1.05
-0.45
0.62
-2.26
2.24
-0.21
Baseline
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Tile
-0.03
0.30
-0.08
-0.23
-0.02
1.33
-3.17
-0.27
$2
-0.05
0.15
-0.89
FeatUp
-0.07
0.14
0.23
-0.07
0.14
0.32
-1.58
-0.13
FeatSharp 0.06
0.16
0.83
0.13
0.17
0.93
0.43
0.39
Table Relative changes (in %) on suite of aggregated benchmarks, with each column reporting 5,, and averaged into 4m %. All relative changes against our baseline run. Raw metries are in section A.1. NOTE: The upsamplers only applied to the DFN CLIP and SigLIP teachers during RADIO training. Metrics are collected from trained RADIO without upsampling methods.
ing to FeatSharp- RADIO's emulation abilities, it allows to estimate these teacher models at arbitrary resolutions, not just integer upsampling factors as restricted Feat- Sharp/FeatUp's core training algorithm. Further, combining RADIO's "ViTDet" (Li al., 2022) mode with these hi-res teacher emulations allows to achieve hi-res feature maps without fully paying the quadratic penalty in number of tokens as required by standard ViTs.
Impact Statement
This paper presents work whose goal is to advance the field of computer vision. By virtue of being a lightweight addi- tion to existing vision models, the work aims to open up doors for higher-resolution perception tasks (e.g. segmenta- tion, depth perception, distillation, etc.) while retaining the original model representations. As such, the ethical impacts are constrained to those of the model being upsampled. The FeatSharp training code will be released to the community.

---

FeatSharp: Your Vision Model Features, Sharper
References
Nvim: Open frontier-class multimodal Ilms, 2024a. URL https:// attess:/aaxxii.oor//ab////// 114022
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y. Lenc, K. Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E.. Cabi, S., Han, T., Gong, Z.,, Samangooei, S., Monteiro, M., Menick, J. Borgeaud, S., Brock, A., Nematzadeh, A., Shar- ifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. Flamingo: vi- sual language model for few-shot learning, 2022. URL https: // aaaii.iiv.vv.... org/ hbtg//bs///ee 2204. 14198.
Dai, W. Lee, N., Wang, B., Yang. Z., Liu, Z., Barker, J., Rintamaki, T., Shoeybi, M., Catanzaro, B., and Ping, W. Nvlm: Open frontier-class multimodal Ilms, 2024b. URL
https: httpp///acaxv...rrr///////0.....
Darcet, T., Oquab, M., Mairal, J., and Bojanowski, P. Vision transformers need registers, 2023.
Dehghani, M., Mustafa, B., Djolonga, J., Heek, J., Minderer,
Awais, M., Naseer, M., Khan, S., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., and Khan, F. S. Founda- tional models defining a new era in vision: A survey and outlook, 2023.
M., Caron, M., Steiner, A. P., Puigcerver, J., Geirhos, R., Alabdulmohsin, L,, Oliver, A., Padlewski, P., Grit- senko, A. A., Lucie, M., and Houlsby, N. Patch pack: Navit, a vision transformer for any aspect ratio and resolution. In Thirly-seventh Conference on Neural Information Processing Systems, 2023. URL https: / //openreveww net/ forum? VetpiiddvVppFGmmmmmmmm
Babenko, A., Slesarev, A., Chigorin, A., and Lempitsky, V. Neural codes for image retrieval. In Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (eds.). Computer Vi- ECCV 2014, pp. 584-599, Cham, 2014. Springer International Publishing. ISBN
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
Beyer, L., Izmailov, P. Kolesnikov, A., Caron, M., Kornblith, S., Zhai, X., Minderer, M., Tschannen, M., Alabdulmohsin, I, and Pavetic, F. Flex- iVIT: One Model for All Patch Sizes In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14496-14506, Los Alamitos, CA, USA, June 2023. IEEE Computer URL :tttp:://oot.ieeeccoomooo/o www.ecooppteresocceeee...oo org/ 10. 11w1119///00 (TW.PPPP2222922222222
Drozdova, M., Kinakh, V., Belousov, Y., Lastufka, E., and Voloshynovskiy, S. Semi-supervised fine-tuning of vi- sion foundation models with content-style decomposi- tion. In NeurIPS 2024 Workshop Fine-Tuning in Modern Machine Learning: Principles and Scalability, 2024. URL https: /oppnrevieeeeeeeeeeeeeeeeeeeeeee net forum? id=zH1 jWq2hqH.
Society. doi: 10.11099CVPR5222222003.0
https:
Beyer, L., Steiner, A., Pinto, A. S., Kolesnikov, A., Wang, X., Salz, D.,. Neumann, M., Alabdulmohsin, I., Tschan- nen, M., Bugliarello, E., Unterthiner, T. Keysers, D., Koppula, S., Liu, F., Gryener, A., Gritsenko, A., Houlsby, N., Kumar, M. Rong, K., Eisenschlos, J., Kabra, R., Bauer, M.. Bošnjak, M.. Chen, X., Minderer, M., Voigt- laender, P., Bica, I., Balazevic, I., Puigcerver, J., Papalam- pidi, P., Henaff, O., Xiong, X., Soricut, R., Harmsen, J., and Zhai, X. Paligemma: A versatile 3b vlm for trans- fer, 2024. URL /tttp::/arxiv. www.g///////ae/eeee/eeeee /407. 07726.
Dumoulin, V. and Visin, F. A guide to convolution arithmetic for deep learning. ArXiv, abs/1603.07285. 2016. URL https: / //ppi..... wwwanti Coo.ooopp...62084...... 6662846.
Banani, M., Raj, A., Maninis, K.-K., Kar, A., Li, Y., Rubinstein, M., Sun, D., Guibas, L., Johnson, J., and Jam- pani, V. Probing the 3D Awareness of Visual Foundation Models. In CVPR, 2024.
et al., O. Gpt-4 technical report, 2024. URL https: //arxiv. ///ar//w.org/aa///333.0.00
Chen, Z., Wang. W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, 7., Ma, J., Wang, J., Dong, X., Yan, H., Guo, H., He, C., Shi, B., Jin, Z., Xu, C., and Wang, W. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67, 12 2024. doi: 10. 1422-024-23111115
Fang, A., Jose, A. M., Jain, A., Schmidt, L.,, Toshev, A., and Shankar, V. Data filtering networks, 2023.
Fu, S., Hamilton, M., Brandt, L. E.,, Feldmann, A., Zhang, Z.,, and Freeman, W. T. Featup: A model-agnostic framework for features at any resolution. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttp:://////eeeeeee//e net/ forum?? id=GkJiNn2@DF.
Dai, W. Lee, N.. Wang, B., Yang, Z., Liu, Z., Barker, J., Rintamaki, T., Shoeybi, M., Catanzaro, B., and Ping, W.
10

---

FeatSharp: Your Vision Model Features, Sharper
Gretton, A., Borgwardt, K. M., Rasch, M. Schölkopf, B., and Smola, A. A kernel two-sample test. Jour- nal of Machine Learning Research, 13(25)723-773, 2012. URL http:// /oler//mmr.... wtw.ll.org/paper////.. gretton1 2a, 2.hmml.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, and world knowledge, January 2024. URL 2024-01-300 11aaa-0-llaaaaxa..aa
io/blog/
Guo, P., Zhao, Z., Gao, J., Wu, C., He, T., Zhang, Z., Xiao, T., and Zhang. W. Videosam: Open- Open-world video seg- mentation, 2024, URL
Lu. H., Liu, w., Ye, Z.. Fu, H., Liu, Y., and Cao. Z. SAPA: Similarity-awand point affiliation for feature upsampling. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https: / openreview .ne/// forum? fiimmidd=Fnnn 381edL.
http://axxvvvorooee abs/
2410.00 08781.
Heinrich, G., Ranzinger, M., Hongxu, Yin, Lu, Y., Kautz, J., Tao, A., Catanzaro, B., and Molchanov, P. Radio ampli- fied: Improved baselines for agglomerative vision foun- dation models, 2024. URL https: atttps:/arxiv..or //t.xii..ggg////////... abs/2412. 07679.
Lu, Y., Cao. S., and Wang, Y.-.. Swiss army knife: Syn- ergizing biases in knowledge from vision foundation models for multi-task learning, 2024. URL https:
/ Maninis, K.-K., Radosavovic, I., and Kokkinos, I.
14433
Kim, D., Angelova, A., and Kuo, W. Region-aware pre- training for open-vocabulary object detection with vision transformers, 2023.
Attentive single-tasking of multiple tasks. 2019
IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1851-1860, 2019. URL bttps://apls http://aappsm.aaiiiiioom wemanticscholar.coo CorpusTD:1211 208399
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L.,, Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., Dollár, P., and Girshick, R. Segment anything. 2023.
Nathan Silberman, Derek Hoiem, P. and Fergus, In- door segmentation and support inference from rghd im- ages. In ECCV, 2012.
Kopf, J., Cohen, M. F., Lischinski, D., and Uyttendaele, M. Joint bilateral upsampling. ACM Trans. Graph., 26(3):96 es, July 2007. ISSN 0730-0301. doi: 10. 1145/1276337.112769977 URL https: http:/oi.orrrr//// org/ 10. 10.1145/1276377..22
Noh, H., Hong, S., and Han, B. Learning Deconvolution Network for Semantic Segmentation In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1520-1528, Los Alamitos, CA, USA, December 2015. IEEE Computer Society. doi: 10.1 10.1109/CC2.2015.1788 URL. https:///oit www.eeemptteresocce.e....... org/10. http///..11//////// 2015.178.
Krizhevsky, A., Sutskever, I., and Hinton, E. Ima- genet classification with deep convolutional neural networks. In Pereira, F., Burges, C., Bottou, L.,, and Weinberger, K. (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL wtt.es//apperr/2222/Ei
Odena, A., Dumoulin, V., and Olah, C. Decon- volution and checkerboard artifacts. Distill, 2016. URL
ce/paper sc/paapr_filleseeeeeeeeeeeeessee c3 029962223b94667664433688888 pdf.
Paper.
deconv-checkent www.uu--ccccerbbard/...
Li, Y., Mao, H., Girshick, R., and He, K. Exploring plain vision transformer backbones for object detection, 2022.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F, El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Y. Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. Dinov2: Learning robust visual features without supervision, 2023.
Lin, J., Yin, H., Ping, W., Lu. Y., Molchanov, P., Tao, A., Mao, H., Kautz, J., Shoeybi, M., and Han, S. Vila: On pre-trat pre-training for visual language models, 2023.
Lin, T.-Y., Maire, M., Belongie. S., Hays, J., Perona, P., Ra- manan, D., Doll'ar, P., and Zitnick, C. L. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Plested, and Gedeon, Deep transfer learning for image classification: survey, 2022.
org /tabb//2255 09904.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning, 2023.

---

FeatSharp: Your Vision Model Features, Sharper
transferable visual models from natural language su- pervision. In Meila, M. and Zhang, T. (eds.), Pro- ceedings of the 38th International Conference Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 18-24 Jul 2021. URL httptt::://proceednngg mww..prees. v139/radford21e. html.
Shi, W., Caballero, J., Theis, L.., Huszar, F., Aitken, A., Ledig, C., and Wang. Z. Is the deconvolution layer the same convolutional layer?, 2016. URL https: 609. 07009.
Tschannen, M., Gritsenko, A., Wang. X., Naeem, M. F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., Hénaff, O., Harmsen, J., Steiner, A., and Zhai, X. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localiza- tion, and dense features, 2025, URL https: /ttpss:///rxvv../ org abs / 2202 14786.
Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Lev- skaya, A., and Shlens, Stand-alone self-attention in vision models. Curran Associates Inc., Red Hook, NY, USA, 2019.
Ranzinger, M. Barker, J., Heinrich, G., Molchanov, P., Catanzaro, B., and Tao, A. Phi-s: Distribution balanc- ing for label-free multi-teacher distillation, 2024a. URL https:/ / arxiv. org/ abs 2210.410...0 01680.
Vaswani, A., Shazeer, N.. Parmar, N., Uszkoreit. J., Jones, L.,, Gomez, A. N., Kaiser, t.,, and Polosukhin, I. Atten- tion is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Ranzinger, M., Heinrich, G., Kautz, J., and Molchanov, P. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12490-12500, June 2024b.
Wang, J., Chen, K., Xu, R., Liu, Z., Loy. C. C., and Lin, D. Carafe: Content-aware reassembly of features. 2019 IEEE/CVE International Confer- ence on Computer Vision (ICCV), pp. 3007-3016. 2019. URL https: btttps://ap..smmettt org httg/CorppsDD:11612000 wtt.apisststtsscsoll..a..
Rudin, L. I., Osher, S., and Fatemi, E. Nonlinear total varia- tion based noise removal algorithms. Physica D: Nonlin- ear Phenomena, 60( 60(1):259-266, 1992. ISSN 0167-2789. doi: http:://ddaa//////00066661177799
Wang. P., Bai. S., Tan, S.,, Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision- -anguan model's perception of the world at any resolution, 2024. URL https: htpps://arxivv /awp/v.ivvvvvv////////.///aaaavv
URL htttp::/www..cceencee cccaanee/atticcee///////22229999
com/
Sariyildiz, M. B., Weinzaepfel, P., Lucas, T., Larlus, D., and Kalantidis, Y. Unic: Universal classification models via multi-teacher distillation, 2024. URL https:// arxiv. www..org/aao abs /2088 05088.
orwoorgg//aooo absg///b///// /40/..2499... 12191.
Yang. J., Luo, K. Z., Li, J., Weinberger, K. Q., Tian, Y., and Wang. Y. Denoising vision transformers, 2024a.
Shang, J., Schmeckpeper. K., May, B. B., Minniti, M. V., Kelestemur, T., Watkins, D., and Herlant, L.. Theia: Distilling diverse vision foundation models for robot learning. In 8th Annual Conference on Robot Learning, 2024. URL https::////.. http:::///ppeeee/eeeeeee net. forum? id=y12HvlwUch.
Yang. Y., Jiang, P.-T., Hou, Q. Zhang, H., Chen, J., and Li, B. Multi-task dense prediction via mixture of low-rank experts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024b.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training, arXiv preprint arXiv:2303.1533 2023.
Shi, B., Wu, Z. Mao, M., Wang, X., and Darrell, T. When do we not need larger vision models? In Com- puter Vision - ECCV 2024: 18th European Confer- ence, Milan, Italy, September 29-October 4, 2024, Pro- ceedings, Part VIII, pp. 444-462, Berlin, Heidelberg, 2024a. Springer Verlag. ISBN 978-3-031-73241-1 doi: 10.1007/978-334311--5922--- org/10.1007/90
Zhang, C. and Liu, J. Content based deep learning image retrieval: A survey. In Proceedings of the 2023 9th International Conference on Communication and Information Processing, ICCIP '23, pp- 158-163,, New York, NY, USA, 2024. Association for Comput- ing Machinery. ISBN 9798400708909, doi: 10.1145/ 3638884.3638988, URL https: //ts:://doo..ee org/10. 1 02236688883363890888888 3638908.
URL https::/tt//
Shi, M. Liu, F., Wang, S., Liao, S., Radhakrishnan, S., Huang, D.-A., Yin, H., Sapra, K., Yacoob, Y., Shi, H., Catanzaro, B., Tao, A., Kautz, J. Yu, Z., and Liu, G. Eagle: Exploring the design space for mul- timodal ilms with mixture of encoders. 2024b. URL https: // //trxivvvvvv//////////eee//aavvl wwwpxiv.orr//aeeeooeo abs///b//20 220/ 15998.
Zhang, H., Li, F., Liu, S., Zhang, L.., Su, H., Zhu, J., Ni, L., and Shum, H.-Y. DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In The
12

---

FeatSharp: Your Vision Model Features, Sharper
Eleventh International Conference Learning Represen- fations, 2023. URL httpt::ttt//ptttt
/tt.ttt////peeeeeeiieee httpss////peeeeeeeeeeeee net./ Forum?id= 3ddddmmwyyS....
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. Scene parsing through ade20k dataset. In 2017 IEEE Conference on Computer Vision and Pottern Recognition (CVPR), pp. 5122-5130, 2017. doi: 10.1109/ CVPR.2017.544.
Zhou, M., Wang. H., Zheng, Y., and Meng, D. A freshed similarity-based upsampler for direct high-ratio feature upsampling, 2025. URL https: / arxiv. org/abs/2407 02283.
13

---

FeatSharp: Your Vision Model Features, Sharper
A. RADIO Results
A.1. Benchmark Results
In this section, we provide detailed benchmark results used to compute the MTL aggregate metrics table 2. We show these results in tables 3, 4, 5, and 6.
Classification
Zero Shot Retrieval
Upsampler
ImageNet-Il 1k
COCO
Flickr30k
Zero Shot
kNN
Text2Im
Im2Text
Text2Im
Im2Text
RADIOv2.5-L
81.01
84.68
51.65
69.06
77.52
90.80
Baseline
81.47
85.00
52.25
68.68
78.64
90.60
Tile
81.41
85.01
51.90
68.30
78.46
91.10
S2
81.44
84.95
51.94
67.98
78.34
90.80
FeatUp
81.39
84.96
51,93
68.40
78.26
91.70
FeatSharp 81.56
85.01
52.13
68.80
78.50
91.30
Table 3. Classification and Zero Shot Retrieval Metries, All zero shot methods use the DFN CLIP Text encoder, paired with RADIO's
respective learned adaptor.
Dense
Probe3d
Upsampler
ADE20k
VOC
SAM COCO
Depth
Surface Normals
Correspondence
SPair71k
RADIOv2.5-L
51.47*
85.49*
75.06
84.69
60.06
58.46
54.36
Baseline
51.58
85.08
75.46
85.03
61.42
59.27
54.49
Tile
51.62
85.55
75.67
85.14 14
60.85
59.65
54,41
S2
51.56
85.28
75.66
85.11
60.49
59.84
54.68
FeatUp 51.67
85.20
74.54
85.39
61.20
59.63
54.63
FeatSharp 51.75
85.13
75.54
85.48
60.76
59.55
56.33
Table 4. Dense and Probe3D (EI Banani al., 2024) metries. *We report numbers for evaluation at 5 512px, which found in Table A5 in RADIOv2.5 (Heinrich et al., 2024).
Pascal Context
NYUDv2 Depth rmsee..
Upsampler
SemSeg f
Parsing mloU 1 Saliency maxF
Surface Normals J.
SemSeg mioU
Surface Normals J.
RADIOv2.5LL
82.87
74.32
81.65
16.15
61.42
0.458
18.57
Baseline
62.88
75.02
80.55
16.49
62.64
0.448
18.09
Tile
03.07
75.28
80.56
16.60
62.91
0.437
17.90
$2
83.09
75.45
80.63
16.56
62.64
0.436
17.86
FeatUp 8
75.21
80.68
16.51
62.74
0.449
17.93
FeatShurp 83.17
75.28
80.64
16.51
62.60
0.439
17.95
Table Pascal Context and NYUDv2 multitask learning metrics. Following the setup of MLoRE (Yang et al., 2024b) and RADIOv2.5 (Heinrich al., 2024) with convolutional probe. NOTE: We're only using their harness with probe, and not using their architecture.
A.2. Additional Qualitative Visualizations
In figure show more PCA feature visualizations coming from our trained RADIO models. We that RADIO learned to mimic how tiling lacks global context, the background-only tiles use a different feature space than those with background+content.
A.3. Difference Visualization
In figure 10, we show the difference heatmaps between FeatSharp/FeatUpp and Bilinear upsampling. For DFN CLIP and SigLIP, we actually see that lot of the differences are with high frequency noise. More intuitively, for the cleaner RADIO and SAM models, the differences are largely concentrated at the edges. Because the PCA projection down to 3D can sometimes distract from the true differences between representations (e.g. color flipping), these difference maps help show where the information is truly different between methods.
14

---

FeatSharp: Your Vision Model Features, Sharper
A12D No Mask
ChartQA
DocVQA
GQA
InfoVQA
MME
MMMU
OCR Bench
POPE
SRED
Te
Upsampler
Val
Overall
Accuracy
Val
Perception
Val
Accuracy
F1
All
Val
Accuracy
Accuracy
RADIOv2.5LL
79.2
$6.4
49.2
63.4
29.8
1592.4
43.3
441
87.6
69.27
Baseline
78.04
57.32
47.12
63.41
28.78
1568.11
40.00
422
87.51
69.08
65.33
Tile
75.71
54.32
42.44
63.60
26.80
1541.61
40.33
400
86.63
68.62
63.78
S2
77.07
55.28
44.89
63.73
28.75
1549.50
42.33
405
87.14
68.96
64.86
FeaUUp
78.40
55.56
45.31
63.60
26.98
1563.57
40.33
407
86.83
68.57
65.05
FeatShurp
79.15
57.56
46.39
63.75
28.25
1564.41
42.22
416
88.06
68.77
66.41
Table VILA metrics, using the same setup from [(Heinrich et al., 2024). Table 9].
SigLIP
512ok
768px
2048px
Tile- SigLIP
52 SigLIP
SipLIP
FearSharp SiPLIP
Figure 9. Visualization of RADIO's Sigl. IIP adaptor, using different teacher upsampling techniques.
B. Architecture Ablations
B.1. De-bias Module
Adding the de-bias module yields a positive improvement fidelity across all featurizers studied. We show the changes in fidelity metries for all featurizers in table 7. We also demonstrate that this module helps for both FeatSharp and FeatUp, occurs prior upsampling, and is thus generally applicable. In figure 11, visualize the learned biases, which are unique to each featurizer, but also how these biases sometimes be directly visible in the output features of these models. Most obvious is SAM, which has windowing artifacts stemming from their use of windowed attention.
B.2. FeatSharp Architecture
Input Feature Selection Based figures 2 and 3, there are important degrees of freedom in the design of the system. We demonstrate in table 7 that including the de-bias module always improves the distribution matching fidelity. Here, we look at some of the other design choices:
Should we use bilinear upsampling, or FeatUp, for the low-res upsampler? (or both)
15

---

FeatSharp: Your Vision Model Features, Sharper
Image
FeatSharp Diff
FeatSharp 4x
Bilinear 4x
FeatUp 4x
FeatUp Diff
DFN CLPP
SigLIP
RADIO
SAM
Figure 10. Visualization of the differences between the FeatSharp or FeatUp algorithm, and bilinear upsampling.
Should bilinear, tiling, or FeatUp be the residual pathway to the output?
Should we use all three upsampling methods?
We show the results of this ablation in table 8 for both noisiest featurizer (SigLIP), and our cleanest (RADIO) as sanity check that we aren't I overfitting to particular featurizer. We also note that we re relatively agnostic to the specific base feature upsampler, allowing us to use other methods, such as ReSFU (Zhou et al., 2025), as future work.
We also visualize the resulting feature maps of the different input configurations in figure 12, as it's hard to get feel for what this multi-view fidelity is telling us. It's clear both in the metrics (table 8) and the visualization that just using one of the three different feature maps largely retains the biases of those views (e.g. the bilinear result is roughly regular bilinear upsampling, the FeatUp input looks like vanilla FeatUp, etc.). We also the profound impact the resulting maps based on which input feature map is the residual pathway. For the single input case, we can observe how the tile-only input results in a distribution shift. apparent because the color space has largely shifted. Once we look at 2+ inputs, the color spaces become consistent, Even though the bilinear-first configurations always have the highest fidelity, they also are clearly the blurriest. This is perhaps not surprising given that FeatUp's 's JBU upsampler has a strong edge prior, so incorporating it into FeatSharp will also hone in on edge boundaries. Also, regardless of 2+ input configuration, we can see that FeatSharp able refine the text, clearly leveraging the tile features. The similarity is very close the tile-only input in that region. We do notice that using "Bilinear + other(s)" yields the highest fidelities, but also that the resulting feature maps are relatively blurry.
In order to not make an entire argument to prefer the use of FeatUp's JBU as the low-res upsampler due to the prettiness of the PCA features, we also consider alternative measures of the produced features. The Total Variation (TV) loss gives us a sense of how much "noise" is present in the produced features, simply based on accumulating the differences between neighbors. On its own, this doesn't tell us much, but in conjunction with the multi-view consiscnss fidelity, and when that
16

---

FeatSharp: Your Vision Model Features, Sharper
Model DEN CLIP
FeatSharp 2x
FeatSharp
FealUp 2x
FeatUp
0.020
0.022
0.033
0.025
DINOv2-L
0.110
0.164
PaliGemma
0.023
RADIOv2.5-L
0.173
0.144
SAM-H
0.076
SigLIP
0.014
0.017
0.033
VIT
0.014
0.009
0.096
Table 7. The delta change in multi-view consistency fidelity when applying the learned de-bias buffer. Positive values mean that the fidelity has improved, which is true for every model and upsampler tested.
SigLIP TV Loss +
RADIO TV Loss
Arch
Fidelity 1
CRF Loss +
Fidelity
CRF Loss
Single Input
Bilinear
1,466
0.135
0.137
4.599
0.061
0.071
FeatUp
1.440
0.020
0.051
3.870
0.025
0.047
Tiles
1.261
0.897
0.237
2.713
0.357
0.094
Two Inputs
Bilinear + Tiles
1.470
0.136
0.135
4.694
0.073
0.074
FeatUp + Tiles
1.460
0.072
0.062
4.173
0.076
0.057
Tiles + Bilinear
1.337
0.812
0.241
3.202
0.336
0.098
Tiles + FeatUp
1.323
0.821
0.228
3.157
0.337
0.094
Three Inputs
Bilinear First
1,469
0.138
0.137
4.682
0,072
0.073
FeatUp First
1.473
0.063
0.065
4.202
0.064
0.057
Tiles First
1.339
0.800
0.238
3.238
0.334
0.098
Table Ablation over different FeatSharp-2x configurations. Single Input means that only supply the respective buffer to the FeatSharp module. For "Two Inputs", we compare different low-res upsamplers in conjunction with tiling, and also the residual pathway, where the first value indicates the residual path. The FeatSharp module must integrate the information Inputs" is similar to Two, except that we only care about which buffer is the residual path, owing to the fact that there's no intrinsic order preference the weights for the secondary buffer(s).
other value into residual. "Three
Loss" stands for Total Variation Loss (Rudin et al., 1992). CRF Conditional Random Field, and is essentially measuring how similar the semantics of two nearby RGB pixel patches are based on how similar the RGB values are. TV and CRF losses were not included in the gradient during training.
fidelity is roughly equal, it might be reasonable assume that less variation is better. We in table 8 that the use of FeatUp does indeed reduce this for SigLIP, but has the opposite effect on RADIO. The other prior that we consider is the CRF loss, which approximately translates to the idea that nearby regions that have similar RGB color should probably also have similar semantics. The JBU also does a good job of reducing this for our noisiest SigLIP model, as well as for RADIO. stands reason that spurious model noise is penalized by CRF because breaks visual/semantic correspondence. For both TV and CRF losses, we capture the metrics, but they do not participate in the gradient. So, purely measuring the latent behaviors.
An alternative argument, which doesn't 't require hand waving about whether less variance is good thing, or if spatio-semantic similarity is necessarily good, we turn to Maximum Mean Discrepancy (MMD, (Gretton et al., 2012)) which is precisely defined as a way to test whether two sets of observations X == {I1, ..., Im } and Y == {81, ... }r are sampled from the same distribution. It has the clear advantage in our setup in that m doesn't have to be equal to n, or rather, we can have a different number samples in X than that in Y. Because we're re upsampling, if we let low-res distribution be X, then the high-res distribution can be Y. and then umm with being the upsampling factor. Given radial basis function kernel (RBF)
k(æ, v) -71xx==-1
(10)
then we have
17

---

FeatSharp: Your Vision Model Features, Sharper
Learned Be-Blas
Image
Image
Image Tiled
Image
S
Palieenin
SAM
VIT
DFN CLIP
DINOv2-L
DINOv2-g
RADIOv2.5.L
Figure Visualization the learned position biases for different models. All models bias signature, however some have very noticeable artifacts, which we visualize for SigLIP, PaliGemma, and SAM, where it's possible to see the artifacts in multiple different images and scales. We display the biases of the less apparent models in the bottom row.
2 mn
MMD [X, v] m(m
k(),,,z) + n(n 1)
k Uÿ)
k(x;, U5)
i.jEm 1pfa
i,j 2pta
(Grettoon et al., 2012). Eq 3)
we select y = med([1z; - =,,13) i * j. We then collect results for Fidelity, TV Loss, CRF Loss, and MMD, for 4x upsampling, and display the results in table We collect these results for SigLIP, DFN CLIP, and RADIO. clear that FeatSharp achieves the highest upsampling fidelities across the board. FeatUp produces the lowest TV and CRF losses. achieving the lowest TV loss is intuitive given how smooth it tends to make object interiors, in the pea visualizations. We can see that the lower TV and CRF losses extends to FeatSharp when we apply JBU upsampling, it achieves lower values than using bilinear upsampling for the residual pathway. The "JBU Tiles" FeatSharp variant also does better on MMD versus "Bilinear + Tiles" across the board. It's curious that JBU alone has the worst MMD (probably due to over-smoothing). but the best when incorporated into FeatSharp (probably owing to smoothing out the noise). We can also see that generally either "X + Tiles" FeatSharp method produces similar fidelities, aside from RADIO, where bilinear actually does do a bit better. Most likely, this because RADIO features are themselves already fairly clean, and at some point, the structural priors of JBU actually hurt, because they's re eliminating some of the raw signal that bilinear upsampling preserves. In this case, the model always has access the low-res signal with bilinear upsampling because integer multiple upsampling factor, and our local attention window size is larger than this multiple. Given the totality of evidence, we choose to select "JBU + Tiles" as the default upsampling mechanism, as it's either the best, or nearly so, across the board, and particularly, it does better with the vision models that are not able to natively change their resolution very well. We also note that newer methods, as they emerge, could serve as better core upsampler modules than bilinear/JBU, and can be trivially swapped in.
an
18

---

FeatSharp: Your Vision Model Features, Sharper
SigLIP
TURKIYS CUMPURIVET
BL + Tiles
CUNNURITED
FeatUp Tiles
Bilinear
Bilinear
Single Input
Three Inputs
Two Inputs
FeatUp
Tiles BL
FeatUp
Tiles + FeatUp
Tiles
Tiles
RADIO
BL + Tiles
CUMMURIFETI POSTALABI
CUMMURIFETI
POSTALABI
FeatUp + Tiles
Billnear
Bilinear
Single Input
Three Inputs
Two Inputs
FeatUp
Tiles + BL
FeatUp
Tiles + FeatUp
Tiles
Tiles
Figure 12, Feature visualizations of different input configurations for 2x upsampling.
19

---

FeatSharp: Your Vision Model Features, Sharper
Featurizer (4x Upsample, Long Recipe)
Upsampler
SigLIP
DFN
CLIP
RADIO
Fidelity
TV
CRF 1
MMD 1
Fidelity
TV 4
CRF 4
MMD +
Fidelity t
TV L
CRF
MMD
Bilinear
1.348
0.048
0.129
0.016
1.284
0.051
0.088
0.015
3.796
0.023
0.071
0.003
FeaUUp (JBU)
1.375
0.009
0.047
0.025
1.326
0.012
0.032
0.023
3.680
0.015
0.064
0.003
Bilinear + Tiles
1.522
0.105
0.093
0.020
1.484
0.167
0.062
0.014
5,921
0.112
0.073
0.001
JBU Tiles
1,580
0.103
0.077
0.017
1.493
0.157
0.046
0.013
5.898
0.095
0.065
0.001
Table 9. Metries for 4x upsampling across SigLIP, DFN CLIP, and RADIO. We primarily compare whether to use bilincar or JBU upsampling for the residual branch of the FeatSharp module, but also report the same values for our two baseline methods, bilincar upsampling itself, and FeatUp (aka JBU upsampling).
SigUP Fidelity Window
Sigl Fidelit Size
Fidelity Window
Fidelity Size
1.41
1.42
4.5
1.42
TearSharp
1.40
FeatSherpp Featlpp
fealls
1.30
3.5-
FuatSharp
FeatSharp
Blincrr
Blinear
3.2
Foot-lo
Restt/p
Window
Window
Window Sice
Window Sux
Figure 13. Ablation study over the choice of window size and upsampling factor for the FeatSharp module.
Local Attention Window Size figure 13 we run an ablation over local attention window sizes between 1 and 11. We
notice that either appear to be optimal.
Do We Even Need Attention/MLPP As be seen in figure 13, the choice of window size has very small impact on the resulting fidelity. However,
also that FeatSharp is achieving much higher fidelity scores than Bilinear and FeatUp. So, we also study what effect the attention block, and the MLP, are having on the resulting quality. We use the "Bilinear+ + Tile" input configuration from section B.2, and when applicable, use window size of 5 from section B.2. We show these results table 10. We notice that it's not until the long recipe where at inclusion of attention is helpful, and that goes a long way toward explaining the relative insensitivity to the window size in figure 13. Inspired by figure 12, we notice that the longer training recipe results in much sharper images. In table 10 we study the effect of running just the MLP for the "Long" recipe. We can see that while the fidelity continues improve, it doesn't keep up with the "Attention + MLP" setting, demonstrating that the attention module is indeed helpful.
C. Implementation Details
Upsampler Training We leverage the same training harness as in FeatUp (Fu et al., 2024), including leveraging the same attention downsampler. We disable the use of the CRF loss that was present in the FeatUp config. Parameters in table 11.
Fidelity
SigLIP
RADIO
Modules
2x Upsample
4x Upsample
2x Upsample
4x Upsample
Short
Long
Short
Long
Short
Long
Short
Long
Linear
1.521
1.581
1.508
1.566
4.702
5.397
4.926
5.701
Attention
1,505
1.566
1.500
1.560
4.410
5.253
4.656
5.446
MLP
1.522
1.581
1.506
1.566
4.741
5.397
4.934
5.707
Attention + MLP
1.513
1.584
1.502
1.568
4.668
5.502
4.849
5.711
Table 10. Fidelity metries for different combinations of blocks in the FeatSharp module (3). The "Long" recipe trains for 3x longer than the short recipe. We only study the "MLP" vs "Attention MLP" configurations the long recipe bocause those were the top two configurations in the short recipe.
20

---

FeatSharp: Your Vision Model Features, Sharper
Hyperparameter
FeatUp JBU
Regular
Long
Num GPUS
1
8
8
Batch Size (per GPU)
4
4
4
Batch Size (total)
4
32
32
Num Steps
2,000
3,000
9,000
Optimizer
NAdam
NAdam
NAdam
Learning Rate
0.001
le-4
le-4
Downsampler
Attention (k=7)
Attention (k=7)
Attention (k=7)
Num Jitters
5
5
5
CRF Weight
0.001
0
0
TV Weight
0
0
0
Feature Normalization
LayerNorm
PHI-S
PHI-S
Dataset
COCO
SA-IB
SA-1B
Multi-view Augs
Scale, Shift
Scale, Shift, HFlip, Rotate, Perspective
Table 11. Training hyperparameters. "FeatUp JBU" refers to the settings in the official https://gtthu..com//// commmmaamitooo/2//aattp Unless otherwise specified, we report numbers based on the "Long" schedule, which includes FeatUp reproduction values, maintain fairness.
RADIO Training We follow the staged setup in (Heinrich et al., 2024) section 4.2, with stages I and 2 being exactly identical. For stage 3. in the hi-res student branch, instead of bilerp downsampling the student features to match DFN CLIP and SigLIP (RADIOv2.5 baseline), we use our various upsampling methods to create hi-res feature maps which the student matches. We use our trained 3x upsamplers for the task, as they 're the smallest factor that produces feature maps larger than RADIO's hi-res partition. For FeatSharp. because we have the learned de-bias buffer which operates on the original model resolution, we also choose apply this the teachers in the low-res partition, as represents the fixed bias of the teacher model, and is thus not particularly useful information.
D. Additional Benchmarks
D.1. Probe3d
In table 12 we show the result of various configurations in Probe3d's (El Banani et al., 2024) depth probing for both DFN CLIP and RADIO. We can see that FeatUp produces the best results, however, we also demonstrate that this is likely due to the strong structural prior to the method, as the single best configuration was to use a FeatUp JBU stack with randomly initialized weights. Both FeatUp and FeatSharp are able to strongly improve over any configuration of regular DFN CLIP. For RADIO, we can see that both FeatUp and FeatSharp are still able to improve over baseline, albeit the margins are much smaller. While FeatSharp x does achieve the highest scores, the margin is too small to be significant compared to 2x and FeatUp, but still better than baseline. We observe essentially the same trend in table 13, where the random JBU stack works the best for DFN CLIP, and then FeatUp/Feattharp are comparable for RADIO.
D.2. NYUDv2
We also report metrics on NYUDv2 (Nathan Silberman & Fergus, 2012) in table 14 for both DFN CLIP and RADIO, similar to Probe3d configurations. We use the MLoRE (Yang et al., 2024b) harness and their conv probing for all configurations. We only use features from the final layer of the models. We can see here that unlike Probe3d, FeatSharp does a noticeably better job than FeatUp across the board, and with FeatSharp 2x, we get the strongest results for DFN CLIP. For RADIO, it's much tighter between FeatSharp and Baseline, however, FeatSharp significantly better than FeatUp.
E. Throughput Analysis
E.1. Empirical Throughput
In (6), using f(x) = c(1 2²) theoretically FeatSharp should always be cheaper than running the base model at the upsampled resolution. FeatSharp's cost is linear in the number of tokens, whereas a VIT is quadratic. In figure 14, we show the results of this prediction on actual hardware. As can be seen with the "Actual" curve, the picture a bit more complex than pure quadratic scaling, as
x?) (c.g. non-progressive tiling), we predict that based on the quadratic scaling of attention,
21

---

FeatSharp: Your Vision Model Features, Sharper
Depth (Scale Aware)
Depth (Scale Invariant)
Vision Encoder
Input Res
Upsampling Method
Output Tokens
d1 1
d21
d3
RMSE..
d11 f
d2 1
d3
RMSE +
378
27
0.303
0.575
0.772
0.168
0,440
0.710
0.842
0.134
756
51º
0.291
0.558
0.757
0.173
0.426
0.695
0.829
0.140
15122
108²
0.280
0.535
0.733
0.181
0.399
0.664
0.805
0.152
378-
2x Upsample features
544
0.301
0.573
0.773
0.443
0.713
0.133
(2 x 3788
Tiling
54
0.248
0,489
0.697
0.193
0.354
0.616
0.771
0.165
DEN CLIP
(4 x x 378"
Tiling
108
0.218
0.434
0.634
0.212
0.317
0.567
0.732
0.184
378'
FeatUp 2x
54"
0.430
0.712
0.851
0.128
0.538
0.793
0.894
0.107
3782
FeatUp 4x
108²
0.435
0.716
0.853
0.128
0.542
0.796
0.896
0.107
3784
FeatUp 4x (Random Weights)
1082
0.440
0.723
0.858
0.126
0.910
0.105
378
FeatSharp 2x
54"
0,398
0.685
0.837
0.136
0.512
0.772
0.882
0.113
378²
FeatSharp 1x
108
0,419
0.705
0.847
0.131
0.527
0.785
0.890
0.109
512"
324
0.472
0.749
0.873
0.118
0.584
0.827
0.097
10242
612
0.478
0.756
0.877
0.115
0.589
0.831
0.095
2048
128*
0.456
0.739
0.868
0.120
0.571
0.820
0.911
0.099
RADIO
512"
Upa 2x
64
0.482
0.764
0.885
0.114
0.606
0.840
0.921
0.092
5122
UpatUp 4x
128²
0.481
0.763
0.885
0.114
0.604
0.838
0.920
0.092
5122
FeatSharp 2x
64"
0,480
0.766
0.887
0.113
0.604
0.840
0.923
0.091
512²
FeatSharp 1x
128"
0.487
0.769
0.888
0.112
0.610
0.843
0.924
0.090
Table 12. Probe3D Depth metries. Linear probe output features.
"Random Weights" refers randomly initialized, untrained,
model.
Vision Encoder
Input Res
Upsampling Method
Output Tokens
Recall
Avg
0.01m
0.02m
0.05m
378
27a
49.26
26.02
44.47
77.30
7562
542
47.06
23.55
41.72
75.89
1512'
108²
41.59
18.08
35,30
71.41
DEN CLIP
378"
FeatU. 2x
54
54.40
30.99
51,20
81.02
3782
FeatU. 4x
1082
54.62
31.05
51.45
81.36
378
FeatUp 4x (Random Weights)
108
55.72
32.23
53.05
81.89
3782
ShatSharp
54
53.00
31,21
49.05
78.73
378
FeatSharp
1082
53.62
31.49
49.60
79.77
512"
324
59.49
37.20
56.44
84.82
102:2
642
58.23
37.21
54.70
82.77
20482
1282
57.22
34.99
53.53
83.15
RADIO
512"
FeatUp 2
612
60.39
38.52
57.51
85.16
512²
FecatU.p 4x
1282
64.72
39.01
57.87
85,29
512"
FeatSharp 2x
64"
60.69
40.11
$7.95
84.02
512"
FeatSharp 4x
128"
60.46
39.95
57.61
83.81
Table 13. Probe3D NAVI Correspondence. "Random Weights" refers randomly initialized, untrained, model.
between 1x and 3x upsample factors, the scaling is actually sub-linear, which likely reflects the period where self-attention is memory bound, and not compute bound, thus adding extra tokens doesn't 't proportionally increase the cost. Specifically, at 1.85x upsampling, we achieve the lowest time per token, and from then on, the cost approximately linearly increases (note that time per token is the first derivative of the time per image, so linear growth implics quadratic scaling, as predicted). Because UeatUp only runs the featurizer once, and its upsampling operation is cheap, we can see that achieves strong scaling regardless of resolution. FeatSharp requires x² + inferences with u being the upsample factor, its cost higher. Likely due to non-optimal kernels, upsampling (z 12602px) px). However, also as predicted by (6), FeatSharp's 's scaling is linear.
see that FeatSharp does start operating faster than the base model until about 3.3x
E.2. Proof of Equation 6
The progressive form of equation 6 is defined as f(x) Ei_ i2, and the regular form of self-attention is g(x) ca with being the upsampling factor per-side, and e being the cost to evaluate at the base resolution. We want to show that:
f())
Vx > 1
(11)
We start by rewriting the series for f() in closed form
f(z) x(x + 12xxx + 1)
(12)
6
22

---

FeatSharp: Your Vision Model Features, Sharper
Vision Encoder
Input
Upsampling Method
Output Tokens
SemSeg mioU
Depth RMSE. + Surf Normals
Edge Loss
378"
27a
53.15
0.551
23.49
0.130
756
51.11
0.589
23.33
0.127
378"
FeatUp 2x
51"
52.51
0,589
23.66
0.129
DEN CLIP
378
FealUp 4x
1082
52.45
0.601
24.15
0.129
378*
FeatSharp 2x
54"
54.29
0,579
23.14
0.126
378²
FeatSharp 4x
108
53.74
0.615
23.93
0.125
512"
32"
60.80
0.486
19.45
0.127
10242
64
62.15
0.479
18.55
0.123
512"
FeatUp 2x
64"
60.64
0,490
19.32
0.124
RADIO
5122
FeatUp 4x
128²
60.55
0.493
19.57
0.125
512
FeatSharp 2x
64
62,23
0.485
19.25
0.123
5122
FeatSharp 4x
128
61.71
0.511
19.82
0.122
Table 14. Multitask metrics NYUDv2 (Nathan Silberman Fergus, 2012) using the MLoRE (Yang et al., 2024b) convolutional probe
harness.
which is the sum of squares sequence multiplied by c. So now
f(x) S g(x) I(x + 1) S x
(13)
6
Given that > and that it's constant factor on both sides, we can eliminate it.
2x + 3=x + x S 6x
(14)
and with 0, we can further simplify to
2x 3x + S 6x
(15)
6x 2n.2 3z - 12
(16)
(r (6x
4x + 2
(17)
and thus 2 Vz 1, and also 6z + 4x +1>0 Vz E R. Therefore. (()) Vr> 1.
F. Effects of "Over-Tiling"
In figure 8, we can see that RADIO had learned some idiosyncratic representations when using the Tile and S2 upsampling algorithms. The effects are also apparent in figure where color spaces can entirely flip. To understand what's happening. we rely on the pretrained RADIOv2.5-L model, which has strong scale equivariance properties (Heinrich et al., 2024), and first that the number of tiles increases, the MSE error between the brute-force inference the tiling of that resolution, increases. We show these results in figure 15. Visually, we argue that the major increases in MSE owes largely to regions that lack context, making it difficult for the encoder (in this case RADIO), to come up with a reasonable representation of the tile-crop. We visualize this in figure 16. Notably, we can see that the 8 x 8 tiling difference images are generally whiter, indicating a general drift towards higher error. We can also see particular tiles that have more error, such as the notecard in row which gets nearly forgotten due to context. We can also see that there are a lot of errors with the car on row The bottom center of the floor on row has the same issue. So, while there appears be general upward error drift, it's exacerbated in regions without much variation.
given resolution and
G. FeatUp's Two Methods
The FeatUp (Fu et al., 2024) paper presented two methods for feature upsampling: The JBU-Stack, and the Implicit network. The resulting quality these two approaches are quite different, with the implicit network producing much finer detailed maps, but having the major drawback that it requires training a network per-image, and is thus computationally prohibitive
23

---

FeatSharp: Your Vision Model Features, Sharper
VIT-H/14 Throughput Analysis
Upsample Factor (1x 730 tok)
1x
2x
3x
5x
6x
7x
8x
0.06
0.05
0.04
Actual
Linear Scale
0.03
Upsampler: FealUp
Upsampler: FeatSharp
0.02
0.01
10000
20000
30000
40000
Tokens
Figure Throughput VIT-H/14 model (e.g. DFN CLIP) achieved an GPU, BS=1. The blue "Actual" curve reflects the time per token spent at various resolutions by the base model. "Linear Scale" assumes constant time token, based on the cost of Ix upsample factor. Note that "Time Per Token" effectively first derivative of "Time Per Image", so linear growth per-token represents quadratic growth in per-image.
(I minute per image). The JBU stack is effective at preserving edges, but also has the effect of over- ever-blurring object interiors. We show Figure 5 from (Fu et al., 2024) in our figure 17.
24

---

FeatSharp: Your Vision Model Features, Sharper
Tile Level vs. MSE
25
15
5
Level
Figure 15. MSE error between brute-force evaluation of RADIOv2.5-L. given resolution (512px2 (tile-level) and the tiling the same resolution.
25

---

FeatSharp: Your Vision Model Features, Sharper
Upsampling
Upeampling
ACIDCHAOS TOUR 1994
MIDGHOOS
BEID CHYOSS
CHASS
CODCH
TOMH4999 4994
TOURI004
TOURH994
TOURISY
LIBA
Tile 4x
DIf
Real 8k
Tile
Diff
Figure 16. Visualization of the emrrs between running RADIOv2.5-L given resolution, and the equivalent of tiling at the same resolution. The difference images are black when there is no difference, and white where there are large differences. The difference computed the euclidean distance of the full features, not their PCA projections.
26

---

FeatSharp: Your Vision Model Features, Sharper
FeatUp (JBU)
FeatUp (Implicit)
Image
Figure 17, FeatUp's 's two upsampler algorithms. Taken directly from their (Fu et al., 2024) Figure
27


Reviews:
[{"review_id": "oOMldx6nGw", "reviewer": "Reviewer_qjbo", "summary": "The paper introduces FeatSharp, a method which builds upon FeatUp (specifically its JBU upsampling variant) [1],  by incorporating higher-resolution tiled views and combining them with the upsampled feature maps from FeatUp. Additionally, FeatSharp includes a de-biasing module designed to remove fixed-pattern noise from the frozen vision encoder.\n\nThe authors find that their method out-performs FeatUp-JBU across a number of pretrained vision encoders (e.g., CLIP, DINOv2, SigLIP, SAM, RadioV2.5) for ADE20K segmentation. Additionally, they utilise their method to improve the training of a pre-existing method, AM-Radio (Multi-teacher distillation method), across a number of datasets/tasks.\n\n[1] Fu, S., Hamilton, M., Brandt, L. E., Feldmann, A., Zhang,\nZ., and Freeman, W. T. Featup: A model-agnostic\nframework for features at any resolution. In The Twelfth\nInternational Conference on Learning Representations,\n2024. URL https://openreview.net/forum?\nid=GkJiNn2QDF.\n\n[2] Ranzinger, M., Heinrich, G., Kautz, J., and Molchanov, P. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12490–12500, June 2024b.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thoughtful review.\n\n> Multi-view consistency / Fidelity Experiments: This measures the mse distance between warped-upsampled features and the encoder’s low-resolution features. While this ensures consistent alignment, it doesn’t guarantee improved semantic detail. One could achieve high fidelity by over-smoothing the features so these results are generally unconvincing but do provide some additional insight to their remaining experiments.\n\nWe argue that over-smoothing is actually what's happening with either bilinear upsampling, or even functionally how the JBU-stack is operating. Figure 6 shows how FeatUp 4x (JBU-stack) leads to over-smoothed results, but with edge preservation when there are strong enough edges in RGB space. This smoothness is why we compare FeatSharp against bilinear and FeatUp on fidelity directly in Figure 5, and FeatSharp consistently does better. So it's not that smoothing in the upsampled space doesn't work, but rather that it doesn't work as well as what FeatSharp is doing. Fidelity is telling us how internally consistent the representations are with respect to a particular model, under arbitrary crops and deformations. Over-smoothing would cause issues with fidelity when the crops are small, since nearly all variation will be gone.\n\n> My main concern is the lack of comparison against FeatUp (with implicit upsampling). The exclusion of FeatUp's implicit model is understandable given its computational cost (~1 min/sample, Appendix G). However, a small empirical comparison—at least in terms of performance and training time trade-offs—would significantly strengthen the paper's claims. Can you better justify why this is not feasible to do, at least for smaller scale experiments and add it to the paper?\n\nUpon your feedback, we have looked into this. It is ~1 min/sample for only the ViT-S/16 featurizer, but balloons quickly for the larger featurizers, like ~5 min/image for SigLIP2-SO400M-512, or ~4 min/image for RADIOv2.5-L/16 at 512px. However, the official implicit upsampler was also projecting the feature dimension down to 128 using PCA to achieve this speed [1, 2]. For ViT-S/16, it’s about 2x slower to use full features (e.g. 2 min/image). Due to the cost of running this mode, we ran a limited study on 100 images from COCO 2017 validation, and found that the fidelity is 3.17, higher than the 2.42 found with FeatSharp. However, FeatSharp at the same 16x upsample ratio takes 250ms/image (500x faster). These speeds are using a single H100 gpu.\n\n> Can you clarify why RADIO-AMP-L is performing worse than the baseline in Table 1?\n\nWe did our best to match the training setting between our reproduction and their claimed results. A key difference seems to come down to the fact that in RADIO-AMP, they bilinear downsample the student to match the teacher in the hi-res-student/low-res-teacher setting, whereas we bilinear upsample the teacher to match the student in our work. We made this change so that we'd have a comparison with a true upsampling baseline.\n\n>The proposed methods and evaluation are sensible, although only segmentation results on a single dataset (ADE20k) are shown for multiple of the vision encoders (DINOv2, SigLIP, SAM). More extensive evaluation is done to show the benefit on incorporating FeatSharp within the AM-Radio method (eg. Classification: imagenet1k - Zero Shot Retrieval: COCO + Flickr30k…) but overall the methods and evaluation are reasonable.\n\nWe have also included further benchmarking of our approach on object detection in response to reviewer oKYc below.\n\n[1] https://github.com/mhamilton723/FeatUp/blob/main/featup/train_implicit_upsampler.py#L197\n\n[2] https://github.com/mhamilton723/FeatUp/blob/main/featup/configs/implicit_upsampler.yaml#L26", "claims_and_evidence": "The main claim that FeatSharp improves upon FeatUp-JBU is well supported. The fidelity results (Figure 5), qualitative PCA visualizations (Figure 6), and ADE20k semantic segmentation results (Figure 7) all demonstrate improvements across a variety of vision encoders. Additionally, FeatSharp is tested in a multi-teacher distillation/agglomerative model training setup ( Section 4.4) on multi-task learning benchmarks (Table 1), where it also it shown to be beneficial. The authors show the FeatSharp-trained RADIO generally performs better than the state-of-the-art RADIO-AMP-L [3]. \n\n[3] Heinrich, G., Ranzinger, M., Hongxu, Yin, Lu, Y., Kautz, J., Tao, A., Catanzaro, B., and Molchanov, P. Radio amplified: Improved baselines for agglomerative vision foundation models, 2024. URL https://arxiv.org/ abs/2412.07679", "methods_and_evaluation_criteria": "The proposed methods and evaluation are sensible, although only segmentation results on a single dataset (ADE20k) are shown for multiple of the vision encoders (DINOv2, SigLIP, SAM). More extensive evaluation is done to show the benefit on incorporating FeatSharp within the AM-Radio method (eg. Classification: imagenet1k - Zero Shot Retrieval: COCO + Flickr30k…) but overall the methods and evaluation are reasonable.", "theoretical_claims": "Only equation (6) which appears sensible and has experimental results validating it in Appendix E.", "experimental_designs_or_analyses": "1. Multi-view consistency / Fidelity Experiments:\nThis measures the mse distance between warped-upsampled features and the encoder’s low-resolution features. While this ensures consistent alignment, it doesn’t guarantee improved semantic detail. One could achieve high fidelity by over-smoothing the features so these results are generally unconvincing but do provide some additional insight to their remaining experiments.\n\n2. The ablations given in Table 6 rely entirely on the measures of fidelity / measures of smoothness. These ablations would be more insightful if they included results using task-specific performance (eg. segmentation).", "relation_to_broader_scientific_literature": "The paper relates to prior work in feature upsampling, vision foundation models and multi-teacher distillation and agglomerative models. It builds directly on FeatUp [1], which introduced a model-agnostic feature upsampling framework using Joint Bilateral Upsampling (JBU) and an implicit upsampler. The authors extended the JBU variant of this work. The work is also relevant to Vision Transformers [2] and their resolution limitations due to quadratic complexity.\n\n[1] Fu, S., Hamilton, M., Brandt, L. E., Feldmann, A., Zhang,\nZ., and Freeman, W. T. Featup: A model-agnostic\nframework for features at any resolution. In The Twelfth\nInternational Conference on Learning Representations,\n2024. URL https://openreview.net/forum?\nid=GkJiNn2QDF.\n[2] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.", "essential_references_not_discussed": "Not to my knowledge.", "other_strengths_and_weaknesses": "The paper is generally well-written. The technical contributions of FeatSharp are incremental, but it effectively integrates ideas from prior works and helps to build upon FeatUp and AM-Radio.\n\nA major limitation I see is that that there are no direct comparisons made with FeatUp using implicit upsampling, which is the better performing variant of FeatUp. This is understandable to an extent as it requires training on a per-image basis, which takes ~1min/sample (Appendix G). If the performance of FeatSharp was shown to be  close/comparable to FeatUp with implicit upsampling, while greatly reducing training time this would strengthen the paper for me.\n\nThe ablations in Table 6 should show the design choice effects on some task-specific performance measures (e.g., segmentation accuracy or retrieval scores), but they instead focus on on multi-view consistency measures.", "other_comments_or_suggestions": "None", "questions_for_authors": "My main concern is that lack of comparison against FeatUp (with implicit upsampling). The exclusion of FeatUp's implicit model is understandable given its computational cost (~1 min/sample, Appendix G). However, a small empirical comparison—at least in terms of performance and training time trade-offs—would significantly strengthen the paper's claims. Can you better justify why this is not feasible to do, at least for smaller scale experiments and add it to the paper?\nCan you clarify why RADIO-AMP-L is performing worse than the baseline in Table 1?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zBOEGsZUg1", "reviewer": "Reviewer_oKYc", "summary": "The paper discusses improving vision model features by refining their sharpness and resolution. It builds on the JBU algorithm to provide more detailed feature maps and studies how to clean features effectively using ViT-Denoiser's methods. The paper also enhances the AM-RADIO framework, achieving better benchmark performance and feature adaptation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review.\n\n> what does \"2x upsampling\" exactly mean? \n\nWe increase the width and height by 2x.\n\n> However, the quantitative results are not very convincing to me. For example, in Fig. 7, the neumerical results on semantic segmentation, why does 3x sampling generally underperform both 2x and 4x? Intuitively, if the upsample method is correct, the semantic segmentation task should continuously benefit from the high-definition feature map. However, the authors observed counterintuitive results but did not provide explanations or analyses.\n\nWe agree that we're missing the analysis for the 3x case. However, it's not necessarily true that increasing the resolution of the features should increase semantic segmentation benchmarks. Our goal is to produce higher-resolution features that are consistent with the featurizer itself, and that is occasionally at odds with doing well on a particular downstream task. For example, resolution is not the reason that PaliGemma is worse than DINOv2-L which is worse than RADIOv2.5-L, because they all encode the same number of tokens. Instead, in figure 7, what we observe is that FeatUp/FeatSharp are cleaning up the noisy models by favoring the view-consistent representations (compared to the baseline horizontal lines). We further observe that FeatSharp consistently does better than FeatUp on the task. An alternative way to look at this is by comparing \"Baseline Inpt-1x\" against \"Baseline Inpt-2x\". This gives us a sense of whether it's merely increasing the resolution that yields better segmentation. We see that DINOv2-L and RADIOv2.5-L improve when increasing resolution. However, the noisy models (DFN CLIP, PaliGemma, SigLIP) either see negligible change, or even get a bit worse.\n\nWe observe a similar effect in tables 11, 12, and 13 where the optimal choice of upsampler depends on the task and the featurizer itself. For DFN CLIP in the NAVI Correspondence task, it appears as though a simple bilateral upsample works the best, with entirely untuned weights, owing to both encoder noise, and also perhaps OOD alignment between the model and task. For table 13, we see that FeatSharp operates similarly to running RADIO at the resolution that matches the number of output tokens, with FeatSharp always being better than the low-res baseline (512px input).\n\n> Additionally, in most experiments in the appendix, I also found that FeatSharp does not consistently bring quantitative gains across various visual prediction tasks. These issues raise concerns about the significance and contributions of the methods in this paper: Merely looking good in visualizations does not prove the method's effectiveness. The authors need more numerical superiority in the results to demonstrate the specific benefits of the new model for visual tasks.\n\nThank you for this feedback, as it has motivated us to run another benchmark setting in order to provide further evidence of the efficacy of our method. We have integrated our method, as well as SAPA [3], and ReSFU [4], into Detectron2, and evaluated on COCO 2017. We used Edge [5], which allowed us to create a <featurizer>+<upsampler>+DINO [6] harness. We chose to use RADIOv2.5-L and SigLIP2-SO400M-512 [7] as featurizers, with the latter to further demonstrate versatility.\n\n\n### RADIOv2.5-L (512px input)\n|Upsampler|Upsample Factor|Fidelity|AP|AP Small|AP Medium|AP Large|\n|-|-|-|-|-|-|-|\n|Baseline|1||51.38|28.73|56.56|73.72|\n|Bilinear|2||51.61|28.43|56.98|74.14|\n|SAPA|2|2.81|41.44|15.92|45.08|69.77|\n|ReSFU|2|3.69|49.81|26.22|55.37|73.55|\n|FeatUp|2|3.71|46.71|21.77|52.01|72.25|\n|FeatSharp|2|**5.35**|**54.83**|**34.72**|**59.40**|**74.40**|\n\n\n### SigLIP2-SO400M-512 (512px input)\n|Upsampler|Upsample Factor|Fidelity|AP|AP Small|AP Medium|AP Large|\n|-|-|-|-|-|-|-|\n|Baseline|1||52.66|30.31|57.94|74.31|\n|Bilinear|2||52.69|30.19|57.84|74.16|\n|SAPA**|2|\n|ReSFU|2|1.62|50.84|28.45|56.18|73.69|\n|FeatUp|2|1.54|47.42|22.87|53.17|72.80|\n|FeatSharp|2|**1.89**|**55.93**|**36.85**|**61.00**|**74.62**|\n\n** *SAPA failed in backprop with a cuda configuration error, due to the size of the feature map and channel count of the model.*\n\nWhile FeatSharp improves the metrics across the board, it makes its largest improvements on AP Small, followed by AP Medium. This makes intuitive sense as the tiling allows for the detection of small objects that otherwise are missed by any upsampler which doesn't have access to multiple views. FeatSharp provides a holistic method which not only keeps representations consistent as resolution increases, but also allows for the incorporation of new details which are missed by the low res encoding, which is an important advancement of the literature.\n\n[3] [SAPA](https://arxiv.org/pdf/2209.12866)\n\n[4] [ReSFU](https://arxiv.org/pdf/2407.02283)\n\n[5] [Edge](https://dgcnz.github.io/edge/part2/adapting.html)\n\n[6] [DINO](https://arxiv.org/abs/2203.03605) (*not to be confused with DINOv2*)\n\n[7] [SigLIP2](https://arxiv.org/abs/2502.14786)", "claims_and_evidence": "Yes, the claims are well supported by their evidence.", "methods_and_evaluation_criteria": "Overall, there are two primary criteria used for evaluating the FeatSharp method.\n\n1. The qualitative results look nice, with the upsampled feature maps produced by FeatSharp look indeed \"sharper\" than baseline reseults.\n\n2. However, the quantitative results are not very convincing to me. For example, in Fig. 7, the neumerical results on semantic segmentation, why does 3x sampling generally underperform both 2x and 4x? Intuitively, if the upsample method is correct, the semantic segmentation task should continuously benefit from the high-definition feature map. However, the authors observed counterintuitive results but did not provide explanations or analyses. Additionally, in most experiments in the appendix, I also found that FeatSharp does not consistently bring quantitative gains across various visual prediction tasks. These issues raise concerns about the significance and contributions of the methods in this paper: Merely looking good in visualizations does not prove the method's effectiveness. The authors need more numerical superiority in the results to demonstrate the specific benefits of the new model for visual tasks.", "theoretical_claims": "I did not find incorrect theoretical claims in the paper.", "experimental_designs_or_analyses": "The authors conducted experiments across a variety of visual tasks which technically sound. However, as I mentioned in “Methods And Evaluation Criteria”, some results are unsatisfactory and lack explanations and analyses. To validate the practical significance of sharp upsampling, I suggest that the authors supplement experiments on vision-language benchmarks, which involve many fine-grained reasoning tasks, such as detecting a small object in a large scene. I believe these tasks could better substantiate the claims of FeatSharp's effectiveness.", "relation_to_broader_scientific_literature": "It has a close relationship to open-vocabulary semantic segmentation tasks witch typically leverage CLIP to perform pixel-level predictions. It might also be a good idea t", "essential_references_not_discussed": "No essential references missing, but I can recommend some papers in open-vocab segmentation that might help evaluating the FeatSharp method:\n\n[1] Extract free dense labels from clip (ECCV'22)\n\n[2] Clip surgery for better explainability with enhancement in open-vocabulary tasks (arxiv'23)\n\n[3] SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference (ECCV'24)\n\n[4] Clearclip: Decomposing clip representations for dense vision-language inference (ECCV'24)", "other_strengths_and_weaknesses": "See comments above.", "other_comments_or_suggestions": "There are some minor issues in the paper which might cause confusion. Eg, what does \"2x upsampling\" exactly mean? Does it mean scale up both width and height by 2x or scale up the area by 2x?", "questions_for_authors": "No more questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JgrguyDYrD", "reviewer": "Reviewer_8ugu", "summary": "The authors proposed a novel method for efficiently upsampling feature maps of low-resolution ViTs (CLIP) to capture fine-grained details typically lost due to limited resolution. Built upon FeatUp, their method adds de-biasing and tiles fusion modules to incorporate detailed tile features, resulting in higher levels of detail, with extensive experiments demonstrating the effectiveness.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your review.\n\n> I'm curious about the performance on fine-grained image image classification datasets, like CUB-200-2011\n\nThank you for your suggestion. Due to time and compute constraints, we were limited to running only one further study comparing upsamplers, and we selected COCO 2017 detection due to its ubiquity in the literature. We have those results in [this rebuttal to reviewer oKYc](https://openreview.net/forum?id=lioemOcq3H&noteId=ZiLr91BWoe). Given the results on COCO, it is plausible that similar effects could be observed with detailed datasets, particularly with small foreground categories, such as Caltech-UCSD Birds-200-2011, but are forced to leave the analysis to future work.\n\n> The paper has some presentation mistakes. One fire sign in Fig. 1 is out of the box, and the colors of the boxes are confusing. The citation in Line 249 is confusing. One captioned data in Fig. 5 is out of the box. The presentation can be further promoted.\n\nThank you, we will absolutely revise these issues in a prospective camera ready.", "claims_and_evidence": "They claim FeatSharp can upsample low-resolution feature maps while picking up on fine-grained details, and it is well evidenced by the fidelity plot in Figure 5 and other visualizations of upsampling feature comparisons.", "methods_and_evaluation_criteria": "No.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is easy to follow.\n2. FeatSharp can upsample low-resolution feature maps while picking up on fine-grained details, and it is well evidenced by the fidelity plot in Figure 5.\n3. The effectiveness is validated through multi-view consistency and semantic segmentation. The performance improvement on ADE20K is convincing.\n4. Compared with FeatUp, the PCA visualization of FeatSharp is much closer to Real 4x.\n\nWeaknesses:\n1. When the number of tokens is larger, the inference cost of FeatSharp is much higher than FeatUp, as shown in Figure 15. Considering its performance improvement evidenced by the fidelity plot and other results, the cost is acceptable, which can be further studied as the future work.\n3. The paper has some presentation mistakes. One fire sign in Fig. 1 is out of the box, and the colors of the boxes are confusing. The citation in Line 249 is confusing. One captioned data in Fig. 5 is out of the box. The presentation can be further promoted.", "other_comments_or_suggestions": "Please refer to strengths and weaknesses.", "questions_for_authors": "I'm curious about the performance on fine-grained image image classification datasets, like CUB-200-2011 [1]. \n\n[1] Wah, Catherine, et al. \"The caltech-ucsd birds-200-2011 dataset.\" (2011).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1ydhTcRMAj", "reviewer": "Reviewer_Lmcr", "summary": "This paper introduces a novel upsampling method designed to address the low-resolution feature map limitations of vision encoders, particularly Vision Transformers. The proposed approach builds upon FeatUp, the current state-of-the-art upsampler, by integrating FeatUp’s Joint Bilateral Upsampling (JBU) with a mosaic of tiles, followed by processing through a single local attention block. The authors demonstrate the effectiveness of this method across various dense prediction tasks, including semantic segmentation, object detection, depth estimation, and surface normal prediction. Additionally, the study highlights how incorporating FeatSharp within RADIO training enables low-resolution-only teacher models to generate high-resolution distillation targets, further enhancing model performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review.\n\n> Limited Novelty: The proposed method primarily builds upon FeatUp, incorporating well-established concepts such as tiling for handling high-resolution images and a simple learnable buffer for de-biasing. The integration of FeatUp, tiling, and attention layers is straightforward, and given these components, the performance improvement over FeatUp is unsurprising. The paper does not introduce fundamentally new techniques or discoveries.\n\nWe disagree that pulling together disparate concepts that are seen in other domains of computer vision itself lends to limited novelty. Tiling is the best example of this. While it is used as a method to work around the inflexibility of most ViTs within the context of VLMs, or implicitly when doing things like sliding-window segmentation, the major insight in our work is that the tiles provide fine-grained guidance to the upsampler. This type of guidance is not present in other upsampler works, as they either rely on the raw pixels (which lack semantic meaning), or they rely on the encoder hierarchy (which isn't applicable to ViTs). ReSFU [2], for example, uses raw pixel guidance. The other critical point about upsampling has to do with the fact that small features are irrecoverable from a combination of raw-pixel + low-res featurizer. Methods like FeatUp's implicit model (>1000 views), and FeatSharp with tiling, however, are able to recover this detail because they're observing the small features. We can see this effect in Figure 13, SigLIP section, left column. These single input visualizations show what happens when you only get a single input source (plus raw pixels). We otherwise use the same model, including the single transformer block. Only the tiles observe enough detail to separate the text lines. For the RADIO section, relying on the JBU stack from FeatUp entirely misses the street lamp, blurring it into the background, and bilinear is unable to recover the latticing. Further, as part of our [rebuttal to oKYc](https://openreview.net/forum?id=lioemOcq3H&noteId=ZiLr91BWoe) we further our upsampler in an object detection setting, with additional comparisons with SAPA and ReSFU, and find that FeatSharp consistently does better. Particularly, we make the largest gains on small objects, directly providing evidence that tiles allow for the introduction of fine-grained detail. The set of techniques we chose to incorporate (multiview consistency, de-biasing, tiling, attention) are not ad-hoc, but form a cohesive novel solution to a problem.\n\n> Limited Generalization and Marginal Gains: The paper evaluates FeatSharp exclusively within the RADIO model as the teacher, without testing its applicability to other architectures. This raises concerns about its generalizability. Moreover, the reported improvement over FeatUp is only 0.39%, which is almost negligible, calling into question the practical significance of the proposed enhancements.\n\nWe believe that this characterization is not supported by the evidence. We chose the RADIO training harness because it is currently the state of the art for general perception foundation models, and because it has a training setting that could directly benefit from a feature upsampler, specifically, the low-res-teacher/hi-res-student part of their training protocol. However, it's also not the case that this setting is a 0.39% improvement over FeatUp. FeatUp damages RADIO training, resulting in a -0.13% change across the benchmark suite. So FeatSharp is +0.52% relative. FeatSharp is also 0.6% better than the RADIO-AMP reported results. While 0.6% may seem small, we stress that this is for a single model over an entire suite of 31 tasks. Further, the RADIO training setting was one of two benchmark evaluations performed in the main body, the other being ADE20k semantic segmentation with six different base featurizers, and in all cases, FeatSharp was shown to be superior. In the appendix, we additionally study upsampling DFN CLIP and RADIO on Probe3d and NYUDv2 benchmarks. For an additional point of clarification, section 4.4 isn’t using an existing RADIO model as a teacher, but rather, we’re using the training protocol, and we’re upsampling DFN CLIP and SigLIP. We further chose SigLIP2-SO400M-512 as an additional featurizer in the rebuttal object detection study to provide even more evidence of broad applicability.\n\n> Insufficient Baseline Comparisons: The work only compares against FeatUp, assuming it to be the sole relevant competitor for feature map upsampling. This narrow comparison overlooks other state-of-the-art methods, limiting the credibility of the results. A broader evaluation against multiple competitive approaches is necessary to justify the effectiveness of FeatSharp.\n\nThank you for this feedback. We have included SAPA and ReSFU in the [object detection study in our rebuttal to oKYc](https://openreview.net/forum?id=lioemOcq3H&noteId=ZiLr91BWoe) upon your advice.", "claims_and_evidence": "The claims made are supported by evidence.", "methods_and_evaluation_criteria": "Yes the proposed methods/evaluation criteria make sense.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "See strength and weakness.", "relation_to_broader_scientific_literature": "Related to vision encoder/model distillation.", "essential_references_not_discussed": "See strength and weakness.", "other_strengths_and_weaknesses": "Strengths:\n\n1. Proposes a comprehensive framework for enhancing upsampling in vision encoder feature maps by effectively integrating multiple techniques, including FeatUp’s Joint Bilateral Upsampling (JBU), de-biasing, and local attention.\n\n2. Demonstrates strong experimental results across a diverse range of vision tasks, including semantic segmentation, object detection, depth estimation, and surface normal prediction. Additionally, the study validates the effectiveness of incorporating FeatSharp into the training framework.\n\n3. Provides a detailed cost analysis, showing that FeatSharp introduces only a minimal computational overhead compared to FeatUp, as evidenced by time-per-token evaluations.\n\nWeaknesses:\n\n1. Limited Novelty: The proposed method primarily builds upon FeatUp, incorporating well-established concepts such as tiling for handling high-resolution images and a simple learnable buffer for de-biasing. The integration of FeatUp, tiling, and attention layers is straightforward, and given these components, the performance improvement over FeatUp is unsurprising. The paper does not introduce fundamentally new techniques or discoveries.\n\n2. Insufficient Baseline Comparisons: The work only compares against FeatUp, assuming it to be the sole relevant competitor for feature map upsampling. This narrow comparison overlooks other state-of-the-art methods, limiting the credibility of the results. A broader evaluation against multiple competitive approaches is necessary to justify the effectiveness of FeatSharp.\n\n3. Limited Generalization and Marginal Gains: The paper evaluates FeatSharp exclusively within the RADIO model as the teacher, without testing its applicability to other architectures. This raises concerns about its generalizability. Moreover, the reported improvement over FeatUp is only 0.39%, which is almost negligible, calling into question the practical significance of the proposed enhancements.", "other_comments_or_suggestions": "I recommend that the authors compare their method with the following state-of-the-art articles.\n\n[1] Yue, Y., Das, A., Engelmann, F., Tang, S., Lenssen, J.E. (2025). Improving 2D Feature Representations by 3D-Aware Fine-Tuning. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024.\n\n[2] Zhou et. al. A Refreshed Similarity-based Upsampler for Direct High-Ratio Feature Upsampling. In: arXiv July 2024. https://arxiv.org/abs/2407.02283.", "questions_for_authors": "See Strengths and Weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No concerns."}}]

Author rebuttal:

Meta-review: All reviewer recognise the benefit of the proposed method and recommend acceptance for the paper. Reviewers agree that the paper  presents a well-structured and comprehensive approach, demonstrating solid performance improvements across various vision tasks. I'm in favor of accepting this paper to ICML
Decision: Accept (poster)