FeatUp: A Model-Agnostic Framework for Features at Any Resolution
Authors: Stephanie Fu
Mark Hamilton
Laura E. Brandt
Axel Feldmann
Zhoutong Zhang
William T. Freeman
Venue: ICLR 2024 poster (2024)
Keywords: deep learning
deep features
computer vision
feature upsampling
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We introduce a method for obtaining high-resolution features from any vision model, trainable end-to-end with the model itself and producing high-quality results on vision tasks.

Abstract: Deep features are a cornerstone of computer vision research, capturing image semantics and enabling the community to solve downstream tasks even in the zero- or few-shot regime. However, these features often lack the spatial resolution to directly perform dense prediction tasks like segmentation and depth prediction because models aggressively pool information over large areas. In this work, we introduce FeatUp, a task- and model-agnostic framework to restore lost spatial information in deep features. We introduce two variants of FeatUp: one that guides features with high-resolution signal in a single forward pass, and one that fits an implicit model to a single image to reconstruct features at any resolution. Both approaches use a multi-view consistency loss with deep analogies to NeRFs. Our features retain their original semantics and can be swapped into existing applications to yield resolution and performance gains even without re-training. We show that FeatUp significantly outperforms other feature upsampling and image super-resolution approaches in class activation map generation, transfer learning for segmentation and depth prediction, and end-to-end training for semantic segmentation.

Published as a conference paper at ICLR 2024
FEATUP: A MODEL-AGNNOTIC FRAMEWORK FOR FEATURES AT ANY RESOLUTION
Stephanie Fu" MIT
Mark Hamilton' MIT, Microsoft
Laura Brandt MIT
Axel Feldmann MIT
Zhoutong Zhang Adobe Research
William T. Freeman
MIT, Google
Original
Ros
FeetUp
FeatUp
Upsampled Features
Maps
Segmentation
Depth Estimation
Figure 1: FeatUp upsamples image features from any model backbone, adding spatial resolution existing semantics. High-res features can be learned either as per-image implicit network or general-purge purposc upsampling operation; the latter is drop-in module to improve downstream dense prediction tasks.
ABSTRACT
Deep features are cornerstone of computer vision research, capturing image se- manties and enabling the community to solve downstream tasks even the zero- or few-shot regime. However, these features often lack the spatial resolution to directly perform dense prediction tasks like segmentation and depth prediction because models aggressively pool information over large areas. In this work, we introduce FeatUp, task- and model-agnostic framework to restore lost spatial information deep features. We introduce two variants of FeatUp: one that guides features with high-resolution signal in a single forward pass, and one that fits an implicit model single image reconstruct features any resolution. Both approaches a multi-view consistency loss with deep analogies to NeRFs. Our features retain their original semantics and can be swapped into existing applications to yield resolution and performance gains even without re- re-training, We show that FeatUp significantly outperforms other feature upsampling and image super-resolution approaches in class activation map generation, transfer learning for segmentation and depth prediction, and end-to-end training for semantic segmentation.
INTRODUCTION
Recently, considerable effort has been made develop methods to extract features from data modalities such as vision (Dalal & Triggs 2005 LoweDavid 2004 Weiss et al. 2016 He et al.
2019 Caron et (021), text (Mikolov al. Devlin Devlin al. 2018 Radford & Narasimhan
2018). and audio (Schneider et al. 2019 Hsu et al. (2021). These features often form the backbone of different methods, including classification (Shao et 2014), weakly-supervised learning (Ahn
et al. 2019 Hamilton et al., (2022), semantic segmentation (Wang et al.) (200)). optical flow (Liu let al. 2010 Teed & Deng (2020), neural rendering (Kobayashí et al. (022), and more recently, image generation (Rombach et al. (021). Despite their immense success, deep features often sacrifice spatial resolution for semantic quality. For example, ResNet-50 (He et al. (2015) produces x 7 deep features from a 224 224 pixel input (32x resolution reduction). Even Vision Transformers (ViTs)
"Equal contribution. Corresponding authors: markth mit.edu and fus csail.mit.ed

---

Published as a conference paper at ICLR 2024
Learned Downsampler
Learned Upsampler Of
Backbone
Transform ter
MultiView Reconstrruttio
Transform TCT
Backbone
Learned Downsampler
Figure 2: The FeatUp training architecture. FeatUp learns upsample features through consistency loss low resolution "views" of model's features that arise from slight transformations of the input image.
DDosovitskiy et al. 2020) incur significant resolution reduction, making it challenging to perform dense prediction tasks such as segmentation or depth estimation using these features alone.
To mitigate these issues, we propose FeatUp: novel framework to improve the resolution of any vision model's 'ss features without changing their original "meaning" orientation. Our primary insight, inspired by 3D reconstruction frameworks like NeRF (Mildenhall et al., 2020), is that multiview consistency low-resolution signals can supervise the construction of high-resolution signals. More specifically, we learn high-resolution information by aggregating low resolution views from a model's 's outputs across multiple "jittered" (e.g. flipped, padded, cropped) images. We aggregate this information by learning upsampling network with multiview consistency loss. Our work explores two architectures for upsampling: single guided upsampling feedforward network that generalizes across images, and an implicit representation overfit to single image.
This feedforward upsampler is parameterized generalization a Joint Bilateral Upsampling (JBU) filter (Kopf et al. 2007) powered by CUDA kernel orders of magnitude faster and less memory- intensive than existing implementations. This upsampler can produce high quality features aligned object edges computational cost comparable
few convolutions. Our implicit upsampler draws direct parallel to NeRF and overfits a deep implicit network to signal, allowing for arbitrary resolution features and low storage costs. In both architectures, our upsampled features can be drop-in replacements in downstream applications because our methods do not transform the semantics of the underlying features. We show that these upsampled features can significantly improve variety of downstream tasks including semantic segmentation and depth prediction. Additionally, we show that model explanation methods such as CAM can be made higher-resolution using upsampled features. In particular, one can study model's behavior with much greater detail without the need for complex methods based on relevance and information propagation (Lee et al.] 2021 Qin et al., 2019). In summary, include short video describing FeatUp at aak..m//featu and make the following contributions:
FeatUp: a new method significantly improve the spatial resolution of any model's features, parametrized as either fast feedforward upsampling network an implicit network.
A fast CUDA implementation of Joint Bilateral Upsampling orders magnitude more efficient than standard PyTorch implementation and allowing guided upsampling in large-scale models.
We show that FeatUp features can be used as drop-in replacements for ordinary features to improve performance on dense prediction tasks and model explainability.
RELATED WORK
Image-adaptive filtering. Adaptive filters are commonly used to enhance images while preserving their underlying structure and content. For example, bilateral filters (Tomasi & Manduchi 1998] Caraffa al.) 2015 Xiao & Gan 2012) apply spatial filter to Now-resolution signal and an intensity filter to a nighrresolution guidance to blend information from the two. Joint Bilateral Upsampling (JBU) Kopf et al. 2007) uses this technique upsample low-resolution signal with high-resolution guidance. JBU has been used successfully for efficient image enhancement and other applications. Recently, some works embed bilateral filtering approaches (Mazzini 2018) and nonlocal
2

---

Published as a conference paper at ICLR 2024
Simple Downsampler 0%
Attention Downsampler 0
softmax
Carmed Blur
Figure 3: We introduce two learned downsamplers. The simple downsampler (Left) is fast learned blur kernel. The attention downsampler (right) combines a predicted salience map with spatially invariant kernels. This downsampler can better adapt to networks with nonlinear and dynamic receptive fields.
means (Buades et al., 2005 into convolutional networks (Gadde et al. 2015; Wang et al.] 2017 and vision transformers (Qian et al. 2021| Lu et al., 2022d). Shape Recipes (Freeman & Torralba 2002) learn the local relationship between signals create up-sample target signals. Pixel-adaptive convolutional (PAC) networks (Su et al. 2019) adapt a convolution operation to input data and has been used to advance performance in segmentation (Araslanov & Roth [2200] Prangemeier et al.) 2020) and monocular depth estimation (Guizilini et al. [200] Choi et al., 2021] 2020). The Spatially- Adaptive Convolution (SAC) in (Xu et al.] 2020) factorizes the adaptive filter into attention map and convolution kernel. (Gadde et al. (016) extend bilateral filtering to superpixels and embed this operation inside of deep network improve semantic segmentation. This class of methods, effective across variety of applications, directly incorporates spatial information into the task while still allowing for flexibility in learning network.
Image super-resolution. One of the earliest deep unsupervised super-resolution methods was Zero- Shot Super-resolution (ZSSR) (Shocher et al. 2018), which learns a single-image network at test time. Local implicit models (Chen et 2021)) use locally-adaptic models to interpolate information, and have been shown to improve the performance of super-resolution networks. Deep Image Priors (Ulyanov et al. 2020) show that CNNs provide inductive biases for inverse problems such zero-shot image denoising and super-resolution. While there extensive literature on image super-resolution, these methods are not well-adapted to handle ultra-low resolution, yet -ishdiinensional deep features as we show in the Supplement.
General-purpose feature upsampling. widely-used approach upsample deep feature maps is bilinear interpolation. Though efficient, this method blurs information and is insensitive to the content or the high-resolution structure in the original image. Nearest neighbor and bicubic interpolation (Keys) 1981) have similar drawbacks. Evaluating a network on larger inputs can achieve higher resolutions but with a steep computational cost. Furthermore, this often degrades model performance and semantics due to the decreased relative receptive field size. For deep convolutional networks, one popular technique is to set final convolution strides to 1 (Long et al. 2015: QQn et al.) 2019). However, this approach yields blurry features, as the model's receptive field is still large. Recent works using visual transformers (Amir et al., 2021: Tumanyan et al., 2022) perform a similar modification on input patch strides and interpolate positional encodings. Though simple and reasonably effective, this approach incurs a steep increase in computational footprint for every 2x increase in resolution, making it impossible to use in practice for larger upsampling factors. This approach can also distort features because of the previously mentioned fixed receptive field of the patches.
|
Image-adaptive feature upsampling. Many different operations exist in the literature create features at higher resolutions. Deconvolutions (Shi et al. 2016 Dumoulin & Visin
201 6a
Noh
al. 2015 Johnson et al. 2016) and transposed convolutions (Dumoulin & Visin 201 use learned kernel to transform features into a new space with a larger resolution. The resize-convolution
(Odena al. 2016) appends learned convolution to deterministic upsampling procedure and reduces checkerboard artifacts that plague deconvolutions (Gauthier 2015) Odena et al.) 2016: Dong et al.
2015). The resize-convolution is now a common component of image decoders such as the U-Net (Ronneberger et al., (015) and has been applied to semantic segmentation (Li et al.] 2018 Huang ct al. 2020 Fu et al. (020) and super-resolution (Lai et al. 2017 Tong et al. 2017 Ledig et al.
2017). Other methods such as IndexNet (Lu et al. 2022a) and Affinity-Aaare U Upsampling (A2U)
Dai et al. 2020)) are effective on image matting but fall short on other dense prediction tasks (Lu
et al. 2022bb. Methods such as Pixel-Adaptive Convolutions (Su et al.) 2019), CARAFE (Wang
ct
al. 2019|SAPA et al. (002c)). and DGF Wu et al. (2019) use learned input-adaptive operators
to transform features. Though PAC is flexible, it does not upsample existing feature maps faithfully
3

---

Published as a conference paper at ICLR 2024
FeatUp Tf (Implicit)
FeatUp (JBU)
dr
Color Feats
Positnn
implictt
MLP
Backbone
Figure 4: Our Implicit version of FeatUp leams an implicit network to upsample a single image's features. Our JBU FeatUp learns stack of JBUs that learns quickly upsample features from large image corpora.
and instead is used to transform features for downstream tasks. Additionally, DGF approximates the JBU operation with learned pointwise convolutions and linear maps, but does not fully implement JBU because the local query/model is computationally intractable. This is precisely the problem we solve exactly with our new efficient CUDA kernel. Additionally, FADE [Lu al, (2022b) introduces a new shifi operator and uses decoder features to produce joint feature upsampling module. Huu iet al. (2022) view feature upsampling in a different light, focusing on a nearest-neighhors approach to align feature maps in encoder-decoder architectures with IFA. While IFA performs well on the specific semantic segmentation benchmarks, does not take advantage of image guidance and fails to learn high quality representations outside of the encode-decoder framework, show in the Supplement.
3 METHODS
The core intuition behind FeatUp is that one can compute high-resolution features by observing multiple different "views" of low-resolution features. We draw a comparison with 3D scene recon- struction models such as NeRF (Mildenhall et al., 2020); in the same way that NeRF builds an implicit representation (Sitzmann et al., 2020b) Chen & Zhang, 2019) of a 3D scene by enforcing consistency across many 2D photos of the scene, FeatUp builds an upsampler by enforcing consistency across many low-resolution feature maps. Like in broader NeRF literature, a variety of methods can arise from this basic idea. In this work, introduce lightweight, forward-pass upsampler based on Joint Bilateral Upsampling (Kopf ct al., (2007) as well as an implicit network based upsampling strategy. The latter learned per-image and query-able at arbitrary resolution. We provide overview of the general FeatUp architecture in Figure 2
The first step in our pipeline is to generate low-resolution feature views to refine into a single high- resolution output. To this end, we perturb the input image with small pads, scales, and horizontal flips and apply the model to each transformed image extract a collection of low-resolution feature maps. These small image jitters allow us to observe tiny differences in the output features and provide sub-feature information to train the upsampler.
Next, we construct a consistent high-resolution feature map from these views. We postulate that we can learn latent high-resolution feature map that, when downsampled, reproduces our low-resolution jittered features (see Figure 2). FeatUp's 's downsampling is a direct analog to ray- marching, just as 3D data is rendered into 2D in this NeRF step. our downsampler transforms high-resolution features into low-resolution features. Unlike NeRF, we do not need to estimate parameters that generate each view. Instead, we track the parameters used to "jitter" each image and apply the same transformation to our learned nigh-resolution features prior downsampling. We then compare downsampled features to the true model outputs using a gaussian likelihood loss (Hamilton et al. 2020). A good high-resolution feature map should reconstruct the observed features across all the different views.
crops, More formally, let & E be from a collection of small transforms such as pads, zooms, horizontal flips, and their compositions. Let be input image, be model backbone, a, be learned downsampler, and of be a learned upsampler. We can form the predicted high-res features For by evaluating Far = a (f(),, 2:. We note that this parameterization allows a' be a guided upsampler (which depends on both æ and (x)), an unguided upsampler (which depends on only f())) ), an implicit network (which depends on only 2), learned buffer of features (which depends on nothing). We can now form our main multi view reconstruction loss term as follows:
Crec
Σ 1ET
25 Ilf (t())) σ2 (t (Fhrr) log(s)
1
(1)
T
4

---

Published as a conference paper at ICLR 2024
FeatUp (JBU)
FeatUp (Implicit)
Image
Low-Res
Bilinear
Resize-conv
Strided
Large image
DIP
CARAFE
SAPA
Figure 5: Low-res ViT features (14 14) from the COCO-Stuff validation upsampled by 16x. Bilinear resize-conv baselines produce blurry outputs. Larger inputs and smaller transformer strides can help, introduce noise or blur and are bound by time and memory constraints (We can only compute 8x upsamplings for these methods, see Figure [7). Our FeatUp methods preserve semantics of the low-res features and recover lost spatial information from the high-res input image.
Where ||-|| is the standard squared l2 norm and s (+)) spatially-varying adaptive uncertainty (Hamilton et al. (2020) parameterized by small linear network W. This turns the MSE loss into proper likelihood capable handling uncertainty. This extra flexibility allows network to learn when certain outlier features fundamentally cannot be upsampled. In the supplement, we show this adaptive uncertainty' 's effectiveness in an ablation study and visualization.
3.1 CHOOSING A DOWNSAMPLER
Our next architectural choice is the learned downsampler σg. We introduce two options: fast and simple learned blur kernel, and a more flexible attention-based downsampler. Both proposed modules do change the "space" or "semantics" of the features with nontrivial transformations, but rather only interpolate features within small neighborhood. We diagram both choices in Figure 3 and demonstrate the effectiveness the attention downsampler in Figure of the Supplement.
Our simple downsampler blurs the features with learned blur kernel and can be implemented as a convolution applied independently to each channel. The learned kernel is normalized to be non-negative and sum to to ensure the features remain in the same space.
Though this blur-based downsampler efficient, cannot capture dynamic receptive fields, object salience, other nonlinear effects. To this end, we also introduce more flexible attention downsam- pler that spatially adapts the downsampling kernel. short, this component uses Ixl convolution to predict saliency map from the high-resolution features. It combines this saliency map with learned spatially-invarian weight and bias kernels and normalizes the result to create a spatially-varying blur kernel that interpolates the features. More formally:
o, (Fir)ss softmax(w Conv(Fr,, b) [Sayl
(2)
Where 0. (F))j is the i, j. component of the resulting feature map and Far [Rig]] refers patch of high resolution features corresponding to the i,j location in the downsampled features. @ and refer the elementwise and inner products respectively. and and b are learned weight and bias kernels shared across all patches. Our main hyperparameter for both downsamplers is the kernel size, which should be larger for models with larger receptive fields such as convolutional nets. We defer discussion of model-specific hyperparameters to the Supplement.
3.2 CHOOSING AN UPSAMPLER
central choice in our architecture the parameterization of o₁. We introduce two variants: "JBU" FeatUp parameterizes a with guided upsampler based stack of Joint Bilateral Upsamplers
5

---

Published as a conference paper at ICLR 2024
DINO
DINOv2
VIT
CLIP
ResNet-50
DeepLab-V3
Image
Original
FeatUp
Figure 6: FealUp can upsample the features any backbone, even convnets with aggressive nonlinear pooling.
(JBU) (Kopf et al. 2007). This architecture learns upsampling strategy that generalizes across corpus of images. The second method, "Implicit" FeatUp, uses an implicit network to parameterize at and can yield remarkably crisp features when overfit to single image. Both methods are trained using the same broader architecture and loss. We illustrate both strategies in Figure
Joint Bilateral Upsampler: upsamplers (JBU) (Kopf et al. 20077:
Our feedforward upsampler uses stack of parameterized joint bilateral
Fhr (JBU(,, (B))
(f())
where is function composition, ((x) is the low-resolution feature map, and x is the original image. This architecture fast, directly incorporates high-frequency details from the input image into the upsampling process, and is independent of the architecture of Our formulation generalizes the original JBU (Kopf et al. 2007) implementation to high-dimensional signals and makes this operation learnable. In joint bilateral upsampling we low-resolution features Fir. We let Ω be a neighborhood of each pixel the guidance. practice, use x 3 square centered at each pixel. Let k(.)) be similarity kernel that measures how "close" two vectors are. We can then form our joint bilateral filter:
high-resolttion signal, G, as guidance for the
Firli
1
Σ (o
bl Krange (Gli, j], G(a,b)) Kspatial (i: [i, [a,
(4)
Z
where Z is normalization factor to ensure the kernel sums Here, ksppaial is learnable Gaussian kernel on the Euclidean distance between coordinate vectors of width spatial:
2112
Kspotiol (I, y) = exp
(5)
20
spatial
Furthermore, Krange is cemperature-weightee softmax (Hamilton al. 2020) applied to the inner products from a multi-layer perceptron (MLP) that operates on the guidance signal G:
(LLLLP (LML((((..) MLPPGG(a
krange (I, 4) = softmax (a,
(6)
Grang
where of acts as the temperature. We note that the original JBU uses a fixed Gaussian kernel on the guidance signal, G. Our generalization performs much better as the MLP can be learned from data to create a better upsampler. In our experiments, we use a two-layer GeLU (Hendrycks & Gimpel| 2016) MLP with 30-dimensional hidden and output vectors. To evaluate [ar the original JBU formulation and use allinear-interpolate features if the guidance pixel does not directly align with a low-resolution feature. For resolution independence, we use coordinate distances normalized to [-1, 1] in the spatial kernel.
follow
One challenge we faced was the poor speed and memory performance of existing JBU implementa- tions. This could explain why this simple approach is not used more widely. To this end, we contribute an efficient CUDA implementation of the spatially adaptive kernel used in the JBU. Compared to a naive PyTorch implementation with the torch. nn. Unfold a operator, orders magnitude less memory and speeds inference by up to 10x. We demonstrate its significant performance improvements in of the supplement.
operation uses up to two

---

Published as a conference paper at ICLR 2024
CAM Score
Semantic Seg.
Depth Estimation
A.D. +
A.I.
Acc. f
mloU f
RMSE +
0 >> 1.25 f
Low-res
10,69
4.81
65.17
40.65
1.25
0.894
Bilinear
10.24
4.91
66.95
42.40
1.19
0,910
Resize-conv
11.02
4.95
67.72
42.95
1.14
0.917
DIP
10.57
5.16
63.78
39.86
1.19
0,907
Strided
11.48
4.97
64.44
40.54
2.62
0.900
Large image
13.66
3,95
58.98
36.44
2.33
0.896
CARAFE
10.24
4.96
67.1
42.39
1.09
0,920
SAPA
10.62
4.85
65.69
41.17
1.19
0.917
FeatUp (JBU)
9.83
5.24
68.77
43.41
1.09
0.938
FeatUp (Implicit)
8.84
5.60
71.58
47.37
1.04
0.927
Table 1: Comparison of feature upsamplers across metries CAM faithfulness, linear probe semantic segmentation, and linear probe depth estimation. Both FeatUp variants consistently outperform other approaches, including other forward-pass upsamplers (CARAFE, SAPA) and features optimized at inference-time (DIP).
Implicit Our second upsampler architecture draws direct analogy with NeRF by parametrizing the high-resolution features of a single image with an implicit function Fhr ML)(2). Several existing upsampling solutions also take this inference-tinge time training approach, including DIP (Ulyanov et al.) 2020) and LIIF (Chen et al. [021]. We use a small MLP to map image coordinates and intensities to high-dimensional feature for the given location. We follow the guidance of prior works (Mildenhall) ct al. 2020 Sitzmann et al. 2020a Tancik et al. 2020) and use Fourier features to improve the spatial resolution of our implicit representations. In addition standard Fourier positional features, show that adding Fourier color features allows the network to use figh-freyuency color information from the original image. This significantly speeds convergence and enables graceful use of high-resolution image information without techniques like Conditional Random Fields (CRFs). We illustrate the profound effect Fourier color features Section 6.44 of the Supplement.
More formally, let h(2, ()) represent the component-wise discrete Fourier transform of an input signal 2, with a vector of frequencies W. Let e and ey represent the two-dimensional pixel coordinate fields ranging in the interval [, Let represent concatenation along the channel dimension. We can now express our high-resolution feature map as:
Fhr MLP(h(e By xI)
2011 network with dropout (Srivastava al. (p = .1) and layer normalization (Ba et al.
Our MLP small 3-layer ReLU GGlorot et al.
2016). We note that, at test time, we can query the pixel coordinate field to yield features Fhr at any resolution. The number of parameters in our implicit representation is over two orders of magnitude smaller than a (224 x 224) explicit representation while being more expressive, significantly reducing convergence time and storage size.
3.3 ADDITIONAL METHOD DETAILS
Accelerated Training with Feature Compression To reduce the memory footprint and further speed up the training of FeatUp's implicit network, we first compress the spatially-varyiine features to their top & 128 principal components. This operation is approximately lossless the top 128 components explain ~ 96% the variance across single image's features. This improves training time by a factor of 60x for ResNet-50, reduces the memory footprint, enables larger batches, and does not have any observable effect on learned feature quality. When training the JBU upsampler, we sample random projection matrices in each batch to avoid computing PCA in the inner loop. This achieves the same effect thanks to the Johnson Linioesssnnnnnssusss lemma (Johnson et al.) 1986).
Total Variation Prior
To avoid spurious noise in the high resolution features, add small (Ato = 0.05) total variation smoothness prior (Rudin et al. 1992) on the implicit feature magnitudes:
Lev Σ (((1 FJJ
1 + 1. (1
1]1)²)
(8)
This is faster than regularizing full features and avoids overprescribing how the individual components should organize. We do not use this in the JBU upsampler because it does not suffer from overfitting. We demonstrate the importance of this regularizer in Section 6.4 in the supplement.
7

---

Published as a conference paper at ICLR 2024
Feutlup (JBu)
Feadup (Implic))
Ground Truth
Image
Low-Ris
Biinnar
Resize-conv
Strided
Large image
DIP
CARAFE
SAPA
twints ta
CAM
Sam Sag-
Protie
Dapat
Profue
Figure 7: A comparison of different upsampling methods across each of the tasks considered in our analysis. FeatUp achieves significant improvements in resolution across each task.
4 EXPERIMENTS
We compare our method against several key upsampling baselines from the literature, in particular: Bilinear upsampling, Resize-conv, Strided (i.e. reducing the stride of the backbone's 's patch extractor), Large Image (i.e. using larger input image). CARAFE (Wang et al., 2019). SAPA (Lu al. 2022c). and FADE (Lu et al., 2022b). We upsample ViT (Dosovitskiy et al.) 2020) features by 16x (to the resolution the input image) with every method except the strided and aarge-image baselines, which are computationally infeasible above 8x upsampling. For additional details on the strided implementation, please refer to Section 6.2 of the Supplement.
4.1 QUALITATIVE COMPARISONS
Visualizing upsampling methods Figure demonstrates the dramatic qualitative improvement FeatUp achieves compared to several baselines. Our visualizations fit a 3-dimensional PCA on each image's low-resolution ViT features and use this PCA to map upsampled features into the same RGB space. We also show that this high-fit seh-fidelity upsampling extends to higher PCA components in Figure 12 and that FeatUp improve small object retrieval in Figure 15 in the Supplement.
Robustness across vision backbones Figure demoostttttess that FeatUp can upsample a variety of modern vision backbones. In particular, show the implicit FeatUp features across variety of backbones spanning transformers, convolutional nets, and both supervised and sulf-supervised models. Even though backbones like ResNet-50 do not precisely localize objects due to their large receptive fields, FeatUp can reasonably associate features to the correct object.
4.2 TRANSFER LEARNING FOR SEMANTIC SEGMENTATION AND DEPTH ESTIMATION
Next, we demonstrate that FeatUp can serve as drop-in replacement for existing features in downstream applications. To demonstrate this, we adopt the widely used experimental procedure of using linear probe transfer learning to evaluate representation quality. More specifically, train linear probes on top of low-resolution features for both semantic segmentation and depth estimation. We then freeze and apply these probes upsampled features measure performance improvement. features are valid drop-in improvements, existing probes should work well without adaptation. For use frozen pre-trained VIT-S/I6 the featurizer, upsample the features (14x 14 + 224x224), and extract maps by applying a linear layer on the features.
all experiments,
follow the experimental setting of both (Alain & Bengio 2016 Hamilton et al.) 2022) and train a linear projection to predict the coarse classes of the COCO- Stuff (27 classes) training dataset using cross-entropy loss. We report mloU and accuracy the validation set in Table I For depth prediction we train on pseudo-labels from the MiDaS (DPT- Hybrid) (Ranftl et al., 2020) depth estimation network using their scale- and shift-invariant MSE, We report root mean square error (RMSE) and the ō > 1.25 metric which is common in monocular depth estimation literature. More specifically this metrie is defined as the percentage of pixels with
For semantic segmentation,
max y y 1.25 where is the depth prediction and y is the ground truth.
We stress that these linear probe evaluations show that FeatUp features can improve downstream tasks without re- re-training models. These analyses do not aim to create SOTA segmentation or depth networks. Both FeatUp variants outperform all basclines across all experiments, showing that either variant can be used drop-in replacement for existing features. Qualitatively, Figure and Figures 19 20 in the supplement show cleaner, more cohesive predictions across both tasks.
8

---

Published as a conference paper at ICLR 2024
Resize- conv
PeatUp
Metric
Bilinear
IndexNet
A2U
CARAFE
SAPA
FADE
(JBU)
mIoU
39.7
41.1
41.5
41.5
42.4
41.6
43.6
44.2
mAce
51.6
51.9
52.2
52.3
53.2
55.3
54.8
55.8
aAcc
78.7
79.8
80.2
79.9
80.1
79.8
80.7
80.7
Params (M)
13.7
+3.54
+12.6
+0.12
+0.78
+0.20
+0.29
+0.16
GFLOPs
16.0
+34.40
+30.90
+0.51
+1.66
+1.15
+2.95
+1.70
Table 2: Semantic segmentation results with the Segformer Xie et al. 2022
architecture trained on the ADE20k train set and evaluated on the val set. FeatUp (JBU) outperforms the standard bilinear and resize-conv upsamplers in U-Net architectures, IndexNet [u al. (2022a). Dai et al. (2020). and other task-agnostic upsamplers
(CARAFE Wang al. (2019), SAPA Lu al. (2022c). FADE Lu et al. (2022b)). Additionally, our upsampler
competitive in parameter and floating-pon operation count.
4.3 CLASS ACTIVATION MAP QUALITY
Attributing model's predictions specific pixels is crucial for diagnosing failures and understanding a model's behavior. Unfortunately, common interpretation methods like Class Activation Maps (CAM) are limited by the low res of the deep feature maps and cannot resolve small objects. We show that FeatUp features can be dropped into existing CAM analyses to yield stronger and more precise explanations. More specifically, we use the literature's ' established metrics, Average Drop (A.D.) and Average Increase (A.I.), that measure CAM quality (refer to Section 6. in the Supplement for a detailed description of these metrics). Intuitively, A.D. and A.I. capture how much an image's 's most salient region changes the classification output. A good CAM should highlight regions with the greatest effect on the classifier's predictions, so censoring these regions will have the largest impact the model's predictions (lower A.D., higher A.I.) Upsamplers are trained the ImageNet training set for 2,000 steps, and we compute metrics across 2,000 random images from the validation set. We use a frozen pre-trained V as the featurizer, and extract CAMs by applying linear classifier after max-pooling. Upsampling is done (14x14 224x224) on the features themselves, and CAMs are obtained from these high-resolution maps. We report results in Table and Figures
4.4 END-TO-ENI SEMANTIC SEGMENTATION
FeatUp not only improves the resolution of pre-trained features but can also improve models learned end-to-end. We adopt the experimental setting of (Lu et al. 2022c b) to show that our JBU upsampler improves end-to-end performance on ADE20K semantic segmentation using the Segformer (Xie et al., 2021) architecture. Specifically, we train SegFormer on ADE20k /hhou et a2.19 2017) (20,210 training and 2,000 val) for 160k steps. To validate that our setup matches that of existing literature despite numerical discrepancies,
also compute FLOPs for SegFormer with various upsamplers in Table 2 These counts are comparable with those Liu et al. (2023), confirming our architectural setup. We report mean IoU, mean class accuracy (mAcc), and all-pixel accuracy (aAce) against several recent baselines in Table 2 including IndexNet (Lu et al.) 2022a), A2U (Dai et al. 2021), CARAFE (Wang et al., 2019), SAPA (Lu al. (022c), and DADE (Lu et 2022b) in addition to more standard bilinear and resize-conv operators. Figure 21 in the Supplement shows examples of segmentation predictions across these methods, FeatUp consistently outperforms baselines with fewer added parameters, showing that FeatUp can also improve a broader, jointly trained architecture.
5 CONCLUSION
We present FeatUp, novel approach to upsample deep features using multiview consistency. Fea- tUp solves critical problem in computer vision: deep models learn high quality features but at prohibitively low spatial resolutions. Our JBU-based upsampler imposes strong spatial priors to accurately recover lost spatial information with a fast feedforward network based on novel general- ization of Joint Bilateral Upsampling. Our implicit FeatUp can learn high quality features at arbitrary resolutions. Both variants dramatically outperform wide range of baselines across linear probe transfer learning, model interpretability, and end-to-end semantic segmentation.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We would like to thank the Microsoft Research Grand Central Resources team for their gracious help performing the experiments in this work. Special thanks to Oleg Losinets and Lifeng Li for their consistent, gracious, and timely help, debugging, and expertise. Without them, none of the experiments could have been run.
This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. 202 1323067. Any opinion, findings, and conclusions or recommenda- tions expressed in this material are those of the authors(s) and do not necessarily reflect the views of the National Science Foundation. This research is based upon work supported in part by the Office of the Director of National Intelligence (Intelligence Advanced Research Projects Activity) via 2021-2011 1000066. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the S Government. The US Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. This work is supported by the National Science Foundation under Cooperative Agreement PHY-201977 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://iaifi.org Research was sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2. 1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed implied, of the United States Air Force or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
REFERENCES
Jiwoon Ahn, Sunghyun Cho, and Suha Kwak. Weakly supervised learning of instance segmentation with inter-pixel relations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2209-22288 2019.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2016. UtL/tttss::/rrxxw.c333 hacg///b///// 610. 01644
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors, 2021. TRL/pttss////xxiv wttp://///.../.......3.1
Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06444 2016.
A. Buades, B. Coll, and J.-M. Morel. A non-local algorithm for image denoising. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) volume 2, pp. 60-65 vol. 2, 2005. doi: 10. 10.1100//VRR.2000.
Laurent Caraffa, Jean-Philippe Tarel, and Pierre Charbonnier. The guided bilateral filter: When the joint/cross bilateral filter becomes robust. IEEE Transactions on Image Processing, 24(4): 1199-1208, 2015. doi: 10. 10.110999122201522288111
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 9650-9660, October 2021.
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8628 8638, 2021.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 5939-5948, 2019.
10

---

Published as a conference paper at ICLR 2024
Hyesong Choi, Hunsang Lee, Sunkyung Kim, Sunok Kim, Seungryong Kim, Kwanghoon Sohn, and Dongbo Min. Adaptive confidence thresholding for monocular depth estimation, 2020. URL
httpss:/aarxi org//bs///aaoooooooo.oooooo httrg/ab//2009.. 200000289900
Jaehoon Choi, Dongki Jung, Yonghan Lee, Deokhwa Kim, Dinesh Manocha, and Donghwan Lee. Selfdeco: Self-supervised monocular depth completion in challenging indoor environments. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 467-474, 2021. doi: 10.1
.956081..
Yutong Dai, Hao Lu, and Chunhua Shen. Learning affinity-ware upsampling for deep image matting, 2020.
Yutong Dai, Hao Lu, and Chunhua Shen. Learning affinity-aware upsampling for deep image matting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2021.
N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'OS) volume 1, pp. 886-893 vol. 2005. doi: 10.1 10.11099CVPR.2005.11
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.048805 2018.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks, 2015. | vittps://arxe org
00092
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
Vincent Dumoulin and Francesco Visin. A guide to convolution arithmetic for deep learning, 2016a. URL petttss:/aaxxxr wttpxxvw.or//a///1333333 07285
Vincent Dumoulin and Francesco Visin. A guide to convolution arithmetic for deep learning. arXiv preprint 1eXiv:/603.072800 2016b.
William Freeman and Antonio Torralba. Shape recipes: Scene representations that refer to the image. Advances in Neural Information Processing Systems, 15, 2002.
Jun Fu, Jing Liu, Yong Li, Yongjun Bao, Weipeng Yan, Zhiwei Fang, and Hanqing Lu. Con- textual deconvolution network for semantic segmentation. Patter Recognition, 101:107152. 2020. ISSN 0031-3203. doi: http:://ob.agg///oooooiiigtgggg2 www. www.sciencediree..
URL https
com/ wwwp://www.....cc.eeeeeeeeeeeeee
Raghudeep Gadde, Varun Jampani, Martin Kiefel, Daniel Kappler, and Peter V. Gehler. Superpixel convolutional networks using bilateral inceptions, 2015. URI.
1511 06739
Raghudeep Gadde, Varun Jampani, Martin Kiefel, Daniel Kappler, and Peter V Gehler. Superpixel convolutional networks using bilateral inceptions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pp. 597- 613. Springer, 2016.
Jon Gauthier. Conditional generative adversarial nets for convolutional face generation. 2015.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 315-323. JMLR Workshop and Conference Proceedings, 2011.
Vitor Guizilini, Rui Hou, Jie Li, Rares Ambrus, and Adrien Gaidon. Semantically-guider representa- tion learning for self-supervised monocular depth, 2020. URL httpp:://axxi..ot wwtp:iiiiivoggs///ooooo 2002. 12319
11

---

Published as a conference paper at ICLR 2024
Mark Hamilton, Evan Shelhamer, and William T Freeman. is likely that your loss should be likelihood. arXiv preprint arXiv:2007.06059, 2020.
Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Unsupervised semantic segmentation by distilling feature correspondences. arXiv preprint arXiv:2203.08414, 2022.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL https: fttt:s://arxiv..rg/aaatte
03385
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning, 2019. URL
911
05722
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprini arXiv: 160.004666 2016.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units, 2021.
07447
Hanzhe Hu, Yinbo Chen, Jiarui Xu, Shubhankar Borse, Hong Cai, Fatih Porikli, and Xiaolong Wang. Learning implicit feature alignment function for semantic segmentation, 2022.
Huimin Huang, Lanfen Lin, Ruofeng Tong, Hongjie Hu, Qiaowei Zhang, Yutaro Iwamoto, Xianhua Han, Yen-Wet Wei Chen, and Jian Wu. Unet 3+: A full-scale connected unet for medical image segmentation. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1055-1059, 2020. doi: 00.100/CCASSP0077722000000000000
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 694-711, Cham, 2016. Springer International Publishing.
William Johnson, Joram Lindenstrauss, and Gideon Schechtman. Extensions of lipschitz maps into banach spaces. Israel Journal of Mathematics, 54(2):129-15 1986.
Robert Keys. Cubic convolution interpolation for digital image processing. IEEE transactions on acousties, speech, and signal processing. 29(6):1 155--110,, 1981.
Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. arXiv preprint arXivv2205. 2022.
Johannes Kopf, Michael F. Cohen, Dani Lischinski, and Matt Uyttendaele. Joint bilateral upsampling. ACM Trans. Graph., 26(3):96-ss, jul 2007. ISSN 0730-0301. doi: 10.1145/1276377.2276997 doi
URL |/L. 12764749711767
Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming- Yang. Deep laplacian pyramid networks for fast and accurate super-resolution, 2017. URL
1704.0391
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6681-4690, 2017.
Jeong Ryong Lee, Sewon Kim, Inyong Park, Taejoon Eo, and Dosik Hwang. Relevance- Your model already knows where to look. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14944-14953, 2021.
Xiaomeng Li, Hao Chen, Xiaojuan Qi, Qi Dou, Chi-Wing Fu, and Pheng-Ann Heng. H-denseunet: Hybrid densely connected unet for liver and tumor segmentation from ct volumes. IEEE Transac- fions Medical Imaging, 37(12):2663-22 2018. doi: 10.1 109/TMI.20 2012 2018.28884 8.2849918.
12

---

Published as a conference paper at ICLR 2024
Ce Liu, Jenny Yuen, and Antonio Torralba. Sift flow: Dense correspondence across scenes and its applications. IEEE transactions on pattern analysis and machine intelligence, 33(5):978- 2010.
Wenze Liu, Hao Lu, Hongtao Fu, and Zhiguo Cao. Learning upsample by learning to sample, 2023.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
G LoweDavid. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004.
Hao Lu, Yutong Dai, Chunhua Shen, and Songcen Xu. Index networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1):242 255, 2022a. doi: 10.1 10.119//PAMM.2220:000444
Hao Lu, Wenze Liu, Hongtao Fu, and Zhiguo Cao. Fade: Fusing the assets of decoder and encoder for task-agnostic upsampling. In Proc. European Conference on Computer Vision (ECCV), 2022b.
Hao Lu, Wenze Liu, Zixuan Ye, Hongtao Fu, Yuliang Liu, and Zhiguo Cao. Sapa: Similarity awwre point affiliation for feature upsampling. In Proc. Annual Conference on Neural Information Processing Systems (NeurIPS), 2022c.
Zhisheng Lu, Juncheng Li, Hong Liu, Chaoyan Huang, Linlin Zhang, and Tieyong Zeng. Transformer for single image super-resolution In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 457-466, June 2022d.
Davide Mazzini. Guided upsampling network for real-time semantic segmentation, 2018. URL
https: pettps:/arriin otl. http://bb/////// 1801181108000 007466
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. arXiv preprint arXiv:1301.3700 2013.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis, 2020. URL
Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for semantic segmentation. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1520-1528, 2015.
Augustus Odena, Vincent Dumoulin, and Chris Olah. Deconvolution and checkerboard arti- facts. Distill, 2016. doi: 0022391/ddistil.000000 URL http::/ httpa/issiil..ppp/..1./p pwwpp/lb///////
(econv-checkerbardd
Tim Prangemeier, Christoph Reich, and Heinz Koeppl. Attention-based transformers for instance segmentation of cells in microstructures, In 2020 IEEE International Conference on Bioinformatics 2200:04449991.222911333333
and Biomedicine (BIBM), pp. 700-707, 2020. doi:
Shengju Qian, Hao Shao, Yi Zhu, Mu Li, and Jiaya Jia. Blending anti-aliasing into vision trans- former. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, volume 34, pp. 5416 5416-5429. Curran As- sociates, Inc., 2021, BeLL:attpps:/pprceediii wtwp:/owwooeenooooe...ee.ee..ee. cc httppppee/222///1eee/
hm3bof3eee2475e0366e2 10tea11110eeeaaaaa PPaerr. pd
Zhenyue Qin, Dongwoo Kim, and Tom Gedeon. Rethinking softmax with cross-entropy: Neural network classifier mutual information estimator. arXiv preprint arXiv:1911.1662 2019.
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. 2018.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE fransactions on pattern analysis and machine intelligence, 2020.
13

---

Published as a conference paper at ICLR 2024
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2021. URL
abs 10752
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networss for biomedical image segmentation. CoRR, abs/1505. 00599, 2015, 5a0.1049954
04597
Leonid Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, 60(1-4):259-268, 1992.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition, 2019. /tlL:ttt////xxx.oor////bbb/ .9886
05862
Ling Shao, Fan Zhu, and Xuelong Li. Transfer learning for visual categorization: A survey, IEEE transactions on neural networks and learning systems, 16(5):101--1034, 2014,
Wenzhe Shi, Jose Caballero, Lucas Theis, Ferenc Huszar, Andrew Aitken, Christian Ledig, and
Zehan Wang. Is the deconvolution layer the same as a convolutional layer?, 2016. https
07009
Assaf Shocher, Nadav Cohen, and Michal Irani. Zero-shot super-resolution using deep internal learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 31 188 3126, 2018. doi: 1..1009/CVR..201..000
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions, 2020a. URL https
//arxiv. abs www.aaa//0 200000000000000000 09661
Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NeurIPS, 2020b.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15())11999-1988 2014.
Hang Su, Varun Jampani, Deqing Sun, Orazio Gallo, Erik G. Learned-Miller, and Jan Kautz. Pixel- adaptive convolutional neural networks. CoRR, abws//904.053777 2019. URL aal.tttp: http:htt::///ett //vxxiv org htch/bb//1ie 1904. 05373
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Kridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains, 2020. URL https: http://axxvvvooooo
abs/2006, 10739
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision, pp. 402-419. Springer, 2020.
C. Tomasi and R. Manduchi. Bilateral filtering for gray and color images. In Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271,, pp. 839-846. 1998. doi: 10.1 10.1109/ ICCV.1998.77 108155.
Tong Tong, Gen Li, Xiejie Liu, and Qinquan Gao. Image super-resolution using dense skip connec- tions. In Proceedings of the IEEE international conference on computer vision, pp. 4799 4807, 2017.
Narek Tumanyan, Omer Bar- Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer, 2022.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. International Journal of Computer Vision, 128(7):1867- 1888, mar 2020. doi: 10. 10.10007/1112--02000001 https://do1 www.oii.rgg11..0
URL
14

---

Published as a conference paper at ICLR 2024
Jiaqi Wang. Kai Chen, Rui Xu, Ziwei Liu, Chen Change Loy, and Dahua Lin. Carafe: Content-awarc reassembly of features. 2019. doi: 1048550/RRXXV.905.22155 URL https: httph:httssaatt. /arxiv org/abs/ 1905. 02188
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks, 2017.
URL htRttt/s////xxivo.o
(os/1111.07971
Yude Wang, Jie Zhang. Meina Kan, Shiguang Shan, and Xilin Chen. Supe-supe equivariant at- tention mechanism for weakly supervised semantic segmentation. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12275-12284, 2020.
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3(1):1-40, 2016.
Huikai Wu, Shuai Zheng, Junge Zhang, and Kaiqi Huang. Fast end-to-end trainable guided filter, 2019.
Chunxia Xiao and Jiajia Gan. Fast image dehazing using guided joint bilateral filter. Vis. Comput., 28(6-8):713-221 721 jun 2012. ISSN 0178-2789. doi: 10. 10.1007/600071-012-0099999 URL https 0od.l..org/1...100/0000003333110 1007
Enze Xie. Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34: 14777777-00 2021.
Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michae, Frahm (eds.), Computer Vision ECCV 2020, pp. 1-19, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58604-1
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):302-321, 2019.
15

---

Published as a conference paper at ICLR 2024
6 SUPPLEMENTAL INFORMATION
6.1 WEBSITE, VIDEO, AND CODE
We provide additional details and short video explaining FeatUp at aka.ms/icatup Additionally, we provide our code at: https//tnnyur.com//88yyyppa
6.2 STRIDED BASELINE IMPLEMENTATION
For the DINO and VIT backbones, we extract patches with a stride of upsample factor to produce a higher density of feature vectors and thus increase feature resolution. We point out that the upsampling factor is limited with this method (as the stride is lower bounded by 1), so this approach can only upsample up to 16x for ViT-S/16. Practically however, these maximum upsampling factors are impractical as they require far more memory than current GPUs provide (see Figure
16
6.3 COMPARISON TO IMAGE- -AGGEUUPSAPPING METHODS
variety of methods have been proposed for image super-resolution. Among the learning-hase based approaches, deep image prior (DIP) (Ulyanov al. 2020) has been used succesfully for enhancing images without additional training data. Figure shows that DIP poorly upsamples features, intro- ducing artifacts and "blob" patterns in the features and downstream outputs. (Shocher et al. 2018) introduced Zero-Shot Resper-esolttion, a method that learns without additional training data, Additionally, images can be represented Local Implicit Image Functions (LIIF) (Chen et al.] 2021 which can be queried at arbitrary resolution. While similar to FeatUp's implicit network, LIIF trained continuously represent a feature map does not produce sharp outputs like FeatUp (Figure 8) Despite these methods' successes in the image super-resolution problem space, they are not equipped to upsample high-dimensional features.
image-specific CNN at test time
Input Image
Low-Ret Res
Featlip
Deep image Prior
ZSSR
LIFF
Figure 8: Comparison of image super-resolution methods using Deep Image Prior, Zero-Shot (7SSR), and Local Implicit Image Function (I.IF). We also include visualization Implicit Feature Alignment (IFA). As shown
whole feature map and zoomed-in section, thse image upsampling methods do not effectively upsample the low-resolution and high high-dimensional feature maps by the large upsampling factors that we are able to handle.
16

---

Published as a conference paper at ICLR 2024
6.4 ABLATION STUDIES
We show the effects of each design decision for FeatUp in Figure 9 features without the uncertainty loss, possibly because it cannot ignore certain nonlinear artifacts resolve the large pooling window present in ResNet-50. The magnitude regularizer provides smoothing and regularization benefits. Our choice to include Fourier color features dramatically improves resolution and high-frequency details. Finally, the attention downsampler helps the system avoid odd edge and halo effects by learning kernels more focused on salient parts of the signal. Using an explicit buffer of features instead of an implicit network yields significant artifacts, though we note that the artifacts are significantly less dramatic if the simple downsampler is also used.
Our upsampler blurs ResNet
We also provide an ablation study of the total variation and magnitude regularizers in Figure Our regularizer is fairly robust to different settings as shown by the 2x multiplication for both terms in the 3rd column. However, there still exists an optimal À range that provide important smoothing properties; larger values can interfere with the main reconstruction objective as shown in the final column.
Current Settings
Image
Low Res DINO
Feat Up
Uncertainty
Mag TV
Color Feats
Simple Down.
Explicit Features
Image
Law Res RN50
Feat Up
Uncertainty
Mag TV
Color Feats
Simple Down.
Explicit Features
Figure Qualitative ablation study across both DINO and Resnet50 Backbones. The biggest improvements arise from the implicit featurizer, color features, and the magnitude TVV regularizer.
Current Settings
Max Pad 0
Max Pad
Max Pad 60
Max Pad = 120
Zoom 1.0
Zoom 1.8
Zoom 2.5
Max Zoom = 3.0
Flips False
Flips True
Figure 10: Ablation of FeatUp's 's training Syper-parameters We are robust to a range of jitter values, though features degrade with large changes in max pad.
17

---

Published as a conference paper at ICLR 2024
Current Settings
Aep = 0.0
A 0.05
Aev = 0.1
Ap = 1.0
Amay 0.0
Amay = 0.001
Amay = 0.002
mma 0.01
Figure 11: Qualitative ablation study of the TV and magnitude regularizers. FeatUp is fairly robust the settig of these parameters.
To further justify design decisions in the context of end-to-end trained architecture, evaluate JBU with the Segformer Xie et al. (2021) decoder by 1) removing the MLP (denoted as MLP in Equation ( the guidance signal, 2) removing the temperature-weigher softmax and replacing it with Euclidean distance between the central feature and its neighborhood, and 3) removing the softmax and replacing with cosine distance. Each ablation degrades segmentation performance, with the MLP exclusion being the most detrimental.
FeatUp (JBU)
Softmax + Euclidean Dist.
Softmax Cosine Dist.
Original
MLP
mloU
44.2
42.9
43.8
43.7
mAce
55.8
54.7
54.5
55.3
aAcc
80.7
79.4
80.0
80.4
Table 3: Semantic segmentation performance with the Segformer architecture trained on the ADE20k train set and evaluated the val set. Ablated FeatUp (JBU) replaces the original feature upsampling in the Segformer decoder.
CAM Score
Semantic Seg.
Depth Estimation
Ablation
A.D. +
A.I. f
Ace. f
mloU f
RMSE +
>1.25 1
Original
9.83
5.24
68.77
43.41
1.09
0.938
MLP
10.04
5.10
68.12
42.99
1.14
0.917
Softmax
9.98
5.19
68.68
43.16
1.10
0.928
Euclidean
Soffmax
9.97
5.21
68.49
43.15
1.12
0.924
+ Cosine
Table 4: FeatUp (JBU) performance with ablated architectural components: removing the MLP, replacing softmax with gaussian kernel Euclidean cosine distance. Across all metries, each ablation degrades performance.
18

---

Published as a conference paper at ICLR 2024
CAM Score
Semantic Seg.
Depth Estimation RMSE
Atm DS.
O.D.
TV Reg.
A.D. 4
A.I. 1
Acc. 1
mloU
á >
8.84
5.60
71.58
47.37
1.04
0.927
V
9.07
5.06
70.95
46.79
1.11
0,916
X
/
8.91
5.55
71.26
46.89
1.08
0,920
/
X
9.10
5.00
68.06
44.36
1.11
0.913
Table 5: Ablation study for implicit FeatUp features with varied downsampler (attention = /, simple X).
outlier detection, λrv (0.05=4 0000 X).
6.5
VISUALIZING ADDITIONAL PCA COMPONENTS
PCA 1-3
PCA 4-6
PCA 7-9
Image
Low Res
Feat Up
Image
Low Res
Feat Up
Figure 12: Visualizing higher PCA components with FeatUp. FeatUp upsamples entire feature maps, so their higher-order principal components also remain in the same space as the original features and upsampled precisely. Higher components tend to separate more fine- anee-grained object categories like the skater from the skateboard, and the trees from the background, and the clouds from the sky. Note that each subobject's features are upsampled precisely to the object it represents.
19

---

Published as a conference paper at ICLR 2024
6.6 SALIENCY MAP DETAILS
Downsampling in FeatUp is analogous to ray-marching in NeRF, which approximates the physics of image formation. FeatUp's downsampler approximates network's 's process of pooling information into features. As shown in Figure 6, most networks preserve the rough location of objects in their features (the objects just appear downsampled and blurred). This observation leads to blur/pooling operators.
The simplest of these is average pooling, but do better by generalizing this operation learned blur/pooling kernel so the downsampler can better match a network's 's receptive field size. To map back to NeRF, this is like adding learned camera lens distortion parameters the ray-marcher so NeRF can better fit the data.
As shown in Figure and described in Section 3.1, even a learned blur/pooling kernel cannot capture dynamic receptive fields or object salience. For example if a small amount of an important object is in transformer's patch, the whole feature changes. We capture effects like this by making the learned pool/blur kernel dependent on image content using 1x conv (we don't need anything bigger than this layer). This generalizes the previously-describn learned blur/pool and allows the downsampler to adaptively pool based on image content. Figure 13 shows that the salience network focuses certain attributes (e.g. object boundaries, some important small objects). We also note that many common pooling strategies such average pooling or nearest/bllinear/bieubb resizing special cases of our learnable attention pooling strategy.
6.7 VISUALIZING DOWNSAMPLER SALIENCE AND KERNELS
Image
Salience
b
Image
Salience
W
b
Figure 13: Visualization of downsampler salience and weight and bias kernels for two images. Note how fine-grained objects have higher salience and regions around important objects (like the sky between the hands and the skateboard) have lower salience. This allows the network to capture nonlinear behavior where embeddings from salient regions dominate the embeddings of other regions.
20

---

Published as a conference paper at ICLR 2024
6.8 VISUALIZING PREDICTED UNCERTAINTY
Downsampled
True Feats
Predicted Uncertainty
Figure 14: An example predicted uncertainty map for set of ViT features. White areas have higher uncertainty. this figure, we can see that nonlinear artifacts like the spurious pink tokens are marked with high uncertainty they change location depending on the given evaluation. These tokens might serve some other role in the network, such as class-token-lit information aggregation. We do not see these types of effects in DINO or convolutional networks.
6.9 IMPROVING IMAGE RETRIEVAL FOR SMALL OBJECTS
Query Point
Query Point (Detail)
Bilinear Retrieval
FeatUp Retrieval
Figure 15: High-resolution FeatUp features can be used to improve the retrieval of small objects and cluttered scenes. A query image (Left) is featurized with DINO and the region marked with red x used as a query point. We show the detailed placement of this query point in the second image from the left. In the two images the right, show the closest matching point in the target image (red x) and we also visualize the similarity heatmap (red means similarity, blue means dissimilarity). The second image from the right depicts the matching point and heatmap when using bilinear feature interpolation on the image and target. The image on the far right shows the results after upsampling with FeatUp prior computing the retrieval. Because the scene is cluttered, bilinear interpolation blurs object features together and the resulting query vector attends over both the ground and the traffic cones. FeatUp's 's features better align with objects allowing only the traffic cones to be retrieved.
21

---

Published as a conference paper at ICLR 2024
6.9.1 LINEAR PROBE DETAILS
In both linear probe tasks, one probe was trained on low-resolution (14x14) features from the COCO training set, and frozen for validation across all methods. FeatUp's performance improvements on this repurposed linear probe show that our methods increase resolution without compromising the original feature space. We highlight that these results are not meant to improve caae--o-thee-de art performance on segmentation and depth estimation; they are meant to showcase feature quality across upsamplers. Because prediction for both tasks is done with a frozen backbone and single trainable linear probe, the segmentation and depth maps are not meant as direct application.
6.10 AVERAGE DROP AND AVERAGE INCREASE DETAILS
Average Drop is expressed as mar(U.X,"-00) 100, where Y is the classifier's softmax output (i.e. Y
confidence) on sample for class c, and Of is the classifier's softmax output on the CAM-masked sample i for class We generate Of by keeping the top 50% of CAM values (and Gaussian blurring the remaining 50% of values with less explainability power). Though we generally expect classifiers to drop in confidence because even masking out less-salient pixels context, a high-quality CAM will target the explainable regions of an image more precisely and thus maintain higher confidence. In the reverse direction, we measure the Average Increase to capture the instances where CAM-masked inputs increase model confidence. Specifically, we define Average N
remove important image
Increase as kyrs 100 where <y is an indicator function equal to when Y, < Of that N
is, when model confidence increases upon classifying a CAM-masked image.
Similar to the RelevanceCAM evaluation in (Lee et al. 2021 we randomly select 2000 images from the ImageNet validation set (limited to images where the label and model prediction match) to measure A.D. and A.I. on.
22

---

Published as a conference paper at ICLR 2024
6.11 PERFORMANCE BENCHMARKING
See Table for performance benchmarking of our adaptive convolution CUDA kernel used in FeatUp (JBU).
Shape (B, H. W. F)
Method
Forward (ms)
Backward (ms)
Peak Mem (Mb)
Ours
0.15
1.05
6.24
x 14 14 2048 x
TorchScript
2455
69367
12.8
Unfold
3.30
2.81
119.
Ours
0.55
2.10
10.2
512 x 512 x
TorchSeript
147.
520.
24.3
Unfold
3.47
4.85
231.
Ours Unfold
8.43
90.8
372.
16 x 32 x 32 x 2048 x 5
118.
218.
6628.
Ours Unfold
17.7
114.
326.
32 X 512 X 512 x 3
36.0
104.
4901.
Ours Unfold
6.12
61.1
400.
64 14 14 2048 x
57.5
170.
5174.
Ours Unfold
6.27
36.1
128.
64 x 224 x 224 x 3 5
16.7
27.4
1878.
Ours Unfold
1.06
8.99
44.5
64 X 64 X 64 x 16 x 5
7.18
14.5
822.
Ours
2.00
8.36
52.6
x x 16 7
Unfold
10.8
25.6
1596.
Table 6: Comparing the performance of our CUDA JBU kernel with with implementations based on Py PoTrch'ss 's Unfold operation and TorchSeript. Our implementation dramatically reduces memory overhead and increases inference speed. Code for this operation available in the provided link.
Upsampling
Spetial
CARAPE (pukkd))
CANAEE lpaccet
GARAPE (guktod)
SAN
SAPA
SANA
FACE
GPLOPS
32
129e Feature Dimension
912
126
Upsampling Factor
Target Spatial Dimension
We evaluate how floating point operations scale with various factors. varying the upsampling factor, feature dimension, and target spatial dimension. FeatUp (JBU) remains competitive in GFLOP usage. each experiment, the attributes not studied are kept constant (upsampling factor = 2, feature dimension = 256, starting spatial dimension = 8x8).
Figure 16:
We analyze peak memory usage and inference time for various upsampling methods. Specifically, upsample ViT features from a (1 x 3 x 224 x 224) image (i.e. low-resolution feature dimensions of (1 x 384 x 14 x 14)) by factors of 2, 4, 8, and 16. Figure 17 shows that FeatUp (JBU)'s peak memory closely follows resize-conv and SAPA baselines and outperforms CARAFE. Additionally, FeatUp is as fast as baselines yet outperforms baselines in all quantitative evaluations. We note
23

---

Published as a conference paper at ICLR 2024
that strided and large image baselines become computationally infeasible after 8x upsampling, using a batch size 1.
Feature Upsampler Memory Analysis
Feature Upsampler Time Analysis
Peak memory usage (GB)
Upsampler
10a
Time elapsed (s)
Biliner
10
CARAFE
FeatUp (JBu)
10a
image
10-2
Resize-corv
10
Strided
10-3
10-2
2
16
Upsample factor
Upsample factor
Figure Analysis of peak memory usage (left) inference time (right) various forward-pas. upsamplers. FeatUp (JBU) competitive with SAPA and resize-conv across upsampling factors and is more efficient than CARAFE for smaller factors. The large image and strided approaches become infeasible at large upsampling factors only show metries for these methods up to 8x upsampling.
24

---

Published as a conference paper at ICLR 2024
6.12 ADDITIONAL QUALITATIVE RESULTS
We provide additional CAM visualizations with supervised ViT features the ImageNet val in Figure 18 As in the main paper, upsample features from 14x14 to 224x224 output before extracting CAMs (except for the "Low-Res" column. where the features are kept as-is). Both FeatUp (JBU)'s s bigge-preeerving bilateral filters and the FeatUp (Implicit)'s feature representation allow resulting CAMs to highlight salient regions more accurately. Our CAMs combine the semantic advantages of low-resolution features and the spatial advantages of large images, producing refined versions of the original CAMs without discontinuous patches present the other upsampling schemes.
FeatUp (Implicit
Image
Resize-conv
Large image
(UBU)
Aeericin a111g0ou
Toig teig-tormed brects
Figure 18: CAMs on the ImageNet validation set from supervised VIT backbone and linear probe classifier.
Both FeatUp variants produce features that more precise with respect the input image, allowing downstream CAMs to better align with object boundarics.
See Figure 19 for examples of linear probe transfer learning for semantic segmentation the COCO- Stuff dataset. The 14x 14 features output from a ViT backbone are upsampled with the following methods achieve 224x224 resolution. Then, linear probe is trained on the low-resolution features and frozen for evaluation on COCO-Stuff semantic class labels. Our methods recover more cohesive labels of objects and backgrounds.
Ground
Image
Large image
(JBU)
!implicit
Figure 19; Examples of linear probe transfer learning for semantic segmentation on the COCO-Stufe datasct. Our methods more closely resemble ground-trudt segmentation and smooth many of the artifacts present in the low-resolution features. Additionally, FeatUp (Implicit) recovers thin structures like the umbrella pole not even present in the ground truth despite being semantically correct.
Figure 20 provides additional examples of linear probe transfer learning for depth estimation. The 14x features output from a ViT backbone are upsampled to achieve 224x224 resolution. Then, a linear probe is trained directly the features to predict depth while supervised with small MiDaS network. Our results show that both FeatUp variants result in high-quality features capable of transfer learning.
25

---

Published as a conference paper at ICLR 2024
Image
Large image
(BUU
(Implicto
Figure 20: Examples linear probe transfer learning depth estimation. Our methods produce sharper
object boundaries and smoother interiors that more closely align with true depth than other methods.
Bilinear
Resize-com conv
CARAFE
SAPA
FADE
FeatUp (JBU)
Ground Truth
Figure 21: End-to-ent training performance different upsampling methods from our Segformer based semantic segmentation experiments. These results do not use linear probes, but instead train the architecture jointly.
26

---

Published as a conference paper at ICLR 2024
6.13 LIMITATIONS
FeatUp (Implicit)
FeatUp (JBU)
Image
Image
Figure 22: Left: Though FeatUp's implicit network can capture fine detail such as the soccer ball or window frame, it can still produce some halo effects (see soccer player). Additionally, because the method relics on the imput image's s spatial signal, certain patterns unrelated to object semantics can be transferred to the feature map (see rug patter), though this is a rare occurrence. Right: FeatUp's JBU network is not as sensitive to fine detail as the implicit network, instead capturing broader contours.
6.14 IMPLEMENTATION DETAILS
All backhones (DINO, DINOV2, VIT, ResNet-50, CLIP, and Deepl a3b3)) used to train FeatUp are frozen, pre-trained models obtained from the community. We outline the hyperparameters used to train FeatUp in table 7
Hyperparameter
FeatUp (Implicit)
FeatUp (JBU)
Num Images
Num Jitters Per Image
10
Downsampler
Attention NAdam
Attention
Optimizer
NAdam
Learning Rate
0.001
0.001
Image Load Size
224
224
Projection Dim
128
30
Training Steps
2000
2000
Max Transform Padding
30px
30px
Max Transform Zoom
1.8x
2x
Kernel Size
29
16
Total Variation Weight
0.05
0.0
Implicit Net Layers
3
n/a
Implicit Net Dropout
0.1
n/a
Implicit Net Activation
ReLU
n/a
Table 7: Hyperparameters used in training FeatUp.
27


Reviews:
[{"review_id": "ruTUZeUv1B", "reviewer": "Reviewer_2Fqt", "summary": "Deep features are usually in a low resolution due to the usage of pooling operations. This paper introduces FeatUp to restore the lost spatial information in deep features. FeatUp has two variants: one that guides features with high-resolution signal in a single forward pass, and one that fits an implicit model to a single image to reconstruct features at any resolution. Both approaches use a novel multi-view consistency loss with deep analogies to NeRFs. FeatUp outperforms other feature upsampling approaches in class activation map generation, semantic segmentation, and depth prediction.", "questions": "Many details are unclear in the paper. Please see the above weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This paper is well motivated and quite novel. The writing is professional and convincing. The improvement over existing approaches is solid and nontrivial. The proposed method can be used as a drop-in replacement for deep feature upsampling, which could be useful for this community if the claimed improvement can be easily obtained.", "weaknesses": "The training details of the FeatUp model when used as a drop-in replacement for existing features are missing. How do you train it? What data do you use for the training? What is the objective and loss for the training?\n\nThe experimental details in Table 1 and Table 2 are unclear, making it difficult to reproduce the results. Which networks/models and backbones are used for various downstream tasks? How do you train these models? Are all upsampling approaches evaluated under the same setting?\n\nThere are no numeric results for ablation study in this paper, and this is no ablation study in the main part. This paper has many designs and components (Eq. (1) – Eq. (8)), and it is important and necessary to evaluate each of these designs and components. Recently, ablation study is also a necessary part of computer vision papers, especially for deep learning papers. This is the biggest weakness of this paper.\n\nWill the code be released? This is not mentioned in the paper. If the claimed improvement can be easily obtained, the code and pretrained models would be valuable to this community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rllGN3ylqA", "reviewer": "Reviewer_2f29", "summary": "This paper introduces a method FeatUp that learns to upsample a low-resolution feature map at any-resolution. The proposed method is supervised by multiview consistency, and has two variants, one with a single forward and one fits an implicit model per image. The authors compare the method to several prior baselines for feature upsampling.", "questions": "Why a predicted salience map is needed in attention downsampler?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Upsampling deep feature maps of neural networks is an important research topic and has wide applications.\n2. The idea of connecting feature upsampling to implicit neural representations and using multiview consistency to supervise the upsampled results is interesting and intuitive.\n3. The effectiveness of the proposed method is well visualized in several figures.", "weaknesses": "1. A main goal of upsampling the feature map is to use it for tasks that requires semantic understanding and dense prediction, e.g. semantic segmentation. How does the proposed method work for more standard semantic segmentation benchmarks (for both mIoU and computation cost) and what are the main advantages of the proposed method? For example, in the context of any-resolution upsampling of feature map, Learning implicit feature alignment function for semantic segmentation (ECCV 2022) seems to be related, does the proposed method show advantages over the prior work?\n2. How does the computation cost (in FLOPS) compare to prior works in main tables? Since the proposed method is a new upsampler, it would better demonstrate the advantages of the proposed method by comparing the computation cost.\n3. In practice, why the reconstruction of features at any-resolution is important? Typically, the encoder takes a fixed-resolution input, the task is more likely to be fixed-resolution rather than any-resolution. If any-resolution is not the key point, can the proposed method improve over state-of-the-art semantic segmentation methods on standard benchmarks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XjeOQoiIHk", "reviewer": "Reviewer_mrZx", "summary": "This paper presents a model-agnostic framework for feature upsampling. The main contribution of the framework is a multi-view cycle consistency loss where low-res features are matched to its upsampled and downsampled ones. Two instantiations are presented. One is inspired by joint bilateral upsampling, and a fast CUDA implementation is provided. The other is inspired by NeRF where upsampled features are optimized per image. The authors have demonstrated good visualizations of the upsampled feature maps and show that the presented upsampling framework can preserve good semantic smoothness and boundary sharpness. The effectiveness of the framework is validated on a number of tasks.", "questions": "Please see the weaknesses for my major concerns. I have also some other suggestions:\n\nIs it appropriate to indicate augmented low-res features as different views? In 3D, different views often imply different view points.\n\nIn addition, the organization of related work does not make sense to me. \n- First, the paper does not conduct experiments on image super-resolution. This sub-section seems redundant. \n- Second, the references in general-purpose feature upsampling are weird. Both Amir et al. (2021) and Tumanyan et al. (2022) do not study feature upsampling at all. Why are the two papers cited? In my opinion, most references discussed in the section of image-adaptive feature upsampling should be discussed in this section. As far as I know, CARAFE is one of the first paper studying general-purpose feature upsampling. \n- In addition, some closely related approaches compared in experiments are not discussed/compared in related work such as FADE and SAPA.\n\nPlease address my major concerns in the rebuttal. I would re-evaluaute the paper conditioned on the response.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ Developing general-purpose feature upsampling operators is a recent emerging topic. The goal of designing model-agnostic framework is interesting and shows significantly better visualizations in terms of feature quality over prior art.\n+ As far as I can tell, viewing feature upsampling as a rendering problem from multiple low-res views is novel.", "weaknesses": "While I generally like the idea of the cycle-consistent training, I have a number of major concerns on the specific operators used/proposed and the results presented.\n- **The focus of the paper seems unclear**. Being a feature upsampling framework, the framework should be agnostic to the specific downsampling or upsampling operator chosen. Yet, the current approach leaves me an impression that the framework only works on the proposed learned downsamplers and upsamplers. If the framework matters, some additional existing downsampler-upsampler pairs should be demonstrated to demonstrate its generality; If the proposed operators matter, the contributions and the title may need to be revised.\n\n- Following my previous comment, I think the problem is that the authors attempt to combine many seemingly related techniques together. For instance, the JBU and its fast CUDA implementation look standalone to me. It is unclear to me what the role of the JBU is in the framework, particularly when it is claimed as a contribution. Please explain why JBU is essential to the FeatUp framework. In addition, applying JBU to feature upsampling is not new. \"Fast End to End Trainable Guided Filter\" CVPR18 and \"Superpixel Convolutional Networks using Bilateral Inceptions\" ECCV16 are two example works, which is also missing in the related work.\n\n- **The ablation study of the paper is poor**. It is unclear to me what exactly leads to the good feature quality. Aside from the loss, there are a number of tricky designs in the downsampler and upsampler used. For example, why a blur kernel must be used in downsampling? And why introducing the saliency map and how it works? Why a simple 1x1 convolution is sufficient to present the saliency map? Would it be precise enough? Also, the implicit FeatUp produces significantly better visualizations according to Fig. 5. Is it because of the per-image optimization. A thorough ablation study is required.\n\n- **The method may be justified in an inappropriate setting**. According to Table 1, FeatUp (Implicit) has reported significant performance improvements on the transfer learning setting. Yet, this setting is rarely used in feature upsampling literature. Why choosing this setting? I wonder how the parameters of comparing upsamplers are updated in this setting. In addition, the comparison between the implicit variant and other upsamplers seems unfair, because other upsamplers do not optimize the per-image upsampling quality. This also leads to the next issue.\n\n- **The practicality of the implicit FeatUp seems poor**. In reality, the per-image optimization of upsampling is unrealistic, it is not likely that the implicit variant is used as a plug-in in existing networks (the experimental setting in Table 2 also confirms my opinion). But I think this approach can somehow indicate an upper bound of upsampling quality.\n\n- The performance of baseline approaches does not align with the number reported in the corresponding papers and is significantly lower. For example, SAPA reports 44.4 mIoU but the table in Table 2 only reports 41.6. Please check.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)