LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision
  Foundation Models
Authors: name: Haiwen Huang, name: Anpei Chen, name: Volodymyr Havrylov, name: Andreas Geiger, name: Dan Zhang

Vision foundation models (VFMs) such as DINOv2 and CLIP have achieved
impressive results on various downstream tasks, but their limited feature
resolution hampers performance in applications requiring pixel-level
understanding. Feature upsampling offers a promising direction to address this
challenge. In this work, we identify two critical factors for enhancing feature
upsampling: the upsampler architecture and the training objective. For the
upsampler architecture, we introduce a coordinate-based cross-attention
transformer that integrates the high-resolution images with coordinates and
low-resolution VFM features to generate sharp, high-quality features. For the
training objective, we propose constructing high-resolution pseudo-groundtruth
features by leveraging class-agnostic masks and self-distillation. Our approach
effectively captures fine-grained details and adapts flexibly to various input
and feature resolutions. Through experiments, we demonstrate that our approach
significantly outperforms existing feature upsampling techniques across various
downstream tasks. Our code is released at https://github.com/andrehuang/loftup.