Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and
  Resolution
Authors: name: Mostafa Dehghani; name: Basil Mustafa; name: Josip Djolonga; name: Jonathan Heek; name: Matthias Minderer; name: Mathilde Caron; name: Andreas Steiner; name: Joan Puigcerver; name: Robert Geirhos; name: Ibrahim Alabdulmohsin; name: Avital Oliver; name: Piotr Padlewski; name: Alexey Gritsenko; name: Mario Lučić; name: Neil Houlsby
Published: 2023-07-12 17:01:03

Abstract: The ubiquitous and demonstrably suboptimal choice of resizing images to a
fixed resolution before processing them with computer vision models has not yet
been successfully challenged. However, models such as the Vision Transformer
(ViT) offer flexible sequence-based modeling, and hence varying input sequence
lengths. We take advantage of this with NaViT (Native Resolution ViT) which
uses sequence packing during training to process inputs of arbitrary
resolutions and aspect ratios. Alongside flexible model usage, we demonstrate
improved training efficiency for large-scale supervised and contrastive
image-text pretraining. NaViT can be efficiently transferred to standard tasks
such as image and video classification, object detection, and semantic
segmentation and leads to improved results on robustness and fairness
benchmarks. At inference time, the input resolution flexibility can be used to
smoothly navigate the test-time cost-performance trade-off. We believe that
NaViT marks a departure from the standard, CNN-designed, input and modelling
pipeline used by most computer vision models, and represents a promising
direction for ViTs.