Ferret-v2: An Improved Baseline for Referring and Grounding with Large
  Language Models
Authors: name: Haotian Zhang, name: Haoxuan You, name: Philipp Dufter, name: Bowen Zhang, name: Chen Chen, name: Hong-You Chen, name: Tsu-Jui Fu, name: William Yang Wang, name: Shih-Fu Chang, name: Zhe Gan, name: Yinfei Yang

While Ferret seamlessly integrates regional understanding into the Large
Language Model (LLM) to facilitate its referring and grounding capability, it
poses certain limitations: constrained by the pre-trained fixed visual encoder
and failed to perform well on broader tasks. In this work, we unveil Ferret-v2,
a significant upgrade to Ferret, with three key designs. (1) Any resolution
grounding and referring: A flexible approach that effortlessly handles higher
image resolution, improving the model's ability to process and understand
images in greater detail. (2) Multi-granularity visual encoding: By integrating
the additional DINOv2 encoder, the model learns better and diverse underlying
contexts for global and fine-grained visual information. (3) A three-stage
training paradigm: Besides image-caption alignment, an additional stage is
proposed for high-resolution dense alignment before the final instruction
tuning. Experiments show that Ferret-v2 provides substantial improvements over
Ferret and other state-of-the-art methods, thanks to its high-resolution
scaling and fine-grained visual processing.