See, Point, Fly: A Learning-Free VLM Framework for Universal Unmanned
  Aerial Navigation
Authors: name: Chih Yao Hu, name: Yang-Sen Lin, name: Yuna Lee, name: Chih-Hai Su, name: Jie-Ying Lee, name: Shr-Ruei Tsai, name: Chin-Yang Lin, name: Kuan-Wen Chen, name: Tsung-Wei Ke, name: Yu-Lun Liu

We present See, Point, Fly (SPF), a training-free aerial vision-and-language
navigation (AVLN) framework built atop vision-language models (VLMs). SPF is
capable of navigating to any goal based on any type of free-form instructions
in any kind of environment. In contrast to existing VLM-based approaches that
treat action prediction as a text generation task, our key insight is to
consider action prediction for AVLN as a 2D spatial grounding task. SPF
harnesses VLMs to decompose vague language instructions into iterative
annotation of 2D waypoints on the input image. Along with the predicted
traveling distance, SPF transforms predicted 2D waypoints into 3D displacement
vectors as action commands for UAVs. Moreover, SPF also adaptively adjusts the
traveling distance to facilitate more efficient navigation. Notably, SPF
performs navigation in a closed-loop control manner, enabling UAVs to follow
dynamic targets in dynamic environments. SPF sets a new state of the art in DRL
simulation benchmark, outperforming the previous best method by an absolute
margin of 63%. In extensive real-world evaluations, SPF outperforms strong
baselines by a large margin. We also conduct comprehensive ablation studies to
highlight the effectiveness of our design choice. Lastly, SPF shows remarkable
generalization to different VLMs. Project page: https://spf-web.pages.dev