LangNav: Language as a Perceptual Representation for Navigation
Authors: name: Bowen Pan, name: Rameswar Panda, name: SouYoung Jin, name: Rogerio Feris, name: Aude Oliva, name: Phillip Isola, name: Yoon Kim

We explore the use of language as a perceptual representation for
vision-and-language navigation. Our approach uses off-the-shelf vision systems
(for image captioning and object detection) to convert an agent's egocentric
panoramic view at each time step into natural language descriptions. We then
finetune a pretrained language model to select an action, based on the current
view and the trajectory history, that would best fulfill the navigation
instructions. In contrast to the standard setup which adapts a pretrained
language model to work directly with continuous visual features from pretrained
vision models, our approach instead uses (discrete) language as the perceptual
representation. We explore two use cases of our language-based navigation
(LangNav) approach on the R2R vision-and-language navigation benchmark:
generating synthetic trajectories from a prompted large language model (GPT-4)
with which to finetune a smaller language model; and sim-to-real transfer where
we transfer a policy learned on a simulated environment (ALFRED) to a
real-world environment (R2R). Our approach is found to improve upon strong
baselines that rely on visual features in settings where only a few gold
trajectories (10-100) are available, demonstrating the potential of using
language as a perceptual representation for navigation tasks.