Flex: End-to-End Text-Instructed Visual Navigation with Foundation Models

Authors: Makram Chahine, Alex Quach, Alaa Maalouf, Tsun-Hsuan Wang, Daniela Rus
Primary area: applications to robotics, autonomy, planning
Keywords: Robot learning, Foundation Models, Visual Navigation

Abstract:
End-to-end learning directly maps sensory inputs to actions, creating highly integrated and efficient policies for complex robotics tasks. However, such models are tricky to efficiently train and often struggle to generalize beyond their training scenarios, limiting adaptability to new environments, tasks, and concepts. In this work, we investigate the minimal data requirements and architectural adaptations necessary to achieve robust closed-loop performance with vision-based control policies under unseen text instructions and visual distribution shifts.
To this end, we design datasets with various levels of data representation richness, refine feature extraction protocols by leveraging multi-modal foundation model encoders, and assess the suitability of different policy network heads. Our findings are synthesized in Flex (Fly-lexically), a framework that uses pre-trained Vision Language Models (VLMs) as frozen patch-wise feature extractors, generating spatially aware embeddings that integrate semantic and visual information. These rich features form the basis for training highly robust downstream policies capable of generalizing across platforms, environments, and text-specified tasks.
We demonstrate the effectiveness of this approach on quadrotor fly-to-target tasks, where agents trained via behavior cloning on a small simulated dataset successfully generalize to real-world scenes, handling diverse novel goals and command formulations.

Decision: reject
Reviewer scores: 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730717486157

thread_type:
official_review

<thread_object>
creator: reviewer_bs4x

<content>
summary:
This paper conducts an empirical study on using features extracted by frozen Vision-Language Pretrained Models (VLMs) for downstream policy learning in object-guided quadrotor visual navigation. The policy is trained with synthetic data in simulation and demonstrates a real-world deployment example, providing an initial test of generalizability.

strengths:
1. The motivation to investigate frozen VLM encoders for semantically rich and spatially aware embeddings without extensive retraining is clear and well-grounded, offering a promising approach to simplify training in complex robotic tasks.
2. The experimental setup is thorough, exploring various dataset configurations and policy architectures, which provides valuable insights into design choices for multi-modal robotic policies.

weaknesses:
1. Limited Task Scope: While the use of VLM features for visual navigation is compelling, the experiments are restricted to quadrotor object navigation. Broader experiments across different robotic tasks or navigation scenarios could help demonstrate Flex’s general applicability, aligning with the paper’s claims of “closed-loop visual navigation agents that generalize across unseen environments.”
2. Feature Extractor Generalizability: Although the study aims to investigate “suitable feature extractors for text and vision-based robotics learning,” the experiments focus solely on BLIP-2 and modifications to its Q-former. Including other VLMs, such as CLIP or ImageBind, would strengthen the claim of generalizability and demonstrate the adaptability of the proposed method across diverse VLM architectures.
3. Task Complexity: In the current setup, the target is observable from the beginning, and language instructions are relatively simple. Visual navigation in complex settings involves learning semantic associations between language and visual cues and exploring the environment effectively to locate and reach the target. Here, the task primarily involves target identification and navigation, with limited indication of the benefits of end-to-end learning over modular approaches like GOAT. Clarifying the advantages of end-to-end learning in this setup would improve the paper’s impact.
4. Language Instruction Scope: The difficulty and complexity of language instructions in the dataset are not discussed. Unlike works in Zero-Shot Object Navigation (ZSON) or Zero-Shot Vision-and-Language Navigation, which leverage the language comprehension capabilities of VLMs, the findings in this paper are limited to basic instructions. This limitation makes it challenging to support claims of Flex being “user-interactive,” as the current level of interaction falls short of handling more complex language instructions effectively.

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730761962525

thread_type:
official_review

<thread_object>
creator: reviewer_mjid

<content>
summary:
This paper introduces FLEX, a method for making drones understand and follow natural language commands. Instead of building everything from scratch, it cleverly uses an existing vision-language model (BLIP-2) to process both camera images and text instructions. The system was trained on simple simulated scenarios with just two colored balls, yet surprisingly, it worked well in real-world tests with various objects. The key innovation is how it breaks down camera images into patches and processes them alongside text commands, allowing the drone to understand both what it sees and what it's looking for.

strengths:
1. Integrating visual navigation with the vision-language foundation model is an interesting direction.
2. The performance of this paper is reasonable with targeting and query on a limited set of objects.
3. The paper writing is in general good except the introduction.
4. The paper evaluates their methods in both simulation and real scenarios.

weaknesses:
1. The paper writing is strange where the introduction section is quite similar to related work.
2. The training data seems to be rich with 1, 1M, 2, 2M, however those four datasets contain only a limited amount of target, and query, which I think is not enough for an open world navigation paper.
3. In addition the paper didn't handle how to solve image sequences for navigation work, which means that they predict action frame by frame.
4. One interesting thing is that in order to correctly navigate in the scene, actually we need to know the depth information, however, the amount of data the paper trained is not enough for that kinds of generalization.

questions:
1. I am mostly interested on how the evaluation data is different from the training data. I am happy to fix my score, if I could understand the performance is not from scene memorization.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730827547172

thread_type:
official_review

<thread_object>
creator: reviewer_2xnd

<content>
summary:
This work studied robust multimodal representations for drones. It combines spatial and lexical features via patch-wise descriptors from VLMs. The training pipeline for closed-loop visual navigation agents that generalize across unseen environments, using real-time natural language instructions to achieve adaptability well beyond the training scope. Extensive experiments on drone fly-to-target tasks, showcasing the ability in generalization and real-world deployment

strengths:
1. This paper is well written and easy to read.
2. The experiment part is good with analysis in feature robustness quality. Real-world experiments are given.

weaknesses:
1. The motivation of using mask-based patch features is unclear to me. I think it can be simply replaced by ViT style patch feature encoding and attention-based interaction. And for some downstream policy nets like MLP, the spatial feature is eliminated by average pooling. Why the spatial dimension is necessary? 
2. Following above, I do not find any ablation studies on different feature designs.

questions:
See weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731983231612

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General reply

comment:
We sincerely thank the reviewers for their positive feedback and valuable constructive criticism. Your professional review and careful reading have already helped us improve our work. We have thoroughly addressed all the comments raised during the initial review. If further clarification is needed, please do not hesitate to reach out. Your engagement is highly appreciated. We will be adapting the manuscript as the discussion phase progresses and provide you with the revised version by the end of the period.

Our response was slightly delayed as we undertook thorough double-checking to ensure that our stated claims, especially regarding the use of other features, are indeed correct. We thank you for your patience.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732204046885

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Looking forward to our discussion

comment:
Dear Reviewers,  

Thank you once again for taking the time to review our work and provide valuable feedback. Your insights have been instrumental in enhancing the quality of our paper. We hope that our clarifications regarding the scope, objectives, and uniqueness of the work have effectively addressed your concerns.  

As the discussion period continues, we are happy to engage in further dialogue and welcome any additional questions or feedback you may have. Please don’t hesitate to reach out—we are more than willing to address them promptly.  

The authors!
</content>
</thread_object>

</thread>
<thread>
creation date:
1732548124010

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Feedback as rebuttal nearing end

comment:
Dear Reviewers, ACs, and SACs,

We value the constructive feedback provided by the initial reviews. However, we hope we can get some benefit from the openly communicated reviewing process and have some engagement from the reviewers to further improve the paper.

At the moment, we have not yet had the chance to engage with any of the reviewers after addressing their comments. 
As the discussion period is coming to an end, we would like to note that we are here to address any lingering issues and comments, even at short notice. We believe that we have adequately addressed all the reviewers’ concerns and issues but remain available to provide clarifications and changes to the manuscript.

Thank you and we look forward to hearing back.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732746581548

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision submitted

comment:
We wish to inform the reviewers and the chairs that we have submitted a revised version of the manuscript taking into account the reviewers' comments namely (changes in purple in the submitted revision):

- Rewriting of the intro and related works section for improved presentation and clarity
- Clarification w.r.t weakness 4 from reviewer BS4x at the end of the intro
- A summary of the test setup to ensure generalization to new environments and objects is clearly stated (new subsection 4.3)
- Clarification regarding the failure of entire image encoding in providing useful features for robot action (section 5.2)

We thank the reviewers for their valuable input, and believe the manuscript changes have addressed their main concerns. With the PDF changes no longer supported, we remain available to answer any additional questions the reviewers might have.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates the use of frozen pre-trained vision-language models (VLMs) with the philosophy of making it lightweight and minimal in terms of both training and data. The approach uses a patch-based representation along with a proposed masking approach to perform feature extraction. Results are demonstrated on a drone scenario tested in simulation and real world. Reviewers appreciated that the topic of VLM integration towards vision-based navigation is interesting and that the paper is easy to read. However, overall reviewers had significant concerns about the paper, including the motivation of the method (e.g. use of masking, 2xnd), the limited amount and complexity of evaluation tasks (BS4x, mJid) and situation with respect to other methods (e.g. GOAT mentioned by BS4x). and lack of comprehensiveness in terms of other feature extractors (BS4x). After reviewing the rebuttal, many of these issues still remain. The use of VLMs, both in terms of frozen models and fine-tuned, is a topic that is both interesting but also being explored by a large number of works (including some cited by the authors, methods like GOAT, etc.). Such methods are often compared across a number of vision-based navigation benchmarks with varying complexity, open-vocabulary variants, etc. to demonstrate the universality of the proposed architectures proposed. The current paper proposes another point in this space of possible designs (which is quite large), but only shows results on very limited settings. Further, aspects such as generalization (e.g. what variations, distribution shifts, new objects/scenes/tasks will it generalize to?) is not explored at all. As such, I agree with all of the reviewers that this paper is below the bar for acceptance. I recommend that the authors address the major concerns of the reviewers and significantly expand both the justification of the design and the scope of evidence for its effectiveness.