# VIPER: Visual Perception and Explainable Reasoning for Sequential Decision-Making

Mohamed Salim Aissi, Clémence Grislain, Mohamed Chetouani, Olivier

Sigaud, Laure Soulier, Nicolas Thome

► To cite this version:

Mohamed Salim Aissi, Clémence Grislain, Mohamed Chetouani, Olivier Sigaud, Laure Soulier, et al.. VIPER: Visual Perception and Explainable Reasoning for Sequential Decision-Making. 2026. [hal-05525519](https://hal.science/hal-05525519v1)

HAL Id: hal-05525519

https://hal.science/hal-05525519v1

Preprint submitted on 24 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
arXiv:2503.15108v3 [cs.LG] 10 Sep 2025

# VIPER: Visual Perception and Explainable Reasoning for Sequential Decision-Making

Mohamed Salim Aissi¹*, Clémence Grislain¹, Mohamed Chetouani¹,
Olivier Sigaud¹, Laure Soulier¹, Nicolas Thome¹,²

¹Sorbonne Université, CNRS, ISIR, F-75005 Paris, France

²Institut universitaire de France (IUF)

* Corresponding author: mohamed-salim.aissi@Sorbonne-universite.fr

## Abstract

While Large Language Models (LLMs) excel at reasoning on text and Vision-Language Models (VLMs) are highly effective for visual perception, applying those models for visual instruction-based planning remains a widely open problem. In this paper, we introduce VIPER, a novel framework for multimodal instruction-based planning that integrates VLM-based perception with LLM-based reasoning. Our approach uses a modular pipeline where a frozen VLM generates textual descriptions of image observations, which are then processed by an LLM policy to predict actions based on the task goal. We fine-tune the reasoning module using behavioral cloning and reinforcement learning, improving our agent's decision-making capabilities. Experiments on the ALFWorld benchmark show that VIPER significantly outperforms state-of-the-art visual instruction-based planners while narrowing the gap with purely text-based oracles. By leveraging text as an intermediate representation, VIPER also enhances explainability, paving the way for a fine-grained analysis of perception and reasoning components.

## 1. Introduction

The impressive capabilities of Large Language Models (LLMs) and Vision-Language Models (VLMs) are now well-demonstrated in various tasks, including respectively translation, summarization, and question answering [21, 26, 31] for Natural Language Processing (NLP) and object detection, image captioning, or visual question answering [7, 34] in computer vision.

These last years, a new line of research leverages recent advancements in LLMs and VLMs for the development of embodied AI agents [30, 35]. One example of task, tackled in this paper, concerns visual-based instruction-based planning, where an agent is tasked with solving a problem

<figure><img src="image_2.png" /><figcaption>Figure 1. VIPER framework for visual instruction-based planning: from image observations and a user's goal, VIPER interacts with the environment to achieve it. VIPER is the first method to split the decision-making process using text as intermediate representation. The VLM-based perception module generates a textual scene description, and the LLM reasoner then selects the most appropriate action. By detecting the most important text tokens and image regions (green in Figure), VIPER has significant potential for explaining decision-making mechanisms.</figcaption></figure>

formulated through a textual prompt (e.g., “heat the apple and put it on the countertop”) while using images as observations of the world — see Figure 1. The direct application of LLMs and VLMs for solving instruction-following tasks remains far from trivial. In particular, the abstract knowledge embedded in LLMs/VLMs often lacks the experiential foundation needed to develop common sense and a grounded understanding of the world [1, 10]. This grounding capacity requires additional skills, such as learning objects’ physical properties and affordances, understanding possible interactions between them, and grasping the temporal causality of the world.

For text-based instruction-based planning, attempts have been made to enhance the relevance of LLMs actions, including prompting strategies such as In-Context Learn-
ing (ICL) and Chain-of-Thought (CoT) reasoning [19, 32], as well as the integration of dedicated grounding modules [6, 12, 14]. More recently, reinforcement learning (RL) has also emerged as a powerful method that enables models to acquire real-world knowledge through interaction with the environment [2, 3, 23]. When turning to visual-based instruction-based planning, recent works have tried to apply similar techniques to VLMs [29, 33] by engaging in realistic multimodal environments and fine-tuning using RL. However, the results obtained in [33] remain significantly lower compared to those achieved with text-based models, while [29] relies on detailed text-based descriptions of each image, including object IDs, to guide the VLM during training—a cumbersome and unrealistic requirement in interactive environments.

This paper proposes a method for visual-based instruction-based planning that only requires images as world's observation, during both training and inference. We introduce the VIPER<sup>1</sup> framework, which integrate the Visual Perception capabilities of VLMs with the Explainable text Reasoning of LLMs.

VIPER follows a modular approach, as illustrated in Figure 1: (1) a **perception module**, which generates textual goal-agnostic descriptions from image observations using a frozen VLM, and (2) a **reasoning module**, which predicts actions based on these descriptions, the task goal, and past interactions. The LLM acts as a policy model, deriving an action distribution from its token probabilities. This structured approach enables explainable, multimodal decision-making, bridging perception and reasoning through text for instruction-based planning.

In summary, our key contributions are as follows:

* **VIPER** is the first method to leverage text as an intermediate representation between perception and reasoning in visual instruction-based planning. We integrate VLM-based perception with LLM-based reasoning, where the VLM describes the image observation to the LLM reasoner, which then generates the appropriate action based on a given textual goal.

* We fine-tune the LLM-based reasoning module using a combination of behavioral cloning (BC) with expert trajectories and RL from environment interactions. By leveraging a zero-shot VLM-based perception module to caption image observation, we enhance the decision-making capabilities of the reasoning module without the need for textual supervision from the environment.

* Our intermediate text representation enables the development of rich explainability techniques. By detecting important tokens in the image description and the corresponding image regions, VIPER can perform weakly supervised instance segmentation, facilitate a fine-grained analysis of the decision-making process, and provide tools for analyzing failure modes.

Extensive experiments conducted on the challenging ALFWorld environment [20] reveal that VIPER significantly outperforms state-of-the-art visual instruction-based planning methods while narrowing the gap with respect to LLM-based oracles that use precise text descriptions as input. The relevance of our intermediate text representation for explainability is further validated by various quantitative results, while ablation studies validate the importance of our architectural and training choices. Source code will be released upon acceptance.

## 2. Related Work

### 2.1. LLM and VLM for Sequential Decision-Making

When tackling long-horizon multimodal sequential decision-making tasks, embodied agents leverage key internal mechanisms such as perception and reasoning to achieve their objectives. Recently, LLMs and VLMs have demonstrated emergent capabilities in understanding complex situations, establishing them as essential architectures for embodied agents [1, 19, 30]. However, these models face challenges related to grounding and alignment with the physical world [16]. A primary line of research focuses on leveraging LLMs for sequential decision-making through text, either by filtering irrelevant actions for a given task [19, 30] or by learning from environmental feedback via reinforcement learning [3, 23]. These methods have demonstrated impressive performance and generalization capabilities. However, they are primarily designed for textual environments, where all state information is transmitted by the environment through text, limiting their applicability to real-world embodied agents. Recently, research has increasingly focused on leveraging VLMs for sequential decision-making tasks in visual environments. For instance, EMMA [29] trained a VLM-based agent for sequential decision-making in visual environments by distilling the actions of a (SOTA) textual agents, such as Reflexion [19], within a parallel textual world. This approach represents an initial step toward utilizing multimodality for instruction generation. However, compared to our work, it remains dependent on a parallel textual environment for task learning, limiting its applicability to real-world scenarios. Closer to our work, RL4VLM [33] employs reinforcement learning to train VLM-based agents within visual environments, enabling them to learn to achieve tasks across diverse environments. However, its performance remains inferior to methods that rely solely on textual observation for reasoning and planning. To address these limitations, we propose a framework that leverages the reasoning capabilities of LLMs in the textual modality and applies them to real-world scenarios through the vision

<footnote n="1">¹code can be found here</footnote>
modality. Our approach VIPER creates a decomposition between perception and reasoning via an intermediate text modality: a VLM component processes visual input to generate textual descriptions of all elements present in a scene, and a fine-tuned LLM utilizes this description to generate actions that achieve the specified goal.

## 2.2. Text as Intermediate Modality

Recently, VLMs have demonstrated impressive performance on mainstream computer vision tasks such as object detection, visual question answering (VQA), and image captioning [7, 34]. These tasks can be addressed using a single architecture and prompting, making VLMs a promising candidate for robotic applications. However, recent studies have shown that these architectures still underperform compared to their LLM counterparts on purely textual data. [24] demonstrated that VLMs often exhibit inferior reasoning capabilities compared to LLMs, highlighting the advantages of text-based reasoning. For example, [37] utilized a VLM for image captioning before performing a question-answering task to enhance image-related information and improve VQA performance, demonstrating the role of VLMs in extracting scene information. Similarly, [9] employed a VLM to extract bounding boxes and object information from a scene for robotic tasks, while [5] proposed Image-to-PDDL, where a VLM extracts a representation in Planning Domain Description Language (PDDL) [17] of the scene, which is then processed by an LLM to generate robotic actions, effectively leveraging the strengths of both architectures.

Our framework follows this line of research by combining the capabilities of VLMs and LLMs. VIPER is, to the best of our knowledge, the first approach for utilizing a VLM to extract scene information in text format and a fine-tuned LLM to reason over these descriptions in visual instruction-based planning.

# 3. The VIPER Framework

We introduce VIPER, a novel model for goal-oriented sequential decision-making in interactive multimodal environments. As shown in Figure 2(a), our framework consists of two key components: (1) a perception module, which generates a textual description of the visual observation (Section 3.2 and Figure 2(b)), and (2) a reasoning module, which processes this description alongside the goal and past actions to determine the next action (Section 3.3 and Figure 2(c)). We use a pre-trained VLM as our perception module and an LLM fine-tuned with a sequential training strategy combining BC and RL for our reasoning module (Section 3.4 and Figure 2(c)), allowing us to improve the grounding capabilities of the reasoning module on textual inputs while avoiding the need for textual supervision from the environment. While VIPER excels at solving sequential decision-making tasks, it also enables interpretable decision-making through the intermediate text modality between the perception and reasoning modules (Section 3.5).

## 3.1. Problem Statement

We consider a goal-conditioned partially observable Markov decision process $M = (S, A, T, R, G, O, \gamma)$, where $S$ is the state space, $A$ is the action space, $G$ is the goal space, $T: S \times A \to S$ is the transition function, $R: S \times G \to \mathbb{R}$ is the goal-conditioned reward function and $\gamma$ the discount factor. More specifically, we consider a multimodal environment where, given a language vocabulary $V$ and a maximum sequence length $N$, the goal $g \in G$ and the action $a \in A$ are in $V^N$, while the observation of the state is an RGB image $o = O(s) \in \mathbb{R}^{3 \times H \times W}$ where $H \times W$ are the image dimensions.

Our goal in this environment is to build an embodied multimodal agent, characterized by a policy $\pi$, that processes an image observation $o$ from the environment at each step and proposes an action $a \in A$ given the task goal $g$.

## 3.2. State Description through VLMs

To incorporate text as an intermediate modality for decision-making, we define a mapping function between image observation $o$ and its textual description $d = (w_1, w_2, ..., w_N) \in D \subset V^N$. Specifically, we leverage the strong captioning capabilities of VLMs to generate these textual descriptions and define a VLM mapping function conditioned on a prompt $p_{vlm}: m_{vlm}(\cdot|p_{vlm}) : \mathbb{R}^{3 \times H \times W} \to D$. As detailed in Figure 2(b), we use a fixed prompt, $p_{vlm} = \text{"Describe in detail the image"}$, which remains agnostic to both the goal and the current step in the trajectory. The VLM is used exclusively for perception, while the decision-making process is entirely handled by the LLM, as detailed in Section 3.3. Further details on VLM prompt engineering can be found in Sup. B.1.

## 3.3. LLMs as Decision-Making Policies

We leverage the reasoning capabilities of LLMs to solve sequential decision-making tasks [3, 19, 23]. We follow the framework introduced in recent literature [3, 23] to derive a policy $\pi$ (a probability distribution over possible actions) from an LLM, as shown in Figure 2(c). More precisely, given a prompt $p$, the probability of an action sequence $a_i \in A$, denoted as $\pi(a_i|p) = P_{LLM}(a_i|p)$, is obtained from the decoding probabilities of the tokens $\{w_j\}$ composing $a_i$. Specifically, this probability is computed as:

$$ \pi(a_i|p) = \frac{1}{|a_i|} \prod_{j=0}^{|a_i|} P_{LLM}(w_j|p, w_{<j}). \quad (1) $$

The recent state-of-the-art RL4VLM agent [33] uses free-form generation of actions, including possible actions
<figure><img src="image_3.png" /><figcaption>Figure 2. **The VIPER framework.** We employ a VLM and an LLM as respectively the perception and reasoning modules of the agent, to solve tasks in a household environment. (a) In this framework, the **perception module** generates a description of the observation, which is then fed to the reasoning module along with the task goal from the environment. The **reasoning module** then produces a textual action that the agent execute in the environment. (b) The **perception module** (VLM) generates a textual description of the image observation based on a fixed prompt. (c) The **reasoning module** constructs a prompt by integrating the observation description, the goal, and past actions. This prompt is then fed into the LLM, which has been fine-tuned for decision-making. The LLM generates a probability distribution over possible actions and selects the most appropriate one based on the agent's current observation and its goal.</figcaption></figure>

in the prompt. In contrast, the policy $\pi(a_i|p)$ in Eq. (1) is directly constrained to the support of possible actions, improving performances on the downstream task (see Sec. 4).

In addition, while existing LLM agents methods [19, 25, 30] receive prompts with textual observations directly from the environment, our method constructs the prompt using a textual description of the image observation generated by a VLM $d = m_{vlm}(o|p_{vlm})$, along with the goal and the past actions.

## 3.4. Training Strategies

Recent works [2, 3] show that while LLMs possess common-sense knowledge, they lack grounding and are misaligned with physical environments [16], leading to poor zero-shot performance in complex interactive decision-making tasks. To address this, following [3, 23], we fine-tune the reasoning module (LLM) using a sequential strategy. First, we align it with a good reference policy using BC on a dataset of expert demonstrations, which is assumed to be available in the environment. Then, we refine its performance through RL via exploration and interaction with the environment. This BC+RL approach is well-established in the control community [15] and has demonstrated its relevance in our ablation study (see Section 4.3). While VIPER's reasoning module operates on textual inputs, the textual description of the image observation is provided by a frozen VLM. Therefore, unlike EMMA [29], which trains its model by distilling a text expert policy and thus requires textual observations from the environment, our training relies solely on the goal and the image observations o.

**BC fine-tuning:** We pre-train the LLM policy $\pi$ parametrized by a parameter $\theta$, using a dataset of expert demonstrations $\mathcal{T} := \{(t_i, g_i)\}_{i=1}^{|\mathcal{T}|}$ of observation-action trajectories $t_i = \{o_0, a_0, o_1, a_1, ...\}$ through BC. In our case, this expert dataset is created by selecting the successful trajectories of a rule-based expert (Sup. A.2). The BC objective is to maximize the log-likelihood of expert actions with respect to the action distribution predicted by the LLM policy $\pi$ along the expert trajectory. Therefore, the BC loss is the negative log-likelihood, defined as:

$$ \mathcal{L}_{BC}(\theta) = - \sum_{t_i \in \mathcal{T}} \sum_{(o_j, a_j) \in t_i} \log(\pi_{\theta}(a_j | P(g_i, d_j, a_{<j}))) \quad (2) $$

where $d_j$ is the VLM description of the observation $o_j$, $d_j = m_{vlm}(o_j)$ and $P$ is the prompting function defined in Sup. B.2.

**RL fine-tuning.** We further enhance the policy through online interaction with the environment and fine-tune $\pi_{\theta}$ using Proximal Policy Optimization (PPO) [18]. We learn a value function $\hat{V}: D \times G \to \mathbb{R}$ that approximates the true value $V(s, g) = \mathbb{E}_{a \sim \pi_{\theta}(a|s)}[R(s, g) + \gamma \times V(T(s, a), g)]$ for PPO. The policy $\pi_{\theta}$ is then optimized to minimize $\mathcal{L}_{PPO}(\theta)$, as described in [18].

## 3.5. Explaining Agent Decision

During inference, VIPER first executes the VLM to generate a textual description of the scene, which is then passed to the LLM to infer the action. This two-step prediction process enhances the explainability of VIPER's behavior, as
the model’s perception can be interpreted through the generated text. We can thus analyze the relevance of the text description in regard to the goal.

Furthermore, we perform weakly-supervised instance segmentation of important concepts in the image description used by the reasoning module by computing the integrated gradient [22] of the chosen action *a*\* with respect to each text token in the image description, and selecting the most significant ones. Following the same procedure, we compute the integrated gradient of the selected tokens, with respect to the image (see Sup. D for more details). This provides a rich tool for explainability, linking the performance of the perception and reasoning modules, and enabling to finely analyze failure cases (see Sec. 4.4).

# 4. Experimental Results

We conduct a series of experiments to evaluate our framework in sequential decision-making tasks within the ALF-World benchmark. Our study aims to address the following research questions: • How does VIPER perform compared to state-of-the-art models? • What are the benefits of our approach in terms of explainability and performance with the integration of an intermediate text modality? • What is the impact of different architectural configurations in the perception and reasoning modules, as well as various training regimes, on overall performance?

## 4.1. Experimental Setup

**Environment.** Our experiments are conducted on the ALF-World benchmark [20], a 3D environment that simulates six categories of household tasks with textual goals. ALFWorld has become a standard benchmark in the literature for evaluating models in long-horizon multimodal decision-making tasks, offering a realistic and interactive environment well suited for studying real-world embodied applications. This benchmark contains sets of training, in-distribution validation, and out-of-distribution (OOD) episodes that encapsulate specific layouts of the environment (room in the house, object and furniture placement, etc.). We follow the standard protocol and train VIPER on the training set of episodes, perform cross-validation on the in-distribution validation set, and report our results on the OOD test set. We evaluate the models using the mean success rate (higher is better, ↑) and the mean length of successful trajectories (lower is better, ↓). See Sup. A.2 for more details about the environment.

**Architecture and Training.** VIPER uses Florence-2-220M [28] as VLM and Mistral-7B [11] as LLM, with further discussions on model choices in Section 4.3. We fine-tune the LLM as described in Section 3.4 using low-rank adaptation (LoRA) with parameters *r* = 32 and *α* = 16 on the attention layers. For BC, we train the model until con-

vergence and for PPO for approximately 5 × 10⁵ interaction steps with the environment. Both fine-tuning processes use a batch size of 2 and the Adam optimizer with a learning rate of 10⁻⁴. For each task, BC fine-tuning takes ≈ 20h and PPO ≈ 200h on 4 H100 GPUs.

**Baselines.** We compare VIPER with various SOTA models from the literature, categorized into two groups. The first group comprises models trained with supervision from a parallel textual environment, including Reflexion [19], DEPS [25], ReAct [30], AutoGen [27] which only receive textual observation of the environment and EMMA [29] which uses both visual and textual observation during training but only access visual observation during inference. Like VIPER, the second group consists of models that exclusively rely on visual information from the environment, including zero-shot models (MiniGPT-4 [36], InstructBLIP [4], Florence 2 [28], Idefics 2 [13]) and a VLM fine-tuned through BC and RL, RL4VLM [33]. All methods of the first group accessing the textual observations benefit from richer scene information, including the number of objects, object IDs within the description, and the correspondence between object IDs and actions. In contrast, the models of the second group, including the zero-shot perception module in VIPER, cannot perfectly generate this level of detail, as this information is internal to the environment and not directly observable from images. Therefore, the models of the first group can be seen as oracle baselines.

## 4.2. Comparison with the state-of-the-art

We compare VIPER with the two baseline families on ALFWorld (see Table 1), evaluating both the mean success rate and the mean trajectory length required to achieve the goals. As observed, VIPER outperforms zero-shot visual-only methods, achieving an average 50pt improvement in success rate and an 8-step reduction in trajectory length across multiple tasks. It also surpasses RL4VLM [33], the SOTA method most closely related to our work that applies both BC and RL fine-tuning. This lower performance of RL4VLM [33] may be explained by the use of CoT reasoning with free generation for action prediction, which prevents it from being constrained to the action space. This leads to irrelevant predictions, as explained in Section 3.3. Another factor could be the absence of past action history in the agent’s prompt, as discussed in Sup. B.2. Besides, VIPER outperforms ReAct [30] and matches the performance of AutoGen [27] and DEPS [25], which rely on textual observations from the environment. Moreover, VIPER approaches the performance of oracle methods such as EMMA [29] and Reflexion [19], achieving more than 70% success rate in the first five tasks and more than 50% in the Pick2 task. The performance gap between Pick2 and the other tasks results from the increased complexity of manipulating multiple objects. Text-based methods, such as
<table><caption>Table 1. : Comparison with the state-of-the-art. Mean success rate (↑) and episode length (↓) into brackets of various methods (*-reported in previous work), and respectively represent access to textual and visual observation from the environment. Florence-2, Idefics-2, MiniGPT-4 and InstructBLIP are used in zero shot. VIPER surpasses SOTA methods solely accessing visual observations and narrows the gap with oracle methods relying on textual observations.</caption><thead><tr><th></th><th>Environnement</th><th>Pick</th><th>Look</th><th>Clean</th><th>Heat</th><th>Cool</th><th>Pick2</th><th>Avg</th></tr></thead><tbody><tr><td>AutoGen* [27]</td><td>Text</td><td>0.92 (-)</td><td>0.83 (-)</td><td>0.74 (-)</td><td>0.78 (-)</td><td>0.86 (-)</td><td>0.41 (-)</td><td>0.77 (-)</td></tr><tr><td>ReAct* [30]</td><td>Text</td><td>0.71 (18.1)</td><td>0.28 (23.7)</td><td>0.65 (18.8)</td><td>0.62 (18.2)</td><td>0.44 (23.2)</td><td>0.35 (25.5)</td><td>0.54 (20.6)</td></tr><tr><td>DEPS* [25]</td><td>Text</td><td>0.93 (-)</td><td>1.00 (-)</td><td>0.50 (-)</td><td>0.80 (-)</td><td>1.00 (-)</td><td>0.00 (-)</td><td>0.76 (-)</td></tr><tr><td>Reflexion* [19]</td><td>Text</td><td>0.96 (17.4)</td><td>0.94 (16.9)</td><td>1.00 (17.0)</td><td>0.81 (19.4)</td><td>0.83 (21.6)</td><td>0.88 (21.6)</td><td>0.91 (18.7)</td></tr><tr><td>EMMA* [29]</td><td>Text + Vision</td><td>0.71 (19.3)</td><td>0.88 (19.6)</td><td>0.94 (17.5)</td><td>0.85 (19.6)</td><td>0.83 (19.9)</td><td>0.67 (22.4)</td><td>0.82 (19.5)</td></tr><tr><td>Florence-2 [28]</td><td>Vision</td><td>0.00 (30.0)</td><td>0.06 (28.5)</td><td>0.0 (30.0)</td><td>0.0 (30.0)</td><td>0.0 (30.0)</td><td>0.0 (30.0)</td><td>0.01 (29.7)</td></tr><tr><td>Idefics-2 [13]</td><td>Vision</td><td>0.04 (29.2)</td><td>0.06 (28.2)</td><td>0.0 (30.0)</td><td>0.0 (30.0)</td><td>0.0 (30.0)</td><td>0.0 (30.0)</td><td>0.02 (29.5)</td></tr><tr><td>MiniGPT-4* [36]</td><td>Vision</td><td>0.04 (29.0)</td><td>0.17 (17.7)</td><td>0.0 (30.0)</td><td>0.19 (26.3)</td><td>0.17 (26.7)</td><td>0.06 (28.9)</td><td>0.16 (26.9)</td></tr><tr><td>InstructBLIP* [4]</td><td>Vision</td><td>0.50 (21.5)</td><td>0.17 (26.8)</td><td>0.26 (25.0)</td><td>0.23 (27.2)</td><td>0.06 (28.9)</td><td>0 (30.0)</td><td>0.22 (26.2)</td></tr><tr><td>RL4VLM* [33]</td><td>Vision</td><td>0.47 (-)</td><td>0.14 (-)</td><td>0.10 (-)</td><td>0.14 (-)</td><td>0.18 (-)</td><td>0.18 (-)</td><td>0.21 (-)</td></tr><tr><td>VIPER</td><td>Vision</td><td>0.80 (13.1)</td><td>0.77 (16.7)</td><td>0.77 (19.5)</td><td>0.92 (14.3)</td><td>0.71 (20.4)</td><td>0.53 (24.0)</td><td>0.75 (18.0)</td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>Figure 3. Impact of BC and RL Training on Mean Success Rate. As observed, the combination VIPER BC+PPO yields the best overall performance in terms of success rate, highlighting the necessity of both BC and RL fine-tuning.</figcaption></figure>

EMMA and Reflexion, do not face this challenge, as they have access to internal environment information, including object identifiers and enumerations, which cannot be obtained solely through vision, as in VIPER (see Section 4.1).

Comparisons between textual observations and VIPER-generated descriptions can be found in Sup. A.2. Furthermore, VIPER achieves the shortest mean episode length compared to both the oracle and vision-only baselines, demonstrating that its successful trajectories are more efficient, taking more goal-directed actions and avoiding unnecessary steps.

### 4.3. Ablation study

**Impact of BC and RL Training.** Through an ablation study, we evaluate the impact of each training strategy defined in Section 3.4 – BC and RL – by comparing different variants of VIPER in Figure 3. Specifically, we report the performance of VIPER trained only with BC (VIPER BC) using rule-based expert demonstrations (see Sup. A.2), only PPO (VIPER PPO), and the sequential approach where PPO follows BC (VIPER BC+PPO). Although using SOTA VLMs and LLMs, the zero-shot model performs poorly across all tasks, highlighting the complexity of the tasks and the necessity of fine-tuning. The results further show that while applying RL fine-tuning alone (VIPER PPO) is sufficient to learn simpler tasks (Pick and Look), using BC beforehand is essential to learn more complex ones (Clean, Heat, Cool and Pick2). Furthermore, while VIPER BC already outperforms vision-only SOTA models, adding RL further improves performance, producing a boost of 3pt to 9pt depending on the task. This performance gain is larger when considering weaker reasoning modules (see Sup. C).

### 4.4. Impact of the Intermediate Perception Module

To assess the impact of the intermediate text modality in VIPER, we introduce a multimodal agent that follows the same prompt structure as our reasoning module but takes the image as input instead of the VLM-generated description. We refer to this baseline as the VLM agent.

**Improved Explainability.** We now examine how the intermediate textual description generated by VIPER can be leveraged to analyze failures. Figure 4 presents two examples of action prediction: a successful action (a) and an unsuccessful one (b). The intermediate textual descriptions generated by the perception module of VIPER play a crucial role in understanding why the agent selects a correct or incorrect action.
<figure><img src="image_5.png" /><figcaption>Figure 4. **Observation Analysis with Perception Modules.** We compare explainability in decision-making of Viper against a VLM agent baseline without intermediate text modality in examples of correct actions ✅ (a) and incorrect actions ✖ (b). VIPER's intermediate text modality filters relevant elements for action generation and retrieves them from the image using integrated gradients, helping to analyze both correct (e.g., identifying cups and sinks in (a)) and incorrect predictions (e.g., missing the egg while focusing on the irrelevant apple in (b)).</figcaption></figure>

In example (a), the integrated gradient of the description with respect to the action highlights elements like 'sink' and 'cups' as important, which helps to comfort the text description relevance with respect to the goal. Additionally, by computing the integrated gradient for these two concepts, we can extract their localizations in the image observation, further validating the correct behavior of the perception module.

In example (b), the agent selects an incorrect action w.r.t. the goal (it chooses an apple instead of an egg). Following the same procedure, we observe that the perception module generates the word apple in the description but does not include an egg. This indicates that the error originates from the perception module, which subsequently affects the reasoning process. Additional visualizations are provided in Sup. D. In contrast, for the baseline VLM agent, computing the integrated gradient of the image with respect to the action results in noisy saliency maps and a bounding box covering the whole image, offering much weaker explainability.

**Impact of Intermediate Text on Performance.** As we have just shown, introducing text as an intermediate modality between perception and reasoning improves Explainability. But it may also impact overall performance due to potential errors in perception.

We train two baseline VLM agents (first part of Table 2) based on the Idefics-2-8B [13] and Florence-2-220M [28] models, using BC with the same dataset and number of iterations as VIPER. For a fair comparison and due to com-

<table><caption>Table 2. **Impact of Intermediate Text modality and Model Analysis.** Upper part: VLM agent baselines, which operate without an intermediate text modality. Lower part: different variants of VIPER, each incorporating distinct perception and reasoning modules. All the agents are trained using **solely BC fine-tuning**.</caption><thead><tr><th>Agent</th><th></th><th>Pick</th><th>Look</th><th>Clean</th><th>Heat</th><th>Cool</th><th>Pick2</th><th>Avg</th></tr></thead><tbody><tr><td rowspan="2">VLM Agent</td><td>Idefics</td><td><strong>0.80</strong></td><td>0.61</td><td><strong>0.77</strong></td><td><strong>0.87</strong></td><td><strong>0.80</strong></td><td>0.48</td><td><strong>0.72</strong></td></tr><tr><td>Florence</td><td>0.42</td><td>0.61</td><td>0.67</td><td>0.52</td><td>0.48</td><td>0.23</td><td>0.48</td></tr><tr><td rowspan="3">VIPER</td><td>Flo — Mistral</td><td><strong>0.80</strong></td><td><strong>0.77</strong></td><td>0.67</td><td><strong>0.87</strong></td><td>0.71</td><td><strong>0.53</strong></td><td><strong>0.72</strong></td></tr><tr><td>Flo — LLaMA</td><td>0.62</td><td>0.61</td><td>0.67</td><td>0.74</td><td>0.40</td><td>0.35</td><td>0.57</td></tr><tr><td>Ide — Mistral</td><td><strong>0.80</strong></td><td>0.67</td><td>0.74</td><td>0.82</td><td>0.62</td><td><strong>0.53</strong></td><td>0.70</td></tr></tbody></table>

putational constraints, we omit the RL training step for the two VLM agents and VIPER. As shown in Table 2, both VIPER and the VLM-based agents achieve comparable performance within the same parameter scale, with Florence/Mistral for VIPER and Idefics for the VLM agent. This shows that VIPER effectively combines the strengths of both worlds, achieving strong decision-making performance while enhancing explainability.

## 4.5. Model Analysis

**Impact of Perception and Reasoning Components.** To evaluate the impact of each component in VIPER, we evaluate its performance with different perception (VLM) and reasoning (LLM) models of varying sizes: Idefics-2-8B and Florence-2-220M (respectively Ide and Flo in Table 2) for perception, and Mistral-7B [11] and LLaMA-1B [8] for reasoning. The second part of Table 2 summarizes the results of the different combinations. We see that the reasoning module is the most critical component of VIPER, as the
<figure><img src="image_6.png" /><figcaption>Figure 5. LoRA Parameterization. (a) Curves showing the performance of the BC framework trained with different LoRA parameters *r*. (b) Performance, in terms of success rate, for various *r* values. *r* = 32 represents the optimal trade-off between performance and LoRA size.</figcaption></figure>

performance changes only of 2% when swapping between
the 8B Idefics and 220M Florence models for perception
but drops from 72% to 57% in success rate on average over
the six tasks when replacing Mistral-7B with LLaMA-1B
for reasoning. These results confirm our initial assumption
that while both perception and reasoning matter, reasoning
is more critical, reinforcing our decision to fine-tune only
the reasoning module.

**LoRA Parameterization.** To fine-tune the pre-trained Mistral-7B reasoning module of **VIPER**, we use BC fine-tuning with LoRA, parameterized by the rank of the adapters $r \in [4, 8, 16, 32, 64, 128]$ and the update scaling factor $\alpha = r/2$ on the *Look* task. Figure 5 (a) shows the training curves. We observe that, as $r$ increases, the convergence speed improves, indicating that the model learns faster with a higher rank. Figure 5 (b) shows the test set success rates. For $r < 32$, the test set success rate is lower, suggesting that the rank is too low to capture the new knowledge required for generalizing sequential decision-making. For $r \ge 32$ the success rates stabilize, showing that the model has enough trainable parameters to effectively learn new knowledge. Therefore, we set $r = 32$ and $\alpha = 16$ in our final framework to maximize performance and minimize the number of trainable parameters.

5. Conclusion

In this paper, we introduce VIPER, a framework that in-
tegrates vision-language models (VLMs) and large lan-
guage models (LLMs) for instruction-based planning. By
leveraging VLMs for perception and LLMs for reasoning,
VIPER enables an embodied agent to generate textual de-
scriptions of visual observations and select actions based
on instructions, improving grounding in real-world envi-
ronments. Experiments on the ALFWorld benchmark show
that VIPER enhances task performance compare to SOTA
models accessing vision-only observations. Additionally,
by generating human-readable descriptions, it facilitates hu-

man understanding and interaction with agents. These re-
sults suggest that **VIPER** can serve as a foundation for ad-
vanced decision-making frameworks.

Future works include extending our approach to low-level control in robotics, enabling fine-grained interactions with the environment and executing real-world tasks with greater autonomy and precision.

Acknowledgments

Experiments presented in this paper were carried out us-
ing the HPC resources of IDRIS under the allocation
2025-[AD011015093R1] made by GENCI. This work was
supported by the European Commission's Horizon Eu-
rope Framework Programme under grant No 101070381
(PILLAR-robots), by RODEO Project (ANR-24-CE23-
5886), by PEPR Sharp (ANR-23-PEIA-0008, ANR,
FRANCE 2030) and by Cluster PostGenAI@Paris (ANR-
23-IACL-0007, FRANCE 2030). We would also like to
thank Clement Romac and Thomas Carta for the various
exchanges held during the project.

References

[1] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irvan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as I can, not as I say: Grounding language in robotic affordances, 2022. 1, 2

[2] Mohamed Salim Aissi, Clément Romac, Thomas Carta, Sylvain Lamprier, Pierre-Yves Oudeyer, Olivier Sigaud, Laure Soulier, and Nicolas Thome. Reinforcement learning for aligning large language models agents with interactive environments: Quantifying and mitigating prompt overfitting. In *Findings of the Association for Computational Linguistics: NAACL 2025*, pages 7030–7046, Albuquerque, New Mexico, 2025. Association for Computational Linguistics. 2, 4, 1

[3] Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. *arXiv preprint arXiv:2302.02662*, 2023. 2, 3, 4

[4] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023. 5, 6
[5] Xuzhe Dang, Lada Kudláčková, and Stefan Edelkamp. Planning with vision-language models and a use case in robot-assisted teaching, 2025. 3

[6] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PALM-E: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. 2

[7] Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Vinija Jain, and Aman Chadha. Exploring the frontier of vision-language models: A survey of current methodologies and future directions, 2024. 1, 3

[8] Aaron Grattafiori and Llama Team al. The llama 3 herd of models, 2024. 7

[9] Nurhan Bulus Guran, Hanchi Ren, Jingjing Deng, and Xianghua Xie. Task-oriented robotic manipulation with vision language models, 2024. 3

[10] Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng, Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, et al. Grounded decoding: Guiding text generation with grounded models for robot control. ArXiv preprint, abs/2303.00855, 2023. 1

[11] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. 5, 7

[12] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv, 2022. 2

[13] Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?, 2024. 5, 6, 7

[14] Jiachen Li, Qiaozhi Gao, Michael Johnston, Xiaofeng Gao, Xuehai He, Suhaila Shakiah, Hangjie Shi, Reza Ghanadan, and William Yang Wang. Mastering robot manipulation with multimodal prompts through pretraining and multi-task fine-tuning. arXiv preprint arXiv:2310.09676, 2023. 2

[15] Yiren Lu, Justin Fu, George Tucker, Xinlei Pan, Eli Bronstein, Rebecca Roelofs, Benjamin Sapp, Brandyn White, Aleksandra Faust, Shimon Whiteson, Dragomir Anguelov, and Sergey Levine. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios, 2023. 4

[16] Kyle Mahowald, Anna A. Ivanova, Idan Asher Blank, Nancy G. Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models: A cognitive perspective. arXiv, abs/2301.06627, 2023. 2, 4

[17] Drew McDermott, Malik Ghallab, Adele E. Howe, Craig A. Knoblock, Ashwin Ram, Manuela M. Veloso, Daniel S. Weld, and David E. Wilkins. Pddl-the planning domain definition language, 1998. 3

[18] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 4

[19] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. 2, 3, 4, 5, 6

[20] Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning, 2021. 2, 5

[21] Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305.09617, 2023. 1

[22] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks, 2017. 5

[23] Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. True knowledge comes from practice: Aligning llms with embodied environments via reinforcement learning. arXiv preprint arXiv:2401.14151, 2024. 2, 3, 4

[24] Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, and Neel Joshi. Is a picture worth a thousand words? delving into spatial reasoning for vision language models, 2024. 3

[25] Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents, 2024. 4, 5, 6

[26] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2023. 1

[27] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation, 2023. 5, 6

[28] Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks, 2023. 5, 6, 7

[29] Yijun Yang, Tianyi Zhou, Kanxue Li, Dapeng Tao, Lusong Li, Li Shen, Xiaodong He, Jing Jiang, and Yuhui Shi. Embodied multi-modal agent trained by an llm from a parallel textworld. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26275–26285, 2024. 2, 4, 5, 6, 1

[30] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. 1, 2, 4, 5, 6

[31] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024. 1
[32] Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh Murthy, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, et al. Retroformer: Retro-spective large language agents with policy gradient optimization. *arXiv preprint arXiv:2308.02151*, 2023. 2

[33] Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. *arXiv preprint arXiv:2405.10292*, 2024. 2, 3, 5, 6

[34] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey, 2024. 1, 3

[35] Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models as commonsense knowledge for large-scale task planning. *Advances in Neural Information Processing Systems*, 36, 2024. 1

[36] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023. 5, 6

[37] Övgü Özdemir and Erdem Akagündüz. Enhancing visual question answering through question-driven image captions as prompts, 2024. 3
# VIPER: Visual Perception and Explainable Reasoning for Sequential Decision-Making

## Supplementary Material

### A. Environment Specifications

#### A.1. ALFWord Dataset

In this section, we provide a detailed description of the data available in the ALFWorld environment. ALFWorld consists of six categories of household tasks, namely: "Pick and Place" (Pick), "Examine in Light" (Look), "Heat and Place" (Heat), "Cool and Place" (Cool), and "Pick2 and Place" (Pick2). The dataset is divided into four subsets: "Train", "Validation", and "Test (Out-of-Distribution)".

Table 3 presents a summary of the number of instances for each task. Additionally, ALFWorld includes a rule-based expert for both the visual and textual environments, designed to solve the various tasks in the training set. In particular, the textual rule-based expert shows superior performance compared to its visual counterpart, as shown in Table 3.

<table><caption>Table 3. Number of episodes per task and the count of successful (labeled) and unsuccessful (unlabeled) trajectories generated by the rule-based expert in ALFWorld. Only the successful trajectories are used for BC fine-tuning.</caption><thead><tr><th rowspan="2">Task</th><th rowspan="2">Total data</th><th colspan="2">Train</th><th rowspan="2">Valid</th><th rowspan="2">Test OOD</th></tr><tr><th>Labeled</th><th>Unlabeled</th></tr></thead><tbody><tr><td>Pick and Place</td><td>790</td><td>783</td><td>7</td><td>46</td><td>24</td></tr><tr><td>Examine in Light</td><td>308</td><td>303</td><td>5</td><td>19</td><td>18</td></tr><tr><td>Clean and Place</td><td>650</td><td>627</td><td>23</td><td>37</td><td>31</td></tr><tr><td>Heat and Place</td><td>459</td><td>438</td><td>21</td><td>25</td><td>23</td></tr><tr><td>Cool and Place</td><td>533</td><td>497</td><td>36</td><td>28</td><td>21</td></tr><tr><td>Pick2 and Place</td><td>813</td><td>734</td><td>79</td><td>45</td><td>17</td></tr></tbody></table>

#### A.2. ALFWord Textual description

The ALFWorld environment includes a parallel textual environment that provides a textual description of observations and the current state of the environment. This textual environment offers internal information that is not available in the visual environment, such as object identifiers corresponding to actions and objects that are not visible in the image. This discrepancy explains the performance gap between text-based and vision-based agents, as observed in Table 1. Figure 6 presents an example of a textual description generated by the environment.

### B. Prompt Definition and Engineering

#### B.1. VLM Prompt Engineering

In VIPER, the perception module (VLM) provides a goal-agnostic description of the current image observation from the environment. The reasoning module (LLM) then uses this description to determine the next action needed to achieve the goal. Therefore, descriptions should be concise, accurate, and free from hallucinations, focusing only on factual elements rather than subjective details like the room's style or atmosphere.

Based on a qualitative evaluation of different prompts using these criteria, we selected $p_{vlm} = "Describe in detail the image"$ when using Florence as VIPER perception module and $p_{vlm} = "Describe in detail the image and all objects present in it."$ when using Idefics in the experiments of Section 4.3. Figure 7 illustrates the descriptions generated by Florence for the same observation with different prompts. Additionally, to reduce hallucinations, we generate textual descriptions using greedy decoding instead of sampling and apply a repetition penalty of value 1.08 and a skip-ngrams penalty with parameter $n = 3$.

#### B.2. LLM Prompt definition

The reasoning module is responsible for selecting the appropriate action based on the goal, the VLM-generated description of the image observation, and past actions. We incorporate past actions based on the findings of the EMMA ablation study [29], which demonstrates that access to past actions is critical to achieving the goal. Specifically, the ALFWorld tasks require exploring the environment to find relevant objects and furniture for the goal. Since the agent is subject to partial observability, recalling past actions is essential for effective decision-making. To balance memory with prompt size, we chose to include a maximum of 5 past actions in our prompt. Building on the work in [2], we use tags for key elements in the prompt—'Goal,' 'Current Observation,' 'Past Action,' and 'Next Action'—with the LLM filling in the final line to propose the next action. An example of this prompt structure is shown in Figure 8.

### C. Training Details and Curves

PPO training Figure 9 and Figure 10 illustrate the evolution of the success rate and episode length during PPO training. We see that online interaction with the environment improves the agent's performance by enabling exploration of new states and interactions. Furthermore, the duration of
<figure><img src="image_7.png" /><figcaption>Figure 6. **Visual and Textual Representations in ALFWorld.** An illustration of an image from the ALFWorld environment alongside its corresponding textual description, which provides detailed observations of the current state.</figcaption></figure>

episodes decreases progressively with interactions, indicat-
ing that the LLM learns more efficient trajectories during
RL training.

For the best combination of VIPER (Florence and Mistral), PPO training improves success rates by 3 to 9 percentage points across tasks. However, as shown in Figure 11, when starting from a weaker initial model trained solely with BC (Florence with LLaMA), the improvement is more significant (improvement between 2pt and 26pt). This suggests that as the performance of the BC-trained model increases, further gains from PPO fine-tuning become smaller, indicating a saturation effect where further fine-tuning yields smaller improvements.

**Results Variability** To assess the robustness of our models, we train with three different seeds on the Examine Light task. The results, presented in Figure 12, show that the models follow a similar trend during training, with a standard error of 3% in success rate and a standard error of 1 step in episode length.

D. Explainability: full interaction scenarios

Figure 13 illustrates how the intermediate text modality
helps explain why the agent succeeds or fails in selecting
the correct action for the goal.

In the first example, the intermediate text description
shows that the VLM accurately identifies and mentions the
objects in the scene, enabling the LLM to successfully com-
plete the task. In contrast, in the second example, the VLM
fails to identify the potato on the table. As a result, the LLM
selects an irrelevant action, since the goal 'put a hot potato
in the fridge' requires finding the potato in the scene, but
the agent mistakenly chooses to interact with the tomato in-
stead. In such examples, the intermediate text description
helps identify a perception mistake from the VLM and a
reasoning mistake from the LLM which should engage in
exploration to find the potato rather than interact with the
tomato or other irrelevant objects for the goal. In the third
example, the text description confirms that the perception
module correctly identifies all objects in the scene, includ-
ing the box, which is relevant for the goal. However, the
selected action is incorrect, choosing to "go to the sofa" de-
spite knowing that a box is present and that the goal is to
"put some box on the dresser", ultimately leading to task
failure. In this case, the selection of an incorrect action
is purely due to a reasoning error from the LLM, which
chooses to "go to the sofa" despite knowing that a box is
present and that the goal is to "put some box on the dresser."

Figure 14 demonstrates how our explainability framework (Section 3.5) allows us to additionally identify the key concepts in the text description that influenced action selection in the reasoning module. Specifically, in these examples, the VLM accurately identifies the objects in the image. The error is therefore attributed to a reasoning mistake from the LLM, rather than a perception error from the VLM, as shown in Figure 14 (f). In (f), the reasoning module ap-
<figure><img src="image_8.png" /></figure>

★ “Describe in detail the image.”

**Description:** This is an animated image. In this image we can see a sink, tap, toilet seat, bathtub, shower, towel, wall and some other objects.

“Describe with a paragraph what is shown in the image..”

**Description:** The image shows a 3D rendering of a bathroom with a white sink and a white toilet. The walls are covered in blue tiles and there is a gold towel rack on the left side of the image. On the right side, there is another white sink with a silver faucet and a green soap dish on the countertop. Above the sink, there are two mirrors hanging on the wall. The overall color scheme of the bathroom is gray and white.

“What does the image shows?”

A bathroom with a sink, toilet and bathtub in it.

<caption>Figure 7. VLM-Generated Descriptions with Different Prompts. Descriptions generated by Florence-2-220M using various prompts. The prompt chosen for VIPER is highlighted in green, as it provides the most concise yet exhaustive description of the image observation.</caption>

**Goal:** Put candle in cart
**Current Observation:** This is an animated image. In this image we can see a toilet seat, flush tank, dustbin, brush and wall.
**Past Action:** go to cart 1
**Past Action:** go to shelf 2
**Past Action:** go to shelf 1
**Past Action:** go to toilet 1
**Past Action:** take candle 1 from toilet 1
**Next action:**

<caption>Figure 8. Example of the Prompt Structure Used by the Reasoning Module (LLM)</caption>

pears to overlook the goal and to incorrectly choose the ac-

<figure><img src="image_9.png" /><figcaption>Figure 9. RL training curves of success rate and episode length for the tasks: Pick and Place, Examine in Light, and Clean and Place.</figcaption></figure>

tion *go to toilet paper hanger 1*— as it holds a toilet paper roll — rather than selecting actions aligned with the goal *put candle in toilet*. In contrast, examples (c) and (d) show the LLM taking actions directly relevant to the goal, as in example (a), as the action *heat [...] cannot be performed while the microwave door is open*. In example (e), the LLM engages in exploration to find relevant objects in the room, making the selected action aligned with the goal.

Beyond failure analysis, our explainability framework enables the identification of key concepts in VIPER intermediate textual descriptions and their corresponding locations in the image observations, as shown in Figure 14. As detailed in Figure 15, we first identify the important concepts in the VLM output description by computing the integrated gradients from the action selected by the LLM to the description generated by the VLM. For each important concept in the textual description (e.g. *egg* in Figure 15), we then generate a heatmap using integrated gradients from the concept to the image. Next, we extract the top *n* connected components from the heatmap, where *n* corresponds to the number of concepts mentioned in the text (e.g. one
<figure><img src="image_10.png" /><figcaption>Figure 10. RL training curves of success rate and episode length: Heat and Place, Cool and place, and Pick two and Place.</figcaption></figure>

for the egg). These components serve as priors for segmentation, and we apply the watershed algorithm to refine object boundaries, ultimately producing a final segmentation mask. The corresponding bounding box is then extracted from this mask.

<figure><img src="image_11.png" /><figcaption>Figure 11. Performance of VIPER with Florence VLM and Llama LLM trained with BC fine-tuning alone and with BC followed by RL fine-tuning, over the six tasks, in terms of mean success rate.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Figure 12. RL training curves of success rate and episode length for the task Examine in Light with three random seeds.</figcaption></figure>
<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /><figcaption>Figure 13. **Failure Analysis with Perception Modules.** The first row (a) shows a successful trajectory, while the next rows (b) and (c) illustrate failure cases. In (a), the perception module provides accurate descriptions for each frame, correctly identifying key elements needed to complete the task (e.g., eggs and microwave, highlighted in green). In (b), however, the agent fails to detect the potato in the scene, leading to incoherent actions that do not align with the goal. Instead, it mistakenly interacts with unrelated objects, such as tomatoes or apples. In (c), although the trajectory is unsuccessful, the text description confirms that the perception module correctly identified and mentioned the box. This suggests that the failure is due to a reasoning error from the LLM rather than a perception issue.</figcaption></figure>
<figure><img src="image_16.png" /><figcaption>Figure 14. Visualization of Key Words and Corresponding Visual Regions in Decision-Making Examples from the *pick and place* (blue) and *heat and place* (green) tasks illustrating VIPER's interaction steps in the ALFWorld environment. Using the integrated gradients method (Section 3.5), we highlight the most relevant words in the VLM-generated description for the selected actions and propagate this information to the images, marking the corresponding visual regions in the observations. In all these examples, the VLM accurately identifies relevant objects for the task, making the reasoning module responsible for the relevance of the actions, labeled as ✅ when correct and ⊗ when incorrect.</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Figure 15. Pipeline for Extracting Segmentation Masks of Key Instances for Decision-Making. Illustration of the step-by-step process used to identify important concepts influencing decision-making and localize their corresponding regions in the image.</figcaption></figure>