SELU: Self-Learning Embodied MLLMs in Unknown Environments

## Abstract

Abstract Recently, multimodal large language models (MLLMs) have demonstrated strong visual understanding and decision-making capabilities, enabling the exploration of autonomously improving MLLMs in unknown environments. However, external feedback like human or environmental feedback is not always available. To address this challenge, existing methods primarily focus on enhancing the decision-making capabilities of MLLMs through voting and scoring mechanisms, while little effort has been paid to improving the environmental comprehension of MLLMs in unknown environments. To fully unleash the self-learning potential of MLLMs, we propose a novel actor-critic self-learning paradigm, dubbed SELU, inspired by the actor-critic paradigm in reinforcement learning. The critic employs self-asking and hindsight relabeling to extract knowledge from interaction trajectories collected by the actor, thereby augmenting its environmental comprehension. Simultaneously, the actor is improved by the self-feedback provided by the critic, enhancing its decision-making. We evaluate our method in the AI2-THOR and VirtualHome environments, and SELU achieves critic improvements of approximately 28% and 30%, and actor improvements of about 20% and 24% via self-learning.

## 1 Introduction

Multimodal Large Language Models (MLLMs) have demonstrated impressive perceptual and understanding capabilities across various domains, *e.g., *web applications (Ma et al., ; Tao et al., ; Liu et al., ), robotics (Xiong et al., ; Li et al., ), gaming (Li et al., ; Qi et al., ; Xu et al., ), and autonomous driving (Wen et al., ; Zhang et al., ). Thanks to their powerful capabilities, many works, *e.g., *Jarvis-1 (Wang et al., ), STEVE-1 (Lifshitz et al., ), and Cradle (Tan et al., ), directly utilize the pre-trained MLLMs to complete various decision-making tasks in different embodied environments.

However, the generalization ability of existing pre-trained MLLMs cannot meet the needs of all environments. For some uncommon environments, embodied MLLMs often exhibit hallucinations and poor visual understanding (Huang et al., ; Jiang et al., ). In more detail, they cannot distinguish left from right and fail to recognize where objects are (Tan et al., ). The reason is that MLLMs have not been further grounded with the environments (Su et al., ; Sun et al., ). Grounding can be realized by fine-tuning on the experiences from interacting with the environments. Based on the evaluation methods, experience can be categorized into three types: human feedback (Dai et al., ; Kirk et al., ), environmental feedback (Tan et al., ; Wang et al., ), and self-feedback (Pang et al., ; Madaan et al., ). The first two types require additional efforts as illustrated in Figure 1(a). Human feedback necessitates expert annotations, which can be costly and influenced by preferences (McAleese et al., ). Environmental feedback assumes we can obtain a dynamics model of the environment and build the reward model (Sontakke et al., ; Urcelay et al., ). Unfortunately, many environments including the real world do not meet such requirements, so these grounding methods are not general. Therefore, we are committed to finding a general way to fill in the remaining gaps.

Figure: (a) External Feedback
Refer to caption: x1.png

Assuming we are facing an unknown embodied environment and cannot obtain external feedback, we can only rely on the capabilities of the MLLM itself. Some work, see Figure 1(b), utilizes the evaluation (discriminative) ability of the pre-trained model itself (Wang et al., ; Huang et al., ) to evaluate its own decision-making, and uses such self-feedback to enhance the model’s decision-making (generative) capabilities.
However, it is easy to see this kind of self-learning is constrained by the static evaluation ability of the model to improve decision-making capability.
Ideally, we hope that self-learning can work similarly to the actor-critic paradigm (Konda & Tsitsiklis, ) in reinforcement learning, where the actor and critic can mutually enhance each other’s performance. If so, the potential for actor enhancement will be greatly expanded. Unlike reinforcement learning, which can obtain external rewards for training, we do not assume that we can get any external feedback. Therefore, we aim to develop a new actor-critic self-learning paradigm for embodied MLLMs in unknown environments.

In this paper, we introduce a novel SElf-Learning paradigm in Unknown environments, dubbed SELU, as illustrated in Figure 1(c). Inspired by the actor-critic paradigm in reinforcement learning, our paradigm learns to simultaneously optimize the MLLM’s ability to understand the environment and to make decisions. For the actor module, we fine-tune the model based on the self-feedback from the critic. As the actor gets improved, it can roll-out more successful trajectories to fine-tune the critic. However, without environmental feedback, the critic may provide inaccurate feedback at the beginning of the training phase, which might mislead the overall optimization. Therefore, we adopt self-asking to correct self-feedback and leverage hindsight relabeling to increase sample efficiency by turning the failure trajectory into a successful one. These high-quality and diverse trajectories are deemed to enhance the critic’s comprehension of the environment. Ultimately, the coupling of these two components mutually promotes the improvement of each other, unleashing the full self-learning potential of MLLMs.

Our key contributions can be summarized as follows:

- •
We propose a self-learning paradigm for embodied MLLMs, SELU, inspired by the actor-critic paradigm in reinforcement learning, which enables MLLMs to self-adapt to unknown environments.
- •
We leverage self-asking and hindsight relabeling to facilitate the improvement of the critic, which greatly increase the sample efficiency and make the self-learning possible.
- •
We demonstrate the effectiveness of SELU in the AI2-THOR and VirtualHome environments, achieving critic improvements of approximately 28% and 30%, and actor improvements of about 20% and 24%, respectively.

## 2 Related Work

### 2.1 MLLMs with External Feedback

In recent years, MLLMs has achieved impressive results across various visual benchmarks (Mathew et al., ; Tang et al., ; Lu et al., ), demonstrating remarkable perception and decision-making capabilities. However, these models still exhibit flaws and often generate unexpected outputs, such as perceptual hallucinations and unreasonable decisions (Yu et al., ; Chen et al., ). Inspired by reinforcement learning, current approaches use external feedback to correct MLLM’s erroneous outputs through a cycle of interaction, feedback, and correction (Pan et al., ; Gero et al., ). Generally, there are two sources of external feedback: human preference feedback and environmental feedback. Utilizing manually annotated data to align MLLMs output with expert preferences (Kirk et al., ; Zhong et al., ), such as DPO (Rafailov et al., ), PRO (Song et al., ), etc., has proven to be effective. Environmental feedback is typically derived from designing a reward function (Pang et al., ; Tan et al., ) or pre-training an evaluation model (Schick et al., ), which often require substantial support from expert data. For instance, McAleese et al. () first trained a critic model using expert data to score program code bugs, and then applied these scores to train ChatGPT with PPO (Schulman et al., ), enhancing its debugging capabilities. Compared to existing studies, we focus on the self-learning potential of MLLMs, as external feedback may be not professional enough in unknown environments.

### 2.2 Self-Improvement in LLMs

Self-improvement in Large Language Models (LLMs) has gained significant attention, as researchers strive to develop models that can learn and adapt from their own outputs, interactions, and internal feedback mechanisms, without relying on external human-labeled data (Yan et al., ; Haluptzok et al., ). Early explorations in this area are based on unsupervised learning techniques, where models learn representations from vast datasets without explicit human guidance (Winter et al., ; Zhao et al., ). Expanded to LLMs, self-improvement goes further by enabling models to critique, refine, and adapt their behavior in a more autonomous manner (Tan et al., ; Choi et al., ). There are two common self-improvement methods: prompt engineering and fine-tuning. The former is an efficient and intuitive approach for large-scale LLMs, as it allows for the establishment of various chains of thought (CoTs) (Wei et al., ) to address the same problem (Huang et al., ; Feng et al., ). For instance, Madaan et al. () demonstrated that GPT-3.5 and GPT-4 can enhance the rationality of responses by simultaneously inputting a question and reflecting on previous answers. The latter one is more useful for small-scale LLMs, as the prompt engineering is unstable. Wang et al. () developed a fine-tuning dataset by generating negative responses to optimize the LLMs, thereby reducing the occurrence of unreasonable answers. Based on these studies, we choose to use fine-tuning to optimize small-scale MLLMs. However, existing methods overlook the enhancement of MLLM’s environment understanding capabilities. Therefore, we employ an actor-critic framework to facilitate comprehensive self-learning in MLLMs, optimizing both perception and decision-making abilities.

## 3 Preliminaries

### 3.1 Actor-Critic in Reinforcement Learning

Actor-critic (Konda & Tsitsiklis, ) is a widely adopted framework in reinforcement learning. The agent consists of two learning modules: an actor and a critic, which are optimized iteratively. The actor selects and executes actions based on current observations. The critic evaluates these observations (and actions) by estimating their values based on reward signals received from the environment, thereby guiding the actor to make improved choices in future. This framework takes advantage of both policy-based learning and value-based learning and is popular nowadays like PPO (Schulman et al., ).

### 3.2 Actor-Critic for MLLMs

With the development of MLLMs, the feedback provided to the agent is no longer constrained to scalar values, like rewards; it can now include diverse modalities, such as natural language (Dong et al., ). This enables the critic gain more specific and informative feedback on the outputs of the actor. Consequently, it can provide more accurate guidance for actor improvement. A prevalent approach in this domain is incorporating human feedback into the critic and building a static evaluation module that can reflect human preference (Ouyang et al., ; Kirk et al., ). Specifically, human annotated data is used to train a critic model (McAleese et al., ) or a reward model (Sontakke et al., ; Wang et al., ) to align the MLLM with human preferences better. More rigorous approaches leverage external evaluation mechanisms, such as tool-interactive learning (Gou et al., ; Chen et al., ), or external knowledge sources like Wikipedia and the Internet (Xu et al., ; Li et al., ). However, regardless of whether preference labels or external tools are used, human intervention remains inevitable. To overcome this reliance, methods like self-consistency (Wang et al., ; Schick et al., ) employ a voting mechanism to enable the model to evaluate its own behavior without relying on external information. However, self-consistency lacks a learnable critic module, thus it cannot improve its grounding knowledge of the environment it interacts with. In contrast to previous work, we propose a novel actor-critic based paradigm aimed at achieving self-learning for both the actor MLLM and critic MLLM, enabling them to iteratively improve decision-making and grounding abilities without external human feedback or environmental rewards.

Figure: Figure 2: The framework of SELU. (lower) The actor MLLM, represented as a robot, collects trajectories for the given instructions. (upper) The critic MLLM, denoted as a brain, evaluates these trajectories and determines whether they complete the tasks, guiding the update of the actor MLLM. In addition, the critic MLLM implements self-asking and hindsight relabeling to build a dataset for optimizing itself. The whole framework does not require any external feedback, such as environmental rewards or human annotations.
Refer to caption: x4.png

## 4 Method

The framework of our method is shown in Figure 2. It consists of two components: the actor MLLM and the critic MLLM. The actor MLLM follows instructions and collects trajectories in the environment. The critic MLLM evaluates the collected trajectories and acquires bootstrapped data via self-asking and hindsight relabeling to optimize itself (Section 4.1). Guided by the success detection results from the critic MLLM, the actor MLLM subsequently improves its decision-making performance in the environment (Section 4.2). By combining the two processes, we can achieve coupled improvements of the critic MLLM and the actor MLLM (Section 4.3).

### 4.1 Critic: Self-Asking and Hindsight Relabeling

As introduced in Section 1, enhancing the interpretation of environmental grounding information is crucial for an MLLM to improve its performance. In our framework, we achieve this objective via self-asking and hindsight relabeling to acquire bootstrapped data for optimizing the critic MLLM.

Specifically, given an instruction $I$, the actor MLLM collects a trajectory by following this instruction. The critic takes the last frame $o_{T}$ of this trajectory as input, using it as the detection frame to determine whether the task depicted by $I$ is completed,

$$ $l_{d}=M_{c}(I,p_{d},o_{T}),$ (1) $$

where $M_{c}$ denotes the critic MLLM, $l_{d}\in\{``yes",``no"\}$ is the result of the success detection, and $p_{d}$ is a prompt for the detection. If the detection result is $l_{d}=``yes"$, we consider this trajectory to be a successful sample for the given instruction and store this trajectory directly into the critic fine-tuning dataset $\mathcal{D}_{\rm critic}$ in the format $(I,p_{d},o_{T},l_{d})$, as shown by trajectory 1 in Figure 2. This trajectory includes environmental grounding information that aligns with the knowledge contained in the critic MLLM.

If the detection result is $l_{d}=``no"$, which means the critic MLLM views this trajectory as a failure for instruction $I$. We first apply self-asking to examine the state of task-related objects, as the decision made by the critic MLLM might not be precise due to potential hallucinations. The critic MLLM is used to obtain the object states,

$$ $l^{\prime}_{d}=M_{c}(l_{s},I),\quad l_{s}=M_{c}(j_{I},p_{s},o_{T})$ (2) $$

where $j_{I}$ is the object name extracted from instruction $I$ by a text-processing function, $p_{s}$ is the prompt for object state analysis, $l_{s}$ is the analysis result, and $M_{c}$ provides a new success detection $l^{\prime}_{d}$ based on $l_{s}$ and $I$. The format is corrected to $(I,p_{d},o_{T},l^{\prime}_{d})$ and will be stored into the critic fine-tuning dataset $\mathcal{D}_{\rm critic}$ if $l_{d}^{\prime}=``yes"$. For example, in trajectory 2 of Figure 2, the critic MLLM initially misjudged the completion of the “open cabinet" task. However, when prompted to focus on the state of the cabinet, it successfully self-corrected its judgment.

If the critic MLLM still considers the trajectory as failure, we propose to use hindsight relabeling to make use of this trajectory, since it might be helpful for learning the environmental grounding of other instructions. Hindsight relabeling is a method that originated in goal-conditioned reinforcement learning (Andrychowicz et al., ). It is based on a simple principle: if a trajectory does not complete the target task, it can be viewed as having accomplished other tasks or subtasks. For example, as shown in trajectory 3 in Figure 2, although it does not complete the task “open cabinet", it successfully completes another task “open drawer". Therefore, we relabel this trajectory with the instruction “open drawer" to help the critic MLLM recognize the completion of the relabeled instruction. We can write this process as,

$$ $I^{\prime}=M_{c}(l_{h},a_{I}),\quad l_{h}=M_{c}(a_{I},p_{h},o_{T})$ (3) $$

where $a_{I}$ is the verb extracted from the instruction $I$ by another text-processing function. $p_{h}$ is the prompt for hindsight relabeling, and $l_{h}$ is the output, which is usually an object name or None. $M_{c}$ checks whether any objects, other than the target object, in the observation $o_{T}$ have completed the task associated with $a_{I}$. $M_{c}$ generates a new instruction $I^{\prime}$ if $l_{h}$ is not None. After that, we store the data $(I^{\prime},p_{d},o_{T},``yes")$ into the critic fine-tuning dataset $\mathcal{D}_{\rm critic}$. Finally, if a failed trajectory proves meaningless after hindsight relabeling, it is considered as not helpful for the MLLM to understand the environment and discarded.

By applying self-asking and hindsight relabeling, we create a fine-tuning dataset $\mathcal{D}_{\rm critic}$ containing the last frames considered as successful by the critic itself and the last frames relabeled as successful after self-asking and hindsight relabeling.

### 4.2 Actor: Critic-Guided Improvement

Recent work has shown that the discriminative ability of an LLM exceeds its generative ability (Pang et al., ). As MLLMs are typically trained the same way as LLMs, we believe that MLLMs’ evaluation abilities would also surpass their generation abilities. For instance, we can easily prompt MLLMs to extract understanding from a given image, but it is challenging to prompt them to choose an appropriate action based on perceived task-relevant information like distance or direction. Our experiment in Section 5.2 also supports this conclusion, where the critic module always performs better than the actor. Therefore, we propose using the critic MLLM to guide the improvement of the actor MLLM in the environment without external feedback.

Specifically, the actor MLLM interacts with the environment and collects online trajectories. At each timestep $t$, the actor generates an action plan $l_{a,t}$ by,

$$ $l_{a,t}=M_{a}(I,p_{a},o_{t}),$ (4) $$

where $M_{a}$ represents the actor MLLM, $I$ is the task instruction, $p_{a}$ is the prompt for action plan and $o_{t}$ is the current image observation. After collecting a whole trajectory, the critic MLLM determines whether this trajectory completes the instruction $I$, as described in Section 4.1. If the answer is yes, we will put the whole trajectory into the actor fine-tuning dataset $\mathcal{D}_{\rm actor}$ with a format of $\{(I,p_{a},o_{t},l_{a,t})\}_{t=0}^{T}$. The relabeled successful trajectories after hindsight relabeling are also added into the actor fine-tuning dataset $\mathcal{D}_{\rm actor}$. Since this dataset only contains task completion trajectories, the actor can quickly converge towards completing tasks in the current environment by fine-tuning on the dataset. Note that the actor fine-tuning dataset $\mathcal{D}_{\rm actor}$ consists of trajectory data, while the critic fine-tuning dataset $\mathcal{D}_{\rm critic}$ only contains the last frames of these trajectories.

### 4.3 Actor-Critic Coupling Improvement

We employ Supervised Fine-Tuning (SFT) (Devlin et al., ; Brown et al., ) and Low-Rank Adaptation (LoRA) (Hu et al., ) to update both the actor and critic MLLMs. Initially, the actor MLLM interacts with the environment to collect online trajectories containing grounding information, as depicted in the lower part of Figure 2. The critic module then evaluates and classifies these trajectories based on the last frame, as shown in the upper part of Figure 2. We select successful trajectories identified by the critic MLLM to create the actor fine-tuning dataset $\mathcal{D}_{\rm actor}$. Subsequently, we utilize the last frame to construct the critic fine-tuning dataset $\mathcal{D}_{\rm critic}$. The update of the actor and critic can be performed iteratively, and make them both improve step-by-step. A detailed pseudo-code for our algorithm is available in Appendix A.1.

## 5 Experiments

### 5.1 Experimental Setup

Environments. In order to simulate embodied MLLM interactions in unknown environments, we select AI2-THOR (Kolve et al., ) and VirtualHome (Puig et al., ) for our experiments. Both environments offer open-ended tasks, various interactive objects, and selectable camera perspectives, facilitating data collection for the actor and critic.

- •
AI2-THOR is an interactive simulation environment designed for embodied AI research. The primary tasks require agents to navigate and interact with household objects. It offers highly realistic 3D environments that simulate kitchens, living rooms, and other indoor settings. AI2-THOR provides locobot and robotic arms as agents and enables agents to perform complex actions like picking up, opening, toggling, and so on.
- •
VirtualHome is also an embodied simulation platform designed to imitate human activities and tasks in home environments. It offers a rich set of virtual household scenes where agents can perform tasks, such as sitting on a bench in the kitchen or grasping a waterglass in the bedroom. This environment focuses on task completion through multi-step action sequences, making it ideal for testing long-term planning.

Task Selection. Considering the training costs while demonstrating the feasibility of the framework, we focus on three typical task categories in the AI2-THOR environment: pick up, open, and break. Locobot is selected as the agent, as this work does not consider low-level control of robotic arms. To ensure task diversity and feasibility, we first prompt the MLLM to explore the environment and use the explore objects to initialize the instruction list which serves as the task set. We randomly sample 2-3 objects for each type of task.
Considering the training costs, we restrict the maximum step for all tasks to 10.
We apply a similar approach in the VirtualHome environment, selecting "female1" as the agent and primarily testing in grab, open, and sit tasks.

MLLMs.
To demonstrate the generalization capability of our framework, we conduct experiments using two MLLMs: LLaVA (Liu et al., ) and Qwen-VL (Bai et al., ).

- •
LLaVA integrates a visual encoder with a language model to effectively process and respond to multimodal inputs. It has gained prominence as one of the most popular MLLMs due to its simple architecture and lower training data requirements. These features enable LLaVA to generate responses more swiftly, and suitable for inference to investigate self-learning.
- •
Qwen-VL is the multimodal version of the large model series. Compared to other open-source MLLMs, Qwen-VL is the first model to use a 448x448 resolution image input. Due to its higher resolution, this model exhibits enhanced visual understanding capabilities. We opt for Qwen-VL with the aim of better success detection, thereby facilitating more efficient self-learning of MLLMs.

Baselines.
We compare SELU with five methods to investigate the feasibility of self-learning of MLLMs in embodied environments:

- •
DG refers to the results obtained through direct generation from the initial MLLM without any fine-tuning.
- •
SC (Wang et al., ) represents an optimization method of MLLMs through self-consistency. Specifically, we employ multiple chains of thought (CoT) to prompt an MLLM to answer the same question, followed by majority voting. In our experiments, we utilize three different CoTs to guide the MLLM, ultimately voting for the most reasonable action.
- •
LMSI (Huang et al., ) is a self-improvement method based on SC. It generates “high-confidence" answers for unlabeled questions to build fine-tuning datasets. This approach enables the LLM to iteratively improve its performance based on the voting mechanism, and we extend this approach to MLLMs.
- •
Self-Refine (Madaan et al., ) involves multiple rounds of self-reflection, followed by self-optimization based on the reflection results. This method focuses on prompt optimization and has been validated for feasibility in large-scale LLMs, such as GPT-4. In our experiments, we reflect 3 rounds to get the final result.
- •
SELU-One represents the method of using the same MLLM to simultaneously perform actor and critic tasks, and fine-tuning with a combination of actor and critic datasets. This approach aims to investigate the feasibility of utilizing a single MLLM to meet the requirements of our framework.

### 5.2 AI2-THOR

LLaVA.
We first demonstrate the effectiveness of SELU in the AI2-THOR environment. After online interactions with the environment and fine-tuning of LLaVA, the critic exhibits an average performance improvement of approximately 27%, while the actor achieves an improvement of around 20% compared to the original model. Table 1 and Table 2 present the accuracy of task success detection and task success rate respectively.

**Table 1: Accuracy of task success detection in the AI2-THOR environment.**
| Method | Pick up | Open | Break | Avg. |
| --- | --- | --- | --- | --- |
| DG | 80.67% | 36.50% | 50.50% | 55.89% |
| SELU-One | 68.67% | 30.50% | 25.50% | 41.56% |
| SELU | 94.33% | 67.50% | 87.50% | 83.11% |

**Table 2: Task success rate in the AI2-THOR environment. SC and Self-Refine use prompt engineering to realize self-learning, whereas LMSI and SELU utilize fine-tuning.**
| Method | Pick Up | Open | Break | Avg. |
| --- | --- | --- | --- | --- |
| DG | 68.33% | 65.00% | 15.50% | 49.61% |
| SC | 65.67% | 68.50% | 17.50% | 50.56% |
| Self-Refine | 69.67% | 70.50% | 14.50% | 51.56% |
| LMSI | 75.67% | 52.50% | 19.50% | 49.22% |
| SELU-One | 91.33% | 85.50% | 27.50% | 68.11% |
| SELU | 94.67% | 83.50% | 30.50% | 69.56% |

In Table 1, it is evidenced that the unified fine-tuning of the actor and the critic (SELU-One) leads to a decline in success detection, even worse than the original critic. Although SELU-One achieves a task success rate comparable to that of SELU as shown in Table 2, the compromised critic will result in SELU-One incorrectly analyzing the trajectory in subsequent epochs.

Table 2 demonstrates that baselines are not suitable for the self-learning of embodied MLLMs. Both prompt-engineering and fine-tuning baselines struggle to improve the decision-making ability of the MLLM. The reason is that the embodied MLLM cannot give a correct task detection with a lack of environmental understanding. As we can see in Table 1, the initial judgment of the MLLM (DG) on tasks is only about 55%. In this case, merely optimizing the prompt to create multiple CoTs for repeated reflection does not help the MLLM gain task achievement details in an unknown environment. Therefore, neither SC nor Self-Refine can substantially enhance the task success rate. For fine-tuning baselines, relying on statistical voting to validate its own behavior even leads to worse performance. For instance, in the Open task, for LMSI the task detection accuracy of 36.5% causes the actor’s performance to drop from 65% to 52.5% after fine-tuning. These results demonstrate the necessity of the critic module in SELU, and optimizing the critic is crucial for enhancing the actor’s performance.

Notably, in the Open task, we can observe a low accuracy of success detection for SELU; however, it still helps the actor improve. This highlights the role of hindsight relabeling, which will be discussed in detail in Section 5.4.

Qwen-VL.
In order to prove that SELU can help different MLLMs achieve self-learning, we select Qwen-VL and test it in the AI2-THOR environment under the same setting. The result is shown in Table 3, which indicates that SELU can help Qwen-VL improve the task evaluation capability by about 24% and the decision-making performance by approximately 23%.

**Table 3: Self-learning performance of SELU on Qwen-VL in the AI2-THOR environment.**
| Task | Critic |  | Actor |  |  |
| --- | --- | --- | --- | --- | --- |
|  | DG-Qwen-VL | SELU-Qwen-VL |  | DG-Qwen-VL | SELU-Qwen-VL |
| Pick Up | 73.33% | 95.67% |  | 57.67% | 95.33% |
| Open | 51.00% | 81.50% |  | 46.50% | 68.00% |
| Break | 63.50% | 83.50% |  | 12.50% | 21.50% |
| Avg. | 62.61% | 86.89% |  | 38.89% | 61.61% |

### 5.3 VirtualHome

We then conduct experiments in the VirtualHome environment, which incorporates a greater variety of items and human agents, thereby enriching the experimental environments to demonstrate the effectiveness of our method. The experimental results are presented in Tables 4 and 5. In this environment, SELU enhances LLaVA task evaluation capability by approximately 30% and improves decision-making performance by around 24%, and it also outperforms baselines. As the environment becomes more complex, the lack of environmental understanding causes SC and Self-Refine to negatively impact the decision-making of the original embodied MLLM, as the performance of SC and Self-Refine are even lower than DG, shown in Table 5.

**Table 4: Accuracy of task success detection in the VirtualHome environment.**
| Method | Grab | Open | Sit | Avg. |
| --- | --- | --- | --- | --- |
| DG | 52.67% | 35.33% | 44.50% | 44.17% |
| SELU-One | 45.33% | 15.67% | 48.50% | 36.50% |
| SELU | 93.67% | 83.33% | 47.50% | 74.83% |

**Table 5: Task success rate in the VirtualHome environment.**
| Method | Grab | Open | Sit | Avg. |
| --- | --- | --- | --- | --- |
| DG | 65.00% | 83.33% | 56.50% | 68.28% |
| SC | 52.67% | 81.67% | 61.50% | 65.28% |
| Self-Refine | 59.67% | 74.33% | 60.50% | 64.83% |
| LMSI | 35.67% | 93.67% | 52.50% | 60.61% |
| SELU-One | 83.67% | 98.67% | 94.50% | 92.28% |
| SELU | 93.33% | 97.67% | 93.50% | 94.83% |

### 5.4 Ablation Study

We conduct ablation experiments on the critic module and hindsight relabeling in the AI2-THOR environemnt, and the results are shown in Table 6. SELU w/o HR means that we do not perform hindsight relabeling to reanalyze the trajectory when evaluating the task. SELU w/o critic means that we remove both self-asking and hindsight relabeling and use the data obtained from environment interaction to directly fine-tune the actor. In this case, the evaluation result of the critic is derived from the original MLLM.

**Table 6: Ablation study in the AI2-THOR environment.**
| Task | Critic (Success Detection Accuracy) |  | Actor (Task Success Rate) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SELU | w/o HR | w/o critic |  | SELU | w/o HR | w/o critic |  |
| Pick Up | 94.33% | 83.67% | 80.67% |  | 94.67% | 67.33% | 56.33% |
| Open | 67.50% | 31.50% | 36.67% |  | 83.50% | 66.50% | 72.50% |
| Break | 87.50% | 83.50% | 50.50% |  | 30.50% | 27.50% | 17.50% |
| Avg. | 83.11% | 66.22% | 55.95% |  | 69.56% | 57.11% | 48.78% |

By comparing SELU and SELU w/o critic, we can see the importance of critic clearly. Only by understanding the environment can we achieve the improvement of decision-making in all tasks. By comparing SELU w/o HR and SELU w/o critic, we find that self-asking can correct the critic’s comprehension of the environmental task, but reflection on a single task is not enough. In Open tasks, we find that the lack of hindsight relabeling directly leads to the disappearance for some instructions, which causes the declined performance of success detection and decision-making. We can observe the improvement from SELU w/o HR to SELU. By incorporating hindsight relabeling, we can perform a comprehensive multi-task evaluation for each trajectory, ensuring that the embodied MLLM achieves self-learning on each task. Consequently, self-asking and hindsight relabeling are essential components of the critic.

### 5.5 Hyperparameter Analysis

Online Dataset Size.
Since the MLLMs fine-tuning process is sensitive to the dataset size, we explore the amount of interaction data required to achieve effective learning for embodied tasks. We conduct multiple tests on this variable based on picking up tasks in the AI2-THOR environment. The results are presented in Figure 3(a).

The size of dataset is described by the number of trajectories for a single task. For instance, dataset-10k indicates that 10k trajectories are collected for a specific task, such as picking up an apple, during online interactions. We evaluate the performance through the actor in terms of task success rate. We can see that the performance of dataset-1k is close to that of dataset-10k. For dataset-10k, it takes about two days to collect one epoch data. To balance experimental performance with sampling efficiency, we opt to sample 1k trajectories per task in our experiments.

Based on this conclusion, we employ a data augmentation method during the actor training process. Since MLLMs tend to prioritize text over images when making decisions, they often focus excessively on the text prompt and overlook image comprehension. To address this issue, we shuffle the action lists in the prompts of training data to provide multiple prompts for the same image. This approach not only increases the dataset size, but also strengthens the connection between the MLLM policy and the observations.

Learning Rate.
The learning rate is a critical factor, which prevents us from using the same MLLM for the actor and critic. We test different learning rates for the actor and critic separately on picking up tasks in the AI2-THOR, and the results are shown in Figure 3(b). We find that, due to the varying sizes of the fine-tuning datasets, using a uniform learning rate inevitably leads to overfitting or underfitting in one of the components, thereby impacting the overall performance of SELU. Therefore, we ultimately use two MLLMs to construct the SELU framework, ensuring effective self-learning. In our experiments, the learning rate for the critic is set to 2e-6, while that for the actor is set to 2e-5.

Figure: (a) Dataset Size
Refer to caption: x5.png

Training Iterations.
The goal of our framework is to achieve multi-iteration self-learning improvement; therefore, we also evaluate the results of multiple rounds of fine-tuning on picking up tasks in the AI2-THOR environment. The results are presented in Figure 3(c).

Our results indicate that multiple iterations of fine-tuning do not consistently improve SELU’s performance in every iteration. Both the actor and critic exhibit significant performance improvements during the first iteration of fine-tuning, but show fluctuations and minimal growth in the subsequent iterations. We attribute this limitation to the performance of the critic. The critic performs the success detection based on the last frame of the trajectory, making it difficult to compare the quality of successful trajectories. Once the actor reaches a level sufficient to roughly complete the task, we lack the nuanced supervisory signals to guide the actor for further improvement. Consequently, while there is a notable improvement after the first iteration, subsequent enhancements are limited. We also attempt to increase the number of frames for success detection. However, the performance of the small-scale MLLM does not meet our needs.

## 6 Conclusion and Limitation

In this paper, we introduce SELU, a method for MLLMs to achieve self-learning in unknown environments. SELU facilitates interaction with the environment, analyzes interaction trajectories, builds an online dataset, and performs coupled optimization of the actor and critic. We employ self-asking and hindsight relabeling to enhance the critic task evaluation capabilities. Ablation experiments demonstrate that relabeling significantly expands the critic task judgment range. By leveraging the principle that MLLMs possess stronger perceptual abilities than decision-making abilities, we improve the performance of the actor policy. We test SELU in the AI2-THOR and VirtualHome environments, achieving critic improvements of approximately 28% and 30%, and policy improvements of about 20% and 24%, respectively. Additionally, to validate the applicability of SELU across different MLLMs, we evaluate it on Qwen-VL, resulting in a 23% performance enhancement.

One limitation of SELU is the lack of a detailed evaluation of the trajectories that complete the tasks. SELU can help embodied MLLM to self-learn how to accomplish tasks in unknown environments. The next urgent problem to address is how to complete tasks more efficiently and optimize actions more effectively. In future work, we will explore finer-grained critic signals to perform more accurate quality assessments of trajectories, guiding embodied MLLMs to tackle more complex tasks, like long-horizon combination tasks.

## Appendix A Appendix

### A.1 Pseudocode of SELU

UTF8gkai

Figure: algorithm 1 SELU

### A.2 Implementation Details

#### A.2.1 Environment Details

Figure 4 shows our experiment environments. Both environments restrict agents to only interact with visible items, limiting their operational range to guarantee behavior plans realistic. Therefore, the actor MLLM makes decisions based on first-person perspective input to ensure accuracy as Figure 4(a) and Figure 4(c) show. Given the limitations of the first-person view, the critic MLLM uses a third-person perspective to evaluate the trajectory, reducing hallucinations and obtaining accurate scene information as Figure 4(b) and Figure 4(d) show.

The positioning of the third-person camera is crucial, as it should accurately capture the agent’s position and the objects it interacts with. Any occlusion or interference can impair the MLLM’s understanding of the image, thereby affecting the results of critic success detection and hindsight relabeling.

Figure: (a) AI2-THOR 1
Refer to caption: x8.png

#### A.2.2 MLLM Details

The specific MLLMs we use are LLaVA-V1.6-Mistral-7B and Qwen-VL (Qwen-7B). We use LoRA to fine-tune them, the hyperparameters are as follows.

**Table 7: Hyperparameters of LLaVA fine-tuning by LoRA**
| Hyperparameters | Value |
| --- | --- |
| train_batch_size | 16 |
| eval_batch_size | 4 |
| gradient_accumulation_steps | 1 |
| learning_rate_actor | 2e-5 |
| learning_rate_critic | 2e-6 |
| warmup_ratio | 0.03 |
| weight_decay | 0.0 |
| model_max_length | 2048 |
| lr_scheduler_type | "cosine" |
| tf32 | True |

**Table 8: Hyperparameters of Qwen-VL fine-tuning by LoRA**
| Hyperparameters | Value |
| --- | --- |
| train_batch_size | 2 |
| eval_batch_size | 1 |
| gradient_accumulation_steps | 8 |
| learning_rate_actor | 1e-5 |
| learning_rate_critic | 1e-6 |
| warmup_ratio | 0.01 |
| weight_decay | 0.1 |
| adam_beta2 | 0.95 |
| model_max_length | 2048 |
| lr_scheduler_type | "cosine" |
| bf16 | True |
| lazy_preprocess | True |

### A.3 Visualization of Actor and Critic on LLaVA in AI2-THOR

We use the embodied actor MLLM to interact with the unknown environment, and collect trajectories for evaluation from the critic MLLM. An example for ’pick up the lettuce’ is as follows.

Figure: Figure 5: A visualization of the actor MLLM interacting with the AI2-THOR environment. The agent is instructed to pick up the lettuce. As the lettuce is far away, the agent needs to move closer before attempting to pick it up.
Refer to caption: x12.png

We use the critic MLLM to perform success detection on each trajectory and use self-asking and hindsight relabeling techniques to build bootstrapped dataset. An example for ’break the mug’ is as follows.

Figure: Figure 6: A visualization of the critic MLLM conducting success detection on a trajectory. Since the trajectory has completed the ’break the mug’ task, it can be added to the fine-tuning dataset.
Refer to caption: x13.png

### A.4 Prompts for SELU

The specific prompts we use for SELU are:

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: See above.
soundness: 2
strengths: - The paper is clearly presented and easy to follow.
- The idea is straightforward and seems effective.
- Strong and competitive performance.
summary: This submission proposes SELU, an implementation of the actor-critic idea into MLLMs. This framework consists of a critic MLLM and an actor MLLM. They are trained and improved iteratively with online environment interactions. Experiments on VirtualHome and AI2-THOR demonstrate competitive performance.
weaknesses: - The proposed framework is not novel. At its core it's a combination of different techniques which are originally adopted under RL context, such as the actor-critic framework and hindsight relabeling.
- Although authors claim the critic learning without any external feedback (e.g., reward) as a strength, I doubt the correctness of resulting critics. For actor-critic methods under RL context, the critic is usually updated with L2 loss between its value estimation and groundtruth. Otherwise, it can easily suffer from issues such as overestimation and hence hinder the actor learning. I can imagine the situation will become worse for MLLMs due to their hallucination.
- The submission claims "unknown environments". However, it's really not unknown per se. The evaluated environments are mostly indoor scenes, which are very common and familiar to LLMs. More practically, at L299 authors mention that MLLMs are first prompted to explore the environment, which really makes the tested environments not unknown.
- Evaluated tasks are too short-horizon with the maximum horizon being 10. Since the low-level action execution is abstracted away to MLLMs, ideally authors should compare on long-horizon tasks.
- Some details need to be clarified. For example, in table 6, what does it mean to evaluate critic but without critic (the third column)?

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - **1** - How does your actor-critic method compare to the other actor-critic approaches for LLMs [1,2]? What are the main differences between SELU and these works (beyond just the use of multiple modalities)?
- **2** - In Section 4.1. (page 4), the authors state that if the detection result of a trajectory is a "yes" then the trajectory is directly added to the critic fine-tuning dataset. But what if that "yes" is, itself, an hallucination from the MLLM? How can your model cope with incorrect data in the training dataset?
- **3** In Section 4.1. (page 5), the authors describe their hindsight relabeling procedure and state that the they extract the "verb" of the action directly from the instruction. What if the agent performs another action (due to, for example, exploration of his environment) in that trajectory?
- **4** - What exactly is an action plan $l_{a, t}? It appears to not be defined in the paper.
- **5** - Why are some of the baselines (SC, Self-Refine, LMSI) missing from Table 3 and Table 4? What are the performance of these methods in AI2-THOR and VirtualHome, respectively?
- **6** - Why does SELU not outperform the baselines in task success detection on the "Sit" task in the VirtualHome environment? There is no discussion of this result in the current version of the work.
- **7** - In Section 5.4., the authors claim that "In Open tasks, we find that the lack of hindsight relabeling directly leads to the disappearance for some instructions, which causes the declined performance of success detection and decision-making.". Can you clarify what this statement means?
- **8** - What is the size of the replay buffer/dataset collected by each agent across the evaluations of Sections 5.2 and Sections 5.3? Is it the same for all baselines?
- **9** - How exactly you evaluate the task success detection metric for the baselines? Do you employ the same test dataset for every baseline? If not, how can you make sure that the performance of the critics is not biased towards easier task succession cases for some models? How many evaluation episodes did you consider? Is it the same for every model?
- **10** - In Section 5.5. (page 10) the authors claim that "our results indicate that multiple iterations of fine-tuning do not consistently improve SELU's performance". Did you collect a novel dataset after each iteration? Or did you reuse the same dataset?
- **11** - Are the MLLMs (LLaVA and Qwen-VL) pre-trained? If so, on what datasets and where did you get the weights? If not, how did you train the MLLMs?
- **12** - Can the authors clarify what is a "step" in this environment? Additionally, what actions are available to the agent? What actions do the agents actually perform on the tasks?
- **13** - How do the authors promote the "exploration" of the environment by the agents? How many steps of exploration are available to each agent? Is the instruction list/objects for each task the same for all agents?

References:
- [1] Zhang, Bin, et al. "Controlling Large Language Model-based Agents for Large-Scale Decision-Making: An Actor-Critic Approach." ICLR 2024 Workshop on Large Language Model (LLM) Agents.
- [2] Kim, Byoungjip, et al. "Prospector: Improving LLM Agents with Self-Asking and Trajectory Ranking." NeurIPS 2023 Foundation Models for Decision Making Workshop.
soundness: 3
strengths: - **Originality**:
  - As highlighted by the authors in their related work section, there has been extensive work on embodied MLLMs in recent years. The authors distinguish their work by focusing on the self-improvement of these models, instead of relying on (expensive) human feedback signals or environmental rewards. In particular, the authors propose a new architecture that takes inspiration from the actor-critic method in reinforcement learning. Actor-critic methods have been extensively explored in the context of RL (e.g., [1], [2]). Actor-critic methods have also been explored in the past for LLM agents, (e.g., [3,4]), which reduces the novelty of the approach. Nonetheless, the use of hindsight relabelling to improve the efficiency of the training of the critic in MLLMs is, to the best of my knowledge, novel.

- **Quality**:
   -  The authors successfully explain in detail their actor-critic approach and positioning against other recent works in embodied MLLMs. The authors also extensively evaluate their approach against relevant baselines, considering multiple MLLM backbones and in multiple simulation environments. Additionally, the authors present an ablation study and hyperparameter analysis that give further support to the claims of the paper and the importance of both the self-reflexive critic and the hindsight relabeling proposed method.

- **Clarity**:
   - The current version of the work is quite clear (except some sentences, as discussed in "Questions") and without any major typos. The authors clearly describe their proposed approach. The figures and tables also also of high-quality and are easily interpretable. Same details in regards to the experimental evaluation are currently missing (see "Questions").

- **Significance**:
   - The authors highlight a significant improvement of the ability of MLLMs to act in unseen environments and the experimental results appear to support the claims of the paper. As such, this work can be of some significant value to practitioners in the field.

**References**: 

- [1] Schulman, John, et al. "Proximal policy optimization algorithms." arXiv preprint arXiv:1707.06347 (2017).
- [2] Haarnoja, Tuomas, et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." International conference on machine learning. PMLR, 2018.
- [3] Zhang, Bin, et al. "Controlling Large Language Model-based Agents for Large-Scale Decision-Making: An Actor-Critic Approach." ICLR 2024 Workshop on Large Language Model (LLM) Agents.
- [4] Kim, Byoungjip, et al. "Prospector: Improving LLM Agents with Self-Asking and Trajectory Ranking." NeurIPS 2023 Foundation Models for Decision Making Workshop.
summary: In this work the authors contribute a self-improvement technique for embodied MLLMs, reminiscent of the actor-critic architecture in reinforcement learning, that does not rely on external human feedback or environmental rewards. Their proposed method (SELU) considers two separate MLLMs: an actor MLLM, responsible for generating action plans for the agent, and a critic MLLM, responsible for evaluating if trajectories are successful. The authors also propose to use hindsight relabelling on failed trajectories of the agent in order to improve the efficiency of the training of the critic. The authors evaluate SELU in two virtual environments across multiple physical tasks (e.g., pick up, open, break) and using two different MLLM models. The authors compare their approach against recently proposed methods for self-improvement of MLLMs and show a significant improvement in both task success evaluation (critic) and decision-making (actor). The authors also show an ablation study on the role of hindsight relabeling and self-asking for the performance of the method, as well as an hyperparameter analysis.
weaknesses: My main concerns with the current version of the work are the following:
- In the current version of the work, significant details regarding the evaluation are missing (see "Questions" for details). There is no description of the evaluation methodology of SELU and the baselines, which would fit quite naturally as another subsection of Section 5.1. As such, it is hard to understand what the evaluation metrics are and how they were computed, and thus the full significance of the results.
- The proposed method employs only a self-reflective supervision signal, i.e., the model itself generates the reward signals to train the policy of the agent (in the form of selecting the successful/unsuccessful trajectories to further improve the MLLMs). However, without any other external reward signal, I'm not convinced that SELU can be robust to potential hallucinations (see "Question" 2).
- No computational code to replicate the results was provided.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What do the error bars represent in Figure 3? Similarly, do you have uncertainty measurements (N/bootstrapped confidence interval) for those numbers reported in the tables? Adding those will help me or other readers interpret the numbers. 
2. In line 301, "we restrict the maximum step for all tasks to 10." Do all trajectories end with 10 steps, or are some trajectories shorter than 10 steps? I wonder whether ending early mixes in an implicit environment feedback of task success because the critic is conditioned only on the last frame.
soundness: 3
strengths: 1. The paper is clear and well-presented.
2. The experiments are thorough, and the ablation studies are informative. The authors also provide helpful analysis of the limitations and error modes.
3. The proposed method to couple comprehension and decision-making in an unfamiliar environment is interesting.
summary: Multimodal large language models (MLLMs) require external feedback from humans or the environment to improve decision-making capabilities in unknown environments. The authors propose a new self-improvement actor-critic-based method, primarily consisting of training the critic, prompting (self-asking and hindsight relabeling) for labels, and curating data for SFT the actor based on labels by the critic. This coupled method improves both environment comprehension and decision-making in the new environment. The authors ran experiments on AI2-THOR and VirtualHome environments with both LLAVA and Qwen-VL, with baseline comparisons and additional ablation studies. The results are promising.
weaknesses: Just a few questions below.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. For the environments, the author mentioned they “randomly sampled 2-3 objects for each type of task”. Can you further provide the details? Whether the object will be randomly replaced at the beginning of each epoch or not? How large is the internal state space, and action space of the environment? These information is useful to evaluate the performance of the agent.

2. the Figure 2 seems a little confusing to me. What does the yellow circles mean? what does the arrows in the critic part mean? why different trajectories are added in sequence? 

3. how do you perform the hindsight relabeling and find the actual task finished? In fig2, it seems the question to distinguish whether a trajectory is useful is to ask "which object is open", which seems to be hand-crafted and cannot be adapted to more complex tasks.

4. How long does it take to collect the dataset and updating for each experiment? By fine-tuning on so many data which is narrowly distributed, whether the MLLM will lost its capacity on general knowledge? Can you discussed the advantages and drawbacks of your method over RAG based method, which seems to be ignored in the related work section.

5. The proposed method SELU interact with the environment to collect 1k trajectories, while baseline (e.g., self-refine) is reported to interact 3 rounds before getting the final answer. How many data are LMSI used? I think it's important to ensure a fair comparison between baselines in terms of the number of interactions.

6. other methods that improves the LLM (MLLM) according to the interactions with the environments are omitted.
soundness: 2
strengths: The paper is well written. It is also a good idea to combine the topics like LLM-based agent and actor-critic style update.
summary: This paper proposes a new actor-critic self learning paradigm, i.e., SELU, for embodied MLLMs in unknown environments. SELU proposes to update both the actor and the critic, via in actor-critic fashion by self-asking. the authors evaluate the efficiency of the proposed method, in 2 simulation environments including AI2thor and virtual home.
weaknesses: The proposed method seems to be simple and not technically contributing. The experiments environments seem to be limited, some settings remain unclear to me (see below).