SELU: Self-Learning Embodied Multimodal Large Language Models in Unknown Environments
Authors: Boyu Li
Haobin Jiang
Ziluo Ding
Xinrun Xu
Haoran Li
Dongbin Zhao
Zongqing Lu
Venue: Accepted by TMLR ()

Abstract: Recently, multimodal large language models (MLLMs) have demonstrated strong visual understanding and decision-making capabilities, enabling the exploration of autonomously improving MLLMs in unknown environments. However, external feedback like human or environmental feedback is not always available. To address this challenge, existing methods primarily focus on enhancing the decision-making capabilities of MLLMs through voting and scoring mechanisms, while little effort has been paid to improving the environmental comprehension of MLLMs in unknown environments. To fully unleash the self-learning potential of MLLMs, we propose a novel actor-critic self-learning paradigm, dubbed SELU, inspired by the actor-critic paradigm in reinforcement learning. The critic employs self-asking and hindsight relabeling to extract knowledge from interaction trajectories collected by the actor, thereby augmenting its environmental comprehension. Simultaneously, the actor is improved by the self-feedback provided by the critic, enhancing its decision-making. We evaluate our method in the AI2-THOR and VirtualHome environments, and SELU achieves critic improvements of approximately 28% and 30%, and actor improvements of about 20% and 24% via self-learning.

Published in Transactions on Machine Learning Research (09/2025)
SELU: Self-Learning Embodied Multimodal Large Language Models in Unknown Environments
Boyu Li Haobin Jiang', Ziluo Ding", Xinrun Xu², Haoran Li
Dongbin Zhaot:2, Zongqing Lu
Institute of Automation, Chinese Academy of Sciences
School of Artificial Intelligence, University of Chinese Academy of Sciences
Beefinng Academy of Artificial Intelligence
School of Computer Science, Peking University
Institute of Software, Chinese Academy of Sciences
Reviewed on OpenReview: https: // openreview. net/ bet/forunt forum? id= G5gROBBAV
Abstract
Recently, multimodal large language models (MLLMs) have demonstrated strong visual understanding and decision-making capabilities, enabling the exploration of autonomously improving MLLMs in unknown environments. However, external feedback like human or environmental feedback not. always available. To address this challenge, existing methods primarily focus on enhancing the decision-making capabilities of MLLMs through voting and scoring mechanisms, while little effort has been paid to improving the environmental comprehension of MLLMs in unknown environments. To fully unleash the self-learning potential of MLLMs, we propose a novel actor-critic self-learning paradigm, dubbed SELU, inspired by the actor-critic paradigm in reinforcement learning. The critic employs self- asking and hindsight relabeling to extract knowledge from interaction trajectories collected by the actor, thereby augmenting its environmental comprehension. Simultaneously, the actor improved by the self-foedback provided by the critic, enhancing decision-making We evaluate our method in the A12-THOR and VirtualHome environments, and SELU achieves critic improvements of approximately 28% and 30% and actor improvements of about 20% and 24% via self-learning
1 Introduction
Multimodal Large Language Models (MLLMs) have demonstrated impressive perceptual and understanding capabilitics across various domains, e.g., web applications (Ma et al., 2024; Tao et al., 2024; Liu et al., 2024), robotics (Xiong et al., 2024; Li et al., 2024c), gaming (Li et al., 2024d; Qi et al., 2024; Xu et al., 2024), and autonomous driving (Wen et al., 2024; Zhang et al., 2024). Thanks to their powerful capabilities, many works, 6.9., , Jarvis-1 (Wang et al., 2023b), STEVE-1 (Lifshitz et al., 2023), and Cradle (Tan et al., 2024b), directly utilize the pre-trained MLLMs to complete various decision-making tasks in different embodied environments.
However, the generalization ability of existing pro-trained MLLMs cannot meet the needs of all environ- ments. For some uncommon environments, embodied MLLMs often exhibit hallucinations and poor visual understanding (Huang et al., 2024; Jiang et al., 2024a). For example, they cannot distinguish left. from right or fail to recognize where objects are (Tan et al., 2024b). The reason is that MLLMs have not been further grounded with the environments (Su et al., 2022; Sun et al., 2024b). Grounding can be realized by fine-tuning on the experiences from interacting with the environments. Based on the evaluation methods, experience can be categorized into three types: human feodback (Dai et al., 2024; Kirk et al., 2024), environ- mental feedback (Tan et al., 2024a; Wang et al., 2024b), and self-feedback (Pang et al., 2024; Madaan et al.,
"Correspondence Zongging Exangging.nn@pkku.uu...o

---

actor-critic Published in Transactions on Machine Learning Research (09/2023)
Memory Buffer
Environment
Memory Buffer
Environment
Memory Buffer
Environment
8008
Vote
Human
Environment
Feedback
Feedback
MLLM Actor
MLLM Actor
MLLM Critic
MLLM Actor
(a) External Feedback
(b) Static Self- Felf-Feedback
(c) Our Self-Feedback
Figure 1: Comparison of our framework with other frameworks in terms of the feedback type.
2023). The first two types roquire additional efforts as illustrated in Figure la. Human feodback requires expert demonstrations and annotations, which can be costly and influenced by individual biases (McAleese et al., 2024). Environmental feedback assumes we can obtain dynamics model of the environment or a well-designed reward model (Fan et al., 2022; Urcelay et al., 2024; Jiang al., 2024b). Unfortunately, many environments including the real world do not meet such requirements, these grounding methods are not general. Therefore, we are committed to finding general way to fill in the remaining gaps.
When operating in an unknown embodied environment without external feedback, the MLLM must rely solely on its inherent capabilities. An unknown environment refers to one in which the MLLM has neither been exposed to its visual data during pretraining, nor fine-tuned on its task-specific data to achieve ground- ing. Some work, see Figure 1b, utilizes the evaluation (discriminative) ability of the pre-trained model itself (Pang et al., 2024) or statistics like voting (Wang et al,, 2023a; Huang et al., 2023) to evaluate its own decision- decision-making, and uses such self- felf-feedback to enhance the model's decision decision-making (generative) capabili- ties. However, it is evident that this form of self-learning is limited by the model's static evaluation capacity, restricting its ability to enhance decision-making. Ideally, we hope that self-learning can work similarly to the actor-critic paradigm (Konda & Tsitsiklis, 1999) in reinforcement learning for embodied lasks, where both the actor and critic iteratively refine their performance. If so, the potential for actor enhancement can be greatly expanded. Unlike reinforcement learning, which relies on external rewards for training, we do not assume the access to any external feedback. Therefore, we aim to develop self-learning paradigm for embodied MLLMs in unknown environments.
In this paper, we introduce a novel SElf-Learning paradigm in Unknown environments, dubbed SELU, as illustrated in Figure Ic. Inspired by the actor-critic paradigm in reinforcement learning, our paradigm learns to simultancously optimize the MLLM's ability to understand the environment and to make decisions. For the actor module, we fine-tune the model based on the self-feedback from the critic. As the actor gets improved, it can roll-out more successful trajectories to fine-tune the critic. However, without environmental feedback, the critic may provide inaccurate feedback at the beginning of the training phase, which might mislead the overall optimization. Therefore, adopt self-asking to correct self-feedback and leverage hindsight relabeling to increase sample elficiency by turning the failure trajectory into successful one for other tasks. These high-quality and diverse trajectories are deemed to enhance the critic's comprehension of the environment. Ultimately, the coupling of these two components mutually promotes the improvement of cach other, unleashing the full self-learning potential of MLLMs.
Our key contributions can be summarized follows:
We propose SELU, a self-lcarning paradigm for embodied MLLMs, inspired by the actor-critic paradigm in reinforcement learning, which enables MLLMs to improve themselves in unknown environments without external feedbacks.
We leverage self-asking and hindsight relabeling to achiove the improvement of the critic, which greatly increase the sample efficiency of our algorithm and make the self-learning possible.
We demonstrate the effectiveness of SELU A12-THOR and VirtualHome, achieving critic im- provements of approximately 28% and 30,, and actor improvements of about 20% and 24%, respectively.
2

---

Published in Transactions on Machine Learning Research (09/2025)
2 Related Work
2.1 MLLMs with External Feedback
In recent years, MLLMs has achieved impressive results across various visual benchmarks (Mathew el al., 2021; Tang et al., 2024; Lu et al., 2024b), demonstrating remarkable perception and decision-making capa- bilitics. However, these models still exhibit flaws and often generate unexpected outputs, such as perceptual hallucinations and unreasonable decisions (Yu et al., 2024; Chen et al., 2024b). Inspired by reinforcement learning, current approaches use external feedback to correct MLLM's erroneous outputs through a cycle of interaction, feedback, and correction (Pan et al., 2024; Gero et al., 2023), RL methods such as PPO (Schulman et al., 2017) and GRPO (Shao et al., 2024) have proven effective (Sun al., 2024a; Zhai et al., 2024; Lu et. al., 2024a). Generally, there are two sources of external feedback: human preference feedback and environmental feedback. Human preference data is used to train reward model aligned with human preferences (Ouyang et al., 2022). In contrast, environmental feedback typically stems from a rule-based re- ward function (Tan et al., 2024a; Guo et al., 2025) or a pre-trained large model (Rocamonde et al., 2023; Lee et al., 2024), which often requires substantial support from expert data. For instance, Konyushkova et al. (2025) help Gemini 1.5 to understand Dialog Games through a rule-based approach, generate a positive evaluation function, and construct a high-quality dataset to improve its image understanding capabilities. Gaven et al. (2024) uses the SAC (Gaven et al., 2024) and sparse context reward from the environment improve the plan ability of LLMs in text-based houschold environments. Compared to cxisting studics, we focus on the self-learning potential of MLLMs, as external leedback may be not professional or available in unknown environments.
2.2 Seff-Improvement in LLMs
Self-improvement in Large Language Models (LLMs) has gained significant attention, as researchers strive to develop models that can learn and adapt from their own outputs, interactions, and internal feedback mechanisms, without relying on external human-labeled data (Yan et al., 2023; Haluptzok et al., 2023). Farly explorations in this area are based on unsupervised learning techniques, where models learn representations from vast datasets without explicit human guidance (Winter et al., 2022; Zhao et al., 2019). Expanded to LLMs, self- imfroveproeetttt goes further by enabling models to critique, refine, and adapt their behavior in a more autonomous manner (Tan et al., 2023; Choi et al., 2024). There are two common self-improvement methods: prompt engineering and fining. The former is an efficient and intuitive approach for large- scale LLMs, as it allows for the establishment of various chains of thought (CoTs) (Wei et al., 2022) to address the same problem (Huang al., 2023; Feng et al., 2023). For instance, Madaan et al. (2023) demonstrate that LLMs enhance the rationality of responses by simultancously inputting question and reflecting previous answers. The latter one is proposed as the prompt engineering is unstable for small- scale LLMs, and shows limitation to solve long-horizon tasks (Huang et al., 2023; Pang et al., 2024; Wang et al., 2024a). Wang et al. (2024a) develop a fine-tuning dataset by generating synthetic negative responses to optimize the LLM evaluator, while other work explores how to improve the instruction-followin abilities of L.LMs (Huang et al., 2023; Pang et al., 2024). Based on these studies, we choose to use fine-tuning to optimize small-scale MLLMs. However, existing methods focus on natural language tasks and overlook the enhancement of MLLM's understanding capabilities in embodied environments. Therefore, we employ an actor-critic framework to facilitate comprehensive self-learning in MLLMs, optimizing both perception and decision-making abilities.
3 Preliminaries
3.1 Actor-Critic in Reinforcement Learning
Actor-critic (Konda & Tsitsiklis, 1999) is a widely adopted framework in reinforcement learning (Li et al., 2024a). The agent consists of two learning modules: an actor and critic, which are optimizod itcratively. The actor selects and executes actions based on current observations. The critic evaluates these observations (and actions) by estimating their values based reward signals received from the environment, thereby
3

---

Published in Transactions on Machine Learning Research (09/2023)
Self-Asking: State
Close
HR: Task
None
CRITIC
Check the state of cobiiet.
Which
SFT
Success Detection
Instruction:
Open
Task instruction:
Task Instruction
cotiined the 2
Open Cablnet
Open Cabiner
Open Drawer
Add
Other Relebel Add
Trajectories
Useful Trajectories
Environment Interaction- Task Instruction: Open Cabinet.
M ACTOR
Trajectory
!
Trajectory C
> 3
Trajectory
Figure 2: The framework of SELU. (lower) The actor MLLM, represented as a robot, collects trajectories for the given instructions. (upper) The critic MLLM, denoted as a brain, evaluates these trajectories and determines whether they complete the tasks, guiding the update of the actor MLLM. In addition, the critic MLLM implements self-asking and hindsight. relabeling to build dataset for optimizing itself. The whole framework does not require any external feedback, such environmental rewards human annotations.
guiding the actor to make improved choices in future. This framework takes advantage of both policy-based learning and value-based learning and popular nowadays like SAC (Haarnoja et al., 2018).
3.2 Actor-Critic for MLLMs
With the development of MLLMs, the feedback provided to the agent is no longer constrained to scalar values, like rewards; it can now include diverse modalities, such as natural language (Dong et al., 2024). This enables the critic gain more specific and informative feedback on the outputs of the actor. Consequently, it can provide more accurate guidance for actor improvement. incorporating human feedback into the critic and building static evaluation module that can reflect human preference (Ouyang et al., 2022; Kirk et al., 2024). Specifically, human annotated data is used to train critic model (McAleese et al., 2024) or reward model (Sontakke et al., 2023; Wang et al., 2024b) to align the MILM with human preferences better. More rigorous approaches leverage external evaluation mechanisms, such as tool-interactive learning (Gou et al., 2024; Chen et. al., 2021), or external knowledge sources like Wikipedia and the Internet (Xu et al., 2023; Li et al., 2024b).
prevalent approach
this domain is
However, regardless of whether preference labels or external tools are used, human intervention remains inevitable. To overcome this reliance, methods like self-consistency (Wang al., 2023a; Schick al., 2023) employ voting mechanism to enable the model to evaluate its behavior without relying external information. In addition, LLM-as-a-Judge approaches utilize an LLM as evaluator to eliminate human intervention (Bai et al., 2022; Lee et al., 2023). However, both self- colfisonnist and LLM-as-a-Judge lack a learnable critic module and therefore cannot improve the model's grounding knowledge of the environment it interacts with. contrast to previous work, we propose a novel actor-critic based paradigm aimed at achieving self-learning for both the actor MLLM and critic MLLM, enabling them ileratively improve decision-male making and grounding abilities without external human feedback environmental rewards.
4 Method
The framework of our method is shown in Figure 2. It consists of two components: the actor MLLM and the critic MLLM. The actor MLLM follows instructions and collects trajectories in the environment. The critic MLLM evaluates the collected trajectories and acquires bootstrapped data via self-asking and hindsight relabeling to optimize itself (Section 4.1). Guided by the success detection results from the critic MLLM, the actor MLLLM subsequently improves its decision-making performance in the environment (Section 4.2).
4

---

Published in Transactions on Machine Learning Research (09/2025)
By combining the two processes, can achieve coupled improvements of the critic MLLM and the actor MLL.M (Section 4.3) without reliance on external feedback.
4.1 Critic: Self-Asking and Hindsight Relabeling
As introduced in Section 1, enhancing the interpretation of environmental grounding information is crucial for an MLLM to improve its performance. In our framework, we propose to achieve this objective via self-asking and hindsight relabeling to acquire bootstrappod data for optimizing the critic MLLM.
Specifically, given an instruction I, the actor ML.LM collects a trajectory by following this instruction. The critic takes the last frame or of this trajectory as input, using as the detection frame to determine whether the task depicted by I is completed,
La MoMol
(1)
where Me denotes the critic MLLM, E E ("yes", "no") } is the result of the success detection, and Pa is prompt for the detection. If the detection result is la "yes", it means this trajectory is considered to be a successful sample by the critic MLLM for the given instruction and store this trajectory directly into the critic fine-tuning dataset Deritic in the format (T,pa, T.Paaor,la Ia), as shown by trajectory in Figure 2. This trajectory includes environmental grounding information that aligns with the knowledge contained in the critic MLLM.
If the detection result is la = "no", it means that the critic MLLM considers the trajectory unsuccessful in fulfilling instruction I. We first apply sclf-asking to examine the state of task-rclated objects, as the decision made by the critic MLLM might not be precise due to potential hallucinations. The critic MLLM is used to obtain the object states,
=c 1a = -=Mo(ji,,e.. oT)
(2)
where jr is the object name extracted from instruction T by a text-processing function, P. is the prompt for object state analysis, 1g is the analysis result, and Mc provides a new success detection 1/2 based on I, and I. The format is corrected to (I,Pa, (2) and stored into the critic fine-tuning dataset Deritic if l' = "yes". For example, in trajectory 2 of Figure 2, the critic MLLM initially misjudged the completion of the "open cabinet" task. However, when prompted to focus the state of the cabinet, it successfully self-corrected its judgment.
If the critic MLLM still considers the trajectory as failure, we propose to use hindsight relabeling to make use of this trajectory, since it might be helpful for learning the environmental grounding of other instructions. Hindsight relabeling is a method that originated in goal-conditioned reinforcement learning (Andrychowicz et al., 2017). Tt is based on a simple principle: if trajectory does not complete the target task, can be viewed as having accomplished other tasks or subtasks. For example, as shown in trajectory in Figure 2, although it does not complete the task "open cabinet', it successfully completes another task "open drawer'. Therefore, we relabel this trajectory with the instruction "open drawer" to help the critic MLLM recognize the completion of the relabeled instruction. We can write this process as,
I' = Mc(h,, 4 = -M(ar,,P Ph. or)
(3)
where ar is the verb extracted from the instruction I by another text-processing function. All text- text-processing functions rely on a classic format-man matthtnt approach, requiring the MLLM to strictly follow the prompt format in its output. Ph is the prompt for hindsight relabeling, and l is the output, which is usually an object name or None. Mc checks whether any objects, other than the target object, in the obscrvation or have completed the task associated with ar. Me generates new instruction I' if is not None. This enables the generation of new instructions, with "closed-book" actions restricted to the robot's capabilities, and "open- book' objects identified independently by the MLLM itself. After that, the data (I',pd; or, "yes") is stored into the critic fine-tuning dataset Derate- Finally, if a failed trajectory proves meaningless after hindsight relabeling, it is considered as not helpful for the MLLM to understand the environment and discarded.
By applying self-asking and hindsight relabeling, a critic fine-tuning dataset Deritic is created by MLLM itself containing the last frames considered as successful and the last frames relabeled as successful after self-asking and hindsight relabeling.
5

---

Published in Transactions on Machine Learning Research (09/2025)
4.2 Actor: Critic-Guided Improvement
Recent work has shown that the discriminative ability of an LLM exceeds its generative ability (Pang et al., 2024). As MLLMs are typically trained the same way as LLMs, we believe that MLLMs evaluation abilitics would also surpass their generation abilitics. For instance, we can casily prompt MLLMs to extract understanding from a given image, but it is challenging to prompt them to choose an appropriate action based on perceived task-relevant, information like distance or direction. Our experiment result. Section 5.2 also supports this conclusion, where the critic module always performs better than the actor. Therefore, we propose using the critic MLLM to guide the improvement of the actor MLLM in the environment without external feedback.
Specifically, the actor MILM interacts with the environment and collects online trajectories. At each timestep t, the actor generates an action plan la, by,
la, = Ma (P.
(4)
where Ma represents the actor MLLM, I is the task instruction, is the prompt for action plan and or is the current image observation. After collecting a whole trajectory, the critic MLLM determines whether this trajectory completes the instruction I, as described in Section 4.1. the answer is yes, the whole trajectory is put into the actor fine-tur ine-tuning dataset Dactor with a format of {(I. Pa. (t, (e,))) The relabeled successful trajectories after hindsight relabeling also added into the actor fine-tuning dataset Dactor- Since this dataset only contains task completion trajectorics, the actor can quickly converge towards completing tasks in the current environment by fine-tuning on the dataset. Note that the actor fine-tuning dataset Dactor consists of trajectory data, while the critic fine-tuning dataset Deritle only contains the last frames of these trajectorics.
4.3 Actor-Critic Coupling Improvement
We employ Supervisod Fine-Tuning (SFT) (Devlin et al., 2019; Brown et al., 2020) and Low-Rank Adaptation (LoRA) (Hu et al., 2022) to update both the actor and critic MLLMs. Initially, the actor MLLM interacts with the environment to collect online trajectories containing grounding information, as depicted in the lower part of Figure 2. The critic module then evaluates and classifies these trajectories based on the last frame, shown in the upper part of Figure 2. We select successful trajectories identified by the critic MLLM to create the actor fine-tuning dataset Dactor- Subsequently, we utilize the last. frame to construct the critic fine-tuning dataset Deritie- The update of the actor and critic can be performed iteratively, and make them both improve step-by-step. A detailed pseudo-code for our algorithm is available in Appendix A.1.
5 Experiments
5.1 Experimental Setup
Environments. In order to simulate embodied MLLM interactions in unknown environments, we select AI2- THOR (Kolve al., 2022) and VirtualHome (Puig et al., 2018) for our experiments. Both environments offer open-ended tasks, various interactive objects, and selectable camera perspectives, facilitating data collection for the actor and critic. More details can be seen in Appendix A.2.1.
A12-THOR is an interactive simulation environment designed for embodied AI research. The primary tasks require agents to navigate and interact. with household objects. Tt. offers highly realistic 3D environments that simulate kitchens, living rooms, and other indoor scttings.
VirtualHome is also an embodied simulation platform designed to imitate human activities and tasks in home environments. This environment focuses task completion through multi-step action sequences, making it ideal for testing long-term planning.
Task Selection. The goal of SELU to enable the MILM to acquire atomic skills in unknown environments through self-feedback. Therefore, the instruction list is derived from the robot's executable actions. In AT2-

---

Published in Transactions on Machine Learning Research (09/2025)
THOR, locobot is selected the agent, this work does not consider low-level control of robotic arms. There are three fundamental actions: pick up, open, and break, which serve as the basis for long-horizon task composition. To ensure task diversity and feasibility, we first prompt the MLLM to explore the environment and use the discovered objects to initialize the instruction list which serves as the task set. We randomly sample 2-3 objects for each lype of task. Considering the requirements of alomic skills and training costs, we restrict the maximum step for all tasks to 10. We apply a similar approach in the VirtualHome environment, selecting 'femalel" the agent and primarily testing in grab, open, and sit tasks.
MLLMs. To demonstrate the generalization capability of our framework, we conduct experiments using two MLLMs: LLaVA (Liu et al., 2023) and Qwen- VL (Bai et al., 2023).
LLaVA has gained prominence as one of the most popular MLLMs due to its simple architecture and lower training data requirements. These features enable LLaVA to generate responses more swiftly, and suitable for inference investigate self-learning.
Qwen- VL is the first model to use a 448x448 resolution image imput. Due to its higher resolution, this model exhibits enhanced visual understanding capabilities. We opt for Qwen- VL with the aim of better success detection, thereby facilitating more efficient self-learning of MLLMs.
Baselines. We compare SELU with five methods to investigate the feasibility of self-learning of MLLMs in embodied environments:
DG refers to the results obtained through direct generation from the initial MLLM without any fine-tuning.
SC (Wang et al., 2023a) represents an optimization method of MLLMs through self-consistency. Specifically, we employ multiple chains of thought (CoT) to prompt an MLLM to answer the same question, followed by majority voting. In our experiments, we utilize three different CoTs to guide the MLLM, ultimately voting for the most reasonable action.
LMSI (Huang et al., 2023) is a self-imprrvemmen method based on SC. It generates "high- confidence' answers for unlabeled questions build fine-tuning datascts. This approach enables the LLM to iteratively improve its performance based the voting mechanism, and extend this approach to MLLMs.
Self-Refine (Madaan et al., 2023) involves multiple rounds of self-reflection, followed by self- optimization based the reflection results. This method focuses on prompt optimization and has been validated for feasibility in large-scale LLMs, such as GPT-4. In experiments, we reflect 3 rounds to get the final result.
LLM-Planner (Song al., 2023) is method specifically designed for multimodal embodied planning with text retrieval-angmented techniques. In experiments, we retrieve 3 trajectories memory for the actor module.
RAP (Kagaya al., 2024) is a method employing a multimodal retrieval technique for embodied tasks. In experiments, retrieved trajectories are used for decision-making.
Sel-Ask represents the method that uses grounding guidance prompt engincering techniques from SELU, substantiate the necessity of critic optimization.
SELU-One represents the method of using the same MLLM to simultaneously perform actor and critic tasks, and finc-tuning with combination of actor and critic datasets. This approach aims to investigate the feasibility of utilizing single MLLM to meet the requirements of our framework.

---

Published in Transactions on Machine Learning Research (09/2025)
Evaluation. For the critic module, the MLLM's detection accuracy of whether the task is completed to reflect its ability to understand the environment. Specifically, we provide the last frame of the trajectory during testing and compare whether the MLLM's judgment is consistent with the environmental feedback. For the actor module, the MLLM's decision-makingg ability is mcasured by its task success rate. In the experiment, each lask tested 100 times get the actor MLLM's task success rate. The final frames of these trajectories are then used evaluate the critic MLLM's success detection accuracy. All results are evaluated at the same dataset. More details for implementation are provided in Appendix A.2.
5.2 A12-THOR
VLaVA. We first demonstrate the elfectiveness of SELU in the A12-THOR environment. After online interactions with the environment and fine-tuning of LLaVA, the critic exhibits an average performance improvement of approximately 27%, while the actor achieves an improvement of around 20% compared to the original model. Table and Table 2 present the accuracy of task success detection and task success rate respectively.
Table 1: Accuracy of task success dctection the A12-THOR environment.
Method
Pick up
Open
Break
Avg.
DG 80.67%
36.50%
50.50%
55.89%
Self-Ask
79.67%
41.50%
53.50%
58.22%
SELU-One
68.67%
30.50%
25.50%
41.56%
SELU 94.33%
67.50%
87.50%
83.11%
Table 2: Task success rate in the A12-THOR environment. SC and Self-Refine use prompt engineering to realize self-learning, whereas LMSI and SELU utilize fine-tuning. LLM Planner and RAP serve as typical embodied MLLM baselines, incorporating memory retrieval mechanisms.
Method
Pick Up
Open
Break
Avg.
DG
68.33%
65.00%
15.50%
49.61%
SC
65.67%
68.50%
17.50%
50.56%
Self-Refine
69.67%
70.50%
14.50%
51.56%
LMSI
75.67%
52.50%
19.50%
49.22%
LLM-Planner
66.67%
52.00%
11.50%
43.39%
RAP 62.33%
63.50%
14.50%
46.78%
SELU-One 91.33%
85.50%
27.50%
68.11%
SELU 94.67%
83.50%
30.50%
69.56%
In Table it is evidenced that the unified fino-tuning of the actor and the critic (SELU-One) leads decline in success detection, even worse than the original critic. Self Ask proves that fine-tuning is more effective than prompt engineering for the critic module. Although SELU-One achieves a task success rate comparable to that of SELU as shown in Table 2, the compromised critic results in SELU-One incorrectly analyzing the trajectory in subsequent epochs.
Table 2 demonstrates that baselines are not suitable for the self-learning of embodied MLLMs. Both prompt- engineering and fine-tuning baselines struggle to improve the decision-making ability of the MLLM. The reason is that the embodied MILM cannot give correct task detection with lack of environmental understanding, As 55%. In this case, merely optimizing the prompt to create multiple CoTs for repeated reflection does not help the MLLM gain lask achievement details in an unknown environment. Therefore, neither SC nor Self- Seff-Refime can substantially enhance the task success rate. LLM-Planner and RAP also don't increase MLLM's ability understand the environment. For finc-tuning basclines, relying on statistical voting to validate behavior even leads worse performance. For instance, in the Open task, for LMSI the task detection accuracy of 36.5% causes the actor's performance to drop from 65% to 52.5% after fine-tuning. These results
in Table 1, the initial judgment of the MLLM (DG) on tasks is only about
own
8

---

Published in Transactions on Machine Learning Research (09/2025)
demonstrate the necessity of the critic module in SELU, and optimizing the critic is crucial for enhancing the actor's 8 performance.
Notably, in the Open task, we can observe a low accuracy of success detection for SELU; however, still helps the actor improve performance. This highlights the role hindsight relabeling, which is discussed in detail in Section 5.4.
In order to prove that SELU can help different. MLLLMs achieve self-learning, we select Qwen- VL and test it in the AI2-THOR environment under the same setting. The result is shown in Table 3, which indicates that SELU can help Qwen-VI improve the task evaluation capability by about 24% and the decision-making performance by approximately 23%.
Qwen-
Table 3: Self-learning performance of SELU Qwen-VT, in the A12-THOR environment.
Qwen- VL-DG
Qwen- VL-SELU
Task
Critic
Actor
Critic
Actor
Pick Up
73.33%
57.67%
95.67%
95.33%
Open
51.00%
46.50%
81.50%
68.00%
Break
63.50%
12.50%
83.50%
21.50%
Avg. 62.61%
38.89%
86.89%
61.61%
5.3 VirtualHome
We then conduct experiments in the VirtualHome environment, which incorporates a greater variety of itemss and human agents, theroby enriching the experimental environments to demonstrate the effectiveness of our method. The experimental results are presented Tables and 5. In this environment, SELU enhances LLaVA task evaluation capability by approximately 30% and improves decision-making performance by around 24%, and it also outperforms baselines. As the environment becomes more complex, the lack of environmental understanding causes SC and Self-Refine to negatively impact the decision- makiinn of the original embodied MLLM. As shown in Table 5, the performance of SC and Self-Refine become even lower than DG.
Table 4: Accuracy of task success detection the VirtualHome environment.
Method
Grab
Open
Sit
Avg.
DG 52.67%
35.33%
44.50%
44.17%
Self-Ask
54.33%
36.67%
43.50%
44.83%
SELU-One
45.33%
15.67%
48.50%
36.50%
SELU
93.67%
83.33%
47.50%
74.83%
Table 5: Task success rate in the VirtualHome environment.
Method
Grab
Open
Sit
Avg.
DG
65.00%
83.33%
56.50%
68.28%
SC
52.67%
81.67%
61.50%
65.28%
Gelf-Refine
59.67%
74.33%
60.50%
64.83%
LMSI
35.67%
93.67%
52.50%
60.61%
LLM-Planner
55.33%
78.67%
50.50%
61.50%
R.A.P 58.67%
79.33%
51.50%
63.17%
SELU-One
83.67%
98.67%
94.50%
92.28%
SELU
93.33%
97.67%
93.50%
94.83%
9

---

Published in Transactions on Machine Learning Research (09/202))
Pick up the Lettace
Pick Up the Bread
Pick the Apple
100
Performance (%)
Performance (%)
100
Success Rate (%)
80
80
60
Actor
Critic
20
40
Critic
40
Critic static
0.1k
0.5k
1k
10k
2e-7
2e-6
2e-5
2e.4
Initial
1st
2nd
3rd
Learning Rate
Training Epoch
(a) Dataset Size
(b) Learning Rate
(c) Training Tterations
Figure 3: Hyperparameter study of SELU on picking tasks in the A12-THOR environment: (a) explores
the size of the interaction dataset required for embodied MLLMs, (b) illustrates why a single MLLM is not suitable for SELU from the perspective of learning rare, and (c) demonstrates that the effect of multiple training iterations.
5.4 Ablation Study
We conduct ablation experiments on the critic module in the A12-THOR environemnt, and the results are shown in Table 6. SELU w/o SA means that we not utilize self aelf-asking to examine the critic output. SELU w/o HR means that we do not perform hindsight relabeling to reanalyze the trajectory when evaluating the task. SELU w/o ALL means that remove both self-asking and hindsight relabeling and the data obtained from environment interaction to directly fine-tune the actor. Tn this case, the evaluation result of the critic is derived from the original MLLM.
Table 6: Ablation study in the A12-THOR environment.
Critic (Success Detection Accuracy)
Actor
(Task Success Rate)
Task
SELU
w/o HR
w/o SA
w/o ALL
SELU
w/o HR
w/o SA
w/o ALL
Pick Up 94.33%
83.67%
78.33%
80.67%
94.67%
67.33%
59.67%
56.33%
Open 67.50%
31.50%
29.00%
36.67%
83.50%
66.50%
65.50%
72.50%
Break 87.50%
83.50%
73.50%
50.50%
30.50%
27.50%
23.00%
17.50%
Avg. 83.11%
66.22%
60.28%
55.95%
69.56%
57.11%
49.39%
48.78%
By comparing SELU and SELU w/o ALL, we can see the importance of whole critic module clearly. Only by understanding the environment can we achieve the improvement of docision-making in all tasks. By comparing SELU w/o SA and SELU w/o ALL, we find that self-asking can correct the critic's comprehension of the euvironmental task. It also serves as premise for hindsight relabeling, misclaesifications without self-asking may lead relabeling data with an entirely opposite distribution, severely impacting SELU's effectiveness. But reflection on single task is not enough. We find that the lack of hinsight relabeling leads to decrease in the diversity of the online fine-tuning dataset, and some tasks can not get enough data to perform well. For example, we can observe declined performance of success detection and decision-making in Open tasks from SELU w/o ALL to SELU w/o HR. By incorporating hindsight relabeling, from SELU w/o HR to SELU, we can perform a comprehensive multi-task evaluation for each trajectory, ensuring that the embodied MILM achieves self-learning on cach task. Consequently, self-asking and hindsight relabcling are essential components of the critic.
5.5 Hyperparameter Analysis
Online Dataset Size. Since the MLLMs fine-tuning process sensitive to the dataset size, we explore the amount of interaction data required achieve effective lcarning for embodied tasks. We conduct multiple tests on this variable based on picking up tasks in the A12-THOR environment. The results are presented in Figure 3a
10

---

Published in Transactions on Machine Learning Research (09/2025)
The size of dataset is described by the number of trajectories collected for single task. For instance, dataset-10k indicates that 10k trajectories are collected for specific task, such as picking up an apple, during online interactions. We evaluate the performance through the actor in terms of task success rate. We can see that the performance of datasct-1k is close to that of dataset-10k. For datasct-10k, it takes about two days lo collect one epoch data. To balance experimental performance with sampling elficiency, we opt to sample 1k trajectories per task in our experiments.
Based on this conclusion, employ data augmentation method during the actor training process. Since MLLMs tend to prioritize text over images when making decisions, they often focus excessively on the text prompt and overlook image comprehension. To address this issue, we shuffle the action lists in the prompts of training data to provide multiple prompts for the same image. This approach not only increases the dataset size, but also strengthens the connection between the MLLM policy and the observations.
Learning Rate. The learning rate is a critical factor, which prevents us from using the same MLLM for the actor and critie. We test different learning rales for the actor and critic separately on picking up lasks in the AI2-THOR, and the results are shown in Figure 3b, We find that, due to the varying sizes of the fine-tuning datasets, using uniform learning rate inevitably leads to overfitting or underfitting in one of the components, thereby impacting the overall performance of SELU. Therefore, we ultimately use two MLLMs to construct the SELU framework, ensuring effective self-learning. In our experiments, the learning rate for the critic is set to 2e-6 and for the actor is set to 2e-5.
Training Iterations. The goal of our framework is to achieve multi-iteration self-learning improvement; therefore, we also evaluate the results of multiple rounds of fine-tuning picking up tasks in the AT2- THOR environment. To ensure the effectiveness of fine-tuning, we retain 30% of the last fine-tuning dataset each time and obtain the remaining 70% of the data through online interaction. The results are presented in Figure 3c. The solid line represents the performance under an optimized critic, while the dashed line indicates the results under a static critic.
The unstable improvement of the actor under a static critic highlights the necessity of fine-tuning the critic. What's more, our results indicate that multiple iterations of fine-tuning do not consistently improve SELU's performance in every itcration. Both the actor and critic exhibit significant performance improvements during the first iteration of fine-tuning, but show fiuctuations and minimal growth in the subsequent iterations. We attribute this limitation to the performance of the critic. The critic performs the success detection based on the last frame, making it difficult to compare the quality of different successful trajectories. Once the actor reaches a level to roughly complete the task, we lack the nuanced supervisory signals to guide the actor for further improvement. Consequently, while there a notable improvement. after the first. iteration, subsequent enhancements are limited. We also attempt to increase the number of frames for success detection. However, the performance of the small-scale MLLM does not meet our needs.
6 Conclusion and Limitation
In this paper, we introduce SELU, method for MLLMs to achieve self-learning in unknown environments. SELU facilitates interaction with the environment, analyzes interaction trajectories, builds an online dataset, and performs coupled optimization of the actor and critic. We employ self-asking and hindsight relabeling to enhance the critic task evaluation capabilities. Ablation experiments demonstrate that relabeling significantly expands the critic task judgment range. By leveraging the principle that ML.LSs possess stronger perceptual abilitics than decision-making abilitics, we improve the performance of the actor policy. We test SELU in the A12-THOR and VirtualHome environments, achieving critic improvements of approximately 28% and 30%, and policy improvements of about 20% and 24%, respectively. Additionally, to validate the applicability of SELU across different MLLMs, we evaluate it on Qwen-VL, resulting in a 23% performance enhancement.
One limitation of SELU is the lack of detailed assessment of successful trajectories. SELU can help embod- ied MLLM to self-learn how to accomplish tasks in unknown environments. How to complete tasks more efficiently is the next problem to solve. In future work, we will explore finer-grained critic signals to perform more accurate quality assessments of trajectories, guiding embodied MLLMs to tackle more complex tasks, such long-horizon combination tasks.
11

---

Published in Transactions on Machine Learning Research (09/2025)
As the current simulator masks the agent in the first-person view, we rely on third-person images to assess task completion. For future work, we aim to explore more suitable simulators or deploy SELU in real-world scenarios, enabling a consistent visual perspective for both the actor and critic modules.
7 Acknowledgements
This research was supported in part by NSFC under Grants 62173324, 62206281, 62450001 and Doubao Fund.
References
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advonces in neural information processing systems, 30, 2017.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprinl arXiv:2308. 12966, 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint ar arXht:2122008073, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christo- pher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain lchter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spa- tialvlm: Endowing vision-language models with spatial reasoning capabilitics. In Proceedings of the IEEF/CVF Conference on Compuler Vision and Pallern Recognition, pp. 14455 14455-14465, 2024a.
Xiang Chen, Chenxi Wang, Ningyu Zhang, Yida Xue, xiaoyan yang, YUE SHEN, Jinjie GU, and Huajun Chen. Unified hallucination detection for multimodal large language models. In ICLR 2024 Workshop on Reliable and Responsible Foundation Models, 2024b.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Tana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Hao Huang, Bryan Routledge, and William Yang Wang. FinQA: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3697-3711. Association for Computational Linguistics, 2021.
Wonje Choi, Woo Kyung Kim, Minjong Yoo, and Honguk Woo. Embodied cot distillation from LLM to off-the-shelf agents. In forty-first International Conference on Machine Learning, 2024.
Josef Dai, Xuchai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. Tn The Twelfth International Conference on Learning Representations, 2024.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019.
Yihong Dong, Kangcheng Luo, Xue Jiang, Zhi Jin, and Ge Li. PACE: Improving prompt with actor-critic editing for large language model. In Findings of the Association for Computational Linguistics ACL 2024. pp. 2024.
12

---

Published in Transactions on Machine Learning Research (09/2025)
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De- An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35:18343 18362, 2022.
Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: A thcorctical perspective. In Advances in Neural Information Processing Systems, volume 36, pp. 70757 70798. Curran Associates, Inc., 2023.
Loris Gaven, Clement Romac, Thomas Carta, Sylvain Lamprier, Olivier Sigand, and Pierre- Oudeyer. Sac-glam: Improving online rl for Ilm agents with soft actor-critic and hindsight relabeling. arXiv preprinl arXiv:2400. 20.112440.12488 12481 2024.
Zelalem Gero, Chandan Singh, Hao Cheng, Tristan Naumann, Michel Galley, Jianfeng Gao, and Hoifung Poon. Sclf-verification improves few-shot clinical information extraction. In ICML 3rd Workshop on Interpretable Machine Learning in Healthcare (IMLH), 2023.
Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Nan Duan, and Weizhu Chen. CRITIC: Large language models can self-correct with tool-interactive critiquing. In The Twelfth International Conference on Learning Representations, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arxiv preprint arXiv:2501.12945 2025.
Tuomas Haarnoja, Aurick Zhou, Picter Abbecl, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861-1870 Pmlr, 2018.
Patrick Haluptzok, Matthew Bowers, and Adam Tauman Kalai. Language models can teach themselves program better. In The Eleventh International Conference Learning Representations, 2023.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-imm sellovee In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1051-1068. Association for Computational Linguisties, 2023.
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospettonn-allcaatio. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14418-1342, 2024.
Chaoya Jiang, Haiyang Xu, Menglan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 77036-27044 2024a.
Haobin Jiang, Junpeng Yue, Hao Luo, Ziluo Ding, and Zongging Lu. Reinforcement learning friendly vision- language model for minecraft. In European Conference on Computer Vision, pp. 1-17. Springer, 2024b.
Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou, Jayashree Karlekar, Sugiri Pranata, Akira Kinose, Koki Oguri, Felix Wick, and Yang You. Rap: Retrieval-augmeetedl planning with contextual memory for multimodal Ilm agents. In NewrIPS 2024 Workshop on Open- World Agents, 2024.
Robert Kirk, Ishita Modiratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefen- stette, and Roberta Raileamu. Understanding the effects of RLHF on LLM generalisation and diversity. In The Twelfth International Conference Learning Representations, 2024.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, Anirdddaa Kembhavi, Abhinav Gupta, and Ali Farhadi. Ai2- thor: An interactive 3d environment for visual ai, 2022.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems. 12, 1999.
Ksenia Konyushkova, Christos Kaplanis, Serkan Cabi, and Misha Denil. Vision-language model dialog games for self-improvement. arXiv preprint -XXi::5502.02/400 2025.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.002: 2023.
Scongyun Lee, Seungone Kim, Sue Park, Geewook Kim, and Minjoon Sco. Prometheus-vision: Vision- language model a judge for fine-grained evaluation. In Findings of the Associalion for Compulational 2024.
Linguistics ACL 2024, pp.
Boyu Li, Haoran Li, Yuanheng Zhu, and Dongbin Zhao. Mat: Morphological adaptive transformer for universal morphology policy learning. IEEE Transactions on Cognitive and Developmental Systems, 16 (4):1611 1621, 2024a.
Boyu Li, Siyuan He, Hang Xu, Haoqi Yuan, Yu Zang, Liwei Hu, Junpeng Yue, Zhenxiong Jiang, Pengbo Hu, Börje F Karlsson, al. Dualthor: A dual-arm humanoid simulation platform for contingency-wware planning. arXiv preprint ar drXiv:2506.16011 2025.
Miaoran Li, Baolin Peng, Michel Galley, Jianfeng Gao, and Zhu Zhang. Self-checker: Plug-and-ple modules for fact-checking with large language models. In Findings of the Association for Compulational Linguistics: NAACL 2024, pp. 163 181. Association for Computational Linguistics, 2024b.
Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 18061-18070, 2024c.
Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus-1: Hybrid multimodal memory empowered agents excel in long-horizon tasks. arXiv preprint arXiv:2408.0061 036115 2024d.
Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila A. McIlraith. STEVE-1: A generative model for text-to-behavior in minecraft. In seventh Conference Neural Information Processing Systems, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. In Advances in Newral Information Processing Systems, volume 36, pp. 84892-34916 Curran Associates, Inc., 2023.
Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yuc. Visual- webbench: How far have multimodal LLMs evolved in web page understanding and grounding? In First Conference Language Modeling, 2024.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepscek-vl: towards real-world vision-language understanding. ar preprint arXiv:2403.0552 2024a.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations, 2024b.
Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. CoCo-agent: A comprehensive cognitive MLLM agent for smartphone GUT automation. pp. 9097-9110 Association for Computational Linguistics, 2024.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Layu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self- feedback. In Advances in Neural Information Processing Systems, volume 36, pp. 46534-46594. Curran Associates, Inc., 2023.
Minesh Mathow, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A datasct for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2021.
Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. Llm critics help catch 1lm bugs. arXiv preprint (07:2407700215, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feodback. Advances in neural information processing systems, 00:27730-27745 2022.
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Au- tomatically correcting large language models: Surveying the landscape of diverse automated correction strategies. Transactions of the Associalion for Compulational Linguislies, 12:484-506 2024.
Jing-Cheng Pang. Pengyuan Wang, Kaiyuan Li, Xiong- Huig-Hui Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. Language model sell-improvement by reinforcement learning contemplation. In The Twelfth International Conference on Learning Representations, 2024.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proccedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Siyuan Qi Shuo Chen, Yexin Li, Xiangyu Kong, Junqi Wang, Bangcheng Yang, Pring Wong, Yifan Zhong, Xiaoyuan Zhang, Zhaowei Zhang, Nian Liu, Yaodong Yang, and Song- Chun Zhu. Civrealm: A learning and reasoning odyssey in civilization for decision-making agents. In The Twelfth International Conference on Learning Representations, 2024.
Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision-language models are zero-shot reward models for reinforcement learning. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023.
Timo Schick, Jane A. Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Scbastian Riedel. PEER: A collaborative language model. In The Eleventh International Conference on Learning Representations, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimiza- tion algorithms. arXio preprint arXiv: 10w.:777.000 007.06647, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint ar arXiv:2402.0333 2024.
Chan Hee Song, Jiaman Wu, Claylon Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Llm- planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2998-30000 3009, 2023.
Sumedh Sontakke, Jesse Zhang, Séb Arnold, Karl Pertsch, Erdem Bi yık, Dorsa Sadigh, Chelsea Finn, and Laurent Itti Roboclip: One demonstration is enough learn robot policies. In Advances in Neural Information Processing Systems, volume 36, pp. 55681 55693. Curran Associates, Inc., 2023.
Dan Su, Xiaoguang Li, Jindi Zhang, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. Read before generate! faithful long form question answering with machine reading. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 744 156.756.66 Association for Computational Linguisties, 2022.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. In Annual Meeting of the Association for Computational Linguistics, 2024a.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented RLHF. In Findings of the Association for Computational Linguistics ACL 2024, pp. 13088 13110. Association for Computational Linguistics, 2024b.
Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. True knowledge comes from practice: Aligning large language models with embodied environments via reinforcement learning. In The Twelith International Conference on Learning Representations, 2024a.
Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, et al. Cradle: Empowering foundation agents towards general computer control. In NewrlPS 2024 Workshop on Open- World Agents, 2024b.
Xiaoyu Tan, Shaojie Shi, Xihe Qiu, Chao Qu, Zhenting Qi, Yinghui Xu, and Yuan Qi. Self-criticism: Aligning large language models with their understanding of helpfulness, honesty, and harmlossness. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 650 662, Singapore, 2023. Association for Computational Linguistics.
Jingqun Tang, Qi Liu, Yongjic Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, al. Mivqa: Benchmarking multilingual text-centric visual question answering. XrXiv preprint 2rXiv:2405. 1 11985, 2024.
Heyi Tao, Sethuraman T V, Michal Golapentokh-Roohhnee
Tanmay Gupla, Heng Ji, and Derek Hoiem. WebWISE: Unlocking web interface control for LLMs via sequential exploration. In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 3693-3711 Association for Computational Linguistics, 2024.
Belen Martin Urcelay, Andreas Krause, and Giorgia Ramponi. Reinforcement learning from human text feedback: Learning a reward model from human text input. In ICML 2024 Workshop on Models of Human Feedback for Al Aligmment, 2024.
Tianlu Wang, Iha Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. Self-taught evaluators. arxiv preprinl 2024a.
arXiv:2408. 026666, 02666,
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Fleventh International Conference on Learning Representations, 2023a.
Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Frickson. RI- VLM-f: Reinforcement learning from vision language foundation model feedback. In Proceedings of the 4 1st International Conference on Machine Learning, volume 235, pp. 51484-51501. PMLR, 21-27 Jul 2024b.
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. JARVIS-1: Open-world multi-task agents with memory-augmentedd multimodal language models. In Second Agent Learning in Open- Fren-nnde Workshop, 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Newral Information Processing Systems, volume 35, pp. 24824 24837. Curran Associates, Inc., 2022.
Licheng Wen, Xuemeng Yang, Daocheng Fu, Xiaofeng Wang, Pinlong Cai, Xin Li, Tao MA, Yingxuan Li, Linran XU, Dengke Shang. Zheng Zhu, Shaoyan Sun, Yoqi BAI, Xinyu Cai, Min Dou, Shuanglu Hu, Botian Shi, and Yu Qiao. On the road with CPT-Av(ision): Explorations of utilizing lasual-language model as autonomous driving agent. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Robin Winter, Marco Bertolini, Tuan Le, Frank Noe, and Djork-Arné Clevert. Unsupervised learning of group invariant and equivariant representations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Chuyan Xiong, Chengyu Shen, Xiaoqi Li, Kaichen Zhou, Jiaming Liu, Ruiping Wang, and Hao Dong. Autonomous interactive correction MLLM for robust robotic manipulation. In 8th Annual Conference Robol Learning, 2024.
Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Wang, and Lei Li. INSTRUCTSCORF: Towards explainable text generation evaluation with automatic feedback. In Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5967-5994.
Association for Computational Linguistics, 2023.
Xinrun Xu, Yuxin Wang, Chaoyi Xu, Ziluo Ding, Jicchuan Jiang, Zhiming Ding, and Börje F. Karlsson. A survey on game playing agents and large models: Methods, applications, and challenges. arXiv preprinl arXiv:2403. 10249, 2024.
Yiming Yan, Tao Wang, Chengqi Zhao, Shujian Huang, Jiajun Chen, and Mingxuan Wang. BLEURT has universal translations: An analysis of automatic metrics by minimum risk training. In Proceedings of the 61st Annual Meeting of the Association for Computational Lánguistics (Volume 1: Long Papers), pp. 5428 5443. Association for Computational Linguistics, 2023.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai- Tao Zheng, Maosong Sun, and Tat-Seng Chua. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-gred une-grained correctional human feedback. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13807-13816, 2024.
Yucxiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alanc Suhr, Saining Xic, Yann LeCun, Yi Ma, and Sergey Levine. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Jiawei Zhang, Chejian Xu, and Bo Li. Chatscene: Knowledge-enabled safety-critical scenario generation for autonomous vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15459- 15469, 2024.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant represen- tations for domain adaptation. In International conference on machine learning, pp- 7523-7532. 7532. PMLR, 2019.
17

---

Published in Transactions on Machine Learning Research (09/2025)
A Appendix
A.1 Pseudocode of SELU
algorithm SELU
Input: critic MLLM Mc. actor MLLM Ma, critic fine-tuning dataset Deritie, actor finc-tuning dataset Dactor maximum timestep T, initial instruction list L, success detection prompt pa and action plan prompl pa Output: critic MLLM Mc. actor MLLM Ma
1:
Deritie: Dactor 6 0
2: function "SELU"
3: for instruct. in do
4: while data collecting not done do for timestep t to do get observation o from env ha, Ma Pa; 0z)
8:
use lar to interact with env end for La = Ma(I,Pa.Or) if la = "yes" then store (I,Pa, OT, la) into Deritie
10:
11:
12:
13:
store (L. ot, la,t), I....T into Dactor
14:
clsc
15:
get 1/ through self-asking
16:
if l/2 = "yes" " then
17:
store (1, oT, 1/2) into Deritic store (I,Pa: oz, la,t), 1....T into Dactor
18:
19:
else
20:
get l' through hindsight relabeling if l' + "None" then
21:
22:
store (I', Pd, Or, yes) into Deritic
23:
store (T', Pa, Ot, la,1), 11, I.... into Dactor
24:
end if end if end if end while
25:
26:
27:
28:
end for
29:
optimization M. and M, by Daritic and Dactor
30:
return critic MILM Mc, actor MLLM Ma end function
31:
A.2
Implementation Details
A.2.1
Environments
Figure shows our experiment environments. Both environments restrict agents to only interact with visible items, limiting their operational range to guarantee behavior plans realistic. Therefore, the actor MLLM makes decisions based on first-person perspective input to ensure accuracy Figure 4a and Figure 4e show. Given the limitations of the first-person view, the critic MLLM uses third-person perspective to evaluate the trajectory, reducing hallucinations and obtaining accurate scene information as Figure 4b and Figure 4d show.
The positioning of the third-person camera crucial, as it should accurately capture the agent's position and the objects it interacts with. Any occlusion or interference can impair the MLLM's understanding of the image, thereby affecting the results of critic success detection and hindsight relabeling.
18

---

Published in Transactions on Machine Learning Research (09/2025)
(a) A12-THOR
(b) A12-THOR 2
(c) VirtualHome 1
(d) VirtualHome
Figure 4:
The diagram of experimental environments. We utilize the first-person perspective for decision- making and a third-person perspective for trajectory evaluation.
A.2.2 Hyperparameters
WL The specific MLLMs we use are LLaVA-VL.6-Mistral--B and Qwen- VL. We use LoRA to fine-tune them, the hyperparameters are as follows.
Table Hyperparameters of LLaVA Fine tuning by LoRA.
Hyperparameters
Value
Train batch size
16
Eval batch size
4
Gradient Gradient_accummuationn accumulation steps Learning rale actor Learning rate critic
1
2e-5
2e-6
Warmup_rat ratio
0.03
Weight Weight__ecay
0.0
Model max length
2048
Lr sr_sccccullr__yye type
cosine
Tf32
True
19

---

Published in Transactions on Machine Learning Research (09/2025)
Table 8: Hyperparameters of Qwen-Vi VL Pine-tuning by LoRA.
Hyperparameters
Value
Train @lain_batch size
2
Eval batch_siize size
1
Gradient sitiiiett_accmmattiioonnoo _tsppp Learning rale actor
8
le-5
Learning mate_critie
1c-6
Warmup_raad ratio
0.01
Weight_
0.1
Adam bela2
0.95
Model @oode__mx__lee
2048
Lr scheduler type
cosine
Bî1
True
Lazy_ preppoccssssss
True
Both models are configured with temperature of and a maximum token length of 2048 for response generation. The maximum number of environment steps set 10. The multiple chains of thought for sclf-consistnncy (Wang al., 2023a) and the multiple rounds of reflection in self-refine (Madaan et al., 2023) are set to 3, following the settings reported in their papers. For LLM-Planner (Song el al., 2023) and RAP (Kagaya et al., 2024), retrieval parameters follow the official implementation to ensure consistency. The number of retrieved trajectories is fixed at the minimum value of 3, as adopted in RAP. The list of hyperparameters provided in Table 9.
Table 9: Hyperparameters List for Baselines.
Hyperparameters
Value
Response generation temperature
0
Maximum token length
2048
Maximum environment steps
10
Number of Co'ls (Wang et al., 2023a)
3
Number of reflection rounds (Madaan et al., 2023)
3
Retrieve buffer
20
KNN retrieves (LLM Planner (Song et al., 2023))
9
Multimodal retrieves (RAP (Kagaya et al., 2024))
5
Retrieve trajectorios input
3
A.2.3 Low-Level Actions of Agents
SELU is designed to enable embodied agents to learn atomic skills through MLLMs themselves. In our experiments, we use the "locobot" agent A12-THOR and the "femalel' agent in VirtualHome. The set of low-level actions to these agents is sunmarized Table 10.
20

---

Published in Transactions on Machine Learning Research (09/2025)
Table 10: Low-Level Actions A12-THOR and VirtualHome.
Action
Env
Description
MoveAhead,
Move the agent forward or backward by a fixed slep (0.25 meters).
Arr-THOR//ittuallomeee
MoveBack
RotateLeft, RotateRight
Al2-THOR/VirtulHHonee
Rotate the agent 90% the left or right.
Pick up visible and graspable object.
Pick
AI2-THOR/Virtullle
The action fails if the object is not visible.
Open an object that is both interactable and openable.
Open
Al2-THORR/Vituulllonme
Break an object that both interactable and breakable.
Break
A12-THOR
Make the agent sit on object that is both interactable and sittable.
Sit
VirtualHome
A.3
Prompts for SELU
The specific prompts for SELU are:
Actor-Interaction with Eav
This is the current observation from a {agent} in a {AI2-THOR/ (ViHHOOR///ttHHoomeee environment. Now the (agent) needs finish the task (instruction), you can only choose the following action to interact with the enviromment, which are {action list]. If you choose (PickupObject, (aajjee//Grabbbbet)) ObenObject, /SitObject), SankkObbeet/Sttbjeeet, you should give specific object name. Now the objects you interact with are (visible objs str). What's your next action to implement the command to (instruction) You should output your action and the reasoning. The output format should be: Action:..
(BreakObject/
Object:...
Reasoning....
Critic-Success Detection
The image shows a environment. Please chock whether the iustrucctinnoojocts. the image is finstruction. verb adj.) or not? You should output yes no, and the reasoning. The output format should be: Result:..
view from the {agent]'s perspective in a Wit-THOOR/VirtuulHHooon
Reasoning...
Critic-Self Asking
The image shows a third-person view from the {agent]'s perspective in a {AI2-THOR, /Vi environment. Please check the state the iestructtinnobjecctt in the image. You should output the state and the reasoning. The output format should be:
State:...
Reasoning:...
21

---

Published in Transactions on Machine Learning Research (09/2025)
Critic-Sell Asking
The image shows a environment. The cnttruuction.objectstt in the observation (objects.state) state, please determine whether the (instruction) / has been completed or not. You should output yes or no, and the reasoning, The output format should be: Result:.. Reasoning....
view from the {agent]'s perspective in a (AD-THORR/ituulHonnn
Critic-hindsight relabeling
The image shows third-person view from the {agent]'ss perspective in AAR=TTHRR/irrtualHoooee VirtualHome)) environment. Please see the image carefully. Determine whether there is any object that is finstruc- tion.ver' + Jdj.} by the {agent}? You should output the object name and the reasoning. The output format should bc: Object:..
Reasoning....
Crilic-hindsight relabeling
The image shows third-person view from the {agent]'s 's perspective a (AT2-THOR/ ( environment. The (relabeling.oby) objectt the observation is (instruction.ver adj-), J you should give new instruction based on it. The original instruction is (instruction), what's the new instruction? The output format should be: New instruction.... Reasoning....
A.4 Visualization of Actor and Critic on LLaVA in AI2-THOR
The embodied actor MLLM is used to interact with the unknown environment, and collect trajectorics from the evaluation of the critie MLLM. An example for 'pick up the lettuce' shown figure 5.
The embodied critic MLLM is used to perform success detection on each trajectory and use self-asking and hindsight relabeling techniques to build bootstrapped datasct. An example for 'break the mug' is shown in figure 6.
A.5 Multi-Frame Critic Study
To further justify our choice of using the final frame, we conduct the frame number parameter experiments to see its effect. As LLaVA can only input single image, tune it first allow multi-image input. The critic performance is evaluated using the DG bascline setting, with detection accuracy of task completion. The results are shown in Table 11. We conduct the parameter experiments with three configurations: the final frame (DG-1), the last 3 frames (DG-3), and the last 5 frames (DG-5).
half of the pretraining dataset to fine-
Table 11: Multi-frame Evaluation for Critic Module.
Pick Up
Open
Break
Avg.
DG-1
80.67%
36.50%
50.50%
55.89%
DG-3
76.67%
37.50%
51.00%
55.06%
DG-5
73.67%
37.50%
53.50%
54.89%
22

---

Published in Transactions on Machine Learning Research (09/202))
ACTOR
ACTOR Prompt:
current observation from robot. Now the robat needs finish the task (instruction), you can only choose the following action interact enviranment, which (action_lise Jist). you choose Pick up Object, OpenObject, SliceObject, or DirtyObject, you should give specific object
Now the objects you can interact with are (visible _bbjssssttt_.. What's your next action implement the (instruction) You should output your action and the reasoning. The output format should be: Action....
Object....
Reasoning...
ACTOR MLLM Response:
Based on the image provided, the robot should perform the following action to pick up the lettuce:
'MoveAhead'
Object: None
Reasoning: The robot is currently positianed at the edge of the counter, and to reach the lettuce, perform
needs forward. Once robot 'PickupObject' action to pick the lettuce.
Figure 5: A visualization of the actor MLLM interacting with the A12-THOR environment. The agent is instructed to pick up the lettuce. As the lettuce is far away, the agent needs to move closer before attempting to pick it up.
CRITIC
CRITIC Success Detection Prompt:
The image shows a third-person view from the robot's perspective in a kitchen environment. Please check whether (instruction, objects) in the image finstruction.verd not? You should output yes no, and the reasoning. The output format should be: Result:... Reasoning...
Response:
Result: Yes
Reasoning: The image shows a rabot with a camera, and the camera is pasitioned in such way counter. the counter, broken state, with pieces missing scattered around. This evident from the visible and the overall appearance of the mug, which suggests has been damaged or shattered.
capturing of
mug apgears be
Figure 6: A visualization of the critic MLLM conducting success detection trajectory. Since the trajectory has completed the 'brcak the mug" task, it can be added to the fine-tuning datasct.
The evaluation with multiple images does not outperform single-image evaluation. There are two reasons explain this result. First, most embodied simulators employ flash-like transitions (Li al., 2025) represent interaction processes. This shortcoming stems from the fact that different agents often require
23

---

Published in Transactions on Machine Learning Research (09/2025)
distinct low-level control policies. To ensure reliable and generalizable execution, simulators always use flash-like transitions for the interaction process. Consequently, multi-images are not helpful to understand the task completion. Second, current MLLMs still exhibit limitations in spatial reasoning (Chen et al., 2024a). Whether provided with egocentric or third-person views, MLLMs can capture the task-related objects but can not give a clear description of the position relationship between the robot and these objects. Given these constraints, we adopt the final frame as the input for the task critic.
A.6 Computational Resource Costs
We run all experiments in 8 x A100 GPUs with 40GB memory. The details of computational resource costs be seen in the Table 12. The training time and GPU hours reflect the training cost, while peak VRAM and inference time represent the inference cost. While SELU incurs higher training costs, its inference speed is quicker than traditional baselines as they need more time for retrieval techniques.
Table 12: Computational Resource Costs of LLaVA.
Method
Training Time (h) GPU Hours (h) Peak VRAM (GB)
Inference Time (s/sample)
DG
None
None
22.57
1.26
SC
None
None
22.62
4.13
LMSI
5.97
47.76
22.57
1.31
Self-Refine
None
None
22.57
5.22
LLM-Plaamer
None
None
23.21
2.76
RAP
None
None
23.21
5.71
Self-Ask
None
None
22.57
2.56
SELU-One
6.65
53.20
22.57
4.72 (critic) 1.26 (actor)
SELU
6.72
53.76
22.57
4.72 (critic) 1.26 (actor)
A.7 Hame-Removeds Prompt Experiments
As introduced in Section 1, an unknown environment refers to one in which the MLLM has neither been exposed to its visual data during pretraining, nor fine-tuned on its task-specific data to achieve grounding. The environment name does not significantly impact the MLLM's initial perception and recision-making ability. This is demonstrated by the experimental results in Table 13 and Table 14, where we replace "a (A12-THOR/ /VirtuaHHome) enviromment' in our prompts with "a household environment'. As the MLLM is unknown to the environment, the name does not notably affect its performance.
Table 13: Critic Performance of LLaVA in A12-THOR Environment with
Prompt
Method
Pick Up
Open
Break
Avg.
DG
78.33%
38.50%
51.50%
56.11%
Self-Ask
78.67%
43.00%
56.50%
59.39%
SELU-One
65.67%
33.50%
31.00%
43.39%
SELU
91.67%
67.00%
89.50%
82.72%
A.8 Long-Horizon Task Experiments
In this work, the self-learning MLLM is designed to learn atomic skills to finish short-horizon tasks. As embodicd MLLMs have been proved to have the ability to decompose complex and long-horizon tasks into atomic skills. Several benchmarks, such as ALFRED, have been developed to evaluate this planning ability. If design self-learning method to help the MLLMM to autonomously acquire atomic skills, it will
24

---

Published in Transactions on Machine Learning Research (09/2025)
Table 14: Actor Performance of LLaVA A12-THOR Environment with Came-Removee Prompt
Method
Pick Up
Open
Break
Avg.
DG
66.67%
66.50%
13.50%
48.89%
SC
67.33%
65.00%
14.50%
48.94%
Self-Refine
72.33%
69.50%
15.50%
52.44%
LMSI
77.67%
60.50%
11.50%
49.89%
LLM-PPlnncr
69.67%
53.50%
13.50%
45.56%
RAP
66.67%
67.50%
14.50%
49.56%
SELU-One 89.33%
80.50%
29.50%
66.44%
SELU
93.67%
83.00%
31.00%
69.22%
significantly facilitate the collection of diverse data and improve the MLLM's capabilities. In ALFRED, long- horizon tasks typically span around 50 timesteps and can be decomposed into 5-6 subtasks by an MLLM planner. Based on this, we sct a horizon of 10 timesteps for the MLLM to learn each atomic skill.
To further evaluate the scalability of SELU, we extend the horizon length to 40 timesteps and use the same exploring prompt initialize the task list. The objects are increased 8-10 (e-g., 10 for "pick up" and 'break' tasks, and 8 for "open' tasks). The corresponding experimental results are shown in Table 15 and Table 16. SELU stiil performs well in learning long-horizon alomic skills. However, in practice, we often decompose these skills into shorter segments to reduce the complexity of low-level skill execution for embodied robots.
Table 15: Critic Performance of LLaVA in A12-THOR Environment with 40 Timesteps
Method
Pick Up
Open
Break
Avg.
DG
73.30%
31.13%
49.30%
51.24%
Self-Ask
74.20%
33.63%
51.60%
53.14%
SELU-One
51.70%
26.50%
36.20%
38%13%
SELU 89.50%
69.38%
89.10%
82.66%
Table 16: Actor Performance of LLaVA in A12-THOR Environment with 40 Timesteps
Method
Pick Up
Open
Break
Avg.
DG
42.80%
51.38%
12.70%
35.63%
SC
45.30%
51.75%
12.10%
36.38%
Self-Refine
47.20%
55.13%
15.30%
39.21%
LMSI
51.60%
59.38%
13.70%
41.56%
LLM-Planner
44.70%
56.50%
16.30%
39.17%
RAP
45.60%
55.25%
17.90%
39.58%
SELU-One
81.30%
86.75%
31.40%
66.48%
SELU
83.50%
84.38%
32.70%
66.86%
25


Reviews:
[{"review_id": "06iDBt7QI2", "reviewer": "Reviewer_6R2S", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nWhile actor-critic frameworks have been utilized to improve decision-making in LLMs, the primary contribution of this work lies in effectively adapting and applying this paradigm to multimodal large language models (MLLMs).\n\nWeaknesses:\nAs noted above, actor-critic frameworks have already been shown to be effective in enhancing the decision-making capabilities of LLMs. Therefore, it is not particularly surprising that a similar effect is observed in MLLMs. From an algorithmic perspective, the paper offers limited novelty. Techniques such as self-asking and hindsight relabeling are not new and have been explored in prior work, see. e.g., \n\n@article{gaven2024sac,\n  title={SAC-GLAM: Improving Online RL for LLM agents with Soft Actor-Critic and Hindsight Relabeling},\n  author={Gaven, Loris and Romac, Clement and Carta, Thomas and Lamprier, Sylvain and Sigaud, Olivier and Oudeyer, Pierre-Yves},\n  journal={arXiv preprint arXiv:2410.12481},\n  year={2024}\n}", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an actor-critic self-learning framework, named SELU, designed to enhance the decision-making capabilities of multimodal large language models (MLLMs). In this framework, the critic leverages self-asking and hindsight relabeling techniques to extract knowledge from interaction trajectories generated by the actor, thereby improving its understanding of the environment. In turn, the actor benefits from self-feedback provided by the critic, leading to more informed decision-making. The effectiveness of SELU is demonstrated in the AI2-THOR and VirtualHome environments, where it yields improvements of approximately 28% and 30% for the critic, and 20% and 24% for the actor through self-learning. While actor-critic frameworks have previously been employed to enhance decision-making in LLMs, the key contribution of this work lies in successfully extending this paradigm to multimodal LLMs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "n/a", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AACMKZcnYX", "reviewer": "Reviewer_UatL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThe hindsight replay in application to robotics is novel and interesting. The closest thing is probably https://arxiv.org/pdf/2302.05206 which also does hindsight relabeling, though not in a robotics context (though this shouldn’t matter for RL methods).\n\nThe writing is clear and the method is generally well described.\n\nUsing hindsight relabeling is a good idea to use the already existing knowledge inside pretrained MLLMs.\n\n# Weaknesses\n\nThe claim of an “Unknown” environment paradigm (SELU) is misleading: the agent is explicitly provided the environment space (e.g., AI2-THOR/VirtualHome). Beyond that, real-world environments cannot be truly unknown. This is why most methods working with LLMs+RL work on e.g. code since it’s easier to estimate whether the model can truly perform in arbitrary environments (though I do understand that as soon as you include vision this is no longer possible).\n\nThe distinction between RL and SELU is really arbitrary: SELU is RL with binary returns (and relabeling). Consider an agent with {0,1} returns, then a naïve policy gradient would compute $-\\log(\\pi(a|s))\\cdot1$ for the 1 rewards  and $-\\log(\\pi(a|s))\\cdot0=0$ for the failures. By only training your agent on the successful rollouts, you end up doing exactly this.\nKeeping this in mind, one should also compare against Motif https://arxiv.org/pdf/2310.00166 which uses preference pairs to build the reward instead of a binary returns as in SELU. \n\nThe experiments are way too limited in complexity: Task horizons are capped at 10 steps with only 2–3 objects and 3 task types, which is too small for generalization. More complex tasks with 4–5× longer horizons are needed, given that low-level control is abstracted away. It's not clear why this method wouldn't collapse as the task horizon is increased. \n\nMany of your baselines also do not train a model, so comparing against SELU is not entirely fair, considering that SELU can use up to 10k examples just for training. You could try to run SELU with the dataset as examples in your prompts to validate whether the actor-critic component gives the performance uplift or just the fact that you train the models.\n\nCritics’ misdetections, even if rare, compound over self-bootstrapped training data, reinforcing model biases. This is a known issue in synthetic/intrinsic-reward RL that’s not discussed. Even if you have a 90% success chance, the missing 10% might introduce a compounding bias during training.\nThis becomes especially relevant if you are in novel environments since this increases the likelihood of misdetections.\n\nThere are some missing aspects for the method implementation as well:\n\n- Action Space: No full description of possible actions.\n- Verb Extraction & Relabeling Mechanics: It’s unclear how verbs are extracted, what aspects of trajectories may be relabeled, and whether relabeling has access to which nouns or words are possible (“open book vs “closed book” relabeling).\n\nLooking at the prompt template: Reasoning is placed after action selection; intuitively, prompting the model to “think” before choosing an action (so it can reflect on its arguments) would be more natural and potentially more effective. \n\nWhy do you list the objects the model can interact with? \n> “Now the objects you can interact with are {visible_objs_str}.” \n\nShouldn’t this be identifiable from the vision itself? Otherwise, why do you need the vision component?\n\nThere is also a major limitation in how information is presented to the MLLM: Figure 4 say that a first-person view is used for acting but a third-person view for critiquing. Why? Especially in the real world, having two completely independent camera systems is not realistic. This could also explain why SELU-One does not perform well. If SELU works in unknown environments, it should also be robust against changes in POV. I recommend you run the experiments with both models using first person and both third person. \n\nMinor note: The acronym SELU conflicts with “Scaled Exponential Linear Units” in existing ML literature.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new method SElf-Learning paradigm in Unknown environments (SELU), which aims to train multi-model large language models (MLLMs) to perform actions in a simulated environment. To achieve this, SELU uses an actor-critic method where the critic MLLM judges whether the actor LLM achieved the goals given to the actor. If the critic finds the actor has not achieved its goal, the MLLM relables the trajectory to synthetically generate a successful rollout.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Would strengthen work\n\nRemove the somewhat arbitrary distinctions between SELU and “normal” RL like \n> With the development of MLLMs, the feedback provided to the agent is no longer constrained to scalar values, like rewards; it can now include diverse modalities, such as natural language (Dong et al., 2024).\n\nUltimately, you still produce scalar (in fact binary) rewards, they are just not explicitly written down.\n\nRun a “prompt SELU” variant to check whether the Actor-Critic filtering or the model training causes the uplift in performance.\n\nCompare performance between “reason-then-act” and the current “act-then-reason” prompt-designs.\n\nDiscuss the impact of estimation errors and compounding errors on Agent/Critic training.\n\n\n# Necessary for acceptance \n\nReframe the „unknown environments“ claim or supply more experiments in more environments. Generally, I recommend removing the information of whether the model sees AI2-THOR or VirtualHome from the prompt to preempt contamination of the pretraining dataset (for unknown environments you wouldn’t be able to name the environment anyways). This also goes to dependent statements like \n> Compared to existing studies, we focus on the self-learning potential of MLLMs, as external feedback may be not professional enough in unknown environments.\n\nThis is not supported by your experiments since they are not “unknown environments” (both in the narrow sense that these are well known simulators, but also in the wide sense that the real-world spaces are well known already).\n\nIn the main text or appendix, list every primitive action available to the agent.\n\nRemove the list of visible objects from the prompt. When studying MLLMs Vision should be enough.\n\nIncrease task-horizon lengths (e.g., 40–50 steps) and object/task variety to test scalability.\n\nDescribe verb‐extraction algorithms in pseudocode. Also describe more thoroughly whether the relabeling is “open book vs “closed book”.\n\nUse a consistent viewpoint (either always first-person or always third-person) for both actor and critic and report the results. The system should perform well in both cases.", "broader_impact_concerns": "I see no broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "o54bo2rWcI", "reviewer": "Reviewer_72Xe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The problem is interesting to make the MLLM as the learnable critic without external feedback.\n2. The paper is easy to follow. \n3. SELU achieves competitive results on 2 benchmarks. \n\nWeakness\n1. Critic looks only at the last frame and cannot compare trajectory quality. When the critic judges success from a single image, it may mislabel longer actions.\n2. Each epoch still needs around 1 k trajectories per task and collecting 10k trajectories for one task costs two days on AI2-THOR, which seems to be less efficient. Moreover, since SELU has separate actor and critic, which doubles GPU memory and training time, please also provide the information about GPU memory, training time and inference time compared with baselines.\n3. Ablation studies are incomplete. It is better to separately analyze the function of self-asking and hindsight relabeling. Moreover, more qualitative examples should be shown in illustrate how hindsight relabeling corrects the trajectory label.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes SELU, an actor–critic style loop for embodied MLLMs in unknown environments. A critic MLLM refines its own success-detection ability through “self-asking” (focus questions) and “hindsight relabeling” (converting failed trajectories into success samples for other tasks); an actor MLLM is then fine-tuned on trajectories the critic judges as successful. Coupled optimisation is shown on AI2-THOR and VirtualHome with LLaVA and Qwen-VL back-bones, achieving SOTA results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Explain why the critic is fed only the final frame and report an ablation that varies the number or range of input frames (e.g., last-1, last-3, full trajectory). This will clarify whether “last-frame” is a design choice or a limitation.\n2. Add training time, GPU hours, peak VRAM, and inference speed for SELU and all baselines so readers can gauge practical cost.\n3. Report performance when each component is removed individually and when both are removed, to isolate their true contributions (self-asking and hindsight relabeling).", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)