# Addressing Long-Horizon Failure in Language-Grounded Robotics via Structured Interaction

Loan Bernat, Matthieu Grard, Ariane Herbulot, Florent Lamiraux

<figure><img src="image_1.png" /></figure>

## To cite this version:

Loan Bernat, Matthieu Grard, Ariane Herbulot, Florent Lamiraux. Addressing Long-Horizon Failure in Language-Grounded Robotics via Structured Interaction. 2026. hal-05514580

HAL Id: hal-05514580

https://hal.science/hal-05514580v1

Preprint submitted on 17 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Addressing Long-Horizon Failure in Language-Grounded Robotics via Structured Interaction

Loan Bernat¹,², Matthieu Grard¹, Ariane Herbulot², Florent Lamiraux²

¹Siléane, Saint-Etienne, France

²LAAS-CNRS, Université de Toulouse, CNRS, Toulouse, France

**Abstract**—Deploying autonomous robots in human-centric environments requires consistent reasoning, memory, and failure recovery across many sequential actions. Although vision-language-action systems have shown impressive low-level generalization, their performance degrades rapidly as action sequences grow longer and constraints evolve. Our insight is that these failures arise from a structural mismatch between training supervision and the states that agents encounter during execution, leading to compounding errors and inconsistent behavior over time. To address this, we introduce MAGMA-GEN, a structured interaction-based data generation framework, where a language model-based agent generates supervision signals directly from its own long-horizon executions. MAGMA-GEN provides structured preference and scoring signals for partial completion, failure, and recovery, enabling continued training without expert demonstrations or trajectory-level human labeling once task components are defined. We further analyze a dual-agent instantiation that separates high-level tool selection from persistent task and safety memories, improving stability over extended action sequences. We evaluate our approach on long-horizon manipulation tasks with evolving constraints, designed to expose long-horizon failure modes. Under a fixed training budget, models trained with our method outperform human-labeled and synthetic-augmentation baselines, while continuing to improve with additional self-generated experience, demonstrating more stable scaling behavior.

## I. INTRODUCTION

Natural language offers an accessible interface for specifying robotic goals, and recent vision-language-action (VLA) systems [20, 59, 4, 42] showcase impressive integration of perception, reasoning, and control, enabling broad task generalization. Yet merging high-level reasoning, perception, planning, and low-level control into a single model yields architectures that are difficult to interpret and maintain. This has motivated modular approaches in which, a language model (LLM) [5] handles task-level reasoning and tool selection, while execution is delegated to reusable motor or sensorimotor skills [10, 1, 38, 50]. While large models partially mitigate these issues, their computational cost limits practical deployment, and existing lightweight policies remain insufficient for long-horizon, semantics-driven manipulation [35, 38]. Crucially, these systems are typically optimized for short-horizon task completion and lack mechanisms to represent and maintain user-preference across extended interactions, as illustrated in Figure 1.

Code will be publicly available upon publication of the paper.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Most existing approaches to language-grounded manipulation are designed around tasks (a) whose objectives are fully specified at the beginning of an episode. We argue that this assumption limits the scope of modeled human-robot interactions, and instead target incrementally specified tasks with evolving goals and constraints (b).</figcaption></figure>

A key obstacle to such long-horizon, semantics-driven embodied decision-making is error compounding. Policies trained on narrow, near-optimal data distributions inevitably encounter out-of-distribution (OOD) states at test time, where uncertainty increases and small mistakes progressively push the system further off-distribution [18]. This effect is amplified in autoregressive Transformer-based pipelines [44], where each generated token becomes future input. It is particularly severe in language-conditioned robotics, where tool-selection errors irreversibly alter the physical environment, necessitating strong
**Human Demonstration**
**Supervision**

<figure><img src="image_4.png" /></figure>

**Reward-Based Interaction**

<figure><img src="image_5.png" /></figure>

**Execution-Aligned Supervision**

<figure><img src="image_6.png" /><figcaption>Fig. 2. Comparison for multi-step tool-calling training with LLM. 1) Where imitation learning from humans labeled data ensure the best performances, it requires to fully cover the observation space which is extremely costly. 2) Reinforcement learning on the other hand, demonstrates an interesting capacity to let the model explore out-of-distribution state. However, its performances with small models remains low due to the sparsity of learned signals. 3) Our method aims to generate offline roll-outs of data which can be used with any learning methods by letting the model explores its own state. We guide the exploration by providing token-level correction within the execution loop.</figcaption></figure>

failure recovery behavior. Our key insight is that **failures in long-horizon tasks mostly arise from a mismatch** between single-step, expert-curated supervision and the multi-step state distributions induced by autonomous execution. Addressing this requires supervision in the regions of the trajectory space that the policy induces (see Fig 2). Reinforcement Learning (RL) addresses this by allowing models to explore their own state distribution [16, 47, 15], but at the cost of substantial computational complexity and data inefficiency. More recent alternatives replace explicit reward modeling with preference-based or simplified supervised objectives [32, 14, 2]. However, these approaches still depend on externally provided supervision, whether human or synthetic, which remains expensive to scale and poorly aligned with the evolving state distributions encountered during autonomous execution. Existing pipelines for generating preference or trajectory-level supervision [52, 56] still rely on expert-annotated trajectories, and therefore fall short of enabling a fully self-generated training loop. Consequently, learning agents that can sustain coherent reasoning, failure recovery, and constraint satisfaction over 10+ sequential actions in dynamic environments remains an open challenge.

We address these challenges by pursuing two objectives: reducing reliance on human-curated data and enabling long-horizon multi-robot orchestration with lightweight models. To this end, we introduce **MAGMA-GEN**, a structured interaction-driven, on-policy data generation framework designed explicitly for long-horizon, language-conditioned manipulation. It generates training data by executing the current policy over stage-decomposed task trees, enabling supervision at intermediate states rather than relying on expert trajectories or binary success signal. MAGMA-GEN combines two components. (i) **Coaching** provides task-consistent validation signals derived from the task specification to identify failures and allow continued execution from failure states. (ii) **SimUser** introduces realistic linguistic interaction during execution, preventing the policy from over-fitting to template-based instructions without injecting task-solving behavior. Together, these mechanisms produce dense, recovery-aware supervision on the

policy's own state distribution, enabling sustained improvement on long-horizon tasks without expert demonstrations. Finally, we investigate a lightweight dual-agent decomposition as a design alternative for structuring long-horizon reasoning, focusing on interpretability and context management rather than raw performance gains. It consists of a lightweight dual-agent design in which an external memorizer maintains an explicit task-state representation, while a policy model focuses on high-level reasoning and tool selection.

We validate our approach through extensive simulation experiments and real-robot demonstrations.

## II. RELATED WORK

### A. Foundation models in Robotics

We distinguish two primary ways of integrating language into robot control with foundation models. The first is the end-to-end approach through Vision-Language-Action models (VLAs) [59, 13, 3]. Due to their scale, they typically rely on auxiliary mechanisms for real-time control, such as lightweight policy heads or action chunking [55, 58]. On the other hand, modular approaches decouple high-level reasoning from low-level execution [12, 50, 38]. A LLM [5] is used to parse and interpret a natural instruction into a sequence of robot actions from a set of predefined skills [1, 12]. These actions could either rely on model-based or learning approaches to generate the movement based on sensors. To improve performances, some works such as Silver et al. [37], Xu et al. [49] augment the number of components by integrating symbolic planners or agents with specific roles to address more complex tasks. However, most of these works use LLM with hundreds of billions of parameters. Szot et al. [40] is giving a detailed comparison between continuous (end-to-end) and discrete methods. In this work, we opt for a modular approach, decomposing the multi-step reasoning into a multi-agents cooperative system while reducing the total number of parameters.

### B. Enhancing Reasoning and Agentic Behavior for LLMs

To enhance reasoning capabilities, recent works encourage LLMs to explicitly articulate intermediate steps [16, 50, 31].
improving interpretability and answer quality. This is typically achieved via multi-phase training pipelines [33, 52, 6, 29]. combining Supervised Fine-Tuning (SFT) on human demonstrations with reinforcement learning (RL) objectives such as RLHF [11, 29] or GRPO [16]. However, RL-based fine-tuning is often unstable and can amplify superficial heuristics rather than promoting genuine causal reasoning [54, 8, 34]. To address this, recent work replaces RL with simplified supervised objectives such as DPO [32], KTO [14] or VAR [2], which avoid training a reward model and greatly improve stability. Critically, all these methods depend on extensive, high-quality human-labeled data. Moreover, policies trained only on expert demonstrations or synthetically generated near-optimal trajectories suffer from compounding errors: small deviations at inference time push the model into out-of-distribution states, where uncertainty accumulates and failures become increasingly likely [18]. This has motivated a growing body of work on self-play and self-evolving methods, where models learn from states they actually reach rather than from idealized behavior. Concurrent approaches [39, 56, 24] use simulated environments to generate self-play data and exploit self-reflection to correct errors. More recent methods rely on auxiliary models to propose training examples from task specifications [17], or train challenger models that generate increasingly difficult prompts to drive policy improvement [21, 51, 48]. These approaches focus on short-horizon reasoning or text-based tasks, rely on privileged signals or expert-provided demonstrations, and typically do not address continuous, tool-based, multi-step robotic execution. In contrast, our pipeline generates structured on-policy data for long-horizon robotic manipulation, using stage-wise evaluation, execution-grounded feedback, and optional coaching signals to support failure recovery and robust skill learning with small language models.

### C. Memories and Multi-agent system with LLMs

Integrating memory in conversational LLMs is done through conversation history, where past messages are concatenated with the current query. This approach is limited by context length: in long interactions, critical information may be truncated or lost [45], which is unacceptable in industrial robotics. Some methods compress past interactions into compact representations, either by using another model [19] or by using a retrieval model (RAG) [22]. Wang et al. [45, 46], Choi et al. [10] augment the model with additional attention-based memory. To preserve explainability and maintainability, we adopt a token-based external memory [30]. Allowing agents to store and access relevant information while enabling users to inspect and understand this shared memory.

## III. PROBLEM FORMULATION

We consider long-horizon language-conditioned robotic tasks in which goals and constraints are not fully specified upfront, but are incrementally introduced, revised, or refined through interaction. Such tasks arise naturally in human-robot collaboration, where instructions are issued sequentially, assumptions are corrected, and additional constraints emerge

as execution unfolds (Fig. 1c). Unlike standard language-conditioned manipulation benchmarks [7, 25, 57], which assume a fixed task specification known at the start of an episode, the tasks we study require the agent to act under an evolving specification. Correct behavior at any timestep depends not only on the current instruction, but on constraints introduced earlier in the interaction. To make these properties explicit, we first formalize the task structure and interaction dynamics, before introducing a Markov Decision Process (MDP) model of execution.

### A. Task Formulation

A task is represented as $\mathcal{T} = (\mathcal{G}, \mathcal{C}, \mathcal{K}, \mathcal{U})$, with $\mathcal{G}$ the sequence of goals to achieve, $\mathcal{C}$ the set of constraints, $\mathcal{K}$ the full set of task attributes (e.g object classes, robot ids) and finally, $\mathcal{U}$, the set of tools available to solve the task :

$$ \mathcal{U} = \{u_i : \mathcal{X}_i \to \mathcal{E}\}_{i=1}^{M} \qquad (1) $$

where $u_i$ are tools names (e.g. pick, place), $\mathcal{X}_i$ is the argument space (JSON dictionary) and $\mathcal{E}$ the execution outcome (success and feedback messages). During execution, the agent acts under a time-varying active goal $g^t \in \mathcal{G}$ and a subset of active constraints $C^t \subseteq \mathcal{C}$, which may be introduced, revised, or overridden through $q^t$. At each timestep $t$, the agent is conditioned on $\mathcal{U}$ and observes $S^t = (\text{Att}^t, Q^{1:t})$ with $\text{Att}^t \in \mathcal{K}$ the current known task attributes, and $Q^{1:t}$ the full interaction history with the user or the execution system.

Although, in theory, the full interaction history $Q^{1:t}$ is sufficient to recover the current task state, it grows unbounded with the task horizon. We therefore denote by $z^t \sim Q^{1:t}$ a bounded latent task abstraction encoding the goals and constraints currently in effect. This abstraction is implicit in the interaction history, encoding the current goal and constraints $z^t = (g^t, C^t)$, but is not directly observable by the agent.

### B. Execution Formulation

Given this interaction setting, the agent must repeatedly map the observable interaction state $S^t$ to a structured action consisting of language and executable commands. We model this decision-making process as a Markov Decision Process (MDP), where the environment state corresponds to the observable interaction history and environment attributes, and actions correspond to parameterized tool invocations and responses.

$$ M = \langle S, A, P, r, \gamma, \rho_0 \rangle, \qquad (2) $$

where $S = (\text{Att}, Q)$, $A$ is the action space, $P$ is the state transition kernel, $r$ is the scalar reward function, $\gamma \in (0, 1]$ is the discount factor, and $\rho_0$ is the initial state distribution. We decompose an action $a^t \in A$ with $a^t = (a_u^t, a_m^t) \in A$, where $a_u^t$ is the natural language response intended for the user, and $a_m^t = (u_i^t, x_i^t)$ the machine-interpretable tool with $x_i^t \in \mathcal{X}_i$ the selected arguments value of tool $u_i$.

The agent is modeled by a parameterized policy

$$ \pi(a^t | S^t) = \pi_{\theta}(a^t | S^t, \mathcal{U}), \qquad (3) $$
<figure><img src="image_7.png" /><figcaption>Fig. 3. Overview of MAGMA-GEN in a simple sorting task. MAGMA-GEN is a data generation framework for long-horizon, language-conditioned manipulation that learns from an agent's own executions without relying on expert demonstrations. Tasks are decomposed into stages and executed in simulation, producing branching trajectories. Two core components structure data generation: Coaching, which provides explicit recovery behavior, corrective feedback following failures or sub-optimal executions, and User Simulation, which generates and evaluates linguistic interactions.</figcaption></figure>

where $\theta$ denotes the parameters of the policy.

While this formulation provides a clean abstraction of the interaction dynamics, it exposes a key learning challenge: correct behavior requires sufficient coverage of transitions in the latent task abstraction $z^t$. In practice, both static datasets from human demonstrations and direct reinforcement learning fail to provide such coverage: the former under-samples transitions induced by failures and constraint revisions, while the latter is highly sample-inefficient and weakly informative through sparse rewards. This motivates the execution-grounded, on-policy data generation framework introduced next (Section IV). In addition, we show that making this abstraction explicit can simplify learning under long-horizon interactions and constrained model capacity (Section V).

## IV. MAGMA-GEN: A GUIDED DATA GENERATION PIPELINE

We now describe MAGMA-GEN, a guided data generation pipeline specially designed to increase coverage of $z^t$ on long-horizon task under evolving constraints with reduced human effort. Rather than relying on sparse rewards or unguided self-play [47], MAGMA-GEN structures the interaction between the agent and the execution environment to expose intermediate failures, corrections, and evolving constraints. Although instantiated with language models in this work, the framework itself is model-agnostic and applies to any policy capable of operating under the task formulation. We first introduce the task decomposition in stage IV-A, then present the two core mechanism enabling dense supervision: Coaching IV-C and SimUser IV-B. Figure 3 illustrates how they interact. We conclude by detailing how tasks are built IV-D and how learning signals are constructed from the collected data IV-E.

### A. Environment and Data Generation Structure

MAGMA-GEN structures data generation by dividing a long-horizon task into a sequence of stages to isolate the evolution of goals and constraints, enabling an effective attributions of failure and allows the coaching system to propose targeted correction, see Section IV-C. Each stage induces a goal $g^t$ and a set of active constraint $C^t$ given through natural language $q^t$. A stage is validated through a physical verification. In this work, we use ManiSkill3 [41] for action $a_m$ execution and evaluation, and the SimUser IV-B to evaluate $a_u$. A stage starts with an instruction $q^t$. Then the agent engages in a multi-step interaction loop: it produces an action $a$, receives tool execution feedback $\epsilon$, and continues from the resulting updated environment state $S^{t+1}$ until the stage is completed or fails, forming the i-stage trajectory $tr_i$. Stage completion is validated through physical execution, using ManiSkill3 [41] to evaluate machine actions $a_m$, and the simulated user (Section IV-B) to evaluate language responses $a_u$.

To capture the multimodality of multiple valid action sequence, each stage specifies an expected number of actions together with a tolerance window defining acceptable sub-optimal completions. This avoids assuming an unique optimal trajectory. The full task execution corresponds to the ordered concatenation of the stage-level trajectories, with each stage continuing from the terminal state reached by the previous one. Allowing to accumulate deviation realistically. In robotic settings, such errors are unavoidable due to motion planning failures or imperfect visuomotor execution, making recovery behavior essential.

To fully explore the agent distribution, we sample $n$ candidate action from each state $S$, creating various trajectory. After an action $a$ execution, the environment transitions to one of three outcomes: (i) a success state, enabling progression to the next stage; (ii) a failure state, corresponding to catastrophic errors or exceeds of the action budget, which triggers coaching (Section IV-C); or (iii) an intermediate state, in which inference continues within the current stage. By combining staged decomposition with tree-structured trajectory generation of goals and constraints, enabling an effective attributions of failure and allows the coaching system to propose targeted correction, see Section IV-C. Each stage induces a goal $g^t$ and a set of active constraint $C^t$ given through natural language $q^t$. A stage is validated through a physical verification. In this work, we use ManiSkill3 [41] for action $a_m$ execution and evaluation, and the SimUser IV-B to evaluate $a_u$. A stage starts with an instruction $q^t$. Then the agent engages in a multi-step interaction loop: it produces an action $a$, receives tool execution feedback $\epsilon$, and continues from the resulting updated environment state $S^{t+1}$ until the stage is completed or fails, forming the i-stage trajectory $tr_i$. Stage completion is validated through physical execution, using ManiSkill3 [41] to evaluate machine actions $a_m$, and the simulated user (Section IV-B) to evaluate language responses $a_u$.
tion, MAGMA-GEN enables data generation for long-horizon
tasks involving more than a dozen sequential interaction steps.

## B. Simulated User

In long-horizon tree-based data generation [52, 47], inter-action histories tend to collapse to a small set of repeated instruction patterns, leading to poor coverage of the state distribution. In our setting, task execution is interleaved with dialogue: constraints are introduced, revised, and confirmed through interaction. MAGMA-GEN addresses this by inte-grating a simulated user that (i) generates state-conditioned variations of user queries at each stage, and (ii) evaluates whether the agent's textual responses are consistent with the currently active goals and constraints.

Concretely, we leverage a frozen, pretrained language model conditioned only on the current stage specification and environment state, enabling grounded interaction without requiring additional human annotations or heuristics. It operates solely at the level of linguistic interaction, while physical correctness is evaluated independently through environment execution. Unlike approaches [51, 9] that rely on separately trained evaluators, the simulated user is frozen and requires no additional training, operating solely at the level of linguistic interaction.

By embedding this interaction directly into the execution loop, MAGMA-GEN exposes the agent to a broader distribution of linguistically induced transitions in $z^t$. Enabling consistent semantic evaluation alongside physical verification without human annotations or trained evaluators.

## C. Coaching

With offline method, exploration is inherently constrained by the current policy distribution: an agent can only sample actions that are already plausible under its learned representation. As a result, behaviors that require revising incorrect assumptions, recovering from failures, or satisfying newly introduced constraints are unlikely to be discovered through unguided exploration alone. To this end, MAGMA-GEN integrates a failure-driven mechanism called Coaching, designed to expose corrective transitions in the latent task abstraction $z^t = (g^t, C^t)$ that are unlikely to be discovered through unguided execution. Rather than providing oracle solutions, coaching intervenes only after execution-time evaluation reveals a failure or sub-optimal outcome, and generates corrective actions at the front-tier of the current policy distribution.

Concretely, coaching is triggered when an action sequence reach a stage failure state. Indicating that the agent's does not satisfy the active goal $g^t$ after the number of steps, or that constraints $C^t$ were not respected We distinguish four classes of failures:

**Textual failures** occur when the agent's language output $a_u^t$ fails to reflect the active constraints $C^t$, indicating an incomplete or incorrect representation of $z^t$. Coaching proposes a corrected response $a'_u$ consistent with all active constraints.

**Goal failures** arise when the executed action $a_m^t$ do not complete the active goal $g^t$ or violated $C^t$. Coaching proposes

an alternative answer $a' = (a'_m, a'_u)$ consistent with the inferred task state.

**Sub-optimal failures** correspond to executions that satisfy $g^t$ and $C^t$ but exceed the ideal action budget. In this case, coaching identifies the earliest decision leading to inefficiency and proposes a local corrective alternative, preserving subsequent recovery behavior.

**Execution failures** result from low-level execution or planning errors. Coaching intervenes only after unsuccessful self-recovery attempts, proposing recovery actions or an explicit abandonment response when infeasibility persists.

In practice, coaching acts as a targeted distillation mechanism: a frozen LLM generates corrective signals conditioned on task annotations and execution diagnostics. Rather than approximating a global expert distribution, coaching focuses on the agent's own failure cases, where the policy is the most brittle. **All coached actions are validated through environment execution, and unsuccessful coached trajectories are discarded.** Through this process, coaching converts execution-time failures into explicit corrective transitions in $z^t$, producing supervision that is sparse or absent in static datasets and prohibitively costly to obtain through human annotation.

## D. Dynamic Task Building

MAGMA-GEN assumes a task definition consisting of (i) a robot interface, defined by available tool APIs; (ii) environment attributes, such as object sets and scene configurations; and (iii) a set of generic stages, each associated with a subgoal verifier. Together, these components define reusable task primitives without prescribing expert trajectories.

From this task definition, training tasks are instantiated dynamically by composing stages and introducing goals and constraints at different points in the execution. Similar to recent work on synthetic task generation [17, 21], we use a language model to generate task instantiations and natural language instructions. In contrast, the model is kept frozen and used solely for task construction, allowing unbounded task instantiation without additional training.

During data generation, task definition are selected based on feedback from prior executions, favoring configurations that expose under-represented or failure-prone transitions in the latent task abstraction $z^t$. By varying when constraints are introduced, how long they must be maintained, and whether they are revised across stages, dynamic task building enables systematic exploration of long-horizon interactions without additional human annotation. Implementation details are provided in the appendix.

## E. Trajectory Scoring and Advantage Computation

Each rollout in MAGMA-GEN produces a tree of alternative execution branches from the same state. Rather than estimating a global value function, our objective is to assign relative preference signals among actions available at the same state, based on how they influence downstream task execution. Concretely, an action is considered preferable if it leads to more consistent stage completion, constraint satisfaction, and
recovery across future stages, reflecting a more accurate inference of the underlying task state abstraction $z^t$. This structure enables the extraction of multiple forms of supervision. By selecting the highest-scoring action at each state, we obtain execution-grounded targets suitable for supervised or weighted supervised fine-tuning. Alternatively, comparisons between sibling branches yield preference pairs that can be used with preference-based learning objectives [32, 14] or to train reward models for reinforcement learning.

We define a score $s(a | S)$ for each action $a$ taken at state $S$ by aggregating its downstream execution outcomes across $D$ future stages:

$$s(a | S) = \sum_{d=1}^{D} \gamma^d \left( \frac{N_d^{opt}}{N_d} + (1 - \lambda) \frac{N_d^{sub}}{N_d} \right), \quad (4)$$

Here, $\gamma \in [0, 1]$ controls temporal discounting across stages, and $\lambda \in [0, 1]$ balances optimal and sub-optimal completions. These parameters are fixed across tasks and not tuned per environment. $N_d$ denotes the number of rollout branches that reach stage $d$, among which $N_d^{opt}$ and $N_d^{sub}$ correspond to optimal and sub-optimal completions, respectively. The score is used solely to rank sibling actions during data construction, not policy evaluation.

## V. DUAL-AGENT DECOMPOSITION

While MAGMA-GEN enables models to learn recovery behaviors from execution-grounded data, monolithic language models often struggle with attention dilution over long interaction histories [23]. To complement our data generation framework, we introduce a lightweight inference-time decomposition that externalizes task-state persistence without modifying environment dynamics or requiring changes to the policy model. The goal of this decomposition is to make task-relevant state explicit and persistent, while preserving the zero-shot capabilities of existing models.

We decompose high-level reasoning and task-state tracking into two cooperating language models, illustrated in Figure 4. The Commander is responsible for high-level reasoning, user interaction, and tool selection, while a lightweight Memorizer maintains an explicit approximation of the latent task state $z^t$, denoted as $Me^t$. Crucially, in this work the Commander is instantiated as a frozen, zero-shot language model and is not updated during training or evaluation. All improvements arise from externalizing and maintaining task state, rather than from modifying the Commander's internal reasoning.

We formalize this decomposition as a structured refinement of the MDP defined in Section III, without altering the underlying environment dynamics. The state is augmented $S^t = (Att^t, Q^t, Me^t)$ and the action space becomes the joint space of the two agents: $a^t = (a_{com}^t, a_{mem}^t) \in A = A_{com} \times A_{mem}$, where:

* $a_{com}^t = (T^t, a_u^t, a_m^t)$ denotes the Commander output, augmented with $T^t$, an explicit natural-language summary of the inferred task intent and active constraints.

<figure><img src="image_8.png" /><figcaption>Fig. 4. Our dual-agent decomposition</figcaption></figure>

* $a_{mem}^t$ denotes the Memorizer output, corresponding to updates of the external memory (e.g., adding or removing task-relevant statements).

Importantly, $T^t$ is an explicit, observable output of the Commander produced within the interaction loop. It does not rely on hidden activations or privileged future information. By operating on an explicit task summary rather than latent representations, the Memorizer remains model-agnostic and can be paired with a wide range of existing language models in a zero-shot manner. Moreover, MAGMA-GEN naturally provides execution-grounded supervision signals that can be used to train or refine the Memorizer, while leaving the Commander entirely unchanged. Although this formulation permits jointly learning action selection and task-state construction in the general case, we deliberately restrict learning to the Memorizer in this work. The Commander is kept fixed and operates in a zero-shot manner throughout, ensuring that any performance gains can be attributed solely to improved task-state externalization and persistence.

## VI. EVALUATION

This section evaluates MAGMA-GEN data-generation pipeline (Section VI-A) and the dual-agent decomposition (Section VI-B). All models are trained under identical compute constraints, using a fixed budget of 2k gradient steps. Reported scores are averaged over three random seeds. All experiments are conducted in a single-robot sorting environment with dynamically assigned object-to-area mappings. We use mplib [27] as the motion planner. Training is performed using two Nvidia RTX 6000 Ada GPUs (48GB each). Data generation, testing, and evaluation are run on a separate machine equipped with a single Nvidia RTX 2000 Ada GPU (8GB). Example task configurations are provided in the Appendix.

### A. Comparing data-generation methods

**Baselines.** We compare several data-generation strategies using Qwen3:1.7B [43]. For reference, we also report results obtained with a 12× larger model, gpt-oss:20b [28].
<figure><img src="image_9.png" /><figcaption>Fig. 5. Task success rates on our long-horizon manipulation single-robot task. All models are trained under the exact same setting. The zero-shot large model is reported for contextual comparison only. Model trained with our method shows significant performance improvement on long-horizon task.</figcaption></figure>

We include a *Human-Only* baseline consisting of 470 expert demonstrations, as well as a stronger *Human+Synthetic* baseline obtained by applying extensive randomization to the expert data. Randomization includes paraphrasing instructions, renaming tools and parameters, and varying task attributes, resulting in approximately 5k training examples. *MAGMA-GEN* is evaluated using two generation-training cycles, producing approximately 800 samples in the first cycle and 4k in the second, matching the overall data budget of the *Human+Synthetic* baseline.

**Metrics.** We report two standard metrics [10, 36]: the Cumulative Success Rate (CSR) measures the percentage of tasks where all sub-goals are achieved and the Cumulative Goal-Conditioned Success Rate (CGC) reports the fraction of individual sub-goals achieved across all tasks before the first failure. We isolate CGC depending on the number of needed action by the agent to entirely solve the task.

**Results and Analysis** Fig 5 summarizes the performance across different task horizons. We observe that while absolute success rates remain modest due to the highly constrained training budget and small model scale, the relative gains from MAGMA-GEN are significant as task complexity increases. At short horizons (2-3 steps), MAGMA-GEN performs on par with or slightly better than human-augmented data. A minor performance dip occurs at mid-horizons (4-5 steps), which we attribute to the high density of expert-written scripts specifically covering this range in the baseline. However, as tasks extend into long and very-long horizons (6-10+ steps), MAGMA-GEN maintains much higher resilience, **achieving absolute gains of 21% and 15% in CGC**, respectively.

This performance gap is a direct result of execution-grounded coverage. Human-written scripts typically describe

"Gold Path" trajectories; they rarely account for the subtle semantic drifts or motion-planning failures that accumulate over time. By autonomously exploring and coaching the agent through these "failure frontiers," MAGMA-GEN provides the high-level planner with the corrective logic necessary to sustain long-term coherence. Beyond raw performance, the efficiency gains are substantial. Drafting and verifying the 470 expert scripts required approximately 50 hours of human effort. In contrast, MAGMA-GEN generated a dataset ten times larger and more diverse in roughly 3 hours of autonomous simulation. This suggests that for small-scale models where data quality is the primary bottleneck, autonomous execution provides a significantly more scalable path to reliability than manual curation.

## B. Benefits of explicit abstraction

**Setup.** We evaluate the effect of explicit task-state abstraction on long-horizon performance using two zero-shot language models at different scales: gpt-oss:20b [28] and Qwen3:4B [43]. Table 1 reports results for three configurations: *Memory-None*, where the model relies solely on prompt history; *Memory-Explicit*, where the model explicitly manages an external task memory via ADD/REMOVE actions; and *Memory-Memorizer*, corresponding to our dual-agent decomposition (Section V), where task-state updates are handled by a separate 0.6B Memorizer trained with MAGMA-GEN.

**Analysis.** Explicit task-state representation consistently improves performance over monolithic baselines for both model scales, indicating that loss of task-relevant information over long interaction histories is a primary failure mode. Introducing a separate Memorizer does not provide uniform gains, but changes the error profile. The Memorizer improves
<table><caption>TABLE I
EFFECT OF MEMORY ABSTRACTION ON PERFORMANCE. PERFORMANCE MEASURED ON OUR LONG-HORIZON MANIPULATION TASK</caption><thead><tr><th>Models</th><th>Parameters</th><th>Memory</th><th>CSR</th><th>CGC 2-3</th><th>CGC 4-5</th><th>CGC 6-9</th><th>CGC 10-16</th></tr></thead><tbody><tr><td>gpt-oss</td><td>20b</td><td>None</td><td>20.83</td><td>51.66</td><td>43.05</td><td>36.57</td><td>9.78</td></tr><tr><td>gpt-oss</td><td>20b</td><td>Explicit</td><td>30.55</td><td>63.33</td><td>52.77</td><td>40.55</td><td>13.94</td></tr><tr><td>gpt-oss</td><td>20b</td><td>Memorizer</td><td>19.44</td><td>70.00</td><td>56.49</td><td>23.90</td><td>25.01</td></tr><tr><td>qwen3</td><td>4b</td><td>None</td><td>11.11</td><td>20.0</td><td>18.51</td><td>32.778</td><td>12.5</td></tr><tr><td>qwen3</td><td>4b</td><td>Explicit</td><td>11.11</td><td>35.00</td><td>29.62</td><td>39.29</td><td>14.80</td></tr><tr><td>qwen3</td><td>4b</td><td>Memorizer</td><td>12.14</td><td>45.00</td><td>20.37</td><td>40.12</td><td>16.17</td></tr></tbody></table>

<table><caption>TABLE II
EFFECT OF REMOVING THE COACHING AND THE SIMUSER MECHANISMS OF MAGMA-GEN</caption><thead><tr><th>Mode</th><th>CGC 2-3</th><th>CGC 4-5</th><th>CGC 6-9</th><th>CGC 10-16</th></tr></thead><tbody><tr><td>no-coaching</td><td>58.33</td><td>42.51</td><td>38.12</td><td>27.25</td></tr><tr><td>no-user-sim</td><td>53.33</td><td>38.89</td><td>21.29</td><td>7.33</td></tr><tr><td>no-coach-user-sim</td><td>50.00</td><td>24.04</td><td>7.41</td><td>5.27</td></tr><tr><td>magma-gen</td><td>75.00</td><td>43.75</td><td>46.48</td><td>36.89</td></tr></tbody></table>

performance at very short and very long horizons, while degrading performance at intermediate horizons, suggesting a coordination mismatch between the frozen Commander and the independently trained task abstraction.

Overall, the results show that explicit task-state abstraction is the dominant factor for long-horizon performance. Externalizing memory through a separate model introduces a robustness-alignment trade-off and should be viewed as a structural mechanism for stabilizing very long-horizon behavior rather than a drop-in replacement for explicit memory management.

## VII. ABLATIONS

To assess the contribution of each component in MAGMA-GEN, we perform ablations under the same training budget and conditions as Section VI-A. As visible in Table II removing one of the components degrades performance, with the effect becoming more pronounced at longer horizons. Notably, removing the SimUser leads to a severe drop in long-horizon success, highlighting its role in maintaining diversity in interaction histories. Without simulated user variability, trajectories collapse to a narrow set of surface instruction patterns, causing the model to overfit and fail to generalize over extended horizons. In contrast, removing the Coaching mechanism results in a more moderate but consistent degradation. This is expected, as Coaching primarily improves recovery and credit assignment in failure-prone regions, rather than defining the overall interaction distribution. When the base model already exhibits moderate results, the impact of Coaching is reduced. Together, these results confirm that SimUser and Coaching play complementary roles in MAGMA-GEN. We also demonstrate the efficacy on other models architecture in Appendix.

## VIII. REAL ROBOT EXPERIMENTS

We conduct real robot experiment using a single robot Franka. We sort object on the table in specific containers according to evolving sorting rules, partial task interruption and interaction. We use Humanoid Path Planner [26] as our motion planner, M2T2 [53] as our 6D grasp pose estimator. We train a custom neural network to detect objects from a RGB image. Videos will be available in supplementary material.

## IX. LIMITATIONS

MAGMA-GEN is currently limited to text-only reasoning and does not incorporate visual inputs, which restricts its applicability to tasks requiring fine-grained spatial understanding. In addition, the framework assumes access to a pretrained model with sufficient reasoning capacity to generate viable initial roll-outs, and does not address learning from scratch. Finally, our experimental evaluation was conducted on a limited set of 100 tasks due to the high cost of collecting human supervision for fair comparison. While sufficient to highlight long-horizon trends, this scale does not fully capture the diversity of real-world robotic scenarios, motivating the development of larger and more diverse benchmarks in future work.

## X. CONCLUSION

This paper introduced MAGMA-GEN, a structured data-generation framework for long-horizon language-conditioned manipulation tasks. Our results show that, when combined with appropriate architectural decomposition, difficult tasks can be addressed by relatively small models through structured generation and execution. Particularly in settings where expert data are costly to obtain, or computing resources are limited. We also explore a dual-agent decomposition in settings where training is not affordable, showing that explicitly exposing task state abstractions can improve agent performance even without parameter updates. These findings suggest that progress on harder embodied tasks can be achieved by improving data quality and system design rather than parameter scaling alone.

## REFERENCES

[1] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian
Ichter, Alex Irpan, Eric Jang, Rosario M Jauregui Ru-
ano, Kyle Jeffrey, Sally Jesmonth, Nikhil Jayant Joshi,
Ryan C. Julian, Dmitry Kalashnikov, Yuheng Kuang,
Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu,
Carolina Parada, Peter Pastor, Jornell Quiambao, Kan-
ishka Rao, Jarek Rettinghouse, Diego M Reyes, Pierre
Sermanet, Nicolas Sievers, Clayton Tan, Alexander To-
shev, Vincent Vanhoucke, F. Xia, Ted Xiao, Peng Xu,
Sichun Xu, and Mengyuan Yan. Do as i can, not
as i say: Grounding language in robotic affordances.
In Conference on Robot Learning, 2022. URL https:
//api.semanticscholar.org/CorpusID:247939706.

[2] Anonymous. RLHF in an SFT way: From optimal solution to reward-weighted alignment. *Submitted to Transactions on Machine Learning Research*, 2025. URL https://openreview.net/forum?id=jewB0UhFuj. Under review.

[3] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. π0: A vision-language-action flow model for general robot control. ArXiv, abs/2410.24164, 2024. URL https://api.semanticscholar.org/CorpusID:273811174.

[4] Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, brian ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Sprin- genberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. π₀.₅: a vision-language-action model with open-world generalization. In Joseph Lim, Shuran Song, and Hae-Won Park, editors, *Proceedings of The 9th Conference on Robot Learning*, volume 305 of *Proceedings of Machine Learning Research*, pages 17–40. PMLR, 27–30 Sep 2025. URL https://proceedings.mlr press/v305/black25a.html.

[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, *Advances in Neural Information Processing Systems*, volume 33, pages 1877–1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/ file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.

[6] Huacan Chai, Zijie Cao, Maolin Ran, Yingxuan Yang, Jianghao Lin, pengxin, Hairui Wang, Renjie Ding, Ziyu Wan, Muning Wen, Weiwen Liu, Weinan Zhang, Fei Huang, and Ying Wen. Parl-mt: Learning to call functions in multi-turn conversation with progress awareness. ArXiv, abs/2509.23206, 2025. URL https://api. semanticscholar.org/CorpusID:281675131.

[7] Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M. Turner, Eric Undersander, and Tsung-Yen Yang. Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. In *International Conference on Learning Representations (ICLR)*, 2025. alphabetical author order.

[8] Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner Fabien Roger Vlad Mikulik, Sam Bowman, Jan Leike Jared Kaplan, et al. Reasoning models don't always say what they think. Alignment Science Team, Anthropic, 2025. URL https://www.anthropic.com/research/ reasoning-models-dont-say-think.

[9] Zhaorun Chen, Zhuokai Zhao, Kai Zhang, Bo Liu, Qi Qi, Yifan Wu, Tarun Kalluri, Sara Cao, Yuanhao Xiong, Haibo Tong, Huaxiu Yao, Hengduo Li, Jiacheng Zhu, Xian Li, Dawn Xiaodong Song, Bo Li, Jason Weston, and Dat Huynh. Scaling agent learning via experience synthesis. ArXiv, abs/2511.03773, 2025. URL https://api.semanticscholar.org/CorpusID:282812949.

[10] Wonje Choi, Jooyoung Kim, and Honguk Woo. Nesyp: Neurosymbolic proceduralization for efficient embodied reasoning. *Advances in Neural Information Processing Systems*, 10 2025. doi: 10.48550/arXiv.2510.19429.

[11] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In *Advances in Neural Information Processing Systems (NeurIPS)*, volume 30, 2017. URL https://arxiv.org/abs/1706.03741.

[12] Murtaza Dalal, Tarun Chiruvolu, Devendra Singh Chap-
lot, and Ruslan Salakhutdinov. Plan-seq-learn: Language
model guided rl for solving long horizon robotics tasks.
In *ICLR 2024*, 2024.

[13] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. PaLM-e: Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/ file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
An embodied multimodal language model. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara En-gelhardt, Sivan Sabato, and Jonathan Scarlett, editors, *Proceedings of the 40th International Conference on Machine Learning*, volume 202 of *Proceedings of Machine Learning Research*, pages 8469–8488. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/driess23a.html.

[14] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. In *Proceedings of the 41st International Conference on Machine Learning*, ICML’24. JMLR, 2024.

[15] Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. *ArXiv*, abs/2504.11536, 2025. URL https://api.semanticscholar.org/CorpusID:277824366.

[16] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, et al. Deepseeker1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948.

[17] Yiduo Guo, Zhen Guo, Chuanwei Huang, Zi-Ang Wang, Zekai Zhang, Haofei Yu, Huishuai Zhang, and Yikang Shen. Synthetic data r1: Task definition is all you need, 2025. URL https://arxiv.org/abs/2505.17063.

[18] Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, Danny Driess, Michael Equi, Adnan Esmail, Yunhao Fang, Chelsea Finn, Catherine Glossop, Thomas Godden, Ivan Goryachev, Lachy Groom, Hunter Hancock, Karol Hausman, Gashon Hussein, Brian Ichter, Szymon Jakubczak, Rowan Jen, Tim Jones, Ben Katz, Liyiming Ke, Chandra Kuchi, Marinda Lamb, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Yao Lu, Vishnu Mano, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Charvi Sharma, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, Will Stoeckle, Alex Swerdlow, James Tanner, Marcel Torne, Quan Vuong, Anna Walling, Haohuan Wang, Blake Williams, Sukwon Yoo, Lili Yu, Ury Zhilinsky, and Zhiyuan Zhou. $\pi_0^{*0.6}$: a vla that learns from experience, 2025. URL https://arxiv.org/abs/2511.14759.

[19] Minki Kang, Wei-Ning Chen, Dongge Han, Huseyin A. Inan, Lukas Wutschitz, Yanzhi Chen, Robert Sim, and Saravan Rajmohan. Acon: Optimizing context compression for long-horizon llm agents, 2025. URL https://arxiv.org/abs/2510.00615.

[20] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Open-

VLA: An Open-Source Vision-Language-Action Model. In *Proceedings of The 8th Conference on Robot Learning*, volume 270 of *Proceedings of Machine Learning Research*, pages 2679–2713, 2025. URL https://proceedings.mlr.press/v270/kim25c.html.

[21] Jakub Grudzien Kuba, Mengting Gu, Qi Ma, Yuandong Tian, and Vijai Mohan. Language self-play for data-free training, 2025. URL https://arxiv.org/abs/2509.07414.

[22] Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. *ArXiv*, abs/2005.11401, 2020. URL https://api.semanticscholar.org/CorpusID:218869575.

[23] Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. *Transactions of the Association for Computational Linguistics*, 12:157–173, 02 2024. doi: 10.1162/tacl_a_00638.

[24] Zhiyuan Ma, Jiayu Liu, Xianzhen Luo, Zhenya Huang, Qingfu Zhu, and Wanxiang Che. Advancing tool-augmented large language models via meta-verification and reflection learning. In *Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2*, KDD ’25, page 2078–2089, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400714542. doi: 10.1145/3711896.3736835. URL https://doi.org/10.1145/3711896.3736835.

[25] Oier Mees, Lukás Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. *IEEE Robotics and Automation Letters*, 7:7327–7334, 2021. URL https://api.semanticscholar.org/CorpusID:244908821.

[26] Joseph Mirabel, Steve Tonneau, Pierre Fernbach, Anna-Kaarina Seppälä, Mylène Campana, Nicolas Mansard, and Florent Lamiraux. Hpp: A new software for constrained motion planning. In *2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*, pages 383–389, 2016. doi: 10.1109/IROS.2016.7759083.

[27] mplib contributors. mplib: Motion planning library, 2023. URL https://motion-planning-lib.readthedocs.io/latest/. Accessed: January 2026.

[28] OpenAI. gpt-oss-120b gpt-oss-20b model card, 2025. URL https://arxiv.org/abs/2508.10925.

[29] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave,
K. Cho, and A. Oh, editors, *Advances in Neural Information Processing Systems*, volume 35, pages 27730–27744. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf.

[30] Charles Packer, Vivian Fang, Shishir G. Patil, Kevin Lin, Sarah Wooders, and Joseph E. Gonzalez. Memgpt: Towards llms as operating systems. *CoRR*, abs/2310.08560, 2023. doi: 10.48550/ARXIV.2310.08560. URL https://doi.org/10.48550/arXiv.2310.08560.

[31] Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tur, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. *ArXiv*, abs/2504.13958, 2025. URL https://api.semanticscholar.org/CorpusID:277955419.

[32] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In *Advances in Neural Information Processing Systems (NeurIPS) 2023*, 2023. URL https://arxiv.org/abs/2205.30351. Oral Presentation.

[33] Ning Shang, Yifei Liu, Yi Zhu, Li Lyna Zhang, Weijiang Xu, Xinyu Guan, Buze Zhang, Bingcheng Dong, Xudong Zhou, Bowen Zhang, Ying Xin, Ziming Miao, Scarlett Li, Fan Yang, and Mao Yang. rStar2-Agent: Agentic Reasoning Technical Report, August 2025. URL http://arxiv.org/abs/2508.20722.

[34] Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, and Luke Zettlemoyer. Spurious rewards: Rethinking training signals in rlvr, 2025. URL https://arxiv.org/abs/2506.10947.

[35] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023. URL https://openreview.net/forum?id=vAElhFcKW6.

[36] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. *2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 10737–10746, 2019. URL https://api.semanticscholar.org/CorpusID:208617407.

[37] Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B. Tenenbaum, Leslie Kaelbling, and Michael Katz. Generalized planning in pddl domains with pretrained large language models. In *Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence*, AAAI’24/IAAI’24/EAAI’24. AAAI Press, 2024. ISBN

978-1-57735-887-9. doi: 10.1609/aaai.v38i18.30006. URL https://doi.org/10.1609/aaai.v38i18.30006.

[38] Chan Hee Song, Jiaman Wu, Clay Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. *2023 IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 2986–2997, 2022. URL https://api.semanticscholar.org/CorpusID:254408960.

[39] Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-based trajectory optimization of LLM agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 7584–7600, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.409. URL https://aclanthology.org/2024.acl-long.409/.

[40] Andrew Szot, Bogdan Mazoure, Harsh Agrawal, R. Von Hjelm, Zsolt Kira, and Alexander Toshev. Grounding multimodal large language models in actions. In *Advances in Neural Information Processing Systems*, volume 37, 2024.

[41] Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Viswesh Nagaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. *Robotics: Science and Systems*, 2025.

[42] Generalist AI Team. Gen-0: Embodied foundation models that scale with physical interaction. *Generalist AI Blog*, 2025. https://generalistai.com/blog/preview-uqlxvb-bb.html.

[43] Owen Team. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.

[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, *Advances in Neural Information Processing Systems*, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.

[45] Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. In *Proceedings of the 37th International Conference on Neural Information Processing Systems*, NIPS ’23, Red Hook, NY, USA, 2023. Curran Associates Inc.

[46] Yu Wang, Dmitry Krotov, Yuanzhe Hu, Yifan Gao, Wangchunshu Zhou, Julian McAuley, Dan Gutfreund,
Rogerio Feris, and Zexue He. M+: Extending memoryllm with scalable long-term memory, 2025. URL https://arxiv.org/abs/2502.00592.

[47] Feijie Wu, Weiwu Zhu, Yuxiang Zhang, Soumya Chatterjee, Jiarong Zhu, Fan Mo, Rodin Luo, and Jing Gao. Portool: Tool-use llm training with rewarded tree. ArXiv, abs/2510.26020, 2025. URL https://apisemanticscholar. org/CorpusID:282592220.

[48] Peng Xia, Kaide Zeng, Jiaqi Liu, Can Qin, Fang Wu, Yiyang Zhou, Caiming Xiong, and Huaxiu Yao. Agent0: Unleashing self-evolving agents from zero data via tool-integrated reasoning, 2025. URL https://arxiv.org/abs/2511.16043.

[49] Mengdi Xu, Peide Huang, Wenhao Yu, Shiqi Liu, Xilun Zhang, Yaru Niu, Tingnan Zhang, Fei Xia, Jie Tan, and Ding Zhao. Creative robot tool use with large language models. In CoRL 2023 Workshop on Language and Robot Learning: Language as Grounding, 2023.

[50] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.

[51] Wenhao Yu, Zhenwen Liang, Chengsong Huang, Kishan Panaganti, Tianqing Fang, Haitao Mi, and Dong Yu. Guided self-evolving llms with minimal human supervision, 2025. URL https://arxiv.org/abs/2512.02472.

[52] Yuanqing Yu, Zhefan Wang, Weizhi Ma, Zhicheng Guo, Jingtao Zhan, Shuai Wang, Chuhan Wu, Zhiqiang Guo, and Min Zhang. Steptool: Enhancing multi-step tool usage in llms via step-grained reinforcement learning, 2024. URL https://api. semanticscholar.org/CorpusID: 273233670.

[53] Wentao Yuan, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. M2t2: Multi-task masked transformer for object-centric pick and place. In 7th Annual Conference on Robot Learning, 2023.

[54] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? *arXiv preprint arXiv:2504.13837*, 2025.

[55] Jianke Zhang, Yanjiang Guo, Xiaoyu Chen, Yen-Jen Wang, Yucheng Hu, Chengming Shi, and Jianyu Chen. Hirt: Enhancing robotic control with hierarchical robot transformers. In Pulkit Agrawal, Oliver Kroemer, and Wolfram Burgard, editors, *Proceedings of The 8th Conference on Robot Learning*, volume 270 of *Proceedings of Machine Learning Research*, pages 933–946. PMLR, 06-09 Nov 2025. URL https://proceedings.mlr.press/v270/zhang25b.html.

[56] Kai Zhang, Xiangchao Chen, Bo Liu, Tianci Xue, Zeyi Liao, Zhihan Liu, Xiyao Wang, Yuting Ning, Zhaorun Chen, Xiaohan Fu, Jian Xie, Yuxuan Sun, Boyu Gou, Qi Qi, Zihang Meng, Jianwei Yang, Ning Zhang, Xian Li, Ashish Shah, and Yifan Wu. Agent learning via early experience, 10 2025.

[57] Shiduo Zhang, Zhe Xu, Peiju Liu, Xiaopeng Yu, Yuan Li, Qinghui Gao, Zhaoye Fei, Zhangyue Yin, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. Vlabench: A large-scale benchmark for language-conditioned robotics manipulation with long-horizon reasoning tasks, 2024. URL https://arxiv.org/abs/2412.18194.

[58] Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. In *Proceedings of Robotics: Science and Systems*, Daegu, Republic of Korea, July 2023. doi: 10.15607/RSS.2023.XIX.016.

[59] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski, Yao Lu, Sergey Levine, Lisa Lee, Tsang-Wei Edward Lee, Isabel Leal, Yuheng Kuang, Dmitry Kalashnikov, Ryan Julian, Nikhil J. Joshi, Alex Irpan, Brian Ichter, Jasmine Hsu, Alexander Herzog, Karol Hausman, Keerthana Gopalakrishnan, Chuyuan Fu, Pete Florence, Chelsea Finn, Kumar Avinava Dubey, Danny Driess, Tianli Ding, Krzysztof Marcin Choromanski, Xi Chen, Yevgen Chebotar, Justice Carbajal, Noah Brown, Anthony Brohan, Montserrat Gonzalez Arenas and Kehang Han. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Jie Tan, Marc Toussaint, and Kourosh Darvish, editors, *Proceedings of The 7th Conference on Robot Learning*, volume 229 of *Proceedings of Machine Learning Research*, pages 2165–2183. PMLR, 06-09 Nov 2023. URL https://proceedings.mlr.press/v229/zitkovich23a.html.
# Supplementary Material for
Addressing Long-Horizon Failure in
Language-Grounded Robotics

## 1 Overview

This supplementary document provides additional material supporting the paper "Addressing Long-Horizon Failure in Language-Grounded Robotics". Its purpose is to clarify the task structure, evaluation protocol, and observed behavior of the proposed pipeline under increased task complexity.

We include extended task examples illustrating long-horizon reasoning, dynamic rule updates, and execution failure handling, as well as additional experimental results and ablations. These experiments are intended to characterize robustness and failure modes in controlled settings, rather than to establish state-of-the-art performance or large-scale generalization.

Specifically:

* real-robot.mp4 presents two real-robot demonstrations (button pressing and object sorting) executed on a Franka Emika Research robot.

* Section 2 details the task formulation and provides representative task examples.

* Section 3 reports additional results on task variations and a failure analysis.

* Section 4 extends the ablation study to alternative model architectures and learning objectives.

* Section 5 briefly describes the dynamic task construction process.

As in the main paper, all methods are evaluated on the same fixed set of tasks across our evaluated sorting scenario. Execution is deterministic except for injected failures. Reported numbers are averaged over 3 runs.
## 2 Examples of Task Configuration

As described in Section 3 of the main paper, a task is defined by a sequence of goals, a set of constraints, task attributes encoding the symbolic state, and the tool API available to the agent. This section provides additional details on the evaluated scenario, task structure, and analysis metrics used in our experiments.

### 2.1 Evaluated Scenario

We evaluate MAGMA-GEN and baselines in a single controlled manipulation scenario. This restriction is intentional: collecting sufficient human demonstrations for long-horizon, dynamically evolving tasks is costly, and using a single scenario allows us to isolate the effect of data generation quality rather than data quantity.

The scenario consists of a single-robot object sorting task. The agent is provided with symbolic information about object identities and available target areas through task attributes, and interacts with the environment using a fixed set of tools: *grab_object*, *deposit_item*, *launch_cycle* (a composite action executing a full sorting cycle), as well as tools to *add* and *remove* areas.

### 2.2 Tool API

The tool API is concatenated to the model input and defines the full action space available to the agent for a given task. The API remains fixed across all tasks within a scenario and is the only mechanism by which task-specific affordances are exposed to the model. An example of the tool API used in our evaluation scenario is shown below.

```json
[
  {
    "name": "grab_object",
    "description": "Take an object from the environment.",
    "parameters": {
      "object_name": {
        "description": "Name of the target entity
        the robot must grab.",
        "type": "str"
      }
    }
  },
  {
    "name": "deposit_item",
```
"description": "Move the robot gripper above
the target area and release the object.",
"parameters": {
  "drop_zone": {
    "description": "The name of the area where the
object should be placed.",
    "type": "str"
  }
}
},
{
  "name": "add_storage_area",
  "description": "Register a new area where objects
may be placed or stored.",
  "parameters": {
    "area_name": {
      "description": "The unique name of the storage
location to define.",
      "type": "str"
    }
  }
},
{
  "name": "remove_location",
  "description": "Discard a known location from
the environment model.",
  "parameters": {
    "target_location": {
      "description": "Name of the area to unregister.",
      "type": "str"
    }
  }
},
{
  "name": "start_global_transfer_cycle",
  "description": "Launch a default cycle to sort
all objects to their assigned areas.",
  "parameters": {
    "distribution_plan": {
      "description": "A dictionary mapping each
object to its corresponding target area.
"type": "dict"
}
}
}
]

## 2.3 Task Attributes

Task attributes encode symbolic information such as object identities, controllable robots, and known target areas. They are represented as a dictionary concatenated to the model input. An example from the evaluation scenario is shown below.

```json
{
  "objects": ["obj_12", "obj_87", "obj_45"],
  "known_robots": ["ur10"],
  "targetareas": ["area2", "area1", "area4"]
}
```

## 2.4 Task Examples

Tables 1 and 2 present representative tasks of horizon length, rule updates, and failure injection frequency, but do not constitute the hardest evaluated instances. Both tasks require constrained reasoning and precise tool invocation across multiple interaction steps. Table 2 further includes an explicit execution failure: at Step 7, the motion planner is forced to fail, and the agent is evaluated on its ability to acknowledge the failure and continue the task without corrupting its internal symbolic state.

Such failure injection is used during evaluation to assess robustness to execution errors, which are common in long-horizon robotic tasks. Success in these tasks therefore reflects not only correct planning, but also stable state tracking and recovery behavior.

## 2.5 Human Expert Baseline Construction

To establish a high-quality ceiling for long-horizon performance, we curated the Human baseline. Due to the lack of existing benchmarks combining high-dimensional interaction with long horizons (10+ steps), we developed a dedicated scripting GUI interface using *gradio*.

**Data Collection Protocol:** A team of three robotics engineers performed 50 aggregate hours of data writing and review. A random situation (instruction, tools,
goals, constraints) is generated from the task definition and the human must answer it. This data is then put into the reviewing queue of another engineer who must review it before adding it to the database.

Despite this intensive manual effort, the resulting 470 scripts were primarily limited to "Gold Path" trajectories. This underscores the scalability challenge: while human experts provide high-precision data, they struggle to exhaustively map the "failure frontiers" that autonomous methods like MAGMA-GEN navigate in a fraction of the time.

**Synthetic Augmentation:** We also use some synthetic augmentation methods inspired by well-known methods such as: API randomization (varying tool names, parameters, etc.), paraphrasing (using *gpt-oss:20b*), and task attribute shuffling.

<table><caption>Table 1: Task 10 : involving dynamic object assignment evolution.</caption><thead><tr><td>Step</td><td>User Instruction</td><td>Expected Robot Behavior</td></tr></thead><tbody><tr><td>1</td><td>Default assignments are ref_obj_1 and ref_obj_2 to area2, and ref_obj_3 to area3.</td><td>Acknowledge and store the object-area assignments.</td></tr><tr><td>2</td><td>Run a cycle now.</td><td>Execute a manipulation cycle placing ref_obj_1 and ref_obj_2 in area2, and ref_obj_3 in area3.</td></tr><tr><td>3</td><td>Wait for future instructions.</td><td>Acknowledge and remain idle without executing actions.</td></tr><tr><td>4</td><td>Add a new area area4. With which area was ref_obj_1 assigned before?</td><td>Add area4 and correctly recall that ref_obj_1 was previously assigned to area2.</td></tr><tr><td>5</td><td>Can I remove area2 or is it still associated with some objects?</td><td>Answer that area2 is still associated with ref_obj_2.</td></tr><tr><td>6</td><td>Run a cycle with this new configuration.</td><td>Execute manipulation using the updated object-area assignments.</td></tr></tbody></table>

## 3 Additional results on task

The results in the main paper show that performance degrades with task horizon length, while MAGMA-GEN consistently improves stage completion at longer horizons (24%). In this section, we provide additional results aimed at clarifying the nature of these improvements by analyzing failure modes, task dynamics, and recovery behavior on harder task variants. Additionally, we show that MAGMA-GEN allows models to generalize across various task objectives.
<table><caption>Table 2: Task 34 : involving dynamic rule overrides and execution failure recovery.</caption><thead><tr><td>Step</td><td>User Instruction</td><td>Expected Robot Behavior</td></tr></thead><tbody><tr><td>1</td><td>The default cycle rule is to send all objects to area1.</td><td>Acknowledge and store the default sorting rule.</td></tr><tr><td>2</td><td>Due to a power failure, area1 must not be used. Use area2 instead.</td><td>Acknowledge that area1 is unavailable and update the default rule to use area2.</td></tr><tr><td>3</td><td>Launch a cycle according to your default assignment.</td><td>Execute a manipulation cycle placing all objects in area2.</td></tr><tr><td>4</td><td>Area1 is back, you can reuse it normally.</td><td>Acknowledge and restore area1 as the default target area.</td></tr><tr><td>5</td><td>Launch a cycle.</td><td>Execute a manipulation cycle placing all objects in area1.</td></tr><tr><td>6</td><td>By the way, area3 must be used now for ref_obj_2.</td><td>Acknowledge and update the sorting rule for ref_obj_2 to use area3.</td></tr><tr><td>7</td><td>Launch a cycle now.</td><td>Attempt execution using the updated rules. A motion planning failure is triggered during execution, the robot must detect and acknowledge</td></tr><tr><td>8</td><td>What are your current sorting rules?</td><td>Correctly report that ref_obj_1 and ref_obj_3 are assigned to area1, and ref_obj_2 to area3.</td></tr></tbody></table>

## 3.1 Failure Analysis

Beyond binary task success, we analyze failures at the level of individual task stages. Each task is decomposed into a sequence of stages, and each stage is annotated according to the primary capability it stresses: (i) constrained reasoning (CR), involving logical consistency under constraints, (ii) multi-step execution (MS), requiring coordinated tool use across multiple actions, and (iii) long-term memorization (LM), requiring recall of the symbolic state established several interaction steps earlier.

For each method, we report the proportion of successfully completed stages for each capability category, aggregated across all evaluated tasks. A stage is considered successful if the expected symbolic or behavioral outcome is satisfied, independently from success or failure in subsequent stages. This stage-level view allows us to localize where failures occur along long-horizon interactions, rather than only measuring final task completion.

Table 3 highlights a fundamental trade-off introduced by the dual-agent architecture. While overall task success remains limited for all methods due to the difficulty of the scenario, MAGMA-GEN significantly improves execution logic
<table><caption>Table 3: Stage-level completion rates by capability category, used to analyze dominant failure modes across methods.</caption><thead><tr><th>Data Generation</th><th>CGC 2-5</th><th>CGC 6-16</th><th>MS</th><th>CR</th><th>LM</th></tr></thead><tbody><tr><td>Human</td><td>51.25</td><td>18.75</td><td>20.00</td><td>13.25</td><td>12.25</td></tr><tr><td>Human+Synthetic</td><td>59.25</td><td>23.65</td><td>20.00</td><td>28.75</td><td>23.00</td></tr><tr><td><strong>MAGMA-GEN</strong></td><td><strong>59.36</strong></td><td>42.10</td><td><strong>55.00</strong></td><td><strong>56.00</strong></td><td>23.75</td></tr><tr><td><strong>MAGMA-GEN + Memorizer</strong></td><td>35.62</td><td><strong>45.6</strong></td><td>50.00</td><td>40.00</td><td><strong>26.25</strong></td></tr></tbody></table>

(CR and MS). However, long-term memorization (LM) remains a bottleneck for the monolithic 1.7B model. The introduction of the Memorizer partially mitigates long-term memorization failures, yielding consistent but modest LM gains, while more clearly improving long-horizon task completion.

However, this gain comes at the cost of a Coordination Penalty in shorter horizons. We observe a significant performance drop in CGC 2-5 and Constrained Reasoning (CR) when the Memorizer is active. We attribute this to a **Semantic Bottleneck**: without joint fine-tuning of the model's intent $\mathcal{T}$ and the Memorizer's abstraction of $z^t$, the two agents occasionally misalign on state updates. For simple tasks, this extra layer of abstraction acts as a noise source; for complex tasks, it acts as a necessary stabilizer. This identifies a critical open challenge for future work: the need for Communication-Aware Training to bridge the language barrier between specialized robotic agents.

## 3.2 Answer Comparison

Tab 4 example illustrates that while both models ultimately fail due to execution-level predicate violations, MAGMA-GEN maintains a consistent symbolic state and successfully executes additional stages. This behavior is consistent with the stage-level failure analysis reported in Table 3 where we see that MAGMA-GEN increases performance on long-horizon tasks.

## 3.3 Qualitative Cross-Domain Versatility

To demonstrate that MAGMA-GEN is a general-purpose framework rather than a task-specific heuristic, we applied the pipeline to four distinct robotic domains. Due to the high cost of collecting human demonstrations in these environments, we report qualitative results only, without quantitative comparison with baselines. We show one trace resulting from agent execution on diverse scenarios:
<table><caption>Table 4: Comparison of failure behavior between the HUMAN+Synthetic model and MAGMA-GEN on the same long-horizon task. Both models fail predominantly due to execution-related predicate violations, but MAGMA-GEN preserves correct symbolic state and completes more stages before failure.</caption><thead><tr><th>Stage</th><th>User Instruction (Summary)</th><th>Human Model</th><th>MAGMA-GEN</th></tr></thead><tbody><tr><td>1</td><td>Define maintenance vs cleaning cycle semantics</td><td>Correctly acknowledges rule</td><td>Correctly acknowledges rule</td></tr><tr><td>2</td><td>Mark ref_obj_1 and ref_obj_3 as maintenance</td><td>Correctly updates state</td><td>Correctly updates state</td></tr><tr><td>3</td><td>Maintenance cycle in area3</td><td>Fails: incomplete assignment (misses ref_obj_1)</td><td>Correctly sorts ref_obj_1 and ref_obj_3</td></tr><tr><td>4</td><td>Cleaning cycle in area1</td><td>Correctly sorts ref_obj_2</td><td>Correctly sorts ref_obj_2</td></tr><tr><td>5</td><td>Unmark ref_obj_1 as maintenance</td><td>Correctly updates state</td><td>Correctly updates state</td></tr><tr><td>6</td><td>Cleaning cycle in area1</td><td>Fails: predicate violation</td><td>Fails: predicate violation</td></tr><tr><td>7</td><td>Maintenance cycle in area2</td><td>Fails: incorrect object set</td><td>Fails: incorrect object set</td></tr></tbody></table>

* **Delivery**: A scenario where a robot must package objects in a box according to a recipe.

* **Bi-Robot Sorting**: A scenario where two robots must sort objects.

* **Laundry**: A scenario where a robot is in charge of putting only specific objects into the washing machine before starting the washing cycle.

* **Press Buttons**: A scenario where a robot must press buttons under constraint (in videos).

We trained one Qwen3:1.7B model for all tasks using zero human demonstrations, relying solely on autonomous simulation roll-outs and coaching. Section 6 shows some extracts of interaction traces from these scenarios. These traces confirm that MAGMA-GEN's coaching and user-simulation mechanisms successfully transfer to tasks requiring temporal sequencing (Button Press), non-unique multi-step tasks (Laundry), and multi-agent coordination (Bi-Robot Sorting). While these qualitative scenarios are not accompanied by detailed quantitative metrics, they highlight recurring challenges in long-horizon and multi-robot task execution under our task formulation. These observations motivate the development of
<table><caption>Table 5: Effect of memory abstraction on performance with a 3b model in zero-shot settings. Performance measured on our long-horizon manipulation task</caption><thead><tr><th>Models</th><th>Memory</th><th>CSR</th><th>CGC 2-3</th><th>CGC 4-5</th><th>CGC 6-9</th><th>CGC 10-16</th></tr></thead><tbody><tr><td>smolLM3</td><td>None</td><td>0.0</td><td>11.67</td><td>0.0</td><td>0.0</td><td>1.4</td></tr><tr><td>smolLM3</td><td>Self-Managed</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>smolLM3</td><td>Memorizer</td><td>2.45</td><td>16.67</td><td>10.19</td><td>17.96</td><td>9.47</td></tr></tbody></table>

a standardized benchmark for multi-robot, long-horizon manipulation, which we are currently designing to enable systematic and reproducible evaluation in future work.

# 4 Extended Ablations

## 4.1 Model Architecture

We evaluate the effect of model architecture and memory abstraction under the same limited training and evaluation conditions as in the main paper. These experiments are intended to assess whether the observed trends persist for smaller models and alternative architectural choices, rather than to establish absolute performance.

Table 5 reports results for a small language model evaluated in a zero-shot setting, with and without an explicit Memorizer module. Introducing a Memorizer substantially increases stage completion across longer horizons, while performance without memory abstraction rapidly collapses. This suggests that, for very limited models, explicit state abstraction plays a critical role in long-horizon task execution.

Table 6 evaluates data-generation strategies on a small model that is not pre-trained for tool calling. Despite this limitation, MAGMA-GEN consistently outperforms human and human+synthetic data under the same constrained training budget. These results indicate that the benefits of execution-grounded data generation are not restricted to models with prior tool-use pretraining.

## 4.2 Preference Learning

We further investigate whether preference learning can improve performance when applied to data generated by MAGMA-GEN. Table 7 compares supervised fine-tuning (SFT) with and without Direct Preference Optimization (DPO), under the same limited training budget.
<table><caption>Table 6: Performance comparison between data-generation method on a llama2.3-1B, a small model not pretrained on function calling task.</caption><thead><tr><th>Training Method</th><th>CSR</th><th>CGC 2-3</th><th>CGC 4-5</th><th>CGC 6-9</th><th>CGC 10-16</th></tr></thead><tbody><tr><td>Human</td><td>0.0</td><td>35.83</td><td>27.78</td><td>24.44</td><td>16.32</td></tr><tr><td>Human+Synthetic</td><td>3.44</td><td>48.33</td><td>33.33</td><td>25.74</td><td>15.21</td></tr><tr><td><strong>MAGMA-GEN</strong></td><td><strong>8.34</strong></td><td><strong>61.67</strong></td><td><strong>37.96</strong></td><td><strong>40.37</strong></td><td><strong>27.02</strong></td></tr></tbody></table>

<table><caption>Table 7: Variation of training methods under the same limited training budget (5k data, 2k steps) using Qwen3:1.7B</caption><thead><tr><th>Training Method</th><th>CSR</th><th>CGC</th></tr></thead><tbody><tr><td>SFT-only <em>HUMAN+SYNTHETIC</em></td><td>16.66</td><td>33.13</td></tr><tr><td><strong>SFT-only MAGMA-GEN</strong></td><td><strong>20.83</strong></td><td><strong>45.40</strong></td></tr><tr><td>SFT+DPO <em>MAGMA-GEN</em></td><td>5.55</td><td>31.29</td></tr></tbody></table>

In this setting, applying DPO after SFT leads to reduced performance and a higher frequency of formatting and tool-call errors. This suggests that optimizing preferences over full model outputs may interfere with strict output constraints required for tool invocation. A more targeted preference formulation is likely necessary to benefit from preference learning in this context.

# 5 Curriculum Task Building

As described in the main paper, tasks are constructed from human-defined task specifications composed of elementary stages, a tool API, and a simulated environment. Without an explicit curriculum, on-policy data generation becomes dominated by task configurations that the current model already solves reliably, leading to highly imbalanced datasets in which failures and corrective behaviors are underrepresented.

To mitigate this effect, we employ a curriculum that selects task configurations based on feedback from prior executions. The curriculum favors task structures that expose under-represented or failure-prone transitions in the symbolic state $z^t$. Concretely, it controls how $z^t$ evolves by varying when constraints are introduced, how many constraints are active, how long they must be maintained, and whether they are revised or reasserted across stages.

Task instances are generated either from human-defined presets or by composing new sequences of goals and constraints. Initialization parameters such as task
difficulty, noise level, and horizon length are adjusted dynamically, while instructions, goals, and constraints are randomized within the predefined task templates. Allowing to sample an unbound set of tasks from a limited task definition.

# 6 Qualitative trace on task variations

These qualitative traces illustrate how the same interaction structure and tool-calling mechanism can be instantiated across distinct task domains. They are provided to demonstrate applicability rather than execution performance.

## Illustrative example (Delivery).

* **User:** Here is the default recipe for all future packaging: apple.

* **Model:** I have set the default recipe for all future packaging to apple.

* **User:** I need four cycles right now. Use manufacturing order BOON.

* **Model:**

initiate_production_run(BOON, [apple], 4)

* **System:** Production run completed successfully.

* **Model:** Four deliveries have been completed using the apple recipe.

* **User:** Could you add apple, banana, and grape to the recipe?

* **Model:** I have updated the recipe to include apple, banana, and grape.

* **User:** Please start 19 cycles using manufacturing order S8WP.

* **Model:**

initiate_production_run(S8WP, [apple, apple, banana, grape], 19)

* **System:** Production run completed successfully.

* **Model:** 19 deliveries have been completed.

* ...
## Illustrative example (Bi-Robot Sorting).

* **User:** Robot1 can access zone1 and robot2 can access zone2. Both can go to zone3.

* **Model:** I understand that robot1 can access zone1 and robot2 can access zone2.

* **User:** Zone1 is the default location for objA and objC, and zone2 for objB.

* **Model:** I understand the default assignments.

* **User:** Please clean the table now.

* **Model:**

robot1:store(objA, zone1)

robot2:store(objB, zone2)

* **System:** objA stored in zone1; objB stored in zone2.

* **Model:**

robot1:store(objC, zone1)

* **System:** objC stored in zone1.

* **Model:** All objects have been stored.

* **User:** The rule has changed: objA must now go to zone2. Can you take it from zone1 and send it to zone2?

* ...

## Illustrative example (Laundry).

* **User:** When I ask you to clean my work clothes, you should wash my hat and jeans and add soap. Correct?

* **Model:** I understand that work clothes include the hat and the jeans, and that soap must be added.

* **User:** Please clean them now.

* **Model:**
put_in_machine(hat)

* **System:** Hat is in the machine.

put_in_machine(jeans)

* **System:** Jeans is in the machine.

put_in_machine(soap)

* **System:** Soap is in the machine.

start_washing_cycle()

* **System:** Washing cycle launched successfully.

* **Model:** The washing cycle has been started as instructed.

* **User:** I need you to clean my work clothes again.

* ...