From Coding Agents to Physical Agents: Code as a New Interface Between AI and the Physical World

How can an AI move a red block from a table into a blue box?

One approach feeds a camera image and a language instruction to a policy model such as a VLA, which then outputs actions. Another has the model write a program first: find the block, choose a grasp pose, move the arm, check whether the grasp succeeded, then place the block in the box.

The second approach looks a lot like how a coding agent works: understand the task, write code, run it, read the feedback, and revise. This time, though, the program moves a robot arm and objects on a table.

Using a program to express a goal and an external system to carry it out is not limited to robotics.

The same pattern shows up in scene generation and simulation, world models, and video editing.

That observation led us to create awesome-code-as-x, a collection of papers and projects organized into four areas: Code as Policy, Code as World, Programmable World Models, and Edit Graph.

This article follows those four threads and asks: when code becomes the interface between a model and external systems, how should we rethink robotics, world models, and coding agents themselves?

01 Why code: turning reasoning into an executable process

Start with a simplified example.

The pseudocode below is only meant to explain the idea; a real system still needs perception, control, and safety interfaces.

block = locate("red block")
box = locate("blue box")

if reachable(block):
    grasp(block)
    if grasp_succeeded():
        place_inside(box)

The program states both the order of operations and the conditions: whether the object is reachable, whether the grasp succeeded, and whether the next step should run. It records these decisions in a form we can inspect.

Code can serve as an executable intermediate representation. Language describes the goal, the program specifies the steps, and the execution stack carries them out in the environment.

Code helps in a few direct ways.

First, logic can be stated explicitly. Loops, branches, calls, and parameters are visible in the program, so a researcher can inspect why a step happened.

Second, existing capabilities can be composed. Interfaces let a program combine detectors, planners, controllers, and learned skills to perform a task.

Third, programs can be edited locally. After a failed grasp, the system can change the approach direction, retry conditions, or step order, then see whether the change helped.

Being able to read the code does not mean you understand the whole system. Behind a grasp() call there may still be a complex neural network. The program makes part of the decision explicit; the underlying modules still need separate reliability checks.

For an agent, external feedback can guide the next edit.

A runtime error is feedback. So is an arm that moves while the object stays put. To interpret the second case, the system needs vision, sensors, and state estimation. Revising the program is more likely to help when it can distinguish a perception error from a failed grasp or a flawed plan.

So the core Code-as-X loop looks like this:

Understand the goal
+ observe the environment
             ↓
Generate or revise a program
             ↓
Execute → Check the result
             ↓
Adjust again ↺

Here X is the role the program plays: a robot policy, a scene description, world state and rules, or an editing pipeline. We use Code-as-X as an umbrella term, but each area has its own tasks, interfaces, and evaluation standards.

02 Code-as-Policy: robots can have a “programmer” too

Code as Policies (2022) is an important starting point for this approach. The researchers used a language model to generate robot policy code from instructions, combining perception and control APIs with logic and numerical computation. They demonstrated language-driven control on several real robot platforms. The original paper already included feedback loops and reactive control, so code policies were never just fixed lists of actions.

This creates a division of labor: the model writes the task program, while the robot stack provides sensing and execution. An instruction such as “put the object on the left” must be translated into spatial relations and a target pose. Planners and controllers then carry out the motion.

This raises a question: how much of the success comes from the model, and how much from tools prepared ahead of time?

If the interface already provides a “tidy the whole table” function, the model mostly decides when to call it. If it only exposes end-effector pose and gripper commands, the model has to work out more of the manipulation details. Both systems use code policies, but they leave different decisions to the model.

From writing programs to learning from failure

CaP-X evaluates this question systematically. It provides the interactive environment CaP-Gym and the benchmark CaP-Bench, testing models under different action abstractions, interaction modes, and perception conditions. Across 12 models, hand-designed abstractions improve performance, while removing those priors reveals clear limitations. Multi-turn interaction and structured feedback also improve results.

Evaluating a robot coding agent therefore means asking more than “does the generated code run?” What did the model see? Which tools could it call? What feedback did it receive after a failure? How many revisions was it allowed?

Act-Observe-Rewrite uses visual observations and structured trial results to regenerate a Python controller between trials. The paper evaluates this approach on three robosuite simulation tasks without gradient updates, using experience from earlier executions to revise the code.

Suppose the gripper approaches from above and hits a neighboring object. On the next trial, the model can change the approach direction. If that still fails, it can recheck the target location and gripper timing. The program gives it something specific to change, but another execution is needed to test whether the change works.

It helps to separate two kinds of “learning”.

One happens in context: the model uses feedback from previous attempts to edit the program without changing its weights.

The other happens in training: task feedback is used as a reward to update the code-generating model. CaP-RL in CaP-X explores this approach, training code-policy generation with verifiable rewards and reporting sim-to-real transfer.

These address two different questions: can the model improve at the current task through repeated attempts, and can training make it better at writing effective policies across tasks? The second takes Code-as-Policy beyond prompting and tool composition into policy learning.

Interfaces themselves are worth studying

Show-Harness (September 2026) takes a related approach: it exposes compact semantic action units to a vision–language model, then an embodiment-specific interpreter maps them to concrete actions. Connecting models to robots does not always require generating long Python programs; the design of the semantic action interface matters too.

Program length is not a measure of intelligence.

More useful comparisons ask whether the interface is expressive enough for the task, whether the model uses it correctly, and whether execution results provide enough feedback for the next decision.

RLE-Bench asks a broader question: can a general-purpose coding agent work as a robot learning engineer?

It evaluates more than whether policy code runs. Agents must observe, experiment, and revise under time, interaction, and compute budgets, then submit artifacts for evaluation under hidden physical conditions.

Tasks fall into four groups: direct robot control, training reusable policies, perception and state estimation, and mechanical design. The site lists 9 task families, 4 workflows, and 48 subtasks.

T01 compares three interface depths: bare action APIs, APIs plus perception and motion tools, then privileged object information. T02 asks an agent to build a harness for another agent and checks whether that second agent can solve a new task zero-shot.

03 How code policies compare with VLAs

If a VLA can combine vision, language, and action in one model, why add a code layer?

OpenVLA, for example, maps image and language inputs to action tokens, then decodes them into continuous actions. The policy learns from robot trajectory data.

Code policies place some of the logic for organizing robot behavior in a generated program. The two approaches offer different ways to change behavior: a VLA is mainly adapted through training or fine-tuning, while a code policy can be changed by editing its program, tool calls, and parameters.

The approaches can also be combined. Code can call neural policies, and learned policies can be part of larger, layered systems. For a multi-stage task, an agent could track progress and handle failures, call a VLA or skill suited to the current scene, and leave motion control to a low-level controller.

“Clear the table,” for example, requires decisions about what to move first, which objects can be placed together, and when the job is done. A single grasp still has to handle pose error, contact, and slip. Different components can handle the decisions they are suited to.

The table describes representative designs. Runtime, generalization, and task success still have to be compared under matched observations, interfaces, and tasks—not decided by the labels “VLA” or “code policy”.

The question is which behaviors should be learned from data, which logic should be written as programs at runtime, and how the two exchange state and feedback.

04 Code-as-World: models start “building worlds” for robots

So far, we have asked how a robot should act. The next question is where it can practice, and how to build a suitable environment.

A photorealistic kitchen image does not tell you whether a drawer opens. A cabinet mesh may look complete but still lack joints, motion ranges, and collision relationships between the drawer and the cabinet body. In robot simulation, these structures determine whether a task is well-posed.

Code-as-World turns scene construction into program generation. A scene program can specify object sizes, part relationships, layout, and motion constraints separately, then pass these descriptions to modeling or simulation tools. When a user asks to “widen the cabinet” or “change how far the drawer opens,” the system can try changing parameters or the code for individual parts.

SceneCode is one example. It generates indoor layouts and object requests from natural language, writes Blender Python code for individual parts, uses execution feedback to repair and refine the result, and exports SDF assets for physics simulation. The paper focuses on editable scenes, object structure, and joint metadata, and also evaluates downstream robot interaction.

A cabinet with drawers needs more than surface geometry. It also needs a parent–child relationship between the body and each drawer, a motion axis, and a travel range. The program records these relationships explicitly so they can be checked and edited later.

Importing a program into a simulator is only one step. Geometry, collisions, and joint settings still need testing. We also need to check whether the environment is suitable for training and evaluation. Scene programs make these problems easier to locate and fix.

The reverse direction: reconstructing an executable scene from observation

If programs can generate worlds, a natural follow-up is: can we infer a program that describes the world from real observations?

NeoWorld-Pro starts from a single RGB image, generates executable programs for object geometry, joints, and physical attributes, and uses physics-engine checks to revise the scene.

You can think of this as writing an “executable manual” for a real scene. The camera captures appearance; the system tries to recover the structure needed for interaction.

The metaphor has limits. A single image rarely provides enough information to uniquely determine mass, friction, or hidden structure. Reconstruction therefore involves inference. A simulator can reject some physically implausible settings, but it cannot prove that the reconstruction exactly matches the real object.

For robots, this could make it easier to go from observing an environment to testing actions in a model of it. Further interaction could help calibrate the program and make the world model better suited to the current task.

05 From writing scenes to writing world state and rules

A scene alone is not a complete world model.

A scene describes what is present and how it is organized. A world model also describes how it changes after an action or event.

Suppose a cup falls and breaks, then the camera looks away and returns. The system still needs to remember that the cup is broken. Whether the cup is intact is state; how falling and collisions change that state is described by rules; how the broken pieces look is a rendering or video-generation question.

Programmable world models try to separate these roles, using explicit state and programs to track how the world changes.

Earlier work explored related ideas. In WorldCoder, an agent builds a Python world model from interaction and uses it for planning. The approach was studied in grid worlds and planning environments.

The more recent Code World Model calls the coding agent a “World Brain”. It reasons about the consequences of events and generates code to maintain persistent world state. That state is then compiled into constraints, such as proxy video, for a video model. The approach combines code-managed changes to the world with visual generation.

Programmable World Model (September 2026) uses a similar division of responsibilities. An agent turns language into entity state and transition rules. A lightweight engine maintains global state, including off-screen entities and non-visual attributes, while intermediate representations constrain video generation.

This separation makes failures easier to diagnose. If an object’s state is wrong, you can inspect the state updates and rules. If the state is right but the image violates the constraints, you can check the visual generator.

Writing state as variables does not automatically make it a correct description of the real world. The model’s usefulness still depends on whether its rules are adequate, whether it recognizes events correctly, and how it handles missing information. Results on the interactive worlds and tasks studied so far do not establish that a model can predict arbitrary real-world processes.

For robotics, one possible design would be to observe the environment, update a world program, compare the consequences of possible actions in the model, then generate and execute a policy. Feedback from the real environment would inform the next round.

This would connect “writing the world” with “writing actions,” but it still needs end-to-end testing on concrete robot tasks.

06 Why image and video editing belong here

Now replace the robot with a video editor.

A user asks for “a 30-second clip: show the product, then how to use it, then a summary”. The system must pick footage, order shots, cut segments, place captions, and check whether the cut matches the brief. These editing steps can also be expressed as a program, executed, and revised.

VisProg generates visual programs from natural language by composing image understanding, editing, and logic modules. It shows intermediate results so users can inspect the process. Language-guided image editing is one of the tasks it demonstrates.

EditDuet assigns video editing to two agents, an Editor and a Critic. The Editor uses the supplied assets and instructions to call editing tools; the Critic checks the resulting sequence and provides feedback.

Here “programs” may be tool-call sequences, edit graphs, or timeline operations—not always traditional source code. What they share is that editing steps can be inspected item by item, reordered, and reused.

This is the connection Code-as-X draws across domains. Controlling a robot and editing a video pose very different practical challenges, but both can organize a task as explicit steps and revise those steps using feedback from the result. The organization pattern is similar; reliability standards still differ by domain.

07 Why these threads are worth reading together

Program generation, tool calling, simulation, and modular composition each have their own research history. Reading these papers together helps us see how these components work as a system.

The papers above point to three related changes.

First, program generation is increasingly tied to execution. Multi-turn interaction in CaP-X and trial-to-trial rewriting in Act-Observe-Rewrite both focus on how models use execution feedback. The first program is a starting point; the model’s ability to improve it through later revisions matters too.

Second, feedback is richer. Images after robot actions, simulation results from scene programs, and intermediate edit sequences can all be used to check results. The question is no longer just how to prompt a model to write a program, but how to turn changes in the environment into feedback it can use.

Third, execution interfaces affect how model ability is evaluated. The available actions, tool descriptions, and failure reports all change what a model can accomplish. Program-generating models and their execution environments need to be designed together, because the interface affects the model’s decisions.

We use Code-as-X to describe this approach to system design: let the model make decisions that can be written down and revised, use specialized components for computation, control, and generation, and connect them through feedback.

This is what we mean by “from Coding Agents to Physical Agents”: models that write software can also work with other programmable environments. How far they can go depends on both the model and the available interfaces, feedback, and execution stack.

08 The hard problems sit between programs and the world

Several practical problems remain, especially where programs interact with the physical world.

Timing matters. Generating or revising a control program happens on a different timescale from running it. A generated controller can run at high frequency, while observing the environment again, reasoning, and rewriting usually take longer. Systems must coordinate both timescales and decide which situations the current controller can handle and which require a new decision.

Perception and object correspondence remain difficult. grasp(red_cup) is easy to write, but identifying the target in an image, locating it, and checking whether it slipped after contact all depend on observation and estimation. A correct variable name does not guarantee that it refers to the right object.

Trial and error comes at a cost. In a simulation that can be reset, failure can provide data for the next attempt. In the real world, one bad move can change the scene or damage objects. Rerunning the same program may mean starting from different conditions. Program checks, action constraints, and stop conditions must be part of the execution stack.

An interface may not transfer directly to another robot. Robots differ in grippers, reach, and sensors. The same action call can mean different things and may not be feasible on every robot. Shared names are only a start; state, units, coordinate frames, and failure semantics also need clear definitions.

There is no single way to divide the work between explicit programs and learned models. Some rules are worth writing explicitly; some skills are hard to capture with a few rules; many settings need both.

When reading papers in this area, check the evaluation conditions:

Is evaluation in simulation or on a real robot? How much can the model see? High-level skills or low-level actions? How many retries? How is success defined?

Those details show which part of the problem a result actually solves.

09 Where to start with awesome-code-as-x

This work is spread across robotics, graphics, world models, and media editing. Keyword searches can find work within each field. Comparing how different fields handle representation, execution, and feedback is harder.

awesome-code-as-x brings this work together by the role code plays, with links to papers and project pages. Readers can start with their own research question, then compare approaches across domains. The repo has English and Chinese READMEs and welcomes PRs.

If you mainly work on robots, start with Code as Policies, CaP-X, and Act-Observe-Rewrite, and compare their action interfaces and feedback. Then look at how RLE-Bench evaluates agents with different interface depths and hidden test conditions.

If you care about simulation assets, compare SceneCode and NeoWorld-Pro to see the difference between generating scenes and reconstructing them from observations.

If you study world models, read WorldCoder, Code World Model, and Programmable World Model together, and compare what code maintains with what the model learns.

The move from coding agents to physical agents raises questions across these fields. awesome-code-as-x is a place to start reading, comparing approaches, and contributing related work.

Project: kkakkkka/awesome-code-as-x

References

1. awesome-code-as-x
2. Code as Policies
3. CaP-X
4. Act-Observe-Rewrite
5. CaP-X project
6. Show-Harness
7. RLE-Bench
8. OpenVLA
9. SceneCode
10. NeoWorld-Pro
11. WorldCoder
12. Code World Model
13. Programmable World Model
14. VisProg
15. EditDuet