NeuralOS: Towards Simulating Operating Systems via Neural Generative Models
Authors: Luke Rivard
Sun Sun
Hongyu Guo
Wenhu Chen
Yuntian Deng
Venue: ICLR 2026 Poster (2026)
Keywords: Generative Models
Neural Simulation
Diffusion Models
Graphical User Interfaces
Primary area: generative models
TL;DR: We introduce a neural generative model that simulates operating system interfaces by directly generating screen images from user inputs.

Abstract: We introduce NeuralOS, a neural framework that simulates graphical user interfaces (GUIs) of operating systems by directly predicting screen frames in response to user inputs such as mouse movements, clicks, and keyboard events. NeuralOS combines a recurrent neural network (RNN), which tracks computer state, with a diffusion-based neural renderer that generates screen images. The model is trained on a dataset of Ubuntu XFCE recordings, which include both randomly generated interactions and realistic interactions produced by AI agents. Experiments show that NeuralOS successfully renders realistic GUI sequences, accurately captures mouse interactions, and reliably predicts state transitions like application launches. Beyond reproducing existing systems, NeuralOS shows that synthesized training data can teach the model to simulate applications that were never installed, as illustrated by a Doom application, and suggests a path toward learning user interfaces purely from synthetic demonstrations.

Published as a conference paper at ICLR 2026
NEURALOS: TOWARDS SIMULATING OPERATING SYSTEMS VIA NEURAL GENERATIVE MODELS
Luke Rivard' Sun Sun? Hongyu Guo Wenhu Chen'
Yuntian Deng'
University of Waterloo "National Research Council Canada {jlrivard, wenhu, chen, yuntian] yuntianJeuwaterlon @uwaterloo. ca ssun.su,, hongyu. guo eureeenrccccc.c gc.ca
ABSTRACT
We introduce NeuralOS, neural framework that simulates graphical user inter- faces (GUIs) of operating systems by directly predicting screen frames in response user inputs such mouse movements, clicks, and keyboard events. NeuralOS combines a recurrent neural network (RNN), which tracks computer state, with a diffusion-based neural renderer that generates screen images. The model is trained on a dataset of Ubuntu XFCE recordings, which include both randomly gener- ated interactions and realistic interactions produced by Al agents. Experiments show that NeuralOS successfully renders realistic GUI sequences, accurately cap- tures mouse interactions, and reliably predicts state transitions like application launches. Beyond reproducing existing systems, NeuralOS shows that synthe- sized training data can teach the model to simulate applications that were never installed, as illustrated by Doom application, and suggests path toward learning user interfaces purely from synthetic demonstrations.
INTRODUCTION
"Chatting" " with LLM feels like using an 80s computer terminal. The GUI hasn't been invented yet, but some properties of it can start to be predicted.
-Andrej Karpathy
Recent breakthroughs in generative models have transformed human-computer interaction, making it increasingly adaptive, personalized, and intuitive. Historically, computing interfaces were rigid and predefined, such as command-line terminals and static graphical menus (Engelbart) 1968). The emergence of large language models (LLMs) and multimodal AI systems expanded this paradigm by enabling interactions through natural language (Radford et al. 2019; Brown et al,, 2020). im- [0022] Radford et al. [202]] Song al. (2020b), and videos (Ho) eet al. 2022b) Sinngrr et al. 2022b OpenAl] 2024). Recently, generative models have even begun simulating dynamic visual environments (Ha & Schmidhuber] [018a| He et al. (202a)), notably in- teractive video games (Feng et al.) [2024] Oh et al.][2113 Valevski et al. (024)) These advancements suggest future where computing interfaces could become fully generative, dynamically adapting in real-time based on user inputs, contexts, and intentions (Deka et al., 2017).
ages (Ho et al. |0000 Lipman al.
In this work, we study whether an operating system-like interface can be modeled using neural generative model, rather than manually programmed system. In doing so, we investigate the mod- eling and optimization challenges, such as precise cursor control and state tracking. Such a neural interface provides a basis for human-computer interaction research on user-controllable UI genera- tion (e.g., prompt-to-Ul personalization (Tolomei al., 2023). and provides a safe environment in which computer-use agents can be trained and evaluated without issuing real system commands.
To this end, we introduce NeuralOS, a first step toward realizing this vision. NeuralOS simulates an operating system's graphical interface entirely using deep neural networks. By modeling the OS interface as a generative process, it directly predicts graphical frames from user input events, such mouse movements, clicks, and keyboard interactions, without manually programmed kernels or applications. FFigure I] illustrates an example sequence generated by NeuralOS, demonstrating realistic cursor movements and window interactions predicted solely from user inputs.

---

Published as a conference paper at ICLR 2026
DOOM
DOOM
Tuan
Fintoc
D00M
DEVICES
PUACE
Must
Fabk
(a)
(c)
TOEX
(d)
(e)
(1)
Figure 1: Real image sequence predicted by NeuralOS, illustrating the model's ability simulate realistic GUT interactions. The sequence shows key frames as a user (a-c) opens and closes the "Home" folder, followed by (d-f) launches and closes Doom application that trained into the model using synthetic demonstrations. Cursor positions are highlighted with red circles. Frames are generated autoregressively, conditioned on previous frames and user inputs.
NeuralOS integrates two complementary architectures, analogous the separation between OS ker- nels and desktop rendering programs: recurrent neural network (Hochreiter & Schmidhuber 1997) maintains internal computer states (such as open applications, hidden windows, and recent actions), while diffusion--assl convolutional neural renderer generates screen images. We train Neura- IOS end-to-end on interaction sequences recorded from Ubuntu XFCE environments, combining randomly generated and realistic Al- Al-generated human-like like interactions.
Developing NcuralOS posed several challenges. (1) Long-term state tracking was essential due to delayed interface responses (e.g., opening Firefox could take up 30 frames); addressed this by using an RNN-based state representation.
(2) Precise cursor modeling required explicit positional encodings within our diffusion model. (3) Without pretrained encoders for GUI interactions, developed novel pretraining method in which the RNN outputs were pretrained via regression losses and subsequently integrated into the diffusion model via finetuning. (4) Exposure bias during inference was mitigated using scheduled sampling techniques (Bengio et al. 2015 Deng et al. 2023 Ranzato et al. 20155) (5) Extensive engineering was necessary for scalable data collection and real- time inference, leveraging parallel Docker environments and Al-generated user interactions.
Experiments show that NeuralOS can generate realistic screen sequences, accurately model mouse interactions, and reliably simulate transitions such as application launches. While computational constraints limit its ability to precisely model fine-grained keyboard inputs, NeuralOS represents a step toward neural operating systems that adapt interfaces in real time, potentially enabling users to interact through natural language or gestures rather than fixed menus.
Beyond mimicking an existing operating system, NeuralOS in principle learn user interfaces from demonstrations even if they are artificially constructed and do not exist in reality. As a proof of concept, created Doom application by combining fabricated desktop interactions with Viz- Doom gameplay recordings. NeuralOS learned to launch, play, and close Doom despite the ap- plication never being installed in the underlying system. This illustrates the broader principle that synthetic demonstrations, once distilled into a generative model, become usable user interfaces.
Our code, pretrained models, and interactive demo at httpt: ///eerr////eaa/aaa -08 com
2

---

Published as a conference paper at ICLR 2026
Time Step
Time Step
Time Step (-
Renderer Consext
Frume X
Frrmm dgg
Trume dz
Cansor Position Map
Upper LSTM
Renderer
Renderer
Renderer
Renderer Context
Renderer Context
Renderer Context
Ft-1
Attention
Lower LSTM
KNN
RNN
RNN
Upst
Input Embeddings
(a) High-level temporal architccture of NeuralOS.
RNN structure step
NeuralOS Model Architecture. (a) High-level architecture of NeuralOS. At each timestep, an RNN tracks the operating system's internal state based on user inputs (cursor positions, mouse clicks, keyboard events) and previously generated frames. This state is then passed as context diffusion--base renderer (UNet) that generates the next graphical frame. (b) Detailed two-level RNN structure at timestep t. The lower-level LSTM encodes user inputs, and then integrates visual information from the previous frame using attention. Its output is passed to the upper-level LSTM, which further processes these iitention-informer representations. Feedback from the upper-level LSTM to the lower-level LSTM (U2-1) ensures that the lower-level LSTM maintains awareness of upper-level state context and previous attention results. The combined outputs of both LSTMs, and cursor position encoding, form the renderer context. This hierarchical structure maintains constant computational complexity per timestep and supports continuous state updates during inference, sential for real-time OS interface simulation.
Figure 2:
GENERATIVE MODELING OF OPERATING SYSTEM INTERFACES
We formalize the task simulating operating system (OS) graphical interfaces as an autoregressive generative modeling problem. At each discrete timestep t, the model predicts the next graphical frame 23 based on the sequence of previously observed frames I<t = Tu, T1 It-1 and user input events age d1, 02 ... at up and including the current timestep.
Formally, each frame I, is represented as image tensor Fa E # with H and W denoting image height and width, and C' the number of color or feature channels. The input event a, at timestep t includes cursor coordinates (æ, y), binary indicators for mouse clicks (left right), and a binary vector indicating which keyboard keys are pressed or released.
The probability distribution of an OS graphical sequence given user inputs can be expressed
P(rer 01::: 0)
II P(re Tiet, agge 0),
where represents the parameters the neural generative model.
Unlike video generation (Singer al.) 2022b Ho al. 2022b OpenAl 2024), OS simulation must respond instantly to unpredictable user inputs, often causing abrupt changes in the interface, such as when a new application is launched. This contrasts with the smooth, predictable transitions typical in video generation. As a result, the model must maintain accurate and responsive state tracking. Next, describe the NeuralOS architecture and training strategies designed for these requirements.
3

---

Published as a conference paper at ICLR 2026
3 NEURALOS ARCHITECTURE
NeuralOS adopts a modular architecture inspired by the functional separation traditional operating systems between kernel-level state management and graphical user interface (GUI) rendering. comprises two primary components: a recurrent neural network (RNN) responsible for maintaining internal system states, and diffusion-bar based renderer that generates graphical frames based on these states (see Figure 2a).
Latent Diffusion Representation NeuralOS uses a latent diffusion framework (Rombach et al, 2022). We train an autoencoder to compress high-resolution OS screen images into lower- dimensional latent representations, reducing spatial dimensions by a scaling factor s. All modeling performed within this latent space. At inference time, the generated latent frames decoded back into pixel-level images only for users.
Hierarchical RNN for State Tracking NeuralOS employs a hierarchical two-level RNN archi- tecture to track the system state (see Figure 2b). Unlike transformers (Vaswani et al., 2017). whose inference complexity increases with context length, the RNN maintains constant complexity per timestep, which is crucial for continuous, long-horizon OS simulation. Existing video-game mod- els (Feng et al. 2024 Oh et al., 2015 Valevski et al. 2024) typically rely on short context windows, which are sufficient because most game states visually observable from recent frames. In con- trast, the RNN state enables NeuralOS to recall previous interactions far in the past (Table 2).
At each timestep t, user inputs a, are encoded into embeddings. Specifically, cursor coordinates are discretized screen positions (af a²), mouse clicks are binary indicators, and keyboard keys are binary press/release states. Each component is embedded separately and then concatenated:
embed(a,) concat(embed((a), embed (a.), embed(a click cllcck
(ambed click)
embed(a; ket)).
key
These embeddings are processed by lower-level LSTM, which also takes its previous hidden state 4-1 and feedback from the previous upper-level LSTM state U,- as inputs:
L 4 LSTM concat (enbeeatteemed((())) U2-1)),
where 4, denotes the hidden state and I., denotes the corresponding output at timestep l.
To handle inherent uncertainties in OS behaviors, such as unpredictable application response times, the lower-level LSTM output L, is used as query vector to attend over the previous graphical frame using multi-headed attention (Vaswani et al. 2017):
C₁ MullsieeadedAAteeetoonqqgrrr
keys/valees W4
Epos):
where W g Wi are learnable projections and Epus encodes positional information of the latent frame.
The attention output e is then merged with the original lower-level LSTM output L;:
C L + Woc:
then processed by the upper-level LSTM:
U,, = LSTM L1S
To ensure that the lower-level LSTM maintains awareness of higher-level contexts, the upper-level LSTM's output U, is fed back as an input to the lower-level LSTM in the next timestep.
Spatial Encoding of Cursor Positions Precise cursor localization is critical for realistic OS in-
teractions. Interactive world and videogame models (Parker-Holdee Holder & Fruchter 2025 Xiang et al.)
2025 Alonso et al. 2024 Valevski et al. 2024) typically use few actions to navigate the world. In contrast, the cursor represents a much larger discrete action space that grows in screen size. This made cursor rendering extremely challenging, where errors in cursor location were often hundreds of pixels as shown in in/Fig 4b) To counter this, NeuralOS explicitly encodes cursor positions using Gaussian spatial map M, E HWW Instead of using a one-hot cursor position (which can lose
4

---

Published as a conference paper at ICLR 2026
Stage RNN Pretraining
Stage Joint Training
MSE
Diffusion
d
Se
Renderer
Stage 3: Scheduled Sampling
Stage 4: Context Length Extension
Diffusion Loss
RNN
RNN
RNN
RNN
RNN
RNN
Nr-21443
Extended Context
Figure 3: Multi-stage training pipeline for NeuralOS, (I) RNN Pretraining: The RNN is pre- trained to predict latent frames using mean squared enror (MSE) loss. (2) Joint Training: The pretrained RNN and the diffusion-based renderer are jointly optimized using a standard diffusion loss. (3) Scheduled Sampling: To mitigate error accumulation caused by exposure bias, the most recent input frame occasionally replaced by a previously generated frame. (4) Context Length Extension:
The input context extended enable the model capture long-term dependencies.
precision due to latent resolution constraints), we construct a Gaussian map centered at the cursor's s scaled coordinates:
Mc(i.)) exp( (i a, /s) + ( a/))
2
As demonstrated in Figure 4b this spatial encoding is vital for accurate cursor rendering. This map
Mr, combined with LSTM outputs L,, U,, forms the renderer context R₂ E "Rxx"C',
R concat(Wa, W,,
Diffusion-Basel Renderer To render the screen image, a UNet-based diffusion renderer gener- ates the latent graphical frames conditioned on the renderer context R, (Ronneberger et al.) 2015).
It Pe Rt).
We concatenate the noisy image with R₂ as input to the UNet, and then predict the clean image.
MULTI-STAGE TRAINING APPROACH
Training NeuralOS is challenging due ineffective of RNN outputs, error accumulation during inference, and difficulties in capturing long-term dependencies due to computational constraints. To address these challenges, take multi-stage training approach (Figure 3).
5

---

Published as a conference paper at ICLR 2026
Stage 1: RNN Prefraining Unlike text-to-image diffusion models (Rombach et al. 2022 which use pretrained textual encoders. NeuralOS uses à customized RNN without pretrained checkpoints. Our initial experiments show that direct joint training leads to the renderer ignoring RNN outputs, as indicated by negligible gradient flow into the RNN. The diffusion- based renderer receives two streams of inputs: noisy latent frames and the RNN output; and without proper initialization of the RNN, it relies solely on the noisy image inputs, resulting in an under-trained RNN.
To address this, we first pretrain the RNN. We structure the RNN output RG E # to match the spatial dimensions the latent frame 2; E R XxC but with more channels (C" C). The RNN is pretrained to predict the latent frames using mean squared error (MSE) loss:
LMSE | C| 22.12²
After pretraining, the RNN- -eneereeed frames alone tend to be blurry due to averaging multiple plausible outcomes, but crucially provide a strong initialization for subsequent joint training.
Stage 2:
Joint Training We jointly optimize the pretrained RNN and the diffusion based renderer with a standard diffusion loss. The meaningful latent representations leamne in RNN pretraining enable the renderer to use the RNN outputs, thus preventing the RNN outputs from being ignored.
Stage 3: Scheduled Sampling During inference, errors accumulate over time and progressively degrade the quality of the generated frames. This issue arises from exposure bias (Ranzato et al,, 2015): model trained exclusively on ground-truth frames becomes overly reliant on perfect inputs and struggles when forced to operate on its own imperfect predictions during inference.
To mitigate this, we introduce a scheduled sampling training stage (Ranzato et al.) 2015 Deng et al. 2023): during training, we replace the most recent input frame 2t-1 with the model model-genenee generated frame -;- with a small probability p. This method makes the model robust against input noise, thus mitigating error accumulation and improving generation stability over extended interactions.
Stage 4: Context Length Extension Although NeuralOS can model sequences of arbitrary length, hardware memory limits justify shorter sequences during training, which limits exposure to long-term dependencies. To address this, we introduce a final stage that extends training to longer contexts, following initial training short context windows for efficiency.
Curriculum Training on Challenging Transitions In our collected OS interaction dataset, a substantial portion of frame transitions involve minor variations, such slight cursor movements, which exhibit limited learning signals. To prioritize learning of significant OS state changes (c.g., opening menus or launching applications), we first train NeuralOS exclusively on challenging transi- tions. Specifically, we define challenging transitions as those frames whose pixel differences exceed a specified threshold: ||x, - - ll e. Subsequently, we expand training to the full dataset. We apply this curriculum strategy for Stage 1 (RNN Pretraining) and Stage 2 (Joint Training).
Additional Finetuning with Real-User Data After deploying NeuralOS, we collect a set real- user interaction demonstrations and find that finetuning the trained model on these real-world ex- amples improves its performance on frequent user tasks. This adaptive finetuning is conducted continuously through an interactive training framework introduced by Zhang et al. (2025). enabling the model to dynamically incorporate real-time collected data and achieve improved alignment with actual user behavior. Full methodological details and results can be found in that work.
DATA COLLECTION
Agent-Based Demonstrations To collect realistic user interactions, we use Anthropic's Claude- 3.5-Sonnet computer-use agent (Anthropic) 2024), which processes screenshots and invokes pro- vided interaction functions. To maximize interaction diversity, we structure the agent's 's exploration process around a state-space search tree representing various OS states (see Figure 13 in Section J).
Specifically, we prompt the agent to identify all interactable GUI elements by moving the cursor to each element's center and reporting its bounding box (center, width, height). Each identified GUI element becomes node in search tree rooted at the initial OS state. The agent is then guided

---

Published as a conference paper at ICLR 2026
Predecec CIWEET
&x
150
22 Qroer Trush Curse
100
3.220
Scheduled Samping
Scheoduled Sempling
0.15
50
0.10
0.05
50
120
125
WV cursor map
cursor map
Random
FFrams
(a)
(b)
(C)
Figure 4: (a) Heatmap illustrating predicted vs. ground truth state transitions. Each cell represents the percentage of predictions assigned to a particular predicted cluster (x-axis), given a ground cluster (y-axis). Only the top 16 clusters heatmap. (b) Comparison of cursor position errors for NeuralOS (with cursor position map), Neu- raIOS without the cursor position map, and random baseline. (c) Pixel RMSE of generated frames of using versus not using stage 3 scheduled sampling training.
displayed here; refer Figure 11 for the complete
through this tree: for each node, it moves the cursor to the corresponding GUI element and performs single or double clicks transition to new OS state, which then becomes a child node in the tree. We iteratively expand the tree until reaching a predefined maximum depth.
Next, we initiate further interactions from each leaf node, allowing the agent to explore freely from these distinct OS states for fixed duration, thereby capturing diverse interaction sequences.
Random Exploration We find that relying exclusively on duces spurious correlations. For example, the model incorrectly associates cursor movement toward window's 8 close button with the action of closing, even in the absence of click. To mitigate such unintended associations, we supplement the dataset with random interaction data.
demonstrations intro-
In generating these random interactions, we simulate mouse movements, clicks, and keyboard inputs (key presses and releases) stochastically. To improve realism, we introduce several constraints and heuristics iteratively developed through experimentation. Cursor movements are modeled using Bezier curves to emulate natural mouse trajectories (Mortenson) 1999). Double-click events, rare under purely random sampling, explicitly generated. Additionally, enforce constraints such as limiting simultaneous key presses and ensuring keys are released only if previously pressed.
EXPERIMENTAL SETUP
Data We collected data using 64 parallel Docker containers, each with Ubuntu 20.04 and XFCE
desktops at a resolution of 512 x 384. To simplify the environment, the desktop was limited to four applications: Home, Trash, Terminal, and Firefox. Data consisted of 2K agent-based and 120K random exploration demonstrations, each 30 seconds long at 15 fps, resulting in 12TB of latent data after compression via an autoencoder. The autoencoder reduced the images by factor of latent resolution of 64 x 48 with 16 channels. Details of the autoencoder are in Sectient
Model The hierarchical RNN has two LSTM modules (each with hidden size 4,096) and a multi- headed attention module heads, 1,024 total dimension). The RNN output is projected to 32 channels and concatenated with the noisy latent frame (16 channels). The UNet uses four resolution levels with channel multipliers of [1, 2, 3, 5]. two residual blocks per level, and attention layers at resolutions 8, 4, and 2. It has base model dimension of 192 and outputs 16 channels. The final model contains 2.2B parameters for the RNN and 263M parameters for the UNet renderer.
Training and Inference NeuralOS was trained using our proposed multi-staee training approach. See (Section for full details hyperparameters. The total data processing and training took

---

Published as a conference paper at ICLR 2026
Doom interactions generated by NeuralOS. Doom was never installed in the underlying operating system used for data collection; instead, the model learned simulate the application from synthesized training data, producing realistic walking and shooting behavior from user inputs.
Figure 5:
Table 1:
Human success rate (%) of identifying the real OS.
Crop Setting
10s
20s
30s
40s
50s
60s
No crop
58.3
55.0
60.0
56.7
61.7
59.3
Bottom cropped (40px)
51.7
53.3
50.0
48.3
46.7
50.0
approximately 4 months, requiring 17,000 GPU hours on a server with 8 NVIDIA H200 GPUs (141GB memory per GPU) and an additional 6,000 GPU hours on a server with 8 NVIDIA H100 GPUs (80GB memory per GPU). At inference time, we used DDIM sampling (Song al. 2020a)
with 2 steps, achieving an inference speed of 18 fps on single NVIDIA H100 GPU.
7 EXPERIMENTS
Given the substantial computational resources required to train NeuralOS, our evaluation focused on NeuralOS variants, ablations, and intermediate training phases. For all evaluations, we used subset of 730 examples from a reserved evaluation dataset, unless mentioned otherwise
Cursor Position Modeling We evaluated cursor-position accuracy by training a regression model
to detect cursor coordinates from the generated images. With the cursor position map. NeuralOS achieved highly accurate cursor localization, with average position error of Ar 1.6 and Ay 1.4 pixels (Figure 4b). Given the 512 x 384 resolution of the images, this corresponds to less than 0.5% of the frame width or height, indicating that cursor locations in generated images are precise. This performance significantly outperformed a baseline without cursor position maps (Av = 130.0, Ay 95.8) and the random baseline (Ar 175.4, Ay 126.9), confirming the importance of explicit spatial encoding for accurate cursor localization.
Modeling Most OS interactions involve only cursor movements without signif- icant visual changes. To focus evaluation on the crucial moments where the interface undergoes meaningful changes, such as opening closing an application, transitions from the evaluation set These transitions were clustered into 73 categories using DB- Scan. NeuralOS predictions were then matched against the nearest cluster labels and compared with the ground truth. As shown in Figure 4al NeuralOS achieved an accuracy of 37.7% (correct state transitions over total state transitions) captured on the diagonal. This substantially outperformed majority voting (1.4%). Note that off- predictions may still correspond to valid outcomes due to inherent timing variability in OS actions (Secion D).
identified challenging frame
Human Evaluation We conducted a human evaluation to test how indistinguishable NeuralOS outputs are from real operating system. Human raters were shown side-by-side video clips of the same user interaction sequence, one generated by NeuralOS and one recorded from an Ubuntu
'This number was chosen because our clustering procedure (detailed later) identified 73 clusters of chal- lenging frame transitions, and selected 10 examples per cluster, resulting total of 730 examples. "This baseline is an earlier model version trained for 700K steps under slightly different conditions. 'These were those with mean pixel distance greater than 0.1 from the previous frame the target frame.
8

---

Published as a conference paper at ICLR 2026
Vyen
(b)
Figure 6: Ablation studies. (a) shows the limitations of directly using RNN outputs. (b) shows error accumulation without scheduled sampling, demonstrating its necessity for maintaining stability,
XFCE desktop, and asked identify the real system. Each clip lasted 10-60 seconds (sampled at 10s intervals) and was drawn from real human interactions collected through our online demo.
To control for artifacts such as fluctuating disk-space counters (a known issue in diffusion), evalua- tion was performed under two conditions: (I) original videos and (2) videos with bottom 40 pixels (10% height) cropped. In each condition, participants judged 30 randomly ordered comparisons.
As shown in Table 1 participants performed only slightly better than chance on up to 20 seconds of interactions, suggesting that NeuralOS is visually realistic for basic interactions.
Learning from Synthetic Demonstrations A benefit of end-to-end learned user interface is its ability to learn from demonstrations that do not correspond to real implementations. To illus- trate this, constructed Doom application entirely from synthetic data, even though Doom was never installed in the docker. Desktop screenshots were augmented with a Doom icon, synthetic double-click actions were inserted simulate launching the game, VizDoom gameplay segments with corresponding input events were spliced in, and each sequence ended with an ESC action re- turning the desktop. Trained on these demonstrations, NeuralOS learned launch, play, and close Doom as if it was a native application (Figures and 5
Long-Term Memory We evaluated whether NeuralOS has long-term memory using folder cre- ation task. We navigated to Home > Documents, created "New Folder" (via right-click + Create Folder), closed the file manager, and later reopened Documents after a delay. During training, the model saw only short delays: we included 500 examples where the file manager was reopened after frames in the training set. test time, we extended the delay to 64, 128, and 256 frames. As control, we also included 500 examples where no folder was created.
Folder presence was evaluated by checking the region where the folder icon should appear, compar- ing it to the ground truth with an MSE threshold. [able 2 shows the model can mostly recall whether a folder had been created even after 256 frames, which far beyond both the 8-frame training delay and its maximum training context of 64 frames. This indicates that NeuralOS develops persistent memory representations of application state that generalize well beyond its training horizon.
Ablations Without joint training (relying solely on the pretrained RNN), the predictions are blurry Figure 6a). This is caused by the MSE loss encouraging the model to predict averaged represen- tations of multiple plausible outcomes rather than committing single clear target. Additionally, cursor positions were absent, despite the model correctly capturing state transitions (c.g.. opening the home folder), indicating that the RNN still implicitly encoded cursor information.

---

Published as a conference paper at ICLR 2026
Folder presence modeling confusion matrix after reopening file manager with increasing gaps. The higher the diagonal values (bolded), the better.
Table 2:
Predicted
Frame Gap
Ground Truth
Has Folder
No Folder
Folder Created
0.52
0.48
64
No Folder Created
0.03
0.97
Folder Created
0.60
0.40
128 No Folder Created
0.00
1.00
Folder Created
0.62
0.38
256
No Folder Created
0.02
0.98
Omitting scheduled sampling led to rapid deterioration in generated frame quality due to compound- ing prediction errors (Figure 6b). In contrast, incorporating scheduled sampling greatly improved the model's robustness (Figure which is also evident from quantitative evaluation in Figure 4c
8 CONCLUSION AND FUTURE WORK
We introduced NeuralOS, a system that simulates OS graphical interfaces using generative mod- els. Trained end-to-end on interaction sequences, NeuralOS produces realistic screen sequences, accurately predicts mouse interactions, and captures transitions such opening applications.
benefit of end-to-end trained interfaces is their ability to learn from synthetic demonstrations. Our Doom experiment illustrated this: the model simulated launching, playing, and closing Doom, even though the application was never installed in the underlying operating system. This points to a broader paradigm for future generative interfaces: if consistent demonstrations can be provided, even they are manually edited, stylized, or fabricated, they can be internalized into usable interfaces.
REPRODUCIBILITY STATEMENT
Our training code and data processing scripts are available at https::/github.. com/ yustiaa-Group/ neural-os An interactive demo is hosted at |ttps::/nnerrallss com/] with demo code at pottps://qithui
The model checkpoints used by the demo are publicly available on Hugging Face (linked directly from the demo repository). Additional details on training procedures and dataset construction are provided in the Appendix.
ETHICS STATEMENT
Generative models capable of imitating software inevitably raise broader considerations around in- tellectual property, licensing, and the economics of software distribution, similar to the concerns that emerged when large (vision) language models began reproducing copyrighted text, API patterns, or proprietary knowledge (He al.] 20256] Wei et al.] 2024 Cooper al., (2025). To avoid these is- sues, training in this work was conducted on Ubuntu XFCE environments. More broadly, this project should be understood an early research prototype investigating the feasibility of generative user interfaces rather than a system intended to replicate or redistribute commercial applications.
ACKNOWLEDGMENTS
This research was supported by Compute Canada through the Resources for Research Groups 2025 competition, awarded to Yuntian Deng (RRG No. 5275). and was also partially supported by collab- orative research funding from the National Research Council of Canada's 's Artificial Intelligence for Design Program 114D-150). Additionally, Yuntian Deng acknowledges support from an NSERC Discovery Grant (RGPIN-2024-05) and Starter Grant provided by the University of Waterloo.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos J Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37:58757-58791, 2024.
Anthropic.
Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku, October 2024. wwwpss--ttt---c---aaaaaaa
URL attps://wwis
Accessed: 2025-05-14.
Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khali- dov, Patrick Labatut, Francisco Massa, Mare Szafraniec, Kapil Krishnakumar, Yong Li, Xiaodong Ma, Sarath Chandar, Franziska Meier, Yann LeCun, Michael Rabbat, and Nicolas Ballas. V-jepa 2: -upe--uue video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.0990 09985, 2025. URL /tttp:::/aaxi..oog///////ooo
09985
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative inter- active environments. In Forty-first International Conference on Machine Learning, 2024.
Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. Gamegen-x: Interactive open- world game video generation. arXiv preprint wwXiv:2411.00766 2024.
A. Feder Cooper, Christopher A. Choquette-Choo, Miranda Bogen, Kevin Klyman, Matthew Jagiel- ski, Katja Filippova, Ken Liu, Alexandra Chouldechova, Jamie Hayes, Yangsibo Huang, Eleni Triantafillou, Peter Kairouz, Nicole Elyse Mitchell, Niloofar Mireshghallah, Abigail Jacobs, James Grimmelmann. Vitaly Shmatikov, Christopher De Sa, Ilia Shumailov, Andreas Terzis, Solon Barocas, Jennifer Wortman Vaughan, Danah Boyd, Yejin Choi, Sanmi Koyejo, Fernando Delgado, Percy Liang, Daniel E. Ho, Pamela Samuelson, Miles Brundage, David Bau, Seth Neel, Hanna Wallach, Amy B. Cyphert, Mark A. Lemley, Nicolas Papernot, and Katherine Lee, Ma- chine unlearning doesn't do what you think: Lessons for generative policy and research, 2025. URL https:tp//// /tapxii//aaxxv..om org. abs 22122 06966
Yufeng Cui, Honghao Chen, Haoge Deng. Xu Huang, Xinghang Li, Jirong Liu, Yang Liu, Zhuoyan Luo, Jinsheng Wang, Wenxuan Wang, Yueze Wang, Chengyuan Wang, Fan Zhang, Yingli Zhao, Ting Pan, Xianduo Li, Zecheng Hao, Wenxuan Ma, Zhuo Chen, Yulong Ao, Tiejun Huang, Zhongyuan Wang, and Xinlong Wang. 6mu3.5: Native multimodal models are world learners, 2025. URL https::////. http://///wxxxx.oo/aaa//////////
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: mobile app dataset for building data-driven design ap- plications. In Proceedings of the 30th annual ACM symposium on user interface software and technology, pp. 845-854, 2017.
Yuntian Deng, Noriyuki Kojima, and Alexander M Rush. Markup-to-image diffusion models with scheduled sampling. In The Eleventh International Conference on Learning Representations, 2023. /
/ttp::::///pe/eieee/// net/ fftp:t/efooromma.toaooooooooooo
Douglas C. Engelbart. The mother of all demos. https://wwit http:://wwwwyouuueee. com wawchwhhhhh ?v= EhEh3tELivg] 1968. Demonstration at the Fall Joint Computer Conference, San Francisco, CA.
11

---

Published as a conference paper at ICLR 2026
Ruili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang. Yu Liu, and Hongyang Zhang. The matrix: Infinite-horizon world generation with real- time moving control. arXiv preprint arXiv:2412.03568, 2024.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31, 2018a.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint 1803:/0003.122 2018b.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.cm 22300.004000 104, 2023. htt.ptttss.//axxi.oor
abs /2/2/311.22 04104
Haoran He, Yang Zhang, Liang Lin, Zhongwen Xu, and Ling Pan. Pre-trained video generative models as world simulators. arXiv preprini arXiv:2502.07885 2025a.
Luxi He, Yangsibo Huang, Weijia Shi, Tinghao Xie, Haotian Liu, Yue Wang, Luke Zettlemoyer,
Chiyuan Zhang, Danqi Chen, and Peter Henderson. Fantastic copyrighted beasts and how (not) to generate them, 2025b. URL htLhttt:: /tttpp::/aaxxi...o www.a.iiiv.oo.... htwp:///bb/244....o 14526
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang. Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022a. URL hictps://arivv.oot abs/
22100. 02305
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, David Fleet, and Mohammad Norouzi. Imagen video: High definition video generation with diffusion models. arXiv preprint 12t.22200.02303, 2022b.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, and Sanja Fidler. Learning to simulate dynamic environments with gamegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1231-1240 2020.
Lambda Cloud. Neuralos neural-native operating system, 2025. URR.pttss://neuralll
lambda ai/
Accessed: 2025- 1-18.
Yann LeCun. Generalist agents and the role of world models, 2022. Unpublished Lecture / Position Paper.
Xin Li, Wenqing Chu, Ye Wu, Weihang Yuan, Fanglong Liu, Qi Zhang, Fu Li, Haocheng Feng, Errui Ding. and Jingdong Wang. Videogen: A reference-guided latent diffusion approach for high definition text-to-video generation, 2023. URL httl/tttt//aaxxitia
org/ abs/ tabs/2300 2209. 00398
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Lc. Flow matching for generative modeling. arXiv preprint arX:v:2210.00277, 2022.
Yu-Ren Liu, Biwei Huang, Zhengmao Zhu, Honglong Tian, Mingming Gong, Yang Yu, and Kun Zhang. Learning world models with identifiable factorization. arXiv preprint arXiv:2306.06566 2023. (RLLittps://art/
org/ abs/
www.bb/...... 22006/3660000 06561
Cristian Meo, Mircea Lica. Zarif Ikram, Akihiro Nakano, Vedant Shah. Aniket Didolkar, Dianbo Liu, Anirudh Goyal, Justin Dauwels, et al. Masked generative priors improve world mod-
sequence modelling capabilities. arXiv preprint arXiv:2410.07836,, 2024. URL https:
/ //arxiv.org wtt.b///411007800
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers sample-efficient
world models. In ICLR 2023, 2023. URL attp:://Eleurer.orr//aaaaa
michel satelli-e-all--- maiiiri22..pppmm
12

---

Published as a conference paper at ICLR 2026
Vincent Micheli, Eloi Alonso, and François Fleuret. Efficient world models with context-aware tokenization. In Proceedings of the 41st International Conference on Machine Learning (ICML). pp. 35623-35633 PMLR, 2024. URL httpss//pprceeeinggg.s... v235/ml chel htmml
Michael E Mortenson. Mathematics for computer graphics applications. Industrial Press Inc., 1999.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L. Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. Advances in neural information processing systems, 28, 2015.
OpenAL. Introducing Sora: OpenAl's Text-to-Vidde Model. htt.p:://openal..com/nbbt sora, February 2024. Accessed: 2025-04-22.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02133 2022. UtL/htpss:/ http:::://aaxvv... http//xxxv....r//../ 2203 02155
Jack Parker-Holder and Shlomi Fruchter. Genie 3: A new frontier for world models. https: 7/ /edepminddd amoogle/blogg wwwlb//gggggeeee// genie-e-a--nnee new fooe
2025. Accessed November 24, 2025.
Virtuals Protocol. Video game generation: A practical study using mario, 2024.
URL http::////gttuuu.coooo/ Mell://sattcc/odff//iieeeeeeeeee
pdf] Preprint.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PmLR, 2021.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level train- ing with recurrent neural networks. arXiv preprint arXiv: 151.16632. 2015.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and computer-assistedd intervention- MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings. pan III 18, pp. 234-241. 241. Springer, 2015.
Julian Schrittwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588 doi:
(7839):604 609, 2020.
00000444115600000005550 1038/s4155 -- 020 03051-4
URL https: /oi....../// orq/10.
D. Shin, Ali Eslami, and Madhavi Sewak. Simulating neural operating system with gem- ini 2.5 flash-lite, June 2025. URL Ittps:://deeloopers www.oopeerr..gggoooogoooooo com/en/ aalaaiiinaa--nuaaaa
-ng- Google Developers Blog. Accessed: 2025-11-18.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-vide videeo generation without text-video data, 2022a. URL https: / //axivvv... org/abs) 2209. 14792
13

---

Ma, Published as a conference paper at ICLR 2026
Uriel Singer, Shelly Sheynin, Adam Polyak, Eliya Nachmani, Oron Ashual, Yonatan Belinkov, Gal Metzer, Gabriel Alalu, Devi Parikh, Yaniv Taigman, et al. Make-a-video: Text-to-video genera- tion without tixt-video data. arXiv preprint 209942200. 2022b.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13356 2020b.
Gabriele Tolomei, Cesare Campagnano, Fabrizio Silvestri, and Giovanni Trappolini. Prompt-to-os (p2os): Revolutionizing operating systems and human computer interaction with integrated ai generative models. arXiv preprint arXiv:2310.04875. 2023.
Sergey Tulyakov, Ming-Yo Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation, 2017. httphtts::/ //apxi////xvv....// https////xvvvv..//aaaeee htrg/abs/1700) 04993
Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408. /4837, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Boyi Wei, Weijia Shi, Yangsibo Huang, Noah Smith, Chiyuan Zhang, Luke Zettlemoye, Kai Li, and Peter Henderson. Evaluating copyright takedown methods for language models, 2024. URL https://arre http:////aaavv.../////aoa/////// wtgg/ass///eooooooooooooo//oooer 2406. 18664
World Labs. Marble: A multimodal world model. https: //t:::/wwwwww.. http://wwwww...bb..oooo.o worldlabs ai ai/blog/ marble-world-moode 2025. Accessed November 24, 2025.
Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer: World models for physical robot learning. In Conference on Robot Learning (CoRL) 2023, pp. 2226-2240. PMLR, 2023. URL. https::/ htttp:://proceedinggs m wwtpsppwwwwww......2..
htm
Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, et al. Pandora: Towards general world model with natural language actions and video states. arXiv preprint arXiv:2406.0944 2024.
Jiannan Xiang. Yi Gu, Zihan Liu, Zeyu Feng, Qiyue Gao, Yiyan Hu, Benhao Huang, Guangyi Liu, Yichi Yang. Kun Zhou, Davit Abrahamyan, Arif Ahmad, Ganesh Bannur, Junrong Chen, Kimi Chen, Mingkai Deng, Ruobing Han, Xinqi Huang, Haoqiang Kang, Zheqi Li, Enze Hector Ren, Yashowardhan Shinde, Rohan Shingre, Ramsundar Tanikella, Kaiming Tao, Dequan Yang, Xinle Yu, Cong Zeng, Binglin Zhou, Hector Liu, Zhiting Hu, and Eric Xing. Pan: A world model for general, interactable, and long-horizon world simulation. arXiv preprint, MrXiv:2511.09055 2025.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers, 2021.
10157
Mengjiao Yang. Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbecl. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114, 1(2)66 2023.
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. In Advances in Neural Information Processing Systems (NeurIPS) 34, 2021.
URL https: / /aaxxxxxc vvx.vvioo org/ abs/211 2111.111 00210
Weipu Zhang, Gang Wang, Jian Sun, Yetian Yuan, and Gao Huang. Storm: Efficient stochastic transformer based world models for reinforcement learning. arXiv preprint arXiv:2310.09615,
2023. URL http::tts///// ptttps://arxiiie
http://aaaiiii..oraaaaaa/////1 wttp://s////..ooooo 09615
14

---

Published as a conference paper at ICLR 2026
Wentao Zhang, Yang Young Lu, and Yuntian Deng. Interactive training: Fredback-driven neural network optimization, 2025. URL
Zangwei Zheng, Xiangyu Peng. Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024. URL httpt:t:///mm /ttpps://arivv org/ abs /212 24402
Rui Zhou et al. Minedreamer: Learning to follow instructions via chain-of-immginatiin for simulated- world control, 2024.
12037
15

---

Published as a conference paper at ICLR 2026
A BASELINE COMPARISON WITH DIAMOND
We adapted the DIAMOND model AAlonso al. 2024 which was developed for action- conditioned video game simulation (e.g., Atari and CS:GO), to the systemyssmmmmmmm setting. DI- AMOND serves representative action- coniionnondiiooeed diffusion model capable of predicting future frames given past frames and actions.
Adaptation DIAMOND was designed for video game environments with short-horizon temporal dependencies and coarse action spaces. To ensure fair comparison, modified it in the following ways:
Context window size: Increased from 4 + 64 past frames/actions to accommodate longer temporal dependencies desktop interactions (e.g., launching Firefox may take 20-30 frames).
Cursor coordinate discretization: Expanded from the original 23 17 grid to match the 512x384 desktop resolution. This supports pixel-accurate pointing (e.g., clicking the close icon).
Keyboard action space: Expanded from the original keys to the full keyboard (179 keys).
Training data: Trained on the same dataset used for NeuralOS to enable comparison.
Apart from these modifications, we followed the original architecture and training setup as closely as possible. Both DIAMOND and NeuralOS were trained from scratch under matched compute constraints (single B200 GPU for around 48 hours). NeuralOS in the main experiments was later trained longer (around 3M steps), but here we report results under equalized compute.
Figure illustrates generations from the three models evaluated under the same interaction sequence used in the main paper (opening the "Home" folder, closing it, launching Doom, and returning to the desktop). The adapted DIAMOND model produces decent frames before the first abrupt transition occurs (opening Home), but generation quality collapses afterwards.
Qualitative Results
To improve the stability and reduce accumulated error over time, we also fine-tuned DIAMOND with scheduled sampling (p = 0.05) for an additional 50k steps. This variant could recover from degenerations, but still failed execute application- transitions.
In contrast, the NeuralOS model produced more stable generations and was able to successfully open and close Doom, although at this training stage had not yet learned open the Home folder. (From the main experiments, we know that with longer training the model eventually learns this behavior.) Interestingly, the cursor not rendered in these generations (we observed that learning to draw the cursor is particularly challenging because it occupies only tiny fraction of the pixels and requires many gradient steps before it begins to appear). However, the model still responds to mouse hovers and double-clicks, indicating that the cursor position is being implicitly tracked even though its visual rendering has not yet emerged.
Quantitative Results We additionally computed pixel RMSE between generated and ground-truth frames as function of the number of generated steps. As shown in Figure 7 eтror for both DIA- MOND variants increases rapidly, particularly after state-changing events such as opening an appli- cation. In contrast, NeuralOS maintains substantially lower error over longer horizons.
This comparison not meant to imply that DIAMOND unsuitable; rather, highlights differ- ences in modeling assumptions. Action-conditioned diffusion models developed for video games generally assume that the necessary state is visually encoded in recent frames, making short context windows sufficient. In operating-system settings, however, critical state may persist far beyond short temporal window (c.g., whether folder was created earlier in the workflow). With a fixed 64-frame context, DIAMOND cannot represent such long-term state by design. NeuralOS, through its recurrent state component, generalizes beyond its training horizon, consistent with the results in Table 2 showing correct folder-state recall after delays of up to 256 frames.
16

---

Published as a conference paper at ICLR 2026
DIAMOND
JEON
(d)
DIAMOND + Scheduled Sampling (p- 0.05)
(d)
NeuralOS
JOJA
(a)
(b)
(d)
Figure 7: Generations from DIAMOND, DIAMOND with scheduled sampling, and NeuralOS evaluated on the same interaction trajectory used in Figure Each pair of rows shows key frames as the model (a-c) opens and closes "Home", followed by (d-f) launching and closing Doom.
17

---

Published as a conference paper at ICLR 2026
0.5
0.4
DIAMOND DIAMOND + Scheduled Sampling
RMSE
0.3
NeuralOS
0.2
0.1
0
20
40
60
80
100
120
# Frames
Figure 8: Pixel RMSE of generated frames of NeuralOS, DIAMOND. and DIAMOND with sched- uled sampling. Both DIAMOND variants show increasing eπror over frames, particularly after state transitions such as application launch events. NeuralOS maintains lower error across longer hori- zons, reflecting better state tracking enabled by its recurrent architecture.
Table 3: Detailed hyperparameters and dataset configurations used for each stage of NeuralOS's multi-stage training. "Challenging transitions" are where the target frame differs from the preceding input frame by mean pixel difference greater than 0.1. These challenging transitions constitute approximately 2.8% of the full dataset.
Stage
Dataset
Batch
Steps
LR
Context
Sampling p
Stage 1: RNN Pretraining
Challenging transitions
2.8% subset 100%
256
50K
8 X 10-5
32
Full dataset
256
200K
8 x 10
32
Stage 2: Joint Training (RNN Renderer)
Challenging transitions
2.8% subset 100%
64
100K
8 x 10
32
Full dataset
IM
10-5
32
Stage 3: Scheduled Sampling
Full dataset
100%
256
500K
8 10
32
0.05
Full dataset (Ir reduced)
100%
256
500K
2 x 10
32
0.05
Stage 4: Context Length Extension
Full dataset
100%
128
100K
2 x 10-5
64
0.05
B LIMITATIONS
The current study uses Ubuntu XFCE as the operating environment. This choice reflects practical rather than technical constraints: Ubuntu provides legally redistributable platform aligned with our goal of releasing all resources openly, and training comparable models across multiple operating systems would substantially increase data collection, storage, and compute requirements. Extending NeuralOS to more operating systems such as Windows or macOS remains an important direction for future work.
18

---

Published as a conference paper at ICLR 2026
2
2 -
bin
Savava as
prtser
multiply prtacr
final
Figure 9: Correct prediction examples by NeuralOS. Each row shows two past frames (columns 1- 2), ground-truth next frame (column 3), and NeuralOS's prediction (column 4). Cursor positions are marked one frame advance with circles (red: move-only, blue: left-click, yellow: right- click). NeuralOS correctly captures various GUI transitions, including opening menus and launching applications.
Our work represents an initial step toward fully generative OS, but several limitations remain. De- spite substantial training compute (17,000 H200 GPU hours and 6,000 H100 GPU hours), NeuralOS still far from replicating the capabilities of a real operating system: screen resolution remains low, fine-grained keyboard interactions are not reliably supported (Section D). For inference, the current system requires an H100 GPU and not yet suitable for cost-efficient or practical deployment. This reflects the exploratory nature of the work. We expect future architectures, training strategies, and hardware to make such systems substantially more efficient. Additionally, many open challenges remain, including enabling the generative OS to interact with external resources (e.g., internet), and introduce controllability beyond traditional OS boundaries.
C MULTI-STAGE TRAINING HYPERPARAMETERS
NeuralOS trained four stages. conceptual overview of role and outcome each stage shown in Table 4. and the corresponding hyperparameter settings and dataset configurations
detailed in Table
In Stage (RNN Pretraining), the RNN was trained first the subset of challenging transitions. defined as examples whose target frame differs from the last input frame by an average pixel differ- ence greater than 0.1. These challenging transitions constitute about 2.8% of the entire dataset. We used a batch size of 256 and an initial learning rate of x 10 1 training for 50K steps. Afterwards, the model was trained the full dataset (100% of data) for additional 200K steps, maintaining
19

---

Published as a conference paper at ICLR 2026
2
2 1-1
i
divice
numb
@
morr Torth
Figure 10: Prediction examples where the generated frame does match the ground truth frame. The layout follows Figure 9 Note that not all mismatches represent errors. For example, the third row illustrates a case where the screenshot tool window closed the ground truth frame but remains open in the prediction. This discrepancy arises because the window-closing action (not shown due to the limited context window) can have variable timing. Thus, both the predicted and ground truth frames are valid outcomes in such scenarios.
the batch size and learning rate. The context window length fixed at 32 frames during this stage.
In Stage 2 (Joint Training). the pretrained RNN and the renderer were jointly trained end-to-end, first focusing on the challenging transitions (2.8% subset) for 100K steps, then extended to the full dataset for an additional 1M steps. The learning rate remained size of 64 to stabilize diffusion training. The context length remained 32 frames, and no scheduled sampling was applied this stage.
10-5, with reduced batch
In Stage (Scheduled Sampling), we trained the full dataset using scheduled sampling with probability = 0.05, where the most recent past frame was occasionally replaced by model- generated frame during training. Initially, training of 256 and a learning rate
conducted for 500K steps batch size 10-5, The learning rate was subsequently reduced to 2 x 10-5 followed by additional 500K training steps. The context window remained 32 frames.
Finally, Stage 4 (Context Length Extension), we increased the context window length from 32 to 64 frames enable NeuralOS to better capture long-term dependencies. Scheduled sampling probability was maintained at p = 0.05. We used a lower learning rate x 10-5, reduced the batch size to 128 to fit GPU memory constraints, and finetuned the model for 100K additional steps.
20

---

Published as a conference paper at ICLR 2026
Table 4: Summary of the four-stage training pipeline. Each stage addresses specific modeling challenge.
Stage
Purpose
Outcome
Remaining limitations
Stage RNN Pretraining
Train the RNN to predict latent frame embeddings that it encodes cursor motion and keyboard inputs, ensuring the RNN is not ignored during later joint training.
Model learns basic transitions: decoding the RNN output via autoencoder yiclds plausible but blurry frames.
Generations are blurry due to the MSB objective.
Stage Joint Training
Combine the pretrained RNN with the diffusion-based renderer so that the decoder can sharpen predictions while retaining temporal structure from the RNN.
Produces stable short-horizon interactions.
Exposure bias: during training model only sees ground truth frames, but at inference it must condition on own imperfect outputs, leading to compounding errors over time.
Stage Scheduled Sampling
Introduce scheduled sampling so model is exposed to its own generated frames during training.
Improves stability over extended interaction sequences.
Context window length limited for training
efficiency, and the model has not yet observed very long interaction horizons.
Stage Context Length Extension
Expand the temporal context so model can learn longer interaction dependencies.
Enables the model observe learn longer-term dependencies.
Future work needed on efficiency, controllability, and external tool integration.
D QUALITATIVE ANALYSIS
We further analyze NeuralOS qualitatively by examining successful and unsuccessful generation examples, shown in Figure 9 and Figure 10, respectively. Each row illustrates two past frames, the ground-truth next frame, and NeuralOS's predicted frame. Cursor positions past frames are annotated with colored circles to indicate the cursor's ' intended position at the next frame: red repre- sents cursor movement only, blue denotes left-click actions, and yellow signifies right-click actions. Additionally, keys pressed at each frame are displayed in red text. Note that cursor annotations are shifted forward by one frame to clearly depict cursor positions expected in the immediate subsequent frame.
In Figure 9 NeuralOS accurately predicts various critical GUI transitions, such as launching appli- cations and opening menus through both mouse clicks and keyboard inputs, demonstrating its ability to capture spatial and functional dynamics.
NeuralOS exhibits limitations, particularly for subtle actions like moving the cursor to a "Close Tab" button without clicking. Moreover, NeuralOS currently struggles to accurately represent fine-graing keyboard inputs, such as specific characters typed in terminal.
However, shown in Figure 10
worth noting that not all mismatches between predictions and ground truth constitute errors; some discrepancies arise from variable timing in GUT responses, exemplified in Figure 10
21

---

Published as a conference paper at ICLR 2026
Predicted Cluster
100
80
Ground Truth Cluster
60
Percentage
40
20
-0
Figure 11: Complete heatmap of NeuralOS state transitions. Each cell represents the percentage of predictions corresponding predicted cluster (X-axis) given ground-truth cluster (y-axis). Diagonal entries indicate exact cluster matches.
22

---

Published as a conference paper at ICLR 2026
E FULL STATE TRANSITION HEATMAP
Due to space constraints, the main text presents only a truncated version of the state transition heatmap. In Figure 11, provide the complete heatmap, showing NeuralOS's predictions across all identified clusters.
F INTERACTIVE WEB DEMO
To make NeuralOS accessible, we built an interactive web demo available at https:// heural-os, com The system consists of a browser-based frontend and a GPU-backed backend.
Frontend The frontend renders the NeuralOS screen inside single HTML. canvas. Mouse move- ments, clicks, and keyboard inputs are streamed to the backend over a persistent WebSocket con- nection.
Because user input often arrives faster than the model can produce frames (about 55 ms per frame), the system performs lightweight input coalescing: redundant mouse-move events are dropped when new meaningful event (e.g., click or keyboard press) occurs. This makes interaction feel smoother.
Some OS actions require multiple generated frames to complete (for example, launching Firefox may take about 20 frames). To handle these cases, the demo includes an "Auto Input" mechanism: when the user stops interacting, the frontend periodically resends the last cursor position (every 67 ms) that the model continues progressing through delayed transitions rather than waiting for input.
The backend uses dispatcher-worker architecture. The dispatcher manages queueing, session lifetime, WebSocket routing, and worker availability. Each worker process runs one instance of NeuralOS and occupies a single GPU. Only one user can actively interact per worker.
Backend
When users are waiting, the system applies time-limited sessions to ensure fair GPU access, The demo also includes an idle timeout, which triggers a "Connection Timeout" warning before discon- necting users who stop interacting.
To reduce bandwidth and improve responsiveness, generated images are split into 32 32 pixel blocks, and only blocks that change meaningfully from the previous frame are transmitted to the browser.
Debug Controls The demo includes internal debugging options such as toggling "Use RNN" (by- passing diffusion), changing sampling steps, resetting state, and modifying Auto Input behavior. These features primarily used during development and are now hidden by default; advanced users can still access them via the browser console using showDebugControls ().
Notably, "Use RNN" mode decodes the RNN state directly, which is only valid after the RNN pretraining stage. After joint training with the diffusion model, this mode produces degraded frames because it is longer the intended decoding path. The default configuration (full RNN + diffusion model) corresponds to the model used the paper.
G TRANSFORMER VS RNN ENCODER
We initially implemented our system using a transformer-bawe encoder architecture. However, in interactive OS simulation, inference sequences can be arbitrarily long, as each user action generates new frame that is appended to the model's 's context unless explicitly truncated. For long or continuous interactions, this results in a steady increase in transformer memory consumption during inference, ultimately leading to out-of-memory (OOM) failures for sufficiently long sequences.
To address this, we replaced the transformer with hierarchical recurrent neural network (RNN) architecture. RNNs maintain a constant-size hidden state between steps, enabling inference- -ime GPU memory usage that is independent of sequence length. Table 5hooo the memory requirements of a RNN encoder vs a Transformer encoder. Moreover, computer state modeling is inherently Markovian, making the RNN suitable choice.
23

---

Published as a conference paper at ICLR 2026
Table Inference-time GPU memory usage (GB) for transformer and RNN architectures of similar parameter counts. The transformer encoder cannot accommodate 1024 frames on an L40 GPU.
Video Length (s)
Transformer (GB)
RNN (GB)
4
21.61
11.96
64
22.69
11.96
128
23.81
11.96
256
26.07
11.96
512
30.60
11.96
1024
40 (OOM)
11.96
Table Application open-close accuracy. We report RMSE and mean absolute error (L.1) between
NeuraIOS predictions and ground-truth frames, with pixel values normalized to [0, 1].
Application
Action
RMSE
L1
Terminal
Open
0.021
0.001
Firefox
Open
0.048
0.006
Trash
Open
0.017
0.001
Home
Open
0.019
0.001
Terminal
Close
0.015
0.001
Firefox
Close
0.016
0.001
Trash
Close
0.016
0.001
Home
Close
0.016
0.001
H APPLICATION OPEN-CLOSE TRANSITIONS
We evaluated the accuracy of application open-close transitions in NeuralOS. Specifically, we con- structed 150 sequences for four main applications (Terminal, Firefox, Trash, Home) by sampling random cursor paths and click locations uniformly within the icon bounding boxes.
We compared the predicted and ground-truth frames at the end of each open and close event using root mean squared error (RMSE) and mean absolute error (L1). To account for rendering latency, evaluated the frame after a fixed delay following each click: 40 frames after Firefox double-clicks, 10 frames after other application double-clicks, and frames after single-click closing. These delays were determined empirically to align with the frame at which the application's 's visual state change completes.
As shown in Table 6) the low RMSE and LI scores confirm that NeuralOS accurately performs application open and close transitions, consistent with observations from the interactive demo.
I AUTOENCODER DETAILS
We trained Variational Autoencoder to compress high-dimensional OS screenshots into low- dimensional latent representations suitable for efficient training of NeuralOS.
Model Architecture The architecture of the autoencoder is based the model proposed by (Rombach et al.) 2022) with some custom adjustments to improve reconstruction quality. The en- coder consisted of four convolutional downsampling blocks with 128 base channels. Each down- sampling stage contained two residual blocks and no attention layers. The latent channel dimension was set 16.
Training The autoencoder was trained using a combined reconstruction and adversarial loss func- tion. We trained the autoencoder using the Adam optimizer with learning rate of 1e-6, batch size 10, and a total of 2 million training steps on our dataset. Training was conducted on a single NVIDIA H200 GPU.
24

---

Published as a conference paper at ICLR 2026
Moring Siaale Rrste
Marking LPVAN PAR
Figure 12: Examples original images (top row) and their corresponding reconstructions (bottom row) from the trained autocncoder. Despite significant spatial compression (8x downsampling), the autoencoder preserves details.
X
Figure 13: Illustration of Search-Tree-Bassee Data Collection. We construct search tree repre- senting OS states, starting from the initial desktop screen (root node). Each node corresponds distinct OS state, created by clicking or double-clicking interactable GUI elements identified by a computer-use agent. For clarity, only first-level transitions (opening applications) and one deeper exploration within Firefox are shown. This approach enables collecting diverse interaction data ef- ficiently.
After training, the encoder was able to compress each 512 x 384 RGB frame into a latent represen- tation of dimension 16 x 64 48 (downsampled by factor of 8 in spatial dimensions), significantly reducing memory requirements for subsequent NeuralOS model training.
Examples of original and reconstructed images provided in Figure 12
J COMPUTER-US USEE AGENT FOR DATA COLLECTION
To build the search tree illustrated in Figure 13
we used structured prompts to guide the computer-
agent. Starting from the initial desktop screen (root node), sequentially instructed the agent to first map and then verify all interactable GUI elements (Figures [4) For subsequent GUI states (non-root nodes), initially prompted the agent transition to each state (Figure 16)
25

---

Published as a conference paper at ICLR 2026
need to generate some training data now, Please 'terat ively map out every appl icat icon/button desktop screen by moving your mouse it. mouse EXACTLY to its center point. DO nor click it though. Be tharough and identify every button icon on the sereen from the cop to the bottom.
icen, your
[(suffix))
Figure 14: Initial GUI Element Mapping (Root Node). Prompt instructing the agent to identify interactable GUI elements on the initial desktop screen.
Now do a final check: Look carefully at every part of the screen for any unmapped buttons. you find any, map exact centers like before. you are absclutely cartain buttoong have been mapped, reapond with ONLY the rinal coordinate nate 1st.
[(uufii)]]
Figure 15: Final Verification of GUT Elements (Root Node). Prompt instructing the agent to perform final check for any missed interactable GUI elements on the initial desktop screen.
naed generate some training data now, First, please click/open object at [1x) I, (y))) Do not progresa untill it la open or elicked. Double al tok to open if neceasary. Onoe you conflrmed it is open or clicked, stop hece.
Figure 16: Transitioning New UI State (Non-Root Node). Prompt instructing the agent to transition to the current state.
Now, iterative related to the ob ject you click/open as (1x) (yy3) especially new buttons. Foz each butten, your EXACILY to its center point. Do NOT click it though.
out all the buttons moving your cursor to them, Focus those
[(uUtii)]]
Figure 17: Initial Mapping of Newly Revealed GUI Elements (Non-Root Node). Prompt instruct- ing the agent to identify GUI elements newly revealed after transitioning to the current UI state.
Please continue mapping out buttons focus mapping those related to the object you click/open at ((x), [fy)] especislly any new ones. As you exhaust the new buttons, move farther out. For each buston, move your RXACTIY its center ((suffix))
screen but
onymore buttons. Again,
NOT click it though.
Figure 18: Continued Mapping of Remaining GUI Elements (Non-Root Node). Prompt instruct- ing the agent to further map any remaining interactable GUI elements in the current UI state.
After transitioning, we issued follow-up prompts to map and verify any newly revealed GUT ele- ments (Figures 17 to 19..
Each prompt included standardized suffix, as shown in Figure 20 to ensure that the agent outputs the coordinates and dimensions of mapped GUI elements in consistent, structured format. This allowed efficient parsing and automated processing collected data.
K CURSOR POSITION PREDICTION MODEL TRAINING
To quantitatively evaluate cursor position accuracy in the generated frames Figure 4b), trained regression model to predict cursor coordinates directly from screen images. The training procedure detailed below.
26

---

Published as a conference paper at ICLR 2026
Now do a rinal check: Took carefully around the screen to aee if there are any new unmapped butlons related exact centers like before. Do NoT click mere buttons though. If you are absclutely certain ALL related buttons have been mapped, respond with ONLY the final coordinate 11st.
objest you click/open at [(x],, :ly]h, If you find any, mag their
[(suffix))
Figure 19: Final Verification of GUI Elements (Non-Root Node). Prompt instructing the agent to perform final check ensuring all interactable GUI elements have been identified at the current UI state.
you find keyboard Input box, Lype Lool specifying its cencer coorcinates input box type via text frem the following possibilities: terminsl, browsez, oF other, with coordinates appended
xty.
the of your response, provide structured List buttons you mapped using this exact format:
For each button provide twe peirs of numbers: Lccation (4): x'y coordinates
you moved the mouse 2. size (8): xty dimensions of the button
Example format:
L000"200 950:30, // Button 1: Located at (00),200) with size 50x30 L303-400 860:40 //
(303, 400) with size 60x40
Important:
EXACTLY formal: Lx"y Sxty
Seperate each button with commas you mapped Numbers only, no text oz descriptiens
Include
Figure 20:
Struetured Output Suffix. A standardized suffix appended to all prompts, guiding the agent report mapped GUI elements using consistent coordinate and dimension format.
Model Architecture We used a ResNet-50 convolutional backbone pretrained on ImageNet, with modifications for fine-grained spatial localization tasks. Specificallyy, we adjusted the stride and dilation parameters in the final convolutional layers reduce downsampling from 32x to 16x, preserving more spatial resolution. The feature extractor output is passed through an additional in- termediate convolutional layer followed by fully-connected regression head, outputting continuous cursor coordinates (xr, y).
Training We used the Adam optimizer with an initial learning rate of 6 x 10-5 and weight decay set to 1 10-5 We clipped gradients maximum norm of 1.0. We optimized predicted and ground-truth cursor positions. We trained with batch size of 16 for total of epochs. Input images were used directly from collected data at the original resolution (512 x 384 pixels), normalized and rearranged to match the input format of ResNet-50. The training data consisted of randomly sampled frames from the full dataset, each labeled with the ground truth cursor positions captured during data collection. Training was performed on single NVIDIA A6000 GPU. The test error 0.5 pixels for both x and y. Given that each image 512 x 384 pixels, this reflects extremely high localization precision. In other words, the regression model can predict the cursor location from a screen image with an average deviation less than a single pixel, making highly sensitive to small differences and suitable for evaluating fine-grained cursor accuracy in generated frames.
L1 loss between
L USE OF LLMS
We used large language models (LLMs) in two ways. First, used LLM-based computer-use agent (Claude-3.. 3,5dee-355-ool Sonneet to generate interaction traces, which were incorporated into our train- ing dataset alongside synthetic data. Second, used LLM--hseed writing assistant to polish
27

---

Published as a conference paper at ICLR 2026
grammar. All ideas, analyses, experiments, and scientific claims own, and take full responsibility for the content of this work.
M SCHEDULED SAMPLING IMPLEMENTATION DETAILS
Scheduled sampling requires generating model-based frames during training, which incurs higher computational costs compared to using only ground-truth frames. In a multi-GPU training setting, naively performing scheduled sampling at random intervals could result in synchronization bottle- necks, as some GPUs would have to wait for others to complete computationally expensive sam- pling steps. To mitigate this issue, we implemented scheduled sampling at regular intervals across all GPUs simultaneously. Specifically, for a scheduled sampling probability o = 0.05, each GPU (and all examples within each GPU's batch) performs scheduled sampling exactly once every 20 steps. This synchronization approach ensures consistent training speed and prevents slowdown due to inter-GPU blocking.
N HUMAN EVALUATION
To evaluate how realistic NeuralOS appears human observers, we conducted perceptual study modeled after GameNGen (Valevski et al.] 2024). Participants were shown pairs of short video clips-one generated by NeuralOS and one recorded from real Ubuntu XFCE desktop- corresponding to the same underlying sequence of user interactions. Their task was to identify which clip came from the real operating system.
We evaluated six interaction durations: 10s, 20s, 30s, 40s, 50s, and 60s. To control for artifacts that could reveal the model's identity (such as flickering free-space counters, a known issue dif- fusion models), the study was conducted under two conditions: (1) full-frame videos and (2) videos cropped to remove the bottom 40 pixels (10% the height).
Each participant completed 30 randomized trials per condition. A sereenshot of the evaluation in- terface is provided in Figure 21, and quantitative results are reported in Table T (main text). In the cropped condition, participants' accuracy fell close chance, suggesting that NeuralOS simulations are visually convincing for short interaction sequences.
O LONG-TERM DEPENDENCY CHALLENGES IN OS SIMULATION
Modeling long-term dependencies is central challenge in NeuralOS, and it arises in two different ways: (1) practical limitations on context window size during training, and (2) the intrinsic difficulty of maintaining long- horizon state in the model.
Practical Constraints on Context Window Size Increasing the temporal context window sub- stantially increases GPU memory usage and decreases training throughput. To keep training feasi- ble, most stages of NeuralOS training use a 32-frame context window, which allows the model to observe more target transitions per unit of compute. Only in the final stage we extended the context to 64 frames for fine-tuning.
This staged approach mirrors common practice in large language model training, where shorter contexts are used for most of training and longer contexts introduced only at the end to maintain efficiency. Although longer contexts are possible in principle, they incur prohibitive cost when used throughout training.
Horizon State Retention Beyond the Context Window Even with extended input con- text, many OS interactions require reasoning over events that happened far earlier (c.g., whether a folder created hundreds of frames ago). Ideally, longer-term information should be encoded and propagated through the model.
illustrates this distinction: NeuralOS recall with decent accuracy whether a folder was created even after 256 frames, well beyond the 64-frame context used during training. This demonstrates generalization but also shows the broader challenge:
Our galder-presence experiment Table 2)
28

---

Published as a conference paper at ICLR 2026
Instructions:
operating sysrem:
you (Al-generotee)
operating system
navigation move examples
Complete ecampiea in selling your eccsrway
Note: Videos are orapped to hide elements that right give away the answer
Video Length Setting: framms (10.0%) Uncropped
Example
Video
Video
Home
Temminal
Temminal
This
This
Previtut
Nexc
Figure 21: Screenshot of the evaluation interface. Participants were shown side-by-side video clips of the same user interaction sequence, one generated by NeuralOS and one recorded from real operating system, and asked to identify the real operating system.
for fully interactive OS simulation, models ideally need to preserve relevant state across arbitrarily long horizons.
RELATED WORK
Existing work has proposed neural operating systems in which generative model mediates between and the operating environment. Prompt-to-OS (Tolomei et al., 2023) introduces visionary paradigm where generative models sit between the user and the kernel, potentially replacing tradi- tional application-specifie interfaces. We see NeuralOS as a concrete step toward that vision on the user-interaction side: our model learns how screens evolve from low-level input events and could in the future be extended with external tools (e.g.. HTTP APIs storage). Recent systems such as Gemini OS (Shin et al., 2025) and Lambda's NeuralOS (Lambda Cloud 2025) take code- generation approach, where user actions are converted into prompts and the model generates HTML or structured UI elements that are rendered by browser existing UI engine. In our testing, this approach works well for structured interfaces composed of predefined UI elements (e.g., buttons, panels, grids), but it does not extend to arbitrary pixel-based applications rich visual content, such as real-time games.
Video generation models such as VideoGPT (Yan al. 2021 and MoCoGAN (Tulyakov et al. 2017) demonstrated early progress in synthesizing temporally coherent videos. More recent diffusion- based systems, including Imagen Video (Ho et al. 2022a), Make-A- A-Video (Singer al., 2022a), Open-Sore (Zheng et al. 2024), and VideoGen (Li et al. 2023), produce high-quality video clips from text prompts reference frames. However, these approaches differ fundamentally from
29

---

Published as a conference paper at ICLR 2026
NeuralOS: they generally operate by generating video segments using temporally compressed latent representations, rather than performing action-conditioned next-frame prediction. This temporal compression makes them unsuitable for interactive simulation, where each frame must directly de- pend on fine-grained cursor and keyboard inputs at step-level resolution.
NeuraIOS is closely related recent generative modeling approaches for simulating interactive environments conditioned on user inputs. GameGAN (Kim et al. 2020) used generative adversarial networks (GANs) for interactive game imitation, and Genie (Bruce et 2024 generated playable 2D platformer worlds. More recently, diffusion-based models have emerged as powerful real-time simulators: GameNGen (Valevski et al. 2024) simulated the game Doom, MarioVGC (Protocol, 2024) simulated Super Mario Bros. DIAMOND (Alonso et al. 2024) simulated Atari and Counter- Strike, GameGen-X (Che et al. 2024) simulated open-world games, and Matrix (Feng et al. 2024) simulated AAA games. Beyond video games, UniSim (Yang et al. 2023) developed simulators for real-world scenarios.
More broadly, NeuralOS relates to the growing body of work on world models. Early work such as World Models (Ha & Schmidhuber 20188) demonstrated predictive simulation for reinforcement learning environments, and subsequent models have scaled this paradigm to large multimodal set- tings, natural-language control, and tool-augmented agents (LeCun 2022: Meo et al. 20244] Micheli Zhang et al. 2023; Liu et al. (2023). Recent systems such as Pandora (Xiang al. Genie-3 (Parker-Holdee & Fruchter 2025), PAN (Xiang et (025). V-JEPA-2 (Assran et al, Emu (Cui al. (225), and Marble (World Labs 2025) explore increasingly open-ended in-
ct al.
2023 2024
2024
2025
leractive simulation, often guided by text. World models have also been used as closed-loop training environments for agents (Wu et al., [2233] Ouyang al.
2022 Ye et al. 2021 Schrittwieser al., Alonso et al., 2024] Zhou et al. [204]] Halner et al. (202)). showing that learned simulations can support planning and skill acquisition.
2020
Compared to these prior works, NeuralOS addresses challenges unique to OS simulation: while most GUI frame transitions involve subtle changes, accurately modeling critical discrete events, such opening applications menus, is essential. Additionally, precise cursor position prediction crucial for interactive usability, NeuralOS introduces targeted model and training innovations specifically addressing these challenges, paving the way toward fully generative OS simulations.
30


Reviews:
[{"review_id": "JIaAUUYnkE", "reviewer": "Reviewer_cFb2", "summary": "This paper proposes NeuralOS, an end-to-end generative model–based operating system capable of simulating and rendering graphical user interfaces (GUIs) through real-time cursor and keyboard interactions. The study adopts a multi-stage training framework to enhance the quality of GUI simulation. Experiments and demo results suggest that NeuralOS can effectively simulate GUIs via GPU rendering without relying on a physically hosted interface. Overall, this work presents an intriguing step toward developing a world model for GUI simulation.", "questions": "Please answer the questions in the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This work provides a clear and detailed description of the model architecture, input feature encoding, and multi-stage training strategy to ensure rendering quality. The use of LLMs to autonomously collect interaction data, thereby removing human involvement, is particularly interesting. Overall, the paper demonstrates a promising approach to constructing a world model for operating systems, capable of generating real-time screens based on user interactions.", "weaknesses": "#1 The paper should better position its work within the context of existing works. For instance, although video generation is briefly mentioned (line 146), no specific papers are cited. Similarly, despite of the discussion on world models in Section L, the main text does not contextualize NeuralOS in world models. The authors clarify how their approach to modeling long-term trajectories and user actions differs from prior methods used in video generation and world modeling.\n\n#2 The discussion of challenges in lines 214–215 is unclear. In Stage 4, the paper notes that capturing long-term dependencies is difficult due to hardware limitations, yet the input context is later extended to address this. If this extension mitigates the issue, the authors should clarify in what sense this remains a challenge.\n\n#3 The paper mentions that transformers suffer from high inference complexity, yet the proposed model incorporates attention heads which is an essential component of transformers. \n\n#4 Cursor coordinates are encoded twice, at line 174 and line 199, respectively. The paper should clarify the rationale behind this encoding design or conduct an ablation study to explain how it contributes to the simulation of cursor movement.\n\n#5 In evaluating the performance of NeuralOS, the performance metrics and their formulas should be clarified and referenced. For example, the formula for accuracy is not explicitly provided in the main text. It is unclear whether the accuracy reported in Table 1 (which measures the identification of the real system) is conceptually the same as the cursor position accuracy and overall image simulation accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0G1pWO8X0q", "reviewer": "Reviewer_oGA3", "summary": "This paper proposes a framework to simulate operating systems’ (OS) graphical user interfaces (GUIs) through single screen frame prediction in response to user’s inputs, using deep neural networks. The proposed approach spans the entire pipeline - from data collection to the final prediction - and employs two neural architectures: a recurrent neural network (RNN) for state-tracking, and a diffusion-based convolutional neural renderer for image generation. The authors formalized the task of next frame prediction as an autoregressive generative modeling problem. \nThe models are trained on synthetic data, utilizing both random and AI-agent human-like generated interactions within Ubuntu XFCE environments. The methodology is clearly stated and well presented. The authors address OS-specific simulation challenges - distinct from those encountered in, for example, game environments. Their proposed solutions include notable design choices, such as the use of an RNN for long-term state tracking, and a Gaussian spatial map for precise cursor modeling. \nAlthough no formal metrics or benchmarks are mentioned, the authors report that, based on  human-evaluation results, the model achieves visually coherent frame prediction. \nThe paper’s main contribution lies in demonstrating that the model can even generalize to previously unseen applications (e.g., clicking an icon for a non-installed app and producing plausible launch behavior). One suggestion regarding this last point, would be to expand on the significance of this finding and its broader implications for the future of generative user interfaces. The work is motivated by the goal of creating learned, interactive simulators of operating systems through synthetic demonstrations, but the research motivation behind this should be stated more clearly.", "questions": "See weaknesses", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The paper is clearly written and well organized; the technical components are formally defined and adequately justified, which makes the work easy to follow.\n* The proposed architectures are coherently structured and integrated. \n* The paper represents an original contribution, as generative simulation of Operating Systems remains novel and unexplored, though it draws some parallels with prior work on generative simulation in other fields (i.e.: gaming).\n* Code is given to reviewers, allowing the reproducibility of the proposed approach.", "weaknesses": "* The motivation remains vague throughout the paper. The key questions left to be answered are: “How can this contribution advance the field?”, “Why do we need to simulate OSs?”. A more explicit problem statement - together with expected applications (i.e.: human-computer interaction research, AI agents) would strengthen the paper’s significance. \n* No closely related previous work is discussed - hence, no competitors are referenced. The authors only cite examples from game or real-time simulations, omitting directly relevant efforts in neural or generative OS simulations. Providing a stronger contextualization of the work, would help in collocating the study - e.g.: by referencing recent work such as “Simulating a Neural Operating System by Google (at https://developers.googleblog.com/en/simulating-a-neural-operating-system-with-gemini-2-5-flash-lite/) or other generative interfaces models (as in  https://arxiv.org/pdf/2310.04875). This would clarify how the proposed approach advances beyond prompt-driven systems. * This too, would aid the authors in the articulation of why a fully autonomous OS would be desirable. \n* Experimental validation is limited. Although ablation studies are included to assess each components’ importance, no quantitative comparison against state-of-the-art baselines is provided. Qualitative evaluations and human-evaluation results are present and informative, but they would benefit from additional quantitative evidence (i.e.: comparison with SOTA, statistical analyses, benchmark comparisons, graphs). \n* Development is limited to synthetic Ubuntu XFCE environments, hence, the generalization ability to other OS interfaces remains to be verified.\n* Regarding the provided demo, the simulation exhibits some technical issues: frame rendering lags; disappearing cursor behavior when the RNN toggle is activated; the “Auto Input” option not working - there is no frame prediction (i.e.: the screen does not change) and after about five seconds of waiting for an autonomous screen update, one gets a “Connection Timeout Warning” and the connection restarts. These issues make it difficult to evaluate the system’s claimed interactivity.\n* An additional concern regards the resources required. The high GPU cost required for the proposed OS simulation - which cannot approach performance comparable to real-time OSs - raises questions about utility and scalability. In simpler words: “Why is it necessary to use all those hours of GPU usage in order to simulate an OS that will never reach performance comparable to a real OS (which is also, way cheaper)?”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8TxUxm1kqr", "reviewer": "Reviewer_9rkZ", "summary": "This paper introduces NeuralOS, a generative framework designed to simulate operating system graphical user interfaces by autoregressively predicting screen frames based on user inputs like mouse movements, clicks, and keyboard events. The architecture combines a hierarchical recurrent neural network to manage and track the internal system state with a diffusion-based neural renderer that generates the final screen image conditioned on the RNN's state. The authors ask an LLM-based computer-use agent (Claude-3.5-Sonnet) to generate interaction traces, and alongside random synthetic traces through a Docker Ubuntu instance to generate the underlying training data with actions.", "questions": "1. Were the human evaluations done on the same distribution of interaction traces used to collect the data?\n\n1. Given that a standard Docker container provides a perfect, zero-cost, high-fidelity environment for training an OS agent, what is the practical justification for doing 23,000-GPU-hours to train this model? What tangible use case does this simulation serve today the real environment doesn't solve more effectively and cheaply?\n\n1. The paper admits that \"fine-grained keyboard interactions are not reliably supported\". How can this be presented as a successful OS simulation if it fails at one of the most basic and critical components of an operating system (the terminal)?\n\n1. How exactly does the \"fabricated\" Doom demo, which was manually constructed from \"spliced in\" gameplay, demonstrate a generalizable capability to simulate new or useful applications? Isn't this just a one-off, manually-guided demo rather than an emergent property of the model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. Cleanly split state tracking (hierarchical RNN with attention over the previous frame) from image synthesis, which makes the problem well-posed for long, interactive sequences.\n\n1. This paper is very instructive in providing a recipe for training video based world-models in general. There are several important tricks here that are more broadly applicable than training a neural OS world model. For instance, the model architecture for long memory, multi-stage training, combination of tricks in Section 4, sampling _transitions_ more often, adding random explorations to mitigate spurious correlations etc.\n\n1. Fast 2-step DDIM giving ~18 fps on a single H100, which is useful for interactive demos, not just offline generation.\n\n1. Reproducibility & openness with code, checkpoints, data scripts, and an interactive demo is linked.", "weaknesses": "1. You can't really use the final artifact for much. If you want to train a model to use an operating system, you'd rather just use the Docker container. I'm finding it a bit hard to justify the contribution of this paper beyond that it is a very interesting demo and some of the tricks used to collect the data and make the world-model work.\n\n1. The demonstration paths through the Docker OS to collect training data were generated by a computer-use agent (Claude-3.5-Sonnet). This is obviously very expensive to collect a very large amount of interaction traces.\n\n1. The quality of the model isn't that particularly great. Notably, in the demo, typing into the terminal barely works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8oGVcOnxIN", "reviewer": "Reviewer_YGuk", "summary": "This paper suggests that it may be possible to simulate computer programs without having their code through a demonstration of using RNNs + Attention + Diffusion Models with next-pixel-prediction to generate realistic responses to computer input (i.e., keyboard or mouse input).  They show through several experiments that this is highly effective in some toy settings.   They also included evidence that this was not nearly as effective without several novelties added to the process, such as their unique way of representing (x,y) coordinates and their specialized training schedule.", "questions": "On line 209 - could you please explain the logic for concatenating the noisy image?\n\nDo you have intuition why your unique spatial encoding did better than just including the x,y coordinates?  I would have guessed that the network could have learned to pay particular attention to these details and to tiny differences in those coordinates without requiring a new map formalism (assuming they were floats passed into the module).\n\nShould we think of the scheduled sampling as a kind of teacher forcing, or could you elaborate on theoretical motivations for using it?\n\nOn line 265, where it states “After pretraining, the RNN-generated frames alone tend to be blurry due to averaging multiple plausible outcomes, but crucially provide a strong initialization for subsequent joint training” - could you be a bit more specific about the strengths and weaknesses of every stage of training, perhaps in the form of a small table?\n\nLine 270 - I'm surprised occasionally replacing the input image with the model-generated frame works at all if they are actually different and the output crucially depends on what the image is; it seems that if you give it a different input, it should produce a different output as \\textit{correct} behavior.  Could you please clarify?\n\nYou said that you were using \"random exploration\" to mine traces, but your search tree looks nothing like random exploration - did you just mean random in the sense of non-deterministic, or am I misunderstanding the role of the search tree in gathering data?\n\nLine 313 - is the use of Bezier curves to model cursor usage novel?  If not, the literature should be cited.  If so, a simple ablation with using a simpler family of curves would be great (although probably more of a camera-ready than necessary-by-rebuttal) item.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper is clearly in line with ICLR's goal, to advance theoretical and empirical knowledge of learning.  It has huge potential implications for both theory and practice, suggesting a) that in a very concrete and systematizable way, neural attention mechanisms can replace symbolic logic, and b) suggesting alternatives to current practice in software development.  While I imagine that these results are limited by the simplicity of the environments that were being simulated, that in itself could become an area of theoretical interest in itself - how do we create complexity classes of software applications which can predict whether or not they can be imitated using such a model?", "weaknesses": "This paper potentially presents significant ethical challenges - if a model can be trained to in effect \"become\" a software package, then it will become difficult to preserve the current economic approach to selling software.  \n\nBesides that - I understood why they chose to rely on an RNN, but those are famously not as expressive as transformers, and, in general, I would have liked to see more analysis of what kinds of applications \\textit{couldn't} be simulated, if any; if literally any software could be simulated, I would have liked to know how they got so much expressivity out of an RNN.  What I would imagine based on their architecture is that the system would be limited in both how complex the software they simulated could be, as well as by how many different software packages a single module could simulate.  I would have liked to see the performance as a function of how many distinct software packages were being imitated by a given module.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)