Octax: Accelerated CHIP-8 Arcade Environments for Reinforcement Learning in JAX
Authors: Waris Radji
Thomas Michel
Hector Piteau
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Benchmarking
CHIP-8
JAX
Environments
Simulation
Acceleration
Primary area: reinforcement learning

Abstract: Reinforcement learning (RL) research requires diverse, challenging environments that are both tractable and scalable. While modern video games may offer rich dynamics, they are computationally expensive and poorly suited for large-scale experimentation due to their CPU-bound execution. We introduce Octax, a high-performance suite of classic arcade game environments implemented in JAX, based on CHIP-8 emulation, a predecessor to Atari, which is widely adopted as a benchmark in RL research. Octax provides the JAX community with a long-awaited end-to-end GPU alternative to Atari games, offering image-based environments, spanning puzzle, action, and strategy genres, all executable at massive scale on modern GPUs. Our JAX-based implementation achieves orders-of-magnitude speedups over traditional CPU emulators. We demonstrate Octax's capabilities by training RL agents across multiple games, showing significant improvements in training speed and scalability compared to existing solutions. The environment's modular design enables researchers to easily extend the suite with new games or generate novel environments using large language models, making it an ideal platform for large-scale RL experimentation. Our open-source framework is available at https://github.com/riiswa/octax/.

Published as a conference paper at ICLR 2026
OCTAX: ACCELERATED CHIP-8 ARCADE ENVIRON- MENTS FOR REINFORCEMENT LEARNING IN JAX
Waris Radji'* Thomas Michel
Hector Piteau
Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189-CRISIAL, France
IIndependent Researcher
(waris.. rrajji, thomas.. michell @inria.. fr
hector, pactor.piteauuegaaiin com
ABSTRACT
Reinforcement learning (RL) research requires diverse, challenging environments that are both tractable and scalable. While modern video games may offer rich dynamics, they are computationally expensive and poorly suited for large-scale experimentation due to their CPU-bound execution. We introduce OCTAX, a high-performance suite of classic arcade game environments implemented in JAX, based on CHIP-8 emulation, a predecessor to Atari, which is widely adopted benchmark in RL research. OCTAX provides the JAX community with long-awaited end-to-end GPU alternative to Atari games, offering image-based environments, spanning puzzle, action, and strategy genres, all executable atmas- sive scale on modern GPUs. Our JAX-based implementation achieves orders-of- magnitude speedups over traditional CPU emulators. We demonstrate OCTAX's capabilities by training RL agents across multiple games, showing significant im- provements in training speed and scalability compared to existing solutions. The environment's modular design enables researchers to easily extend the suite with new games or generate novel environments using large language models, making it an ideal platform for large-scale RL experimentation. Our open-source frame- work available at Itttp:: // tighub com/ http:riiiswaaaaaaaa
INTRODUCTION
Modern reinforcement learning (RL) research (Sutton & Barto 2018) demands extensive experi- mentation to achieve statistical validity, yet computational constraints severely limit experimental scale. RL papers routinely report results with fewer than five random seeds due prohibitive train- ing costs (Henderson et al.] 2018)) Colas et al.) 2018 Agarwal et al., [021]] Mathieu et al.] [023] Gardner al. (025).. While understandable from practical standpoint, this undersampling un- dermines statistical reliability and impedes algorithmic progress. Environment execution creates this bottleneck: while deep learning has embraced end-to-end GPU acceleration, RI. environments remain predominantly CPU-bound. Originally designed under severe hardware constraints, classic arcade games represent solution for scalable RL experimentation. The Atari Learning Environ- ment (ALE) (Bellemare et al. 2013) has established itself as a standard RI. benchmark, although existing implementations remain fundamentally CPU-bound. As noted by OOaando-Ceron & Castro 2020), the Rainbow paper (Hessel et al.) 2018) required 34,200 GPU hours (equivalent to 1,425 days) experiments, computational cost that is prohibitively high for small research laboratories. In this paper, we propose alternative approach for training RL agents in environments that share mechanisms with ALE, but which is not intended as a drop-in in replacement and offers significantly reduced computational cost.
Contributions. We introduce OCTAX, suite of arcade game environments implemented in JAX (Bradbury et al. 2018a) through CHIP-8 emulation. CHIP-8, a 1970s virtual machine specification contemporary with early Atari systems, became the foundation for numerous classic games spanning puzzle, action, and strategy genres, CHIP-8's constraint-driven design creates games with similar
"The project page is available httph:ttt////.... http://sssaaiicccoootoooa-aaaaa
wwtp:/ow/ooot...tetteto. page//

---

Published as a conference paper at ICLR 2026
000
10
D22
IF
D00
13
Figure I: Overview of CHIP-8 game environments implemented in OCTAX.
cognitive demands to Atari while enabling efficient vectorized emulation that scales to thousands of parallel instances. The JAX ecosystem has rapidly emerged solution for scalability in RL research but lacks native environments, particularly image-based ones. Our framework addresses this gap by transforming classic games into fully vectorized, GPU-accelerated simulations. These simulations run thousands of game instances in parallel while maintaining perfect fidelity to the orig- inal mechanics. This approach dramatically reduces experiment times. Experiments that previously required days weeks can be completed in hours. This efficiency makes comprehensive hy- perparameter sweeps and ablation studies computationally feasible. The modular design facilitates extension with new games automated generation using large language models that directly output CHIP-8 assembly code. Figure Souure[ppe provides an overview of the integrated CHIP-8 games.
Outline. First, we present end-to-end JAX implementation of classic arcade environments through CHIP-8 emulation (Section Second, we demonstrate diverse learning dynamics through PPO evaluation across 16 games (Section [41].. Third, achieve 350,000 environment steps per second (1.4 million frames per second) on consumer-grade hardware, substantially outperforming CPU-based solutions (Section 4.2). Fourth, establish L.-M-assisted pipeline for automated environment generation that creates meaningful difficulty gradients (Section
RELATED WORK
Game environments have proven essential for RL research because they provide engaging, human- relevant challenges with clear success metrics. The Arcade Learning Environment (ALE) Bellemare al. (2013) demonstrated this principle by establishing Atari 2600 games as the standard RI. bench- mark, enabling breakthrough algorithms like DQN (Mnih et al. 2015) and Rainbow (Hessel al.) 2018). The success of these classic arcade games stems from their constraint-driven design: simple rules that yield complex behaviors, deterministic dynamics that enable reproducible experiments, and visual complexity that tests spatial reasoning without overwhelming computational resources. While algorithmic advances demand increasingly large-scale experiments with dozens of parallel environments and extensive hyperparameter sweeps, traditional game environments remain CPU- bound and poorly suited for parallel execution. This mismatch has driven progression of solutions, each addressing different aspects of the scalability problem.
Game-based RL environment platforms. Increasingly sophisticated gaming platforms have been developed to test different dimensions of learning performance. NetHack Learning Environment (Küttler et al. 2020) provides procedurally generated roguelike challenges that test long-term plan- ning, while Crafter (Hafner) 22221) offers simplified Minecraft-like like environments focused on source management. These environments expand cognitive challenges beyond arcade games, but their CPU-based implementations compound the scalability problem.
2

---

Published as a conference paper at ICLR 2026
CPU high--peforrmance solutions. Several projects have focused on optimizing CPU-based envi- ronment execution. EnvPool (Weng et al. 2022) achieves substantial speed improvements through highly optimized C++ implementation, demonstrating up to million Atari frames per second on high-end hardware. Pufferl.bb Jib (Suarez) 2025) provides environments written entirely in C, achieving millions of steps per second through 20,000 lines of optimized code. While these approaches improve CPU throughput, they retain fundamental limitations: costly CPU-GPU data transfers dur- ing training and require implementation in Python-dominate field.
GPU-accelerated RL environments. GPU-accelerated solutions target the constraint more directly by moving environment execution accelerators. CUDA Learning Environment (CuLE) (Dalton et al., (020)) provides a pioneering CUDA port of ALE, achieving 1901990 million frames per hour on single GPUs. Isaac Gym (Makoviychuk et al. 202T) demonstrates similar principles for robotics tasks, achieving 2-3 orders of magnitude speedups over CPU approaches by running thousands of environments simultaneously. These GPU approaches solve computational bottlenecks but intro- duce NVIDIA hardware dependence and substantial per-environment engineering costs.
--bed environments. The adoption JAX (Bradbury al. 2018b) has enabled natively accelerated environments that combine portability across hardware with end-to-end GPU acceler- ation. Brax (Freeman et al. 202I) established viability through MuJoCo-like physics simulation, while Gymnax (Lange 2022) provides JAX implementations of classic control tasks and simplified environments from BSuite (Osband et al.) 2019) and MinAtar (Young & Tian 2019). Specialized environments target specific research needs: Xinnn-riii (Nikulin al.
2024) and Navix (Pig-) natelli et al.) 2024) focus on gridworld navigation, Jumanji (Bonnet et al. 2023) spans domains
from simple games to NP-hard combinatorial problems, Pgx (Koyamada al. 2023) provides clas-
sic board games, and PuzzleJAX Earle et al. (2025) enables dynamic compilation of puzzle games.
Despite this coverage, a critical gap remains: classic arcade games. While MinAtar provides sim- plified versions of Atari games, the full visual complexity and authentic game mechanics of classic arcade games remain absent from the JAX ecosystem. OCTAX addresses this gap by providing the first end-to-end JAX implementation of classic arcade games through CHIP-& emulation, delivering computational benefits while preserving the engaging gameplay mechanics that made arcade games valuable for algorithmic development.
3 OCTAX: THE ACCELERATED CHIP-8 PLATFORM
OCTAX
Action Set
Function
ROM Looding
Iolillzztion
Fetch- www.odd--eeeottee (JAX vedtorbee)
Octax Environment Wrapper
Cassic RL Intaractinn Loop
CHIP-S
Agent
35 CHEP-8 Instructions
Startup Instructions
Score Function
Figure 2: OCTAX architecture: ROM loading, CHIP-& emulation pipeline, and RL environment in- tegration. The system transforms game ROMs through JAX operations suitable for GPU acceleration.
execute cycles into vectorized
This section presents our JAX implementation of CHIP-8 emulation. We detail the design deci- sions that enable GPU acceleration while maintaining behavioral fidelity to original games, and ex- plain how CHIPP-'s architecture provides an optimal foundation for scalable experimentation RL. Figure 2 summarizes this section.
3.1 WHY CHIP-8 FOR RL RESEARCH?
CHIP-8 represents a strategic choice for RL environment design. Created in the 1970s as a virtual machine specification, CHIP-8 features 64x32 monochrome display, 16 registers, 4KB memory,
3

---

Published as a conference paper at ICLR 2026
and 35-instruction set. These constraints, originally imposed by early microcomputer limitations, create several research advantages.
The platform provides image-based environments comparable to Atari games while offering some computational advantages. The 4KB memory footprint allows thousands of simultaneous game instances without memory constraints. The simple instruction set reduces emulation overhead com- pared to complex modern processors. The deterministic execution model ensures experimental re- producibility across different hardware configurations.
The platform supports everything from precise action games requiring split-second timing to com- plex puzzles demanding long-horizon planning. The 16-key input system provides sufficient com- plexity for interesting control challenges while remaining tractable for systematic analysis. Most importantly, CHIP-8 games are inherently modifiable and analyzable: their simple assembly code can be automatically generated, modified, and assessed for difficulty, enabling novel research direc- tions in environment design and curriculum learning. This combination of Atari-like visual com- plexity with modern computational efficiency makes CHIP-8 well-suited for the JAX ecosystem, where extensive parallelization can transform week-long experiments into hour-long runs.
3.2 How DOES OCTAX WORK?
OCTAX converts CHIP-8 ROMe into vectorized RL environments while maintaining compatibil- ity with original games. The implementation leverages JAX's functional programming model and vectorization capabilities to enable GPU acceleration.
ROM loading and initialization. Game data is loaded from ch8 files into the emulator's 4KB memory space starting at address 0x200, following the standard CHIP-8 program layout first in- troduced in Weisbeeker (1978). The system initializes with font data at address 0x50, sixteen purrroe- registers (VO-VF), an index register (I), program counter (PC), and the 64x32 monochrome display buffer.
Fetch-decode-exeeute cycle. The core emulation loop implements the classic processor cycle using JAX primitives. The fetch () function retrieves 16-bit instructions from memory and advances the program counter. The decode () function extracts instruction components through bitwise operations, identifying opcodes, register indices, and immediate values. The execute () function uses JAX's lax switch ch for GPU-compatible instruction dispatch to specialized handlers.
Vectorized instruction execution. Instruction handlers follow JAX's functional programming model, treating state immutable and returning updated copies. ALU operations handle arithmetic and bitwise logic with carry/borrow flag management. Control flow instructions implement jumps, calls, and conditional operations using lax. cond. The display system uses vectorized operations to render sprites across the entire framebuffer simultaneously.
Environment integration. The Oct wrapper transforms the emulator into standard RL interface. Each RL step executes multiple CHIP-8 instructions to maintain authentic game timing relative to the original 700Hz instruction frequency. The default frame skip setting preserves realistic game dynamics. Observations consist of the 64x32 display with 4-frame stacking, producing (4, 64, 32) boolean arrays. Actions map from discrete RL outputs to game-specific key subsets plus no- op option. The wrapper manages delay and sound timers at 60Hz and executes startup sequences to bypass menu screens. Even though some games use the RND Chip-8 instruction and are therefore stochastic, we provide additional wrappers for no-op reset and sticky actions.
3.3 HOW TO TRANSFORM GAMES INTO RL ENVIRONMENTS?
Converting CHIP-8 games into RL environments requires extracting reward signals and termination conditions from game-specific memory layouts and register usage patterns.
Score function design. Games store scores in different registers using various encoding schemes. OCTAX provides game-specific score_fi fn functions that extract scores from appropriate memory locations. Brix stores its score in register V5, incrementing with each destroyed brick. Pong encodes
ROM stands for Read-Only Memory, type of storage originally used in game cartridges to hold software that cannot be modified by the user.
4

---

Published as a conference paper at ICLR 2026
scores in BCD format within register V14, requiring score = (V[14] // 10) (V[14] 10) to compute player advantage. Our decision to support modular rewards intentional: how- it is incompatible with adopting numan-normalle
scoring schemes like those used in ALE.
Termination logic. Games signal completion through different register states that must be identi- fied through analysis. Brix terminates when lives (V14) reach zero, while Tetris uses dedicated game-state register (V1) that equals 2 on game over. Some games require compound conditions: Space Flight ends when either lives reach zero or a level completion counter exceeds a threshold, implemented as terminated = (V[9) = 0) | (V[12) >= Cx3E).
Action space optimization. Most games use subsets of the 16-key hexadecimal keypad. OCTAX supports custom act _on_set arryys that map RL action indices to relevant keys. Pong requires only keys and for paddle movement, while Worm uses directional keys 4, 6, This reduces action space size and accelerates learning by eliminating irrelevant inputs.
Initialization handling. Many games include menu screens that interfere with RL training. OCTAX supports startup Cetrrtupiinstuutt instrruct ons parameters that automatically execute instruction sequences during environment reset, bypassing menus begin gameplay immediately.
We address CHIP-8's non-standardized scoring and termination by combining static ROM analysis and dynamic memory monitoring during gameplay, as detailed in Appendix
3.4 WHICH GAMES DOES OCTAX SUPPORT?
OCTAX provides curated collection of classic CHIP-8 games across multiple genres and difficulty levels. The current implementation includes 21 titles, with additional games planned for future releases. All environments maintain full compatibility with both Gymnasium and Gymnax APIs.
Category
Available Games
Required Capabilities
Puzzle
Tetris, Blinky, Worm
Long-horizen planning, spatial reasoning Timing, prediction, reactive control
Action
Brix, Pong. Squash, Vertical Brix, Wipe
Off, Filter
Strategy
Missile Command, Rocket, Submarine,
Resource management, tactical decisions
Tank Battle, UFO
Exploration
Cavern (7 levels), Flight Runner, Space
Spatial exploration, continuous navigation
Flight (10 levels), Spacejam!
Shooter
Airplane, Deep8, Shooting Stars
Simple reaction, basic timing
Table I: Currently implemented games in OCTAX.
The games (Figure I vary across multiple dimensions of difficulty and cognitive demand. Tem- poral complexity ranges from immediate reactions to long-term planning requirements. Spatial complexity spans single-screen environments to multi-screen worlds requiring navigation. Reward structures include both dense scoring mechanisms and sparse achievement-bact tematic variation enables controlled studies of algorithmic performance across different challenge types while maintaining unified technical framework for fair comparison. A categorization of these games is provided in Table with more detailed descriptions available in Appendix CK
systems. This sys-
4 EXPERIMENTAL EVALUATION
We evaluate OCTAX through RL training experiments across 16 diverse CHIP-8 games. Our goal is to demonstrate that the environments present varied difficulties and learning dynamics suitable for RL research. We then evaluate the platform's computational performance.
4.1 HOw DO RL AGENTS LEARN IN OCTAX?
We train Proximal Policy Optimization (PPO) (Schulman et al. 2017) agents across our game suite due to its widespread adoption and proven scalability with parallel environments (Rudin et al.. (222), and Parallel Q-Network (PQN) (Galiic et al. 2024) as a modern value-basee based method that also benefits from parallel environments.
5

---

Published as a conference paper at ICLR 2026
airplane
blinky
brix
deep
200
20
0.0
0
0
0
filter
missile
pang
spacejam
40
5
10
0
20
Returns (IQM)
-5
5
0
squash
submarine
tank
tetris
75 0
25
0.02
mmm
-5
25
0.00
ufo
vertical brix
wipe off
worm
30
100
20
40
1
50
10
20
0
Timesteps
PPO
PQN
Figure 3: PPO and PQN learning curves across 16 games: interquartile mean returns using 10th-90th percentile ranges over 5M timesteps, with confidence intervals computed across 12 random seeds.
Network architecture. Both agents] uses the same architecture, for fair comparison: convolu- tional neural network designed for OCTAX's (4, 64, 32) stacked observations. The feature extractor consists of three convolutional layers with 32, 64, and 64 filters, respectively. These layers use kernel sizes of (8,4), 4, and 3 with corresponding strides of (4.2), 2, and 1. Extracted features are flattened and fed separate actor and critic heads, each containing a single 256-unit hidden layer with ReLU activation throughout (and LayerNorm for PQN).
Training configuration. We combine grid search optimization (detailed in Appendix D on Pong with CleanRL's standard Atari PPO hyperparameters (Huang et al. 2022). This yields GAE lambda of 0.95, clipping epsilon of 0.2, value function coefficient of 0.5, and entropy coefficient of 0.01 for PPO. For PQN we use = 0.9 and an ppsilon-greede exploration decaying from 1. to 0.05 during 10% of the training. Each experiment uses 512 parallel environments with 32-step rollouts, training epochs per update, and 32 minibatches for gradient computation. We apply the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 5 10-4 and gradient clipping ensure stable training across 5 million timesteps per environment.
Experimental setup. We conduct 12 independent training runs per game using different random seeds. All experiments run single NVIDIA A100 GPU with 24 concurrent training sessions. Agent performance is assessed every 131,072 timesteps on 128 parallel environments.
Results analysis. The training curves in Figure rreval distinct learning profiles across games. We observe three main patterns that reflect different cognitive demands and exploration needs. Rapid plateau games (Airplane, Brix, Deep, Filter, Blinky) show quick initial learning followed by stable performance, suggesting clear reward signals. Gradual improvement games (Pong, Tank, Vertical Brix) learn continuously over the course of training, indicating either sparser reward structures or more complex strategic requirements. Limited performance games, like Tetris, show little absolute
Based Rejax implementation Liesen al.
2024

---

Published as a conference paper at ICLR 2026
progress, making them difficult for methods without targeted exploration. Similarly, in Worn (a Snake clone), agents often manage to eat only a single apple before dying.
These learning profiles support the diversity of CHIP-8 environments, demonstrating that different games test varied aspects learning and control. Individual training runs averaged 65 minutes cach, with 24 experiments running concurrently, achieving approximately 30,800 environment steps per second across all parallel sessions.
4.2 How DOES OCTAX SCALE WITH PARALLELIZATION?
10
Octax
Steps per Second
300K
EnvPool
8
6
Time (5)
200K
4
100K
2
0
0
2
2
2
24
2
26
27
28
2
20
2
2
2
Number of Environments
Figure 4: Performance scaling of OCTAX and EnvPool across parallelization levels. The solid purple line is the number of steps per second (higher is better), and the dashed green line is the total execution time seconds (lower better).
Experimental setup. We measure environment throughput across different parallelization lev- els to quantify OCTAX'S computational advantages. This experiment isolates pure computational benefits by fixing the game (Pong) and agent behavior (constant action) while varying parallel en- vironment instances. Since all environments execute identical CHIP-8 computational cycles, these performance measurements apply uniformly across the entire game suite. To better interpret our results, we compare against EnvPool because it is widely adopted in RL research, using ALE Pong to assess CPU vs. GPU-based environment scalability.
Configuration. We benchmark on a consumer-grade workstation with an RTX 3090 (24GB VRAM), 32GB RAM, and an Intel i7 processor (20 cores). We measure execution time for 100- step rollouts across varying parallel environment counts, with 50 independent measurements per configuration. The primary metric is environment steps per second, calculated (number of en- vironments x 100 steps) divided by execution time, where each step represents 4 frames due to OCTAX's default frame skip setting.
Performance results. Figure 4 demonstrates near-linear scaling up 350,000 steps (or 1.4M frames) per second with 8, 1.92 parallel environments before hitting VRAM limitations. EnyPool running ALE Pong with all available CPU cores shows reduced scaling, plateauing around 25. 25,000 steps per second due to CPU saturation. OCTAX achieves a 14x improvement in computational efficiency at high parallelization levels, reducing the computational cost of large-scale RL exper- iments. We also measured GPU memory usage across different environment counts, finding that execution memory scales linearly with the number of parallel environments with our benchmark script, consuming approximately 2 MB of GPU memory per environment.
4.3 How DO LLMS ASSIST ENVIRONMENT CREATION?
Large language models (LLMs) have demonstrated strong capability in code generation across diverse programming languages, enabling the automated creation of environments RL research. Here explore OCTAX'S capacity accelerate research by leveraging L.MMs to generate novel

---

Published as a conference paper at ICLR 2026
tasks, extending beyond manually designed game suites toward automated environment synthesis, as explored in Faldor et al. (2024).
Context. During OCTAX's development, we encountered a few games where reward and termina- tion logic proved difficult to extract through manual analysis of game mechanics. In these cases, we decompiled ROMs to obtain CHIP-8 assembly code and successfully employed LLMs to explain the code and guide us in defining the correct score_ fn and terminated. fn functions. Using gpt-4o-minii from the OpenAl API, we evaluated our 21 games by checking whether the model could reproduce the same score and termination functions as our hand-written implementations. While the LLM might, in principle, discover alternative reward definitions that are still semantically valid, our goal here was to assess direct match accuracy under a limited context. We found that the model performs reliably when the score is stored in single register (57% perfect matches), but termination logic proved harder, with only 19% correct due to missed multi-register OR/AND conditions or encoded state variables. This small feasibility study illustrates that LLMS can recover simple reward signals but still require human oversight, especially with no interactive debugging or additional context. Full details and per-game analyses are provided in Appendix E This exper- iment motivated us to investigate the reverse pipeline: using LLMs to generate complete CHIP-8 games from high-level descriptions, then leveraging OCTAX's scalable simulation to evaluate these procedurally created environments.
Automated environment generation pipeline. Our pipeline consists of seven replicable steps for automatic CHIP-8 game generation. In Step 1, we construct corpus of CHIP-8 tutorials, documentation, and pro- gramming examples, ensuring the LLM understands the architecture's instruction set, memory layout, and common coding patterns. In Step 2, we embed this corpus into prompt (detailed in Appendix F.I that guides the LLM to produce syntactically correct CHIP-8 programs from high-level instructions. In Step 3, we provide a description of the game with desired mechanics, objectives, and constraints. In Step 4, the LLM generates the initial CHIP-8 code based the provided description. In Step 5, an automated feedback loop between the LLM and a CHIP-8 compiler iteratively refines the code based on compilation errors until successful. In Step 6, Python wrapper functions for score En and cerminated fn are automatically generated, translating CHIP-8 reg- isters into RI-compatible reward and termination signals. Finally, in Step 7, the game description augmented to increase difficulty or in- troduce new challenges. Both the new description and the previously generated game are added to the LLM's context before next iteration. Figure 5 summarizes the automated environment generation pipeline.
1. Build Corpus
Prepare Prompt
3. Define Game
Generate Code
Retry on Error
Check & Reflne
Repeat
Success
Target Shooter case study. We validated this pipeline using Claude Opus 4.1, known for its proficiency in programming, with the following description: "Target Shooter Targets appear randomly on the screen, and the player moves crosshair to shoot them. Score increases per hit, and the game ends after a fixed number of targets." The system suc- cessfully generated three progressive difficulty levels: static targets for basic aiming skills, time-limited targets introducing decision pressure, and moving targets with time constraints requiring predictive aiming. Each level maintains consistent register mappings for score and termina- tion, simplifying OCTAX compatibility. Figure 6 shows how the LLM- generated environment visual appearance. All the code generated by the L.M is given in F2
6. Generate Score
Terminate funcs
7. Augment Difficulty
Figure 5: Environment generation pipeline.
RL experiments. Using identical PPO configurations from Section 4.1 we trained agents on the three generated difficulty levels over 5M timesteps. Figure ( demonstrates clear performance strat- ification across difficulty levels: Level 1 agents achieved optimal returns of 10.0 with rapid con- vergence by 1M timesteps, Level 2 agents plateaued at 9.0 returns with moderate learning speed, while Level 3 agents reached 8.0 returns with the slowest progression. The inverse relationship between difficulty level and both final performance and sample efficiency indicates that our LLM- generated environments successfully create a meaningful difficulty gradient. This proof-of-concept
8

---

Published as a conference paper at ICLR 2026
demonstrates the feasibility of automated environment generation for RL research via OCTAX, with promising applications in curriculum learning, open-endedness, and continual learning scenarios.
10.0
Returns (IQM)
7.5
5.0
Level
Level 2
2.5
Level 3
2
5
Timesteps
1e6
Figure 6: Rendering of the Target Shooter game showing the player (left, circular ob- ject) and target (right, cross-shaped object).
Figure 7: PPO training performance on gen- erated environments with varying difficulty.
5 CONCLUSION
We introduced OCTAX, a JAX-based CHIP-8 emulation platform that provides GPU-accelerated cade game environments for reinforcement learning research. Our implementation achieves signifi- cant performance improvements of parallel environments while maintaining perfect behavioral fidelity to original games. Through PPO evaluation across 16 diverse games, demonstrated varied learning dynamics that highlight the cognitive diversity within classic arcade environments. The platform's 's modular design enables both manual game integration and automated environment generation using large language models, providing researchers with flexible experimental design options.
CPU-based alternatives, enabling experiments with thousands
Societal and environmental impact. OCTAX enables more rigorous evaluation with larger sam- ple sizes, addressing reproducibility concerns that affect institutions with limited computational resources. This implementation can reduce energy consumption compared resource-intensing benchmarks such as ALE: experiments that once required top-tier clusters can now run efficiently on a single GPU, potentially saving significant compute time and resources.
The GPU-based architecture faces performance constraints due to CHIP-8's variable instruction execution complexity. JAX synchronization across parallel environments means each step's execution time depends on the slowest instruction among CHIP-8's S 35 operations, typically display rendering or complex ALU operations. The absence of established maximum scores across game suite prevents the assessment of whether agents approach theoretical performance limits, limiting evaluation of algorithmic performance ceilings.
Limitations.
Future work. OCTAX expand through community contributions, with hundreds of compati- ble ROMs available online. The LLM-assisted environment generation pipeline enables curriculum learning and open-ended research through procedurally generated games that provide task diversity. We plan to investigate emulator optimizations including instruction-level parallelization strategies and adaptive batching address synchronization bottlenecks from variable execution times. We also aim to extend platform support to Super-CHIP8 and XO-CHIP variants: Super-CH CHIP8 offers higher resolution displays (128x64) and extended instruction sets originally developed for HP48 calculators, while XO-CHIP provides color graphics, improved audio, and expanded memory while maintaining backward compatibility. These extensions would enable OCTAX to support more so- phisticated games and visual complexity while preserving the computational efficiency advantages of the JAX-native architecture. Many CHIP-8 games feature multi-agent or multi multi-player mechanics, which plan to support in future platform releases. The platform's 's high-throughhu capabilities also position it well for offline RL research, enabling the efficient creation of large-scale datasets and the comprehensive evaluation of offline algorithms across diverse game environments.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The authors Waris Radji and Thomas Michel are affiliated with the Inria Scool team project. This work has been supported by the French Ministry of Higher Education and Research, the Hauts- de-Francce region, Inria, and the MEL. Additional support was provided by the French National Research Agency under the PEPR IA FOUNDRY project SANR-23-PEIA-OR sented in this paper were carried out using the PlaFRIM experimental testbed, supported by Inria, CNRS (LABRI and IMB), Université de Bordeaux, Bordeaux INP and Conseil Régional d' Aquitaine (see httptttt////. www. ffrmmm.. Fr
Experiments pre-
REPRODUCIBILITY STATEMENT
We provide complete resources to ensure reproducibility of results. The OCTAX source code, including all 21 game environment implementations, JAX-based CHIP-8 emulator, and training scripts, available as supplementary material. Our experimental setup uses standard PPO hyperpa- rameters detailed in Section 4.1 with hardware specifications and performance benchmarking con- figurations provided in Section 4.2 All training experiments
identical network architectures and hyperparameters across games, enabling direct replication of our learning curves in Figure For the LLM-assisted environment generation pipeline in Section
include the prompt templates and generated CHIPP-8 assembly code in Appendix The modular design of OCTAX allows researchers extend our game suite using the technical specifications in Section The repository containing all source code, experiments, and data is available at: https: /tapp:///gtte thub.. com/ http://iiawaaaa//e /ttp///t///////////////////////t
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural informa- fion processing systems, 34:29304-29320, 2021.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of artificial intelligence research, 47: 253-279, 2013.
Clément Bonnet, Daniel Luo, Donal Byrne, Shikha Surana, Sasha Abramowitz, Paul Duck- worth, Vincent Coyette, Laurence Midgley, Elshadai Tegegn, Tristan Kalloniatis, et al. Ju- manji: a diverse suite of scalable reinforcement learning environments in jax. arXiv preprint arXiv:2306. 09884, 2023.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018a. URL http:
//github. com/
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Millen et al. Jax: composable transformations of python+ numpy programs. 2018b.
Cédric Colas, Olivier Sigaud, and Pierre- Yves Oudeyer. How many random seeds? statistical power analysis in deep reinforcement learning experiments. arXiv preprint arXiv: /ariv:/806. /806.08295 2018.
Steven Dalton et al. Accelerating reinforcement learning through gpu atari emulation. Advances in Neural Information Processing Systems, 33: 1377337719978,, 2020.
Sam Earle, Graham Todd, Yuchen Li, Ahmed Khalifa, Muhammad Umair Nasir, Zehua Jiang, An- drzej Hanburski-Fahey and Julian Togelius. Puzzlejax: benchmark for reasoning and learning. arXiv preprint a50iv 2eei:22088....... /6821, 2025.
Maxence Faldor, Jenny Zhang, Antoine Cully, and Jeff Clune. Omni-epic: Open-endedness via mod- els human notions of interestingness with environments programmed in code. arXiv preprint arXiv:2405. 15568, 2024.
10

---

Published as a conference paper at ICLR 2026
C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a a differentiable physics engine for large scale rigid body simulation. arXiv preprint 1rXiv:2/06.13281, 2021.
Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning. arXiv preprint arXiv:2407.04811, 2024.
Jason Gardner, Ayan Dutta, Swapnoneel Roy, O Patrick Kreidl, and Ladislau Boloni. Greener deep reinforcement learning: Analysis of energy and carbon efficiency across atari benchmarks. arXiv preprint arXiv:2509.055 0090223,, 2025.
Danijar Hafner. Benchmarking the spectrum of agent capabilities. arXiv preprint (7X0022009.067800 2021.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Ki- nal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-fil implemennatioons of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
URL http://
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980, 2014.
Sotetsu Koyamada, Shinri Okano, Soichiro Nishimori, Yu Murata, Keigo Habara, Haruka Kita, and Shin Ishii Pgx: Hardwware-acceleaate parallel game simulators for reinforcement learning. Advances in Neural Information Processing Systems, 36:45716-45743. 2023.
Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. Advances in Neural Information Processing Systems, 33:7671-7684, 2020.
Robert Tjarko Lange. gymnax: A based reinforcement learning environment library, 2022.
URL /att:://qithui com httm///ooerr.taggeee//ee gymnaxx
Jarek Liesen, Chris Lu, and Robert Lange. /
rejax, 2024. URL https:// com/
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gau-based physics simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021.
Timothée Mathieu, Riccardo Della Vecchia, Alena Shilova, Matheus Medeiros Centa, Hector Kohler, Adblric-Ambrym Maillard, and Philippe Preux. Adastop: adaptive statistical testing for sound comparisons of deep agents. arXiv preprint arXiv:2306. 17.3066.8882 10882, 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 817754)):529-533 2015.
Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman, Artem Agarkov, Viacheslav Sinii, and Sergey Kolesnikov. Xland-minigrid: Scalable meta-reinforcemmen learning environments in jax. Ad- vances in Neural Information Processing Systems, 37:43809 2024.
Johan S Obando-Ceron and Pablo Samuel Castro. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. arXiv preprint arXiv:2011.1443 2020.
11

---

Published as a conference paper at ICLR 2026
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, et al. Behaviour suite for reinforce- ment learning. arXiv preprint arXiv:1908.03566. 2019.
Eduardo Pignatelli, Jarek Liesen, Robert Tjarko Lange, Chris Lu, Pablo Samuel Castro, and Laura Toni. Navix: Scaling minigrid environments with jax. arXiv preprint arXiv:2407. 19396, 2024.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning walk in minutes using massively parallel deep reinforcement learning. In Conference on robot learning, pp. 91-100. PMLR. 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Joseph Suarez. Pufferlib 2.0: Reinforcement learning at Im steps/s. In Reinforcement Learning Conference, 2025.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 2 edition, 2018.
Joseph Weisbecker. An easy programming system. BYTE, 3(12):108-122, December 1978. First introduction of CHIP-8 programming language.
Jiayi Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu, Yufan Song, Ting Luo, Yukun Jiang, et al. Envpool: A highly parallel reinforcement learning environment execution engine. Advances in Neural Information Processing Systems, 35:22409- 22421, 2022.
Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv: 2019.
A USE OF LARGE LANGUAGE MODELS
We used Large Language Models three capacities during this research. First, Claude Opus serves core research component in Section 4.3 where we demonstrate automated CHIP-8 game generation from high-level descriptions. This represents a novel research contribution, with all gen- erated code validated through compilation and RL experiments. Second, employed Claude Son- net 4 for writing assistance, including text refinement, rephrasing technical concepts, and improving academic tone. Third, LLMs generated code documentation, docstrings, and tutorial content. All research ideas, experimental design, and scientific claims originate from the authors. We did not LLMs for ideation, hypothesis formation, result interpretation. We manually reviewed and validated all LLM-assisted content for accuracy and take full responsibility for all presented content.
B CHIP-8 TECHNICAL SPECIFICATIONS
B.1 PLATFORM OVERVIEW
CHIP-8 was created by Joseph Weisbecker at RCA in the mid-1970s as a virtual machine for early microcomputers. The platform established one of the first successful portable gaming ecosystems by providing hardware abstraction layer that enabled games to run across different systems.
B.2
SYSTEM ARCHITECTURE
The CHIP-8 architecture consists of:
Memory: 4KB total, with programs loaded at address 0x200
Registers: 16 8-bit registers (VO-VF), with VF serving as a flag register
Display: 64x32 pixel monochrome screen with XOR-based rendering
12

---

Published as a conference paper at ICLR 2026
Input: 16-key hexadecimal keypad (0-9, A-F)
Timers: 60Hz delay timer and sound timer
Audio: Single-tone buzzer
B.3
INSTRUCTION SET HIGHLIGHTS
CHIP-8's 35-instruction set includes specialized gaming primitives:
Sprite Drawing (DXYN): XOR- rendering enabling collision detection
Key Input (EX9E, EXA1): Skip instructions based key state
BCD Conversion (FX33): Convert register values decimal display
Memory Operations: Bulk register loading/storing (FX55, FX65)
The XOR-based sprite system is particularly elegant: drawing the same sprite twice erases it, abling simple animation and automatic collision detection when pixels turn off.
B.4 FONT SYSTEM
CHIP-8 includes built-in 4x5 pixel font data for hexadecimal digits (O-F), stored at addresses 0x050- 0x09F. Games reference these fonts for score and text display by setting the index register to the appropriate font location.
C GAME ENVIRONMENT IMPLEMENTATION DETAILS
C.1 SCORE DETECTION METHODOLOGY
CHIP-8 8 games store scoring information in arbitrary memory locations using game-specific formats. Our automated detection operates in two phases:
Static Analysis: We analyze ROM structure for common programming patterns, particularly binary-
coded decimal (BCD) operations (FX33 instruction) that suggest numeric display routines.
Dynamic Monitoring: During human gameplay sessions, we monitor memory changes to correlate locations with scoring events. Register trend analysis identifies increasing values (likely scores) versus decreasing values (likely lives/health).
C.2 REWARD DESIGN
Each environment implements reward function that extracts scoring information from the emula- tor state. The reward is computed by reading specific CHIP-8 registers (V0-VF) that track game- relevant metrics. Most environments use direct score extraction where the reward equals the value stored in score register. Some environments apply transformations to the raw register values to shape the reward signal for reinforcement learning.
C.3 GAME LIST
C.3.1
LONG-HORIZON PLANNING SPATIAL REASONING
Requires strategic thinking, spatial awareness, and multi-step planning
tetris - Tetris by Fran Dachille (1991): Classic Tetris with piece rotation, movement, and dropping. Uses keys (rotate left), (move left), (move right), and (drop). Speed increases every 5 lines and peaks at 45 lines. Reward: Register V[10] containing the score. Termination: Game over when V[1] equals 2 (Board overflow).
blinky Blinky by Hans Christian Egeberg (1991): Pac-Man clone where the player nav- igates maze to collect pills while avoiding ghosts (Packlett and Heward). The maze
13

---

Published as a conference paper at ICLR 2026
contains one gateway and four energy pills near corners. Points awarded for each pill, ergy pill, catching ghosts, and finishing the maze. Player has lives. Uses keys 3, 6. 7, and 8 for movement. Reward: Register V[6] containing the score. Termination: Game over when V[3] equals 255 (Collision with ghost).
worm SuperWorm V4 by RB-Revival Studios (2007): Snake-like game for Chip8. The player navigate the worm to collect items while avoiding walls and self-collision. Uses keys 2, 8, 4, and 6 for directional movement. Reward: Register V[5] containing the score. Termination: Game over when V[7] equals 255 (Collision).
C.3.2 TIMING, PREDICTION & REACTIVE CONTROL
Requires precise timing, trajectory prediction, and fast reactive responses
brix Brix by Andreas Gustafsson (1990): Breakout clone where the player controls paddle to bounce a ball and destroy bricks. Player has 5 lives. Uses keys 4 (left) and 6 (right) for paddle movement. Reward: Register V[5] increments per brick destroyed. Game over when V[14] equals 4 (No more lives).
Termination:
pong Pong: Single player pong game where the player controls a paddle to hit the ball. Uses keys I and 4 for paddle movement. Reward: Computed as (V[14] // 10) (V[14] % 10), representing player score minus opponent score. Termination: Game over when either score reaches 9.
squash - Squash by David Winter (1997): Bounce a ball around a squash court with paddle control. Uses keys (up) and 4 (down) for paddle movement. Player has 5 lives. Reward: Register V[1I] (Number of lives left). Termination: Game over when V[11] equals 0 (No more lives).
vertical brix - Vertical Brix by Paul Robson (1996): Breakout variant with vertical brick layout and paddle movement. In the original game you need to press 7 to start, but we skip that in the environment. Uses keys I and 4 to move the paddle vertically. Reward: Register V[8] (bricks climinated). Termination: Game over when V[7] (remaining lives) equals O.
wipe_ofe off Wipe Off by Joseph Weisbecker: Move paddle left right to wipe out spots on screen. Each spot counts point. Player gets 20 balls. Uses keys 4 (left) and (right) for paddle movement. Reward: Register V[6] (Number of points wiped out). Termination: Game over when V[7] (balls left) equals O.
filter Filter: Catch drops falling from a pipe at the top of the screen with paddle. Uses keys 4 (left) and 6 (right) for paddle movement. 7 lives. Reward: Register V[14] (number of drops caught). Termination: Game over when V[13] (remaining lives) equals
C.3.3 RESOURCE MANAGEMENT & TACTICAL DECISIONS
Requires managing limited resources and making strategic tactical choices
missile- - Missile Command by David Winter (1996): Shoot 8 targets on screen using key 8. The shooter moves faster with each shot. Player has 12 missiles total and earns 5 points per target hit. Reward: Register V[7] (points). Termination: Game over when V[6] (missiles
left) equals
rocket Rocket by Joseph Weisbecker (1978): An enemy UFO moves from left to right across the top of the screen. Launch rockets vertically by pressing key F (15). Rockets appear at random horizontal positions at the bottom. Score increments by when UFO is hit. Player has 9 rockets total. Reward: Register V(1] (points). Termination: Game over when V[2] (rocket launched) equals 9.
submarine Submarine by Carmelo Cortez (1978): Fire depth charges at submarines be- low using key 5. Score 15 points for hitting a small submarine and points for a large submarine. Player starts with 25 depth charges. Reward: Register V[7] (points). Termina- tion: Game over when V[8] (remaining lives) equals
tank Tank Battle: Control tank with 25 bombs hit mobile target. Uses keys 2, 4, 5, 6, and to move. the tank hits the target, player loses bombs. Reward: Register V[14] (points). Termination: Game over when V(6] (bombs left) equals O.
14

---

Published as a conference paper at ICLR 2026
ufo UFO by Lutz V (1992): Stationary missile launcher at the bottom of the screen shoots at flying objects. Uses keys 4 (left diagonal), (straight up), and (right diagonal) to fire. Player has 15 missiles. Score displayed left, remaining missiles right. Reward: Register V[7] (score). Termination: Game over when V[8] (missiles left) equals O.
C.3.4 EXPLORATION CONTINUOUS NAVIGATION
Requires spatial exploration, obstacle avoidance, and continuous movement control
cavern Cavern by Matthew Mikolay (2014): Navigate through a cave without crashing into walls. Uses keys 2, 4, 6, and 8 for movement. Modified ROM with leftward progress reward system where V[0] increments for each new leftmost position reached, encour- aging leftward exploration. V[A] tracks leftmost position ever visited. Reward: Register Game over when V[14] equals (crash into wall).
V[0]. Termination:
flight runner Flight Runner by TodPunk (2014): Simple flight navigation game. Uses keys 5. 7, 8, and 9 for movement controls. Reward: Register V[7]. Termination: Game
over when V[5] or V[7] equals 255.
Fly through asteroid field from left to right avoiding obsta- cles. Uses keys and 4 to navigate the spaceship. Modified ROM with single life mode and immediate termination
space_flight flight Space Flight:
space
collision. V[0] increments per frame survived as distance V[9] tracks lives. Reward: Register V[0]. Termination: Game over when V[9] equals 0 or V[12] least Ox3E.
spacejam Spacejam! by William Donnelly (2015): Ship tunnel navigation game based on ShipTunnel from 2014 Octolam. Uses keys 5, 8, 7, and 9 for movement. Reward: Register V[9]. Termination: Game over when V[10] equals 0 (Ship destroyed).
C.3.5 SIMPLE REACTION TIMING
Requires basic reaction time and simple decision making
airplane- Airplane: Bombing game where bombs are dropped by pressing key 8. V[II]
tracks remaining targets and V[12] tracks level progression. Reward: Computed as -V[11] V[[2], rewarding target hits and penalizing level progression. Termination: Game over when V[11] equals or V[12] equals 6.
deep Deep8 by John Earnest (2014): Move boat left and right with keys and 9. Press key 8 to drop bomb and release to detonate it. Destroy incoming squid before they tip the boat. Reward: Register V[9]. Termination: Game over when V[B] does not equal I.
shooting stars Shooting Stars by Philip Baltzer (1978): Classic shooting game. Uses keys 2, 8, 4, and 6 for movement. Reward: Register V[0] with capping at 128 (returns V[0] exceeds 128). Termination: Never terminates.
D HYPERPARAMETER OPTIMIZATION RESULTS
We conducted a comprehensive grid search on the Pong environment to identify optimal PPO hyper- parameters before evaluating across the full game suite. The search explored four key dimensions: number of parallel environments, rollout length, minibatch size, and learning rate. All experiments used 4 epochs per update, GAE lambda 0.95, and gradient clipping at 0.5.
D.1 SEARCH SPACE
The hyperparameter search explored the following ranges:
Environments: (128, 256, 512, 1024)
Rollout steps: (32, 64, 128,512) 512)
Minibatches: (4, 16, 32)
Learning rate: (2.5 10-1,5 10-4 1 10-3)
15

---

Published as a conference paper at ICLR 2026
Each configuration was trained for 1M timesteps with evaluation every 65,536 steps. Final eval- uation scores represent the last recorded performance, where less negative values indicate better performance.
D.2 RESULTS SUMMARY
Table 2 presents the key configurations and their final evaluation scores. Higher scores indicate better performance (scores negative, with values closer zero being better).
Table 2: Hyperparameter search results Pong environment. Configurations sorted by final evalu-
ation score.
Envs
Steps
Minibatches
LR
Score
S12
32
32
0.0005
-2.34
512
0.001
-2.48
512
32
32
0.001
-2.69
128
128
16
0.00025
-2.95
128
64
0.00025
-3.19
512
0.00025
33.00
0.00025
-3.38
128
32
0.00025
-3.44
128
64
16
0.00025
-3.53
512
32
16
0.0005
3373
256
0.00025
33778
128
128
0.00025
-3.11
256
128
32
0.00025
-4.03
512
64
16
0.00025
-4.17
1024
32
0.00025
-4.34
1024
0.00025
-4.4
256
128
16
0,00025
-4.66
1024
64
32
0.00025
-4.96
D.3 ANALYSIS AND KEY FINDINGS
Learning rate impact. Higher learning rates significantly improved performance, with 5 10-1 and x 10-3 substantially outperforming 2.5 x 10-4 The top three configurations all used learning rates above the commonly used 2.5 10-4
Environment scaling. 512 parallel environments provided the optimal balance between computa- tional efficiency and sample diversity. Configurations with 1024 environments showed diminishing returns, possibly due to computational overhead or reduced gradient update frequency.
Rollout length. Shorter rollouts (32 steps) consistently outperformed longer ones, indicating more frequent policy updates may be beneficial for this environment.
Minibatch size. Larger minibatch sizes (16-32) generally improved performance by providing more stable gradient estimates, though the effect was less pronounced than learning rate changes.
D.4
FINAL CONFIGURATION
Based on these results, we selected the following hyperparameters for all subsequent experiments:
Parallel environments: 512
Rollout steps: 32
Training epochs:
Minibatches: 32
Learning rate: 5 x 10 10-4
GAE lambda: 0.95
Clip epsilon: 0.2
16

---

Published as a conference paper at ICLR 2026
Value function coefficient: 0.5
Entropy coefficient: 0.01
E EVALUATION STUDY ON LLM-GENERATED REWARD FUNCTIONS
We evaluate whether LLMs can extract reward and termination functions from raw CHIP-8 as- sembly code, comparing LLM outputs against our human-defined implementations. Given CHIP-8 assembly code without symbolic information and any documentation, the LLM must implement two functions using only 16 registers: score_ fn (state) to extract the current game score, and rmmm nated_ (state) to determine if the game has ended.
Important cavear: Exact register matches indicate correct understanding the original implemen- tation, but alternative register choices may still represent valid game semantics.
We used the model gpt gpt-4o-ming from the OpenAl API due its low price. The full prompt we use is shown below:
You are analyzing CHIP-8 assembly code to extract reward and termination logic for reinforcement learning.
CHIP-8 ASSEMBLY CODE for (rom__narr _oom__amm)); (game_code)
TASK: Implement score_fn() and terminated_fnf using CNLY the 16 registers in state.V[c-15).
STEP-BY-STEP ANALYSIS REQUIRED:
1. REGISTER IDENTIFICATION:
Scan the assembly for registers that track score, lives, or game state
Look for patterns: incrementing (score), decrementing (lives),
flag checks (game over)
Note: Multiple registers may be involved (e.g., V[9] AND v[12])
2. SCORE FUNCTION:
Which register (s) hold the score value?
Is it simple read, or does it require calculation
(e.g., BCD decode, multi-register)? Provide evidence from assembly code
3. TERMINATION FUNCTION:
Which register (s) indicate game over?
What are the exact conditions? (equal, not equal, AND, OR?)
Check for: 1ives==0, game_state==X, score>=limit, etc.
Provide evidence from assembly code
OUTPUT FORMAT:
Game: [One-line description]
ANALYSIS:
Score (egister(s): V[X] because [reason from code]
Termination register (s) V[Y] because [reason from code]
def score_fn (state): return state.V(X) def ferminated_fn fn (state) return state.V[Y) == Z
CRITICAL REMINDERS:
Look for ALL conditions in termination (often OR/AND combinations)
Veri fy register choices against actual assembly operations
17

---

Published as a conference paper at ICLR 2026
Don't guess justify each register choice with code evidence
E.1 EVALUATION
We classify LLM outputs into four categories. Perfect: exact register(s) and logic match ground truth; Partial: correct register(s) but incomplete/nccoreet logic; Wrong Logic: uses ground truth registers in wrong context; Failure: completely unrelated register(s) ground truth. Results summarized in T
Table 3: LLM Function Generation Results (21 CHIP-8 Games)
Game
Score Function
Termination Function
Airplane
Partial
Failure
Blinky
Failure
Failure
Brix
Perfect
Failure
Cavern
Perfect
Perfect
Deep8
Perfect
Wrong Logic
Filter
Perfect
Perfect
FlightRunner
Perfect
Failure
Missile
Perfect
Partial
Pong
Partial
Failure
Rocket
Perfect
Failure
Shooting Stars
Partial
Failure
SpaceFlight
Wrong Logic
Failure
Spacejam
Perfect
Failure
Squash
Perfect
Wrong Logic
Submarine
Perfect
Perfect
Tank
Failure
Failure
Tetris
Failure
Failure
UFO
Perfect
Perfect
VerticalBrix
Wrong Logic
Failure
WipeOff
Perfect
Failure
Worm
Failure
Failure
Perfect
12/21 (57%
4/21 (19%)
Partial
3/21 (14%)
1/21 (5%)
Wrong Logic
3/21 (14%)
2/21 (10%)
Failure
3/21 (14%)
14/21 (67%)
Perfect matches: simple single-register cases. The LLM succeeded when game logic used triv- ial register assignments. In Submarine, both functions were correctly identified because the as- sembly showed clear v7 += OxC5 for scoring and if v8 Cx00 for game-over checks- unambiguous patterns that the LLM easily recognized.
Partial success: correct registers, wrong logic. In Pong, the LLM identified V[14] as the score register but missed Binary-Coded Decimal (BCD) encoding where 0x23 represents player=2, oppo- nent=3. The ground truth decodes this as (V [V[4]] /10) (V [14] % $10) while the LLM simply returned V V[14]. Similarly, in Shooting Stars, the LLM missed overflow protection: the ground truth returns 0 when V[0] exceeds 128, but the LLM returned raw V[0]. These cases show the LLM can identify score registers but struggles with encoding schemes and boundary conditions.
Wrong logic: register confusion within context. In Squash, the LLM confused adjacent regis- ters: correctly identified V[11] for scoring but used V[12] for termination when V[11] actually serves dual purpose (score and lives). This "off-by-one" pattern appeared repeatedly (Missile: V[5] vs V[6], Tank: V[6] vs V[13]). Both registers appear in the assembly's game logic, but the LLM incorrectly assumed separate registers for each function rather than recognizing dual-purpose usage.
18

---

Published as a conference paper at ICLR 2026
Complete failures: unrelated register selection. In Blinky, the LLM chose entirely wrong reg- isters (V[9] for score vs ground truth V[6], and V[6]/V[7] for termination vs ground truth V[3]- Analysis of the LLM's reasoning revealed it focused on sprite drawing operations involving these registers rather than actual score-tracking logic
Termination logic:
the major challenge Termination functions achieved only 19% perfect accu-
racy, primarily due to multi-condition logic. In SpaceFlight, the game ends when either lives (V[9]) reach zero OR distance (V[12]) exceeds threshold: [V[9]==) (V [12] =12]>=0x3E). The LLM output only v [[14] ==0-a single condition using an unrelated register. This pattern repeated: the LLM rarely captured compound conditions with multiple registers. In Airplane, the LLM detected that V[12] reaches at game end but used wrong register V[10] and completely missed the V[U]] lives condition. Even with explicit prompt instructions to check for OR/AND combinations, the LLM showed strong bias toward single-condition checks.
E.2 DISCUSSION AND IMPLICATIONS
Score functions achieved 57% perfect accuracy, with 71 % correctly identifying the primary register. Success correlated with simple increment patterms (v7 += 0x01) and direct reads. Failures oc- curred with multi-register calculations, encoding schemes like BCD, and negative scores requiring subtraction. Termination functtons struggled 19% accuracy with 67% failures, stemming from multi-condition OR/AND logic, multiple register interactions, and flag-based state machines.
Practical implications: LLMs can assist with reward function extraction for simple game mechan- ics but require human verification for games with multiple termination conditions, complex scoring systems, or multi mugtt-reggiseer state dependencies.
Limitations: Our evaluation measures implementation matching. not semantic equivalence. Alter- native register choices may provide valid rewards for RL training even when differing from human implementations.
F LLM-ASSISTED ENVIRONMENT GENERATION
This appendix details the automated environment generation pipeline using large language models (LLMMs) to create novel CHIP-8 games for reinforcement learning research. We demonstrate the complete process from prompt engineering to code generation across three difficulty levels of Target Shooter game.
F.1 PROMPT ENGINEERING
Our LLM generation pipeline relies on carefully crafted prompts that provide comprehensive CHIP- 8 programming context and specific game requirements. The core prompt structure includes CHIP-8 architectural constraints, Octo assembly language syntax, and reinforcement learning compatibility requirements.
Listing 1: LLM prompt template for CHIP-8 game generation
You are a *xproiessional CHIP-8 (classic version) game developer**. Your task is to **design and implement new CHIP-8 games in Ccto assembly language**. I will provide you with tutorials and references for Octo assembly. You must be rigorous and ensure that your code is syntactically correct, runnable, and follows CHIP-8 conventions**.
cdbumenttaiioonx/cccmmetttttttt
stutoriall>x//utorriaa1
<tutoriaal2+//ttooriallss
<example></example>
19

---

Published as a conference paper at ICLR 2026
The goal is to create **game suitable for reinforcement learning (RL) research**, which means:
The **CCCrC* must be stored in clear and consistent register or memory location.
The **termination condition (gane over) must also be easily extractable (e.g., through specific flag register value).
The game should have exdeterministic rules* and be lightweight enough for training agents.
Here is the description of the game you must implement:
edescription> hetrr///eeeerrrrroooo/oeee//errr
The prompt incorporates several key components:
Role specification: Establishes the LLM as a professional CHIP-8 developer
Technical constraints: Emphasizes syntactic correctness and CHIP-& -8 compliance
RL compatibility: Specifies requirements for tracking and termination detection
Reference material: Includes comprehensive CHIP-8 documentation and examples
Game description: Placeholder for specific game mechanics and objectives
The prompt template includes placeholder tags that are populated with comprehensive CHIP-8
resources: <documentation> contains the official Octo Manual <tutoriall> includes the Beginner's
piithbb,i//Ottt//bcs///aaaaa httml,
<tutorial2s incorporates the Intermediate Guide http:////ccoeeeegnnneeeeeeeeeeae .ht.lileeeemmmmm.......m..m...
Guide httts::/1ohnearreee github wtt.tuut...otto///eeeeoe.o http:///ooooaaoooooo///////////a
htt//etto/docc//ntteeee
html), and <example> provides complete
game implementation httpps://gtthhtiei www.com/J///ooooo. wtt.///aaaaeetttti/aerrr...
naster/src/outlaw/ www.ww/wwwww.......... demonstrate best practices and coding patterns.
F.2
GENERATED TARGET SHOOTER IMPLEMENTATION
Using the prompt template, we generated three progressive difficulty levels of a Target Shooter game. Each level maintains consistent register mappings for score and termination while introducing increasing complexity in target behavior and timing constraints.
F.2.1 LEVEL 1: STATIC TARGETS
The first difficulty level features stationary targets that appear at random locations, focusing on basic aiming and shooting mechanics.
Listing 2: Level Target Shooter Static targets
Target Shooter RL Training Game
ceterministic shooting game designed for reinforcement learning research.
Controls:
move crosshair E to shoot
Score is stored in register v2 (score_reg) Game flag register v3 (gameover_reg)
Game ends after hitting 10 targets.
20

---

Published as a conference paper at ICLR 2026
Sprite data crosshair
0b10000001
0b01011010
0b00100100
0b01011010
0b01011010
0b00100100
0b01011010
0b10000001
targgtt
0b00111100
0b01000010
0b10011001
0b10100101
0b10100101
0b10011001
0b01000010
0b00111100
Register Map Critical for RL extraction
#*#****##****** #*w.:E/***AP*.....
:alias crosshair_x
vO Crosshair X position
:alias crosshair_y
v1
Crosshair position
:alias score_reg
SCORE RL agents read this!
:alias gameover_reg
v3
GAME OVER FLAG (0=playing, l=over)
:alias target_x
v4
Target X position
:alias target_y
v5
Target Y position
:alias target_active
Target active flag
:alias templ
Temporary register
:alias temp2
v8
Temporary register
:alias shet_activve
v9
Shot in progress flag Count of targets hit (max 10)
:alias targets_hit
:alias key_reg
Key input register
:const MAX TARGETS 10
Game ends after 10 targets Target sprite size
:const TARGET SIZE 8
:const CROSSHAIR_
Crosshair sprite size
:const FOINTS_PER_HIT
Points awarded per target hit
**************
Main Game Entry Point
###***************
main # Initialize game state
score_reg
Score starts at
gameover_reg :- Game is not over
targets_hit :- No targets hit yet
active :=
No target active initially
shot active I=
No shot in progress
Initial crosshair position (center) crosshair_x := 28
crosshair_y := 12
clear
# Draw initial UI draw-crosshair
21

---

Published as a conference paper at ICLR 2026
# Main game 1oop 1oop
Check if game should end if hargets_hit = TARCETS then jump game-over
Spawn new target if active
if target_active 0 then spawn-target.
Handle player input hanale-input
Check for hit if shot was fired if shot_active -- then check-hit
Small delay for playability templ := 1
delay := temp1
wait-delay
again
Game Over Handler
*
game-over
gameover_reg
Set game over flag for RL agent
Flash screen indicate game
templ :=
loop
clear
temp2 5
delay ;= temp2
wait-delar
draw-crosshair
if target_active 1 then draw-target:
temp2 := 5
delay := temp2
wait-delay
templ
if temp1 != 3 then
again
Infinite loop game loop
RL agent should detect gameover_reg == again
Input Handling ing
handle-input
Save current position
temp1 := crosshair_x
temp2 == crosshair_y
Movement controls (WASD) use consistent key codes key_reg :- 7 # key (left) if key key then templ += -2
key reg 9 key (right)
22

---

Published as a conference paper at ICLR 2026
if key_rep key then templ +=
key_reg = 5 koy (up)
if key_reg key then temp2 += -2
key_reg 8 key (down)
key_reg key then temp2 += 2
# Bouncary checking
if templ 254 then templ 0
# Left boundary (wrapping check)
if templ
then templ : 56
Right boundary
if temp2 > 254 then temp2 :- 0
Top boundary (wrapping check)
if temp2 >- 24 then temp2. : 24 # Bottom boundary
Check if position changed
templ crosshair_x then jump update-crosshaizz if tomp2 crosshair_y then jump updato-crosshair
# Check for shoot (E key)
key_reg 6 key_reg key then shot_active :-
return
update-crosshair
I Erase old crosshair
i :- crosshair
sprite crosshair_x crosshair_y CROSSHAIR_S12E
# Update position crosshair_x := templ
crosshair_y := temp2
Draw new crosshair sprite crosshair_x crosshair_y CROSSHA IR_ST2R
# Check sheet after movement
key_reg 6 if key reg key then shot active
Target Management АеннААлАААААААА
spawn-target
Generate random position for target
target_x random 0x37 0-55 range
target_y := random 0x17
0-23 range
# knsure minimum distance from edges if target_x then target_x 3
if target_y 2 then target_y :- 3
:= 1
draw-target
draw-target
:= target sprite target_x target_y TARGET_ST24
drax-crosshair crosshair
23

---

Published as a conference paper at ICLR 2026
sprite crossbair crosshair CROSSHAIR_SIAE SIZE
Hit Detection
check-hit
0 # Reset shot flag shot_active
Check if target is active target_active -- then return
# Simple hit detection check if crosshair center is near target center
# Calculate distance
templ == crosshair_x
temp1 += # Crosshair center
temp2 target_x
temp2
Target center
# Check x proximity
if templ temp2 then jump check-x-greater
crosshair left of or at target temp2 - templ
if temp2 > 6 then return A Tec far jump check-y-axis
check-x-greater
# crosshair is right of target temp1 - temp2
if templ 6 then return & Tec far
check-y-axis
# Calculate Y distance
templ crosshair_y
templ
Crosshair center
temp2 := target_y
temp2 += Target center
I Check proximity
if templ temp2 then jump check-y-greater
# crosshair is above or at target temp? - templ
if temp2 6 then return $ Tec far jump register-hit
check-y-greater
# crosshair is below target
templ temp2 templ 6 then return 4 Toc far
register-hit Hit confirmed! Erase target
draw-target
target_active := 0
Update score (for agent)
ecore_reg +- POINTS_PER_HE targets_hir hit += 1
Sounc. feedback
24

---

Published as a conference paper at ICLR 2026
templ
buzzer = templ
Utility Functions
wait-celay
1oop templ delay
if templ !- 0 then
again
F.2.2 LEVEL 2: TIME-LIMITED TARGETS
The second level introduces time pressure by making targets disappear after fixed duration, requir- ing faster decision-making from RL agents.
Listing 3: Level Target Shooter Time-limited targets
Target Shooter RI. Training Game
deterministic shooting game designed for reinforcement learning research.
Controls:
WASD crosshair
E shoot
# Score is stored in register v2 (score_reg) Game over tlag in register v3 (gameover_reg)
Game ends after 10 targets (hit or missed) Targets disappear after =3 seconds if not hit.
Sprite data crosshair
0b10000001
0b01011010
0b00100100
0b01011010
0b01011010
0b00100100
0b01011010
0b10000001
target
0b00111100
0b01000010
0b10011001
0b10100101
0b10100101
0b10011001
0b01000010
0b00111100
25

---

Published as a conference paper at ICLR 2026
Critical for RL extraction CILAATEAAR
Register Map
:alias
crosshair_x
V0
Crosshair X position
:alias cresshair_y
V1
Crosshair Y position
:alias score_reg
SCORE RL agents read this!
:alias gameover_reg
V3
GAME OVER FIAG (0-playing, 1-over)
:alias target_x
V4
Target X position
:alias target_y
V5
Target Y position
:alias target_active
Target active flag
:alias templ
Temporary register
:alias temp2
v8
Temporary register
:alias shot_active
v9
Shot in progress flag Total targets appeared (max 10) Key input register Timer for current target Count of missed targets
:alias targets targets_total total
:alias key_reg
:alias target_ timer
VC
:alias missed_targets
vd
:const TAX_TTS 10
Game ends after 10 targets total Target sprite size
:const TARGET_SI2E
const CROSSHAIR_SIPE
Cresshair sprite size Points awarded per target hit
:onstt
POINTS_PERE ______R__TTTT____________
:onnst TARGET_TINEOUT 60
Franes before target disappears (-3 sea at
201ps)
Main Game Entry Point еин.нлаааАлннееес
main # Initialize game state
scorc_reg := 0
# Score starts at
gameover_reg
0
# Game is not over
targets_total := 0
# No targets appeared yet
missed_targets :- 0
# No missed targets yet
target_active :- 0
# No target active initially
shot_active 0
# No shot in progress
target_timer
Timer
# Initial crosshair position (center)
crosshair_x := 28
crosshair_y 12
clear
# Draw initial UI draw-crosshair
# Main game 1oop loop
Check if game should end (10 total targets) if targets_total
MAX_TARGETS then jump game-over
# Spawn new target if none active if target_active == 0 then spawn-target
Check target timeout
if target_active = 1 then check-target-timcoor
Handle player input handle-input.
Check for hit if shot was fired if shot_active == then check-hit
26

---

Published as a conference paper at ICLR 2026
Small delay for playability templ
delay : temp1
wait-delay
again
Target Timeout Check ###
check-target-timeott
# Decrement timer target_tinme timer_ += -1
Check timer expired if target_timer != 0 then return
# Target timed out count miss draw-target Erase target target_active :- missed_targets +-
Brief sound to indicate miss templ := buzzer :- templ
**
Game Over Handler
game-over
gameover_reg
Set game ever flag agent
# Plash screen to indi cate game ever templ
loop
clear
temp2 := 5
delay := temp2
wait-delay
draw-crosshair if target_actinne == 1 then trawetarget.
temp2 : 5
delay :- temp2 wait-delay
templ += 1
if templ !- 3 then again
# Infinite loop game is over loop
RL agent should detect gameover_ reg again
Input Handling
#********
hancle-input
# Save current position templ crosshair
27

---

Published as a conference paper at ICLR 2026
temp2 crosshair
# Movement controls (WASD) use consistent key codes key_reg :- 7 # koy (left)
if key_reg key then templ += -2
key_reg :- 9 # key (right)
if key_rey key then templ += 2
key_rey reg 5 if key_reg key then temp2
key (up)
key_reg :- 8 H S key (down)
if key key then temp2 += 2
Boundary checking
if templ >= 254 then temp1 = 0
# Left boundary (wrapping check)
if temp1 >= 56 then temp1 == 56
# Right boundary
if temp2 >= 254 then temp2 0
# Top boundary (wrapping check)
if temp2
then temp2 24
Bottom boundary
$ Check if position changed
if templ != crosshair_x then jump update-crosshair
if temp2 == crosshair_y then jump update-crosshaizz
# Check for shoot (E key) key_reg :- 6
if key_reg key then shot_active
return
update-crosshair
# Erase cld crosshair
:= crosshair sprite crosshair_x crosshair_y CROSSHA IR_ST2R
Update position erosshair_x :- templ
crosshair_y == temp2
Draw new crosshair sprite cresshair_x crosshair_y CROSSHAIR_SI2E
# Check shoot after movement
key_reg : 6
if key_reg key then shot_active
Target Management #**.:АаААААААееее
spawn-target
# Generate random position for target
target_x x random 0x37 # 0-55 range
target_y random 0x17
0-23 range
# Ensure minimum distance from edges if target_x <= 2 then target_x := 3 if target_y <= 2 then target_y :=
carget_active :-
harget_times := TIMEOUT Set timeout timer
targets_total += 1
Increment total targets count
draw-target
28

---

Published as a conference paper at ICLR 2026
draw-target
1 := target
sprite target_x target_y TARGET_SI2A
draw-crosshair
i := crosshair
sprite crosshair_x erosshair_y CROSSHAIR_SI22 SIZE
Hit Detection ********************************
check-hit
shot_active ;= 0 # Reset shot flag
Check if target is active target_active
then return
Simple hit detection check if crosshair center is near target center
Calculate distance
templ :- crosshair_x
templ +- # Crosshair center
temp2 target_x Target center
temp2
# Check X proximity
if temp1 > temp2 then jump check-x-greater
# crosshair left of or at target temp2 templ
if temp2 > 6 then return # Tec far jump check-y-axis
check-x-greater
# crosshair is right of target templ - temp2
if templ 6 then return & Tec far
check-y-axis
# Calculate Y distance
templ : cresshair_y
templ
Crosshair center
temp2 target_y
temp2 += Target center
# Check proxinity
if templ temp2 then jump checker-
# crosshair is above or at target temp2 -= templ
if temp2 6 then return $ Tec far jump register-hit
check-y-greater
# crosshair is below target
templ temp2 templ 6 then return $ Tee far
register-hit
Hit contirmed!
29

---

Published as a conference paper at ICLR 2026
# Erase target
draw-target
target_active ;=
target_timer = 0 # Clear timer
Update score (for agent) score_reg +- POINTS_PER
# Sounc teedback templ
buzzer :- templ
#* Utility Eunctions
wait-delay 1oop templ delay
if templ !- 0 then
again
F.2.3 LEVEL 3: MOVING TARGETS WITH TIME CONSTRAINTS
The most challenging level combines target movement with time limits, requiring predictive aiming and rapid response times.
Listing 4: Level Target Shooter Moving targets with time constraints
#
Target Shooter Level 3 RL Training Game
aeterministic shooting game designed for reinforcement Learning research.
Controls:
WASD move crosshair
E shoot
# Score is stored in register v2 (score_reg) Game over flag in register v3 (gameover_rog)
Features:
Moving targets that bounce off walls
Targets disappear after >3 secends if not hit Game ends after 10 targets (hit missed)
Sprite data crosshair
0b10000001
0b01011010
0b00100100
0b01011010
0b01011010
0500100100
0b01011010
0b10000001
target
30

---

Published as a conference paper at ICLR 2026
0b00111100
0b01000010
0b10011001
0b10100101
0b10100101
0b10011001
0b01000010
0b00111100
Register Map Critical for RL extraction
:alias
crosshair_x
Crosshair position
:alias crosshair_y
v1
Crosshair position
:alias scorc_reg
V2
SCORE RL agents read this!
:alias gamcover_reg
V3
GAME OVER FLAG (0=playing, 1=over)
:alias target_x
Target X position
:alias target_y
Target Y position
:alias target_active
V6
Target active flag
:alias templ
v7
Temporary register
:alias temp2
v8
Temporary register
:alias shot_active
Shot in progress flag
:alias pargets_total
va
Total targets appeared (max 10) Key input register Timer for current target vd
:alias key_reg
vb
:alias target_timer
:alias target_vx
Target X velocity Target Y velocity
:alias target_vy
:const MAX_ TARGETS 10
MAX_TARGETS
Game ends after 10 targets total Target sprite size
:const
TARCET_ SIIE 8
:const
CROSSHAIR_ SIZE POINTS_PER const "AAGRT_TTMROUD 80
Crosshair sprite size
:const
Points awarded per target
Frames before target dasppeasss (-4 sec with
movement.)
Main Game Entry Point
main Initialize game state
scorc_reg := 0
F Score starts at
gameover_reg : 0
# Game is not over
targets_total :- 0
# No targets appeared yet target active initially
target_active :-
shot_active := 0
# No shot in progress
target_time := 0
# Timer at 0
target_vx vx == O
# No initial velocity
target_vy
# No initial velocity
Initial crosshair position (center)
crosshair_x := 28
crosshair_y 12
clear
# Draw initial UI draw-crosshair
# Main game loop loop
Check if game should end (10 total targets)
if targets_tote total MAX TARGETS then jump game-over
31

---

Published as a conference paper at ICLR 2026
Spawn new target if none active if target_active = 0 then spawn-target
Update target position if active if target_active 1 then mave-target
Check target timeout if target_active = 1 then check-target-timeout
Handle player input handle-input
# Check for hit if shot was fired if shot_active = then check-hit
Small delay for playability templ := 1
delay ;= temp1 wait wail-delay
again
**
Target Movement чАКЯЯААААААААА
move-target
Erase target current positicn draw-target
# Update position target_x += target_vx
# Check boundaries and bounce
if target_x >- 250 then jump bcunce-1eft left if target_x >- 56 then jump bounce-right
Hit 1eft edge 4 Hit right edge
check-y-movement
# Update position
target_y += target_vy
Check boundaries and bounce
if target_y >= 250 then jump bounce-top
Hit top edge
if target_y >= 24 then jump bounce-bottan
Hit bottor. edge
finish-move
Draw target at new position draw-target return
bounce-left.
target_x :-
target vx := 1 Reverse to move right jump check-y-novement
bounce-right
target_x := 55
target_vx := 255 # -1 to move left
jump check-y-movement
bounce-top
target_y ==
target_ vy := 1 # Reverse to move down jump finish-move
32

---

Published as a conference paper at ICLR 2026
bounce-botten target_y = 23
target_vy == 255 # -1 to move
jump finish-move
Target Timeout Check #
check-target-timauu
# Decrement timer target_timer +- +=-1 -1
Check if timer expired target_timer != then return
# Target timed out count as miss draw-target Erase target target_active
# Brief scund to indicate miss templ := 1
buzzer templ
Game Over Handler
#*******************************
game-over gamcover_reg
Set game over flag for RL agent
Flash screen indicate game ever
templ :-
loop
clear
temp2 :-
delay := temp2
wait-delan
draw-crosshair
if target_active 1 then draw-target
temp2 := 5
delay :- temp2 wait-delay
templ
if templ != 3 then
again
Infinite loop
game over
loop
RL agent should detect gameover_reg reg ==
again
Input Handling
handle-input
# Save current position templ := crosshair_x x
temp2 == crosshair
33

---

Published as a conference paper at ICLR 2026
Movement controls (WASD) consistent key codes key_reg :- key (left)
if key_reg key then templ += -2
key_reg : 9 # key (right) if key_reg key then templ
key_reg == 5 # key (up)
if key_reg key then temp2 += -2
key_reg :- key_reg key then temp2 +- 2
key (down)
# Boundary checking
if templ 254 then templ 0
Left boundary (wrapping check)
templ >= then templ == 56
# Right boundary
if tomp2 >= 254 then temp2 == 0
d Top boundary (wrapping chock)
if tcmp2 >= 24 then temp2 == 24 # Bottom boundary
Check if position changed
if templ crosshair_x then jump update-crosshaiz
if temp? !- crosshair_y then jump update-crosshal
Check for shoot key) key _eggeegg_ := 6
if key_reg key then shot_active :
return
update-crosshair
# Erase old crosshair
i : crosshair
sprite crosshait_x crosshair_y CROSSHAIR_SI2E
# Update position crosshair_x :- templ
crosshair_y temp2
Drax new crosshair
sprite crosshair x crosshair CROSSHAIR_ SIZE
I Check shoot after movement
key_reg 6 if key_reg key then shot_active :
**
Target Management ###
spawn-target.
Generate random position target
target_x :- random 0x37 0-55 range
target_ := random 0x17
# 0-23 range
Ensure minimum distance from edges if target_x
then target_x 3 if target_y <= 2 then target_y :=
# Generate random velocity (-1, 0, 1 for each axis) target_vx random 0x03 target_vx then target_vx vx :- 255 # Convert to -1
target_vy := random 0x03
if target_vy vy == then target_ vy 255 # Convert to -1
34

---

Published as a conference paper at ICLR 2026
Ensure target moving (not beth velocities zero) if target_vx = then jump ensure-movement jump finish-spawn
ensure-movemen
target_vy then target_vy vy :-
finish-spawn
tariet_active
target_timer "ARGET_TIMEOO
timeout timer
targets_total +- draw-target
Increment total targets count
draw-target
i := target
sprite target_x target_y TARGET_SI2E
draw-crosshair
:- cresshair
sprite crosshair_x crosshair CROSSHAIR_SI22
Hit Detection
check-hit
thot_active := 0 # Reset shot flag
# Check if target is active
if target_active
then return
# Simple hit detection - check if crosshair center is near ta target center
Calculate distance
templ := crosshair
templ += Crosshair center
temp2 == target_x
temp2 +=
Target center
# Check x proximity
if templ > temp2 then jump check-x-greate
crosshair left of target temp2 templ
if temp2 6 then return & Too far jump check-y-axis
chect-x-greater
I crosshair is right of target templ - temp2 if templ 6 then return # Tee far
check-y-axis f Calculate y distance
templ == crosshair_y
templ += # Crosshair center
temp2 target_y Target center
temp2 +-
# Check Y proxinity
if templ temp2 then jump check-y-greater
35

---

Published as a conference paper at ICLR 2026
crosshair above or target temp2 - templ
if temp2 > 6 then return $ Toc far jump register-hit
check-y-greater
# crosshair is below target
templ - temp2
if templ 6 then return $ Tec far
register-hit
# Hit confirmed!
# Erase target
draw-target
carget_actise := targot_timer := # clear timer
# Update score (for agent) score_reg SEINTS_PER_HTT
# Sound feedback
templ :=
buzzer templ
Utility Functions
#
wait-delay loop
templ delay
if templ != then again
F.2.4 ENVIRONMENT INTEGRATION AND WRAPPER IMPLEMENTATION
Once the LLM generates the CHIP-8 assembly code for each difficulty level, the games require in- tegration with OCTAX's reinforcement learning interface. The environment wrapper extracts reward signals and termination conditions from the consistent register mapping established during code generation.
The Target Shooter implementation demonstrates the integration between LLM-generated content and the OCTAX framework. Each level maintains identical register assignments patibility across the difficulty progression, enabling curriculum learning experiments without code modifications.
com-
Listing 5: Target Shooter environment wrapper implementation
from octax import EmulatorState
def cstre_f EmulatorState) -> float:
***
Extract score from register V(2]
Score increments by 1 for each successful hit Range: 0-10 points
"**
return state.v(2)
def terminated EmulatorState) -> bool:
Check game termination flag in register v[3]
36

---

Published as a conference paper at ICLR 2026
Game ends after 10 total targets (hit missed in levels 2-3) нMм
return state.V[3] == 1
CHIP-8 key mapping for controls W=5 (up), A=7 (left), 5-8 (down), D=9 (right), E-6 (shoot) action_set [5, 7, 8, 9, 6]
metadata =
"titie":
"authors"; heglly..ii--Geeerrteddd_...ii "description"; www.aaattteoooorrreeeeeeeffeooon
"roms":
sorrget_shhoterttt levell";e
"file": "target shocter levettrr__vell.cch"
"description": "Static,targets, B Basicc, aimigg, skills"
"target_shoattr_oovll222 {
"file": "target saarisst_shhottr_eeeeee ch8"
"description": "Time- [
"Mile": strrgee_sshttee_eeeeeeeee "description": wtwiiintiime-immmeeddeeeeeeeddee b sharee__shooer__eeel1
The consistent register mapping across all three levels enables direct comparison of agent perfor- mance and facilitates automated curriculum progression. Register V[2] consistently stores the score for reward calculation, while V[3] serves the binary termination flag. The five-action control scheme (WASD movement plus shoot) provides sufficient complexity for interesting policies while remaining tractable for systematic analysis.
37


Reviews:
[{"review_id": "osjVF486zo", "reviewer": "Reviewer_6XnA", "summary": "In this paper, the authors present a JAX implementation, known as OCTAX, of the CHIP-8 platform amenable to GPU acceleration. Octax converts CHIP-8 ROMs into RL environments that can take advantage of vectorized JAX functions. Octax provides a wide selection of games of interest to RL researchers spanning a number of different genres and difficulty levels. The performance of Octax is shown to be substantially higher than traditional methods for environment simulation using parallel instances of games with EnvPool. The authors note that some CHIP-8 ROMs had difficult reward functions to implement and understand, but this process was made easier with the help of LLMs to study the machine code. By reversing this process, they were able to train LLMs to produce new games that could be added to the RL training environment.", "questions": "- How many attempts were required to generate Target Shooter using Claude? Was the success rate?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The performance improvement of Octax over vectorized environments is impressive and will decrease the training time for many researchers using JAX to engage in RL experiments. With a shorter training time on simpler games, many new experiments to understand the training dynamics and tradeoffs between the large number of RL hyperparameters may be investigated more effectively.\n- On top of improved acceleration, the authors demonstrate the ability of LLMs to aid in the understanding of existing ROMs and the generation of new environments based on properly constructed prompts. I believe this may be a simplified and interesting sandbox to explore LLM coding in a way that is faster than traditional code generation procedures.\n- Providing this code as a testbed will facilitate the construction of more complex environments, such as Super-CHIP8, to add further complexity and interest.", "weaknesses": "- The utility of arcade environments for SOTA RL has passed, so the real value is to use the environment to accelerate understanding of RL training dynamics and other metrics. Although the added challenge of generating new environments may mitigate this point.\n- Supporting yet another training environment may have a limited impact and lower the contributions.\n- There's no discussion regarding the success rate of generating ROMs using LLMs. I assume this is for the sake of space and to focus the discussion of the paper on the acceleration of arcade environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CV7R75C55G", "reviewer": "Reviewer_iK9J", "summary": "The paper presents OCTAX, a JAX-native, vectorized CHIP-8 emulator and RL environment suite that runs many GPU-accelerated game instances. OCTAX exposes (21) CHIP-8 titles as Gym/Gymnax-compatible environments, includes wrappers to extract score and termination signals, reports high throughput (claims up to hundreds of thousands env steps/s / millions of frames/s), provides PPO training experiments across multiple games, and demonstrates an LLM-assisted pipeline to generate CHIP-8 games and corresponding reward/termination wrappers. An anonymized code repository is provided.", "questions": "Re Emulator fidelity: Do you have systematic validation tests? E.g. for something like Pong, can you provide instruction-by-instruction equivalence tests, frame-by-frame rendering comparisons, and unit tests across representative ROMs against a trusted CHIP-8 interpreter.\n\nBaselines and fairness: If you retain EnvPool/ALE comparisons, justify differences in environment complexity or compare EnvPool on a matched low-resolution workload. Additionally, consider adding DQN-style results.\n\nProfiling and memory breakdown: Provide GPU profiler traces (kernel times, GPU utilization) and a detailed memory breakdown per environment (state arrays, framebuffers, intermediate buffers, compiled executable memory). Explain how you measured ~2 MB/env and the causes of scaling limits.\n\nPPO protocol: Were hyperparameters tuned per game or held fixed? Provide ablations showing sensitivity to frame-skip and observation stacking and report whether reported learning curves use per-game tuning.\n\nLLM pipeline evaluation: What fraction of LLM-generated ROMs compiled and ran without manual edits? How often did generated score_fn/terminated_fn require human correction? Provide statistics and typical failure cases and describe any automated validation used.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses a clear practical gap in the JAX ecosystem: image-based, GPU-native environments for RL. It thereby comes with significant engineering effort: a fully vectorized CHIP-8 emulator in JAX that can execute many parallel instances and integrates with standard RL training loops (Gym/Gymnax). Furthermore, this work directly empowers the community to work on more complex JAX-accelerated environments, which wasn't directly possible beyond Craftax before.\n\nIt introduces a broad catalog of 21 CHIP-8 games across multiple genres, useful for rapid prototyping and curriculum experiments. The reported throughput and scaling is promising, and if validated could substantially reduce wall-clock time for many experiments allowing for low resource algorithmic developments.\n\nFinally, the authors for provide evidence for a novel auxiliary idea: an LLM-assisted pipeline to generate lightweight CHIP-8 games and wrappers, enabling rapid environment prototyping. I find this especially exciting since it may enable open-ended training and novel autocurricula approaches.", "weaknesses": "Emulation fidelity claims ('perfect fidelity') may be partially unsubstantiated: no instruction-trace equivalence, frame-by-frame comparison, or unit tests against a trusted CHIP-8 interpreter are reported. Most of the functional correctness assertions come from the successful training of agents.\n\nArguably, there is no really fair CPU baseline for CHIP-8 measured on the same machine, and comparisons to EnvPool/ALE conflate environment complexity differences. I understand that this is not trivial to accomplish, but I think it would make sense to maybe show runtimes of CPU/GPU-enabled environments just for contextualization. Even if they are not the same as the ones implemented in OCTAX.\n\nThe GPU memory/accounting claims (~2 MB per environment) lack a principled breakdown and appear implausible without explanation of XLA/JAX buffer and compiled executable overheads.\n\nThe LLM-assisted game generation is presented as a single case study with no statistics on compile/run success rates, human edit frequency, or failure modes.\n\nSome broader claims (energy savings, enabling small labs) are asserted without corresponding measured energy or cost data. This could and should be better substantiated.\n\nAll experiments only consider PPO as the single RL algorithm tested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LcpS3rVCNE", "reviewer": "Reviewer_VWtY", "summary": "This paper presents Octax, a high-performance arcade-style benchmark for Reinforcement Learning in Jax. The paper utilizes end-to-end GPU training, allowing extremely high throughput. Primarily, this work looks to provide a viable alternative to the long-standing Atari benchmark, but using considerably less computational resources. Furthermore, the paper presents a way to quickly use LLMs to generate new environments, on top of the current set of games presented.", "questions": "- Is this benchmark only usable to those who have algorithms written in JAX?\n- Does this benchmark have a set of rigorous set of specifications (number of frames, number of games, way to compare different overall performance) to ensure that researchers can easily compare their work?\n- How challenging are the existing environments in this benchmark? Can you benchmark some different algorithms and provide code for these in the repository?\n- Do all environments have a source of stochasticity? Or can they be solved by brute-force style algorithms?\n\nAlso, it appears Line 79 has a mistake. While I'm strongly in favor of benchmarks which make research easier, I feel this work still has a long way to go before it could replace something like Atari, thus I cannot yet recommend acceptance.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper aims to achieve the important goal of reducing the cost of running Reinforcement Learning experiments, clearly succeeding, achieving an impressive 1.4 million frames per second on consumer-grade hardware. The code already being open-source is also a nice positive. LLM-generated environments using the provided prompts provide a nice way for developers to test environments with specific properties if they wish.", "weaknesses": "While I appreciate the attempt to reduce the computational burden of the Atari, I feel that the authors missed many key reasons why the Atari benchmark became so popular, and many of the features that still make it so popular today. In its current form, I don’t believe this work is capable of replacing the Atari benchmark without major reforms for a few key reasons:\n\n- **A clear, fixed way to evaluate** - One reason Atari is useful is that there is a relatively clear and fixed protocol for evaluation [1]. This means researchers know what environments they need to run, the settings they need to use, and know that their work will be comparable to others. Furthermore, there are precise guidelines on how scores should be reported [2]. The paper would benefit from being very clear about what environments are included in the benchmark, how many frames/timesteps algorithms should be trained for, and how results should be reported, with examples. Currently Figure 3 does not meet the same standards as many papers which use the Atari benchmark.\n- **Aggregate Scores** - Atari is useful as a benchmark, as the performance of an algorithm can be boiled down to a single number or single graph. In Atari, this has become human-normalized IQM performance with 95\\% confidence intervals, which provides a score for the given algorithm. This prevents games with different score magnitudes from dominating the final score. Currently, this appears to be missing, despite games having scores of different magnitudes.\n- **No-op Starts, Sticky Actions and Random Actions** - In the Atari benchmark, features are provided that prevent the agent from exploiting determinism (preventing brute-force approaches). Specifically, no-op starts randomly uses up to 30 no-op steps at the start of episodes; sticky actions give a 25% chance for actions to be repeated, and agents are forced to take random actions 1% of the time. While Octax appears to have some environments that have randomness, it's unclear whether brute-force approaches could work in some environments. \n- **JAX only** - While JAX has its advantages in speed, for a benchmark, I see it as a significant weakness if this benchmark is exclusive only to JAX users. A significant portion of RL researchers and users use other frameworks, such as PyTorch, which appear to be excluded from using this benchmark.\n- **Historic data** - One major advantage of Atari is that a huge number of algorithms have been evaluated, making it useful to compare against new algorithms, while Octax only has PPO. Please consider adding more algorithms such as DQN [3], SAC [4], PQN [5], and also more state-of-the-art algorithms would be appreciated.\n- **Game categorizations** - Currently, in Octax games are categorized into groups such as Puzzle, Action, etc. I think it would be more beneficial for the research community if environments were grouped by the aspects of the algorithm they challenge. In Atari, there are well known groups such as hard exploration (Montezuma’s Revenge), long term-credit assignment (Skiing) and many more. For a good example of this, please look at BSuite [6]. While Table 1 somewhat provides this, I still don’t think it is up to the standards of recent work. Furthermore, I’d appreciate a more detailed description of the tasks, including the frequency and magnitude of rewards. It is currently unclear if  Octax provides environments that are as challenging as Atari - for example, even after years of research, environments such as Pitfall and Montezuma’s Revenge are still extremely challenging.\n\n\n[1] Machado, Marlos C., et al. \"Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents.\" Journal of Artificial Intelligence Research 61 (2018): 523-562.\n\n[2] Agarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\" Advances in neural information processing systems 34 (2021): 29304-29320.\n\n[3] Mnih, Volodymyr, et al. \"Playing atari with deep reinforcement learning.\" arXiv preprint arXiv:1312.5602 (2013).\n\n[4] Haarnoja, Tuomas, et al. \"Soft actor-critic algorithms and applications.\" arXiv preprint arXiv:1812.05905 (2018).\n\n[5] Gallici, Matteo, et al. \"Simplifying Deep Temporal Difference Learning.\" The Thirteenth International Conference on Learning Representations.\n\n[6] Osband, Ian, et al. \"Behaviour Suite for Reinforcement Learning.\" International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pWfpa8yfvK", "reviewer": "Reviewer_E8PB", "summary": "This paper introduces OCTAX, a suite of classic arcade-style RL environments implemented as a fully vectorized CHIP-8 emulator in JAX. The goal is to provide an Atari-like benchmark that (a) runs end-to-end on GPU, (b) scales to thousands of parallel environments, and (c) maintains authentic game dynamics. The authors show throughput up to ~350k steps/sec (≈1.4M frames/sec) with 8,192 parallel environments on a single consumer GPU (RTX 3090), which they report is roughly 14× faster than CPU-bound baselines like EnvPool on Atari Pong. \n\nThe benchmark covers 20+ CHIP-8 games across genres (puzzle, action, navigation, resource management, etc.), and the paper provides PPO training curves for 16 of them. These curves exhibit meaningful diversity in difficulty and learning dynamics (fast plateau vs. gradual improvement vs. failure-to-learn cases like Tetris/Worm), suggesting this is not just “16 clones of Breakout,” but a range of cognitive burdens. \n\nFinally, the paper explores automatic environment generation: they use an LLM to synthesize new CHIP-8 games (e.g. a “Target Shooter” with increasing difficulty levels), plus score/termination logic, and then successfully train PPO on these generated tasks with clean difficulty gradients. This is pitched as a path toward scalable curriculum / rapid task creation. \n\nOverall, the paper proposes OCTAX as “the missing GPU-native Atari for JAX RL,” arguing that this makes statistically reliable RL experiments cheaper and more reproducible for smaller labs.", "questions": "**Generalization / external validity.**\nDo you have any evidence that OCTAX performance correlates with Atari performance, Procgen performance, or other visual RL benchmarks? Even a tiny pilot (e.g. rank-correlation of seed-averaged scores across algorithms) would strengthen the “Atari alternative” positioning. \n\n**EnvPool/CuLE comparison.**\nCan you report CuLE numbers (GPU Atari) on the same GPU you used for OCTAX, or at least discuss why that wasn’t feasible? That would make the “14× faster” claim feel less like apples vs oranges. Also, have you tried running OCTAX on CPU only to show the CPU→GPU delta cleanly? \n\n**Automatic reward/termination inference.**\nFor a new arbitrary CHIP-8 ROM (unseen by you), how robust is score/termination extraction? Do you have quantitative success rates for the static+dynamic heuristics or the LLM-generated wrappers? For RL practitioners, “plug in a ROM and it just works” is a killer feature—please convince us it's realistic.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "**S1. Clear problem, clear value to the community.**\nThe paper correctly identifies a real bottleneck in RL: environments are still mostly CPU-bound, even though policy learning is GPU-bound. This prevents large-scale sweeps, high seed counts, and rigorous statistics, especially for vision-based tasks like Atari. OCTAX directly targets that gap with an image-based, arcade-style benchmark that runs entirely in JAX on GPU. \n\n**S2. Strong engineering contribution.**\nThe authors implement a full CHIP-8 fetch/decode/execute loop in JAX using functional state passing, vectorized dispatch (lax.switch), and batched framebuffers, and wrap it as Gym/Gymnax-style RL envs with reward extraction, termination logic, action pruning, frame stacking, etc. They argue that fidelity to original game mechanics is preserved. \nThis is nontrivial and, as far as I know, not previously available in the JAX ecosystem for Atari-like games.\n\n**S3. Throughput + scaling results are compelling.**\nReported numbers (350k steps/sec, linear-ish scaling up to 8k envs on a single 3090; 14× over EnvPool Pong at high parallelism) are impressive and very relevant to labs that don't have multi-node clusters. The memory footprint (2 MB per env, linear in env count) is also concretely discussed. \nThis is an unusually thorough systems evaluation for an RL benchmark paper.\n\n**S4. Diversity of tasks and empirical characterization.**\nThey don’t just dump environments; they also measure PPO learning curves across 16 games and group them into qualitative regimes (fast plateau vs. gradual improvement vs. hard/sparse). This suggests these games could be used for algorithm diagnostics, curriculum, etc.", "weaknesses": "**W1. How “Atari-like” is it, really? (External validity.)**\nCHIP-8 games are dramatically simpler than Atari 2600 in terms of resolution (64×32 monochrome), action semantics, and world complexity. The paper asserts “Atari-like cognitive demands,” but the qualitative gap (e.g., long-horizon exploration, partial observability, rich object interactions) is not deeply quantified. We see PPO struggling on Tetris/Worm, but I'd like more systematic evidence that success on OCTAX predicts anything on Atari, NetHack, Procgen, etc. \nRight now OCTAX looks great for intra-JAX benchmarking and ablations, but it's less clear if it’s meaningful as a drop-in Atari replacement for algorithm claims.\n\n**W2. Reward shaping / termination extraction feels artisanal.**\nFor each game, they manually or semi-automatically infer score registers, life counters, game-over flags, menu skips, etc. (e.g. “Brix stores score in V5, Pong encodes BCD in V14, etc.”). This is powerful, but also fragile and slightly underspecified. \nIf a new contributor adds a weird ROM, how confident are we that OCTAX's automatic heuristics won't silently produce a broken reward function (e.g. rewarding losing health)? The paper mentions static+dynamic analysis plus some LLM help, but I’d like stronger guarantees or validation.\n\n**W3. No baselines beyond PPO.**\nAll learning curves are PPO only. There's no DQN-style baseline, no lightweight world-model baseline, no offline RL baseline, etc. PPO is reasonable (and popular), but I'd like to know if these environments produce meaningful rankings across algorithms, or whether they’re PPO-biased (e.g. continuous control style tuning, frame stacking assumptions, frame-skip assumptions). \nRight now we mainly learn “PPO can learn some of them.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)