grid-policy-mlp-self-improved

Can a model learn from the exact situations where it fails?

This model is a small, playable experiment in failure-driven learning. We start with an existing PPO-trained Breakout policy, record its failures, search the same simulator for reachable alternative paths, and train the policy to make the choices that produce those better outcomes. Every proposed checkpoint must then prove itself through actual gameplay before it can replace the current model.

The result is a real change to the model's weights. Across the game's 96 legal starting configurations, mean return increased from 8.895833 to 16.750000, with 56 improved starts, zero return regressions, and 40 ties. Misses within the fixed 200-step horizon fell from 56 to 6.

The game is not solved: the final model has six misses, 90 time limits, and zero board clears. These results describe a repeatedly used development population, not an unseen test set. The experiment demonstrates measurable improvement under this particular protocol.

Play it, inspect it, reproduce it

Resource	What you can see
Live paired-game simulator	Run the original and gradient-trained policies on the same starting seed. Select the gradient-trained model and run paired games.
W&B experiment report	Accepted-policy curves, optimizer metrics, and the training-versus-surgery comparison.
Training dashboard	Hosted pipeline progress, results, and curves.
Trajectory dataset	Episode traces, diagnoses, candidate evidence, retained repairs, and deployment verification.
FailureIsAllYouNeed source	The exact executable source used for this run.
Completed Hugging Face Job	The hosted execution of both experimental methods.

What we are testing

The central question is: what reachable state earlier in a failed trajectory would have changed its outcome, and can we teach the model to reach that state without losing its existing performance?

A failure at the end of an episode does not imply the decisive mistake occurred on the last action. A different choice twelve steps earlier may have led to a different ball position, paddle position, collision, and continuation. A useful correction may also solve another starting configuration that encounters a related observation.

This experiment therefore searches from nodes throughout the failed trajectory. It does not restrict diagnosis to a single terminal action. A proposed rescue is a sequence of legal transitions under the existing collision rules. We do not fabricate an advantageous world state or change the game's physics to make a correction succeed.

There is no unique, reliable inverse from a terminal failure back to neural-network weights. Discrete choices and collisions make that relationship difficult. Here, the simulator supplies candidate corrective behavior, gradient training proposes new parameters, and exact replay establishes whether the resulting policy actually works.

“Self-improved” describes this automated feedback pipeline. The policy does not explain its mistakes in language, rewrite its own code, or run a search planner during normal inference. An external training process diagnoses failures and updates the policy.

The starting model and environment

Both experimental arms began from the exact original grid-policy-mlp PPO checkpoint, at revision 28b137f6b049eaf3d9f8fa848edb87b83f46f94a. They did not start from Kimi v4 or an earlier repaired checkpoint. The original PyTorch tensors were converted while preserving their float32 values, and the baseline is included here as initial_baseline.npz.

The policy is a 17,253-parameter multilayer perceptron:

500 observation values → Linear(500, 32) → tanh
                       → Linear(32, 32)  → tanh
                       → Linear(32, 5)   → action logits

The observation is a flattened 10 × 10 grid with five channels: paddle, ball, bricks, walls, and previous-frame paddle/ball positions. Its five action outputs are no-op, up, down, left, and right; in this Breakout engine, up and down have the same physical effect as no-op. Scored gameplay always uses the greedy action choice.

The board starts with 24 bricks. Removing a brick earns one reward; missing the ball incurs a one-point penalty and ends the episode. Clearing every brick also ends the episode. Otherwise, the episode stops after 200 steps.

Randomness occurs at reset. Six possible paddle positions, eight ball columns, and two horizontal directions produce 96 distinct legal initial configurations. The evaluation enumerates that complete reset support. Additional seed numbers can select the same configurations; they do not automatically create novel worlds. There is no exploratory action noise in scored play, and the rules, physics, rewards, and horizon remain fixed.

How the recursive improvement loop works

Original PPO checkpoint
        ↓
Run greedy episodes and save complete trajectories
        ↓
Select a miss and reproduce its exact failure
        ↓
Search reachable alternatives from earlier trajectory nodes
        ↓
Train candidate weights on a simulator-validated corrective path
        ↓
Replay the target, previous repairs, and all 96 starts
        ↓
Pass: retain the checkpoint and repeat from its new behavior
Fail: preserve the evidence and keep the incumbent checkpoint
        ↓
Audit → publish weights → load the pinned release in the simulator

1. Capture the failure, including its history

The loop records the actions and world states leading to a miss, not just the final score. It verifies that the unchanged policy reproduces that failure. These records make it possible to distinguish an actual repair from a change in the starting conditions or evaluation procedure.

2. Search for reachable corrective paths

The diagnostic search considers pre-action nodes across the original failed trajectory, with a bounded search of up to six branching steps, 2,048 state expansions, and 512 continuation rollouts per diagnosis. These are search limits, not a restriction to the final six steps of the episode. Budget limits also mean the search is not exhaustive.

Each diagnostic branch uses legal actions in a cloned simulator. The incumbent policy then supplies the continuation. A promising branch must improve return and survive beyond the original miss, or clear the board earlier. Candidate paths are ranked using factors including the number of action deviations, shared observations with other episodes, and the model's current action preferences. The loop selects up to two alternatives from different anchors.

Forced choices exist only inside this diagnostic process. A model cannot pass the final evaluation merely because an external controller forced it along a good path: its own greedy behavior must reproduce a useful outcome from reset.

3. Train the policy on the correction

The successful arm uses PyTorch backpropagation and AdamW over all three layers and all six weight/bias tensors. It is failure-conditioned supervised fine-tuning of a PPO-trained policy. It does not run a new PPO optimization phase.

The training objective combines:

• Correction cross-entropy: increase the probability of the legal actions along the selected corrective sequence.
• KL preservation: keep the candidate's action probabilities close to the current accepted policy on other observed states.
• Parameter displacement penalty: discourage unnecessary movement away from the current weights.

In compact form, the objective is correction CE + λ × preservation KL + 0.0001 × squared parameter displacement. Exact corrective observations are excluded from the preservation examples so that the optimizer is not asked to copy the old choice and correct it simultaneously.

Training setting	Value
Optimizer	AdamW
Steps per candidate	200
Learning rates tried	0.0001 and 0.0005
KL preservation weights tried	0.1, 1, 10, 100, 1000
Preservation minibatch size	Up to 128
AdamW weight decay	0.0001
Gradient norm clipping	1.0
Parameters eligible for training	All 17,253

Each proposal starts from the current accepted checkpoint. Rejected proposals do not accumulate into it. Once a proposal passes, the next cycle diagnoses the behavior of that newly accepted policy. That feedback is what makes the learning recursive.

4. Require behavioral evidence before keeping the weights

A lower training loss is insufficient. The candidate must pass:

1. Target replay: the corrected continuation succeeds when replayed from the recorded prefix.
2. Full reset replay: the candidate's own greedy choices produce the required improvement from the original starting configuration.
3. Retained-repair replay: previously protected corrections still pass their checks.
4. Complete development census: no starting configuration loses return relative to either the incumbent or original base, and no previously achieved board clear is lost.

Passing these checks protects the measured behaviors; it does not guarantee identical actions or preservation in untested environments. The retained bank also captures incidental repairs of other starting configurations, so one accepted update can contribute multiple correction records.

Results: what changed

Metric	Original PPO	Published trained policy
Mean return over 96 starts	8.895833	16.750000
Misses within 200 steps	56	6
Episodes reaching the time limit	40	90
Board clears	0	0
Starts with improved return versus base	—	56
Starts with lower return versus base	—	0
Starts with equal return versus base	—	40

The gradient arm evaluated 226 candidate trainings over 14 cycles, accepting six checkpoints. It performed 45,200 AdamW steps across separate candidate trainings; that number should not be read as one uninterrupted training trajectory. The final repair bank contains 74 correction records covering 56 distinct starting configurations.

The run stopped with all_current_failure_cases_attempted: the remaining current failure cases had been attempted under this protocol's bounded diagnostic search. That does not establish that those failures are impossible to fix.

The accepted-policy curve and optimizer history answer different questions. Optimizer losses describe individual candidate training. The accepted-policy curve tracks gameplay performance only after a checkpoint passes the behavioral gate. A rejected candidate cannot advance that curve.

What happened to parameter surgery?

The same hosted job ran a separate direct parameter-surgery arm from the same original PPO checkpoint. That arm used numerical W2/W3 correction estimators with the shared diagnosis and replay framework. It evaluated 170 candidates across 10 cycles and accepted none. Its mean return remained 8.895833, and no self-improved surgery model was published.

The weights in this repository come from the successful gradient-training arm. Earlier local surgery experiments began from different checkpoints and used their own protocols; their results should not be combined with this run. This comparison does not establish that gradient training always beats surgery: the arms have different proposal mechanisms and compute costs, and this is one small environment.

What physically changes in the model?

Training changes the numerical weights and biases that map an observation to action logits. A corrective label can raise the preferred action's score, while changes in earlier layers alter the features used to make decisions across many observations. Shared features are why an update may fix multiple starts—and why it may also create regressions that replay must catch.

The network architecture, observation format, and action meanings stay the same. During deployment, the simulator feeds each observation into the published network and executes its greedy action. It does not retrieve a memorized rescue sequence from the dataset or run the diagnostic search on the model's behalf.

The published NumPy and safetensors exports contain the same trained tensors. Their identity is checked independently from the gameplay evaluation.

From training job to playable release

The pipeline ran on Hugging Face Jobs using a Docker image built from a frozen GitHub commit. This small policy used CPU training. After committed cycles, the job uploaded experiment evidence to the trajectory dataset and logged metrics to W&B.

Publication required a passed final replay audit, at least one accepted update, positive mean improvement, and zero measured return regressions. A release manifest then pinned the published model revision. The inference Space loads that revision and checks the policy fingerprint.

After deployment, all 96 episodes served by the live gradient-model simulator matched the saved final trajectories, including actions, world states, rewards, and termination. See the deployment verification receipt. All 13 prelaunch software checks also passed.

Release provenance

Item	Identifier
Run	self-improved-20261104-01
Method	failure-conditioned-gradient-v1
Original model revision	28b137f6b049eaf3d9f8fa848edb87b83f46f94a
Trained weight release revision	49d08edd1d60af4a8b82dfabb74b4c8edc7c3187
Executable source commit	e7845f15f172344d918c9d8085898fad2a9e8768
W&B gradient run	6288c5a5

The trained policy fingerprint is ea45539e3b1d58c5eae723ead9e9204c13a53900e762672949a3cc4b0c6c9e3d. The original policy fingerprint is c818ab03938861603c446365e414296c1e250667e9814f85168d166fd714c6cf. Documentation updates may advance this repository's main revision; the immutable weight-release revision above remains the reproducible reference.

Load the published weights

The easiest way to compare behavior is the live simulator. For local inference, install numpy and huggingface_hub, then load the helper and weights from the same pinned release:

import importlib.util
from huggingface_hub import hf_hub_download

repo = "HarleyCooper/grid-policy-mlp-self-improved"
revision = "49d08edd1d60af4a8b82dfabb74b4c8edc7c3187"

helper = hf_hub_download(repo, "inference.py", revision=revision)
spec = importlib.util.spec_from_file_location("grid_policy_inference", helper)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)

weights = hf_hub_download(repo, "policy_weights.npz", revision=revision)
policy = module.Policy.load(weights)

# observation: the simulator's 500-value float32 observation
# action = policy.act(observation, None, "greedy")

The helper is Python code from this repository; inspect it before executing it. model.safetensors provides the same weights under standard fc1, fc2, and fc3 PyTorch keys for the included model.py architecture. config.json preserves the original architecture metadata; its PPO training label describes the base model's origin. The additional training method for this release is documented here and in summary.json and plan.json.

Included file	Purpose
policy_weights.npz	Trained NumPy inference weights
model.safetensors	Equivalent trained PyTorch-format tensors
inference.py	NumPy policy loader and forward pass
model.py, config.json	Original architecture and observation/action definitions
initial_baseline.npz	Exact starting policy for comparison
summary.json, audit.json, plan.json	Results, verification, and recorded protocol

Limits and the next experiment

The main unresolved result is survival without completion. Ninety starts reach the time limit, but none clear the board. A timeout is not a completed game. This protocol targets misses; time-limit episodes are reported but are not themselves selected as repair targets.

The 96-start census is repeatedly used to select updates, so it is development evidence even though it covers the current reset distribution. It does not prove generalization to different layouts, altered physics, longer horizons, other games, or language models. Equal return also does not mean identical behavior. Bounded diagnostic search can miss useful alternatives, and preservation constraints may reject changes that could help after a more coordinated training update.

The next proposed experiment is to continue from this published checkpoint, preserve its existing repairs, and investigate the six remaining misses with deeper or differently allocated reachable-state search. A separate, explicitly defined completion experiment should examine stalled time-limit trajectories and board-clearing progress. Longer-horizon tests should be reported separately from this fixed 200-step comparison.

Those are research directions, not results claimed for these weights. The useful evidence so far is concrete: simulator-validated corrections became training targets, six weight updates survived the replay gates, and the released model reproduces the measured improvement in a live simulator.