MarshCurrant/agilex_pour_ft_idm

AgileX pour checkpoint with four-task IDM adaptation

This repository contains the complete native LingBot-VA transformer after action-specific supervised fine-tuning of the pour checkpoint. Despite the repository name, adaptation used four tasks: pour, pick, opendraw, and stack3bowls. It is not a LoRA adapter and does not require overlaying weights on the original transformer.

The exported checkpoint is step 950, selected by the lowest fixed validation action loss, 0.0159488310, in a 1,000-step training run. It is the checkpoint used in our subsequent paired IDM and imagined-pouring evaluations. This is not a real-robot success-rate claim.

Required files and format

transformer/
  config.json
  diffusion_pytorch_model.safetensors
effective_policy_config.json
normalization.json
provenance.json
training/
  recipe.yaml
  split.json
  trainable_parameters.json
  freeze_verification.json
release_manifest.json

Use the included normalization.json, not the original pour-only statistics. The 14-channel q01/q99 values are also mapped into the 30-channel model space in effective_policy_config.json. Do not recompute statistics from deployment or test data. Camera ordering and action channel ordering must remain unchanged.

This is a native transformer export, not a converted LeRobot PreTrainedPolicy.from_pretrained package. It does not contain LeRobot processors, the frozen VAE, T5 encoder, or tokenizer. Use a compatible LingBot-VA implementation and the same frozen components as the base policy. effective_policy_config.json is the effective inference configuration, not a replacement for transformer/config.json or the training recipe.

Download:

from huggingface_hub import snapshot_download

model_root = snapshot_download(
    repo_id="MarshCurrant/agilex_pour_ft_idm",
    local_dir="checkpoints/agilex_pour_ft_idm",
    # Set revision to the desired commit SHA for reproducibility.
)

Point the native loader's transformer_path to model_root + "/transformer". Load effective_policy_config.json and supply your local frozen-component root as wan22_pretrained_model_name_or_path / wan_pretrained_path, as required by your loader. The published wan_pretrained_path is the relative placeholder frozen; no machine-specific path is required. Apply the included action normalization and channel mapping before executing any generated commands.

What was trained

• Base: Ame233/lingbot-va-agilex, revision b859abc4a859b6da9b2b95eaa343da5ade80b671, checkpoint-1000/transformer.
• Transformer parameters: 5,088,872,670 total; 67,058,718 trainable.
• Trained: action_embedder, action_proj_out, and action time-conditioning modules. The unused action-condition text embedder is excluded.
• Frozen: shared transformer blocks and video-specific modules, plus VAE/T5.
• Ten action-specific parameter tensors changed; the remaining 829 tensors were unchanged in the checkpoint audit. Frozen parameter hashes match before and after adaptation.

This architecture shares transformer blocks across modalities. "IDM FT" here means action-specific adaptation, not unfreezing a separate action transformer. Changed generated actions can affect later imagined video through action-token history even though video parameters are frozen.

Data and training

Setting	Value
Episodes	pour 100, pick 100, opendraw 100, stack3bowls 200
Adaptation split	450 train / 50 validation, episode-level, seed 20261105
Normalization	Exact frame-weighted q01/q99 over all 500 episodes, 280,532 command rows
Training window	Four chunks, eight video latents, 96 action slots
Sampling	Uniform task, then episode, then valid window
Loss	Masked action flow-matching velocity MSE with native timestep weight
Trainable mode	Non-LoRA action-specific parameters only
Optimizer	AdamW, betas 0.9/0.95, weight decay 1e-4
Peak LR	1e-5; 50-step warmup, cosine decay to 0.1 of peak
Batch	One GPU, microbatch 1, accumulation 8, global batch 8 windows
Gradient clip	1.0
Training length	1,000 optimizer steps; release is best step 950
Validation	16 fixed held-out windows, every 50 steps

Normalization intentionally includes the validation episodes, as specified for this experiment. Thus validation is held out from gradient updates, but not from normalization-statistic estimation. Pour validation episodes were also available during the original base checkpoint's training. Use new collection sessions for an independent deployment evaluation.

Evaluated inference settings

• Front, left-wrist, right-wrist images, in that order; width concatenation, each view resized to 256 x 256; no horizontal flip.
• 14 recorded command channels mapped to model channels [14,15,16,17,18,19,28,21,22,23,24,25,26,29].
• Video denoising: 10 FlowMatch Euler steps, guidance 1.0, shift 5.0.
• Action denoising: 5 Euler steps, guidance 1.0, shift 1.0.
• Attention window: 8 latent positions; BF16 inference.
• Two video latents per chunk, 12 action slots per latent; native initial action padding is retained. The first chunk has 12 executable commands; subsequent chunks have 24.

Source actions are 30 Hz. Do not equate MP4 frames with action slots or change camera/action conventions without an explicit deployment validation.

Evidence and limitations

The fixed validation loss supports improved action reconstruction under real video conditioning. It is not a measure of closed-loop success. Earlier test80 per-task IDM tables used inconsistent upstream task labels and are not published here as clean generalization results; corrected-label evaluation must use the repaired dataset and rerun text conditioning.

On 20 visually audited test80 pour initial conditions, paired autoregressive inspection found visible transfer in 19/20 base and 19/20 adapted trajectories, with the same non-transfer case. These are assistant visual observations, not environment labels or hardware execution. No real-robot improvement is claimed.

Check joint limits, gripper conventions, action scaling and emergency-stop behavior before hardware use. Inspect commands offline before enabling motion.

Transformer SHA256: 36ce1ce8ec1dd88c0c3785f79aa77fb7a4de342ddba0a5d4a86eb2a85f10d533

Normalization SHA256: cd379a4d4b8ae5fe119489ec716b9589c9d2fc102861ede8fc924127e15cfd33

The manifest records file hashes and source revisions. Training recipe paths are relocatable examples; optimizer state, credentials, raw videos and dataset files are deliberately not included in this inference release.