Hugging Face's logo

SimpleJev
/
JevAny-Robotics-4B-LoRA

Robotics
PEFT
Safetensors
jevany
lora
decision-model
pointer
reward-model
judge
Model card Files
xet
 Community

JevAny-Robotics-4B-LoRA (v0.2, SFT)

A 4B Jev-style (System One) judge for robot manipulation: one forward pass returns a probability for every option. It is a robotics continuation of SimpleJev/JevAny-Qwen3.5-4B-LoRA (pointer readout on Qwen/Qwen3.5-4B), fine-tuned on JevAny-Robotics-Train-v0.2 (to be released) β€” 184,942 judgement questions over real and simulated robot images and videos (task progress, 1–5 rollout scores, execution success, failure mode, plan checking, failure localisation, trajectory preference), built only from train-side splits and decontaminated against every test set below.

Highlights (every model below was run by us in the same harness, on the same items; dedicated robot reward models included, see Dedicated robot reward models):

  • No single model wins every benchmark; this one stays near the top on all of them. Specialists win their own format: Robo-Dopamine-GRM-2.0-8B on RoboPulse (90.1% vs 71.1%) and RoboReward-8B on RoboRewardBench (MAE 0.358 vs 0.443).
  • Real robots, human ratings: on RoboRewardBench's RoboArena split (1,000 real-robot rollouts) it has the lowest error of every model we ran, 0.687 (best reward model: Robometer-4B 0.834, paired βˆ’0.147 [βˆ’0.245, βˆ’0.048]; RoboReward-8B 0.968; 27B judges 0.867–0.919). When two policies attempted the same task, it orders them like the human raters 88.4% of the time (805 pairs; OneJev-27B 88.8%, RoboReward-8B 85.3%, Robo-Dopamine 78.3%, Robometer-4B 75.5%).
  • RoboPulse++: AUROC 0.728, statistically tied with Robo-Dopamine (0.698–0.730 across its readouts) and the 27B judges (0.692–0.700); above Robometer-4B (+0.098 [+0.062, +0.131]).
  • Breadth: it answers all 13 question families of JevAny-Robotics-Public (skill_role 0.567, micro accuracy 80.4%, tied with the strongest 27B judge). The reward models can answer 6 of them; on success detection (5 sets) its mean AUROC is 0.872 vs 0.756 for the best reward model (Robometer-4B, paired +0.116 [+0.088, +0.144]); OneJev-27B is higher (0.903).
  • Calibrated probabilities: on RoboRewardBench its probabilities have the lowest NLL of all models we ran, 1.02 vs 1.39 for RoboReward-8B (paired βˆ’0.37 [βˆ’0.52, βˆ’0.22]) at similar exact-score accuracy (62.1% vs 63.5%). Robo-Dopamine and Robometer output no per-option probabilities.
  • Speed: batch 1 on one A100, a median 185 ms per RoboPulse pair with the flash-linear-attention / causal-conv1d kernels (294 ms with the reference kernels used for the tables) vs 316 ms for Robo-Dopamine-8B (vLLM) and 732–838 ms for 27B judges on two GPUs.

Robot benchmarks

Jev-style judges and base VLMs

Model Size RoboPulse acc ↑ RoboPulse AUROC ↑ RoboPulse++ AUROC ↑ RoboPulse++ bal. acc ↑ RoboRewardBench MAE ↓ JevAny-Robotics-Public skill_role ↑
JevAny-Robotics-4B-LoRA (this model) 4B 71.1 0.806 0.728 65.9 0.443 0.567
JevAny-Qwen3.5-4B-LoRA (parent) 4B 52.7 0.588 0.641 59.1 1.057 0.149
Qwen3.5-4B zero-shot (constrained choice) 4B 49.9 0.519 0.592 50.0 1.261 0.203
OneJev-4B 4B 53.9 0.586 0.617 58.3 1.005 0.425
Intern-Decision-4B 4B 51.6 0.547 0.653 60.5 0.820 0.303
OmniJev-4B 4B 49.7 0.536 0.556 54.6 0.811* 0.264
OneJev-9B 9B 57.6 0.595 0.564 54.0 0.927 0.442
Jev-Omni-12B 12B 49.4 0.492 0.537 52.3 1.085 0.194
Qwen3.8-27B zero-shot (constrained choice) 27B 56.7 0.604 0.569 55.2 0.992 0.472
JevAny-Qwen3.8-27B-LoRA 27B 56.6 0.627 0.692 55.7 0.692 0.549
OneJev-27B 27B 60.4 0.651 0.700 62.7 0.805 0.605

RoboPulse: 1,800 pairs, native image resolution. RoboPulse++: 7,720 consecutive-step pairs, native resolution. RoboRewardBench: 2,831 videos, full media, official group-wise MAE (RoboArena 0.687 / OXE subsets 0.432 for this model). JevAny-Robotics-Public v0.1: 3,990 items, skill_role is the role-balanced chance-corrected skill. *OmniJev-4B's training data overlaps RoboRewardBench. Every model answered every item.

Paired comparisons (this model minus baseline, 95% cluster-bootstrap CI over episodes / task groups):

Baseline RoboPulse acc RoboPulse++ AUROC RoboRewardBench MAE JevAny-Robotics-Public skill_role
Parent JevAny-4B +18.3 [+15.1, +21.4] +0.087 [+0.049, +0.131] βˆ’0.614 [βˆ’0.725, βˆ’0.554] +0.419 [+0.365, +0.463]
Qwen3.5-4B zero-shot +21.2 [+18.1, +24.4] +0.136 [+0.091, +0.187] βˆ’0.817 [βˆ’0.875, βˆ’0.732] +0.364 [+0.318, +0.411]
OneJev-4B +17.2 [+14.1, +20.1] +0.111 [+0.072, +0.151] βˆ’0.562 [βˆ’0.811, βˆ’0.541] +0.142 [+0.100, +0.189]
Intern-Decision-4B +19.4 [+16.5, +22.5] +0.075 [+0.047, +0.104] βˆ’0.376 [βˆ’0.518, βˆ’0.341] +0.264 [+0.218, +0.312]
JevAny-27B +14.4 [+11.4, +17.0] +0.036 [βˆ’0.003, +0.076] βˆ’0.249 [βˆ’0.374, βˆ’0.220] +0.019 [βˆ’0.027, +0.052]
OneJev-27B +10.7 [+8.0, +13.3] +0.028 [βˆ’0.004, +0.063] βˆ’0.362 [βˆ’0.488, βˆ’0.336] βˆ’0.038 [βˆ’0.081, +0.006]

Every difference in favour of this model is significant except RoboPulse++ AUROC and JevAny-Robotics-Public against the two 27B judges (ties). Per-benchmark reports with all slices are in eval/comparison/.

Dedicated robot reward models (same harness)

Model Size RoboPulse acc ↑ RoboPulse AUROC ↑ RoboPulse++ AUROC ↑ RoboPulse++ bal. acc ↑ RoboRewardBench MAE ↓ RoboArena MAE ↓
JevAny-Robotics-4B-LoRA (this model) 4B 71.1 0.806 0.728 65.9 0.443 0.687
RoboReward-8B 8B 49.9ᡃ 0.499ᡃ 0.489ᡃ 50.0ᡃ 0.358 0.968
RoboReward-4B 4B 49.6ᡃ 0.501ᡃ 0.507ᡃ 51.6ᡃ 0.407 0.999
Robometer-4B 4B 63.6ᡇ 0.700ᡇ 0.630ᡇ 58.6ᡇ 0.777ᢜ 0.834ᢜ
Robo-Dopamine-GRM-2.0-8B 8B 90.1 0.953 0.717ᡈ 68.6ᡈ 1.023ᡉ 0.928ᡉ

How each model was run (all decisions fixed before scoring the test sets):

  • RoboReward-8B / 4B (teetone/RoboReward-*): model-card prompt verbatim, assistant turn prefilled with ANSWER: , softmax over the digit tokens 1–5; every frame of the same derived clips. Free generation gives the same RoboRewardBench MAE (0.357 vs 0.358 for 8B, 0 unparsable). ᡃ RoboReward scores whole rollouts; for the pairwise benchmarks we score the 2-frame video (before, after) minus (after, before). It was not trained for pairs, so treat these as an out-of-scope use.
  • Robometer-4B (robometer/Robometer-4B, official code): ᡇ progress difference between the two observations, with the two frames also scored in reversed order to remove a position bias. ᢜ 1–5 levels from the last-frame progress, with cut points fitted on a label-stratified subset of the RoboReward validation split (never the test set).
  • Robo-Dopamine-GRM-2.0-8B (tanhuajie2001/Robo-Dopamine-GRM-2.0-8B-Preview): the inference of PRM-as-a-Judge's GRMInference (vLLM, same prompt, sampling and model length), its native 8-image layout (reference start / end + three views before and after). ᡈ RoboPulse++: hop from the earlier to the later observation, reference start = the episode's first observation, blank goal image (the model card's default when no goal image exists), missing views repeated as in the model card; 146 of 23,559 requests exceeded the 8,192-token limit and were rerun with images downscaled to ≀76,800 px. Its other readouts: forward 0.698, position-debiased 0.730. ᡉ RoboRewardBench: hop from the first to the last frame, mapped to 1–5 with cut points fitted on the RoboReward validation subset. Robo-Dopamine's RoboPulse lead is not explained by test overlap: on the 939 items whose episodes do not appear in its public training data (GRM), it scores 87.5% (this model 68.8%).

Paired comparisons with the reward models (this model minus the reward model, 95% cluster-bootstrap CI):

Reward model RoboPulse acc RoboPulse++ AUROC RoboRewardBench MAE RoboArena MAE RoboRewardBench NLL
RoboReward-8B +21.2 [+17.9, +24.2] +0.239 [+0.204, +0.272] +0.085 [+0.039, +0.116] βˆ’0.281 [βˆ’0.380, βˆ’0.190] βˆ’0.367 [βˆ’0.517, βˆ’0.220]
RoboReward-4B +21.4 [+18.2, +24.6] +0.221 [+0.185, +0.256] +0.036 [+0.005, +0.093] βˆ’0.312 [βˆ’0.410, βˆ’0.216] βˆ’0.191 [βˆ’0.332, βˆ’0.060]
Robometer-4B +7.5 [+4.6, +10.3] +0.098 [+0.062, +0.131] βˆ’0.334 [βˆ’0.565, βˆ’0.316] βˆ’0.147 [βˆ’0.245, βˆ’0.048] –
Robo-Dopamine-8B βˆ’19.1 [βˆ’21.3, βˆ’16.8] +0.011 [βˆ’0.026, +0.045] βˆ’0.580 [βˆ’0.773, βˆ’0.521] βˆ’0.241 [βˆ’0.345, βˆ’0.132] –

Full reports: eval/comparison/reward_models_{robopulse,robopulsepp,roboreward,jevany_robotics_public,calibration_ranking_speed}.md. Positive RoboRewardBench MAE differences mean the reward model is better. Our harness measures the RoboReward models with lower error than published on the official leaderboard (8B: 0.358 vs 0.665; 4B: 0.407 vs 0.845) on the same 2,831 test videos (HF test split, revision 469b9af, all labels identical). Free generation reproduces our number and a zero-shot Qwen3-VL-4B-Instruct control lands near its published value (0.939 vs 1.032), so we report what we measured and do not compare with leaderboard entries.

Ranking policies: on RoboArena sessions where two policies attempted the same task and received different human scores (805 policy pairs, 134 sessions), the model orders the two policies like the human raters 88.4% of the time (RoboReward-8B 85.3%, RoboReward-4B 83.7%, Robometer-4B 75.5%, OneJev-27B 88.8%; paired vs RoboReward-8B +3.1 [βˆ’0.2, +6.5], vs RoboReward-4B +4.7 [+1.4, +8.0]).

Baselines

  • Zero-shot Qwen3.5-4B / Qwen3.8-27B (constrained choice): the untrained base VLM with the same judge prompt; options are lettered and the answer is restricted to the option letters (one forward pass, probabilities = softmax over the letter logits). This is the same scoring as vLLM structured-output choice decoding (structured_outputs.choice + letter logprobs); the robot suites were run with the transformers implementation, JevAny-Multimodal-agentic-Public with vLLM.
  • OneJev / Intern-Decision / OmniJev / Jev-Omni: released checkpoints run with their own inference code inside the same harness and items.
  • JevAny-Qwen3.8-27B-LoRA and the parent JevAny-Qwen3.5-4B-LoRA: released JevAny checkpoints, same media settings as this model.

General judgement (retention)

Benchmark Qwen3.5-4B zero-shot (vLLM choice) Parent JevAny-4B This model This βˆ’ parent
Transfer v9 (1,046) acc / NLL β€” 78.68% / 0.587 78.39% / 0.664 acc βˆ’0.29 [βˆ’1.75, +1.05]
JevBench public (local rebuild, 231) acc β€” 77.92% 77.92% 0.00 [βˆ’3.10, +2.64]
JevAny-Multimodal-agentic-Public v0.4 (3,208) skill_role / acc 0.223 / 53.6% 0.201 / 52.6% 0.176 / 51.4% acc βˆ’1.2 (n.s., McNemar p = 0.12)

Accuracy on general (non-robot) judgement is retained within noise; probabilities become over-confident outside robotics (NLL +0.077 on Transfer, +0.163 on JevAny-Multimodal-agentic-Public), so recalibrate before using them there. For general-purpose judging use the parent checkpoint; this one is specialised for robotics. The JevBench number is on a local rebuild of the public suite (same 231 items, different manifest hash) and is not directly comparable to the official 80.09%.

Training

  • Init: LoRA adapter + pointer head of SimpleJev/JevAny-Qwen3.5-4B-LoRA (delta mode); base revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
  • 1 epoch, 5,780 optimizer steps, global batch 32, lr 1e-5 (OneCycle), LoRA r=8 on all projection modules, head dim 256 + residual 256, bf16.
  • Full media: images at native resolution, every frame of the derived video clips (β‰ˆ1–2 fps, 8–32 frames), up to 16,384 packed tokens.
  • training_config.json / training_metrics.json hold the exact arguments; eval/training_eval_history.jsonl the in-training evaluations.
step development acc / NLL (3,026) calibration acc / NLL (2,228)
0 (parent) 0.463 / 1.069 0.496 / 0.969
1,000 0.640 / 0.757 0.688 / 0.633
3,000 0.723 / 0.650 0.778 / 0.499
5,780 (this checkpoint) 0.733 / 0.638 0.786 / 0.475

Limits

  • Source overlap. RoboPulse and RoboRewardBench draw on the same public robot datasets as parts of the training data (GRM, RoboReward train). Training data were decontaminated against all test sets (episode blocklists + perceptual-hash frame dedup), but scenes and robots overlap.
  • RoboPulse++ threshold. Training balanced rising/falling 1:1, while the RoboPulse++ test is 81% rising; at the default 0.5 threshold the model under-predicts "rising" (accuracy 60.4% vs 69.9% for the parent; a constant "rising" answer scores 81.1%). Ranking quality (AUROC) and balanced accuracy are the meaningful metrics here and both improve; set the threshold on your own data. Multi-view (2–3 camera) items and long, slow-progress intervals remain the weakest cases.
  • RoboRewardBench's synthetic OXE items. On relabelled-instruction and clipped-video items the RoboReward models, trained on exactly that construction, are clearly better (MAE 0.223 vs 0.297 and 0.418 vs 0.584 for RoboReward-8B vs this model); clipped videos are often scored as near-complete.
  • Speed vs RoboReward. Our full-media protocol gives the model ~2.5Γ— more visual tokens per video than RoboReward's processor budget, and Qwen3.5's linear-attention layers are slower than Qwen3-VL attention: per RoboRewardBench video this model takes a median 279 ms (fast kernels) vs 142–160 ms for a single RoboReward forward pass and 361 ms for RoboReward-8B's normal generation.
  • Calibration outside robotics (see above).

Usage 

# --media is in a pending JevAny PR; until it is merged, use the full-media branch of the fork:
git clone -b full-media https://github.com/BruceSheng1202/JevAny && cd JevAny
pip install -e '.[serve,multimodal]'
jevany serve --checkpoint SimpleJev/JevAny-Robotics-4B-LoRA --device cuda --dtype bf16 \
    --media full --max-state-tokens 16384 --max-packed-tokens 32768

Use --media full (or JEVANY_MEDIA=full; also on jevany decide --checkpoint): images at native resolution and every frame of a video, the inputs this model was trained and evaluated with. With it, our evaluation harness returns exactly the probabilities of the reported runs (checked on 300 RoboRewardBench videos and 300 RoboPulse pairs). The JevAny default (capped: the images of a request share 512Γ—512 pixels, a video is 8 frames within 512Γ—512) costs accuracy, most of all on video:

full capped
RoboRewardBench MAE ↓ (RoboArena MAE) 0.443 (0.687) 0.607 (1.289)
RoboPulse accuracy / AUROC 71.1 / 0.806 69.4 / 0.786

Send short clips like the evaluation's (about 1 frame per second, 8–32 frames); every frame becomes visual tokens. For speed, install the optional flash-linear-attention and causal-conv1d kernels (see JevAny's DEPLOYMENT.md): median 185 ms per RoboPulse pair on an A100.

The repository contains LoRA adapter weights and JevAny readout metadata, not the base weights; the base model's license and access terms also apply, as do the licenses of the training data sources (RoboReward, Robo-Dopamine GRM, Guardian BDV2 / RLBench, DenseReward, RoboFAC, ViFailback, RoboVQA).

Downloads last month
45
Video Preview
Robotics
loading

Model tree for SimpleJev/JevAny-Robotics-4B-LoRA

Base model

Qwen/Qwen3.5-4B-Base
Finetuned
Qwen/Qwen3.5-4B
Adapter
(931)
this model