| --- |
| license: other |
| tags: |
| - computer-use |
| - video |
| - action-decoder |
| - onevision |
| - qwen |
| --- |
| |
| # OneVision Task-Aware Action Decoder — checkpoints |
|
|
| Lightweight task-aware action decoder for computer-use / instructional-video understanding. |
| Built on a frozen **OneVision vision encoder** (`lmms-lab-encoder/onevision-encoder-large-lang`) |
| and **Qwen/Qwen3-4B** as the text encoder/decoder, **LoRA**-tuned (r=8, alpha=16, dropout=0.05, |
| applied to both vision and text). Trained with `train_onevision_lw_decoder.py` in |
| [proteusagi/onevision-compuse](https://github.com/proteusagi/onevision-compuse) under |
| `onevision-lightweight-decoder/`. |
|
|
| Given video visible-token embeddings plus a task prompt, the model predicts, per frame: |
| - discrete **action / key / modifier** classes, |
| - a **click heatmap** (actor query attending over visual tokens), |
| - (new_arch only) an **autoregressive per-frame transcript**. |
| |
| ## Checkpoints (~8.7 GB each — LoRA adapters + task heads) |
| |
| ### `new_arch/best.pt` — "parallel heads" architecture (2026-03-16) — RECOMMENDED |
| - Adds an **autoregressive transcript decoder** (per-frame transcript via the Qwen LM head: |
| visual features + transcript history of frames 0..t-1 predict frame t) on top of the |
| action/key/modifier classification heads and the click-heatmap head. |
| - Modularized `encode`/`decode`, **parallel prediction heads**, and an **efficient-inference** |
| path (git commits: "parallel heads", "efficient inference", "further speedup"). |
| - Training performance: **train action accuracy ~64-66%**, train loss ~0.9 (see `accuracy.png` |
| and `loss.png` in the repo). The epoch ~85-117 plateau is a curriculum/data change; metrics |
| recover afterward. |
| - Choose this by default: most capable (also generates transcripts) and the actively developed arch. |
|
|
| ### `onevision_task_action_decoder/best.pt` — original architecture (2026-03-10) |
| - The original `OneVisionTaskAwareActionDecoder`: action/key/modifier classification heads plus a |
| click-heatmap head over OneVision visual tokens, with frame self-attention and prompt |
| cross-attention. **Predates the transcript decoder** (which was added 2026-03-16), so it does |
| action/key/modifier + click only. |
| - Choose this if you specifically want the lighter, action-only baseline or to reproduce the |
| March-10 results. |
| |
| > Also in this repo: `onevision_task_action_decoder/epoch_*.pt` (earlier epoch snapshots of the |
| > original run) and `onevision_task_action_decoder_8F/best.pt` (an 8-frame-clip variant). |
| |
| ## Which to choose |
| - **`new_arch/best.pt`** — full capability (action + transcript), latest architecture. Default pick. |
| - **`onevision_task_action_decoder/best.pt`** — original action-only model / March-10 baseline. |
| |
| ## Loading |
| ```python |
| import torch |
| ckpt = torch.load("new_arch/best.pt", map_location="cpu") |
| # State for OneVisionTaskAwareActionDecoder: |
| # vision_encoder_name = "lmms-lab-encoder/onevision-encoder-large-lang" |
| # text_encoder_name = "Qwen/Qwen3-4B" |
| # LoRA r=8, alpha=16 on both encoders |
| ``` |
| See `onevision-lightweight-decoder/src/model.py` in proteusagi/onevision-compuse for the module |
| definition and `train_onevision_lw_decoder.py` for the training/eval pipeline. |
|
|
