FEGE16/robot-task-observer-subtask-recognition-history

Robot Task Observer: Current-Subtask Recognition

Recognizes the robot's current observed subtask from visual history. This LoRA adapter is an observation component of the Robot Task Observer VLM-assisted VLA framework.

Base model: Qwen3-VL-4B-Instruct. Load it together with this adapter.

Usage

• Input: Global task instruction and an ordered history of scene and wrist images. Use up to eight timestamps with 224 x 224 images from both views.
• Output: A short English description of the current action, object, target, and relation, or UNKNOWN when unclear.

Loading

Use PyTorch, Transformers with Qwen3-VL support, PEFT, and Accelerate. This example assumes a bfloat16-capable GPU.

import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
from peft import PeftModel

BASE_ID = "Qwen/Qwen3-VL-4B-Instruct"
BASE_REVISION = "ebb281ec70b05090aa6165b016eac8ec08e71b17"
ADAPTER_ID = "FEGE16/robot-task-observer-subtask-recognition-history"

processor = AutoProcessor.from_pretrained(BASE_ID, revision=BASE_REVISION)
base = Qwen3VLForConditionalGeneration.from_pretrained(
    BASE_ID,
    revision=BASE_REVISION,
    dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, ADAPTER_ID).eval()

Provide the task prompt and image history using the base model's multimodal chat format, then call model.generate.

Project overview and method figure

License: Apache 2.0. See LICENSE and NOTICE.