Qwen3-VL-8B-Instruct-UI-Genie-scoring

A Bradley-Terry reward model fine-tuned from Qwen/Qwen3-VL-8B-Instruct on the UI-Genie-RM-517k dataset. This is a merged sequence-classification checkpoint (task_type="seq_cls", num_labels=1, problem_type="regression") trained with the Bradley-Terry pairwise loss. A learned score.weight linear head projects the last non-padding token's hidden state to a scalar reward.

Score = score_head(last non-padding token hidden state) — a single float per input trajectory. Higher score = higher quality action.

Intended Use

Drop-in scalar reward signal for GUI agent training (e.g. PPO/GRPO, RLHF) or best-of-N selection. Unlike the discrete <|+|> / <|-|> classifier, this model outputs a continuous unbounded reward that is directly usable as a value signal.

Inference

Requirements

pip install transformers torch pillow

Note: Use HuggingFace transformers directly — vLLM's Qwen3VLForConditionalGeneration loader does not support the additional score.weight tensor.

Scoring with HuggingFace Transformers

import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
from PIL import Image

MODEL_PATH = "Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie-scoring"
MAX_LEN = 8192

model = Qwen3VLForConditionalGeneration.from_pretrained(
    MODEL_PATH,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(MODEL_PATH, max_pixels=1_048_576)
tokenizer = processor.tokenizer
score_head = torch.nn.Linear(model.config.hidden_size, 1, bias=False).to(model.device, dtype=torch.bfloat16)

# Load the trained score.weight from the checkpoint
from safetensors.torch import load_file
state = load_file(f"{MODEL_PATH}/model-00001-of-00004.safetensors")  # or whichever shard contains score.weight
# Alternatively, it is loaded automatically if the model class supports it.


def score_action(prefix_messages, response_text, images=None):
    """
    Score a GUI agent action.

    Args:
        prefix_messages: Chat messages up to (not including) the assistant turn.
                         E.g. [{"role": "system", "content": "..."},
                               {"role": "user", "content": [{"type":"text","text":"..."},
                                                             {"type":"image"}]}]
        response_text:   The assistant tool-call response to score.
        images:          List of PIL.Image objects matching image placeholders.

    Returns:
        float: Scalar BT reward (higher = better action).
    """
    prefix_text = processor.apply_chat_template(
        prefix_messages, tokenize=False, add_generation_prompt=True
    )
    full_text = prefix_text + response_text.strip()

    inputs = tokenizer(
        full_text,
        return_tensors="pt",
        truncation=True,
        max_length=MAX_LEN,
    ).to(model.device)

    if images:
        pixel_data = processor.image_processor(images=images, return_tensors="pt")
        inputs["pixel_values"] = pixel_data["pixel_values"].to(model.device, dtype=torch.bfloat16)
        inputs["image_grid_thw"] = pixel_data["image_grid_thw"].to(model.device)

    with torch.no_grad():
        outputs = model(**inputs, output_hidden_states=True)

    # Last non-padding hidden state → score head
    hidden = outputs.hidden_states[-1]              # (1, seq_len, hidden_dim)
    last_idx = inputs["attention_mask"].sum(dim=1) - 1
    last_hidden = hidden[0, last_idx[0], :]         # (hidden_dim,)
    reward = model.score(last_hidden.unsqueeze(0)).squeeze().item()
    return reward


# Example: compare two candidate actions
screenshot = Image.open("screenshot.png").convert("RGB")

SYSTEM_PROMPT = "You are a helpful assistant."  # use full mobile_use tool spec in practice

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user",   "content": [
        {"type": "text",  "text": "The user query: tap the search button\n"},
        {"type": "image"},
    ]},
]

action_a = '<tool_call>\n{"name": "mobile_use", "arguments": {"action": "click", "coordinate": [540, 120]}}\n</tool_call>'
action_b = '<tool_call>\n{"name": "mobile_use", "arguments": {"action": "click", "coordinate": [100, 800]}}\n</tool_call>'

score_a = score_action(messages, action_a, images=[screenshot])
score_b = score_action(messages, action_b, images=[screenshot])

print(f"Action A: {score_a:.4f}")
print(f"Action B: {score_b:.4f}")
print(f"Preferred: {'A' if score_a > score_b else 'B'}")

Pairwise evaluation with rm_eval

python eval_rm.py \
    --rm_path Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie-scoring \
    --datasets ui-genie \
    --mode bt \
    --uigenie_jsonl /path/to/reward_data_rm_pairs_last5.jsonl \
    --uigenie_images_dir /path/to/images \
    --output_dir results/

Training Details

Field	Value
Base model	Qwen/Qwen3-VL-8B-Instruct
Training method	Bradley-Terry pairwise loss (LoRA, merged)
Architecture	Qwen3VLForConditionalGeneration + score.weight linear head
Task type	seq_cls (regression, num_labels=1)
Score	Last non-padding token hidden state → linear head → scalar
dtype	bfloat16

Related Models

• Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie — SFT discrete preference classifier (<|+|> / <|-|>), also fine-tuned from Qwen3-VL-8B-Instruct on UI-Genie-RM-517k.

Citation

@misc{qwen3technicalreport,
      title={Qwen3 Technical Report},
      author={Qwen Team},
      year={2025},
      eprint={2505.09388},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
}