Fara 1.5-9B W8 baseline evaluation

This repository preserves the exact Fara 1.5-9B checkpoint used for the W8 DOM browser-agent baseline, together with commit-level provenance for the task dataset and evaluation environment.

The checkpoint was not trained or fine-tuned for this run. Its files are an exact mirror of the pinned Microsoft checkpoint. The W8 integration translated the recorder's DOM-based OpenAI Responses contract to the model-serving endpoint; that infrastructure code does not modify the weights.

Baseline result

Measure Result
Selected tasks 90
Passed 6
Verified failures 60
Execution errors 24
Infrastructure errors 0
Raw Pass@1 6.67%
Pass rate among verified tasks 9.09%

The 24 execution errors require a separate audit. The raw Pass@1 should not be treated as a clean estimate of model capability until those errors are classified and any invalid trials are rerun.

Provenance

Component Repository Exact revision
Base checkpoint microsoft/Fara1.5-9B 1a93677cd89d5601bc2ed759791e981f3a520032
Fara reference source microsoft/fara a675d6d61c41c47ae87bacefeab22caad18e3e84
Task dataset WootzappLab/cua-bench 745eebb8f07abb92fdaae356bb6bee3a1f2975af
Task recorder, browser harness, environments, and verifier tokenbender/w8-cua-bench 66037db6bae4b00a2b5860aa08ebb69b559e6871
DOM serving infrastructure Pending infrastructure PR Pending

The Hugging Face revision pins the task dataset. The Git revision separately pins the recorder, browser harness, browser environments, and verifier used to execute and score those tasks.

Serving configuration

  • Model dtype: BF16
  • KV-cache dtype: BF16 through vLLM auto
  • Maximum model length: 32,768 tokens
  • Tensor parallel size: 1
  • Runtime: vLLM 0.19.1, PyTorch 2.10.0, Transformers 5.6.2
  • Hardware: one NVIDIA H100 80 GB HBM3
  • GCP instance: w8-fara-9b-8115f370-head-3ibtw2ah-compute
  • Run ID: fara15-dom-baseline
  • Run date: 2026-09-25

Repository contents

  • Model configuration, tokenizer, processor, chat template, and Safetensors weights at the repository root, copied from the pinned upstream revision.
  • CHECKPOINT_FILES.json: upstream Git blob identities and LFS SHA256 hashes, including all four Safetensors shards.
  • MANIFEST.json: machine-readable checkpoint, code, and run provenance.
  • PROVENANCE.md: concise revision and experiment-boundary receipt.
  • CHECKSUMS.sha256: hashes for the repository's reproducibility metadata.

Evaluation artifacts, the detailed results report, and the infrastructure PR will be linked after they are published. Their absence here does not affect the identity of the preserved checkpoint.

Load the checkpoint

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "WootzappLab/fara-9b-baseline-eval"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

License and attribution

The mirrored base checkpoint is released by Microsoft under the MIT License. See the upstream Fara 1.5-9B model card for its original documentation, intended use, and limitations.

Downloads last month
21
Safetensors
Model size
9B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for WootzappLab/fara-9b-baseline-eval

Finetuned
Qwen/Qwen3.5-9B
Finetuned
(1)
this model