JevForge-0.8B

JevForge-0.8B is an experimental candidate-conditioned scoring model for structured web decisions. It starts from the pretrained Qwen3.5-0.8B backbone, adds a shared scalar decision head, and returns a probability distribution over the candidates supplied for each question.

This is not a chat model and is not intended to be loaded as a standard text-generation pipeline. Use the jevforge.predict.Predictor implementation from the JevForge source repository.

Evaluation

Evaluation uses frozen website-disjoint splits and temperature 0.9717.

Metric	Test (2,400)	OOD (1,158)
Choice top-1 in positives	0.5787	0.6373
Choice exact gold	0.5675	0.6192
Noul accuracy	0.8263	0.8601
Noul Brier	0.1283	0.1173
Score MAE	0.3674	0.3898

These results apply only to the recorded offline candidate-scoring protocol. They are not evidence of autonomous live-site operation or a browser-agent leaderboard result.

Raw-backbone reference

On the first 200 records of each frozen split, the unmodified Qwen/Qwen3.5-0.8B chat model was prompted to score every candidate from 0 to 10. This is a zero-shot reference rather than an architectural ablation because the raw model has no JevForge scalar head.

Choice top-1 in positives	Raw backbone	JevForge-0.8B
Test (200)	0.235	0.505
OOD (200)	0.340	0.630

Training and model lineage

• Base model: Qwen/Qwen3.5-0.8B, loaded from pretrained weights.
• Objective: supervised candidate-distribution prediction.
• Training steps: 1,200.
• Maximum sequence length: 768 tokens.
• Best checkpoint: step 1,200.
• Calibration temperature: 0.9717.

The upstream Qwen3.5-0.8B model is licensed under Apache 2.0. Its notice and license are retained in this release.

Data lineage

Training records were transformed from Mind2Web annotations obtained through LangAGI-Lab/Mind2Web-axtree-cleaned-lite. Mind2Web is distributed under CC BY 4.0. No raw or unzipped official Mind2Web test files, and no complete JevForge training or evaluation split, are included in this model repository. Obtain benchmark data from the official upstream project and follow its current access and attribution requirements.

Intended use

• Offline scoring and ranking of supplied candidate actions.
• Research on choice, binary noul, and ordered score decisions.
• Reproduction of the fixed evaluation protocol using the JevForge codebase.

Limitations

• Candidate discovery is out of scope; callers must supply candidate paths.
• Offline page snapshots differ from live browser environments.
• The model has only been evaluated on the recorded task construction and frozen splits described above.