SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models
Paper • 2610.17727 • Published • 2
SkillScreen-2B-Markdown
Current deployment release v0.1.0 · 8K context
SkillScreen-2B-Markdown screens AI-agent skills directly from SKILL.md. It returns a maliciousness score and a review decision in one forward pass. Complete merged weights, the original LoRA adapter and frozen operating points are included.
Package model · Markdown model · Data · Code · Release v0.1.0 · Paper
Run locally
Use Linux, Python 3.12 and a CUDA GPU with BF16 support. The bundled runtime and full context limit are verified on an A100 80GB.
python -m pip install huggingface_hub==1.33.0
hf download SkillScreen/SkillScreen-2B-Markdown --revision v0.1.0 --local-dir SkillScreen-2B-Markdown
python -m pip install -r SkillScreen-2B-Markdown/requirements/inference.txt
python SkillScreen-2B-Markdown/scan.py path/to/SKILL.md --output score.json
For a bundled example, replace the input path with SkillScreen-2B-Markdown/examples/local-inventory/SKILL.md. The scanner reads files as text and never executes them. Scoring runs locally. --inspect lists input files without loading the model.
The default is the native 0.5 threshold. Use --operating-point high-recall for the frozen 99% OOF recall threshold, approximately 0.011642. --operating-point native selects 0.5. Full-precision thresholds are loaded from deployment_settings.json. Decisions compare raw entailment log-odds, avoiding rounded threshold values.
Out-of-fold results
Three-fold OOF evaluation over 13,514 skills, including 7,500 malicious and 6,014 benign examples.
| Operating point | Score threshold | Recall | Benign FPR | Precision | F1 |
|---|---|---|---|---|---|
| Native 0.5 | 0.500000 | 96.96% | 0.88% | 99.28% | 98.10% |
| 99% OOF recall | 0.011642 | 99.00% | 15.71% | 88.71% | 93.57% |
AUPRC (average precision) 0.9966 · AUROC 0.9946. Per-skill OOF predictions and full metrics are included.
Each OOF score comes from a fold model trained without that skill's content group. Thresholds were selected on these pooled predictions and transferred to the final model trained on all eligible records. These results describe the OOF procedure, not an independent test of the final refit. Recall and FPR can change on new collections.
Training and input
Initialized from OpenJev 2B, then trained for one epoch with seed 42, LoRA rank 8 and a trainable NLI classification head. The final weights use all 13,514 eligible records from SkillScreen-Data at the pinned revision. Labels are inherited from upstream benchmarks. This deployment release is separate from the paper's train-only experiment checkpoints.
The fixed hypothesis is “This AI agent skill is malicious.” The score is the entailment softmax output, not a separately calibrated probability. The input is the original Markdown paired with the fixed hypothesis. The report records token truncation. Inputs above 8,192 tokens are truncated while retaining the hypothesis. A below-threshold score means no model alert, not a safety certification.
Training configuration · Adapter · Release checksums
Citation and license
Please cite SkillScreen for the method and identify this deployment release when reporting results. Machine-readable citation details are in CITATION.cff.
Jerrold Huang, Junwei Quan, Saiyue Lyu, Zihan Guo and Yingqiang Ge. SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models. 2026.
Weights, code and configuration are Apache-2.0 for rights held by the contributors. Documentation and predictions are CC BY 4.0. Upstream model and dataset terms remain applicable. See third-party notices.
- Downloads last month
- -