SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models
Paper • 2610.17727 • Published • 2
SkillScreen-2B-Package
Recommended deployment release v0.1.0 · 32K context
SkillScreen-2B-Package screens AI-agent skills using SKILL.md and supporting text files. It returns a maliciousness score and a review decision in one forward pass. Complete merged weights, the original LoRA adapter and frozen operating points are included.
Package model · Markdown model · Data · Code · Release v0.1.0 · Paper
Run locally
Use Linux, Python 3.12 and a CUDA GPU with BF16 support. The bundled runtime and full context limit are verified on an A100 80GB.
python -m pip install huggingface_hub==1.33.0
hf download SkillScreen/SkillScreen-2B-Package --revision v0.1.0 --local-dir SkillScreen-2B-Package
python -m pip install -r SkillScreen-2B-Package/requirements/inference.txt
python SkillScreen-2B-Package/scan.py path/to/skill-package --output score.json
For a bundled example, replace the input path with SkillScreen-2B-Package/examples/local-inventory. The scanner reads files as text and never executes them. Scoring runs locally. --inspect lists input files without loading the model.
The default is the frozen 99% OOF recall threshold, approximately 0.288. --operating-point native selects 0.5. Full-precision thresholds are loaded from deployment_settings.json. Decisions compare raw entailment log-odds, avoiding rounded threshold values.
Out-of-fold results
Three-fold OOF evaluation over 11,831 skills, including 6,436 malicious and 5,395 benign examples.
| Operating point | Score threshold | Recall | Benign FPR | Precision | F1 |
|---|---|---|---|---|---|
| Native 0.5 | 0.500000 | 98.93% | 0.33% | 99.72% | 99.32% |
| 99% OOF recall | 0.287766 | 99.01% | 0.35% | 99.70% | 99.35% |
AUPRC (average precision) 0.9990 · AUROC 0.9985. Per-skill OOF predictions and full metrics are included.
Each OOF score comes from a fold model trained without that skill's content group. Thresholds were selected on these pooled predictions and transferred to the final model trained on all eligible records. These results describe the OOF procedure, not an independent test of the final refit. Recall and FPR can change on new collections.
Training and input
Initialized from OpenJev 2B, then trained for one epoch with seed 42, LoRA rank 8 and a trainable NLI classification head. The final weights use all 11,831 eligible records from SkillScreen-Data at the pinned revision. Labels are inherited from upstream benchmarks. This deployment release is separate from the paper's train-only experiment checkpoints.
The fixed hypothesis is “This AI agent skill is malicious.” The score is the entailment softmax output, not a separately calibrated probability. Package text is ordered as SKILL.md, code, configuration, documentation, then bulk text. The report identifies fully retained, partial and omitted files. Inputs above 32,768 tokens are truncated while retaining the hypothesis. A below-threshold score means no model alert, not a safety certification.
Training configuration · Adapter · Release checksums
Citation and license
Please cite SkillScreen for the method and identify this deployment release when reporting results. Machine-readable citation details are in CITATION.cff.
Jerrold Huang, Junwei Quan, Saiyue Lyu, Zihan Guo and Yingqiang Ge. SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models. 2026.
Weights, code and configuration are Apache-2.0 for rights held by the contributors. Documentation and predictions are CC BY 4.0. Upstream model and dataset terms remain applicable. See third-party notices.
- Downloads last month
- 36