GrimGuard-01-Prompt-Guard

Published by Grimlabs LLC.

GrimGuard-01-Prompt-Guard is a small, instruction-conditioned prompt guard. It reads a natural-language policy containing DO_FLAG and DO_NOT_FLAG rules plus a user prompt, and returns allow, flag, or review — without fine-tuning a new classifier for every application.

[CONTEXT] A digital storefront support assistant for ebooks and subscriptions.
DO_FLAG:
- Requests for a refund for a digital product.
- Requests to cancel an active subscription.
DO_NOT_FLAG:
- Questions about the refund policy or cancellation terms.

This repository contains:

Path	Model	Params	Purpose
root	GrimGuard-Pro	110.27M	Full policy conditioning; accuracy path
fast/	GrimGuard-Fast	22.76M	Compact policy; strict single-core CPU path
onnx/	both	—	fp32 and per-channel int8 ONNX artifacts

The main model is an encoder-only classifier with an optional rule-attribution head. The Fast model is a MiniLM-L6 student distilled from Pro.

Quickstart

The model uses a custom architecture, so load it with trust_remote_code=True.

import torch
from transformers import AutoModel, AutoTokenizer

repo_id = "Grimlabs/GrimGuard-01-Prompt-Guard"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True).eval()

policy = """\
[CONTEXT] A digital storefront support assistant for ebooks and subscriptions.
DO_FLAG:
- Requests for a refund for a digital product.
- Requests to cancel an active subscription.
DO_NOT_FLAG:
- Questions about the refund policy or cancellation terms.
"""

for prompt in [
    "I want a refund for the ebook I bought yesterday.",
    "What does your refund policy say about digital goods?",
]:
    result = model.classify(tokenizer, policy, prompt, threshold=0.5)
    print(prompt)
    print(result)

classify returns:

{
  "verdict": "flag" | "allow" | "review",
  "score": float,                # calibrated probability
  "logit": float,                # raw binary logit
  "matched_rule_id": "df_1" | None,
  "matched_rule_text": str | None,
  "policy_text": str,            # the exact rendered policy input
  "too_long": bool,
  "omitted_rule_ids": list[str],
}

Fast CPU model

fast_tokenizer = AutoTokenizer.from_pretrained(repo_id, subfolder="fast")
fast = AutoModel.from_pretrained(repo_id, subfolder="fast", trust_remote_code=True).eval()
print(fast.classify(fast_tokenizer, policy, prompt, threshold=0.5743))

The Fast model uses a compact rendered policy (52-token policy budget, 72-token prompt budget, 128-token maximum). It is intended for high-volume, established policies. The first request under a new policy should use the Pro model or return review; this is the safety routing used by the full serving stack.

Thresholds

• The config stores the F1-optimal calibration threshold (threshold in config.json).
• Thresholds do not transfer across domains. The F1 threshold has a high false-positive rate on unseen domains. For conservative application-specific use, pass a higher threshold (e.g., threshold=0.5) and calibrate on the target deployment's benign traffic.
• review is returned when the policy+prompt exceeds the model's sequence budget; do not treat it as allow.

ONNX

onnx/ contains:

• pro.onnx (fp32) and pro_int8.onnx (per-channel int8)
• fast.onnx (fp32) and fast_int8.onnx (per-channel int8)

See onnx/README.md for input names and a minimal ONNX Runtime example. The Fast int8 artifact is the recommended strict-CPU deployment: measured 8.61 ms p50 / 8.76 ms p95 model-only at 128 tokens on one pinned Intel i9-14900KF core.

Evaluation

Final held-out evaluation, 3,663 examples (909 in-distribution, 2,185 unseen-domain, 569 adversarial):

Model / suite	AUROC	F1	FPR
Pro — all	0.9289	0.8387	0.1129
Pro — in-distribution	0.9767	0.9082	0.0415
Pro — unseen domains	0.8927	0.8011	0.1595
Pro — adversarial	0.9641	0.8741	0.0468
Fast — all	0.8868	0.8030	0.1206
Fast — unseen domains	0.8301	0.7500	0.1705
Lexical rule-keyword baseline	0.6452	0.6757	0.8074

Policy controls (Pro): empty policy 100% allow, policy-shuffled 95.5% correct, rule-added 100% flag, rule-removed 76.7% allow. Rule attribution top-1 85.3%, top-3 99.2%.

Counterfactual pair ranking is 0.743 overall and 0.706 on unseen policies, below the 0.80 target. Aggregate operating points and confidence intervals are in eval_results_pro.json and eval_results_fast.json.

Training data

Synthetic, English, single-turn policy-conditioned data authored by domain-specific LLM sub-agents:

• 10,452 authored examples across 804 policies (612 training/raw, 192 held-out/eval).
• 30,219 processed training pairs after counterfactual augmentation (rule removal, rule addition, empty policy, cross-policy pairing, authored counterfactual groups) and custom-policy oversampling.
• Pro was trained on BERT-base with binary BCE, an optional rule-attribution BCE, and a pairwise margin loss on counterfactual pairs.
• A short warm-started fine-tune on ordinary-action/refund custom policies produced the released Pro checkpoint.
• Fast was distilled from Pro with hard-label BCE + KL.

No real user data was used. Validation and a 200-example blind QA audit (Cohen's kappa 0.98) were performed before training.

Latency

Artifact	Hardware	Result
Fast int8, 128 tokens	1 pinned i9-14900KF core	8.61 ms p50 / 8.76 ms p95
Fast, warm HTTP path	same	2.1 ms p50 / 2.6 ms p95 client-observed
Pro fp16, 256 tokens	RTX 4070	2.2 ms p50 / 3.1 ms p95
Pro int8, 128 tokens	8 CPU threads	15.0 ms p50 / 15.1 ms p95

A single dense 80–120M encoder cannot meet <15 ms on one CPU core at useful context; the Fast path is the strict single-core model, and Pro is the GPU/multi-thread accuracy path.

Limitations

• English, single-turn only.
• Synthetic training data; performance on real traffic is not established.
• Unseen-policy generalization and low-FPR calibration are the main open gaps.
• Ordinary-action custom policies (e.g., "flag every refund request") required a targeted fine-tune and threshold discipline; a low threshold can over-flag the same prompt under an allowing policy.
• Counterfactual pair accuracy is below target.
• Fast int8 has 98.0% label agreement with PyTorch (target 99%).
• A guardrail is not a substitute for human review in high-stakes settings. Adversarial robustness is measured, not guaranteed.
• The model follows the written policy, including unusual or adversarial policies supplied by the deployer.

License

Apache-2.0. The Fast backbone derives from sentence-transformers/all-MiniLM-L6-v2; the Pro backbone derives from google-bert/bert-base-uncased.

Citation

@misc{grimguard01promptguard,
  title  = {GrimGuard-01-Prompt-Guard: A Small Instruction-Conditioned Prompt Guard},
  year   = {2026},
  author = {Grimlabs LLC},
  note   = {Dynamic DO_FLAG/DO_NOT_FLAG policies with a Fast CPU path and a Pro accuracy path}
}