Privoid-ai/Privoid-Confidential-1B

Privoid Confidential-1B

Policy-conditioned detection of confidential business information, with exact evidence. Give it your company's confidentiality policy and a passage; it returns which information is confidential under that policy, which rule it falls under, and the exact text that proves it — or says needs_review when the policy and context do not settle the question.

The same passage can be confidential under one policy and permitted under another: the model reads the policy instead of reacting to keywords like "secret" or "private", and it is not a PII detector.

Base model	ibm-granite/granite-4.0-1b @ 6a7381ba (Apache-2.0, 1.63B parameters)
Method	LoRA (r=16, alpha=32) on attention + shared MLP, assistant-only loss
Adapter size	76 MB
Language	English
GGUF	Privoid-ai/Privoid-Confidential-1B-GGUF
License	Apache-2.0

Results at a glance

Locked test set: 2,274 examples in 1,750 scenario groups across seven groups, including policy wordings, categories and document templates never seen in training. Evaluated once, after acceptance criteria were recorded.

System	Decision acc.	Doc F1	Missed confidential ↓	False alarms on no-match ↓	Evidence span F1	Raw output valid	needs_review F1	Counterfactual sets correct
Keyword baseline	40.4%	0.191	87.7%	11.6%	0.119	100.0%	0.000	0.4%
Granite 4.0 1B (base, same prompt)	37.3%	0.328	11.4%	15.7%	0.176	2.0%	0.339	1.5%
Privoid Confidential-1B	96.9%	0.963	5.4%	0.3%	0.941	99.9%	0.980	94.4%
Privoid Confidential-1B GGUF Q4_K_M	96.8%	0.961	6.2%	0.3%	0.929	99.7%	0.985	93.6%

Privoid Confidential-1B, 95% cluster-bootstrap intervals: decision accuracy 96.9% [96.1%, 97.7%]; evidence span F1 0.941 [0.925, 0.954]; missed-confidential rate 5.4% [3.8%, 7.3%]; false alarms on no-match 0.3% [0.0%, 0.8%].

By test group (decision accuracy)

Test group	n	Keyword baseline	Granite 4.0 1B (base, same prompt)	Privoid Confidential-1B	Privoid Confidential-1B GGUF Q4_K_M
Familiar policy types, new passages	313	44.4%	39.3%	99.7%	99.7%
Held-out rule wordings and category names	308	44.8%	38.3%	99.7%	99.7%
Held-out confidentiality categories	292	45.5%	43.5%	89.0%	88.0%
Held-out fact templates and document formats	316	40.2%	34.2%	90.5%	90.8%
Negation, hypotheticals, exceptions	250	58.4%	27.2%	99.6%	100.0%
Embedded instructions / fake system markers	261	36.0%	41.0%	98.5%	97.7%
Inputs that require needs_review	534	26.6%	36.9%	99.8%	100.0%

Privoid Confidential-1B, all metrics by test group

Test group	Decision acc.	Evidence span F1	Missed confidential	False alarms	needs_review F1
Familiar policy types, new passages	99.7%	1.000	0.0%	0.0%	0.987
Held-out rule wordings and category names	99.7%	0.986	0.8%	0.0%	1.000
Held-out confidentiality categories	89.0%	0.788	19.0%	0.7%	0.873
Held-out fact templates and document formats	90.5%	0.842	16.4%	0.8%	0.944
Negation, hypotheticals, exceptions	99.6%	0.988	0.0%	0.0%	n/a (no review cases)
Embedded instructions / fake system markers	98.5%	0.988	1.5%	1.4%	0.990
Inputs that require needs_review	99.8%	0.993	0.0%	0.0%	0.998

Acceptance criteria (recorded before the test was run)

Criterion	Threshold	Measured	Result
Decision accuracy	≥ 80.0%	96.9%	✅ pass
Missed-confidential rate	≤ 10.0%	5.4%	✅ pass
False alarms on no-match	≤ 10.0%	0.3%	✅ pass
Evidence span F1	≥ 0.750	0.941	✅ pass
Raw output validity	≥ 95.0%	99.9%	✅ pass
needs_review F1	≥ 0.600	0.980	✅ pass
Counterfactual sets correct	≥ 70.0%	94.4%	✅ pass
GGUF Q4_K_M accuracy drop vs. adapter	≤ 3 pts	0.1 pts	✅ pass

Recorded on 2026-10-25 (Claude (AI), delegated by project owner), after the development-set results and before any test result existed. The test set was then evaluated once.

Rationale for the thresholds

Set by Claude (AI) under explicit delegation from the project owner, who was unavailable; recorded before any locked test result was produced. Thresholds apply to metrics pooled over all seven locked test groups, including held-out policy wordings, held-out categories and held-out templates, so they are deliberately below the in-distribution dev results (decision accuracy 0.986, span F1 0.986). They require a clear margin over the dev baselines (keyword 0.486 / base model 0.424 decision accuracy), few missed confidential documents and few false alarms (the costly errors for a review aid), near-perfect raw output validity, meaningful use of needs_review, and policy-sensitivity on counterfactual pairs. The GGUF limit allows at most 3 points of decision-accuracy loss versus the HF model on the same test examples.

How to use

# pip install "transformers>=4.56" peft accelerate jsonschema
# git clone https://huggingface.co/Privoid-ai/Privoid-Confidential-1B  (provides privoid_confidential/)
from privoid_confidential import inference

model, tok = inference.load_model("ibm-granite/granite-4.0-1b", revision="6a7381ba1f54d684ff508d991aeb7dc580157103",
                                  adapter_path="Privoid-ai/Privoid-Confidential-1B", local_files_only=False)
policy = {"id": "acme-sales-v3", "precedence": "exclusions_override_rules", "rules": [{"id": "R1", "category": "negotiated_pricing", "description": "Customer-specific negotiated discounts and pricing terms are confidential."}, {"id": "R2", "category": "unreleased_product_plans", "description": "Plans, timelines, and features of products under development are confidential."}], "exclusions": [{"id": "E1", "description": "Published list prices are not confidential.", "applies_to": ["R1"]}]}
result = inference.classify(model, tok, {
    "policy": policy,
    "trusted_context": [],
    "document": "The Standard plan is $49 per seat. Acme received a private 28% discount for three years.",
})
print(result["output"])   # contract-valid JSON with deterministic start/end offsets
print(result["raw_valid"])  # was the raw generation valid before post-processing?

Input and output contract

Input (one JSON object): policy (trusted: rules R<n> with category + description, optional exclusions E<n>, conditions C<n>, precedence), optional trusted_context (operator-verified facts T<n>), and document (untrusted text, ≤ 6,000 characters).

Output (schema 1.0):

{"schema_version": "1.0",
 "decision": "confidential_under_policy | no_match_under_policy | needs_review",
 "findings": [{"rule_id": "R1", "category": "negotiated_pricing", "evidence_text": "exact substring", "occurrence": 1}],
 "review_reasons": [{"code": "policy_conflict | missing_context | unverified_public_claim | ambiguous_scope", "refs": ["R1", "E1"], "evidence_text": "..."}]}

• no_match_under_policy means nothing matched this policy. It does not mean a document is safe to publish.
• Evidence is copied verbatim from the document; character offsets are computed deterministically by the bundled code, never generated.
• No confidence scores. Claims inside the document ("approved for release", "ignore the policy") are treated as data, not facts or instructions.
• Always validate with contract.validate_output and use contract.postprocess, which never adds findings and turns ungrounded output into needs_review.

Limits of the first release

• English; short passages or bounded sections (tested up to 1,145 tokens of prompt + output; documents ≤ 6,000 characters).
• Structured policies: ≤ 8 rules, ≤ 6 exclusions, ≤ 4 conditions, ≤ 6 trusted facts. Not for unrestricted, lengthy legal policies.
• It cannot verify that something is public: supply that as trusted_context.
• It identifies information; it does not enforce access control or take any action. A locally runnable model is not by itself a secure deployment.

Training data

Fully synthetic and fictional (no real company data), generated by the deterministic, auditable template generator included in privoid_confidential/synth.py (version 1.0.1, seed 42). Labels follow from construction, not from a model. No public dataset with policy-conditioned business-confidentiality labels and evidence spans was found.

• Splits: train 5,156, dev 527, test_familiar 313, test_heldout_policy 308, test_heldout_category 292, test_heldout_template 316, test_negation_exception 250, test_adversarial 261, test_ambiguous 534 examples.
• Covers confidential facts with and without trigger words, public vs. confidential mixes, multiple rules, uncovered categories, irrelevant "secret/private" wording, PII-only text, negation, hypotheticals, quoted emails, repeated evidence, exclusions confirmed by trusted context, rule/exclusion conflicts, missing context, unverified public claims, ambiguous scope, and prompt-injection text.
• Counterfactual pairs keep the document fixed while changing the policy (or vice versa) so the right answer flips.
• Leakage controls: split-disjoint name/filler/phrasing pools; held-out rule wordings, categories, fact templates and document formats; all variants of a scenario stay in one split.
• A stratified sample of 120 labels was reviewed before training (AI pre-review, approved by the project owner; 0 rejected).

Labelling conventions worth knowing: an applicable exclusion with no precedence set is a policy_conflict (needs review), and a document's own claim of being public or approved never counts as proof.

Training procedure

• 5,156 training examples, 646 optimizer steps (2 epochs, effective batch 16), learning rate 0.0002, cosine schedule, bf16, gradient checkpointing.
• Best checkpoint by development loss (0.00058); over-length examples excluded, never truncated (0 excluded).
• Hardware: 1× NVIDIA GeForce RTX 3060; peak GPU memory 6.75 GB allocated.
• Software: transformers 5.5.0, peft 0.20.0, torch 2.11.

Other evaluations

Development set (in-distribution; used for model selection):

System	Decision acc.	Evidence span F1	Raw output valid	Missed confidential
Keyword baseline	48.6%	0.196	100.0%	73.7%
Granite 4.0 1B (base)	42.4%	0.293	2.8%	8.8%
Privoid Confidential-1B	98.6%	0.986	100.0%	0.0%
GGUF Q8_0	98.6%	0.986	100.0%	0.0%
GGUF Q4_K_M	97.9%	0.979	100.0%	0.0%

Out-of-distribution probe on 96 public PII-heavy documents from RedactionBench (CC BY 4.0) under a business-only policy (no PII rules): 89 no match, 6 needs review, 1 confidential. No gold labels exist for this probe; flagged outputs were exported for human review.

Merged checkpoint check: merging the adapter into the base weights reproduced the adapter's greedy outputs on 100% of 9 examples.

Transformers latency (merged bf16, batch 1, ~833-token prompts, NVIDIA GeForce RTX 3060): median 2.77 s, p95 3.68 s; peak 3.52 GB GPU memory.

General-purpose benchmark scores of the base model do not indicate performance on this task and are not reported here.

Limitations and risks

• Trained on synthetic data; real documents are messier. Validate on your own reviewed documents before relying on it.
• Weakest on entirely new confidentiality categories and unseen document formats (see the test-group table).
• Misses are possible: use it as a review aid, not a sole safeguard.
• Results reflect the labelling conventions above; organisations with different conventions should re-evaluate.

Files

• adapter_model.safetensors, adapter_config.json — LoRA adapter (base ibm-granite/granite-4.0-1b @ 6a7381ba)
• privoid_confidential/ — contract, validation, post-processing, scoring, inference helpers, data generator
• evaluation/ — measured results behind every number on this card
• NOTICE, LICENSE

Citation

@misc{privoid_confidential_1b_2026,
  title  = {Privoid Confidential-1B: policy-conditioned confidential information detection},
  author = {Privoid},
  year   = {2026},
  url    = {https://huggingface.co/Privoid-ai/Privoid-Confidential-1B}
}

Base model: IBM Granite 4.0 1B, Apache-2.0.