LocalAI-io/privacy-filter-GGUF

privacy-filter — GGUF (F16)

GGUF conversion of openai/privacy-filter, OpenAI's bidirectional PII token-classification model. It labels every token with a BIOES tag over 8 PII categories (33 classes) in a single forward pass, then decodes coherent spans with a constrained Viterbi procedure — so it can be served locally with no Python as the encoder/NER tier of a PII redactor.

For the full model description, training, evaluation, operating points, limitations, and citations, see the source model card — this card only covers the GGUF packaging and how to run it.

For broader language coverage (54 categories across 16 languages), see the multilingual fine-tune privacy-filter-multilingual GGUF.

Runtimes

This GGUF uses a custom architecture, openai-privacy-filter, that is not (yet) part of upstream llama.cpp. It runs on:

1. privacy-filter.cpp (recommended) — a small standalone GGML engine for exactly this model family, on stock upstream ggml with no patches (CPU / CUDA / Vulkan). This is the reference runtime.

   # build (see the repo README for CUDA/Vulkan)
   cmake --preset release && cmake --build --preset release -j
   # run
   echo "My name is Alice Smith" | \
     build/release/pf-cli --classify privacy-filter-f16.gguf 0.5
   

   It exposes a flat C API (pf_load / pf_classify → entity spans with UTF-8 byte offsets; pf_tokenize / pf_logits) shaped for FFI — see the repo README.

2. LocalAI — install from the model gallery; LocalAI serves it behind the gRPC TokenClassify RPC and runs the constrained BIOES Viterbi decode, returning entity spans. LocalAI drives it through the privacy-filter backend (which wraps privacy-filter.cpp). The model is not a chat/completion model — it is a PII detector that other models opt into via a pii.detectors list.

3. llama.cpp — only with a patch. Stock llama.cpp, llama-cpp-python, Ollama, and LM Studio will fail to load this file (unknown model architecture: 'openai-privacy-filter'). The arch can be added with carry-patches (TOKEN_CLS pooling, the architecture + HF→GGUF converter, the bidirectional banded-attention graph, and an all-SWA no-cache mask fix; TOKEN_CLS pooling tracks the still-open PR #19725). Until that support lands upstream, privacy-filter.cpp above is the patch-free alternative.

Pooling note (llama.cpp path only): the model must be loaded with TOKEN_CLS pooling (the GGUF's default). If you drive llama-embedding directly for testing, do not pass --pooling none. privacy-filter.cpp handles this automatically.

Files

File	Precision	Size	Notes
privacy-filter-f16.gguf	F16	2.82 GB	156 tensors; 33 classifier.output_labels; pooling_type = TOKEN_CLS.

sha256: eb71312b6b9370d0fe582e576b840567bb06603c4de241c6d899205d1b04dc81

F16 is the validated, shipped precision. Quantized variants are deferred until they can be evaluated with a task metric (span-F1) + KL-vs-F16 — perplexity is meaningless for a classifier, so a naively-quantized GGUF is not published here yet.

Architecture & conversion

gpt-oss-style sparse MoE (8 layers, d_model=640, 128 experts, top-4 routing; ~1.5B total / ~50M active per token), bidirectional banded attention (symmetric sliding window, attention sinks retained), interleaved (GPT-J) RoPE with YaRN (θ=150000, factor 32), o200k (o200k_base) tokenizer, and a 33-way token-classification head (score → cls.output). privacy-filter.cpp re-derives the YaRN truncate=false frequencies at load time (fed to ggml_rope_ext as freq_factors) so the GGUF is interchangeable across runtimes.

Label space

O plus B-/I-/E-/S- for each of 8 categories (1 + 8×4 = 33): account_number, private_address, private_date, private_email, private_person, private_phone, private_url, secret. The ordered id2label table is embedded in the GGUF (classifier.output_labels).

Limitations & intended use

Identical to the source model: trained for high-throughput data sanitization, not a substitute for legal/compliance review, and not a clinical PHI model. Use it as one tier behind deterministic regex pre-filters and human review. For multilingual text, prefer the multilingual fine-tune.

License

Apache-2.0, inherited from openai/privacy-filter.

Credits & citation

Model by OpenAI (openai/privacy-filter). GGUF conversion and runtime support (privacy-filter.cpp) by the LocalAI project. Please cite OpenAI per the source card.