privacy-filter-multilingual-v2-GGUF

GGUF conversion of OpenMed/privacy-filter-multilingual-v2 for the privacy-filter.cpp runtime, with the Mixture-of-Experts weights requantized to Q8_0.

Files

file	bytes	sha256
pf-multilingual-v2-q8.gguf	1,637,811,520	5e1a6121eff570febfca19c6c54641e0f84e9f11f5f24897c596207e31c5d074

How it was produced

Converted from OpenMed/privacy-filter-multilingual-v2 at revision 0d0c0430fa386435ace1d9f842f0f966903a9ac8, using the conversion scripts from privacy-filter.cpp at commit 735a6c28607ee82afc3a670383f41b55266a3b9a.

Step 1 - Hugging Face checkpoint to f16 GGUF, with scripts/convert.py:

hf download OpenMed/privacy-filter-multilingual-v2 --local-dir pf-src

python scripts/convert.py \
    --model pf-src \
    --outfile pf-multilingual-v2-f16.gguf \
    --outtype f16

convert.py is self-contained: it reads config.json, model.safetensors and tokenizer.json directly and does not depend on llama.cpp or its conversion script. The f16 output is 2,817,459,424 bytes.

Step 2 - requantize the expert weights to Q8_0, with scripts/requant_q8.py:

python scripts/requant_q8.py \
    --in pf-multilingual-v2-f16.gguf \
    --out pf-multilingual-v2-q8.gguf

Environment. torch (CPU build is sufficient), safetensors, and gguf>=0.10, as pinned by scripts/requirements.txt. Roughly 30 seconds for the conversion and 60 seconds for the requant on a CPU-only machine.

The upstream project's own ci.yml converts checkpoints to f16 and f32 on its nightly tier and gates them against reference fixtures; it does not run the Q8_0 requant, so step 2 above is the part that is not covered by upstream CI.

Architecture

The checkpoint is the openai/privacy-filter body (a gpt-oss style MoE) repurposed as a bidirectional token classifier. As recorded in the GGUF metadata:

key	value
general.architecture	openai-privacy-filter
block_count	8
embedding_length	640
attention.head_count / head_count_kv	14 / 2
expert_count / expert_used_count	128 / 4
context_length	131072
tokenizer.ggml.model / pre	gpt2 / gpt-4o
classifier.output_labels	217 BIOES classes

Notes

• The requant is partial. Only tensors matching ffn_gate_exps.weight, ffn_up_exps.weight and ffn_down_exps.weight (the MoE expert weights) become Q8_0. Every other tensor is copied verbatim and stays f16. This is what makes the file 1.64 GB rather than the 2.8 GB f16 intermediate, and it is the configuration the int8 mul_mat_id kernel is fastest on.
• general.file_type still reports f16 (value 1). requant_q8.py copies metadata fields verbatim, so the declared file type is not updated to reflect the Q8_0 expert tensors.
• gguf-py cannot reopen these files. The reader raises KeyError: Duplicate GGUF.version already in list. The cause is not a duplicate in the file: requant_q8.py copies the reader's synthetic fields into the output as real metadata, so the file carries literal GGUF.version, GGUF.tensor_count and GGUF.kv_count entries that collide with the ones gguf-py injects when reading. llama.cpp and privacy-filter.cpp both load the files normally. The published LocalAI-io q8 files share this quirk.