Canonical Antfly layout (2026-10-17)

This repository follows the upstream Fastino checkpoint name. The previous -gguf repository URL redirects here. The paired filenames identify the Q8_0 bundle variant: the encoder matrices are Q8_0, while the auxiliary head remains FP32 despite its Q8_0 filename suffix. The weight bytes, tokenizer, and source configuration have not changed. The bundle descriptor and antfly_inference_variants.json name both required components explicitly. Historical validation reports retain the original paths and measurements; gliner_head.gguf in those reports is now gliner2-head.Q8_0.gguf.

This exact Q8 bundle has independent Native CPU and physical Metal classification qualification for one request item, at most 64 prepared sequences, and at most 1024 complete prompt tokens per sequence (65,536 request tokens at the Cartesian maximum). The expanded runtime contract is implemented in Antfly commit 10e6c371d5. Earlier Antfly builds retain their 198-token contract; the model rename does not change an installed runtime's limits. This is Q8 evidence, separate from the FP32 qualification. See the checksummed capacity receipts. Native's exact maximum used a 16 GiB operator process envelope. Metal's exact maximum used 24 GiB after admission refused it at 16 GiB. Physical-memory admission and emergency reserves remained active; the envelopes are ceilings, not measured RAM consumption. Classification quality is separate from parity: the small corpus preserves upstream's 7/13 supersession and 0/8 concept-triage answers, so these receipts are not a general application-quality endorsement.

Entity and relation extraction are not advertised: the pinned upstream FP32 span heads produce empty results in the retained extraction investigation. Long-document graph extraction and ModernBERT fine-tuning remain unsupported for this artifact. See the investigation and issue #1083.

The runtime patch and commands below document the original qualification. Current Antfly main includes that runtime integration; do not reapply the historical patch to current main. For exact reproduction against the old base, use the pre-migration repository revision or the original head filename.

GLiNER2.5-Decide-1B — Q8_0 GGUF

Q8_0 conversion of Fastino's GLiNER2.5-Decide-1B, validated for classification with Antfly on Apple Silicon. The complete weights occupy 1,735,552,448 bytes (1.74 GB / 1.62 GiB), 63.5% less than the original FP32 checkpoint.

This is an Antfly split GGUF bundle with a ModernBERT encoder. Download the entire repository: the encoder, auxiliary head, tokenizer, configurations, and bundle marker are required together. Compatibility with llama.cpp, Ollama, and other GGUF runtimes has not been established.

Precision and provenance

• All 113 two-dimensional encoder matrices, including token embeddings: Q8_0, 32-value blocks.
• Encoder normalization parameters, decision classifier, and auxiliary heads: FP32.
• Original tensor names and shapes are preserved; no transposition, pruning, or calibration-data tuning.
• All 199 tensors / 1,188,796,693 parameters are retained. The decision classifier is stored in the encoder file for runtime inspection.
• Source revision: 688cd7ba8917a0855ad3ce929cba5a9998932e79.
• Source model.safetensors SHA-256: 02c567d791aed26550d300064c7f0c0094fd65291503c65969b45b30786e33b3.
• The original tokenizer and configuration sidecars are copied byte-for-byte. Both local and global RoPE use theta 160000.

The upstream configuration was subsequently restructured. This conversion pins the earlier configuration compatible with the validated runtime; it does not combine configuration files from different revisions.

conversion.json records every tensor, precision, output digest, and reconstruction error. convert.py reproduces the conversion using the pinned packages in requirements-convert.txt:

python convert.py --source /path/to/pinned-source --output /path/to/new-bundle

Quality checks

• 14 frozen classification cases: identical selected labels after Q8 conversion; maximum probability change 0.00341 against source FP32.
• Public development sample: 170 examples across 17 domains, comprising 290 classification tasks. Q8 and FP32 agree on 289/290 (99.66%) decisions; each scores 184/290 (63.45%) exact task accuracy. No measured accuracy loss on this sample.
• Sampling was fixed before evaluation: zero-based rows 0, 10, …, 90 from each domain in fastino/fast-decisions revision 35afb79a0f611a1db44380e855ac2a821b0a4da1. Multi-label tasks use threshold 0.5. These are development-sample results, not the upstream private test benchmark.
• The larger sample uses FP32 CPU computation with the exact GGUF weights decoded. Direct runtime and HTTP validation are reported separately in performance.json.

The Python reference environment and RoPE contract are pinned in oracle-runtime.json. See validation.json and evaluation.json for the recorded comparisons. Classification is the validated task here; retaining auxiliary head weights does not establish quality for other extraction tasks.

Apple Silicon execution

Measured on an Apple M4 with 16 GiB RAM, using two CPU threads. The same Q8 artifact was compared with direct quantized execution and reusable FP16 Metal matrices in three alternating, paired runs. The table reports the median of three process medians; each process measured ten requests per case after warmup.

Prepared tokens	Direct Q8 (ms)	Optimized Q8 (ms)
40	150.0	87.5
79	138.8	129.8
87	145.9	127.8
93	144.8	129.9
158	203.1	188.7
174	203.7	200.7

The strongest improvement is on the short 40-token case (about 42% lower latency). Longer requests show smaller gains; individual paired runs include regressions. This was an active shared Mac, and the separate FP32 comparison also varied substantially. Raw samples, run order, binary hashes, memory observations and all completed comparisons are in performance.json. Do not infer a fixed speedup for every request.

Both Native CPU and Metal passed all 14 frozen requests through the actual HTTP server with the backend explicitly required. On Metal, the first request took 948 ms including GPU matrix preparation; subsequent requests in that qualification sweep took 69–225 ms. Those individual HTTP observations are separate from the repeated pipeline measurements above.

Across the recorded optimized Q8 runs, 948 requests completed across nine processes without an observed kernel failure. One earlier direct-Q8 control hit the 120-second watchdog; three later paired controls completed. The cause remains unresolved and the failed run is retained in the report. The recommended default is the optimized path. Peak process footprint was approximately 2.5 GB in the optimized runs; this is the OS-reported footprint, not a complete accounting of all shared file-cache residency.

The optimized path prepares reusable FP16 Metal matrices from the Q8 encoder weights at first use. The download remains Q8; warmed computation uses the faster FP16 matrix path with FP32 activations and normalization. This increases runtime memory relative to direct Q8 execution. The 112 encoder matrices consume 1,875,378,176 bytes in FP16; per-matrix temporary staging is capped at 64 MiB. TERMITE_METAL_MODERNBERT_1B_F16_MIRRORS=0 selects the direct path for controlled comparison.

Runtime requirement and usage

This bundle requires Antfly base revision 97c5dc3f8725978c75dd737840d562df81c85a4a plus the included antfly-runtime.patch. The patch adds exact split-artifact qualification and the measured Metal execution policy. It is supplied here because these changes are not yet an upstream Antfly release. It applies cleanly to the pinned base and includes seven focused tests and the standalone benchmark. Tested toolchain: Zig 0.17.0 on macOS 26.5. See provenance.json.

The qualified server contract supports Native CPU and Metal, one request / one prepared sequence, and at most 198 prepared tokens, including schema and text. The model's architectural 7999-token capacity is not a validated serving limit. CUDA, batching, longer requests, and browser execution are unqualified for this artifact.

Download the complete bundle into the runtime's model directory:

hf download antflydb/GLiNER2.5-Decide-1B \
  --local-dir ./models/antflydb/GLiNER2.5-Decide-1B

From a clean Antfly checkout at the pinned base, with its build prerequisites installed, download the model as above, then:

MODEL_ROOT="$PWD/models"
git apply "$MODEL_ROOT/antflydb/GLiNER2.5-Decide-1B/antfly-runtime.patch"
(cd zig && python3 tools/run_bounded_zig_build.py build \
  inference-bench-server inference-bench-gliner-decide-quant-build \
  inference-test-gliner-decide-quant \
  -Doptimize=fast -Dmetal=true -Dcuda=false -j1)
ANTFLY_INFERENCE_REQUIRED_BACKEND=metal \
ANTFLY_INFERENCE_CPU_THREADS=2 VECLIB_MAXIMUM_THREADS=2 \
./zig/zig-out/bin/antfly-inference-bench-server run \
  --host 127.0.0.1 --port 18769 --models-dir "$MODEL_ROOT" \
  --max-loaded-models 1 --max-concurrent-requests 1 \
  --process-memory-budget-mb 8192 --host-budget-mb 6144 \
  --backend-budget-mb 4096 --combined-budget-mb 8192 \
  --preload-model extractor:metal:antflydb/GLiNER2.5-Decide-1B

The build target above uses the production HTTP server. The explicit memory limits are the configuration tested on this 16 GiB Mac. For the validated CPU path, change both backend selections from metal to native.

In another terminal:

curl http://127.0.0.1:18769/ai/v1/decide \
  -H 'Content-Type: application/json' \
  -d '{"model":"antflydb/GLiNER2.5-Decide-1B","state":"The package arrived damaged.","questions":{"refund":{"type":"choice","instructions":"Choose the action.","criteria":{"refund":"Refund the order","keep":"Keep the order"}}}}'

The validated choice for this example is refund. benchmark-cases.json and benchmark-six.json contain the exact frozen inputs, prepared token IDs, and FP32 reference logits for reproducing the standalone benchmark.

License

Apache-2.0, inherited from the upstream model. See LICENSE. Original model and training work: Fastino. Conversion, validation, and Antfly integration: Antfly.