CollinEvers/fermijn-scan-zoek

SigLIP2 Base patch16-224 — Core ML (fp16)

Core ML conversion of google/siglip2-base-patch16-224 (revision 75de2d55ec2d0b4efc50b3e9ad70dba96a7b2fa2), split into an image encoder and a text encoder, for fully on-device text-to-image search.

• Licence: Apache-2.0 (same as the upstream model). See LICENSE and NOTICE.
• Original authors: Google — SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features (Tschannen et al., 2025, arXiv:2502.14786).
• This is a format conversion only: no retraining or fine-tuning. Not affiliated with or endorsed by Google.

Intended use

Used by the Herken Lab "Zoek" mode in the Fermijn Scan iOS app (admin-only, experimental): the user types a phrase (e.g. "blue vase") and the app ranks regions of the live camera frame by image-text similarity. The model is an opt-in download; everything runs on the iPhone and camera images never leave the device. The app downloads the zip, verifies its SHA-256, unzips it and compiles the .mlpackage files on device with MLModel.compileModel(at:).

Not intended for identifying people, biometric use, or any safety-critical decision. Inherits the limitations and biases of the upstream model (see Google's model card).

Files

Path	Description
image.mlpackage	Vision encoder, Core ML mlprogram (spec v8, iOS 17+), fp16 weights (~185 MB)
text.mlpackage	Text encoder, Core ML mlprogram (spec v8, iOS 17+), fp16 weights (~565 MB)
tokenizer.json	Gemma SentencePiece vocabulary (256,000 pieces + scores) as JSON for a Swift tokenizer. Not a Hugging Face tokenizers file.
config.json	Shapes, I/O names, special-token ids, logit_scale / logit_bias
LICENSE, NOTICE, README.md	Licence, attribution, this card

Input / output

image.mlpackage

	Name	Type	Shape
input	image	Image (RGB, 224×224; BGRA/RGBA CVPixelBuffer accepted)	1×3×224×224
output	embedding	MultiArray Float16, L2-normalised	1×768

Preprocessing is baked in: pixel x → x / 127.5 − 1 (= SigLIP mean 0.5, std 0.5). Upstream resizes the whole frame to 224×224 (bilinear, no crop).

text.mlpackage

	Name	Type	Shape
input	input_ids	MultiArray Int32	1×64
output	embedding	MultiArray Float16, L2-normalised	1×768

Tokenisation must match upstream: Gemma SentencePiece BPE (byte fallback), no BOS, no dummy space prefix, append EOS (1), pad with 0 to exactly 64 tokens (padding="max_length"), no attention mask. Google recommends lower-casing the text, since the model was trained on lower-cased text.

Scoring (SigLIP sigmoid): p = sigmoid(cos(img, txt) * logit_scale + logit_bias) with logit_scale = 112.66889953613281 (already exponentiated) and logit_bias = -16.771724700927734.

Conversion

• Script: scripts/convert_siglip2_coreml.py (Fermijn Scan repo), run on Linux x86_64.
• Toolchain: Python 3.12, torch 2.7.0+cpu, transformers 4.56.2, coremltools 9.0, sentencepiece.
• torch.jit.trace → coremltools.convert(convert_to="mlprogram", minimum_deployment_target=iOS17, compute_precision=FLOAT16).
• The vision pooling head's nn.MultiheadAttention is swapped for an equivalent plain-matmul attention before tracing (coremltools cannot convert the traced aten::Int cast inside nn.MultiheadAttention).
• Checks (on Linux, where Core ML prediction can't run): patched fp32 PyTorch vs. original fp32 gives cosine 0.9999999 (image and text). PyTorch fp16 vs. fp32 gives cosine ≥ 0.999998. Core ML on-device output has not been measured yet.

SHA-256 of the contents

051c889adfb251985dc78fc7a59f42744e3f004bf164761ca02a5965df16bb9c  config.json
f2dabf26affcb7674c5b58aaf582d7cdb3c010a47c374f0b3640e928065cc665  image.mlpackage/Data/com.apple.CoreML/model.mlmodel
6fdf1936f91d40a4b9b4fcaabe5b2dcadfc6d8c0e76ab1cc9e18c38e95d60787  image.mlpackage/Data/com.apple.CoreML/weights/weight.bin
44efd6feb21f62a583ff4edf8c0728a343812f564903f095e45388b5d4adcd8d  image.mlpackage/Manifest.json
68622af73ac6b432d1c78fe7e8c4d61a95142fe492134561b3dfe84c5e5e2f10  text.mlpackage/Data/com.apple.CoreML/model.mlmodel
5de78e1774aebc080a0877afd3db7c6737c51ad9a4a3c8e581f4c9ff53f4f7c8  text.mlpackage/Data/com.apple.CoreML/weights/weight.bin
d7bf057a664a4899223c6ca4d6a30052bdd6edd9f3a80981be5f857acef2a4eb  text.mlpackage/Manifest.json
f5b5d7e0ea07a0f12742e66e7dfae8e3c389929d3fcba1e77f3b5bb07b02411a  tokenizer.json

Download bundle

File	Bytes	SHA-256
siglip2-base-patch16-224-coreml.zip	703534586 (703.5 MB)	84d2ba93cdbfdb421dd8126ba6e268e8cd7d1cccb9c4b061c605834db9eed2b7

Download: https://huggingface.co/CollinEvers/fermijn-scan-zoek/resolve/main/siglip2-base-patch16-224-coreml.zip

The zip holds image.mlpackage/, text.mlpackage/, tokenizer.json, config.json, LICENSE, NOTICE and README.md at its root (deflate, no ZIP64).

Citation

@misc{tschannen2025siglip2,
  title  = {SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features},
  author = {Michael Tschannen and Alexey Gritsenko and Xiao Wang and Muhammad Ferjad Naeem and Ibrahim Alabdulmohsin and Nikhil Parthasarathy and Talfan Evans and Lucas Beyer and Ye Xia and Basil Mustafa and Olivier H{\'e}naff and Jeremiah Harmsen and Andreas Steiner and Xiaohua Zhai},
  year   = {2025},
  eprint = {2502.14786},
  archivePrefix = {arXiv}
}