EvidenceMem-SigLIP2

EvidenceMem classifies images into 11 visual categories using a frozen SigLIP2 encoder and a compact memory of representative training-image embeddings. It returns a class, class probabilities, and the IDs of retrieved prototypes. This is a memory-based classifier built on SigLIP2, not a new ViT architecture or a fine-tuned backbone.

Install and classify an image

Requires Python 3.11 or newer. Download this model repository and install its included Python package. The source is included here so deployment does not depend on an unreleased GitHub change.

python -m pip install "huggingface_hub>=0.34,<1"
hf download Prathamesh333/EvidenceMem-SigLIP2 --local-dir EvidenceMem-SigLIP2
cd EvidenceMem-SigLIP2
python -m pip install ".[deployment]"

from evidencemem.image_inference import EvidenceMemImageClassifier

model = EvidenceMemImageClassifier.from_pretrained(".", seed=7, device="cpu")
result = model.predict_images("your-image.jpg")
print(result["labels"][0])
print(result["probabilities"][0])
print(result["source_indices"][0])

After installation, the loader also accepts Prathamesh333/EvidenceMem-SigLIP2 instead of a local folder. Pass a Hub commit hash through revision= to pin the memory release. Use device="cuda" for a CUDA GPU. The first image inference downloads the separate SigLIP2 weights, about 1.5 GB. No image dataset is required.

How it works

1. Convert the image to RGB and apply SigLIP2's pinned slow image processor at 384 × 384.
2. Encode it as a normalized 768-dimensional vector without updating the encoder.
3. Retrieve nearby prototypes separately within each class and calculate visual scores.
4. Combine visual and saved class-text scores using the selected reliability-dependent rule.
5. Apply the saved temperature and return the highest-probability class.

Each checkpoint stores 440 prototypes: 40 per class. Each prototype refers to a training row. prototype_indices addresses rows of that checkpoint's memory; source_indices addresses the original training order. Neither field contains image pixels or a URL. Retrieved examples are supporting references, not causal explanations.

The output class order is: apparel, artwork, cars, dishes, furniture, illustrations, landmark, meme, packaged, storefronts, toys. This is closed-set classification: an image outside these categories still receives one of these labels.

Evaluation

The experiment uses a balanced 22,000-image subset of the 512 × 512 Universal Image Embeddings dataset. The split contains 13,200 training, 2,200 validation, 3,300 development, and 3,300 confirmatory images. The encoder processes the source images at 384 × 384.

Memory seed	Confirmatory accuracy
7 (default)	95.0303%
17	94.9394%
29	94.8182%
43	95.2121%
61	94.6364%
Mean across five seeds	94.9273%

These are five memories evaluated on the same 3,300 images, not five independent test sets. The mean is not the accuracy of the default checkpoint. Seed 7 is the first predefined seed, not the best-performing test seed. Seeds 29 and 61 assign zero weight to text under their selected settings. No settings were refitted for this release.

Recovered memories and T4-regenerated class-text vectors reproduce every archived class prediction for all five seeds. The largest absolute probability difference is 0.000007144, below the fixed 0.00002 verification tolerance. See verification_report.json. Private evaluation embeddings and images are not included.

What has and has not been tested

• Saved-embedding inference: all five checkpoints, all 3,300 confirmatory queries each.
• Image adapter: CPU float32 smoke test on synthetic RGB and grayscale images, compared with direct encoder preprocessing and scoring. See image_smoke_report.json.
• CUDA mode uses float16, as in the experiment. The new end-to-end image adapter has not been rerun on the full image benchmark or independently validated on a GPU.
• CPU and GPU image embeddings can differ numerically. The table is not a fresh CPU accuracy measurement, and the smoke test is not an accuracy benchmark.
• The release does not establish out-of-distribution reliability, demographic fairness, calibrated confidence on new domains, or suitability for high-stakes decisions.
• This repository hosts downloadable artifacts. It does not provide a hosted prediction service or standard transformers.pipeline integration.

Files and reproducibility

Each seed-* folder contains memory.npz, text_vectors.npy, and config.json. The loader checks their SHA-256 hashes and uses NumPy loading without pickle. The included source implements the original scoring primitives; it does not execute remote model code. SHA256SUMS.json records the release file hashes.

The encoder and processor are pinned to google/siglip2-base-patch16-384 revision f775b65a79762255128c981547af89addcfe0f88. The original experiment did not record its exact Transformers version or backbone revision. Upstream history supports this pin; the independent saved-prediction comparison provides the numerical recovery check. Text recovery used Transformers 4.57.6 and PyTorch 2.10.0+cu128 on a Tesla T4.

Licenses and project source

The included EvidenceMem code is MIT-licensed. The separately downloaded Google SigLIP2 model is Apache-2.0-licensed. The dataset publisher lists CC0 for its dataset; that statement does not independently establish the rights of every underlying image. This release does not redistribute source-image pixels. Users must assess applicable data and model terms for their intended use.

Research code and experiment documentation: Prathmesh333/evidencemem.