PrimeNeural/pnn-vit-b16-v41-d50

PNN ViT-B/16 — V4.1 connectivity, 50 % of the weights

© PreciseStatistics GmbH (Prime Neural Network). Proprietary weights, published for evaluation. Loading requires the PNN evaluation or licence package, version V4.1 or later.

A Prime Neural Network (PNN) version of ViT-B/16. The 48 linear layers of the 12 transformer blocks (attention qkv and projection, MLP fc1 and fc2) are PNN layers: each output reads a fixed, structured half of its inputs. The patch embedding, position embeddings, layer norms and the classification head stay dense.

Model at a glance

Base model	vit_base_patch16_224.augreg_in21k_ft_in1k (timm): 86.6 M parameters, 84.51 % top-1
PNN connectivity	V4.1, uniform density 50 % on all 48 converted layers
Parameters	44.1 M (51 % of the base model)
ImageNet-1k top-1 (validation, all 50,000 images)	78.47 % fp32 · 78.13 % int8 (W8A8) · 74.87 % int4 (W4A8)
Input	224 × 224 RGB, the base model's preprocessing (resize, center crop, ImageNet mean/std)
Output	1,000 ImageNet class logits

Size against the original

Model	Weights	File size	vs. the dense original	ImageNet-1k top-1
Dense original (timm ViT-B/16, fp32)	86.6 M	346 MB	1×	84.51 %
PNN 50 %, fp32 (best.pt)	44.1 M	176 MB	2.0× smaller	78.47 %
PNN 50 %, int8	44.1 M	49 MB	7.0× smaller	78.13 %
PNN 50 %, int4	44.1 M	28 MB	12.3× smaller	74.87 %
PNN 50 %, 1-bit (separate repo)	44.1 M	8.9 MB	39× smaller	71.19 %

The accuracy cost of the smaller files is in the last column and under "Limitations".

Speed on CPUs

The 48 PNN layers of one image (197 tokens per call), one thread, on the PNN V4.2 CPU runtime. Ratios, each from a single run on that machine:

CPU (kernel)	int8 weights vs fp32	4-bit weights: time relative to int8
Apple M1 (NEON)	1.95× faster	0.95
AMD EPYC 9555P, Zen 5 (AVX-512)	1.97× faster	0.97
Arm Neoverse-N1 (NEON)	1.49× faster	1.11
Intel Core (AVX2)	1.29× faster	1.05

On the M1: fp32 408 ms, int8 209 ms, 4-bit 200 ms per image for these layers. On CPUs, 4-bit weights run at about int8 speed and halve the weight memory; the runtime measures both per machine and keeps the faster. The rest of the network (attention, layer norms, patch embedding, head) is not included in these times.

Files

Path	Format	Size	Use
int8/model.safetensors + int8/config.json	int8 weights (symmetric, one scale per output row) with static int8 input scales	49 MB	integer inference, CPU runtime
int4/model.safetensors + int4/config.json	int4 weights (signed, two per byte, one scale per output row) with static int8 input scales	28 MB	smallest file; needs PNN V4.2 or later
bf16/model.safetensors + bf16/config.json	bfloat16 weights	88 MB	GPU inference, fine-tuning
best.pt	PyTorch checkpoint, fp32 (research layout)	176 MB	the original training checkpoint
history.json	top-1 after reconstruction and each epoch	—	—

The int8/, int4/ and bf16/ folders use the PNN model file format (pnn-safetensors/1): safetensors + config.json, with quantization_config in the compressed-tensors vocabulary (quant_method: "pnn") and a pnn block that records each layer's widths, density and connectivity version. Each PNN layer stores only the weights it has, shape (outputs, connections); the connections themselves are not stored. The PNN package recomputes them when the model is loaded.

config.json also states the encoding in words, so no compressed-tensors knowledge is needed: pnn.requires_pnn is the minimum PNN package version (V4.1); pnn.weight_encoding and pnn.activation_encoding give sign, range, zero point and scale. For the int8 files: weights are signed int8 in [−127, 127], zero point 0, one fp32 scale per output row; inputs are signed int8, zero point 0, one static fp32 scale per layer. For the int4 file: weights are signed int4 in [−8, 7], zero point 0, one fp32 scale per output row, stored as q + 8 in the compressed-tensors pack-quantized layout (eight per int32 word); inputs as for int8. Its requires_pnn is V4.2.

Loading

import timm
from pnn_circulant import PNNLinearCirculant        # PNN package
from pnn_quant import load_pnn_safetensors

model = timm.create_model("vit_base_patch16_224", pretrained=False)
for name, m in list(model.named_modules()):          # swap the 48 block linears for PNN layers
    if name.endswith(("attn.qkv", "attn.proj", "mlp.fc1", "mlp.fc2")):
        parent, leaf = name.rsplit(".", 1)
        setattr(model.get_submodule(parent), leaf,
                PNNLinearCirculant(m.in_features, m.out_features, 0.5))
f = load_pnn_safetensors("bf16", model)                # or "int8": dequantised on load

For the int8 model, f.input_scales holds each layer's static input scale. The model was measured with every PNN layer's input quantised as clamp(round(x / scale), −127, 127).

Training

• Conversion: least-squares reconstruction of each PNN layer from the base model's layer, followed by 10 epochs of fine-tuning on ImageNet-1k.
• Recipe: batch 256, AdamW lr 1e-4, weight decay 0.05, cosine schedule with 1 warm-up epoch; distillation from the base model (α 0.5, T 2). One NVIDIA H100, about 18.5 minutes per epoch.
• int8: post-training quantisation without retraining. Weights are symmetric per output row. Activations are symmetric per layer with a static scale at the 99.999th percentile of |x| on 1,024 training images.
• int4: post-training quantisation without retraining. Weights are symmetric 4-bit per output row; activations are int8, as for the int8 file.

Limitations

• At 50 % of the weights this model is 6.0 points below its dense base model (78.47 % vs 84.51 %) after 10 epochs. Longer training or distillation may close part of the gap; that has not been measured.
• Accuracy is measured on the ImageNet-1k validation split only.
• The layers need the PNN package; standard PyTorch, timm or transformers loaders cannot build them. quant_method: "pnn" stops standard loaders from misreading the files.

Contact

Prime Neural Network · primeneural.net · info@primeneural.net