burner22

/

numbat

Like 202

The author is full of shit. The NVFP4 matrices are generated straight from their tensor names, and together with their block scales, that's roughly 99.6% of the checkpoint.

Use the exact same seed setup as #13: first 8 bytes of SHA-256(name), big-endian, into NumPy's PCG64. If you draw z1 and z2 with the matrix shape and z3 with one value per row, the weights are just this:

w = bf16(z1 * exp(0.4z2) * s * max(1 + 0.12z3, 0.7))

then run that through standard NVFP4 quantization.

Checking just the signs against z1 across 5,503 tensors (from 770 of the 772 shards) gave 162,246,496 out of 162,246,496 matches. When I rebuilt 10 tensors from layers 2 to 108, 465,566,255 out of 465,567,744 4bit codes came out identical. All norms and convs match the recipe from #13, and lm_head is just 0.891 * embed_tokens, even though the config claims they're untied.

the embedding table is pretty much the only part of this entire file with actual language structure in it and that's like 0.04% of the bytes. + the BF16 dense and router matrices follow the exact same noise distribution. I mean either way, this definitely isn't a trained model lol

Script that pulls about 20 MB from shard 346 and rebuilds one of the experts so you can verify it yourself:

import hashlib, json, struct
import numpy as np, requests

REV = "6b7f8fc548cbafe9bec441f03af2d92a55cb4012"
URL = f"https://huggingface.co/burner22/numbat/resolve/{REV}/model-00346-of-00772.safetensors"
NAME = "model.layers.50.mlp.experts.7.gate_proj.weight"
S = np.float32(0.006558356340974569)  # per-tensor scale, the only fitted constant
f = np.float32

def get(a, b):
    r = requests.get(URL, headers={"Range": f"bytes={a}-{b}"}, timeout=120)
    assert r.status_code == 206 and len(r.content) == b - a + 1
    return r.content

n = struct.unpack("<Q", get(0, 7))[0]
hdr = json.loads(get(8, 7 + n))
def tensor(k):
    s, e = hdr[k]["data_offsets"]
    return get(8 + n + s, 8 + n + e - 1)

p = np.frombuffer(tensor(NAME), np.uint8)
sb_file = np.frombuffer(tensor(NAME[:-6] + "weight_scale"), np.uint8)
ws2 = np.frombuffer(tensor(NAME[:-6] + "weight_scale_2"), "<f4")[0]
O, I = hdr[NAME]["shape"][0], hdr[NAME]["shape"][1] * 2
codes = np.empty(p.size * 2, np.uint8); codes[0::2] = p & 15; codes[1::2] = p >> 4
codes = codes.reshape(O, I); sb_file = sb_file.reshape(O, I // 16)

g = np.random.Generator(np.random.PCG64(int.from_bytes(hashlib.sha256(NAME.encode()).digest()[:8], "big")))
z1 = g.standard_normal((O, I), dtype=f)
z2 = g.standard_normal((O, I), dtype=f)
z3 = g.standard_normal(O, dtype=f)

nz = (codes & 7) > 0
print("signs matching z1:", int((((codes & 8) > 0) == (z1 < 0))[nz].sum()), "/", int(nz.sum()))

w = z1 * np.exp(f(0.4) * z2) * (S * np.maximum(f(1) + f(0.12) * z3, f(0.7)))[:, None]
u = w.view(np.uint32)
w = (((u + 0x7FFF + ((u >> 16) & 1)) >> 16) << 16).astype(np.uint32).view(f)  # to bf16

e4 = np.array([(1 + m / 8) * 2.0 ** (e - 7) if e else m / 8 * 2.0 ** -6 for e in range(16) for m in range(8)][:127], f)
bmax = np.abs(w).reshape(O, I // 16, 16).max(-1)
sb = np.minimum(np.searchsorted(e4, bmax / f(6) / ws2), 126).astype(np.uint8)
x = np.abs(w / (np.repeat(e4[sb], 16, 1) * ws2))
c = np.minimum(np.searchsorted(np.array([.25, .75, 1.25, 1.75, 2.5, 3.5, 5], f), x), 7).astype(np.uint8)
c = np.where(w < 0, c | 8, c)

print("block scales reproduced:", int((sb == sb_file).sum()), "/", sb.size)
print("4-bit codes reproduced:", int((c == codes).sum()), "/", c.size)

Output:

signs matching z1: 31134993 / 31134993
block scales reproduced: 2162684 / 2162688
4-bit codes reproduced: 34602991 / 34603008

so bro's just wasting HF's storage atp

I am against taking down models but if there's a case where it is justified it is this one. A waste of resources and time.

Great work actually ruling it out instead of yelling "fake" with no evidence. You saved my NAS from wasting its day downloading this garbage.