It is FAKE
This is not real Anthropic model
Checked the model config, layout, architecture etc. This is well made fake, nobody will be able to run it because of the size. Probably price manipulation attmpt before Anthropic IPO.
Found issues with the model:
• Reversed arguments: in the GDN layer, o_proj = linear(hidden, H*V) passes input and output sizes the wrong way round. It only works because 48 × 256 = 12288 = hidden_size.
• Ignored parameter: NumbatMoE(config, quantized=...) never uses quantized; experts are always NVFP4.
• Weight init does nothing: _init_weights is a no-op. Any tensor missing from a non-strict load keeps whatever was in memory from torch.empty (the router weight, for example).
• Padding produces NaN: with left padding, the attention mask turns padded query rows entirely to -inf, so the outputs at those positions are NaN.
• Unusable for real inference: there's no KV cache, the GDN layer loops over tokens and the MoE layer loops over experts, all weights are dequantized every call, and everything assumes one device. Running it on ~7.7 TB of weights isn't realistic. It works as a readable spec of how the file layout maps to the model, and it's tidy, correctly written code. That fits a careful fake: its job is to make load_state_dict(strict=True) succeed and look legitimate, not to run the model.
Yeah, they even added noise to the initialized weights that could make a naive test assume they were actually trained.
Atericparker, have you actually checked the weights? They are not noise
Found issues with the model:
• Reversed arguments: in the GDN layer, o_proj = linear(hidden, H*V) passes input and output sizes the wrong way round. It only works because 48 × 256 = 12288 = hidden_size.
• Ignored parameter: NumbatMoE(config, quantized=...) never uses quantized; experts are always NVFP4.
• Weight init does nothing: _init_weights is a no-op. Any tensor missing from a non-strict load keeps whatever was in memory from torch.empty (the router weight, for example).
• Padding produces NaN: with left padding, the attention mask turns padded query rows entirely to -inf, so the outputs at those positions are NaN.
• Unusable for real inference: there's no KV cache, the GDN layer loops over tokens and the MoE layer loops over experts, all weights are dequantized every call, and everything assumes one device. Running it on ~7.7 TB of weights isn't realistic. It works as a readable spec of how the file layout maps to the model, and it's tidy, correctly written code. That fits a careful fake: its job is to make load_state_dict(strict=True) succeed and look legitimate, not to run the model.
This reads like AI slop. It mixes genuine implementation issues with incorrect generalizations.
The reversed projection dimensions and ignored quantized flag are real. Neither breaks the posted configuration, since the projection is square and the MoE experts are supposed to be NVFP4.
The NaN claim is not generally true. Modern PyTorch’s math SDPA returns zero attention outputs for fully masked rows. Its regression test checks exactly this. Ordinary softmax on an entirely -inf row produces NaNs, but assuming SDPA necessarily does the same is the mistake. Older versions and other backends can behave differently.
The claim that every missing tensor contains uninitialized memory is also wrong. The router’s raw torch.empty allocation is a legitimate concern, but under normal eager construction, ordinary PyTorch layers initialize themselves, and several custom tensors explicitly use zeros or ones. Fast-init and meta loading need separate consideration.
“All weights are dequantized every call” is wrong too. Invoked quantized layers dequantize their weights, unselected experts are skipped, and dense weights need no dequantization.
The lack of caching, Python loops, and missing distributed runtime are legitimate performance criticisms. They do not establish that the weights are fake or that the implementation was deliberately written to deceive people.
None of this proves the checkpoint is genuine, either. It means the evidence presented does not support that conclusion about fakery.
If you want to debunk the model, I have no issue with that. But for the love of God, at least get the technical claims right. Mixing real problems with inaccurate claims and speculation about intent does not make a convincing debunk.
If it was real it would be down already
There's actually a more direct way to investigate this than inspecting the loader. Discussion #13 reports that 43 complete tensors (730,368 BF16 values) can be regenerated bit-for-bit using a PRNG seeded from their tensor names. An independent check also reproduced all 12,288 values of another tensor. That's considerably stronger evidence of synthetic weights than the implementation issues discussed here. It still doesn't establish that every tensor is synthetic, since the bulk of the NVFP4 expert weights hasn't been independently verified.
There's actually a more direct way to investigate this than inspecting the loader. Discussion #13 reports that 43 complete tensors (730,368 BF16 values) can be regenerated bit-for-bit using a PRNG seeded from their tensor names. An independent check also reproduced all 12,288 values of another tensor. That's considerably stronger evidence of synthetic weights than the implementation issues discussed here. It still doesn't establish that every tensor is synthetic, since the bulk of the NVFP4 expert weights hasn't been independently verified.
Yeah this is true, thats the analysis i wanna see, where there is hard evidence that actually proves stuff (;