Beware of this model, Hugging Face automatically reports 6.8T parameters, while the uploader claims approximately 14T.
Hugging Face automatically reports 6.8T parameters, while the uploader claims approximately 14T.
Could this be an error due to quantization?
Anyway still very hard to believe this is from Anthropic. Like why would they use HuggingFace-compatible formats like "configuration_numbat.py", "modeling_numbat.py" etc
The uploader says the checkpoint was quantized to NVFP4 for evaluation on approximately 85 AWS Trainium2 chips.
However, AWS's published documentation does not list native FP4 support for Trainium2. Software dequantization could make such a setup possible, but the stated explanation is technically questionable without further details.
burner22/numbat at main+2
The repository also provides claimed MMLU accuracy of 88.49%, ARC-Challenge accuracy of 72.01%, and HellaSwag accuracy of 71.85%. These are numbers in a metadata file, not independently reproduced benchmark results.
arc-challenge only at 72% is such a small number for frontier models
fake 100%
I would like to give this the benefit of the doubt. Give it time and let's see what it materializes to! It could be legit and we could finally be able to train something smaller from the weights.
The uploader says the checkpoint was quantized to NVFP4 for evaluation on approximately 85 AWS Trainium2 chips.
However, AWS's published documentation does not list native FP4 support for Trainium2. Software dequantization could make such a setup possible, but the stated explanation is technically questionable without further details.
burner22/numbat at main+2The repository also provides claimed MMLU accuracy of 88.49%, ARC-Challenge accuracy of 72.01%, and HellaSwag accuracy of 71.85%. These are numbers in a metadata file, not independently reproduced benchmark results.
The eval config includes a Neuron compiler flag, --experimental-unsafe-fp8e4m3fn-as-fp8e4m3, which is aimed exactly at the FP8 scale-format mismatch you’d hit running NVFP4 block scales on Trainium.
Not saying that this proves it's real, but whoever made this put substantial effort into making it. Tokenizer seems custom at first glance, but anyone can make one of those pretty easily.
Hugging Face automatically reports 6.8T parameters, while the uploader claims approximately 14T.
hugging face is famously bad at reporting parameter numbers
that was upload speed lag
Hugging Face automatically reports 6.8T parameters, while the uploader claims approximately 14T.
hugging face is famously bad at reporting parameter numbers
yeah iirc qwen 3.8 27b is reported as 28b