ArthurConmy
/

redwood_tokenizer

Model card Files Files and versions Community

Edit model card

YAML Metadata Warning: empty or missing yaml metadata in repo card (https://huggingface.co/docs/hub/model-cards#model-card-metadata)

WARNING: this adds the Redwood 2L Attn Only tokenizer for compatibility with transformer lens. I think this implementation is correct, though check against the reference tokenizer here: https://github.com/redwoodresearch/rust_circuit_public/blob/42c3fcbffbc367897d3a810b20c12d7c1c99a00d/python/rust_circuit/module_library.py#L1012

Downloads last month: -; Downloads are not tracked for this model. How to track

Unable to determine this model's library. Check the docs .