fix: prefer `| trim` over `.strip()`in tokenizer_config

#21
by drbh HF staff - opened

this PR simply replaces .strip() with trim to align with standard jinja: https://jinja.palletsprojects.com/en/2.11.x/templates/#list-of-builtin-filters

Code Llama org

Thank you! Tested locally, works for me. cc @Rocketknight1 in case he sees some incompatibility.

Code Llama org

Also cc @Xenova for info (.strip() not supported by TGI so changing to | strip, not sure if this will be needed in more models and whether transformers.js would work with it).

Thanks for the ping @pcuenq ! Filters (| operator) are supported by @huggingface/jinja, but trim has not yet been added yet. Luckily, it’s really simple, and I’ll open a PR for it tomorrow.

@huggingface/jinja v0.1.3 is now out which adds support for | trim (link). Feel free to merge :)

Code Llama org

Thanks all, merging!

pcuenq changed pull request status to merged

Sign up or log in to comment