How to disable BOS token?
#1
by
anon22342134
- opened
I'm using hugging face's transform, and every output starts with <|startoftext|>
I'm not sure of the precise Python code to do it, and I think this is only an issue when running GGUF quantizations? Native BOS is transformers should be fine, I believe.
...Still, I would recommend using something, anything, other than plain transformers. Its insanely slow and vram inefficient at high context.