fix eos_token and bos_token in config

by mayank31398 - opened
No description provided.

fixes the eos and bos token id in the config to match the tokenizer
The generation config picks eos_token from the config and not the tokenizer.
So, when specifying, min_new_tokens, the code crashes with the following error:

Traceback (most recent call last):
  File "/dccstor/mayankgpfs/conda/envs/bloom/lib/python3.8/", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/dccstor/mayankgpfs/conda/envs/bloom/lib/python3.8/", line 87, in _run_code
    exec(code, run_globals)
  File "/dccstor/mayankgpfs/scratch/GPTQ-for-SantaCoder/", line 90, in <module>
  File "/dccstor/mayankgpfs/scratch/GPTQ-for-SantaCoder/", line 86, in main
    simple_generation_test(tokenizer, model, args.prompt)
  File "/dccstor/mayankgpfs/scratch/GPTQ-for-SantaCoder/", line 55, in simple_generation_test
    generated = model.generate(batch["input_ids"], do_sample=False, min_new_tokens=100, max_new_tokens=100)
  File "/dccstor/mayankgpfs/conda/envs/bloom/lib/python3.8/site-packages/torch/autograd/", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/dccstor/mayankgpfs/conda/envs/bloom/lib/python3.8/site-packages/transformers/generation/", line 1438, in generate
    return self.greedy_search(
  File "/dccstor/mayankgpfs/conda/envs/bloom/lib/python3.8/site-packages/transformers/generation/", line 2263, in greedy_search
    next_tokens_scores = logits_processor(input_ids, next_token_logits)
  File "/dccstor/mayankgpfs/conda/envs/bloom/lib/python3.8/site-packages/transformers/generation/", line 92, in __call__
    scores = processor(input_ids, scores)
  File "/dccstor/mayankgpfs/conda/envs/bloom/lib/python3.8/site-packages/transformers/generation/", line 162, in __call__
    scores[:, i] = -float("inf")
IndexError: index 50256 is out of bounds for dimension 1 with size 49280
mayank31398 changed pull request status to open
mayank31398 changed pull request status to merged

Sign up or log in to comment