TypeError: 'NoneType' object cannot be interpreted as an integer
Author: andysingal
Pull request: no

### System Info

RTX 3090

### Who can help?

@ArthurZucker  @younesbelkada  

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```py
import torch
import transformers

class Llama3:
    def __init__(self, model_path):
        self.model_id = model_path
        self.pipeline = transformers.pipeline(
            "text-generation",
            model=self.model_id,
            model_kwargs={
                "torch_dtype": torch.float16,
                "quantization_config": {"load_in_4bit": True},
                "low_cpu_mem_usage": True,
            },
        )
        self.terminators = [
            self.pipeline.tokenizer.eos_token_id,
            self.pipeline.tokenizer.convert_tokens_to_ids(""),
        ]
  
    def get_response(
          self, query, message_history=[], max_tokens=4096, temperature=0.6, top_p=0.9
      ):
        user_prompt = message_history + [{"role": "user", "content": query}]
        prompt = self.pipeline.tokenizer.apply_chat_template(
            user_prompt, tokenize=False, add_generation_prompt=True
        )
        outputs = self.pipeline(
            prompt,
            max_new_tokens=max_tokens,
            eos_token_id=self.terminators,
            do_sample=True,
            temperature=temperature,
            top_p=top_p,
        )
        response = outputs[0]["generated_text"][len(prompt):]
        return response, user_prompt + [{"role": "assistant", "content": response}]
    
    def chatbot(self, system_instructions=""):
        conversation = [{"role": "system", "content": system_instructions}]
        while True:
            user_input = input("User: ")
            if user_input.lower() in ["exit", "quit"]:
                print("Exiting the chatbot. Goodbye!")
                break
            response, conversation = self.get_response(user_input, conversation)
            print(f"Assistant: {response}")

if __name__ == "__main__":
    bot = Llama3("/workspace/meta-llama/Meta-Llama-3-8B-Instruct")
    bot.chatbot()            
```

ERROR:
```py
Loading checkpoint shards: 100%
 4/4 [00:24<00:00,  5.24s/it]
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
User:  
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[7], line 51
     49 if __name__ == "__main__":
     50     bot = Llama3("[/workspace/meta-llama/Meta-Llama-3-8B-Instruct](https://8lcf99knl425bd-8888.proxy.runpod.net/lab/tree/workspace/workspace/meta-llama/Meta-Llama-3-8B-Instruct)")
---> 51     bot.chatbot()            

Cell In[7], line 46, in Llama3.chatbot(self, system_instructions)
     44     print("Exiting the chatbot. Goodbye!")
     45     break
---> 46 response, conversation = self.get_response(user_input, conversation)
     47 print(f"Assistant: {response}")

Cell In[7], line 28, in Llama3.get_response(self, query, message_history, max_tokens, temperature, top_p)
     24 user_prompt = message_history + [{"role": "user", "content": query}]
     25 prompt = self.pipeline.tokenizer.apply_chat_template(
     26     user_prompt, tokenize=False, add_generation_prompt=True
     27 )
---> 28 outputs = self.pipeline(
     29     prompt,
     30     max_new_tokens=max_tokens,
     31     eos_token_id=self.terminators,
     32     do_sample=True,
     33     temperature=temperature,
     34     top_p=top_p,
     35 )
     36 response = outputs[0]["generated_text"][len(prompt):]
     37 return response, user_prompt + [{"role": "assistant", "content": response}]

File [/usr/local/lib/python3.10/dist-packages/transformers/pipelines/text_generation.py:263](https://8lcf99knl425bd-8888.proxy.runpod.net/lab/tree/workspace/usr/local/lib/python3.10/dist-packages/transformers/pipelines/text_generation.py#line=262), in TextGenerationPipeline.__call__(self, text_inputs, **kwargs)
    261         return super().__call__(chats, **kwargs)
    262 else:
--> 263     return super().__call__(text_inputs, **kwargs)

File [/usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py:1243](https://8lcf99knl425bd-8888.proxy.runpod.net/lab/tree/workspace/usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py#line=1242), in Pipeline.__call__(self, inputs, num_workers, batch_size, *args, **kwargs)
   1235     return next(
   1236         iter(
   1237             self.get_iterator(
   (...)
   1240         )
   1241     )
   1242 else:
-> 1243     return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)

File [/usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py:1250](https://8lcf99knl425bd-8888.proxy.runpod.net/lab/tree/workspace/usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py#line=1249), in Pipeline.run_single(self, inputs, preprocess_params, forward_params, postprocess_params)
   1248 def run_single(self, inputs, preprocess_params, forward_params, postprocess_params):
   1249     model_inputs = self.preprocess(inputs, **preprocess_params)
-> 1250     model_outputs = self.forward(model_inputs, **forward_params)
   1251     outputs = self.postprocess(model_outputs, **postprocess_params)
   1252     return outputs

File [/usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py:1150](https://8lcf99knl425bd-8888.proxy.runpod.net/lab/tree/workspace/usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py#line=1149), in Pipeline.forward(self, model_inputs, **forward_params)
   1148     with inference_context():
   1149         model_inputs = self._ensure_tensor_on_device(model_inputs, device=self.device)
-> 1150         model_outputs = self._forward(model_inputs, **forward_params)
   1151         model_outputs = self._ensure_tensor_on_device(model_outputs, device=torch.device("cpu"))
   1152 else:

File [/usr/local/lib/python3.10/dist-packages/transformers/pipelines/text_generation.py:350](https://8lcf99knl425bd-8888.proxy.runpod.net/lab/tree/workspace/usr/local/lib/python3.10/dist-packages/transformers/pipelines/text_generation.py#line=349), in TextGenerationPipeline._forward(self, model_inputs, **generate_kwargs)
    347         generate_kwargs["min_length"] += prefix_length
    349 # BS x SL
--> 350 generated_sequence = self.model.generate(input_ids=input_ids, attention_mask=attention_mask, **generate_kwargs)
    351 out_b = generated_sequence.shape[0]
    352 if self.framework == "pt":

File [/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py:115](https://8lcf99knl425bd-8888.proxy.runpod.net/lab/tree/workspace/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py#line=114), in context_decorator.<locals>.decorate_context(*args, **kwargs)
    112 @functools.wraps(func)
    113 def decorate_context(*args, **kwargs):
    114     with ctx_factory():
--> 115         return func(*args, **kwargs)

File [/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1565](https://8lcf99knl425bd-8888.proxy.runpod.net/lab/tree/workspace/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py#line=1564), in GenerationMixin.generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, **kwargs)
   1562 batch_size = inputs_tensor.shape[0]
   1564 device = inputs_tensor.device
-> 1565 self._prepare_special_tokens(generation_config, kwargs_has_attention_mask, device=device)
   1567 # decoder-only models must use left-padding for batched generation.
   1568 if not self.config.is_encoder_decoder and not is_torchdynamo_compiling():
   1569     # If `input_ids` was given, check if the last id in any sequence is `pad_token_id`
   1570     # Note: If using, `inputs_embeds` this check does not work, because we want to be more hands-off.

File [/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1405](https://8lcf99knl425bd-8888.proxy.runpod.net/lab/tree/workspace/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py#line=1404), in GenerationMixin._prepare_special_tokens(self, generation_config, kwargs_has_attention_mask, device)
   1400     generation_config.decoder_start_token_id = self._get_decoder_start_token_id(
   1401         generation_config.decoder_start_token_id, generation_config.bos_token_id
   1402     )
   1404 bos_token_id = _tensor_or_none(generation_config.bos_token_id, device=device)
-> 1405 eos_token_id = _tensor_or_none(generation_config.eos_token_id, device=device)
   1406 pad_token_id = _tensor_or_none(generation_config.pad_token_id, device=device)
   1407 decoder_start_token_id = _tensor_or_none(generation_config.decoder_start_token_id, device=device)

File [/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1396](https://8lcf99knl425bd-8888.proxy.runpod.net/lab/tree/workspace/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py#line=1395), in GenerationMixin._prepare_special_tokens.<locals>._tensor_or_none(token, device)
   1394 if token is None or isinstance(token, torch.Tensor):
   1395     return token
-> 1396 return torch.tensor(token, device=device, dtype=torch.long)

TypeError: 'NoneType' object cannot be interpreted as an integer
```

### Expected behavior

works like a chatbot

Comments:
Hi @andysingal 
Which transformers version are you using?

> Hi @andysingal Which transformers version are you using?

4.41

Hi @andysingal , can you please **change this section in your code:**
```
self.terminators = [
            self.pipeline.tokenizer.eos_token_id,
            self.pipeline.tokenizer.convert_tokens_to_ids(""),
        ]
```
**to this:**
```
self.terminators = [
            self.pipeline.tokenizer.eos_token_id,
            self.pipeline.tokenizer.convert_tokens_to_ids("
"),
        ]
```
It was causing the `TypeError: 'NoneType' object cannot be interpreted as an integer`

Cheers!


> self.terminators = [
>             self.pipeline.tokenizer.eos_token_id,
>             self.pipeline.tokenizer.convert_tokens_to_ids("
"),
>         ]

Thank you very much , works now 
<img width="1514" alt="Screenshot 2024-05-31 at 7 52 53 PM" src="https://github.com/huggingface/transformers/assets/20493493/9fcb06ca-1710-485a-80c0-79840529bb4f">



you're welcome :+1: 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@andysingal hi! i have the exact same issue as you, and i made the suggested change. now there is no error but it has been running for 18 minutes with no result or prompt anywhere.

`Loading checkpoint shards: 100%|██████████| 4/4 [01:20<00:00, 20.08s/it]
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Setting `pad_token_id` to `eos_token_id`:128009 for open-end generation.`

what should it look like when the chatbot asks for user input? i don't see how to input anything anywhere... 

thanks for any help you have provide!