Multi turn conversation with Blender Bot 
Author: mailong25
Pull request: no

# 🚀 Feature request

Hi there. is there any way to predict a response using multi-turn dialog context for the Blender Bot Model. From your [example](https://huggingface.co/transformers/model_doc/blenderbot.html) I saw that it only use single-turn context.

I tried to use a ` </sep>` token to separate human/bot turn like the following example:

```
Human: I am from Vietnam
Bot: I've never been there, but I've always wanted to go. How do you like it?
Human: pretty good actually , where you are from ?
```

Concatenate input: 
`I am from Vietnam</sep> I've never been there, but I've always wanted to go. How do you like it?</sep> pretty good actually , where you are from ?`

huggingface's model response: `I am from the United States. I have never been to Vietnam, but I have always wanted to go.`
Facebook_ParlAI's model response: `I'm from the United States. I've heard it's a great place to visit, though.`


Comments:
Hi @mailong25 for blenderbot the dialogs are separated by the newline `\n`.

So the text should be 
`I am from Vietnam\nI've never been there, but I've always wanted to go. How do you like it?\npretty good actually , where you are from ?`

which model are you using, 90M or 3B?

Also, could you post the `parlai` command that you used?

Thanks for a quick response. I use the `facebook/blenderbot-1B-distill` model

For parlai, I use the cmd:

`python parlai/scripts/interactive.py -t blended_skill_talk -mf zoo:blender/blender_1Bdistill/model --include_personas=False`

I tried to use the `'\n'` separator with `blenderbot-1B` and `blender_1Bdistill` and the results are still the same with ` </sep>`, which are different than parlai version.


Also, when I tried to move the model and the input sentence to "cuda", the following errors occur:

```
import os
os.environ["TRANSFORMERS_CACHE"] = '/mnt/disks/blender/'
from transformers import BlenderbotTokenizer, BlenderbotForConditionalGeneration
mname = 'facebook/blenderbot-3B'
model = BlenderbotForConditionalGeneration.from_pretrained(mname)
tokenizer = BlenderbotTokenizer.from_pretrained(mname)
model.to('cuda')

import torch
with torch.no_grad():
    UTTERANCE = []
    UTTERANCE.append("I am from Vietnam")
    UTTERANCE.append("I've never been there, but I've always wanted to go. How do you like it?")
    UTTERANCE.append("pretty good actually , where you are from ?")
    
    UTTERANCE = '\n'.join(UTTERANCE)
    print(UTTERANCE)
    inputs = tokenizer([UTTERANCE], return_tensors='pt')
    reply_ids = model.generate(**inputs)
    print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in reply_ids])

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-9-dff8c43ffc48> in <module>
     10     print(UTTERANCE)
     11     inputs = tokenizer([UTTERANCE], return_tensors='pt')
---> 12     reply_ids = model.generate(**inputs)
     13     print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in reply_ids])

~/.local/lib/python3.7/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     13         def decorate_context(*args, **kwargs):
     14             with self:
---> 15                 return func(*args, **kwargs)
     16         return decorate_context
     17 

~/.local/lib/python3.7/site-packages/transformers/generation_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, num_return_sequences, decoder_start_token_id, use_cache, num_beam_groups, diversity_penalty, prefix_allowed_tokens_fn, **model_kwargs)
    501         if self.config.is_encoder_decoder:
    502             # add encoder_outputs to model_kwargs
--> 503             model_kwargs = self._prepare_encoder_decoder_kwargs_for_generation(input_ids, model_kwargs)
    504 
    505             # set input_ids as decoder_input_ids

~/.local/lib/python3.7/site-packages/transformers/generation_utils.py in _prepare_encoder_decoder_kwargs_for_generation(self, input_ids, model_kwargs)
     84             argument: value for argument, value in model_kwargs.items() if not argument.startswith("decoder_")
     85         }
---> 86         model_kwargs["encoder_outputs"]: ModelOutput = encoder(input_ids, return_dict=True, **encoder_kwargs)
     87         return model_kwargs
     88 

~/.local/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~/.local/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py in forward(self, input_ids, attention_mask, inputs_embeds, output_attentions, output_hidden_states, return_dict)
    750 
    751         if inputs_embeds is None:
--> 752             inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale
    753 
    754         embed_pos = self.embed_positions(input_shape)

~/.local/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~/.local/lib/python3.7/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    124         return F.embedding(
    125             input, self.weight, self.padding_idx, self.max_norm,
--> 126             self.norm_type, self.scale_grad_by_freq, self.sparse)
    127 
    128     def extra_repr(self) -> str:

~/.local/lib/python3.7/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1812         # remove once script supports set_grad_enabled
   1813         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1814     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1815 
   1816 

RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select
```

The error occurs because the `inputs` is not on GPU, putting `inputs` on GPU should fix the error. 

Hi. I tried to use the blenderbot example code from huggingface. I just copy pasted it in colab and ran it. But it is showing me the following error : 

TypeError: forward() got an unexpected keyword argument 'token_type_ids'

Please help me out. What can be done to solve this?

According to the Parlai [documentation](https://parl.ai/docs/tutorial_task.html), the Parlai format is to use a `<\t>` or 4 spaces as a separator of turns in a conversation.
I ran your example using 4 spaces and got the Parlai response
```
from transformers import BlenderbotForConditionalGeneration, BlenderbotTokenizer

MODEL_ID = "facebook/blenderbot-400M-distill"
model = BlenderbotForConditionalGeneration.from_pretrained(MODEL_ID)
tokenizer = BlenderbotTokenizer.from_pretrained(MODEL_ID)

text = ["I am from Vietnam    I've never been there, but I've always wanted to go. How do you like it?    pretty good actually , where you are from ?"]

inputs = tokenizer(text, return_tensors='pt')
res = model.generate(inputs['input_ids'])
tokenizer.batch_decode(res)

#["
 I'm from the United States. I've heard it's a beautiful place to visit. 
"]
```



> According to the Parlai [documentation](https://parl.ai/docs/tutorial_task.html), the Parlai format is to use a `<\t>` or 4 spaces as a separator of turns in a conversation.
> I ran your example using 4 spaces and got the Parlai response
> 
> ```
> from transformers import BlenderbotForConditionalGeneration, BlenderbotTokenizer
> 
> MODEL_ID = "facebook/blenderbot-400M-distill"
> model = BlenderbotForConditionalGeneration.from_pretrained(MODEL_ID)
> tokenizer = BlenderbotTokenizer.from_pretrained(MODEL_ID)
> 
> text = ["I am from Vietnam    I've never been there, but I've always wanted to go. How do you like it?    pretty good actually , where you are from ?"]
> 
> inputs = tokenizer(text, return_tensors='pt')
> res = model.generate(inputs['input_ids'])
> tokenizer.batch_decode(res)
> 
> #["
 I'm from the United States. I've heard it's a beautiful place to visit. 
"]
> ```

I did a whole bunch of testing with different turn separation tokens, and the only one that consistently separated the turns and created outputs that made sense was 4 spaces. 

Everything else would sometimes separate a turn, but sometimes not (tab, new line, `
 
`, two spaces)

The documentation there mentioned that tabs are rendered as 4 spaces in the browser, but it still should be tab as a separator? 

Any new insights into why?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Was there a finalized way to construct inputs for Blenderbot? According to this https://github.com/huggingface/transformers/blob/v4.5.1/src/transformers/models/blenderbot/modeling_blenderbot.py#L499 should be using `
 
` not sure if I should be following this or `\n` as indicated in parlai

I have tried four spaces,  \n  and  \t. Four spaces gave the best results.


Should the first sentence in the encoder input be prepended with an extra space ' '? Because I note that the first token generated by the decoder has the space prefix (e.g., ' I' or ' yes').

I noticed the space prefix in the generation too. But I didn't check if adding extra space to encoder input gives better results.

Check out this example. This example is crafted in such a way that it could be a 1 turn or 2 turn depend on how you separate it.  I am using the 3B model.
```
Using \n as seperator
UTTERANCE = "I am from tokyo\nwhere you are from?"
inputs = tokenizer([UTTERANCE], return_tensors="pt")
reply_ids = model.generate(**inputs)
print(tokenizer.batch_decode(reply_ids,skip_special_tokens=True))
#[' I was born and raised in Tokyo, the capital of Japan. How about you?']


Using four spaces as seperator
UTTERANCE = "I am from tokyo    where you are from?"
inputs = tokenizer([UTTERANCE], return_tensors="pt")
reply_ids = model.generate(**inputs)
print(tokenizer.batch_decode(reply_ids,skip_special_tokens=True))
#[" I'm from Tokyo. It's the capital of Japan. It's a big city"]

Using <t> as seperator
UTTERANCE = "I am from tokyo<t>where you are from?"
inputs = tokenizer([UTTERANCE], return_tensors="pt")
reply_ids = model.generate(**inputs)
print(tokenizer.batch_decode(reply_ids,skip_special_tokens=True))
#[" I'm from the United States. I've never been to Tokyo, but I've always wanted to go."]
 

# Using 
 
 as seperator
​UTTERANCE = "I am from tokyo
 
where you are from?"
inputs = tokenizer([UTTERANCE], return_tensors="pt")
reply_ids = model.generate(**inputs)
print(tokenizer.batch_decode(reply_ids,skip_special_tokens=True))
#[' I am also from Tokyo, the capital and most populous metropolitan area in Japan.']

```

It looks like using ```\n``` and four spaces,  the model interpret it as 2 turn but 1 turn for using ```\<t> and \
 \
```