run_generation.py gives TypeError when using xlnet due to empty dict being passed as token
Author: nanne-aben
Pull request: no
Labels: name: wontfix

## 🐛 Bug

When I run
```
python run_generation.py  --model_type=xlnet  --model_name_or_path=xlnet-large-cased
```
I get the following error
```
Traceback (most recent call last):
  File "run_generation.py", line 236, in <module>
    main()
  File "run_generation.py", line 214, in main
    encoded_prompt = tokenizer.encode(prompt_text, add_special_tokens=False, return_tensors="pt")
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 820, in encode
    **kwargs
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 912, in encode_plus
    first_ids = get_input_ids(text)
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 904, in get_input_ids
    return self.convert_tokens_to_ids(text)
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 751, in convert_tokens_to_ids
    ids.append(self._convert_token_to_id_with_added_voc(token))
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 758, in _convert_token_to_id_with_added_voc
    if token in self.added_tokens_encoder:
TypeError: unhashable type: 'dict'
```

I think the problem lies in the following code in run_generation.py:
```
def prepare_xlnet_input(args, _, tokenizer, prompt_text):
    prompt_text = (args.padding_text if args.padding_text else PADDING_TEXT) + prompt_text
    return prompt_text, {}
```
This returns a tuple of (string, dict). As this gets passed down to _convert_token_to_id_with_added_voc(), it will first try to check whether prompt_text is in self.added_tokens_encoder, and then whether {} is in self.added_tokens_encoder (which gives a TypeError, because you cannot check ```{} in list```.

I'm not yet sure where the empty dict is supposed to be used, so I can't fix it myself. Would be happy to contribute though.

## Important information

Model I am using (Bert, XLNet....): XLNet
Language I am using the model on (English, Chinese....): English
The problem arise when using:
* [x] the official example scripts: run_generation.py
* [ ] my own modified scripts: (give details)
The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: NA
* [ ] my own task or dataset: NA

## To Reproduce

Steps to reproduce the behavior:

1. cd examples
2. python run_generation.py  --model_type=xlnet  --model_name_or_path=xlnet-large-cased

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

```
Traceback (most recent call last):
  File "run_generation.py", line 236, in <module>
    main()
  File "run_generation.py", line 214, in main
    encoded_prompt = tokenizer.encode(prompt_text, add_special_tokens=False, return_tensors="pt")
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 820, in encode
    **kwargs
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 912, in encode_plus
    first_ids = get_input_ids(text)
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 904, in get_input_ids
    return self.convert_tokens_to_ids(text)
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 751, in convert_tokens_to_ids
    ids.append(self._convert_token_to_id_with_added_voc(token))
  File "/Users/nanneaben/Documents/Projects/2019/intelmatch/src/transformers/transformers/src/transformers/tokenization_utils.py", line 758, in _convert_token_to_id_with_added_voc
    if token in self.added_tokens_encoder:
TypeError: unhashable type: 'dict'
```

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

Running without an error.

## Environment

* OS: Mac OS Mojave 10.14.4
* Python version: 3.7.3
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): current master (8c67b529f615cc24c46864b8323d2d47a15ccd58)
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information: NA

## Additional context

<!-- Add any other context about the problem here. -->
NA

Comments:
I have the same problem, did you find any solutions?
@nanne-aben 

No, not really. I removed the empty dictionary, which makes the code run,
but the generated text is just kinda bad. GPT2 (in which case {} is not
added) creates much better text. So I guess that the {} was added for a
reason, but I can't figure out why. Also, when using
https://transformer.huggingface.co/ XLNET seems to make good text, so it
does seem that something is wrong with simply removing the {}.

I'm not sure how to proceed though... Still interested in resolving this
though, so please let me know if you find anything! Would be happy to
contribute something here.

On Wed, Feb 5, 2020 at 2:46 PM pooya khandel <notifications@github.com>
wrote:

> I have the same problem, did you find any solutions?
> @nanne-aben <https://github.com/nanne-aben>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/2326?email_source=notifications&email_token=ALOBCX4FHNIAMCNWJMQ3D4DRBK7KRA5CNFSM4J7LJOR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK3PAYY#issuecomment-582414435>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALOBCXYOF27U4R54BY3WZILRBK7KRANCNFSM4J7LJORQ>
> .
>


Hi, thank you for opening this issue. I'm fixing this in #2749.

Awesome, thanks!

On Wed, 5 Feb 2020 at 22:19, Lysandre Debut <notifications@github.com>
wrote:

> Hi, thank you for opening this issue. I'm fixing this in #2749
> <https://github.com/huggingface/transformers/pull/2749>.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/2326?email_source=notifications&email_token=ALOBCX3PHAP4E6EUC6VB2SLRBMUNDA5CNFSM4J7LJOR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK5ATRI#issuecomment-582617541>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALOBCX4HMZSVJHAHMQAQUSTRBMUNDANCNFSM4J7LJORQ>
> .
>


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.