[QuestionGeneration] RuntimeError: Integer division of tensors using div or / is no longer supported
Author: jqueguiner
Pull request: no

## Environment info
- `transformers` version: 4.12.0
- Platform: Linux-5.4.0-88-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.10
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.6.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
Models:
- T5ForConditionalGeneration : @patrickvonplaten, @patil-suraj

## Information

Model I am using T5conditionalGeneration through Questgen.ai

The problem arises when using:
* [x] my own modified scripts: (give details below)
```python
from transformers import AutoTokenizer, AutoModelWithLMHead

def predict(sentence):
    tokenizer = AutoTokenizer.from_pretrained("flexudy/t5-base-multi-sentence-doctor")

    model = AutoModelWithLMHead.from_pretrained("flexudy/t5-base-multi-sentence-doctor")

    input_text = f"repair_sentence: {sentence}
"

    input_ids = tokenizer.encode(input_text, return_tensors="pt")

    outputs = model.generate(input_ids, max_length=32, num_beams=1)

    sentence = tokenizer.decode(outputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=True)

    return sentence
```

The tasks I am working on is:
* [x] my own task : 
```python
  File "apis/text/text/boolean-question-generations/questgen/questgen.py", line 12, in predict
    output = qe.predict_boolq(payload)
  File "/opt/conda/lib/python3.7/site-packages/Questgen/main.py", line 238, in predict_boolq
    output = beam_search_decoding (input_ids, attention_masks,self.model,self.tokenizer)
  File "/opt/conda/lib/python3.7/site-packages/Questgen/encoding/encoding.py", line 18, in beam_search_decoding
    early_stopping=True
  File "/opt/conda/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/transformers/generation_utils.py", line 1064, in generate
    **model_kwargs,
  File "/opt/conda/lib/python3.7/site-packages/transformers/generation_utils.py", line 1839, in beam_search
    next_indices = (next_tokens / vocab_size).long()
RuntimeError: Integer division of tensors using div or / is no longer supported, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.
```

## To reproduce

Steps to reproduce the behavior:

1.
```
pip install git+https://github.com/ramsrigouthamg/Questgen.ai
pip install git+https://github.com/boudinfl/pke.git

python -m nltk.downloader universal_tagset
python -m spacy download en 
```
2.
```python
from transformers import AutoTokenizer, AutoModelWithLMHead

def predict(sentence):
    tokenizer = AutoTokenizer.from_pretrained("flexudy/t5-base-multi-sentence-doctor")

    model = AutoModelWithLMHead.from_pretrained("flexudy/t5-base-multi-sentence-doctor")

    input_text = f"repair_sentence: {sentence}
"

    input_ids = tokenizer.encode(input_text, return_tensors="pt")

    outputs = model.generate(input_ids, max_length=32, num_beams=1)

    sentence = tokenizer.decode(outputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=True)

    return sentence
```

## Expected behavior
No error

## Fix found

https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L1839

use // instead of /


Comments:
oups, I guess we are all colliding here 
here is the related change 3months ago that removed the // operator because of torch working.
https://github.com/huggingface/transformers/pull/13013

my 2 cents is that implicit operator are kind of broken

I feel like https://github.com/StevenTang1998 push toward explicit operation https://github.com/huggingface/transformers/pull/13013#issuecomment-894198148 but not supported for torch < 1.8

@sgugger 
@nreimers
@patrickvonplaten

any thoughs on that matter ?

I think we will need to do a version check and use @StevenTang1998 solution for recent versions of PyTorch and leave the old code for older versions.

Do you want to take a stab at a PR?

I could try to write it if you want.

I think I just asked you to ;-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I am still getting the error in generation_utils.py. The issue is not yet resolved in Transformers 4.15.0



@JeetRoy97,

Could you also post your environment info? 

Just run `transformes-cli env` and copy paste the output here. Especially your Python and PyTorch versions would be important to know

I run into the same division error
torch: 1.6.0
transformers: 4.15.0
onnxruntime: 1.10.0
python: 3.6
GPU

tokenizer =M2M100Tokenizer.from_pretrained( path )
tokenizer.src_lang ='xx'
example =['xxxx']
encoded = tokenizer(example, return_tensors='pt')
generated_tokens = model.generate(**encoded, forced_bos_token_id=tokenizer.get_lang_id('en'))

Error occurred at the `model.generate `step.
error message was directed to "transformers/generation_utils.py" 
line 1955: next_indices = (next_tokens / vocab_size).long()

I am not entirely sure if you can reproduce because I used the onnx version of the M2M model. 

This one looks like a similar problem: https://discuss.pytorch.org/t/runtimeerror-integer-division-of-tensors-using-div-or-is-no-longer-supported-and-in-a-future-release-div-will-perform-true-division-as-in-python-3-use-true-divide-or-floor-divide-in-python-instead/99427 and it was solved by changing the input type `image = image.float()` , but in the case of NLP transformer, the inputs are ids. I tried to use encoded = encoded.to(torch.long), no luck. 

Any advice would be highly appreciated! 

Same error when running the fine-tuning script, `run_summarization.py` with `--predict_with_generate`. The stack trace:
```
Traceback (most recent call last):
  File "run_summarization.py", line 698, in <module>
    main()
  File "run_summarization.py", line 650, in main
    predict_results = trainer.predict(
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer_seq2seq.py", line 119, in predict
    return super().predict(test_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 2319, in predict
    output = eval_loop(
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 2419, in evaluation_loop
    loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer_seq2seq.py", line 172, in prediction_step
    generated_tokens = self.model.generate(
  File "/usr/local/lib/python3.8/dist-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/generation_utils.py", line 1239, in generate
    return self.beam_search(
  File "/usr/local/lib/python3.8/dist-packages/transformers/generation_utils.py", line 2027, in beam_search
    next_indices = (next_tokens / vocab_size).long()
RuntimeError: Integer division of tensors using div or / is no longer supported, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.
```
Environment:
```
- `transformers` version: 4.17.0.dev0
- Platform: Linux-5.4.0-90-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.3.1 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO
```


Hey @mwojnars, 

Thanks for reporting the error here. I'm attaching a PR that should fix it.

@patrickvonplaten Thanks. It seems to be working fine now. 

The error seems to persist when running the RAG-Token model: https://huggingface.co/facebook/rag-token-nq. Running the exact code from the aforementioned link with a fresh install of the `transformers` library yields the following error:
```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_35233/3517742745.py in <module>
      1 input_dict = tokenizer.prepare_seq2seq_batch("who holds the record in 100m freestyle", return_tensors="pt")
      2 
----> 3 generated = model.generate(input_ids=input_dict["input_ids"])
      4 print(tokenizer.batch_decode(generated, skip_special_tokens=True)[0])
      5 

~/anaconda3/lib/python3.8/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     13         def decorate_context(*args, **kwargs):
     14             with self:
---> 15                 return func(*args, **kwargs)
     16         return decorate_context
     17 

~/anaconda3/lib/python3.8/site-packages/transformers/models/rag/modeling_rag.py in generate(self, input_ids, attention_mask, context_input_ids, context_attention_mask, doc_scores, max_length, min_length, early_stopping, use_cache, num_beams, num_beam_groups, diversity_penalty, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, encoder_no_repeat_ngram_size, repetition_penalty, bad_words_ids, num_return_sequences, decoder_start_token_id, n_docs, prefix_allowed_tokens_fn, forced_bos_token_id, forced_eos_token_id, remove_invalid_values, **model_kwargs)
   1614                 num_beam_hyps_to_keep=num_return_sequences,
   1615             )
-> 1616             return self.beam_search(
   1617                 input_ids,
   1618                 beam_scorer,

~/anaconda3/lib/python3.8/site-packages/transformers/generation_utils.py in beam_search(self, input_ids, beam_scorer, logits_processor, stopping_criteria, max_length, pad_token_id, eos_token_id, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, synced_gpus, **model_kwargs)
   1853             )
   1854 
-> 1855             next_indices = (next_tokens / vocab_size).long()
   1856             next_tokens = next_tokens % vocab_size
   1857 

RuntimeError: Integer division of tensors using div or / is no longer supported, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.
```

and the following warnings:
```
/home/myname/anaconda3/lib/python3.8/site-packages/transformers/models/rag/tokenization_rag.py:92: FutureWarning: `prepare_seq2seq_batch` is deprecated and will be removed in version 5 of 🤗 Transformers. Use the regular `__call__` method to prepare your inputs and the tokenizer under the `with_target_tokenizer` context manager to prepare your targets. See the documentation of your specific tokenizer for more details
  warnings.warn(
/home/myname/anaconda3/lib/python3.8/site-packages/transformers/generation_utils.py:1747: UserWarning: `max_length` is deprecated in this function, use `stopping_criteria=StoppingCriteriaList(MaxLengthCriteria(max_length=max_length))` instead.
  warnings.warn(
```

I cannot rule out that this is an error specific to my installation. But as I've just installed the `transformers` library, I suspect that this might be a more general issue.