gpt2.generate breaks on FP16 Apex training.
Author: Laksh1997
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT2

Language I am using the model on (English, Chinese ...): Molecule

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Run Pytorch lightning with Apex (or just default apex training), then during validation try and generate samples with the model (which is on fp16).

## Trace

```
2020-06-23T18:30:00.337+01:00 | THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1579022034529/work/aten/src/THC/THCReduceAll.cuh line=327 error=716 : misaligned address
-- | --
  | 2020-06-23T18:30:00.338+01:00 | Validation sanity check: 0it [00:00, ?it/s] Validation sanity check: 50% 1/2 [00:01<00:01, 1.08s/it]generating smiles
  | 2020-06-23T18:30:00.341+01:00 | Traceback (most recent call last):
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/bin/transformervae", line 11, in <module>
  | 2020-06-23T18:30:00.341+01:00 | load_entry_point('exs-transformervae', 'console_scripts', 'transformervae')()
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  | 2020-06-23T18:30:00.341+01:00 | return self.main(*args, **kwargs)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/click/core.py", line 782, in main
  | 2020-06-23T18:30:00.341+01:00 | rv = self.invoke(ctx)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
  | 2020-06-23T18:30:00.341+01:00 | return _process_result(sub_ctx.command.invoke(sub_ctx))
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  | 2020-06-23T18:30:00.341+01:00 | return ctx.invoke(self.callback, **ctx.params)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  | 2020-06-23T18:30:00.341+01:00 | return callback(*args, **kwargs)
  | 2020-06-23T18:30:00.341+01:00 | File "/app/transformervae/cli.py", line 404, in pretrain
  | 2020-06-23T18:30:00.341+01:00 | trainer.fit(model)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/uwandb: Waiting for W&B process to finish, PID 28
  | 2020-06-23T18:30:00.341+01:00 | ser/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 918, in fit
  | 2020-06-23T18:30:00.341+01:00 | self.single_gpu_train(model)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/distrib_parts.py", line 176, in single_gpu_train
  | 2020-06-23T18:30:00.341+01:00 | self.run_pretrain_routine(model)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1076, in run_pretrain_routine
  | 2020-06-23T18:30:00.341+01:00 | False)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 330, in _evaluate
  | 2020-06-23T18:30:00.341+01:00 | eval_results = model.validation_epoch_end(outputs)
  | 2020-06-23T18:30:00.341+01:00 | File "/app/transformervae/models/base.py", line 95, in validation_epoch_end
  | 2020-06-23T18:30:00.341+01:00 | return self._shared_eval_end(output, "val")
  | 2020-06-23T18:30:00.341+01:00 | File "/app/transformervae/models/lm.py", line 152, in _shared_eval_end
  | 2020-06-23T18:30:00.341+01:00 | use_cache=True,
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
  | 2020-06-23T18:30:00.341+01:00 | return func(*args, **kwargs)
  | 2020-06-23T18:30:00.341+01:00 | File "/app/transformervae/models/lm.py", line 173, in generate_no_grad
  | 2020-06-23T18:30:00.341+01:00 | generated_ids = self.encoder.generate(input_ids, **kwargs)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
  | 2020-06-23T18:30:00.341+01:00 | return func(*args, **kwargs)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/transformers/modeling_utils.py", line 1181, in generate
  | 2020-06-23T18:30:00.341+01:00 | model_specific_kwargs=model_specific_kwargs,
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/transformers/modeling_utils.py", line 1285, in _generate_no_beam_search
  | 2020-06-23T18:30:00.341+01:00 | if unfinished_sents.max() == 0:
  | 2020-06-23T18:30:00.341+01:00 | RuntimeError: cuda runtime error (716) : misaligned address at /opt/conda/conda-bld/pytorch_1579022034529/work/aten/src/THC/THCReduceAll.cuh:327


```

## Expected behavior

Should generate samples.


## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.11.0
- Platform: Ubuntu
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 cu101
- Tensorflow version (GPU?): n/a
- Using GPU in script?: V100
- Using distributed or parallel set-up in script?: no but FP16 training


Comments:
This may be an error on my part, apologies. Just confirming.

This is supposed to work, misaligned address is usually a flavor of OOM. I would try to cut your batch size.

@sshleifer Batch size is 16, am on a V100 on FP16. Also I'm training with a batch size of 512, so surely batch size is fine in this case?

Okay, turns out the problem is still there.

Here is my code:

```
    @torch.no_grad()
    def generate_no_grad(self, num_samples, batch_size, **kwargs):
        device = next(self.parameters()).device
        num_iters = num_samples // batch_size
        all_smiles = []
        for idx in range(num_iters):
            input_ids = torch.empty(batch_size, 1).fill_(self.tokenizer.bos_token_id)
            input_ids = input_ids.to(device).long()
            generated_ids = self.encoder.generate(input_ids, **kwargs)
            smiles = self.tokenizer.decode(generated_ids)
            all_smiles.extend(smiles)
        return all_smiles

    def generate(self):
            smiles = self.generate_no_grad(
                num_samples=50,
                batch_size=16,
                max_length=self.collater.max_length,
                do_sample=True,
                num_beams=1,
                temperature=1.0,
                top_k=500,
                top_p=1.0,
                repetition_penalty=1.0,
                pad_token_id=self.tokenizer.pad_token_id,
                bos_token_id=self.tokenizer.bos_token_id,
                eos_token_id=self.tokenizer.eos_token_id,
                length_penalty=1,
                no_repeat_ngram_size=0,
                num_return_sequences=1,
                use_cache=True,
            )
            ....
```

Bugs out on batch size 16, num_samples=10_000, max_length=100

@patrickvonplaten  Any idea with this?

Also, I'm on torch 1.4 and cu101, could upgrading to 1.5.1 and cu102 fix this?

I cannot reproduce the error in the notebook. Looking into it more. 

Hmmmm, getting the same error on batch size 8 and num samples = 50.

here's my colab where I'm trying to reproduce: https://colab.research.google.com/drive/13uvd_Y_VHoZqQxyZ0OdyyNAJEurrW2LX?usp=sharing

@sshleifer I'm guessing the CICD checks for torch 1.4 cu101?

Okay. I can only think that this is a pytorch 1.4 cu101 error. Will update image to 1.5.1, try to tomorrow and update this accordingly. Cheers!

Is there anything from the code above that would warrant this error?

Well, well, well @sshleifer I moved to the pytorch nightly conda builds (1.6-dev, which has native AMP) and now the issue is no longer there.

Weird ...

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.