ddp: fp16 misaligned address
Versions:
torch==1.5
pytorch-lightning==0.8.0rc1
cuda=10.1

In both O1 and O2 mode and the new ddp trainer.fit fails with the following traceback. This does not happen with ddp_spawn.
fp16 O2 mode:
  File "finetune.py", line 791, in <module>
    main(args)
  File "finetune.py", line 723, in main
    trainer: pl.Trainer = generic_train(model, args, early_stopping_callback=True)
  File "/home/shleifer/transformers_fork/examples/lightning_base.py", line 428, in generic_train
    trainer.fit(model)
  File "/home/shleifer/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 890, in fit
    self.spawn_ddp_children(model)
  File "/home/shleifer/pytorch-lightning/pytorch_lightning/trainer/distrib_data_parallel.py", line 396, in spawn_ddp_children
    self.ddp_train(local_rank, model, is_master=True)
  File "/home/shleifer/pytorch-lightning/pytorch_lightning/trainer/distrib_data_parallel.py", line 472, in ddp_train
    self.run_pretrain_routine(model)
  File "/home/shleifer/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 1050, in run_pretrain_routine
    self.train()
  File "/home/shleifer/pytorch-lightning/pytorch_lightning/trainer/training_loop.py", line 363, in train
    self.run_training_epoch()
  File "/home/shleifer/pytorch-lightning/pytorch_lightning/trainer/training_loop.py", line 445, in run_training_epoch
    _outputs = self.run_training_batch(batch, batch_idx)
  File "/home/shleifer/pytorch-lightning/pytorch_lightning/trainer/training_loop.py", line 621, in run_training_batch
    loss, batch_output = optimizer_closure()
  File "/home/shleifer/pytorch-lightning/pytorch_lightning/trainer/training_loop.py", line 599, in optimizer_closure
    model_ref.backward(self, closure_loss, optimizer, opt_idx)
  File "/home/shleifer/pytorch-lightning/pytorch_lightning/core/hooks.py", line 155, in backward
    scaled_loss.backward()
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/apex/amp/handle.py", line 123, in scale_loss
    optimizer._post_amp_backward(loss_scaler)
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/apex/amp/_process_optimizer.py", line 190, in post_backward_with_master_weights
    models_are_masters=False)
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/apex/amp/scaler.py", line 119, in unscale
    self.unscale_python(model_grads, master_grads, scale)
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/apex/amp/scaler.py", line 89, in unscale_python
    self.dynamic)
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/apex/amp/scaler.py", line 9, in scale_check_overflow_python
    cpu_sum = float(model_grad.float().sum())
RuntimeError: CUDA error: misaligned address
feel free to cross post if this is a pytorch issue