TransformerXL: StopIteration: Caught StopIteration in replica 0 on device 0
Author: davidliujiafeng
Pull request: no

## Environment info
- `transformers` version: 3.4.0
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-debian-stretch-sid
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

@TevenLeScao 

## Error I get
```
Traceback (most recent call last):
  File "/ai/fzc/minGPT/transformerXLtest.py", line 163, in <module>
    input_ids=inputs["input_ids"].to(device),
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 155, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 165, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/opt/conda/lib/python3.6/site-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
StopIteration: Caught StopIteration in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_transfo_xl.py", line 866, in forward
    mems = self.init_mems(bsz)
  File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_transfo_xl.py", line 800, in init_mems
    param = next(self.parameters())
StopIteration
```
## To reproduce the problem

Run Code below:

```python
import torch
from torch.nn import DataParallel
from transformers import TransfoXLTokenizer, TransfoXLModel

device = "cuda:0"

# Get model
tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
model = TransfoXLModel.from_pretrained('transfo-xl-wt103', return_dict=True)
model = DataParallel(model, device_ids=list(range(torch.cuda.device_count())))
model.to(device=device)

# Run forward
inputs = tokenizer(["This is an example"], return_tensors="pt")

outputs = model(
    input_ids=inputs["input_ids"].to(device),
)

print(f"outputs: {outputs}")
print("Success.")
```


Comments:
The same code I tested on GPT-2, works fine for me. Guess something wrong with transformer-xl

GPT-2 code below:
```Python
import torch
from torch.nn import DataParallel
from transformers import GPT2Tokenizer, GPT2LMHeadModel

device = "cuda:0"

# Get model
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")
model = DataParallel(model, device_ids=list(range(torch.cuda.device_count())))
model.to(device=device)

# Run forward
inputs = tokenizer(["This is an example"], return_tensors="pt")
outputs = model(
    input_ids=inputs["input_ids"].to(device),
    attention_mask=inputs["attention_mask"].to(device),
    labels=inputs["input_ids"].to(device),
)

print(f"outputs: {outputs}")
print("Success.")
```

Seems like this was overlooked in #4300 ! I'll update TransfoXL in the same way.

As of now, Pytorch [doesn't support calling](https://github.com/pytorch/pytorch/issues/40457) `self.parameters()` within `DataParallel`, which causes the current issue. Even after fixing that, which was straightforward, Pytorch [also doesn't support calling](https://github.com/pytorch/pytorch/issues/36035) `self.ParameterList` and `self.ParameterDict`, which are also used in TransfoXL, which will cause another issue. As Pytorch is moving people away from `DataParallel`, they are unlikely to fix this anytime soon on their end. On our end, this is going to be much harder to fix in a non-BC way, as changing the way the model is organized means previous checkpoints cannot be loaded. In the meantime, you could use `DistributedDataParallel` instead. 

I used `torch.nn.parallel.DistributedDataParallel` to run the model in forward pass with the script below:
```python
import os
import sys
import tempfile
import torch
import torch.distributed as dist
import torch.nn as nn
import torch.optim as optim
import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP
from transformers import TransfoXLTokenizer, TransfoXLLMHeadModel

def setup(rank, world_size):
     os.environ['MASTER_ADDR'] = 'localhost'
     os.environ['MASTER_PORT'] = '12355'

     # initialize the process group
     dist.init_process_group("gloo", rank=rank, world_size=world_size)

def demo_model_parallel(rank, world_size):
    print(f"Running DDP with model parallel example on rank {rank}.")
    setup(rank, world_size)

    # transfoXL model
    tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
    mp_model = TransfoXLLMHeadModel.from_pretrained('transfo-xl-wt103', return_dict=True)
    ddp_mp_model = DDP(mp_model, find_unused_parameters=True)
    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_mp_model.parameters(), lr=0.001)
    
    for i in range(10):
        optimizer.zero_grad()
        # check to see if the model returns different losses
        if rank == 0:
            inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
        else:
            inputs = tokenizer("Borat and the republic of Kazakhistan!", return_tensors = "pt")
        outputs = ddp_mp_model(input_ids = inputs["input_ids"], labels=inputs["input_ids"], return_dict = True)
        _l = outputs.losses.mean() # documentation is incorrect there is no `loss` but `losses`
        print(_l)
        _l.backward()
        optimizer.step()

def run_demo(demo_fn, world_size):
    mp.spawn(demo_fn,
             args=(world_size,),
             nprocs=world_size,
             join=True)
    
    
if __name__ == "__main__":
    run_demo(demo_model_parallel, 2)
```

However during backward pass I get this error:
```
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that
your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1)
passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making
sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the
distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward`
function. Please include the loss function and the structure of the return value of `forward` of your module when reporting
this issue (e.g. list, dict, iterable).
```

The code is modified from [tutorial](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html)

@TevenLeScao  
I have the same error when trainning TransformerXL.

File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_transfo_xl.py", line 1056, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_transfo_xl.py", line 866, in forward
    mems = self.init_mems(bsz)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_transfo_xl.py", line 800, in init_mems
    param = next(self.parameters())
StopIteration

how to solve it quikly?

@yashbonde this seems to be an unrelated issue! I'll take a look tomorrow.
@ismymajia see my message above - this is a Pytorch issue that we cannot fix without breaking backwards compatibility of checkpoints, as they're slowly stopping support for `DataParallel`.

Now the problem is that huggingface tansformer-xl model cannot be trained. huggingface tansformer-xl model will not be supported?  Do you plan to update the tansformer-xl code? 
@TevenLeScao

As I said in my previous post, you can just use single-GPU or distributed training instead. Of course transformer-xl is supported ; but we cannot update its code to bypass the Pytorch issues with `DataParallel` without breaking backwards compatibility with previous checkpoints.

when i train tansformer-xl as below:

python -m torch.distributed.launch --nproc_per_node 4 run_language_modeling.py --output_dir ${model_dir} \
	    --tokenizer_name $data_dir/wordpiece-custom.json \
	     --config_name $data_dir/$config_file \
            --train_data_files "$data_dir/train*.txt" \
            --eval_data_file $data_dir/valid.txt \
            --block_size=128 \
            --do_train \
	    --do_eval \
            --per_device_train_batch_size 4  \
            --gradient_accumulation_steps 1 \
            --learning_rate 6e-4 \
            --weight_decay 0.01 \
            --adam_epsilon 1e-6 \
            --adam_beta1 0.9 \
            --adam_beta2 0.98 \
            --max_steps 500_000 \
            --warmup_steps 24_000 \
            --fp16 \
            --logging_dir ${model_dir}/tensorboard \
            --save_steps 5000 \
            --save_total_limit 20 \
            --seed 108 \
            --max_steps -1 \
            --num_train_epochs 20 \
	    --dataloader_num_workers 0  \
            --overwrite_output_dir 

occur error:

[INFO|language_modeling.py:324] 2020-11-11 13:50:49,520 >> Loading features from cached file /opt/ml/input/data/training/mm/huggingface/data/train40G/cached_lm_PreTrainedTokenizerFast_126_train3.txt [took 93.739 s]
[INFO|language_modeling.py:324] 2020-11-11 13:52:30,959 >> Loading features from cached file /opt/ml/input/data/training/mm/huggingface/data/train40G/cached_lm_PreTrainedTokenizerFast_126_train2.txt [took 101.436 s]
Traceback (most recent call last):
  File "run_language_modeling.py", line 350, in <module>
    main()
  File "run_language_modeling.py", line 313, in main
    trainer.train(model_path=model_path)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 657, in train
    else True
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/distributed.py", line 333, in __init__
    self.broadcast_bucket_size)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/distributed.py", line 549, in _distributed_broadcast_coalesced
    dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:518, unhandled cuda error, NCCL version 2.4.8
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/launch.py", line 261, in <module>
    main()
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/launch.py", line 257, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python', '-u', 'run_language_modeling.py', '--local_rank=3', '--output_dir', '/opt/ml/input/data/training/mm/huggingface/data/20201107/checkpoints/transfo-xl_1L_dembed1024_dhead64_dInner4096_dmodel1024_heads16_1', '--tokenizer_name', '/opt/ml/input/data/training/mm/huggingface/data/20201107/wordpiece-custom.json', '--config_name', '/opt/ml/input/data/training/mm/huggingface/data/20201107/config-transfo-xl.json', '--train_data_files', '/opt/ml/input/data/training/mm/huggingface/data/20201107/train*.txt', '--eval_data_file', '/opt/ml/input/data/training/mm/huggingface/data/20201107/valid.txt', '--block_size=128', '--do_train', '--do_eval', '--per_device_train_batch_size', '16', '--gradient_accumulation_steps', '1', '--learning_rate', '6e-4', '--weight_decay', '0.01', '--adam_epsilon', '1e-6', '--adam_beta1', '0.9', '--adam_beta2', '0.98', '--max_steps', '500_000', '--warmup_steps', '24_000', '--fp16', '--logging_dir', '/opt/ml/input/data/training/mm/huggingface/data/20201107/checkpoints/transfo-xl_1L_dembed1024_dhead64_dInner4096_dmodel1024_heads16_1/tensorboard', '--save_steps', '5000', '--save_total_limit', '20', '--seed', '108', '--max_steps', '-1', '--num_train_epochs', '20', '--overwrite_output_dir']' died with <Signals.SIGKILL: 9>.
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************

my env is below:
pytorch:1.6+cu101
transformer 3.4
tokenizer 0.9.3

@TevenLeScao  How to solve it ? 

Hey, looking at the error message (the SIGKILL) this looks more like the machine killing the process than like a bug. What's your setup? This happens if the machine runs out of RAM for example. 

I am trainning the transformer-xl on one machine with multi-gpus by ddp.
I don't know if this is a problem.
@TevenLeScao 
 

Hey, usually when you get a mysterious CUDA error like this ("RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:518, unhandled cuda error, NCCL version 2.4.8") it's because of GPU memory. I'll close the issue now as this is unrelated, and does not particularly look like a library bug. You should probably post on the forums at https://discuss.huggingface.co/ to see if you can get help with debugging!

[INFO|language_modeling.py:242] 2020-11-11 11:54:46,363 >> Loading features from cached file /opt/ml/input/data/training/kyzhan/huggingface/data/train40G/cached_lm_PreTrainedTokenizerFast_126_train3.txt [took 116.431 s]
/ _th_index_copy_
    main()
  File "run_hf_train_lm_ti.py", line 338, in main
    trainer.train(model_path=model_path)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 758, in train
    tr_loss += self.training_step(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1056, in training_step
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1082, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/distributed.py", line 511, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_transfo_xl.py", line 1056, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_transfo_xl.py", line 888, in forward
    word_emb = self.word_emb(input_ids)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_transfo_xl.py", line 448, in forward
    emb_flat.index_copy_(0, indices_i, emb_i)
RuntimeError: Expected object of scalar type Float but got scalar type Half for argument #4 'source' in call to _th_index_copy_

Now encounter this problem.  @TevenLeScao

I'm closing this issue as it concerns an unrelated problem that we cannot solve. Can you open a new one with a complete description ?