StopIteration error happened
Author: brightbsit
Pull request: no

I'm using cuda11.0 with RTX3090.(using ubuntu 18.04)
I don't know how can I solve this problem.

I saw some people solve this StopIteration error with downgrade torch.

But rtx3090 is only compatible with cuda 11.

Please help!


[pip list]
anytree           2.8.0
apex              0.1
boto3             1.16.63
botocore          1.19.63
certifi           2020.12.5
chardet           4.0.0
cycler            0.10.0
decorator         4.4.2
future            0.18.2
idna              2.10
imageio           2.9.0
jmespath          0.10.0
kiwisolver        1.3.1
matplotlib        3.3.4
mkl-fft           1.2.0
mkl-random        1.1.1
mkl-service       2.3.0
networkx          2.5
numpy             1.20.0
olefile           0.46
oscar             0.1.0               /home/u2/바탕화면/Kim_Project/Oscar
Pillow            8.1.0
pip               20.3.3
pyparsing         2.4.7
python-dateutil   2.8.1
PyWavelets        1.1.1
PyYAML            5.4.1
regex             2020.11.13
requests          2.25.1
s3transfer        0.3.4
scikit-image      0.18.1
scipy             1.6.0
setuptools        52.0.0.post20210125
six               1.15.0
tifffile          2021.1.14
torch             1.7.1+cu110
torchaudio        0.7.2
torchvision       0.8.2+cu110
tqdm              4.56.0
typing-extensions 3.7.4.3
urllib3           1.26.3
wheel             0.36.2



here is the full traceback.
```
2021-02-16 03:37:03,483 vlpretrain WARNING: Device: cuda, n_gpu: 2
2021-02-16 03:37:07,380 vlpretrain INFO: Training/evaluation parameters Namespace(adam_epsilon=1e-08, add_od_labels=True, config_name='', data_dir='', device=device(type='cuda'), do_eval=False, do_lower_case=True, do_test=False, do_train=True, drop_out=0.1, eval_model_dir='', evaluate_during_training=True, gradient_accumulation_steps=1, img_feature_dim=2054, img_feature_type='frcnn', learning_rate=3e-05, length_penalty=1, logging_steps=20, loss_type='sfmx', mask_prob=0.15, max_gen_length=20, max_grad_norm=1.0, max_img_seq_length=50, max_masked_tokens=3, max_seq_a_length=40, max_seq_length=70, max_steps=-1, min_constraints_to_satisfy=2, model_name_or_path='pre_trained/base-vg-labels/ep_67_588997', n_gpu=2, no_cuda=False, num_beams=5, num_keep_best=1, num_labels=2, num_return_sequences=1, num_train_epochs=30, num_workers=4, output_dir='output/', output_hidden_states=False, output_mode='classification', per_gpu_eval_batch_size=128, per_gpu_train_batch_size=64, repetition_penalty=1, save_steps=5000, scheduler='linear', scst=False, seed=88, temperature=1, test_yaml='oscar/coco_caption/test.yaml', tokenizer_name='', top_k=0, top_p=1, train_yaml='oscar/coco_caption/train.yaml', use_cbs=False, val_yaml='oscar/coco_caption/val.yaml', warmup_steps=0, weight_decay=0.05)
/home/u2/바탕화면/Kim_Project/Oscar/oscar/utils/misc.py:33: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  return yaml.load(fp)
2021-02-16 03:37:09,671 vlpretrain INFO: ***** Running training *****
INFO:vlpretrain:***** Running training *****
2021-02-16 03:37:09,672 vlpretrain INFO:   Num examples = 566747
INFO:vlpretrain:  Num examples = 566747
2021-02-16 03:37:09,672 vlpretrain INFO:   Num Epochs = 30
INFO:vlpretrain:  Num Epochs = 30
2021-02-16 03:37:09,672 vlpretrain INFO:   Batch size per GPU = 64
INFO:vlpretrain:  Batch size per GPU = 64
2021-02-16 03:37:09,672 vlpretrain INFO:   Total train batch size (w. parallel, & accumulation) = 128
INFO:vlpretrain:  Total train batch size (w. parallel, & accumulation) = 128
2021-02-16 03:37:09,672 vlpretrain INFO:   Gradient Accumulation steps = 1
INFO:vlpretrain:  Gradient Accumulation steps = 1
2021-02-16 03:37:09,672 vlpretrain INFO:   Total optimization steps = 132840
INFO:vlpretrain:  Total optimization steps = 132840
oscar/run_captioning.py:110: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
  return torch.Tensor(features)
oscar/run_captioning.py:110: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
  return torch.Tensor(features)
oscar/run_captioning.py:110: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
  return torch.Tensor(features)
oscar/run_captioning.py:110: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
  return torch.Tensor(features)
Traceback (most recent call last):
  File "oscar/run_captioning.py", line 884, in <module>
    main()
  File "oscar/run_captioning.py", line 863, in main
    global_step, avg_loss = train(args, train_dataset, val_dataset, model, tokenizer)
  File "oscar/run_captioning.py", line 434, in train
    outputs = model(**inputs)
  File "/home/u2/anaconda3/envs/oscar/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/u2/anaconda3/envs/oscar/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 161, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/u2/anaconda3/envs/oscar/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 171, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/u2/anaconda3/envs/oscar/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/home/u2/anaconda3/envs/oscar/lib/python3.7/site-packages/torch/_utils.py", line 428, in reraise
    raise self.exc_type(msg)
StopIteration: Caught StopIteration in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/home/u2/anaconda3/envs/oscar/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/home/u2/anaconda3/envs/oscar/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/u2/바탕화면/Kim_Project/Oscar/oscar/modeling/modeling_bert.py", line 440, in forward
    return self.encode_forward(*args, **kwargs)
  File "/home/u2/바탕화면/Kim_Project/Oscar/oscar/modeling/modeling_bert.py", line 448, in encode_forward
    encoder_history_states=encoder_history_states)
  File "/home/u2/anaconda3/envs/oscar/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/u2/바탕화면/Kim_Project/Oscar/oscar/modeling/modeling_bert.py", line 223, in forward
    extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype) # fp16 compatibility
StopIteration

```

Comments:
Hi! It seems that you're on a very old transformers library version. I would recommend you upgrade to a more recent versions, as this particular has been patched several months ago.

@LysandreJik Hi! thank you for answering. But I'm using version 4.3.2

Are you sure? The error happens on the following line:

```py
    extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype) # fp16 compatibility
```

This does not exist in recent Transformers versions.

It was changed in may: https://github.com/huggingface/transformers/pull/4300

@LysandreJik Oh.. sorry. i was confused. I installed and uninstalled because it interupted run the code, saying it can't import pytorch_transformers.

The code can run with transformers from "https://github.com/huggingface/transformers/tree/067923d3267325f525f4e46f357360c191ba562e."

Is there any way that i can run?
this is the full traceback when i install transformers
```
Traceback (most recent call last):
  File "oscar/run_captioning.py", line 20, in <module>
    from oscar.utils.cbs import ConstraintFilter, ConstraintBoxesReader
  File "/home/u2/바탕화면/Oscar/oscar/utils/cbs.py", line 13, in <module>
    from oscar.modeling.modeling_utils import BeamHypotheses
  File "/home/u2/바탕화면/Oscar/oscar/modeling/modeling_utils.py", line 8, in <module>
    from transformers.pytorch_transformers.modeling_bert import (BertConfig,
ModuleNotFoundError: No module named 'transformers.pytorch_transformers'
```



Oh that's a very old version indeed! Unfrotunately, without knowing what's in your `oscar` folder it's a bit complicated to help you.

For example, the following line is erroneous:

```
from transformers.pytorch_transformers.modeling_bert import (BertConfig,
```

It should simply be 

```
from transformers import BertConfig
```

@LysandreJik It imports many .py files from pytorch_transformers/modeling_bert.

Do you think if i chage below code, I can fix it?
`   extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype) # fp16 compatibility`

![start](https://user-images.githubusercontent.com/75205503/108102129-79db0180-70cb-11eb-82be-4b56ffd3f143.PNG)
![second](https://user-images.githubusercontent.com/75205503/108102132-7ba4c500-70cb-11eb-8e05-dd749e81bdb6.PNG)




This is the try/except we implemented to prevent that error from happening. You can add it to your file:

```py
    try:
        dtype = next(self.parameters()).dtype
    except StopIteration:
        # For nn.DataParallel compatibility in PyTorch 1.5

        def find_tensor_attributes(module: nn.Module) -> List[Tuple[str, Tensor]]:
            tuples = [(k, v) for k, v in module.__dict__.items() if torch.is_tensor(v)]
            return tuples

        gen = self._named_members(get_members_fn=find_tensor_attributes)
        first_tuple = next(gen)
        dtype = first_tuple[1].dtype

    extended_attention_mask = extended_attention_mask.to(dtype)
```

@LysandreJik I added it on my modeling_bert.py. 

I got this traceback.
I guess it's hard to fix it just editing some code from original file.

I really appreciate your help!

if you have any idea to solve it, please share with me. 
Thank you again.
```
Traceback (most recent call last):
  File "oscar/run_captioning.py", line 886, in <module>
    main()
  File "oscar/run_captioning.py", line 865, in main
    global_step, avg_loss = train(args, train_dataset, val_dataset, model, tokenizer)
  File "oscar/run_captioning.py", line 436, in train
    outputs = model(**inputs)
  File "/home/u2/anaconda3/envs/oscar/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/u2/anaconda3/envs/oscar/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 161, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/u2/anaconda3/envs/oscar/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 171, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/u2/anaconda3/envs/oscar/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/home/u2/anaconda3/envs/oscar/lib/python3.7/site-packages/torch/_utils.py", line 428, in reraise
    raise self.exc_type(msg)
NameError: Caught NameError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/home/u2/바탕화면/Oscar/oscar/modeling/modeling_bert.py", line 224, in forward
    dtype = next(self.parameters()).dtype
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/u2/anaconda3/envs/oscar/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/home/u2/anaconda3/envs/oscar/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/u2/바탕화면/Oscar/oscar/modeling/modeling_bert.py", line 453, in forward
    return self.encode_forward(*args, **kwargs)
  File "/home/u2/바탕화면/Oscar/oscar/modeling/modeling_bert.py", line 461, in encode_forward
    encoder_history_states=encoder_history_states)
  File "/home/u2/anaconda3/envs/oscar/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/u2/바탕화면/Oscar/oscar/modeling/modeling_bert.py", line 227, in forward
    def find_tensor_attributes(module: nn.Module) -> List[Tuple[str, Tensor]]:
NameError: name 'List' is not defined

```


These are the type hints. Just remove them if you don't need them or don't want to import them:

```py
    try:
        dtype = next(self.parameters()).dtype
    except StopIteration:
        # For nn.DataParallel compatibility in PyTorch 1.5

        def find_tensor_attributes(module):
            tuples = [(k, v) for k, v in module.__dict__.items() if torch.is_tensor(v)]
            return tuples

        gen = self._named_members(get_members_fn=find_tensor_attributes)
        first_tuple = next(gen)
        dtype = first_tuple[1].dtype

    extended_attention_mask = extended_attention_mask.to(dtype)
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.