Error when running run_squad.py in colab
Author: vamshi975
Pull request: no
Labels: name: wontfix

Hi I used the below code which was given as an example:

!python -m torch.distributed.launch --nproc_per_node=8 ./examples/run_squad.py \
    --model_type bert \
    --model_name_or_path bert-large-uncased-whole-word-masking \
    --do_train \
    --do_eval \
    --do_lower_case \
    --train_file SQUAD_DIR/train-v1.1.json \
    --predict_file SQUAD_DIR/dev-v1.1.json \
    --learning_rate 3e-5 \
    --num_train_epochs 2 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir wwm_uncased_finetuned_squad/ \
    --per_gpu_eval_batch_size=3   \
    --per_gpu_train_batch_size=3   \

 and I tested it colab notebook but it throwed some error as following:

Traceback (most recent call last):
  File "./examples/run_squad.py", line 527, in <module>
    main()
  File "./examples/run_squad.py", line 439, in main
    torch.distributed.init_process_group(backend='nccl')
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py", line 406, in init_process_group
    store, rank, world_size = next(rendezvous(url))
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/rendezvous.py", line 143, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, world_size, start_daemon)
RuntimeError: Address already in use
THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=33 error=10 : invalid device ordinal
Traceback (most recent call last):
  File "./examples/run_squad.py", line 527, in <module>
    main()
  File "./examples/run_squad.py", line 437, in main
    torch.cuda.set_device(args.local_rank)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 265, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (10) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:33
THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=33 error=10 : invalid device ordinal
Traceback (most recent call last):
  File "./examples/run_squad.py", line 527, in <module>
THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=33 error=10 : invalid device ordinal
    main()
  File "./examples/run_squad.py", line 437, in main
Traceback (most recent call last):
  File "./examples/run_squad.py", line 527, in <module>
    torch.cuda.set_device(args.local_rank)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 265, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (10) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:33
    main()
  File "./examples/run_squad.py", line 437, in main
    torch.cuda.set_device(args.local_rank)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 265, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (10) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:33
THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=33 error=10 : invalid device ordinal
Traceback (most recent call last):
  File "./examples/run_squad.py", line 527, in <module>
    main()
  File "./examples/run_squad.py", line 437, in main
    torch.cuda.set_device(args.local_rank)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 265, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (10) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:33
THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=33 error=10 : invalid device ordinal
Traceback (most recent call last):
  File "./examples/run_squad.py", line 527, in <module>
    main()
  File "./examples/run_squad.py", line 437, in main
    torch.cuda.set_device(args.local_rank)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 265, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (10) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:33
THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=33 error=10 : invalid device ordinal
Traceback (most recent call last):
  File "./examples/run_squad.py", line 527, in <module>
    main()
  File "./examples/run_squad.py", line 437, in main
    torch.cuda.set_device(args.local_rank)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 265, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (10) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:33
THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=33 error=10 : invalid device ordinal
Traceback (most recent call last):
  File "./examples/run_squad.py", line 527, in <module>
    main()
  File "./examples/run_squad.py", line 437, in main
    torch.cuda.set_device(args.local_rank)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 265, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (10) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:33
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/launch.py", line 235, in <module>
    main()
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/launch.py", line 231, in main
    cmd=process.args)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-u', './examples/run_squad.py', '--local_rank=0', '--model_type', 'bert', '--model_name_or_path', 'bert-large-uncased-whole-word-masking', '--do_train', '--do_eval', '--do_lower_case', '--train_file', 'SQUAD_DIR/train-v1.1.json', '--predict_file', 'SQUAD_DIR/dev-v1.1.json', '--learning_rate', '3e-5', '--num_train_epochs', '2', '--max_seq_length', '384', '--doc_stride', '128', '--output_dir', 'wwm_uncased_finetuned_squad/', '--per_gpu_eval_batch_size=3', '--per_gpu_train_batch_size=3']' returned non-zero exit status 1.


Previously I used other bert package by huggingface (before pytorch-transformers), It worked fine and was fast when used fp16 argument. But after changing it to Pytorch-transformers this is not working.

Can anyone help me in this regard?



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.