torchelastic multinode training fail
I am trying to run a 2 node distributed training on kubernetes cluster. I am using pytorch lightning with torchelastic. However I keep getting RuntimeError: NCCL error in:  invalid argument, NCCL version 2.4.8. I tried to run the code without pytorch lightning the code works fine.  Here is the logs from the two worker nodes
Packages:
       - torchelastic: 0.2.0
        - pyTorch_version:   1.6.0
        - pytorch-lightning: 0.8.5

worker-0
Running torchelastic.distributed.launch with args: ['/opt/conda/lib/python3.7/site-packages/torchelastic/distributed/launch.py', '--rdzv_backend=etcd', '--rdzv_endpoint=etcd-service:2379', '--rdzv_id=mnist12', '--nnodes=1:2', '--nproc_per_node=1', '/workspace/script.py', '--gpus=1', '--distributed_backend=ddp', '--num_workers=0', '--replace_sampler_ddp=False']
[INFO] 2020-08-18 05:25:52,881 launch: Running torchelastic.distributed.launch with args: ['/opt/conda/lib/python3.7/site-packages/torchelastic/distributed/launch.py', '--rdzv_backend=etcd', '--rdzv_endpoint=etcd-service:2379', '--rdzv_id=mnist12', '--nnodes=1:2', '--nproc_per_node=1', '/workspace/script.py', '--gpus=1', '--distributed_backend=ddp', '--num_workers=0', '--replace_sampler_ddp=False']
[INFO] 2020-08-18 05:25:52,882 launch: Using nproc_per_node=1.
INFO 2020-08-18 05:25:52,888 Etcd machines: ['http://0.0.0.0:2379']
[default] starting workers for function: wrapper_fn
[INFO] 2020-08-18 05:25:53,661 api: [default] starting workers for function: wrapper_fn
[default] Rendezvous'ing worker group
[INFO] 2020-08-18 05:25:53,661 api: [default] Rendezvous'ing worker group
INFO 2020-08-18 05:25:53,661 Attempting to join next rendezvous
INFO 2020-08-18 05:25:53,670 New rendezvous state created: {'status': 'joinable', 'version': '1', 'participants': []}
INFO 2020-08-18 05:25:53,756 Joined rendezvous version 1 as rank 0. Full state: {'status': 'joinable', 'version': '1', 'participants': [0]}
INFO 2020-08-18 05:25:53,756 Rank 0 is responsible for join last call.
INFO 2020-08-18 05:25:53,823 Rank 0 finished join last call.
INFO 2020-08-18 05:25:53,823 Waiting for remaining peers.
INFO 2020-08-18 05:25:53,825 All peers arrived. Confirming membership.
INFO 2020-08-18 05:25:53,843 Waiting for confirmations from all peers.
INFO 2020-08-18 05:25:53,891 Rendezvous version 1 is complete. Final state: {'status': 'final', 'version': '1', 'participants': [0, 1], 'keep_alives': ['/torchelastic/p2p/run_mnist12/rdzv/v_1/rank_0', '/torchelastic/p2p/run_mnist12/rdzv/v_1/rank_1'], 'num_workers_waiting': 0}
INFO 2020-08-18 05:25:53,892 Creating EtcdStore as the c10d::Store implementation
/opt/conda/lib/python3.7/site-packages/torchelastic/utils/store.py:53: FutureWarning: This is an experimental API and will be changed in future.
  "This is an experimental API and will be changed in future.", FutureWarning
[default] Rendezvous complete for workers.
Result:
	restart_count=0
	group_rank=0
	group_world_size=2
	master_addr=mnist12-worker-0
	master_port=55169
	workers={'local_ranks': [0], 'global_ranks': [0], 'role_ranks': [0], 'world_size': 2, 'role_world_size': 2}

[INFO] 2020-08-18 05:25:53,905 api: [default] Rendezvous complete for workers.
Result:
	restart_count=0
	group_rank=0
	group_world_size=2
	master_addr=mnist12-worker-0
	master_port=55169
	workers={'local_ranks': [0], 'global_ranks': [0], 'role_ranks': [0], 'world_size': 2, 'role_world_size': 2}

[default] Starting worker group
[INFO] 2020-08-18 05:25:53,905 api: [default] Starting worker group
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:25: UserWarning: WORLD_SIZE environment variable (2) is not equal to the computed world size (1). Ignored.
  warnings.warn(*args, **kwargs)
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/1
----------------------------------------------------------------------------------------------------
distributed_backend=ddp
All DDP processes registered. Starting ddp with 1 processes
----------------------------------------------------------------------------------------------------
mnist12-worker-0:26:26 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.60.90<0>
mnist12-worker-0:26:26 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).

mnist12-worker-0:26:26 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
mnist12-worker-0:26:26 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.60.90<0>
NCCL version 2.4.8+cuda10.1
mnist12-worker-0:26:34 [0] NCCL INFO Setting affinity for GPU 0 to 0f
mnist12-worker-0:26:34 [0] NCCL INFO Using 256 threads, Min Comp Cap 7, Trees enabled up to size -2
mnist12-worker-0:26:34 [0] NCCL INFO comm 0x7f5438002280 rank 0 nranks 1 cudaDev 0 nvmlDev 0 - Init COMPLETE

  | Name      | Type             | Params
-----------------------------------------------
0 | model     | ResNet           | 11 M
1 | criterion | CrossEntropyLoss | 0
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:25: UserWarning: The dataloader, val dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  warnings.warn(*args, **kwargs)
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:25: UserWarning: The dataloader, train dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  warnings.warn(*args, **kwargs)
[default] Detected 1 new nodes from group_rank=0; will restart worker group
[INFO] 2020-08-18 05:26:08,936 api: [default] Detected 1 new nodes from group_rank=0; will restart worker group
[default] Stopping worker group
[INFO] 2020-08-18 05:26:08,936 api: [default] Stopping worker group
[default] Rendezvous'ing worker group
[INFO] 2020-08-18 05:26:09,195 api: [default] Rendezvous'ing worker group
INFO 2020-08-18 05:26:09,196 Attempting to join next rendezvous
INFO 2020-08-18 05:26:09,200 Observed existing rendezvous state: {'status': 'final', 'version': '1', 'participants': [0, 1], 'keep_alives': ['/torchelastic/p2p/run_mnist12/rdzv/v_1/rank_0', '/torchelastic/p2p/run_mnist12/rdzv/v_1/rank_1'], 'num_workers_waiting': 1}
INFO 2020-08-18 05:26:09,232 Added self to waiting list. Rendezvous full state: {"status": "final", "version": "1", "participants": [0, 1], "keep_alives": ["/torchelastic/p2p/run_mnist12/rdzv/v_1/rank_0", "/torchelastic/p2p/run_mnist12/rdzv/v_1/rank_1"], "num_workers_waiting": 2}
INFO 2020-08-18 05:26:13,992 Keep-alive key /torchelastic/p2p/run_mnist12/rdzv/v_1/rank_1 is not renewed.
INFO 2020-08-18 05:26:13,992 Rendevous version 1 is incomplete.
INFO 2020-08-18 05:26:13,992 Attempting to destroy it.
INFO 2020-08-18 05:26:13,995 Rendezvous attempt failed, will retry. Reason: Compare failed : [{"status": "final", "version": "1", "participants": [0, 1], "keep_alives": ["/torchelastic/p2p/run_mnist12/rdzv/v_1/rank_0", "/torchelastic/p2p/run_mnist12/rdzv/v_1/rank_1"], "num_workers_waiting": 2} != {"status": "setup"}]
INFO 2020-08-18 05:26:14,996 Attempting to join next rendezvous
INFO 2020-08-18 05:26:15,000 Observed existing rendezvous state: {'status': 'joinable', 'version': '2', 'participants': [0]}
INFO 2020-08-18 05:26:15,031 Joined rendezvous version 2 as rank 1. Full state: {'status': 'frozen', 'version': '2', 'participants': [0, 1], 'keep_alives': []}
INFO 2020-08-18 05:26:15,032 Waiting for remaining peers.
INFO 2020-08-18 05:26:15,033 All peers arrived. Confirming membership.
INFO 2020-08-18 05:26:15,051 Waiting for confirmations from all peers.
INFO 2020-08-18 05:26:15,104 Rendezvous version 2 is complete. Final state: {'status': 'final', 'version': '2', 'participants': [0, 1], 'keep_alives': ['/torchelastic/p2p/run_mnist12/rdzv/v_2/rank_1', '/torchelastic/p2p/run_mnist12/rdzv/v_2/rank_0'], 'num_workers_waiting': 0}
INFO 2020-08-18 05:26:15,104 Creating EtcdStore as the c10d::Store implementation
[default] Rendezvous complete for workers.
Result:
	restart_count=0
	group_rank=1
	group_world_size=2
	master_addr=mnist12-worker-1
	master_port=40213
	workers={'local_ranks': [0], 'global_ranks': [1], 'role_ranks': [1], 'world_size': 2, 'role_world_size': 2}

[INFO] 2020-08-18 05:26:15,114 api: [default] Rendezvous complete for workers.
Result:
	restart_count=0
	group_rank=1
	group_world_size=2
	master_addr=mnist12-worker-1
	master_port=40213
	workers={'local_ranks': [0], 'global_ranks': [1], 'role_ranks': [1], 'world_size': 2, 'role_world_size': 2}

[default] Starting worker group
[INFO] 2020-08-18 05:26:15,114 api: [default] Starting worker group
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/1
Epoch 1:   2%|▏         | 32/1720 [00:09<07:57,  3.54it/s, loss=6.834, v_num=0]mnist12-worker-0:46:46 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.60.90<0>
mnist12-worker-0:46:46 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).

mnist12-worker-0:46:46 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
mnist12-worker-0:46:46 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.60.90<0>

mnist12-worker-0:46:46 [0] init.cc:981 NCCL WARN Invalid rank requested : 1/1
Traceback (most recent call last):
  File "/workspace/script.py", line 236, in <module>
    run_cli()
  File "/workspace/script.py", line 232, in run_cli
    main(args)
  File "/workspace/script.py", line 211, in main
    trainer.fit(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 982, in fit
    self.ddp_train(process_idx=task, q=None, model=model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 557, in ddp_train
    model = model.configure_ddp(model, device_ids)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/core/lightning.py", line 899, in configure_ddp
    find_unused_parameters=True
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 333, in __init__
    self.broadcast_bucket_size)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 549, in _distributed_broadcast_coalesced
    dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1595629403081/work/torch/lib/c10d/../c10d/NCCLUtils.hpp:82, invalid argument, NCCL version 2.4.8
[default] Worker group failed
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 224, in _monitor_workers
    if self._process_context.join(timeout=-1):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 119, in join
    raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 101, in _wrap
    ret = fn(*args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/distributed/launch.py", line 401, in wrapper_fn
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', '/workspace/script.py', '--gpus=1', '--distributed_backend=ddp', '--num_workers=0', '--replace_sampler_ddp=False']' returned non-zero exit status 1.

[ERROR] 2020-08-18 05:26:25,134 local_elastic_agent: [default] Worker group failed
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 224, in _monitor_workers
    if self._process_context.join(timeout=-1):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 119, in join
    raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 101, in _wrap
    ret = fn(*args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/distributed/launch.py", line 401, in wrapper_fn
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', '/workspace/script.py', '--gpus=1', '--distributed_backend=ddp', '--num_workers=0', '--replace_sampler_ddp=False']' returned non-zero exit status 1.

[default] Worker group FAILED. 3/3 attempts left; will restart worker group
[INFO] 2020-08-18 05:26:25,134 api: [default] Worker group FAILED. 3/3 attempts left; will restart worker group
[default] Stopping worker group
[INFO] 2020-08-18 05:26:25,134 api: [default] Stopping worker group
[default] Rendezvous'ing worker group
[INFO] 2020-08-18 05:26:25,134 api: [default] Rendezvous'ing worker group
INFO 2020-08-18 05:26:25,134 Attempting to join next rendezvous
INFO 2020-08-18 05:26:25,138 Observed existing rendezvous state: {'status': 'final', 'version': '2', 'participants': [0, 1], 'keep_alives': ['/torchelastic/p2p/run_mnist12/rdzv/v_2/rank_1', '/torchelastic/p2p/run_mnist12/rdzv/v_2/rank_0'], 'num_workers_waiting': 0}
INFO 2020-08-18 05:26:25,164 Added self to waiting list. Rendezvous full state: {"status": "final", "version": "2", "participants": [0, 1], "keep_alives": ["/torchelastic/p2p/run_mnist12/rdzv/v_2/rank_1", "/torchelastic/p2p/run_mnist12/rdzv/v_2/rank_0"], "num_workers_waiting": 1}
INFO 2020-08-18 05:26:35,491 Keep-alive key /torchelastic/p2p/run_mnist12/rdzv/v_2/rank_1 is not renewed.
INFO 2020-08-18 05:26:35,491 Rendevous version 2 is incomplete.
INFO 2020-08-18 05:26:35,492 Attempting to destroy it.
INFO 2020-08-18 05:26:35,494 Rendezvous attempt failed, will retry. Reason: Compare failed : [{"status": "final", "version": "2", "participants": [0, 1], "keep_alives": ["/torchelastic/p2p/run_mnist12/rdzv/v_2/rank_1", "/torchelastic/p2p/run_mnist12/rdzv/v_2/rank_0"], "num_workers_waiting": 2} != {"status": "setup"}]
INFO 2020-08-18 05:26:36,494 Attempting to join next rendezvous
INFO 2020-08-18 05:26:36,498 Observed existing rendezvous state: {'status': 'joinable', 'version': '3', 'participants': [0]}
INFO 2020-08-18 05:26:36,538 Joined rendezvous version 3 as rank 1. Full state: {'status': 'frozen', 'version': '3', 'participants': [0, 1], 'keep_alives': []}
INFO 2020-08-18 05:26:36,538 Waiting for remaining peers.
INFO 2020-08-18 05:26:36,540 All peers arrived. Confirming membership.
INFO 2020-08-18 05:26:36,635 Waiting for confirmations from all peers.
INFO 2020-08-18 05:26:36,637 Rendezvous version 3 is complete. Final state: {'status': 'final', 'version': '3', 'participants': [0, 1], 'keep_alives': ['/torchelastic/p2p/run_mnist12/rdzv/v_3/rank_0', '/torchelastic/p2p/run_mnist12/rdzv/v_3/rank_1'], 'num_workers_waiting': 0}
INFO 2020-08-18 05:26:36,638 Creating EtcdStore as the c10d::Store implementation
[default] Rendezvous complete for workers.
Result:
	restart_count=1
	group_rank=1
	group_world_size=2
	master_addr=mnist12-worker-1
	master_port=38529
	workers={'local_ranks': [0], 'global_ranks': [1], 'role_ranks': [1], 'world_size': 2, 'role_world_size': 2}

[INFO] 2020-08-18 05:26:36,647 api: [default] Rendezvous complete for workers.
Result:
	restart_count=1
	group_rank=1
	group_world_size=2
	master_addr=mnist12-worker-1
	master_port=38529
	workers={'local_ranks': [0], 'global_ranks': [1], 'role_ranks': [1], 'world_size': 2, 'role_world_size': 2}

[default] Starting worker group
[INFO] 2020-08-18 05:26:36,647 api: [default] Starting worker group
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/1
mnist12-worker-0:58:58 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.60.90<0>
mnist12-worker-0:58:58 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).

mnist12-worker-0:58:58 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
mnist12-worker-0:58:58 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.60.90<0>

mnist12-worker-0:58:58 [0] init.cc:981 NCCL WARN Invalid rank requested : 1/1
Traceback (most recent call last):
  File "/workspace/script.py", line 236, in <module>
    run_cli()
  File "/workspace/script.py", line 232, in run_cli
    main(args)
  File "/workspace/script.py", line 211, in main
    trainer.fit(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 982, in fit
    self.ddp_train(process_idx=task, q=None, model=model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 557, in ddp_train
    model = model.configure_ddp(model, device_ids)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/core/lightning.py", line 899, in configure_ddp
    find_unused_parameters=True
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 333, in __init__
    self.broadcast_bucket_size)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 549, in _distributed_broadcast_coalesced
    dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1595629403081/work/torch/lib/c10d/../c10d/NCCLUtils.hpp:82, invalid argument, NCCL version 2.4.8
[default] Worker group failed
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 224, in _monitor_workers
    if self._process_context.join(timeout=-1):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 119, in join
    raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 101, in _wrap
    ret = fn(*args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/distributed/launch.py", line 401, in wrapper_fn
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', '/workspace/script.py', '--gpus=1', '--distributed_backend=ddp', '--num_workers=0', '--replace_sampler_ddp=False']' returned non-zero exit status 1.

[ERROR] 2020-08-18 05:26:46,661 local_elastic_agent: [default] Worker group failed
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 224, in _monitor_workers
    if self._process_context.join(timeout=-1):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 119, in join
    raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 101, in _wrap
    ret = fn(*args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/distributed/launch.py", line 401, in wrapper_fn
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', '/workspace/script.py', '--gpus=1', '--distributed_backend=ddp', '--num_workers=0', '--replace_sampler_ddp=False']' returned non-zero exit status 1.

[default] Worker group FAILED. 2/3 attempts left; will restart worker group
[INFO] 2020-08-18 05:26:46,662 api: [default] Worker group FAILED. 2/3 attempts left; will restart worker group
[default] Stopping worker group
[INFO] 2020-08-18 05:26:46,662 api: [default] Stopping worker group
[default] Rendezvous'ing worker group
[INFO] 2020-08-18 05:26:46,662 api: [default] Rendezvous'ing worker group
INFO 2020-08-18 05:26:46,662 Attempting to join next rendezvous
INFO 2020-08-18 05:26:46,665 Observed existing rendezvous state: {'status': 'final', 'version': '3', 'participants': [0, 1], 'keep_alives': ['/torchelastic/p2p/run_mnist12/rdzv/v_3/rank_0', '/torchelastic/p2p/run_mnist12/rdzv/v_3/rank_1'], 'num_workers_waiting': 0}
INFO 2020-08-18 05:26:46,730 Added self to waiting list. Rendezvous full state: {"status": "final", "version": "3", "participants": [0, 1], "keep_alives": ["/torchelastic/p2p/run_mnist12/rdzv/v_3/rank_0", "/torchelastic/p2p/run_mnist12/rdzv/v_3/rank_1"], "num_workers_waiting": 1}
INFO 2020-08-18 05:26:56,992 Keep-alive key /torchelastic/p2p/run_mnist12/rdzv/v_3/rank_1 is not renewed.
INFO 2020-08-18 05:26:56,992 Rendevous version 3 is incomplete.
INFO 2020-08-18 05:26:56,992 Attempting to destroy it.
INFO 2020-08-18 05:26:56,994 Rendezvous attempt failed, will retry. Reason: Compare failed : [{"status": "final", "version": "3", "participants": [0, 1], "keep_alives": ["/torchelastic/p2p/run_mnist12/rdzv/v_3/rank_0", "/torchelastic/p2p/run_mnist12/rdzv/v_3/rank_1"], "num_workers_waiting": 2} != {"status": "setup"}]
INFO 2020-08-18 05:26:57,995 Attempting to join next rendezvous
INFO 2020-08-18 05:26:57,999 Observed existing rendezvous state: {'status': 'joinable', 'version': '4', 'participants': [0]}
INFO 2020-08-18 05:26:58,015 Joined rendezvous version 4 as rank 1. Full state: {'status': 'frozen', 'version': '4', 'participants': [0, 1], 'keep_alives': []}
INFO 2020-08-18 05:26:58,015 Waiting for remaining peers.
INFO 2020-08-18 05:26:58,016 All peers arrived. Confirming membership.
INFO 2020-08-18 05:26:58,116 Confirm membership CAS unsuccessful, retrying
INFO 2020-08-18 05:26:58,196 Waiting for confirmations from all peers.
INFO 2020-08-18 05:26:58,198 Rendezvous version 4 is complete. Final state: {'status': 'final', 'version': '4', 'participants': [0, 1], 'keep_alives': ['/torchelastic/p2p/run_mnist12/rdzv/v_4/rank_0', '/torchelastic/p2p/run_mnist12/rdzv/v_4/rank_1'], 'num_workers_waiting': 0}
INFO 2020-08-18 05:26:58,198 Creating EtcdStore as the c10d::Store implementation
[default] Rendezvous complete for workers.
Result:
	restart_count=2
	group_rank=1
	group_world_size=2
	master_addr=mnist12-worker-1
	master_port=41561
	workers={'local_ranks': [0], 'global_ranks': [1], 'role_ranks': [1], 'world_size': 2, 'role_world_size': 2}

[INFO] 2020-08-18 05:26:58,207 api: [default] Rendezvous complete for workers.
Result:
	restart_count=2
	group_rank=1
	group_world_size=2
	master_addr=mnist12-worker-1
	master_port=41561
	workers={'local_ranks': [0], 'global_ranks': [1], 'role_ranks': [1], 'world_size': 2, 'role_world_size': 2}

[default] Starting worker group
[INFO] 2020-08-18 05:26:58,207 api: [default] Starting worker group
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/1
mnist12-worker-0:70:70 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.60.90<0>
mnist12-worker-0:70:70 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).

mnist12-worker-0:70:70 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
mnist12-worker-0:70:70 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.60.90<0>

mnist12-worker-0:70:70 [0] init.cc:981 NCCL WARN Invalid rank requested : 1/1
Traceback (most recent call last):
  File "/workspace/script.py", line 236, in <module>
    run_cli()
  File "/workspace/script.py", line 232, in run_cli
    main(args)
  File "/workspace/script.py", line 211, in main
    trainer.fit(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 982, in fit
    self.ddp_train(process_idx=task, q=None, model=model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 557, in ddp_train
    model = model.configure_ddp(model, device_ids)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/core/lightning.py", line 899, in configure_ddp
    find_unused_parameters=True
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 333, in __init__
    self.broadcast_bucket_size)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 549, in _distributed_broadcast_coalesced
    dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1595629403081/work/torch/lib/c10d/../c10d/NCCLUtils.hpp:82, invalid argument, NCCL version 2.4.8
[default] Worker group failed
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 224, in _monitor_workers
    if self._process_context.join(timeout=-1):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 119, in join
    raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 101, in _wrap
    ret = fn(*args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/distributed/launch.py", line 401, in wrapper_fn
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', '/workspace/script.py', '--gpus=1', '--distributed_backend=ddp', '--num_workers=0', '--replace_sampler_ddp=False']' returned non-zero exit status 1.

[ERROR] 2020-08-18 05:27:08,227 local_elastic_agent: [default] Worker group failed
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 224, in _monitor_workers
    if self._process_context.join(timeout=-1):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 119, in join
    raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 101, in _wrap
    ret = fn(*args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/distributed/launch.py", line 401, in wrapper_fn
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', '/workspace/script.py', '--gpus=1', '--distributed_backend=ddp', '--num_workers=0', '--replace_sampler_ddp=False']' returned non-zero exit status 1.

[default] Worker group FAILED. 1/3 attempts left; will restart worker group
[INFO] 2020-08-18 05:27:08,227 api: [default] Worker group FAILED. 1/3 attempts left; will restart worker group
[default] Stopping worker group
[INFO] 2020-08-18 05:27:08,227 api: [default] Stopping worker group
[default] Rendezvous'ing worker group
[INFO] 2020-08-18 05:27:08,227 api: [default] Rendezvous'ing worker group
INFO 2020-08-18 05:27:08,227 Attempting to join next rendezvous
INFO 2020-08-18 05:27:08,231 Observed existing rendezvous state: {'status': 'final', 'version': '4', 'participants': [0, 1], 'keep_alives': ['/torchelastic/p2p/run_mnist12/rdzv/v_4/rank_0', '/torchelastic/p2p/run_mnist12/rdzv/v_4/rank_1'], 'num_workers_waiting': 0}
INFO 2020-08-18 05:27:08,257 Added self to waiting list. Rendezvous full state: {"status": "final", "version": "4", "participants": [0, 1], "keep_alives": ["/torchelastic/p2p/run_mnist12/rdzv/v_4/rank_0", "/torchelastic/p2p/run_mnist12/rdzv/v_4/rank_1"], "num_workers_waiting": 1}
INFO 2020-08-18 05:27:18,492 Keep-alive key /torchelastic/p2p/run_mnist12/rdzv/v_4/rank_1 is not renewed.
INFO 2020-08-18 05:27:18,492 Rendevous version 4 is incomplete.
INFO 2020-08-18 05:27:18,492 Attempting to destroy it.
INFO 2020-08-18 05:27:18,494 Rendezvous attempt failed, will retry. Reason: Compare failed : [{"status": "final", "version": "4", "participants": [0, 1], "keep_alives": ["/torchelastic/p2p/run_mnist12/rdzv/v_4/rank_0", "/torchelastic/p2p/run_mnist12/rdzv/v_4/rank_1"], "num_workers_waiting": 2} != {"status": "setup"}]
INFO 2020-08-18 05:27:19,495 Attempting to join next rendezvous
INFO 2020-08-18 05:27:19,499 Observed existing rendezvous state: {'status': 'joinable', 'version': '5', 'participants': [0]}
INFO 2020-08-18 05:27:19,506 Joined rendezvous version 5 as rank 1. Full state: {'status': 'frozen', 'version': '5', 'participants': [0, 1], 'keep_alives': []}
INFO 2020-08-18 05:27:19,507 Waiting for remaining peers.
INFO 2020-08-18 05:27:19,508 All peers arrived. Confirming membership.
INFO 2020-08-18 05:27:19,606 Waiting for confirmations from all peers.
INFO 2020-08-18 05:27:19,607 Rendezvous version 5 is complete. Final state: {'status': 'final', 'version': '5', 'participants': [0, 1], 'keep_alives': ['/torchelastic/p2p/run_mnist12/rdzv/v_5/rank_0', '/torchelastic/p2p/run_mnist12/rdzv/v_5/rank_1'], 'num_workers_waiting': 0}
INFO 2020-08-18 05:27:19,608 Creating EtcdStore as the c10d::Store implementation
[default] Rendezvous complete for workers.
Result:
	restart_count=3
	group_rank=1
	group_world_size=2
	master_addr=mnist12-worker-1
	master_port=33769
	workers={'local_ranks': [0], 'global_ranks': [1], 'role_ranks': [1], 'world_size': 2, 'role_world_size': 2}

[INFO] 2020-08-18 05:27:19,618 api: [default] Rendezvous complete for workers.
Result:
	restart_count=3
	group_rank=1
	group_world_size=2
	master_addr=mnist12-worker-1
	master_port=33769
	workers={'local_ranks': [0], 'global_ranks': [1], 'role_ranks': [1], 'world_size': 2, 'role_world_size': 2}

[default] Starting worker group
[INFO] 2020-08-18 05:27:19,618 api: [default] Starting worker group
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/1
mnist12-worker-0:82:82 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.60.90<0>
mnist12-worker-0:82:82 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).

mnist12-worker-0:82:82 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
mnist12-worker-0:82:82 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.60.90<0>

mnist12-worker-0:82:82 [0] init.cc:981 NCCL WARN Invalid rank requested : 1/1
Traceback (most recent call last):
  File "/workspace/script.py", line 236, in <module>
    run_cli()
  File "/workspace/script.py", line 232, in run_cli
    main(args)
  File "/workspace/script.py", line 211, in main
    trainer.fit(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 982, in fit
    self.ddp_train(process_idx=task, q=None, model=model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 557, in ddp_train
    model = model.configure_ddp(model, device_ids)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/core/lightning.py", line 899, in configure_ddp
    find_unused_parameters=True
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 333, in __init__
    self.broadcast_bucket_size)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 549, in _distributed_broadcast_coalesced
    dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1595629403081/work/torch/lib/c10d/../c10d/NCCLUtils.hpp:82, invalid argument, NCCL version 2.4.8
[default] Worker group failed
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 224, in _monitor_workers
    if self._process_context.join(timeout=-1):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 119, in join
    raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 101, in _wrap
    ret = fn(*args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/distributed/launch.py", line 401, in wrapper_fn
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', '/workspace/script.py', '--gpus=1', '--distributed_backend=ddp', '--num_workers=0', '--replace_sampler_ddp=False']' returned non-zero exit status 1.

[ERROR] 2020-08-18 05:27:29,637 local_elastic_agent: [default] Worker group failed
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 224, in _monitor_workers
    if self._process_context.join(timeout=-1):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 119, in join
    raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 101, in _wrap
    ret = fn(*args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/distributed/launch.py", line 401, in wrapper_fn
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', '/workspace/script.py', '--gpus=1', '--distributed_backend=ddp', '--num_workers=0', '--replace_sampler_ddp=False']' returned non-zero exit status 1.

Local worker group finished (WorkerState.FAILED). Waiting 300 seconds for other agents to finish
[INFO] 2020-08-18 05:27:29,637 api: Local worker group finished (WorkerState.FAILED). Waiting 300 seconds for other agents to finish

Worker-1
Running torchelastic.distributed.launch with args: ['/opt/conda/lib/python3.7/site-packages/torchelastic/distributed/launch.py', '--rdzv_backend=etcd', '--rdzv_endpoint=etcd-service:2379', '--rdzv_id=mnist12', '--nnodes=1:2', '--nproc_per_node=1', '/workspace/script.py', '--gpus=1', '--distributed_backend=ddp', '--num_workers=0', '--replace_sampler_ddp=False']
[INFO] 2020-08-18 05:25:52,909 launch: Running torchelastic.distributed.launch with args: ['/opt/conda/lib/python3.7/site-packages/torchelastic/distributed/launch.py', '--rdzv_backend=etcd', '--rdzv_endpoint=etcd-service:2379', '--rdzv_id=mnist12', '--nnodes=1:2', '--nproc_per_node=1', '/workspace/script.py', '--gpus=1', '--distributed_backend=ddp', '--num_workers=0', '--replace_sampler_ddp=False']
[INFO] 2020-08-18 05:25:52,910 launch: Using nproc_per_node=1.
INFO 2020-08-18 05:25:52,914 Etcd machines: ['http://0.0.0.0:2379']
[default] starting workers for function: wrapper_fn
[INFO] 2020-08-18 05:25:53,722 api: [default] starting workers for function: wrapper_fn
[default] Rendezvous'ing worker group
[INFO] 2020-08-18 05:25:53,723 api: [default] Rendezvous'ing worker group
INFO 2020-08-18 05:25:53,723 Attempting to join next rendezvous
INFO 2020-08-18 05:25:53,725 Observed existing rendezvous state: {'status': 'joinable', 'version': '1', 'participants': []}
INFO 2020-08-18 05:25:53,821 Joined rendezvous version 1 as rank 1. Full state: {'status': 'frozen', 'version': '1', 'participants': [0, 1], 'keep_alives': []}
INFO 2020-08-18 05:25:53,822 Waiting for remaining peers.
INFO 2020-08-18 05:25:53,822 All peers arrived. Confirming membership.
INFO 2020-08-18 05:25:53,890 Waiting for confirmations from all peers.
INFO 2020-08-18 05:25:53,891 Rendezvous version 1 is complete. Final state: {'status': 'final', 'version': '1', 'participants': [0, 1], 'keep_alives': ['/torchelastic/p2p/run_mnist12/rdzv/v_1/rank_0', '/torchelastic/p2p/run_mnist12/rdzv/v_1/rank_1'], 'num_workers_waiting': 0}
INFO 2020-08-18 05:25:53,891 Creating EtcdStore as the c10d::Store implementation
/opt/conda/lib/python3.7/site-packages/torchelastic/utils/store.py:53: FutureWarning: This is an experimental API and will be changed in future.
  "This is an experimental API and will be changed in future.", FutureWarning
[default] Rendezvous complete for workers.
Result:
	restart_count=0
	group_rank=1
	group_world_size=2
	master_addr=mnist12-worker-0
	master_port=55169
	workers={'local_ranks': [0], 'global_ranks': [1], 'role_ranks': [1], 'world_size': 2, 'role_world_size': 2}

[INFO] 2020-08-18 05:25:53,902 api: [default] Rendezvous complete for workers.
Result:
	restart_count=0
	group_rank=1
	group_world_size=2
	master_addr=mnist12-worker-0
	master_port=55169
	workers={'local_ranks': [0], 'global_ranks': [1], 'role_ranks': [1], 'world_size': 2, 'role_world_size': 2}

[default] Starting worker group
[INFO] 2020-08-18 05:25:53,902 api: [default] Starting worker group
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/1
mnist12-worker-1:24:24 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.65.152<0>
mnist12-worker-1:24:24 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).

mnist12-worker-1:24:24 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
mnist12-worker-1:24:24 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.65.152<0>

mnist12-worker-1:24:24 [0] init.cc:981 NCCL WARN Invalid rank requested : 1/1
Traceback (most recent call last):
  File "/workspace/script.py", line 236, in <module>
    run_cli()
  File "/workspace/script.py", line 232, in run_cli
    main(args)
  File "/workspace/script.py", line 211, in main
    trainer.fit(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 982, in fit
    self.ddp_train(process_idx=task, q=None, model=model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 557, in ddp_train
    model = model.configure_ddp(model, device_ids)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/core/lightning.py", line 899, in configure_ddp
    find_unused_parameters=True
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 333, in __init__
    self.broadcast_bucket_size)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 549, in _distributed_broadcast_coalesced
    dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1595629403081/work/torch/lib/c10d/../c10d/NCCLUtils.hpp:82, invalid argument, NCCL version 2.4.8
[default] Worker group failed
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 224, in _monitor_workers
    if self._process_context.join(timeout=-1):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 119, in join
    raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 101, in _wrap
    ret = fn(*args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/distributed/launch.py", line 401, in wrapper_fn
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', '/workspace/script.py', '--gpus=1', '--distributed_backend=ddp', '--num_workers=0', '--replace_sampler_ddp=False']' returned non-zero exit status 1.

[ERROR] 2020-08-18 05:26:03,919 local_elastic_agent: [default] Worker group failed
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 224, in _monitor_workers
    if self._process_context.join(timeout=-1):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 119, in join
    raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/agent/server/local_elastic_agent.py", line 101, in _wrap
    ret = fn(*args)
  File "/opt/conda/lib/python3.7/site-packages/torchelastic/distributed/launch.py", line 401, in wrapper_fn
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', '/workspace/script.py', '--gpus=1', '--distributed_backend=ddp', '--num_workers=0', '--replace_sampler_ddp=False']' returned non-zero exit status 1.

[default] Worker group FAILED. 3/3 attempts left; will restart worker group
[INFO] 2020-08-18 05:26:03,919 api: [default] Worker group FAILED. 3/3 attempts left; will restart worker group
[default] Stopping worker group
[INFO] 2020-08-18 05:26:03,920 api: [default] Stopping worker group
[default] Rendezvous'ing worker group
[INFO] 2020-08-18 05:26:03,920 api: [default] Rendezvous'ing worker group
INFO 2020-08-18 05:26:03,920 Attempting to join next rendezvous
INFO 2020-08-18 05:26:03,922 Observed existing rendezvous state: {'status': 'final', 'version': '1', 'participants': [0, 1], 'keep_alives': ['/torchelastic/p2p/run_mnist12/rdzv/v_1/rank_0', '/torchelastic/p2p/run_mnist12/rdzv/v_1/rank_1'], 'num_workers_waiting': 0}
INFO 2020-08-18 05:26:03,975 Added self to waiting list. Rendezvous full state: {"status": "final", "version": "1", "participants": [0, 1], "keep_alives": ["/torchelastic/p2p/run_mnist12/rdzv/v_1/rank_0", "/torchelastic/p2p/run_mnist12/rdzv/v_1/rank_1"], "num_workers_waiting": 1}
INFO 2020-08-18 05:26:13,990 Keep-alive key /torchelastic/p2p/run_mnist12/rdzv/v_1/rank_1 is not renewed.
INFO 2020-08-18 05:26:13,990 Rendevous version 1 is incomplete.
INFO 2020-08-18 05:26:13,990 Attempting to destroy it.
INFO 2020-08-18 05:26:13,991 Destroyed rendezvous version 1 successfully.
INFO 2020-08-18 05:26:13,991 Previously existing rendezvous state changed. Will re-try joining.
INFO 2020-08-18 05:26:13,991 Attempting to join next rendezvous
INFO 2020-08-18 05:26:13,996 New rendezvous state created: {'status': 'joinable', 'version': '2', 'participants': []}
INFO 2020-08-18 05:26:14,087 Joined rendezvous version 2 as rank 0. Full state: {'status': 'joinable', 'version': '2', 'participants': [0]}
INFO 2020-08-18 05:26:14,087 Rank 0 is responsible for join last call.
INFO 2020-08-18 05:26:15,032 Rank 0 finished join last call.
INFO 2020-08-18 05:26:15,032 Waiting for remaining peers.
INFO 2020-08-18 05:26:15,033 All peers arrived. Confirming membership.
INFO 2020-08-18 05:26:15,103 Waiting for confirmations from all peers.
INFO 2020-08-18 05:26:15,104 Rendezvous version 2 is complete. Final state: {'status': 'final', 'version': '2', 'participants': [0, 1], 'keep_alives': ['/torchelastic/p2p/run_mnist12/rdzv/v_2/rank_1', '/torchelastic/p2p/run_mnist12/rdzv/v_2/rank_0'], 'num_workers_waiting': 0}
INFO 2020-08-18 05:26:15,104 Creating EtcdStore as the c10d::Store implementation
[default] Rendezvous complete for workers.
Result:
	restart_count=1
	group_rank=0
	group_world_size=2
	master_addr=mnist12-worker-1
	master_port=40213
	workers={'local_ranks': [0], 'global_ranks': [0], 'role_ranks': [0], 'world_size': 2, 'role_world_size': 2}

[INFO] 2020-08-18 05:26:15,111 api: [default] Rendezvous complete for workers.
Result:
	restart_count=1
	group_rank=0
	group_world_size=2
	master_addr=mnist12-worker-1
	master_port=40213
	workers={'local_ranks': [0], 'global_ranks': [0], 'role_ranks': [0], 'world_size': 2, 'role_world_size': 2}

[default] Starting worker group
[INFO] 2020-08-18 05:26:15,111 api: [default] Starting worker group
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:25: UserWarning: WORLD_SIZE environment variable (2) is not equal to the computed world size (1). Ignored.
  warnings.warn(*args, **kwargs)
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/1
----------------------------------------------------------------------------------------------------
distributed_backend=ddp
All DDP processes registered. Starting ddp with 1 processes
----------------------------------------------------------------------------------------------------
mnist12-worker-1:36:36 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.65.152<0>
mnist12-worker-1:36:36 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).

mnist12-worker-1:36:36 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
mnist12-worker-1:36:36 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.65.152<0>
NCCL version 2.4.8+cuda10.1
mnist12-worker-1:36:44 [0] NCCL INFO Setting affinity for GPU 0 to 0f
mnist12-worker-1:36:44 [0] NCCL INFO Using 256 threads, Min Comp Cap 7, Trees enabled up to size -2
mnist12-worker-1:36:44 [0] NCCL INFO comm 0x7f9f6c002280 rank 0 nranks 1 cudaDev 0 nvmlDev 0 - Init COMPLETE

  | Name      | Type             | Params
-----------------------------------------------
0 | model     | ResNet           | 11 M
1 | criterion | CrossEntropyLoss | 0
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:25: UserWarning: The dataloader, val dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  warnings.warn(*args, **kwargs)
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:25: UserWarning: The dataloader, train dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  warnings.warn(*args, **kwargs)
[default] Detected 1 new nodes from group_rank=0; will restart worker group
[INFO] 2020-08-18 05:26:30,134 api: [default] Detected 1 new nodes from group_rank=0; will restart worker group
[default] Stopping worker group
[INFO] 2020-08-18 05:26:30,134 api: [default] Stopping worker group
[default] Rendezvous'ing worker group
[INFO] 2020-08-18 05:26:30,391 api: [default] Rendezvous'ing worker group
INFO 2020-08-18 05:26:30,391 Attempting to join next rendezvous
INFO 2020-08-18 05:26:30,393 Observed existing rendezvous state: {'status': 'final', 'version': '2', 'participants': [0, 1], 'keep_alives': ['/torchelastic/p2p/run_mnist12/rdzv/v_2/rank_1', '/torchelastic/p2p/run_mnist12/rdzv/v_2/rank_0'], 'num_workers_waiting': 1}
INFO 2020-08-18 05:26:30,434 Added self to waiting list. Rendezvous full state: {"status": "final", "version": "2", "participants": [0, 1], "keep_alives": ["/torchelastic/p2p/run_mnist12/rdzv/v_2/rank_1", "/torchelastic/p2p/run_mnist12/rdzv/v_2/rank_0"], "num_workers_waiting": 2}
INFO 2020-08-18 05:26:35,490 Keep-alive key /torchelastic/p2p/run_mnist12/rdzv/v_2/rank_1 is not renewed.
INFO 2020-08-18 05:26:35,490 Rendevous version 2 is incomplete.
INFO 2020-08-18 05:26:35,490 Attempting to destroy it.
INFO 2020-08-18 05:26:35,491 Destroyed rendezvous version 2 successfully.
INFO 2020-08-18 05:26:35,491 Previously existing rendezvous state changed. Will re-try joining.
INFO 2020-08-18 05:26:35,491 Attempting to join next rendezvous
INFO 2020-08-18 05:26:35,495 New rendezvous state created: {'status': 'joinable', 'version': '3', 'participants': []}
INFO 2020-08-18 05:26:35,563 Joined rendezvous version 3 as rank 0. Full state: {'status': 'joinable', 'version': '3', 'participants': [0]}
INFO 2020-08-18 05:26:35,563 Rank 0 is responsible for join last call.
INFO 2020-08-18 05:26:36,539 Rank 0 finished join last call.
INFO 2020-08-18 05:26:36,539 Waiting for remaining peers.
INFO 2020-08-18 05:26:36,539 All peers arrived. Confirming membership.
INFO 2020-08-18 05:26:36,610 Waiting for confirmations from all peers.
INFO 2020-08-18 05:26:36,635 Rendezvous version 3 is complete. Final state: {'status': 'final', 'version': '3', 'participants': [0, 1], 'keep_alives': ['/torchelastic/p2p/run_mnist12/rdzv/v_3/rank_0', '/torchelastic/p2p/run_mnist12/rdzv/v_3/rank_1'], 'num_workers_waiting': 0}
INFO 2020-08-18 05:26:36,635 Creating EtcdStore as the c10d::Store implementation
[default] Rendezvous complete for workers.
Result:
	restart_count=1
	group_rank=0
	group_world_size=2
	master_addr=mnist12-worker-1
	master_port=38529
	workers={'local_ranks': [0], 'global_ranks': [0], 'role_ranks': [0], 'world_size': 2, 'role_world_size': 2}

[INFO] 2020-08-18 05:26:36,644 api: [default] Rendezvous complete for workers.
Result:
	restart_count=1
	group_rank=0
	group_world_size=2
	master_addr=mnist12-worker-1
	master_port=38529
	workers={'local_ranks': [0], 'global_ranks': [0], 'role_ranks': [0], 'world_size': 2, 'role_world_size': 2}

[default] Starting worker group
[INFO] 2020-08-18 05:26:36,644 api: [default] Starting worker group
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:25: UserWarning: WORLD_SIZE environment variable (2) is not equal to the computed world size (1). Ignored.
  warnings.warn(*args, **kwargs)
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/1
----------------------------------------------------------------------------------------------------
distributed_backend=ddp
All DDP processes registered. Starting ddp with 1 processes
----------------------------------------------------------------------------------------------------
Epoch 1:   2%|▏         | 31/1720 [00:08<08:02,  3.50it/s, loss=6.841, v_num=0]mnist12-worker-1:56:56 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.65.152<0>
mnist12-worker-1:56:56 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).

mnist12-worker-1:56:56 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
mnist12-worker-1:56:56 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.65.152<0>
NCCL version 2.4.8+cuda10.1
mnist12-worker-1:56:64 [0] NCCL INFO Setting affinity for GPU 0 to 0f
mnist12-worker-1:56:64 [0] NCCL INFO Using 256 threads, Min Comp Cap 7, Trees enabled up to size -2
mnist12-worker-1:56:64 [0] NCCL INFO comm 0x7f0e80002280 rank 0 nranks 1 cudaDev 0 nvmlDev 0 - Init COMPLETE

  | Name      | Type             | Params
-----------------------------------------------
0 | model     | ResNet           | 11 M
1 | criterion | CrossEntropyLoss | 0
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:25: UserWarning: The dataloader, val dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  warnings.warn(*args, **kwargs)
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:25: UserWarning: The dataloader, train dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  warnings.warn(*args, **kwargs)
[default] Detected 1 new nodes from group_rank=0; will restart worker group
[INFO] 2020-08-18 05:26:51,666 api: [default] Detected 1 new nodes from group_rank=0; will restart worker group
[default] Stopping worker group
[INFO] 2020-08-18 05:26:51,666 api: [default] Stopping worker group
[default] Rendezvous'ing worker group
[INFO] 2020-08-18 05:26:51,937 api: [default] Rendezvous'ing worker group
INFO 2020-08-18 05:26:51,937 Attempting to join next rendezvous
INFO 2020-08-18 05:26:51,940 Observed existing rendezvous state: {'status': 'final', 'version': '3', 'participants': [0, 1], 'keep_alives': ['/torchelastic/p2p/run_mnist12/rdzv/v_3/rank_0', '/torchelastic/p2p/run_mnist12/rdzv/v_3/rank_1'], 'num_workers_waiting': 1}
INFO 2020-08-18 05:26:51,951 Added self to waiting list. Rendezvous full state: {"status": "final", "version": "3", "participants": [0, 1], "keep_alives": ["/torchelastic/p2p/run_mnist12/rdzv/v_3/rank_0", "/torchelastic/p2p/run_mnist12/rdzv/v_3/rank_1"], "num_workers_waiting": 2}
INFO 2020-08-18 05:26:56,990 Keep-alive key /torchelastic/p2p/run_mnist12/rdzv/v_3/rank_1 is not renewed.
INFO 2020-08-18 05:26:56,990 Rendevous version 3 is incomplete.
INFO 2020-08-18 05:26:56,990 Attempting to destroy it.
INFO 2020-08-18 05:26:56,992 Destroyed rendezvous version 3 successfully.
INFO 2020-08-18 05:26:56,992 Previously existing rendezvous state changed. Will re-try joining.
INFO 2020-08-18 05:26:56,992 Attempting to join next rendezvous
INFO 2020-08-18 05:26:56,996 New rendezvous state created: {'status': 'joinable', 'version': '4', 'participants': []}
INFO 2020-08-18 05:26:57,043 Joined rendezvous version 4 as rank 0. Full state: {'status': 'joinable', 'version': '4', 'participants': [0]}
INFO 2020-08-18 05:26:57,044 Rank 0 is responsible for join last call.
INFO 2020-08-18 05:26:58,015 Rank 0 finished join last call.
INFO 2020-08-18 05:26:58,015 Waiting for remaining peers.
INFO 2020-08-18 05:26:58,016 All peers arrived. Confirming membership.
INFO 2020-08-18 05:26:58,114 Waiting for confirmations from all peers.
INFO 2020-08-18 05:26:58,196 Rendezvous version 4 is complete. Final state: {'status': 'final', 'version': '4', 'participants': [0, 1], 'keep_alives': ['/torchelastic/p2p/run_mnist12/rdzv/v_4/rank_0', '/torchelastic/p2p/run_mnist12/rdzv/v_4/rank_1'], 'num_workers_waiting': 0}
INFO 2020-08-18 05:26:58,196 Creating EtcdStore as the c10d::Store implementation
[default] Rendezvous complete for workers.
Result:
	restart_count=1
	group_rank=0
	group_world_size=2
	master_addr=mnist12-worker-1
	master_port=41561
	workers={'local_ranks': [0], 'global_ranks': [0], 'role_ranks': [0], 'world_size': 2, 'role_world_size': 2}

[INFO] 2020-08-18 05:26:58,205 api: [default] Rendezvous complete for workers.
Result:
	restart_count=1
	group_rank=0
	group_world_size=2
	master_addr=mnist12-worker-1
	master_port=41561
	workers={'local_ranks': [0], 'global_ranks': [0], 'role_ranks': [0], 'world_size': 2, 'role_world_size': 2}

[default] Starting worker group
[INFO] 2020-08-18 05:26:58,205 api: [default] Starting worker group
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:25: UserWarning: WORLD_SIZE environment variable (2) is not equal to the computed world size (1). Ignored.
  warnings.warn(*args, **kwargs)
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/1
----------------------------------------------------------------------------------------------------
distributed_backend=ddp
All DDP processes registered. Starting ddp with 1 processes
----------------------------------------------------------------------------------------------------
Epoch 1:   2%|▏         | 31/1720 [00:08<08:00,  3.52it/s, loss=6.832, v_num=1]mnist12-worker-1:76:76 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.65.152<0>
mnist12-worker-1:76:76 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).

mnist12-worker-1:76:76 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
mnist12-worker-1:76:76 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.65.152<0>
NCCL version 2.4.8+cuda10.1
mnist12-worker-1:76:84 [0] NCCL INFO Setting affinity for GPU 0 to 0f
mnist12-worker-1:76:84 [0] NCCL INFO Using 256 threads, Min Comp Cap 7, Trees enabled up to size -2
mnist12-worker-1:76:84 [0] NCCL INFO comm 0x7faf24002280 rank 0 nranks 1 cudaDev 0 nvmlDev 0 - Init COMPLETE

  | Name      | Type             | Params
-----------------------------------------------
0 | model     | ResNet           | 11 M
1 | criterion | CrossEntropyLoss | 0
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:25: UserWarning: The dataloader, val dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  warnings.warn(*args, **kwargs)
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:25: UserWarning: The dataloader, train dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  warnings.warn(*args, **kwargs)
[default] Detected 1 new nodes from group_rank=0; will restart worker group
[INFO] 2020-08-18 05:27:13,224 api: [default] Detected 1 new nodes from group_rank=0; will restart worker group
[default] Stopping worker group
[INFO] 2020-08-18 05:27:13,224 api: [default] Stopping worker group
[default] Rendezvous'ing worker group
[INFO] 2020-08-18 05:27:13,492 api: [default] Rendezvous'ing worker group
INFO 2020-08-18 05:27:13,492 Attempting to join next rendezvous
INFO 2020-08-18 05:27:13,495 Observed existing rendezvous state: {'status': 'final', 'version': '4', 'participants': [0, 1], 'keep_alives': ['/torchelastic/p2p/run_mnist12/rdzv/v_4/rank_0', '/torchelastic/p2p/run_mnist12/rdzv/v_4/rank_1'], 'num_workers_waiting': 1}
INFO 2020-08-18 05:27:13,536 Added self to waiting list. Rendezvous full state: {"status": "final", "version": "4", "participants": [0, 1], "keep_alives": ["/torchelastic/p2p/run_mnist12/rdzv/v_4/rank_0", "/torchelastic/p2p/run_mnist12/rdzv/v_4/rank_1"], "num_workers_waiting": 2}
INFO 2020-08-18 05:27:18,490 Keep-alive key /torchelastic/p2p/run_mnist12/rdzv/v_4/rank_1 is not renewed.
INFO 2020-08-18 05:27:18,490 Rendevous version 4 is incomplete.
INFO 2020-08-18 05:27:18,490 Attempting to destroy it.
INFO 2020-08-18 05:27:18,491 Destroyed rendezvous version 4 successfully.
INFO 2020-08-18 05:27:18,491 Previously existing rendezvous state changed. Will re-try joining.
INFO 2020-08-18 05:27:18,491 Attempting to join next rendezvous
INFO 2020-08-18 05:27:18,496 New rendezvous state created: {'status': 'joinable', 'version': '5', 'participants': []}
INFO 2020-08-18 05:27:18,546 Joined rendezvous version 5 as rank 0. Full state: {'status': 'joinable', 'version': '5', 'participants': [0]}
INFO 2020-08-18 05:27:18,546 Rank 0 is responsible for join last call.
INFO 2020-08-18 05:27:19,507 Rank 0 finished join last call.
INFO 2020-08-18 05:27:19,507 Waiting for remaining peers.
INFO 2020-08-18 05:27:19,508 All peers arrived. Confirming membership.
INFO 2020-08-18 05:27:19,519 Waiting for confirmations from all peers.
INFO 2020-08-18 05:27:19,605 Rendezvous version 5 is complete. Final state: {'status': 'final', 'version': '5', 'participants': [0, 1], 'keep_alives': ['/torchelastic/p2p/run_mnist12/rdzv/v_5/rank_0', '/torchelastic/p2p/run_mnist12/rdzv/v_5/rank_1'], 'num_workers_waiting': 0}
INFO 2020-08-18 05:27:19,606 Creating EtcdStore as the c10d::Store implementation
[default] Rendezvous complete for workers.
Result:
	restart_count=1
	group_rank=0
	group_world_size=2
	master_addr=mnist12-worker-1
	master_port=33769
	workers={'local_ranks': [0], 'global_ranks': [0], 'role_ranks': [0], 'world_size': 2, 'role_world_size': 2}

[INFO] 2020-08-18 05:27:19,615 api: [default] Rendezvous complete for workers.
Result:
	restart_count=1
	group_rank=0
	group_world_size=2
	master_addr=mnist12-worker-1
	master_port=33769
	workers={'local_ranks': [0], 'global_ranks': [0], 'role_ranks': [0], 'world_size': 2, 'role_world_size': 2}

[default] Starting worker group
[INFO] 2020-08-18 05:27:19,615 api: [default] Starting worker group
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:25: UserWarning: WORLD_SIZE environment variable (2) is not equal to the computed world size (1). Ignored.
  warnings.warn(*args, **kwargs)
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/1
----------------------------------------------------------------------------------------------------
distributed_backend=ddp
All DDP processes registered. Starting ddp with 1 processes
----------------------------------------------------------------------------------------------------
Epoch 1:   2%|▏         | 31/1720 [00:08<08:04,  3.48it/s, loss=6.837, v_num=2]mnist12-worker-1:96:96 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.65.152<0>
mnist12-worker-1:96:96 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).

mnist12-worker-1:96:96 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
mnist12-worker-1:96:96 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.65.152<0>
NCCL version 2.4.8+cuda10.1
mnist12-worker-1:96:104 [0] NCCL INFO Setting affinity for GPU 0 to 0f
mnist12-worker-1:96:104 [0] NCCL INFO Using 256 threads, Min Comp Cap 7, Trees enabled up to size -2
mnist12-worker-1:96:104 [0] NCCL INFO comm 0x7f0970002280 rank 0 nranks 1 cudaDev 0 nvmlDev 0 - Init COMPLETE

  | Name      | Type             | Params
-----------------------------------------------
0 | model     | ResNet           | 11 M
1 | criterion | CrossEntropyLoss | 0
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:25: UserWarning: The dataloader, val dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  warnings.warn(*args, **kwargs)
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:25: UserWarning: The dataloader, train dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  warnings.warn(*args, **kwargs)