Issue 1:
Flaky occurrence:

https://circle.pytorch.org/build-details.html?build_id=2161720

Jul 09 19:35:19 test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackend) ... Process process 0:
Jul 09 19:35:19 Traceback (most recent call last):
Jul 09 19:35:19 File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
Jul 09 19:35:19 self.run()
Jul 09 19:35:19 File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 93, in run
Jul 09 19:35:19 self._target(*self._args, **self._kwargs)
Jul 09 19:35:19 File "test_distributed.py", line 1661, in _run
Jul 09 19:35:19 getattr(self, self.id().split(".")[2])()
Jul 09 19:35:19 File "test_distributed.py", line 1597, in wrapper
Jul 09 19:35:19 fn(self)
Jul 09 19:35:19 File "test_distributed.py", line 120, in wrapper
Jul 09 19:35:19 return func(*args, **kwargs)
Jul 09 19:35:19 File "test_distributed.py", line 136, in wrapper
Jul 09 19:35:19 return func(*args, **kwargs)
Jul 09 19:35:19 File "test_distributed.py", line 1548, in test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value
Jul 09 19:35:19 model = nn.parallel.DistributedDataParallel(ONLY_SBN_NET.cuda(rank), device_ids=[rank])
Jul 09 19:35:19 File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 298, in __init__
Jul 09 19:35:19 self.broadcast_bucket_size)
Jul 09 19:35:19 File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 480, in _distributed_broadcast_coalesced
Jul 09 19:35:19 dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
Jul 09 19:35:19 RuntimeError: NCCL error in: /var/lib/jenkins/workspace/torch/lib/c10d/ProcessGroupNCCL.cpp:272, unhandled system error
Jul 09 19:35:19 Process process 1:
Jul 09 19:35:19 Traceback (most recent call last):
Jul 09 19:35:19 File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
Jul 09 19:35:19 self.run()
Jul 09 19:35:19 File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 93, in run
Jul 09 19:35:19 self._target(*self._args, **self._kwargs)
Jul 09 19:35:19 File "test_distributed.py", line 1661, in _run
Jul 09 19:35:19 getattr(self, self.id().split(".")[2])()
Jul 09 19:35:19 File "test_distributed.py", line 1597, in wrapper
Jul 09 19:35:19 fn(self)
Jul 09 19:35:19 File "test_distributed.py", line 120, in wrapper
Jul 09 19:35:19 return func(*args, **kwargs)
Jul 09 19:35:19 File "test_distributed.py", line 136, in wrapper
Jul 09 19:35:19 return func(*args, **kwargs)
Jul 09 19:35:19 File "test_distributed.py", line 1548, in test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value
Jul 09 19:35:19 model = nn.parallel.DistributedDataParallel(ONLY_SBN_NET.cuda(rank), device_ids=[rank])
Jul 09 19:35:19 File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 298, in __init__
Jul 09 19:35:19 self.broadcast_bucket_size)
Jul 09 19:35:19 File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 480, in _distributed_broadcast_coalesced
Jul 09 19:35:19 dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
Jul 09 19:35:19 RuntimeError: Connection reset by peer
Jul 09 19:40:19 FAIL

Jul 09 19:40:48 ======================================================================
Jul 09 19:40:48 FAIL: test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackend)
Jul 09 19:40:48 ----------------------------------------------------------------------
Jul 09 19:40:48 Traceback (most recent call last):
Jul 09 19:40:48 File "test_distributed.py", line 1595, in wrapper
Jul 09 19:40:48 self._join_and_reduce(fn)
Jul 09 19:40:48 File "test_distributed.py", line 1677, in _join_and_reduce
Jul 09 19:40:48 "Timeout waiting for rank %d to terminate" % rank)
Jul 09 19:40:48 AssertionError: True is not false : Timeout waiting for rank 2 to terminate
Jul 09 19:40:48
Jul 09 19:40:48 ----------------------------------------------------------------------

cc @mrshenli @unlimblue

Issue 2:
🐛 Bug

TestDistBackend.test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value seems to be flaky

23:43:08 ======================================================================
23:43:08 FAIL: test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackend)
23:43:08 ----------------------------------------------------------------------
23:43:08 Traceback (most recent call last):
23:43:08 File "test_distributed.py", line 1986, in wrapper
23:43:08 self._join_and_reduce(fn)
23:43:08 File "test_distributed.py", line 2079, in _join_and_reduce
23:43:08 "Timeout waiting for rank %d to terminate" % rank)
23:43:08 AssertionError: True is not false : Timeout waiting for rank 1 to terminate

cc @ezyang @gchanan @zou3519 @jerryzh168 @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528