Issue 1:
🚀 Feature/Enhnacement

During RPC initialization, each participating process initializes a singleton DistAutogradContainer. During shutdown, we cleanup items such as the RRef context and RPC agent, but do not reset this singleton dist autograd container. This prevents RPC from being reinitialized (i.e. a group of processes cannot call rpc.shutdown() ; dist.barrier() ; rpc.init_rpc(...). It will fail with the following error:

/test/distributed/rpc/faulty_agent/rpc_spawn_faulty#binary,link-tree
/torch/distributed/rpc/__init__.py", line 85, in init_rpc
> dist_autograd._init(rank)
> RuntimeError: Container is already initialized! Cannot initialize it twice!

Motivation

I was looking into seeing if we could reuse spawned subprocesses across our RPC tests, which would decrease the amount of time it takes to run the RPC test suite; however, this would require processes to be able to re-init RPC, which is blocked by this error. Also, I was writing a set of similar tests in #38590, and wanted to use this pattern to reduce code duplication, but couldn't (the tests need to use different initializations of RPC since they test the RRef deletion behavior triggered by the shutdown sequence).

Pitch

I don't know if we necessarily need to destroy the container, maybe we could just de-initialize it, and then re-init if RPC is being started back up again.

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @rohan-varma @xush6528 @jjlilley @osalpekar

Issue 2:
OS: Ubuntu 16.04

PyTorch version: source code@master

How you installed PyTorch (conda, pip, source): source

Python version: Python3.6@conda

CUDA/cuDNN version: 9.0

GPU models and configuration: 4x1080Ti

GCC version (if compiling from source): gcc5.4

A script to reproduce the bug. Please try to provide as minimal of a test case as possible.

dataloader(..., numworkers=2)

then iterate over dataset

Error messages and/or stack traces of the bug

Files already downloaded and verified

Files already downloaded and verified

ERROR: Unexpected segmentation fault encountered in worker.

ERROR: Unexpected segmentation fault encountered in worker.

Traceback (most recent call last):

File "./main.py", line 445, in

main()

File "./main.py", line 426, in main

train_TR(dataloader, dataloader_test, net, loss_f, opt.lr, opt.radius, opt.max_inner, opt.epoch)

File "./main.py", line 214, in train_TR

for x, y in data_iter:

File "/home/luinx/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 275, in next

idx, batch = self._get_batch()

File "/home/luinx/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 254, in _get_batch

return self.data_queue.get()

File "/home/luinx/miniconda3/lib/python3.6/multiprocessing/queues.py", line 335, in get

res = self._reader.recv_bytes()

File "/home/luinx/miniconda3/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes

buf = self._recv_bytes(maxlength)

File "/home/luinx/miniconda3/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes

buf = self._recv(4)

File "/home/luinx/miniconda3/lib/python3.6/multiprocessing/connection.py", line 379, in _recv

chunk = read(handle, remaining)

File "/home/luinx/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 175, in handler

_error_if_any_worker_fails()

RuntimeError: DataLoader worker (pid 21010) is killed by signal: Segmentation fault.

Context around what you are trying to do

Nothing special, just loading data. When I set numworkers=0, then it works fine. I guess it is triggered by interprocess communication.