Issue 1:
Log:

21:31:08 test_proper_exit (__main__.TestDataLoader)
21:31:08 There might be ConnectionResetError or leaked semaphore warning (due to dirty process exit), but they are all safe to ignore ... Traceback (most recent call last):
21:31:10 File "<string>", line 1, in <module>
21:31:10 File "C:\Jenkins\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
21:31:10 exitcode = _main(fd)
21:31:10 File "C:\Jenkins\Miniconda3\lib\multiprocessing\spawn.py", line 115, in _main
21:31:10 self = reduction.pickle.load(from_parent)
21:31:10 File "C:\Jenkins\Miniconda3\lib\multiprocessing\connection.py", line 942, in rebuild_pipe_connection
21:31:10 handle = dh.detach()
21:31:10 File "C:\Jenkins\Miniconda3\lib\multiprocessing\reduction.py", line 129, in detach
21:31:10 self._access, False, _winapi.DUPLICATE_CLOSE_SOURCE)
21:31:10 PermissionError: [WinError 5] Access is denied
21:31:25 FAIL

21:32:15 ======================================================================
21:32:15 FAIL: test_proper_exit (__main__.TestDataLoader)
21:32:15 There might be ConnectionResetError or leaked semaphore warning (due to dirty process exit), but they are all safe to ignore
21:32:15 ----------------------------------------------------------------------
21:32:15 Traceback (most recent call last):
21:32:15 File "C:\Jenkins\workspace\pytorch-builds\pytorch-win-ws2016-cuda9-cudnn7-py3-test2\test\common_utils.py", line 117, in wrapper
21:32:15 fn(*args, **kwargs)
21:32:15 File "test_dataloader.py", line 726, in test_proper_exit
21:32:15 self.assertTrue(setup_event.is_set(), 'loader process setup timed out')
21:32:15 AssertionError: False is not true : loader process setup timed out
21:32:15
21:32:15 ----------------------------------------------------------------------

Context: https://ci.pytorch.org/jenkins/job/pytorch-builds/job/pytorch-win-ws2016-cuda9-cudnn7-py3-test2/16532/console #14490

Issue 2:
🚀 Feature

When I try to load a plugin library built with 2 cpp file,

torch.ops.load_library("libtorch_plugins.so")

I got the following error:

Only a single TORCH_LIBRARY can be used to register the namespace decode; please put all of your definitions in a single TORCH_LIBRARY block. If you were trying to specify implementations, consider using TORCH_LIBRARY_IMPL (which can be duplicated). Previous registration of TORCH_LIBRARY was registered at /home/cloudhan/workspaces/torch_plugins/src/readjpeg_cpu.cpp:146; latest registration was registered at /home/cloudhan/workspaces/torch_plugins/src/readpng_cpu.cpp:87

Motivation

Currently, the only valid way to build a plugin library is to collect all op registration into one translation unit.

That is

// file foo.cpp

// define op foo

// file bar.cpp

// define op bar

// registration.cpp

#include "foo.h"
#include "bar.h"

// register ops foo and bar

In this way, it is hard to maintain flexible extension op set. Since to add a new op, you need to directly modify registration.cpp

To remove unneeded ops, you also need to modify registration.cpp

If distributed registration is supported, all I need is

// file foo.cpp

// define op foo
// register op foo

// file bar.cpp

// define op bar
// register op bar

Simply control which cpp file to be compiled and linked, I can now control the registration easier.

cc @yf225 @glaringlee @ezyang @bhosmer @smessmer @ljk53