Can't pickle model torch._C._distributed_c10d.ProcessGroupNCCL' object 
Labels: oncall: distributed, module: c10d, module: ddp

### 🐛 Describe the bug

I can't pickle a model. I have a line where i'm effectively doing something like

```
class DataParallelModel(torch.nn.Module):
    def __init__(self, model, group, ):
        super().__init__()
        self.shadow_model = deepcopy(self.model)
        for param in self.shadow.parameters():
            param.detach_()

     ... other stuff
```
Keep getting the following error trace
  File "/opt/conda/lib/python3.8/copy.py", line 172, in deepcopy                                                                                                                                                                                                                                                                         
    y = _reconstruct(x, memo, *rv)                                                                                                                                                                                                                                                                                                       
y[deepcopy(key, memo)] = deepcopy(value, memo)  File "/opt/conda/lib/python3.8/copy.py", line 270, in _reconstruct                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                                                                         
  File "/opt/conda/lib/python3.8/copy.py", line 172, in deepcopy                                                                                                                                                                                                                                                                         
        y = _reconstruct(x, memo, *rv)state = deepcopy(state, memo)                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                                         
  File "/opt/conda/lib/python3.8/copy.py", line 296, in _reconstruct                                                                                                                                                                                                                                                                     
  File "/opt/conda/lib/python3.8/copy.py", line 146, in deepcopy                                                                                                                                                                                                                                                                         
    y = copier(x, memo)                                                                                                                                                                                                                                                                                                                  
  File "/opt/conda/lib/python3.8/copy.py", line 230, in _deepcopy_dict                                                                                                                                                                                                                                                                   
    value = deepcopy(value, memo)                                                                                                                                                                                                                                                                                                        
  File "/opt/conda/lib/python3.8/copy.py", line 172, in deepcopy                                                                                                                                                                                                                                                                         
    y[deepcopy(key, memo)] = deepcopy(value, memo)                                                                                                                                                                                                                                                                                       
  File "/opt/conda/lib/python3.8/copy.py", line 161, in deepcopy                                                                                                                                                                                                                                                                         
    y = _reconstruct(x, memo, *rv)                                                                                                                                                                                                                                                                                                       
  File "/opt/conda/lib/python3.8/copy.py", line 270, in _reconstruct                                                                                                                                                                                                                                                                     
    rv = reductor(4)                                                                                                                                                                                                                                                                                                                     
TypeError: cannot pickle 'torch._C._distributed_c10d.ProcessGroupNCCL' object                                                                                                                                                                                                                                                            
    state = deepcopy(state, memo)                                                                                                                                                                                                                                                                                                        
  File "/opt/conda/lib/python3.8/copy.py", line 146, in deepcopy                                                                                                                                                                                                                                                                         
    y = copier(x, memo)                                                                                                                                                                                                                                                                                                                  
  File "/opt/conda/lib/python3.8/copy.py", line 230, in _deepcopy_dict                                                                                                                                                                                                                                                                   
    y[deepcopy(key, memo)] = deepcopy(value, memo)                                                                                                                                                                                                                                                                                       
  File "/opt/conda/lib/python3.8/copy.py", line 172, in deepcopy                                                                                                                                                                                                                                                                         
    y = _reconstruct(x, memo, *rv)                                                                                                                                                                                                                                                                                                       
  File "/opt/conda/lib/python3.8/copy.py", line 296, in _reconstruct                                                                                                                                                                                                                                                                     
    value = deepcopy(value, memo)                                                                                                                                                                                                                                                                                                        
  File "/opt/conda/lib/python3.8/copy.py", line 172, in deepcopy                                                                                                                                                                                                                                                                         
    y = _reconstruct(x, memo, *rv)                                                                                                                                                                                                                                                                                                       
  File "/opt/conda/lib/python3.8/copy.py", line 270, in _reconstruct                                                                                                                                                                                                                                                                     
    state = deepcopy(state, memo)                                                                                                                                                                                                                                                                                                        
  File "/opt/conda/lib/python3.8/copy.py", line 146, in deepcopy                                                                                                                                                                                                                                                                         
    y = copier(x, memo)                                                                                                                                                                                                                                                                                                                  
  File "/opt/conda/lib/python3.8/copy.py", line 230, in _deepcopy_dict                                                                                                                                                                                                                                                                   
    y[deepcopy(key, memo)] = deepcopy(value, memo)                                                                                                                                                                                                                                                                                       
  File "/opt/conda/lib/python3.8/copy.py", line 161, in deepcopy                                                                                                                                                                                                                                                                         
    rv = reductor(4)                                                                                                                                                                                                                                                                                                                     
TypeError: cannot pickle 'torch._C._distributed_c10d.ProcessGroupNCCL' object  




### Versions

PyTorch version: 1.12.0a0+bd13bc6
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.22.3
Libc version: glibc-2.31

Python version: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10)  [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-5.4.0-107-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.6.124
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-80GB
GPU 1: NVIDIA A100-SXM4-80GB
GPU 2: NVIDIA A100-SXM4-80GB
GPU 3: NVIDIA A100-SXM4-80GB
GPU 4: NVIDIA A100-SXM4-80GB
GPU 5: NVIDIA A100-SXM4-80GB
GPU 6: NVIDIA A100-SXM4-80GB
GPU 7: NVIDIA A100-SXM4-80GB

Nvidia driver version: 470.103.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] pytorch-quantization==2.1.2
[pip3] torch==1.12.0a0+bd13bc6
[pip3] torch-tensorrt==1.1.0a0
[pip3] torchtext==0.13.0a0
[pip3] torchvision==0.13.0a0
[conda] magma-cuda110             2.5.2                         5    local
[conda] mkl                       2019.5                      281    conda-forge
[conda] mkl-include               2019.5                      281    conda-forge
[conda] numpy                     1.22.3           py38h1d589f8_2    conda-forge
[conda] pytorch-quantization      2.1.2                    pypi_0    pypi
[conda] torch                     1.12.0a0+bd13bc6          pypi_0    pypi
[conda] torch-tensorrt            1.1.0a0                  pypi_0    pypi
[conda] torchtext                 0.13.0a0                 pypi_0    pypi
[conda] torchvision               0.13.0a0                 pypi_0    pypi

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang