Repository: pytorch/pytorch

Issue:
AttributeError: 'NoneType' object has no attribute '_free_weak_ref'
### 🐛 Describe the bug

https://github.com/pytorch/pytorch/pull/66970#issuecomment-1063643284 produces these warnings

```
(/home/ezyang/local/pytorch-tmp-env) [ezyang@devvm066.ash0 ~/local/labs] MASTER_ADDR=localhost MASTER_PORT=29500 python worker0.py  
tensor([-0.6383,  0.2935,  0.6883, -0.9899], requires_grad=True)
Exception ignored in: <function StorageWeakRef.__del__ at 0x7f15538914c0>
Traceback (most recent call last):
  File "/data/users/ezyang/pytorch-tmp/torch/multiprocessing/reductions.py", line 36, in __del__
  File "/data/users/ezyang/pytorch-tmp/torch/storage.py", line 520, in _free_weak_ref
AttributeError: 'NoneType' object has no attribute '_free_weak_ref'
Exception ignored in: <function StorageWeakRef.__del__ at 0x7f15538914c0>
Traceback (most recent call last):
  File "/data/users/ezyang/pytorch-tmp/torch/multiprocessing/reductions.py", line 36, in __del__
  File "/data/users/ezyang/pytorch-tmp/torch/storage.py", line 520, in _free_weak_ref
AttributeError: 'NoneType' object has no attribute '_free_weak_ref'
(/home/ezyang/local/pytorch-tmp-env) [ezyang@devvm066.ash0 ~/local/labs] echo $?
0
```

this shouldn't happen. Investigate.

### Versions

master

cc @ezyang @bhosmer @smessmer @ljk53 @bdhirsh


Discussion:
@ezyang started seeing this in YOLOv5 CI recently:
https://github.com/ultralytics/yolov5/runs/5541120744?check_suite_focus=true

<img width="1788" alt="Screenshot 2022-03-14 at 17 55 30" src="https://user-images.githubusercontent.com/26833433/158222024-4b3b5389-515c-474c-97f8-95c17c054a8b.png">
Sorry I' m a starter. It seems like this warning makes no influence on the training result. (Forget about my bad English : )
well the proximal cause of the problem is `__del__` is being run during process shutdown so the class has already been deleted
During the process I solve another problem, I found that this warning disappear and it didn't raise any error after this operation: You may find 1 libiomp5md.dll in your virtual environment like D:\ANACONDA\envs\yolov5\Lib\site-packages, but another in D:\ANACONDA\envs\yolov5\Library\lib or somewhere in Library. Just move the second one to other place. This could be helpful in some cases like memory error. 
> During the process I solve another problem, I found that this warning disappear and it didn't raise any error after this operation: You may find 1 libiomp5md.dll in your virtual environment like D:\ANACONDA\envs\yolov5\Lib\site-packages, but another in D:\ANACONDA\envs\yolov5\Library\lib or somewhere in Library. Just move the second one to other place. This could be helpful in some cases like memory error.

Sry, I failed. 
same error

Pytorch1.11has same error, howerver you need to degrade pytorch version to 1.10! 
A basic example using the `dataloader` is giving me the error. This happens when using multithreading. When running in the main process, things are OK.

```python
import torch
from torch.utils.data import TensorDataset, DataLoader

print("Torch version:", torch.__version__)

x = torch.arange(12).reshape(6, 2)
ds = TensorDataset(x)
dl = DataLoader(ds, num_workers=8)

for x in dl:
    print(x)
```

Gives output

```
Torch version: 1.11.0

[tensor([[0, 1]])]
[tensor([[2, 3]])]
[tensor([[4, 5]])]
[tensor([[6, 7]])]
[tensor([[8, 9]])]
[tensor([[10, 11]])]
Exception ignored in: <function StorageWeakRef.__del__ at 0x7f8c7695daf0>
Traceback (most recent call last):
  File "anaconda3/lib/python3.9/site-packages/torch/multiprocessing/reductions.py", line 36, in __del__
  File "anaconda3/lib/python3.9/site-packages/torch/storage.py", line 520, in _free_weak_ref
AttributeError: 'NoneType' object has no attribute '_free_weak_ref'
```
I am facing the same issue while training my YOLOV5 model.
`Exception ignored in: <function StorageWeakRef.__del__ at 0x00000248E64DF700>
Traceback (most recent call last):
  File "miniconda3\envs\yo1\lib\site-packages\torch\multiprocessing\reductions.py", line 36, in __del__
  File "miniconda3\envs\yo1\lib\site-packages\torch\storage.py", line 520, in _free_weak_ref
AttributeError: 'NoneType' object has no attribute '_free_weak_ref'`

I also had the same problem while training my model. This problem does not occur if the parameter of workers is set to 0
Same problem
@kurtamohler I think this should be easier to fix once torch/csrc/generic/StorageSharing.cpp becomes not generic. Then we can just refer to the C binding directly and... hopefully that fixes the module unloading ordering issue.
I can look into it after #75459
i am running into the same issue "'NoneType' object has no attribute '_free_weak_ref'" while training yolov5 model on custom data.
Any update on resolution?
hi guys
sorry my english not very well
ı uninstallpytorch 11.3
ı installed 10.0 and error fixed
conda install pytorch==1.10.0 torchvision==0.11.0 torchaudio==0.10.0 cudatoolkit=10.2 -c pytorch
Still present in `torch=1.11.0`
I've solved the error by installing torch 1.10.1 with along cuda 11.1.Following command was executed
(pip install torch==1.10.1+cu111 torchvision==0.11.2+cu111 torchaudio==0.10.1 -f https://download.pytorch.org/whl/torch_stable.html)
ERROR: Could not find a version that satisfies the requirement torchvision==0.11.2+cu111 (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3, 0.8.2, 0.8.2+cpu, 0.8.2+cu101, 0.8.2+cu110, 0.9.0, 0.9.0+cpu, 0.9.0+cu101, 0.9.0+cu111, 0.9.1, 0.9.1+cpu, 0.9.1+cu101, 0.9.1+cu102, 0.9.1+cu111, 0.10.0, 0.10.0+cpu, 0.10.0+cu102, 0.10.0+cu111, 0.10.1, 0.10.1+cpu, 0.10.1+cu102, 0.10.1+cu111, 0.11.0, 0.11.0+cpu, 0.11.0+cu102, 0.11.0+cu113, 0.11.1, 0.11.1+cpu, 0.11.1+cu102, 0.11.1+cu113, 0.11.2, 0.11.2+cpu, 0.11.2+cu102, 0.11.2+cu113, 0.11.3, 0.11.3+cpu, 0.11.3+cu102, 0.11.3+cu113, 0.12.0, 0.12.0+cpu, 0.12.0+cu113, 0.12.0+cu115)
ERROR: No matching distribution found for torchvision==0.11.2+cu111
> I've solved the error by installing torch 1.10.1 with along cuda 11.1.Following command was executed (pip install torch==1.10.1+cu111 torchvision==0.11.2+cu111 torchaudio==0.10.1 -f https://download.pytorch.org/whl/torch_stable.html)

thank you very much!
@glenn-jocher I currently get this problem on the visDrone Dataset. Any new on this or any better solutions? I installed a older torch version and that did not help. Not sure what @iyxv meant with the workers parameter.

Error during training with:

`!python train.py --data VisDrone.yaml --cfg yolov5s.yaml --epochs 20 --batch-size 128 --name visDrone_Model_yolov5s`

```
Traceback (most recent call last):
  File "c:\Users\Tobias\anaconda3\lib\site-packages\torch\multiprocessing\reductions.py", line 36, in __del__
  File "c:\Users\Tobias\anaconda3\lib\site-packages\torch\storage.py", line 520, in _free_weak_ref
AttributeError: 'NoneType' object has no attribute '_free_weak_ref'
```

I also tried to playing around with the epoch/batch count and it still occurred randomly.

Edit: batchsize 16 seems to make it occure less atleast but still happened after 9/10 Epochs :(
I will start working on fixing this now

Edit: Actually, probably going to wait until #78032 is merged
@kurtamohler thank you! This has been a real headache for YOLOv5 users the last couple months.

Resolving patch:
diff --git a/torch/storage.py b/torch/storage.py
--- a/torch/storage.py
+++ b/torch/storage.py
@@ -63,6 +63,8 @@ def is_shared(self) -> bool: ...  # noqa: E704
     @classmethod
     def _new_shared_cuda(cls, *args, **kwargs) -> T: ...  # noqa: E704
     def _shared_incref(self, *args, **kwargs): ...  # noqa: E704
+    @classmethod
+    def _free_weak_ref(cls, *args, **kwargs): ...  # noqa: E704
 
     def __str__(self):
         data_str = ' ' + '\n '.join(str(self[i]) for i in range(self.size()))
@@ -633,7 +635,7 @@ def resize_(self, size):
 
     @classmethod
     def _free_weak_ref(cls, *args, **kwargs):
-        return eval(cls.__module__)._UntypedStorage._free_weak_ref(*args, **kwargs)
+        return _UntypedStorage._free_weak_ref(*args, **kwargs)
 
     def _weak_ref(self, *args, **kwargs):
         return self._storage._weak_ref(*args, **kwargs)


Test patch:
diff --git a/test/distributed/rpc/test_share_memory.py b/test/distributed/rpc/test_share_memory.py
new file mode 100644
--- /dev/null
+++ b/test/distributed/rpc/test_share_memory.py
@@ -0,0 +1,85 @@
+#!/usr/bin/env python3
+# Owner(s): ["oncall: distributed"]
+
+import torch
+import torch.distributed as dist
+
+if not dist.is_available():
+    print("Distributed not available, skipping tests", file=sys.stderr)
+    sys.exit(0)
+
+import copyreg
+import os
+import contextlib
+
+from torch import multiprocessing
+import torch.multiprocessing.reductions as TorchMpReductions
+import torch.distributed.rpc as rpc
+from torch.distributed.rpc.internal import _InternalRPCPickler
+from torch.distributed.rpc.api import _use_rpc_pickler
+from torch.testing._internal.common_utils import TestCase, run_tests
+
+@contextlib.contextmanager
+def fs_sharing():
+    prev_strategy = multiprocessing.get_sharing_strategy()
+    multiprocessing.set_sharing_strategy('file_system')
+    try:
+        yield
+    finally:
+        multiprocessing.set_sharing_strategy(prev_strategy)
+
+class ShareMemoryRPCPickler(_InternalRPCPickler):
+    def __init__(self) -> None:
+        super().__init__()
+        self._dispatch_table
+        # pyre-fixme[4]: Attribute must be annotated.
+        self._dispatch_table = copyreg.dispatch_table.copy()
+
+        for t in torch._storage_classes:
+            self._dispatch_table[t] = TorchMpReductions.reduce_storage
+
+        for t in torch._tensor_classes:
+            self._dispatch_table[t] = TorchMpReductions.reduce_tensor
+        self._dispatch_table[torch.Tensor] = TorchMpReductions.reduce_tensor
+        self._dispatch_table[
+            torch.nn.parameter.Parameter
+        ] = TorchMpReductions.reduce_tensor
+
+def worker_loop(a):
+    rpc.init_rpc('worker1', rank=1, world_size=2)
+    rpc.shutdown()
+
+def worker_fn(m):
+    pass
+
+class TestRPCPickler(TestCase):
+    def setUp(self):
+        super().setUp()
+
+    def test_case(self):
+        os.environ['MASTER_ADDR'] = 'localhost'
+        os.environ['MASTER_PORT'] = '29500'
+
+        with fs_sharing():
+            r = multiprocessing.spawn(worker_loop, join=False)
+
+            try:
+                with _use_rpc_pickler(ShareMemoryRPCPickler()):
+                    rpc.init_rpc(
+                        'worker0',
+                        rank=0,
+                        world_size=2)
+                    m = torch.nn.Linear(1, 2)
+                    m.share_memory()
+                    rref = rpc.remote(
+                        'worker1',
+                        worker_fn,
+                        args=(m,))
+
+                    rref.to_here()
+            finally:
+                rpc.shutdown()
+                r.join()
+
+if __name__ == '__main__':
+    run_tests()
diff --git a/test/run_test.py b/test/run_test.py
--- a/test/run_test.py
+++ b/test/run_test.py
@@ -176,6 +176,7 @@ def skip_test_p(name: str) -> bool:
     "distributed/nn/jit/test_instantiator",
     "distributed/rpc/test_faulty_agent",
     "distributed/rpc/test_tensorpipe_agent",
+    "distributed/rpc/test_share_memory",
     "distributed/rpc/cuda/test_tensorpipe_agent",
     "distributed/pipeline/sync/skip/test_api",
     "distributed/pipeline/sync/skip/test_gpipe",
@@ -227,6 +228,7 @@ def skip_test_p(name: str) -> bool:
     "distributed/nn/jit/test_instantiator",
     "distributed/rpc/test_faulty_agent",
     "distributed/rpc/test_tensorpipe_agent",
+    "distributed/rpc/test_share_memory",
     "distributed/rpc/cuda/test_tensorpipe_agent",
     "distributed/_shard/checkpoint/test_checkpoint"
     "distributed/_shard/checkpoint/test_file_system_checkpoint"
@@ -612,6 +614,7 @@ def test_distributed(test_module, test_directory, options):
     "distributed/test_pg_wrapper": get_run_test_with_subprocess_fn(),
     "distributed/rpc/test_faulty_agent": get_run_test_with_subprocess_fn(),
     "distributed/rpc/test_tensorpipe_agent": get_run_test_with_subprocess_fn(),
+    "distributed/rpc/test_share_memory": get_run_test_with_subprocess_fn(),
     "distributed/rpc/cuda/test_tensorpipe_agent": get_run_test_with_subprocess_fn(),
 }