Repository: open-mmlab/mmpretrain

Issue:
RuntimeError: Pin memory thread exited unexpectedly
2021-10-04 15:14:24,792 - mmcls - INFO - workflow: [('train', 1)], max: 200 epochs
2021-10-04 15:14:33,806 - mmcls - INFO - Epoch [1][100/391]	lr: 1.000e-01, eta: 1:57:16, time: 0.090, data_time: 0.022, memory: 76, loss: 2.0658
2021-10-04 15:14:39,876 - mmcls - INFO - Epoch [1][200/391]	lr: 1.000e-01, eta: 1:38:01, time: 0.061, data_time: 0.000, memory: 76, loss: 1.8056
2021-10-04 15:14:45,923 - mmcls - INFO - Epoch [1][300/391]	lr: 1.000e-01, eta: 1:31:25, time: 0.060, data_time: 0.000, memory: 76, loss: 1.6816
2021-10-04 15:14:51,428 - mmcls - INFO - Saving checkpoint at 1 epochs
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 10000/10000, 7696.3 task/s, elapsed: 1s, ETA:     0s2021-10-04 15:14:52,802 - mmcls - INFO - Epoch(val) [1][79]	accuracy_top-1: 32.6600, accuracy_top-5: 84.0200
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/home/songyuan/anaconda3/envs/pytorch1.7.1/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/songyuan/anaconda3/envs/pytorch1.7.1/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/songyuan/anaconda3/envs/pytorch1.7.1/lib/python3.7/site-packages/torch/utils/data/_utils/pin_memory.py", line 28, in _pin_memory_loop
    idx, data = r
ValueError: not enough values to unpack (expected 2, got 0)

Traceback (most recent call last):
  File "tools/train.py", line 157, in <module>
    main()
  File "tools/train.py", line 153, in main
    meta=meta)
  File "/home/songyuan/yty/mmclassification-0.15.0/mmcls/apis/train.py", line 160, in train_model
    runner.run(data_loaders, cfg.workflow)
  File "/home/songyuan/anaconda3/envs/pytorch1.7.1/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 127, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/songyuan/anaconda3/envs/pytorch1.7.1/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 47, in train
    for i, data_batch in enumerate(self.data_loader):
  File "/home/songyuan/anaconda3/envs/pytorch1.7.1/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 349, in __iter__
    self._iterator._reset(self)
  File "/home/songyuan/anaconda3/envs/pytorch1.7.1/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 852, in _reset
    data = self._get_data()
  File "/home/songyuan/anaconda3/envs/pytorch1.7.1/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1029, in _get_data
    raise RuntimeError('Pin memory thread exited unexpectedly')
RuntimeError: Pin memory thread exited unexpectedly


我用mmclassification-0.15.0在3090上跑的，pytorch版本为1.7，mmcv为1.3.14，然后训了一个epoch并完成验证后，就出现了这个问题。请问这是什么原因?


Discussion:
refer to https://github.com/open-mmlab/mmclassification/issues/392

Resolving patch:
diff --git a/mmcls/datasets/builder.py b/mmcls/datasets/builder.py
--- a/mmcls/datasets/builder.py
+++ b/mmcls/datasets/builder.py
@@ -1,14 +1,13 @@
 # Copyright (c) OpenMMLab. All rights reserved.
 import platform
 import random
-from distutils.version import LooseVersion
 from functools import partial
 
 import numpy as np
 import torch
 from mmcv.parallel import collate
 from mmcv.runner import get_dist_info
-from mmcv.utils import Registry, build_from_cfg
+from mmcv.utils import Registry, build_from_cfg, digit_version
 from torch.utils.data import DataLoader
 
 from .samplers import DistributedSampler
@@ -98,7 +97,7 @@ def build_dataloader(dataset,
         worker_init_fn, num_workers=num_workers, rank=rank,
         seed=seed) if seed is not None else None
 
-    if LooseVersion(torch.__version__) >= LooseVersion('1.7.0'):
+    if digit_version(torch.__version__) >= digit_version('1.8.0'):
         kwargs['persistent_workers'] = persistent_workers
 
     data_loader = DataLoader(


Test patch: