Issue 1:
🐛 Bug

BrokenPipeError on Windows when setting the num_workers as 4 in DataLoader but 2 num_workers is fine.

To Reproduce

Steps to reproduce the behavior:

I have submitted this issue in #31465, and the contributor commented as

See https://pytorch.org/docs/stable/notes/windows.html#multiprocessing-error-without-if-clause-protection.

I opened the link and found the relevant content Multiprocessing error “Broken pipe” as

Multiprocessing error “Broken pipe”

ForkingPickler(file, protocol).dump(obj)

BrokenPipeError: [Errno 32] Broken pipe

This issue happens when the child process ends before the parent process finishes sending data. There may be something wrong with your code. You can debug your code by reducing the num_worker of DataLoader to zero and see if the issue persists.

But my code ran normally if I set the num_workers as 2.

However, the 4 num_workers would still cause the BrokenPipeError.

Expected behavior

I want to set the num_workers as 4 instead of 2.

Environment

Please copy and paste the output from our

environment collection script

(or fill out the checklist below manually).

You can get the script and run it with:

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

PyTorch Version (e.g., 1.0): 1.3.1

OS (e.g., Linux): Win10

How you installed PyTorch (conda, pip, source): conda

Build command you used (if compiling from source): by PyCharm

Python version: 3.7.3

CUDA/cuDNN version: CUDA 10.1

GPU models and configuration: using CPU

Any other relevant information:

Additional context

Issue 2:
🐛 Bug

BrokenPipeError on Windows when setting the num_workers as 4 in DataLoader

To Reproduce

Steps to reproduce the behavior:

I used code like this,

testloader = torch.utils.data.DataLoader(testset, batch_size=4,
shuffle=False, num_workers=4)

Here are the error messages:

Traceback (most recent call last):

File "", line 1, in

File "E:\Path\Python\Anaconda3\envs\env_general\lib\multiprocessing\spawn.py", line 105, in spawn_main

exitcode = _main(fd)

File "E:\Path\Python\Anaconda3\envs\env_general\lib\multiprocessing\spawn.py", line 114, in _main

prepare(preparation_data)

File "E:\Path\Python\Anaconda3\envs\env_general\lib\multiprocessing\spawn.py", line 225, in prepare

_fixup_main_from_path(data['init_main_from_path'])

File "E:\Path\Python\Anaconda3\envs\env_general\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path

run_name="mp_main")

File "E:\Path\Python\Anaconda3\envs\env_general\lib\runpy.py", line 263, in run_path

pkg_name=pkg_name, script_name=fname)

File "E:\Path\Python\Anaconda3\envs\env_general\lib\runpy.py", line 96, in _run_module_code

mod_name, mod_spec, pkg_name, script_name)

File "E:\Path\Python\Anaconda3\envs\env_general\lib\runpy.py", line 85, in run_code

exec(code, run_globals)

File "D:\Programs\Kaggle\Digit Recognizer\digit-recognizer\main_predict.py", line 7, in

import torch

File "E:\Path\Python\Anaconda3\envs\env_general\lib\site-packages\torch_init.py", line 81, in

from torch._C import *

ImportError: DLL load failed: 页面文件太小，无法完成操作。

Traceback (most recent call last):

File "D:/Programs/Kaggle/Digit Recognizer/digit-recognizer/main_predict.py", line 62, in

main()

File "D:/Programs/Kaggle/Digit Recognizer/digit-recognizer/main_predict.py", line 47, in main

for data in testloader:

File "E:\Path\Python\Anaconda3\envs\env_general\lib\site-packages\torch\utils\data\dataloader.py", line 278, in iter

return _MultiProcessingDataLoaderIter(self)

File "E:\Path\Python\Anaconda3\envs\env_general\lib\site-packages\torch\utils\data\dataloader.py", line 682, in init

w.start()

File "E:\Path\Python\Anaconda3\envs\env_general\lib\multiprocessing\process.py", line 112, in start

self._popen = self._Popen(self)

File "E:\Path\Python\Anaconda3\envs\env_general\lib\multiprocessing\context.py", line 223, in _Popen

return _default_context.get_context().Process._Popen(process_obj)

File "E:\Path\Python\Anaconda3\envs\env_general\lib\multiprocessing\context.py", line 322, in _Popen

return Popen(process_obj)

File "E:\Path\Python\Anaconda3\envs\env_general\lib\multiprocessing\popen_spawn_win32.py", line 89, in init

reduction.dump(process_obj, to_child)

File "E:\Path\Python\Anaconda3\envs\env_general\lib\multiprocessing\reduction.py", line 60, in dump

ForkingPickler(file, protocol).dump(obj)

BrokenPipeError: [Errno 32] Broken pipe

Process finished with exit code 1

And here is the code sample:

source = "../data/digit-recognizer/test.csv"
transform = transforms.Compose(
[transforms.ToTensor(),
transforms.Normalize((0.5,), (0.5,))])
# 其中transforms.ToTensor()的作用为将维度顺序为H*W*C的图像张量装换为适合于PyTorch的C*H*W的张量
testset = MNISTTestDataset(csv_file=source,
transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
shuffle=False, num_workers=4)
# 如果有问题的话，就把num_workers设成0

recognizer = DigitRecognizer()
index = 0

for data in testloader:
index = index + 1
if index > 6:
break

Expected behavior

I want setting the num_workers as 4 in DataLoader.

Environment

Please copy and paste the output from our

environment collection script

(or fill out the checklist below manually).

You can get the script and run it with:

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

PyTorch Version (e.g., 1.0): 1.3.1

OS (e.g., Linux): Win10

How you installed PyTorch (conda, pip, source): conda

Build command you used (if compiling from source): by PyCharm

Python version: 3.7.3

CUDA/cuDNN version: CUDA 10.1

GPU models and configuration: using CPU

Any other relevant information:

Additional context

cc @ssnl @peterjc123