BrokenPipeError: [Errno 32] Broken pipe When i :"run cifar10_tutorial.py"

username: keloli
cooked: <p>Hi Soumith,</p>
<p>I download “cifar10_tutorial.py"from <a href="http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Training a Classifier — PyTorch Tutorials 2.1.1+cu121 documentation</a>.<br>
When I run this file in Spyder on Windows10, it reports"BrokenPipeError: [Errno 32] Broken pipe”, the details information are as follows:</p>
<blockquote>
<p>runfile(‘D:/PyTorch/Code/LearnPyTorch/cifar10_tutorial.py’, wdir=‘D:/PyTorch/Code/LearnPyTorch’)<br>
Files already downloaded and verified<br>
Files already downloaded and verified<br>
Traceback (most recent call last):<br>
File “”, line 1, in<br>
runfile(‘D:/PyTorch/Code/LearnPyTorch/cifar10_tutorial.py’, wdir=‘D:/PyTorch/Code/LearnPyTorch’)<br>
File “D:\Anaconda\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.p y”, line 880, in runfile<br>
execfile(filename, namespace)<br>
File “D:\Anaconda\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.p y”, line 102, in execfile<br>
exec(compile(f.read(), filename, ‘exec’), namespace)<br>
File “D:/PyTorch/Code/LearnPyTorch/cifar10_tutorial.py”, line 99, in<br>
dataiter = iter(trainloader)<br>
File “D:\Anaconda\Anaconda3\lib\site-packages\torch\utils\data\dataloader.p y”, line 303, in iter<br>
return DataLoaderIter(self)<br>
File “D:\Anaconda\Anaconda3\lib\site-packages\torch\utils\data\dataloader.p y”, line 162, in init<br>
w.start()<br>
File “D:\Anaconda\Anaconda3\lib\multiprocessing\process.p y”, line 105, in start<br>
self._popen = self._Popen(self)<br>
File “D:\Anaconda\Anaconda3\lib\multiprocessing\context.p y”, line 223, in _Popen<br>
return _default_context.get_context().Process._Popen(process_obj)<br>
File “D:\Anaconda\Anaconda3\lib\multiprocessing\context.p y”, line 322, in _Popen<br>
return Popen(process_obj)<br>
File “D:\Anaconda\Anaconda3\lib\multiprocessing\popen_spawn_win32.p y”, line 65, in init<br>
reduction.dump(process_obj, to_child)<br>
File “D:\Anaconda\Anaconda3\lib\multiprocessing\reduction.p y”, line 60, in dump<br>
ForkingPickler(file, protocol).dump(obj)<br>
BrokenPipeError: [Errno 32] Broken pipe</p>
</blockquote>
<p>Could you give me any advise to help me run “cifar10_tutorial.py” successfully? By the way, other examples in “Deep Learning with PyTorch: A 60 Minute Blitz” have been run suceessfully.</p>

username: Julia_mdr
cooked: <p>Found a solution that works for me on: <a href="https://medium.com/@mackie__m/running-a-cifar-10-image-classifier-on-windows-with-pytorch-9094e29089cd" rel="nofollow noopener">https://medium.com/@mackie__m/running-a-cifar-10-image-classifier-on-windows-with-pytorch-9094e29089cd</a></p>

username: karmus89
cooked: <p>Hi <a class="mention" href="/u/keloli">@keloli</a>!</p>
<p>The problem has to do with multiprocessing, <code>DataLoader</code> class and Windows broadly, but I’m not familiar with the details. What helped me was to set the  <code>num_workers</code> parameter to either <code>0</code> or <code>1</code> with the data loaders.</p>
<p><strong>Original (not working):</strong></p>
<pre><code class="lang-auto"># ... code ...
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)
# ... code ...
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)
# ... code ...
</code></pre>
<p><strong>Modified (working):</strong></p>
<pre><code class="lang-auto"># ... code ...
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=0)
# ... code ...
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=0)
# ... code ...
</code></pre>

username: karmus89
cooked: <p>However, the real way around this problem lies in re-factoring your code to comply to Python’s Windows-specific multiprocessing guidelines as discussed here in <a href="https://stackoverflow.com/a/18205006" rel="noopener nofollow ugc">this</a> StackOverflow thread.</p>
<p>This subject is touched upon in Python 2 documentation for <a href="https://docs.python.org/2/library/multiprocessing.html#windows" rel="noopener nofollow ugc"><code>multiprocessing: Programming Guidelines, Windows</code></a>. While Python 3 documentation shares similar guidelines (see <a href="https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods" rel="noopener nofollow ugc">here</a>), the Python 2 is more explicit with Windows:</p>
<blockquote>
<p>Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process).</p>
</blockquote>
<p>In short, the the idea here would be to wrap the example code inside an <code>if __name__ == '__main__'</code> statement as follows:</p>
<pre><code class="lang-auto"># Deep Learning with PyTorch: A 60 Minute Blitz » Training a classifier 
# Load the CIFAR10 data
import torch
import torchvision
import torchvision.transforms as transforms

# Safe DataLoader multiprocessing with Windows
if __name__ == '__main__':
    # Code to load the data with num_workers &gt; 1
</code></pre>
<p>While the tutorial seems to define multiple scripts, the best way around this would be to wrap all operations in functions and then call them inside an <code>if __name__ == '__main__'</code> clause:</p>
<pre><code class="lang-auto"># Imports for dataset generation, training, etc

def load_datasets(...):
    # Code to load the datasets with multiple workers

def train(...):
    # Code to train the model

if __name__ == '__main__':
    load_datasets()
    train()
</code></pre>

username: keloli
cooked: <p>Hi <a class="mention" href="/u/karmus89">@karmus89</a>!<br>
I’m so appreciative for your solution,i will try this later!</p>

username: wangyujie
cooked: <p>I meet the same problems,your solution is so good.<img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=5" title=":+1:" class="emoji" alt=":+1:"></p>

username: ConeyYan
cooked: <p>Yes, this can solve the problem and thank you very much!</p>

username: changchao_xi
cooked: <p>It’s work, thank you!</p>

username: Shariq_Ali
cooked: <p>thank you so much friend. i really stuck on that problem and cant find any solution. but this work. thank you so much…</p>

username: yajlan2015
cooked: <p>Thanks for your solve karmus89!</p>

username: resilere
cooked: <p>Oh thank you it solved my problem too.</p>

username: Tianyu_Ding
cooked: <p><a class="mention" href="/u/karmus89">@karmus89</a> reply to set number of workers to 0 is to not using multiprocessing while <a class="mention" href="/u/julia_mdr">@Julia_mdr</a> 's answer can help using multiprocess in win by just adding if <strong>name</strong>=“<strong>main</strong>” to the code. Correct me if I am wrong.</p>

username: FarisHijazi
cooked: <p>I’m experiencing this problem in windows 10 but in Jupyter notebook, does anyone have a solution for this?</p>