BrokenPipeError: [Errno 32] Broken pipe in Windows even with if __name__ == '__main__':

abodh_ltd: <p>Hello everyone,</p>
<p>I get broken pipeline error when num of workers &gt; 0 in the dataloader. I did look at various solutions (which worked for multiple people) that suggested the code snippet as shown below:</p>
<pre data-code-wrap="python"><code class="lang-python">def run():
     # actual code

if __name__ = '__main__':  
       run()
</code></pre>
<p>However, this does not seem to work for me. I have a PyTorch version of 1.0.0 , python version of 3.6.8 running on Windows 10 and my GPU details are shown below.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9257697cdde783dc94b883de6913a3ed3b0bd53.png" data-download-href="https://discuss.pytorch.org/uploads/default/a9257697cdde783dc94b883de6913a3ed3b0bd53" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9257697cdde783dc94b883de6913a3ed3b0bd53.png" alt="image" data-base62-sha1="o8kXljkECrVS9l2BSFo2ZyvAJQT" width="690" height="391" data-dominant-color="1E1E1E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">887×503 18.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>cuda version:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6ae45cf2d7e7a16fbea73ae285365799a19ff525.png" alt="image" data-base62-sha1="ffBSp9sKPCAi5zzSLBDQb2yjg3z" width="501" height="101"></p>
<p>Can you please help me with this?</p>

peterjc123: <p>Which terminal / IDE are you using? Could you please post a minimal reproducible example?</p>

abodh_ltd: <p>I am using Jupyter notebook to simulate this. Here is the minimal reproducible example. Since my input and ouput are of the shape [1700, 402] and [1700,] I used the similar size here.</p>
<pre><code class="lang-python">import torch
import numpy
from torch.utils.data import Dataset, DataLoader
import pdb

# dataset class
class data(Dataset):
    def __init__(self, x, y):
        self.x = x
        self.y = y
        self.len = self.x.shape[0]

    def __getitem__(self, idx):
        return self.x[idx], self.y[idx]

    def __len__(self):
        return self.len

if __name__ == "__main__": 
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu") 
    x = numpy.random.rand(1700,402)
    y = numpy.random.rand(1700)
    dataset = data(torch.Tensor(x),torch.Tensor(y))
    print('the length of the dataset = ', len(dataset))
    
    print ('before loader')
    data_loader = DataLoader(dataset, batch_size = 50, shuffle = True, pin_memory=True, num_workers = 1)
    print ('after loader')
    
    for idxs, (xx, yy) in enumerate(data_loader):
        xx, yy = xx.to(device), yy.to(device)
        print ('idx: {} x: {} y: {}'.format(idxs, xx, yy))
</code></pre>
<p>This time, the error is not shown however, the notebook gets stuck without displaying anything . It does print ‘after loader’ after which it gets stuck until I restart the kernel. It works perfectly fine if I set num_workers = 0.</p>
<p>Note: the brokern pipe error is still consistent with my original case.</p>

peterjc123: <p>For Jupyter Notebook, you need to refactor your code a bit more. Please refer to this post for solution. <a href="https://medium.com/@grvsinghal/speed-up-your-python-code-using-multiprocessing-on-windows-and-jupyter-or-ipython-2714b49d6fac" rel="nofollow noopener">https://medium.com/@grvsinghal/speed-up-your-python-code-using-multiprocessing-on-windows-and-jupyter-or-ipython-2714b49d6fac</a></p>

abodh_ltd: <p>Thank you for your response. The workaround works for now but I am getting a new error at the line:</p>
<p>p.map(workers.run(used_device), [i for i in range(0,3)])</p>
<pre><code class="lang-python">    642             return self._value
    643         else:
--&gt; 644             raise self._value
    645 
    646     def _set(self, i, obj):

TypeError: 'NoneType' object is not callable
</code></pre>
<p>Here is the code snippet from the python file</p>
<pre><code class="lang-python">import torch
import numpy
from torch.utils.data import Dataset, DataLoader
import pdb

class data(Dataset):
    # Constructor
    def __init__(self, x, y):
        self.x = x
        self.y = y
        self.len = self.x.shape[0]

    # Getter
    def __getitem__(self, idx):
        return self.x[idx], self.y[idx]

    # Return the length
    def __len__(self):
        return self.len

def run(device):
    x = numpy.random.rand(1700,402)
    y = numpy.random.rand(1700)
    dataset = data(torch.Tensor(x),torch.Tensor(y))
    print('the length of the dataset = ', len(dataset))
    data_loader = DataLoader(dataset, batch_size = 100, shuffle = True)
    
    for idxs, (xx, yy) in enumerate(data_loader):
        xx, yy = xx.to(device), yy.to(device)
        print ('idx: {} x: {} y: {}'.format(idxs, xx, yy))
</code></pre>
<p>this is the code snippet from the Jupiter notebook:</p>
<pre><code class="lang-python">from multiprocessing import Pool
import workers
    
if __name__ == "__main__": 
    used_device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    num_processors = 2
    p=Pool(processes = num_processors)
    p.map(workers.run(used_device), [i for i in range(0,3)]) 
</code></pre>
<p>Can you tell me where the error comes from?</p>

peterjc123: <p>No no no, I don’t think you understand the post. The correct thing is to do is to create a new source file <code>worker.py</code> with all your code in it and then in Jupyter Notebook you call <code>worker.run("cuda")</code>. You don’t need to use multiprocessing manually.</p>

abodh_ltd: <p>That’s exactly what I did. I created a new file <code>workers.py</code>. The <code>data</code> class and <code>run</code> function are in a separate file <code>workers.py</code> which is loaded in the jupyter notebook as <code>import workers.</code></p>
<p>I did a mistake using multiprocessing pool here. Thanks for notifying me. Now it works smoothly. Cheers.</p>