[DataLoader] num_of_worker>0 make error

Moonlight: <h1>i use window env.</h1>
<p>code1:<br>
for ii , (X, y, y_weight) in enumerate(dataLoader[phase]): <span class="hashtag">#for</span> each of the batches<br>
X = X.to(device)  # [Nbatch, 3, H, W]<br>
y_weight = y_weight.type(‘torch.FloatTensor’).to(device)<br>
y = y.type(‘torch.LongTensor’).to(device)</p>
<p>error:</p>
<p>File “”, line 1, in <br>
debugfile(‘C:/Users/mbmhm/Desktop/unet/train_unet.py’, wdir=‘C:/Users/mbmhm/Desktop/unet’)</p>
<p>File “C:\Users\mbmhm\ansel\Anaconda3\envs\moongpu\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 856, in debugfile<br>
debugger.run(“runfile(%r, args=%r, wdir=%r)” % (filename, args, wdir))</p>
<p>File “C:\Users\mbmhm\ansel\Anaconda3\envs\moongpu\lib\bdb.py”, line 585, in run<br>
exec(cmd, globals, locals)</p>
<p>File “”, line 1, in </p>
<p>File “C:\Users\mbmhm\ansel\Anaconda3\envs\moongpu\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 827, in runfile<br>
execfile(filename, namespace)</p>
<p>File “C:\Users\mbmhm\ansel\Anaconda3\envs\moongpu\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 110, in execfile<br>
exec(compile(f.read(), filename, ‘exec’), namespace)</p>
<p>File “c:/users/mbmhm/desktop/unet/train_unet.py”, line 267, in <br>
for ii , (X, y, y_weight) in enumerate(dataLoader[phase]): <span class="hashtag">#for</span> each of the batches</p>
<p>File “C:\Users\mbmhm\ansel\Anaconda3\envs\moongpu\lib\site-packages\torch\utils\data\dataloader.py”, line 193, in <strong>iter</strong><br>
return _DataLoaderIter(self)</p>
<p>File “C:\Users\mbmhm\ansel\Anaconda3\envs\moongpu\lib\site-packages\torch\utils\data\dataloader.py”, line 469, in <strong>init</strong><br>
w.start()</p>
<p>File “C:\Users\mbmhm\ansel\Anaconda3\envs\moongpu\lib\multiprocessing\process.py”, line 112, in start<br>
self._popen = self._Popen(self)</p>
<p>File “C:\Users\mbmhm\ansel\Anaconda3\envs\moongpu\lib\multiprocessing\context.py”, line 223, in _Popen<br>
return _default_context.get_context().Process._Popen(process_obj)</p>
<p>File “C:\Users\mbmhm\ansel\Anaconda3\envs\moongpu\lib\multiprocessing\context.py”, line 322, in _Popen<br>
return Popen(process_obj)</p>
<p>File “C:\Users\mbmhm\ansel\Anaconda3\envs\moongpu\lib\multiprocessing\popen_spawn_win32.py”, line 89, in <strong>init</strong><br>
reduction.dump(process_obj, to_child)</p>
<p>File “C:\Users\mbmhm\ansel\Anaconda3\envs\moongpu\lib\multiprocessing\reduction.py”, line 60, in dump<br>
ForkingPickler(file, protocol).dump(obj)</p>
<p>File “stringsource”, line 2, in tables.hdf5extension.Array.<strong>reduce_cython</strong></p>
<h1><strong>TypeError: self.dims,self.dims_chunk,self.maxdims cannot be converted to a Python object for pickling</strong></h1>
<p>code 2:<br>
for x,y,w in dataLoader[‘train’]:<br>
print(x.shape, y.shape, w.shape)</p>
<p>error:<br>
File “”, line 1, in <br>
debugfile(‘C:/Users/mbmhm/Desktop/unet/train_unet.py’, wdir=‘C:/Users/mbmhm/Desktop/unet’)</p>
<p>File “C:\Users\mbmhm\ansel\Anaconda3\envs\moongpu\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 856, in debugfile<br>
debugger.run(“runfile(%r, args=%r, wdir=%r)” % (filename, args, wdir))</p>
<p>File “C:\Users\mbmhm\ansel\Anaconda3\envs\moongpu\lib\bdb.py”, line 585, in run<br>
exec(cmd, globals, locals)</p>
<p>File “”, line 1, in </p>
<p>File “C:\Users\mbmhm\ansel\Anaconda3\envs\moongpu\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 827, in runfile<br>
execfile(filename, namespace)</p>
<p>File “C:\Users\mbmhm\ansel\Anaconda3\envs\moongpu\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 110, in execfile<br>
exec(compile(f.read(), filename, ‘exec’), namespace)</p>
<p>File “c:/users/mbmhm/desktop/unet/train_unet.py”, line 200, in <br>
for w, y, z in dataLoader[‘train’]:</p>
<p>File “C:\Users\mbmhm\ansel\Anaconda3\envs\moongpu\lib\site-packages\torch\utils\data\dataloader.py”, line 576, in <strong>next</strong><br>
idx, batch = self._get_batch()</p>
<p>File “C:\Users\mbmhm\ansel\Anaconda3\envs\moongpu\lib\site-packages\torch\utils\data\dataloader.py”, line 543, in _get_batch<br>
success, data = self._try_get_batch()</p>
<p>File “C:\Users\mbmhm\ansel\Anaconda3\envs\moongpu\lib\site-packages\torch\utils\data\dataloader.py”, line 519, in _try_get_batch<br>
raise RuntimeError(‘DataLoader worker (pid(s) {}) exited unexpectedly’.format(pids_str))</p>
<p><strong>RuntimeError: DataLoader worker (pid(s) 7744, 4584) exited unexpectedly</strong></p>

ptrblck: <p>Is this error related to <a href="https://discuss.pytorch.org/t/dataloader-num-of-worker-0-error/52121">this topic</a>, <a href="https://discuss.pytorch.org/t/dataloader-num-of-worker-0-error-occurs/52250">this one</a> and <a href="https://discuss.pytorch.org/t/datloader-problem-in-unet-num-of-worker-0-error/52112">this</a>?<br>
If so, I would recommend to continue the discussion in one of the mentioned threads to avoid confusion.</p>
<p>Let me know, if these issues are related and where you would like to continue the discussion.</p>

Moonlight: <p>for ii , (X, y, y_weight) in enumerate(dataLoader[phase]): <span class="hashtag">#for</span> each of the batches<br>
X = X.to(device) # [Nbatch, 3, H, W]<br>
y_weight = y_weight.type(‘torch.FloatTensor’).to(device)<br>
y = y.type(‘torch.LongTensor’).to(device)</p>
<p>error:<br>
TypeError: self.dims,self.dims_chunk,self.maxdims cannot be converted to a Python object for pickling</p>
<p>====================================</p>
<p>for x,y,w in dataLoader[‘train’]:<br>
print(x.shape, y.shape, w.shape)</p>
<p>error:<br>
*RuntimeError: DataLoader worker (pid(s) 7744, 4584) exited unexpectedly</p>

ptrblck: <p>Could you post the line of code, which is raising this error:</p>
<pre><code class="lang-python">TypeError: self.dims,self.dims_chunk,self.maxdims cannot be converted to a Python object for pickling
</code></pre>
<p>or alternatively the definition of your <code>Dataset</code>? I assume something is wrong in <code>__getitem__</code>.</p>

Moonlight: <p>it is my code</p>
<p>class Dataset(object):<br>
def <strong>init</strong>(self, fname ,img_transform=None, mask_transform = None, edge_weight= False):<br>
<span class="hashtag">#nothing</span> special here, just internalizing the constructor parameters<br>
self.fname=fname<br>
self.edge_weight = edge_weight</p>
<pre><code>    self.img_transform=img_transform
    self.mask_transform = mask_transform
    
    self.tables=tables.open_file(self.fname)
    self.numpixels=self.tables.root.numpixels[:]
    self.nitems=self.tables.root.img.shape[0]
    self.tables.close()
    
    self.img = None
    self.mask = None
    
def __getitem__(self, index):
    #opening should be done in __init__ but seems to be
    #an issue with multithreading so doing here
    with tables.open_file(self.fname,'r') as db:
        self.img=db.root.img
        self.mask=db.root.mask
   
        #get the requested image and mask from the pytable
        img = self.img[index,:,:,:]
        mask = self.mask[index,:,:]
    
    #the original Unet paper assignes increased weights to the edges of the annotated objects
    #their method is more sophistocated, but this one is faster, we simply dilate the mask and 
    #highlight all the pixels which were "added"
    if(self.edge_weight):
        weight = scipy.ndimage.morphology.binary_dilation(mask==1, iterations =2) &amp; ~mask
    else: #otherwise the edge weight is all ones and thus has no affect
        weight = np.ones(mask.shape,dtype=mask.dtype)
    
    mask = mask[:,:,None].repeat(3,axis=2) #in order to use the transformations given by torchvision
    weight = weight[:,:,None].repeat(3,axis=2) #inputs need to be 3D, so here we convert from 1d to 3d by repetition
    
    img_new = img
    mask_new = mask
    weight_new = weight
    
    seed = random.randrange(sys.maxsize) #get a random seed so that we can reproducibly do the transofrmations
    if self.img_transform is not None:
        random.seed(seed) # apply this seed to img transforms
        img_new = self.img_transform(img)

    if self.mask_transform is not None:
        random.seed(seed)
        mask_new = self.mask_transform(mask)
        mask_new = np.asarray(mask_new)[:,:,0].squeeze()
        
        random.seed(seed)
        weight_new = self.mask_transform(weight)
        weight_new = np.asarray(weight_new)[:,:,0].squeeze()

    return img_new, mask_new, weight_new
def __len__(self):
    return self.nitems</code></pre>