Num_workers in dataloader always gives this error

username: jwillette
cooked: <p>Whenever I try to use num_workers with a dataloader I always get this error</p>
<p><code>OSError: [Errno 16] Device or resource busy: '.nfs00000000087c7062000001de'</code></p>
<p>I tried googling for it but nothing came up which is always scary. Below is the full stacktrace.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/multiprocessing/util.py", line 277, in _run_finalizers                          
    finalizer()
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/multiprocessing/util.py", line 201, in __call__                                 
    res = self._callback(*self._args, **self._kwargs)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/multiprocessing/util.py", line 277, in _run_finalizers                          
    finalizer()
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/multiprocessing/util.py", line 110, in _remove_temp_dir                         
    rmtree(tempdir)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/multiprocessing/util.py", line 201, in __call__                                 
    res = self._callback(*self._args, **self._kwargs)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/shutil.py", line 494, in rmtree                                                 
    _rmtree_safe_fd(fd, path, onerror)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/multiprocessing/util.py", line 110, in _remove_temp_dir                         
    rmtree(tempdir)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/shutil.py", line 452, in _rmtree_safe_fd                                        
    onerror(os.unlink, fullname, sys.exc_info())
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/shutil.py", line 494, in rmtree                                                 
    _rmtree_safe_fd(fd, path, onerror)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/shutil.py", line 450, in _rmtree_safe_fd                                        
    os.unlink(entry.name, dir_fd=topfd)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/shutil.py", line 452, in _rmtree_safe_fd                                        
    onerror(os.unlink, fullname, sys.exc_info())
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/shutil.py", line 450, in _rmtree_safe_fd                                        
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '.nfs00000000087c7066000001db'                                                         
OSError: [Errno 16] Device or resource busy: '.nfs00000000087c7064000001dc'                                                         
Traceback (most recent call last):
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/multiprocessing/util.py", line 277, in _run_finalizers                          
    finalizer()
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/multiprocessing/util.py", line 201, in __call__                                 
    res = self._callback(*self._args, **self._kwargs)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/multiprocessing/util.py", line 110, in _remove_temp_dir                         
    rmtree(tempdir)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/shutil.py", line 494, in rmtree                                                 
    _rmtree_safe_fd(fd, path, onerror)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/shutil.py", line 452, in _rmtree_safe_fd                                        
    onerror(os.unlink, fullname, sys.exc_info())
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/shutil.py", line 450, in _rmtree_safe_fd                                        
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '.nfs00000000087c7055000001dd'                                                         
Traceback (most recent call last):
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/multiprocessing/util.py", line 277, in _run_finalizers                          
    finalizer()
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/multiprocessing/util.py", line 201, in __call__                                 
    res = self._callback(*self._args, **self._kwargs)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/multiprocessing/util.py", line 110, in _remove_temp_dir                         
    rmtree(tempdir)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/shutil.py", line 494, in rmtree                                                 
    _rmtree_safe_fd(fd, path, onerror)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/shutil.py", line 452, in _rmtree_safe_fd                                        
    onerror(os.unlink, fullname, sys.exc_info())
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/shutil.py", line 450, in _rmtree_safe_fd                                        
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '.nfs00000000087c7062000001de'    
</code></pre>

username: ptrblck
cooked: <p>Are you using a network file system? If so, could you try to use the local drive?<br>
Based on the error message, my best guess is that your NFS has too many open files, which conflicts with the creation of shared memory.</p>

username: jwillette
cooked: <p>9 months later and I finally figured out what this was. In one of my environments there was some issue with the <code>/home</code> folder drive being full and I got tired of getting errors so I moved the <code>tmp</code> folder to another drive that was a nfs filesystem and caused this error.</p>

username: jxmorris12
cooked: <p>I fixed it with <code>unset TMPDIR</code>.</p>

username: LalchandPandia
cooked: <p>Hi,<br>
I recently encountered the same issue. I removed the num_worker option. Can you elaborate the issue?<br>
For me the following code caused the error:</p>
<pre><code class="lang-auto">train_loader = DataLoader(train_dataset, shuffle=True, pin_memory=True, batch_size=16, num_workers=1)
for epoch in range(3):
	pbar = tqdm(train_loader)
	for model_inputs, labels in pbar:
		model_inputs = model_inputs.to(device)
</code></pre>
<p>Is there a locking mechanism in place that causes the issue?</p>

username: Saurabh_Kataria
cooked: <p>getting this issue continuously now</p>

username: JuliaWasala
cooked: <p>I also have this issue.<br>
But my code still runs 3x faster with num workers &gt;0.<br>
so I’m wondering what this error means and whether it’s affecting my results or not? Or is it just temporarily blocking the workers until resource is released?</p>