"No space left on device" with DistributedDataParallel despite space left

dstrohmaier: <p>Python=3.11.4 | packaged by conda-forge<br>
pytorch=2.0.1<br>
pytorch-cuda=11.7</p>
<p>I use DistributedDataParallel for training and testing a model and getting the following error</p>
<pre><code class="lang-auto">Traceback (most recent call last):
File "/local/scratch/ds858/nlsa/lib/python3.11/multiprocessing/queues.py", line 244, in _feed
   obj = _ForkingPickler.dumps(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/local/scratch/ds858/nlsa/lib/python3.11/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "/local/scratch/ds858/nlsa/lib/python3.11/site-packages/torch/multiprocessing/reductions.py", line 358, in reduce_storage
    metadata = storage._share_filename_cpu_()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
</code></pre>
<p>A similar error has been reported here: <a href="https://discuss.pytorch.org/t/unable-to-write-to-file-torch-18692-1954506624/9990" class="inline-onebox">Unable to write to file &lt;/torch_18692_1954506624&gt;</a><br>
I am, however, not using Docker or anything like it. The shared memory should also be more than sufficient. It has been checked repeatedly.<br>
The num_workers of the dataloader should also be 0, as it is left at its default value. (There are, however, multiple processes running on the GPUs at the same time.)</p>
<p>Having a brief glance at reductions.py, I found that the line above at which it had a problem is specific to the file_system sharing strategy. As a test, I switched to the file_descriptor strategy. This led to another error, which you can find below.</p>
<pre><code class="lang-auto">File "/local/scratch/ds858/nlsa/lib/python3.11/multiprocessing/queues.py", line 122, in get
  return _ForkingPickler.loads(res)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/local/scratch/ds858/nlsa/lib/python3.11/site-packages/torch/multiprocessing/reductions.py", line 307, in rebuild_storage_fd
   fd = df.detach()
        ^^^^^^^^^^^
 File "/local/scratch/ds858/nlsa/lib/python3.11/multiprocessing/resource_sharer.py", line 57, in detach
   with _resource_sharer.get_connection(self._id) as conn:
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/local/scratch/ds858/nlsa/lib/python3.11/multiprocessing/resource_sharer.py", line 86, in get_connection
   c = Client(address, authkey=process.current_process().authkey)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/local/scratch/ds858/nlsa/lib/python3.11/multiprocessing/connection.py", line 507, in Client
   answer_challenge(c, authkey)
 File "/local/scratch/ds858/nlsa/lib/python3.11/multiprocessing/connection.py", line 756, in answer_challenge
   response = connection.recv_bytes(256)        # reject large message
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/local/scratch/ds858/nlsa/lib/python3.11/multiprocessing/connection.py", line 215, in recv_bytes
   buf = self._recv_bytes(maxlength)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/local/scratch/ds858/nlsa/lib/python3.11/multiprocessing/connection.py", line 413, in _recv_bytes
   buf = self._recv(4)
         ^^^^^^^^^^^^^
 File "/local/scratch/ds858/nlsa/lib/python3.11/multiprocessing/connection.py", line 382, in _recv
   raise EOFError
OFError
</code></pre>
<p>I am grateful for any help or advice for addressing this issue.</p>

kumpera: <p>Can you share the full error message and a repro script.</p>
<p>From your description it might be that your system has too little shm setup. You can check the value of <code>kernel.shmmax</code> by using <code>/sbin/sysctl -a</code></p>

dstrohmaier: <p>I cannot share the whole code at the moment. Generally, though, I am using huggingface BERT models. The error occurred when I was creating embeddings using the model and passing them back to the main process using a queue. I am not even training the model at this point. It is set to eval() and I am using a no_grad() context. The error also has been intermittent.</p>
<p>We have checked the value of kernel.shmmax and it should far exceed what my process could possibly ever use. Speficially, it is a kernel.shmmax value of 18,446,744,073,692,774,399 or 0xFFFF FFFF FEFF FFFF = 2^64 - 2^24 - 1, which we assume is the default value for a 64bit machine.</p>
<p>I have not seen the same error on another machine I am using, however, so it might be some other bit of machine-specific configuration.</p>