Pytorch dataloaders : OSError: [Errno 9] Bad file descriptor

username: zhaoworking
cooked: <p><strong>Description of the problem</strong></p>
<p>The error will occur if the num_workers &gt; 0 , But when I set num_workers = 0 , the error disappeared, though, this will slow down the trainning speed. I think the multiprocessing really matters here .How can I solve this problem?</p>
<p><strong>env</strong></p>
<p>docker python3.8 Pytorch 1.11.0+cu113</p>
<p><strong>error output</strong></p>
<pre><code>Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/multiprocessing/resource_sharer.py", line 149, in _serve
    send(conn, destination_pid)
  File "/opt/conda/lib/python3.8/multiprocessing/resource_sharer.py", line 50, in send
    reduction.send_handle(conn, new_fd, pid)
  File "/opt/conda/lib/python3.8/multiprocessing/reduction.py", line 184, in send_handle
    sendfds(s, [handle])
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/multiprocessing/reduction.py", line 149, in sendfds
  File "save_disp.py", line 85, in &lt;module&gt;
    sock.sendmsg([msg], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, fds)])
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/multiprocessing/resource_sharer.py", line 151, in _serve
    test()
  File "save_disp.py", line 55, in test
    close()
    for batch_idx, sample in enumerate(TestImgLoader):
  File "/opt/conda/lib/python3.8/multiprocessing/resource_sharer.py", line 52, in close
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 530, in __next__
    os.close(new_fd)
OSError: [Errno 9] Bad file descriptor
    data = self._next_data()
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1207, in _next_data
    idx, data = self._get_data()
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1173, in _get_data
    success, data = self._try_get_data()
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1011, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/opt/conda/lib/python3.8/multiprocessing/queues.py", line 116, in get
    return _ForkingPickler.loads(res)
  File "/opt/conda/lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 295, in rebuild_storage_fd
    fd = df.detach()
  File "/opt/conda/lib/python3.8/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/opt/conda/lib/python3.8/multiprocessing/reduction.py", line 189, in recv_handle
    return recvfds(s, 1)[0]
  File "/opt/conda/lib/python3.8/multiprocessing/reduction.py", line 159, in recvfds
    raise EOFError
EOFError
</code></pre>

username: srishti-git1110
cooked: <p>Hey,<br>
please share the code and also maybe the format of file you are loading the data from.</p>

username: zhaoworking
cooked: <p><strong>Dataloader</strong></p>
<pre><code class="lang-auto">TrainImgLoader = DataLoader(train_dataset, args.batch_size, shuffle=True, num_workers=0, drop_last=True)

TestImgLoader = DataLoader(test_dataset, args.test_batch_size, shuffle=False, num_workers=0, drop_last=False)
</code></pre>
<pre><code class="lang-auto">    def __getitem__(self, index):
        left_img = self.load_image(os.path.join(self.datapath, self.left_filenames[index]))
        right_img = self.load_image(os.path.join(self.datapath, self.right_filenames[index]))
        disparity = self.load_disp(os.path.join(self.datapath, self.disp_filenames[index]))
        roi = self.load_mask(os.path.join(self.datapath, self.mask_filenames[index]))
        try:
            if self.training:
                w, h = left_img.size
                crop_w, crop_h = 512, 256

                x1 = random.randint(0, w - crop_w)
                y1 = random.randint(0, h - crop_h)

                # random crop
                left_img = left_img.crop((x1, y1, x1 + crop_w, y1 + crop_h))
                right_img = right_img.crop((x1, y1, x1 + crop_w, y1 + crop_h))
                disparity = disparity[y1:y1 + crop_h, x1:x1 + crop_w]
                roi = roi[y1:y1 + crop_h,x1:x1 + crop_w]
                # to tensor, normalize
                processed = get_transform()
                left_img = processed(left_img)
                right_img = processed(right_img)

                return {"left": left_img,
                        "right": right_img,
                        "disparity": disparity,
                        "left_filename": self.left_filenames[index],
                        "right_filename": self.right_filenames[index],
                        "roi":roi}
            else:
                w, h = left_img.size
                # crop_w, crop_h = 1024, 1024

                # left_img = left_img.crop((w - crop_w, h - crop_h, w, h))
                # right_img = right_img.crop((w - crop_w, h - crop_h, w, h))
                # disparity = disparity[h - crop_h:h, w - crop_w: w]
                # roi = roi[h - crop_h:h, w - crop_w: w]

                processed = get_transform()
                left_img = processed(left_img)
                right_img = processed(right_img)

                return {"left": left_img,
                        "right": right_img,
                        "disparity": disparity,
                        "top_pad": 0,
                        "right_pad": 0,
                        "left_filename": self.left_filenames[index],
                        "right_filename": self.right_filenames[index],
                        "roi":roi}
        except Exception as e:
            print(e.args)
            print(str(e))
            print(repr(e))
            print('here is get_item error')
</code></pre>
<p><code>File format</code><br>
TIFF image and txt file</p>

username: srishti-git1110
cooked: <p>From what I am able to see, it’s an EOF error and even before that it’s a bad file descriptor error which in this case looks like stemming from an attempt to close a file that wasn’t open in the first place. See this-</p>
<blockquote>
<p>File “/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 530, in <strong>next</strong><br>
os.close(new_fd)<br>
OSError: [Errno 9] Bad file descriptor</p>
</blockquote>
<p><a href="https://pytorch.org/docs/stable/notes/windows.html#multiprocessing-error-without-if-clause-protection" rel="noopener nofollow ugc">this</a> might help. Multiprocessing on windows is error prone due to many reasons like pickling etc. Potentially, check <a href="https://discuss.pytorch.org/t/pytorch-windows-eoferror-ran-out-of-input-when-num-workers-0/25918/19">this</a>.</p>
<p>Not sure though, if these links would solve your error.</p>

username: zhaoworking
cooked: <aside class="quote no-group" data-username="zhaoworking" data-post="3" data-topic="157944">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhaoworking/48/51932_2.png" class="avatar"> zhaoworking:</div>
<blockquote>
<p><code>taLoader(test_dataset, args.test_batch_size, s</code></p>
</blockquote>
</aside>
<p>I am using ubuntu to run my program , and I also do if-clause operation which put my trian() code wrapped into if <strong>name</strong> == ‘<strong>main</strong> ’. I think it is not the source of error.</p>

username: esaracino
cooked: <p>I’m using torch 1.12.1 with pytorch-lightning 1.7.6 and getting the exact same issue, sporadically. I have 96 vCPUs available, and when I allow the dataloader to use them, training takes ~1 hour, but I will get Bad File descriptors (no matter what sharing strategy I set my workers to use (file-system, or file-descriptor)).</p>
<p>Seems to occur with any num_workers &gt; 0, and when num_workers == 0, training times explodes (4+ hours from the 1 with multiprocessing).</p>

username: bo_xu
cooked: <p>Hi, have you solved this problem? I also had this problem.</p>

username: bo_xu
cooked: <p>I have sloved this problem.Adding this configuration to the dataset script works:</p>
<pre><code class="lang-auto">import torch.multiprocessing
torch.multiprocessing.set_sharing_strategy('file_system')
</code></pre>

username: karaspd
cooked: <p>Hi <a class="mention" href="/u/zhaoworking">@zhaoworking</a> , I am getting similar error with similar file extension. I was wondering how you load the image, are you using Pillow? if yes what version of Pillow you used? Also, have you been able to fix the issue? Thanks</p>