Error when running tensor.to(device) in dataset.__getitem__()

richmengsix: <p>Hello,</p>
<p>I am currently working on a 3D video processing program, and I have encountered some performance issues while using the PyTorch dataset and data loader. After conducting some benchmarking, I found that a significant amount of time is spent waiting on <code>img.to(device)</code> in the main loop. To speed up the processing, I experimented with offloading the <code>img.to(device)</code> work onto <code>dataset.__get_item__</code>.</p>
<p>This approach seemed to work well, as I observed a nice speed improvement. However, I frequently encountered the <code>CUDA error: mapping of buffer object failed</code> error during processing, which I have been unable to debug. Here’s a code snippet of what I was trying to do:</p>
<pre><code class="lang-auto">class FrameDepthFloDataset(Dataset):
    def __init__(self, src_in_files, depth_dir, device):
        self.img_files = sorted(src_in_files)
        self.depth_files = sorted(list(glob.iglob(depth_dir+'/*')))
        self.device = device


    def __getitem__(self, index):
        img_file = self.img_files[index]
        depth_file = self.depth_files[index]
        depth = load_torch_gray_img(depth_file)
        img = load_torch_image(img_file)
        depth = depth.to(self.device) # Here move data to GPU before returning
        img = img.to(self.device) # Here move data to GPU before returning
        return {'img_file': img_file, 'img': img, 'depth':depth}

    def __len__(self):
        return len(self.img_files)
</code></pre>
<p>And exception:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\multiprocessing\queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\multiprocessing\reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "C:\Users\richa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\Local\pypoetry\Cache\virtualenvs\3dengine-4gtTLmD6-py3.9\lib\site-packages\torch\multiprocessing\reductions.py", line 261, in reduce_tensor
    event_sync_required) = storage._share_cuda_()
  File "C:\Users\richa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\Local\pypoetry\Cache\virtualenvs\3dengine-4gtTLmD6-py3.9\lib\site-packages\torch\storage.py", line 920, in _share_cuda_
    return self._untyped_storage._share_cuda_(*args, **kwargs)
RuntimeError: CUDA error: mapping of buffer object failed
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
</code></pre>
<p>The data loader is configured with <strong>batch_size=1</strong> and <strong>num_workers=4</strong>. Unfortunately, running with <code>CUDA_LAUNCH_BLOCKING=1</code> showed the same error.</p>
<p>I would like to ask for help with troubleshooting this issue. I could not reason what <code>CUDA error: mapping of buffer object failed</code> means and why this error did not happen deterministically. Is it legitimate to load data onto the GPU within the data loader? My program is running on Windows 10 with PyTorch 2.0 and CUDA 11.8.</p>
<p>Thank you very much. Any help is greatly appreciated!</p>

MAYANK_PUNETHA: <p>Same Issue for a different Context. Do you define number of workers anywhere?</p>

richmengsix: <p>Yes, I used num_workers=4 for data loader. I have to use extra workers to load data efficiently.</p>