Two set of batches from two different dataloaders

username: Sumesh_Sankar
cooked: <p>How do i get two set of batches at the same time from two different dataloaders for iteration,<br>
for example we can get batches of data from a single data loader as</p>
<p>for i,data in enumerate(dataloader,0):<br>
inputs,labels = data</p>
<p>but i need to get two batches simultaneously from two data loaders, something like<br>
for i,j,data_1,data_2 in enumerate(dataloade1,dataloader2,0):<br>
inputs1,labels1 = data_1<br>
inputs2,labels2 = data_2</p>

username: ptrblck
cooked: <p>You can use <code>zip</code>:</p>
<pre><code class="lang-python">datasetA = TensorDataset(torch.randn(10, 1), torch.randn(10, 1))
datasetB = TensorDataset(torch.randn(10, 1), torch.randn(10, 1))
loaderA = DataLoader(datasetA, batch_size=5)
loaderB = DataLoader(datasetA, batch_size=5)

for (dataA, targetA), (dataB, targetB) in zip(loaderA, loaderB):
    print(dataA.shape)
</code></pre>
<p>to iterate both <code>DataLoader</code>s.</p>

username: Sumesh_Sankar
cooked: <p>Hai ptrblck,<br>
i found this run time error.<br>
RuntimeError: Caught RuntimeError in DataLoader worker process 0.<br>
Original Traceback (most recent call last):<br>
File “/home/sinnu/.local/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py”, line 302, in _worker_loop<br>
data = fetcher.fetch(index)<br>
File “/home/sinnu/.local/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py”, line 58, in fetch<br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “/home/sinnu/.local/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py”, line 58, in <br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
RuntimeError: CUDA error: initialization error<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.</p>

username: ptrblck
cooked: <p>It seems you are using <code>CUDATensor</code>s inside the <code>Dataset</code>, which then fails while trying to initialize multiple contexts. Load the data on the CPU, change the startmethod of multiprocessing to <code>spawn</code>, or use the main thread (<code>num_workers=0</code>).</p>

username: Sumesh_Sankar
cooked: <p>Yeah, it worked, but if you use num_worker = 0 and pin_memory = False , we are blocking the option of making GPU fast. One of the reasons, i use dataloader is to make training fast by making num_worker = 50 and  pin_memory = True. Now i dont have these options? then how to solve this problem with these options also. Also can you tell what is startmethod of multiprocessing?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Sumesh_Sankar" data-post="5" data-topic="188290">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sumesh_sankar/48/45448_2.png" class="avatar"> Sumesh_Sankar:</div>
<blockquote>
<p>One of the reasons, i use dataloader is to make training fast by making num_worker = 50 and pin_memory = True. Now i dont have these options?</p>
</blockquote>
</aside>
<p>You are creating <code>CUDATensor</code>s inside the <code>Dataset</code> so setting <code>pin_memory=True</code> would be useless as the data is already on the GPU.<br>
I’ve suggested a few alternatives, one which is to use tensors on the CPU allowing you to use multiple workers and pinned memory.</p>