How do I train using multiple data loaders

username: ysleer
cooked: <p>Hi all.</p>
<p>I am trying to train my model using various image sizes and batch sizes.</p>
<p>I want to train my model using multiple dataloaders.</p>
<p>Here’s how I can simply think about it in code.</p>
<pre><code class="lang-auto">dataloader1(image_size=(1024, 512), batch_size=2,...)
dataloader2(image_size=(512, 256), batch_size=4,...)
dataloader3(image_size=(256, 128), batch_size=8,...)

for epoch in range(0, epochs):
    for idx, (input, target) in enumerate(dataloader1):
            loss1
    for idx, (input, target) in enumerate(dataloader2):
            loss2
    for idx, (input, target) in enumerate(dataloader3):
            loss3

    tot_loss = loss1 + loss2 + loss3
    tot_loss.backward()
</code></pre>
<p>The method that comes to mind right now is the same as above. Does the above method work properly? Or is there another efficient way?</p>
<p>In summary, I want to train one model using multiple data loaders, add up each loss and update it back to one model.</p>
<p>Please let me know if there is any possible example or method.</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>Your posted code could work assuming your model is small and you have enough memory.<br>
Assuming you are summing the losses inside the <code>DataLoader</code> loop, this approach would store the computation graphs, which would allocate more memory in each iteration.<br>
Depending on the size of your dataset, you might be quickly running out of memory.<br>
As an alternative, you could call <code>backward()</code> on each loss in each iteration, so that Autograd can free the intermediate tensors, as they are not needed anymore.</p>

username: ysleer
cooked: <p>Thank you for answer.</p>
<p>I have additional questions. If you want to use memory efficiently when trying to learn using multiple dataloaders, is there any way to modify the inside of the dataloader?</p>
<p>Or Is there a way to share the queue of each data loader?</p>

username: ptrblck
cooked: <p>You could concatenate the datasets via <code>ConcatDataset</code> and pass it to a single <code>DataLoader</code>.<br>
However, you would have to make sure that the output sizes of all tensors are equal (doesn’t seem to be the case in your example) or you would have to use a custom <code>collate_fn</code>.</p>

username: ysleer
cooked: <p>Thanks for the good answer.</p>
<p>But what I want to do is use multiple data loaders. Sorry, is there any way to utilize each data loader’s queue??</p>

username: ptrblck
cooked: <p>Each <code>DataLoader</code> would use its own queue and I don’t think you can share it between different loaders.</p>

username: ysleer
cooked: <p>I’ll ask you one more thing.</p>
<p>If so, is there any other way to receive and train different image sizes and different batch sizes as inputs except collate_fn, concatdataset??</p>
<p>As far as I know, the batch size should be the same when concatdataset is applied.</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>If you are using different loaders, as posted in your initial code, then yes, each loader can use different input shapes and batch sizes.<br>
If you concatenate the dataset and use a single loader, the batch size would be set in the <code>DataLoader</code> creation unless you disable <a href="https://pytorch.org/docs/stable/data.html#loading-batched-and-non-batched-data">automatic batching</a> and customize the loading pipeline.</p>

username: ysleer
cooked: <p>Sorry.</p>
<p>I don’t understand the second paragraph a bit. Could you please tell me more specifically?</p>
<p>Thank you</p>