Concatenated dataloader have different image and label order

username: chanakin
cooked: <p>So the story is that I am doing a building segmentation for various sites. This means that I need to randomly split the train val test to equal proportion (e.g. to ratio of 60%, 20%, 10% respectively) for different location, and subsequently concatenating them into a big dataset.</p>
<p>I have manage to first split the images and then match the label to them. However, I’ve found out that once I concatenated the datasets together, although both resulting pytorch datasets of image and label contain both the correct images and labels, they are not in the same order. I am struggling to find a way to match and sort the order within the pytorch dataset object or pytorch dataloader object. Have anyone had similar issues before:</p>
<p>See attached output of print(Train.png_dir, Train.lbl_dir) in the same Train &lt;pytorch.dataset&gt; object:<br>
The folder structure indicates the location:<br>
KBY = Kalobeyei<br>
DZK = Dzaleka<br>
DZKN = Dzaleka North</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/e/cebaccc5def27aa6f4226f2f94a84ad45395fbc8.png" data-download-href="https://discuss.pytorch.org/uploads/default/cebaccc5def27aa6f4226f2f94a84ad45395fbc8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/cebaccc5def27aa6f4226f2f94a84ad45395fbc8_2_690x202.png" alt="image" data-base62-sha1="tuOzHwQfmpg5XW8qAYHqAHlLDvO" width="690" height="202" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/cebaccc5def27aa6f4226f2f94a84ad45395fbc8_2_690x202.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/cebaccc5def27aa6f4226f2f94a84ad45395fbc8_2_1035x303.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/e/cebaccc5def27aa6f4226f2f94a84ad45395fbc8.png 2x" data-dominant-color="333333"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1086×319 110 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="chanakin" data-post="1" data-topic="144221">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chanakin/48/47257_2.png" class="avatar"> chanakin:</div>
<blockquote>
<p>I have manage to first split the images and then match the label to them. However, I’ve found out that once I concatenated the datasets together, although both resulting pytorch datasets of image and label contain both the correct images and labels, they are not in the same order.</p>
</blockquote>
</aside>
<p>I assume this means the labels do not correspond to the right sample anymore?<br>
If so, how did you concatenate the data and made sure the correspondence is not broken?<br>
Assuming you are using a custom <code>Dataset</code>, I guess you’ve created the data/label arrays in the <code>__init__</code>?</p>

username: chanakin
cooked: <p>Hi ptrblck,</p>
<p>Yes this is correct, the concatenation was done using, would it make sense to add a sorting element embedded within the <strong>init</strong>:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/9/89248ad6bf0e15f472cfaa568418d0431c20a844.png" alt="image" data-base62-sha1="jzdHIdI4o3wmvc1jwWkp2cbhNbK" width="637" height="146"></p>

username: chanakin
cooked: <p>I solved this by a rediculously simple solution, is to copy the current png_dir list and then re.sub the required <em>IMG</em> to <em>LBL</em>.</p>
<p>Solved, sorry for the stupid question <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:"></p>