How to create custom dataset for training NN

misiak: <p>Hey guys,<br>
I´m new in developing with pytorch and i would appreciate if someone could help me with creating my own dataset. I checked many topics and forums but it seemed like no one was in the same situation. I got data with following directory hierarchy <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8ada9399974bc1dcb35ffd1771b0d6d7497afb58.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8ada9399974bc1dcb35ffd1771b0d6d7497afb58" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8ada9399974bc1dcb35ffd1771b0d6d7497afb58_2_690x233.jpeg" alt="1" data-base62-sha1="jOmbIHcVH8nXR0vVNPHcbeSJYCY" width="690" height="233" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8ada9399974bc1dcb35ffd1771b0d6d7497afb58_2_690x233.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/8/a/8ada9399974bc1dcb35ffd1771b0d6d7497afb58.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/a/8ada9399974bc1dcb35ffd1771b0d6d7497afb58.jpeg 2x" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">873×295 35.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I also got an CSV file formats, as you can see is called test-480_ImgPats. Inside this CSV is a path to images which looks like this - this screenshot will be in comment below.<br>
I have also created a directory called “labels” for each image with corresponding class (0,1,2,3,4).<br>
Another additional data for each image called text-480_data have 4 numbers corresponding to each image.</p>
<p>For example. In directory Test is another directory called 4_24, where is image with number 1.jpg. This image had created label in directory labels/Test/4_24/1.jpg with output 0-4 classes. Every image has metadata (CSV file called Test-480_data) that contains 4 numbers separated with comma. And every image has a CSV with paths.</p>
<p>How to create my own dataset with this annotations ?<br>
I hope that someone will understand <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"><br>
Thank you everyone !</p>

misiak: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d8a53d7f0503f6ebd5b1c661c3255db52c48d3f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8d8a53d7f0503f6ebd5b1c661c3255db52c48d3f" title="2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d8a53d7f0503f6ebd5b1c661c3255db52c48d3f_2_690x219.jpeg" alt="2" data-base62-sha1="kc7GHe3TrXMUXBVL6hffTyz0cwv" width="690" height="219" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d8a53d7f0503f6ebd5b1c661c3255db52c48d3f_2_690x219.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d8a53d7f0503f6ebd5b1c661c3255db52c48d3f_2_1035x328.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d8a53d7f0503f6ebd5b1c661c3255db52c48d3f_2_1380x438.jpeg 2x" data-dominant-color="EBECEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">1395×444 171 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This is screenshot 2.<br>
I forgot to mention that each image have 480x360 px with RGB.</p>

ptrblck: <p>I’m not sure I understand the data structure completely, but would recommend to use e.g. <code>pandas</code> and to make sure you can load all images with their label.<br>
Once you’ve created the mapping between the image paths, their metadata, and targets, I would create a csv (or any other format you like), such that indexing this file will give you everything you need to load the data sample and create or load the target.</p>
<p>If that’s done, you could load this new csv in <code>Dataset.__init__</code>, and lazily load each sample in its <code>__getitem__</code>.</p>