Question about WRITING CUSTOM DATASETS, DATALOADERS AND TRANSFORMS tutorial

Anna_Inberg: <p>Good afternoon!<br>
I have questions about the following tutorial:<br>
<a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html</a></p>
<p>I have a similar dataset (images + landmarks) and the dataset is labeled.</p>
<ol>
<li>I didn’t quite understand from the tutorial when the labels are incorporated into the dataloader.</li>
<li>When the tutorial does image rescaling, do the landmark coordinates change accordingly?</li>
<li>In terms of image augmentation options, the tutorial only demonstrates Rescale, RandomCrop and ToTensor. In case more augmentations are needed, should I write them as a separate class (like Rescale in the tutorial) or maybe it’s possible to use the built-in transforms options? (for example, transforms.RandomHorizontalFlip() or transforms.Normalize())</li>
</ol>

Andrei_Cristea: <p>Hi Anna,</p>
<ol>
<li>The <code>Dataset</code> (<code>FaceLandmarksDataset</code>) is the one that returns both the image and the coordinates in its <code>__getitem__</code> method. This is typical, the dataloaders handle things like in what order to go through the dataset, using what minibatch size, and so on, but the core data is returned by the dataset rather than the dataloader.</li>
<li>In the tutorial, the transforms they use take in a dictionary that contains both the image and the landmarks, and indeed make the appropriate transformation to both of those. You can see that in the definition of the <code>Rescale</code> and <code>RandomCrop</code> transforms:</li>
</ol>
<pre><code class="lang-auto">        img = transform.resize(image, (new_h, new_w))
        # h and w are swapped for landmarks because for images,
        # x and y axes are axis 1 and 0 respectively
        landmarks = landmarks * [new_w / w, new_h / h]  # &lt;-- here
</code></pre>
<p>So the landmark changes are <em>not</em> something that PyTorch does for you automatically, you have to take care of that yourself explicitly by writing your own custom logic.</p>
<ol start="3">
<li>Since you need to also transform the landmarks, unfortunately you will not be able to just use most built-in functions out of the box, and you’ll have to write your own, like they do in the tutorial. If you had been working on a different project that didn’t have localized data such as bounding boxes or landmarks, such as  image classification, then you absolutely could use the builtins.<br>
To be clear, you <em>can</em> use builtins that don’t affect the geometry of the image, such as ColorJitter.</li>
</ol>
<p>Hope this helps!</p>

Anna_Inberg: <p>Andrei, thank you so much for the detailed answer.<br>
This is my first experience with PyTorch, hence tons of questions!<br>
Just to clarify:</p>
<ol>
<li>If I understood your answer correctly, I need to incorporate labels in the dataset itself, not in the dataloader, right?</li>
<li>Got it</li>
<li>Could you, please, give some examples of possible augmentations, in addition to ColorJitter, for which built-ins can be used?</li>
</ol>
<p>Thanks for your help!</p>

Andrei_Cristea: <ol>
<li>
<p>That’s right. Dataloader just iterates over the dataset in various ways that you can specify (whether or not to go in random order, how many payloads at a time to deliver, etc.) but the actual stuff that’s being yielded, in your case the pairs of (image, landmarks), comes from the dataset itself.</p>
</li>
<li>
<p>Just going down the list <a href="https://pytorch.org/vision/stable/transforms.html" rel="noopener nofollow ugc">here</a> in order: <a href="https://pytorch.org/vision/stable/generated/torchvision.transforms.ColorJitter.html#torchvision.transforms.ColorJitter" rel="noopener nofollow ugc"><code>ColorJitter</code></a>, <a href="https://pytorch.org/vision/stable/generated/torchvision.transforms.Grayscale.html#torchvision.transforms.Grayscale" rel="noopener nofollow ugc"><code>Grayscale</code></a>, <a href="https://pytorch.org/vision/stable/generated/torchvision.transforms.RandomGrayscale.html#torchvision.transforms.RandomGrayscale" rel="noopener nofollow ugc"><code>RandomGrayscale</code></a>, <a href="https://pytorch.org/vision/stable/generated/torchvision.transforms.GaussianBlur.html#torchvision.transforms.GaussianBlur" rel="noopener nofollow ugc"><code>GaussianBlur</code></a>, <a href="https://pytorch.org/vision/stable/generated/torchvision.transforms.RandomInvert.html#torchvision.transforms.RandomInvert" rel="noopener nofollow ugc"><code>RandomInvert</code></a>, <a href="https://pytorch.org/vision/stable/generated/torchvision.transforms.RandomPosterize.html#torchvision.transforms.RandomPosterize" rel="noopener nofollow ugc"><code>RandomPosterize</code></a>, <a href="https://pytorch.org/vision/stable/generated/torchvision.transforms.RandomSolarize.html#torchvision.transforms.RandomSolarize" rel="noopener nofollow ugc"><code>RandomSolarize</code></a>, <a href="https://pytorch.org/vision/stable/generated/torchvision.transforms.RandomAdjustSharpness.html#torchvision.transforms.RandomAdjustSharpness" rel="noopener nofollow ugc"><code>RandomAdjustSharpness</code></a>, <a href="https://pytorch.org/vision/stable/generated/torchvision.transforms.RandomAutocontrast.html#torchvision.transforms.RandomAutocontrast" rel="noopener nofollow ugc"><code>RandomAutocontrast</code></a>, <a href="https://pytorch.org/vision/stable/generated/torchvision.transforms.RandomEqualize.html#torchvision.transforms.RandomEqualize" rel="noopener nofollow ugc"><code>RandomEqualize</code></a>, <a href="https://pytorch.org/vision/stable/generated/torchvision.transforms.Normalize.html#torchvision.transforms.Normalize" rel="noopener nofollow ugc"><code>Normalize</code></a>, <a href="https://pytorch.org/vision/stable/generated/torchvision.transforms.RandomErasing.html#torchvision.transforms.RandomErasing" rel="noopener nofollow ugc"><code>RandomErasing</code></a></p>
<p>Basically anything that alters the color of pixels but not their position is OK.</p>
<p>Note that some of those work on both Tensors and Images (two different data types that you can convert back and forth bewteen using the ToTensor() and ToPILImage() transforms) while others work on just Tensor (such as Normalize).</p>
</li>
</ol>

Anna_Inberg: <p>Thank you so much, you’ve helped me a lot!</p>