Data Augementation on the fly

Emma00: <p>Hi everyone.</p>
<p>I have this training set of 2997 samples, where each sample has size 24x24x24x16. I would like to augment it by 24 times through rotation. Ideally the rotation should have been of 90 degrees, thus in order to get 23 different sample (the first one is the orignal) i would have to change the ax of rotation [(0,1), (1,0), (2,0), (0,2)] ecc. Six permutations are required.</p>
<p>Everthing should be on the fly, so once the sample is retrived with <code>__getitem__</code> ,  i should generate 24 samples.</p>
<p>By now, i don’t think that an actual augmentation is done (i adjusted the original version of the code i am working on). I printed some tensors and indeed i don’t see any rotation, only the same sample replied.<br>
Plus, i was considering to move everthing to tensors and not working with Numpy arrays, so i need to modify the code anyway.</p>
<p>This the class in which i perform the augmentation on the fly:</p>
<pre><code class="lang-auto">class AugmentedDataGenerator(Dataset):
    # It initializes the instance of the class
    def __init__(self, x, y, aug_count=24):
        # features and labels 
        self.x = x
        self.y = y
        self.aug_count = aug_count

    def __len__(self):
        # multiplication for the augmentation
        return len(self.x) * self.aug_count

    def __getitem__(self, idx):
        # Original index (actual divided by the augumentation index)
        index = idx // self.aug_count 
        # rotation index will be in the range from 0 to 23. 
        rotation_index = idx % self.aug_count

        # retriving the sample
        sample_x = self.x[index]
        sample_y = self.y[index]
                
        aug_x = self._rotate_sample(sample_x, rotation_index)
        print(aug_x.shape)
        aug_y = sample_y

        return aug_x, aug_y

    # This method performs rotation augmentation on the input sample
    def _rotate_sample(self, sample, rotation_index):
        output = np.zeros_like(sample)
        # axes of rotation
        axes = [(1, 2), (0, 2), (0, 1)]
    
        for axis in axes:
            # rotates the sample
            rotated_sample = np.rot90(sample, k=rotation_index, axes=axis)
            # maximum value across all rotations along different
            output = np.maximum(output, rotated_sample)

        # it holds the sample that has been rotated along all specified axes
        return output
</code></pre>
<p>This is the beginning of the training:</p>
<pre><code class="lang-auto"> # Loops over each batch in train_loader
    for i, (input, target) in enumerate(train_loader): 
        # Measure data loading time
        data_time.update(time.time() - end)
                
        # Moves both the input data and the target labels to the CPU or GPU 
        print("batch", i, "and input shape", input.shape)
        input = input.reshape(-1, 16, 24, 24, 24).to(device, non_blocking=True)
        target = target.view(-1, 1).to(device, non_blocking=True)
</code></pre>
<p>It would much easier to augment the data once and save to disk, but i runned out of memory.</p>
<p>I tried to apply this transformation:</p>
<pre><code class="lang-auto">transform =v2.Compose([
    v2.ToDtype(torch.float64),
    v2.RandomRotation(degrees=(0,360))    
])
</code></pre>
<p>by doing this in the <code>__getitem__</code> method:</p>
<pre><code class="lang-auto"># Retrieving a sample from the dataset given its index
    def __getitem__(self, idx):
        if self.transform is None: 
            features= self.x_data[idx]
            label =  self.y_data[idx]
        else:
            index = idx // self.aug_count 
            rotation_index = idx % self.aug_count
            features = torch.stack([torch.tensor(self.transform(self.x_data[index])) for _ in range(24)], dim=0)
            label = self.y_data[index]
    
        return features, label
</code></pre>
<p>But then i have that each batch has its size multiplicated by 24 and the dimensions don’t make sense.<br>
<code>Batch 0 - Shape of the input torch.Size([8, 24, 24, 24, 24, 16])</code></p>
<p>To conclude, i think i’m losing the thread here. I would like to understand better these concept:</p>
<ul>
<li>how to apply augmentation on the fly formally (in which class / methods)</li>
<li>how to deal with the resulting augmented data and the dimensions of the batches</li>
</ul>
<p>Thank you in advance.</p>

KFrank: <p>Hi Emma!</p>
<aside class="quote no-group" data-username="Emma00" data-post="1" data-topic="202877" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/48db29/48.png" class="avatar"> Emma00:</div>
<blockquote>
<p>each sample has size 24x24x24x16.</p>
</blockquote>
</aside>
<p>Do I understand correctly that each sample is a three-dimensional image with<br>
height = width = depth = 24 (and with 16 channels)?</p>
<aside class="quote no-group">
<blockquote>
<p>I would like to augment it by 24 times through rotation.</p>
</blockquote>
</aside>
<p>This makes sense to me – a cube has 24 (non-reflecting) rotations (including<br>
the trivial rotation that doesn’t change anything).</p>
<p>You can count them like this:  A cube has six faces, so you have six choices of<br>
which face to rotate to the top.  Then you have a choice of four rotations you<br>
can make around the bottom-to-top axis (0, 90, 180, and 270 degrees).  Then<br>
six times four gives you 24.</p>
<p>Note that (assuming that I understand your use case correctly) you <em>can’t</em> get<br>
all 24 rotations of the cube by rotating around a specific <code>(0, 1)</code>-style axis.  You<br>
have three such axes:  bottom-to-top, left-to-right, and front-to-back.  Around<br>
each axis you have three non-trivial rotations. (The 0-degree rotation is trivial.)<br>
So you have only ten such “around-an-axis” rotations:  the trivial rotation plus<br>
nine (three times three) non-trivial rotations.</p>
<aside class="quote no-group">
<blockquote>
<p>Ideally the rotation should have been of 90 degrees, thus in order to get 23 different sample (the first one is the orignal) i would have to change the ax of rotation [(0,1), (1,0), (2,0), (0,2)] ecc. Six permutations are required.</p>
</blockquote>
</aside>
<p>I don’t follow your logic here.</p>
<aside class="quote no-group">
<blockquote>
<p>Everthing should be on the fly, so once the sample is retrived with <code>__getitem__</code> ,  i should generate 24 samples.</p>
</blockquote>
</aside>
<p>This makes sense and is certainly doable (along the general lines sketched in<br>
the code you posted).</p>
<aside class="quote no-group">
<blockquote>
<p>i was considering to move everthing to tensors and not working with Numpy arrays</p>
</blockquote>
</aside>
<p>Yes, I would recommend writing all of this in pytorch.  This will simplify things a<br>
little bit by not using numpy where pytorch can do the same thing.  Plus using<br>
pytorch tensors will get you the benefits of a gpu, if you have one.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">    def __getitem__(self, idx):
        # Original index (actual divided by the augumentation index)
        index = idx // self.aug_count 
        # rotation index will be in the range from 0 to 23. 
        rotation_index = idx % self.aug_count
</code></pre>
</blockquote>
</aside>
<p>Yes, <code>rotation_index</code> will run from 0 to 23.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        # retriving the sample
        sample_x = self.x[index]
        sample_y = self.y[index]
</code></pre>
</blockquote>
</aside>
<p>You don’t say what <code>sample_y</code> is.  I assume that it is some sort of ground-truth<br>
target.  Depending on your use case, you may or may not have to rotate <code>sample_y</code><br>
along with <code>sample_x</code>.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">    def _rotate_sample(self, sample, rotation_index):
        ...    
        for axis in axes:
            # rotates the sample
            rotated_sample = np.rot90(sample, k=rotation_index, axes=axis)
</code></pre>
</blockquote>
</aside>
<p>Here, you are passing <code>rotation_index</code> – which runs up to 23 – to <code>rot90()</code>.  This<br>
will only give you four distinct results (for <code>k = 0, 1, 2, 3</code>).  (I assume that <code>rot90()</code><br>
returns the same result for, say, <code>k = 1</code> and <code>k = 5</code>.)  You need a scheme to map your<br>
<code>rotation_index</code> to the 24 distinct rotations of a cube.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">            # maximum value across all rotations along different
            output = np.maximum(output, rotated_sample)
</code></pre>
</blockquote>
</aside>
<p>I don’t understand why you are taking the <code>maximum()</code> here.  If this is really what<br>
you want, could you explain your use case in greater detail?</p>
<aside class="quote no-group">
<blockquote>
<p>It would much easier to augment the data once and save to disk, but i runned out of memory.</p>
</blockquote>
</aside>
<p>Performing augmentation on the fly make good sense here.  Your general approach<br>
looks sound (although some of the details look wrong to me).</p>
<p>Best.</p>
<p>K. Frank</p>

Emma00: <p>Thank you Frank for your quick and extensive the response!</p>
<p>I am going to clarify some aspects because I was hasty in writing the answer.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="202877">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Do I understand correctly that each sample is a three-dimensional image with<br>
height = width = depth = 24 (and with 16 channels)?</p>
</blockquote>
</aside>
<p>Yes. Each sample is representing a protein-drug complex that has been voxelized in 3D grids. Each cell of the grid has 16 channels. So in the end each sample has shape (24,24,24,16).</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="202877">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>You can count them like this: A cube has six faces, so you have six choices of<br>
which face to rotate to the top. Then you have a choice of four rotations you<br>
can make around the bottom-to-top axis (0, 90, 180, and 270 degrees). Then<br>
six times four gives you 24.</p>
<p>Note that (assuming that I understand your use case correctly) you <em>can’t</em> get<br>
all 24 rotations of the cube by rotating around a specific <code>(0, 1)</code>-style axis. You<br>
have three such axes: bottom-to-top, left-to-right, and front-to-back. Around<br>
each axis you have three non-trivial rotations. (The 0-degree rotation is trivial.)<br>
So you have only ten such “around-an-axis” rotations: the trivial rotation plus<br>
nine (three times three) non-trivial rotations.</p>
</blockquote>
</aside>
<p>I think that what you explained here was pratically what i was trying to saying very confusingly  in the next lines (the ones you didn’t follow).<br>
I posted and was working on because, as you said, it was incorrect.<br>
Here is how I rewrote it. It is basic and not optimized, but I needed a better understanding of how the cube was rotated.</p>
<pre><code class="lang-auto">def rotate_sample(sample):
    # Initialize output array to store rotated samples
    output = np.zeros((10,) + sample.shape) 
    
    # Initialize counter to keep track of the current index
    counter = 0
    
    # Define axes representing different planes of rotation
    axes = [(0, 1), (1, 2), (0, 2)]
    
    # Save the original sample
    output[counter] = sample
    counter += 1
    
    # For angle in [0, 90, 180, 270]
    for plane in axes:
        for angle in [90, 180, 270]:
            # Rotate the sample (k=1,2,3)
            rotated_sample = np.rot90(sample, k=angle//90, axes=plane)

            # Save the rotated sample
            output[counter] = rotated_sample
            counter += 1
    
    # x times the rotated sample 
    return output


def aug_data_generator(sample_x, sample_y):
    aug_count = 10 # Origin + rotation
    
    # Arrays for stroing 
    aug_data_x = np.zeros((sample_x.shape[0] * aug_count,) + sample_x.shape[1:])
    aug_data_y = np.zeros((sample_y.shape[0] * aug_count,))
    
    # Take the given dataset
    for i in range(sample_x.shape[0]):
        # Apply the rotation on each sample
        aug_x = rotate_sample(sample_x[i])
        # Repeat the label
        aug_y = np.repeat(sample_y[i], aug_count)
        
        # Assign augmented data to the output arrays
        start_index = i * aug_count
        end_index = start_index + aug_count
        aug_data_x[start_index:end_index] = aug_x
        aug_data_y[start_index:end_index] = aug_y

    return aug_data_x, aug_data_y
</code></pre>
<p>I decided to keep 10, as you indicated, because I don’t understand how you can get to 24 different rotations of the cube. In fact, my reference code was not augmenting the data. It was creating 23 copies for each sample.</p>
<p>I have to do everything on the fly because I get a memory out of memory if I try to save the data to disk at once.  There are some ways to efficiently write a lot of data to disk, but I need to read up on it because I have never done it before.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="202877">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Yes, I would recommend writing all of this in pytorch. This will simplify things a<br>
little bit by not using numpy where pytorch can do the same thing. Plus using<br>
pytorch tensors will get you the benefits of a gpu, if you have one.</p>
</blockquote>
</aside>
<p>Thank you for the suggestion. I will try to use tensors and not array. I have a gpu so i will also move the tensors to it.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="202877">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>You don’t say what <code>sample_y</code> is. I assume that it is some sort of ground-truth<br>
target</p>
</blockquote>
</aside>
<p>Yes is the ground-truth target. Is not rotating but it is repeated for each augmented sample.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="202877">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Here, you are passing <code>rotation_index</code> – which runs up to 23 – to <code>rot90()</code>. This<br>
will only give you four distinct results (for <code>k = 0, 1, 2, 3</code>). (I assume that <code>rot90()</code><br>
returns the same result for, say, <code>k = 1</code> and <code>k = 5</code>.) You need a scheme to map your<br>
<code>rotation_index</code> to the 24 distinct rotations of a cube</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="202877">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>I don’t understand why you are taking the <code>maximum()</code> here. If this is really what<br>
you want, could you explain your use case in greater detail</p>
</blockquote>
</aside>
<p>As you can see i modified these 2  part because they weren’t clear to me either. Especially the maximum was random.</p>
<p>Apart of this problems around the rotation I was more concerned on generating data dynamically or not, and if there were some well defined cases that i could follow for augmentation. This last concern is due to the fact that I am just beginning to learn about the world of deep leraning and Pytorch, so I am trying to keep the code as clean and simple as possible, following the structures found in the Pytorch tutorials and examples.</p>
<p>Thank you a lot!</p>
<p>Best,<br>
Emma.</p>
<p>P.S:<br>
This is the final version that should work correctly:</p>
<pre><code class="lang-auto">class AugmentedDataGeneratorPytorch(Dataset):
    def __init__(self, x, y, aug_count=10):
        self.x = torch.tensor(x, dtype=torch.float32)
        self.y = torch.tensor(y, dtype=torch.long)
        self.aug_count = aug_count

    def __len__(self):
        return len(self.x) * self.aug_count

    def __getitem__(self, idx):
        original_index = idx // self.aug_count
        augmentation_index = idx % self.aug_count
        
        sample_x = self.x[original_index]
        sample_y = self.y[original_index]
        
        augmented_sample = self._rotate_sample(sample_x, augmentation_index)
        
        return augmented_sample, sample_y

    def _rotate_sample(self, sample, augmentation_index):
        if augmentation_index == 0:
            return sample

        axes = [(1, 2), (0, 2), (0, 1)]
        
        counter = 1
        for plane in axes:
            for angle in [90, 180, 270]:
                if counter == augmentation_index:
                    rotated_sample = torch.rot90(sample, k=angle//90, dims=plane)
                    return rotated_sample
                counter += 1

        raise ValueError("Invalid augmentation index")
</code></pre>

KFrank: <p>Hi Emma!</p>
<aside class="quote no-group" data-username="Emma00" data-post="3" data-topic="202877" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/48db29/48.png" class="avatar"> Emma00:</div>
<blockquote>
<p>I was more concerned on generating data dynamically or not, and if there were some well defined cases that i could follow for augmentation.</p>
</blockquote>
</aside>
<p>Your scheme where you use an indexed <code>Dataset</code> (one that implements<br>
<code>__getitem__(self, idx)</code>), where <code>idx</code> encodes both the actual image <code>index</code><br>
and also the <code>rotation_index</code>, and then use <code>rotation_index</code> to generate<br>
your rotated image on the fly is fine.</p>
<p>(You could also follow the pytorch <a href="https://pytorch.org/vision/0.18/transforms.html" rel="noopener nofollow ugc">transforms</a> approach where you would add<br>
a random-cube-rotation transform to your <code>Dataset</code>.  Now each <code>idx</code> would map<br>
to a specific real image – no <code>rotation_index</code> encoded in it – but each time you<br>
fetch an image with the same <code>idx</code>, you would get a different randomly-rotated<br>
version of the same real image.  This would be an alternative approach – I tend<br>
to like your scheme better.)</p>
<p>If you then access your <code>AugmentedDataGenerator</code> using a <code>DataLoader</code> with<br>
<code>shuffle = True</code>, the random <code>idx</code> that the <code>DataLoader</code> uses to retrieve an image<br>
will map to a specific (but random) rotation of the “real” image, thus giving you<br>
your desired data augmentation.  There’s nothing the matter with this scheme.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">class AugmentedDataGeneratorPytorch(Dataset):
    def __init__(self, x, y, aug_count=10):
</code></pre>
</blockquote>
</aside>
<p>It’s perfectly legitimate to augment your data using only 10 of the 24 rotations<br>
of the cube.  Nothing will break or give you wrong results.  However, there really<br>
are 24 rotations of the cube.</p>
<p>Here’s another way to count them:</p>
<p>A cube has 12 edges.  Pick one of those edges and think about rotating the cube<br>
so that that chosen edge becomes the lower front edge.  But there are two ways<br>
a given edge can become the lower front edge: it can either run left-to-right or<br>
right-to-left.  So there are 12 times 2 = 24 rotations of the cube.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">                    rotated_sample = torch.rot90(sample, k=angle//90, dims=plane)
</code></pre>
</blockquote>
</aside>
<p>Again, it is not possible to get all 24 rotations of the  cube by using a single<br>
<code>rot90()</code>.  Consider the three (face-to-face) axes of a cube.  (You have labelled<br>
them <code>[(1, 2), (0, 2), (0, 1)]</code>.)  Any <code>rot90</code> rotation of the cube leaves one<br>
of the three axes unchanged, while swapping the other two with one another.<br>
This makes it easy to construct a specific example of a cube-rotation for which<br>
it is easy to see that it is not a <code>rot90()</code>-rotation.</p>
<p>Consider a rotation that permutes all three axes:</p>
<p><code>[(1, 2), (0, 2), (0, 1)]</code> → <code>[(0, 1), (1, 2), (0, 2)]</code></p>
<p>Such a rotation leaves <em>no</em> axis unchanged, so it can’t be a <code>rot90()</code>-rotation.</p>
<p>It is not necessary to use all of the cube rotations to augment your data, but why<br>
not?  In for a penny, in for a pound, as they say.</p>
<p>Best.</p>
<p>K. Frank</p>

Emma00: <p>Thank you again Frank.</p>
<p>I will try augmenting with 24 rotations, even if the training will slow down.<br>
So basically what i have to do is</p>
<ul>
<li>Fix one face as the top face</li>
<li>Rotate the cube around the vertical axis passing through the top face</li>
</ul>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="202877">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>But there are two ways<br>
a given edge can become the lower front edge: it can either run left-to-right or<br>
right-to-left. So there are 12 times 2 = 24 rotations of the cube.</p>
</blockquote>
</aside>
<p>So, for example:</p>
<ul>
<li>(0, 1) represents an edge running from left to right.</li>
<li>(1, 0) represents the same edge but described from right to left.</li>
</ul>
<p>Thank you!<br>
Best,<br>
Emma.</p>