Dynamic Dataloaders for on the fly modifications

kpprasa: <p>Hello,<br>
I am working on a project where we are trying to modify the data every n epochs. I have two questions related to this:</p>
<ol>
<li>
<p>Can we use a single dataloader and dataset to do this?<br>
ie every 5 epochs jitter the images in the trainset</p>
</li>
<li>
<p>Would it be better from a computational standpoint to perform these custom transforms in a modified dataset class or in the training loop itself after getting the natural images?</p>
</li>
</ol>
<p>Thank you,</p>

ptrblck: <p>If your use case allows to manipulate the dataset after a full epoch, I would recommend to perform the manipulations directly on the data inside the <code>Dataset</code> and just recreate the <code>DataLoader</code>.<br>
Instantiation of <code>DataLoader</code>s should be cheap, so you shouldn’t see any slow down.</p>
<p>Also, manipulating the data on the fly inside a <code>DataLoader</code> loop might now work, if you are using multiple workers, so you would be forced to use <code>num_workers=0</code> or use some shared memory approach.</p>

kpprasa: <p>Ah great, thanks! Just as a followup, the data loading is what is distributed, right? So if I modify the data in a lazy format just prior to running the model on it, then there is no need to worry about the number of worker nodes if I don’t want to save the new data, right?</p>

ptrblck: <p>In a distributed setup using <code>DistributedDataParallel</code>, you would use a single process for each device.<br>
Each process would then take care of its data loading.</p>
<p>Could you give an example of your use case and how it could interact in a bad way?</p>

kpprasa: <p>Ah thank you,<br>
I think I miscommunicated. We are not moving to a truly distributed setup. I was more curious as to the distribution of the data loading when we specify <code>num_workers</code> amongst the workers. But I took a look at the source code and think I understand.</p>
<p>Your first post helped a lot and we have the dynamic dataloader working.<br>
Thank You!</p>

jasg: <p>I dont understand, why num_workers = 0 improves performance</p>

kpprasa: <p>It’s not actually improving performance; it’s attempting to allow modification of a dataloader while training. Usually, the data loading is done by multiple workers and the changes wouldn’t be reflected across all worker threads. Thus, if <code>num_workers= 0</code>, then we do not have this issue.</p>

kpprasa: <p>Note for anyone trying to do this in the future:<br>
If you create a separate member variable in the dataset with the transforms, you can change that variable and just modify the <code>__get_item__</code> method to have an expanded length and index</p>
<p>here’s a simple example:</p>
<pre><code class="lang-auto">class Dynamic_dataset(dataset):


    def __init__(
        self , data):
        

        self.orig_train_size = self.data.shape[0]
        
        self.transforms = []
     

    

    def add_transforms(self, transforms):
        self.transforms = transforms


    

    def __getitem__(self, index):
        """
        Args:
            index (int): Index
        Returns:
            tuple: (image, label) where target is class_index of the target class.
        """
        # For indexes past the original data, find from transforms
        if index &gt;= self.orig_train_size:
                # Use the original data
                image, label = (
                    self.data[int(index % len(self.data))],
                    self.targets[int(index % len(self.data))],
                )
                image = torch.FloatTensor(image)
                pert_index = index - self.orig_train_size
                image = image + self.perturbations[pert_index]
                image = torch.clamp(image, 0.0, 1.0)
        else:
            image, label = self.data[index], self.targets[index]
            image = torch.FloatTensor(image)

        return image, label

    def __len__(self):
        return self.data.shape[0] + len(self.transforms)
</code></pre>
<p>You can access the method by<br>
<code>train_loader.dataset.add_transforms(transforms)</code> (once you define your train_loader…)</p>

jasg: <p>I don’t understand how I would adapt that in my code, please support me</p>
<pre><code class="lang-auto">          train_transform = transforms.Compose([
                    transforms.RandomRotation(10),      # rotate +/- 10 degrees
                    transforms.RandomHorizontalFlip(),  # reverse 50% of images
                    transforms.Resize(image),             # resize shortest side to 224 pixels
                    transforms.CenterCrop(image),         # crop longest side to 224 pixels at center
                    transforms.ToTensor(),
                    transforms.Normalize([0.485, 0.456, 0.406],
                                         [0.229, 0.224, 0.225])
                ])
            
                test_transform = transforms.Compose([
                    transforms.Resize(image),
                    transforms.CenterCrop(image),
                    transforms.ToTensor(),
                    transforms.Normalize([0.485, 0.456, 0.406],
                                         [0.229, 0.224, 0.225])
                ])
                
                inv_normalize = transforms.Normalize(
                mean=[-0.485/0.229, -0.456/0.224, -0.406/0.225],
                std=[1/0.229, 1/0.224, 1/0.225]
                )
                
                print("RGB")
                
            
            train_data = datasets.ImageFolder(os.path.join(root, 'train_real'), transform=train_transform)
            test_data = datasets.ImageFolder(os.path.join(root, 'validation'), transform=test_transform)
            
            
            
            torch.manual_seed(42)
            

            train_loader = DataLoader(train_data, batch_size=10,num_workers=2, pin_memory=False,shuffle=True)
            test_loader = DataLoader(test_data, batch_size=10, num_workers=2,pin_memory=False,shuffle=True)
            
            #obtiene los labels o clases del dataset
            class_names = train_data.classes
            
            print(class_names)
            print(f'Training images available: {len(train_data)}')
            print(f'Testing images available:  {len(test_data)}')</code></pre>

kpprasa: <p>Do you need custom dynamic transformations? Most standard applications can just use:<br>
<a href="https://pytorch.org/docs/stable/torchvision/transforms.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/transforms.html</a><br>
the transforms above. I am working on some research regarding adversarial robustness that required custom transformations on the fly.</p>
<p>If you tell us your use case, we can point you in the right direction.</p>

Jaideep_Valani: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> my requirement in this regard is to update the labels after epoch ,as I consider those labels incorrect ,so correcting it for next epoch .<br>
How can we do that …</p>

ptrblck: <p>If you are using <code>persistent_workers=False</code> in the <code>DataLoader</code> (the default setup), you should be able to manipulate the underlying <code>Dataset</code> after each epoch even with <code>num_workers&gt;0</code>.</p>

sqiangcao: <p>@ <a href="https://discuss.pytorch.org/t/dynamic-dataloaders-for-on-the-fly-modifications/78870/2">ptrblck</a>, hi, thanks for you answer.  I use <code>data_loaders.sampler.__init__(data_loaders.dataset)</code> to solve this problem in my case. Are there potential bugs?</p>