How to load dataset more efficient

xin_du: <p>I created a dataset class by myself, but I need to change the labels of the data points for different iterations. Using for loop to go through each point will be quite slow as it needs to be repeated several times in the training process.</p>
<p>I tried to use dataset[:][‘labels’] to revalue the labels, but the error said TypeError: join() argument must be str or bytes, not ‘Series’. For the data loader, it cannot extract all the labels without the iteration too.  Do you have any solution for solving it more efficiently? Thanks</p>

Matias_Vasquez: <p>If you created your own <code>Dataset</code>, then you could do this lazily.</p>
<p>You could define a variable inside your <code>Dataset</code> that keeps track of what iteration it is, and according to it you could change what the returned label is. If you want, you can also add methods to update this variable, or change it directly when your epoch/iteration is done.</p>
<p>Here is a small example of how you could do it. Then, when you call <code>__getitem__</code> you will get the corresponding label. In this example the label is the same as <code>self.iteration</code>, but you would need to change it to whatever it is that you need.</p>
<pre><code class="lang-python"># Dataset example

class MyCustomDataset(torch.utils.data.Dataset):
    def __init__(self, *args, **kwargs):
        self.data = list(range(10))
        self.iteration = 0

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        data_point = self.data[idx]

        # Here comes your logic to change the label according to the iteration
        label = self.iteration

        return data_point, label

    def change_iteration(self, iteration):
        self.iteration = iteration
      
    def reset_iteration(self):
        self.iteration = 0
</code></pre>
<pre><code class="lang-python"># Small usage example

ds = MyCustomDataset()
dl = torch.utils.data.DataLoader(ds, batch_size=10)

for epoch in range(3):
    ds.change_iteration(epoch)
    for i, (data, lbl) in enumerate(dl):
        print(data, lbl)
</code></pre>
<pre><code class="lang-python"># Output

tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) tensor([2, 2, 2, 2, 2, 2, 2, 2, 2, 2])
</code></pre>

xin_du: <p>Thanks, it is helpful. I have a question that the <strong>getitem</strong> function that will be called in every epoch for changing all labels. Is that an efficient way to go through labels compared to changing labels outside the dataset class where we only need to go through all the labels in the first epoch (if the iteration means the run time of training a models instead of epochs)? Thanks</p>

Matias_Vasquez: <p>If you have created a <code>Dataset</code> class similar to mine, then every time you iterate through it with a <code>DataLoader</code> you will get the items through the <code>__getitem__</code> function.</p>
<p>This means that adding this little bit of logic should not really affect the performance. On the other side, if every time you want to change the label you iterate through your entire dataset, then you will be doing double the work (iterating once to change the label and iterating once to get the item).</p>
<p>What we are doing at the beginning of each epoch (<code>ds.change_iteration(epoch)</code>) is only changing <strong>one</strong> number inside the <code>Dataset</code>, <strong>not</strong> going through every datapoint and changing its label. The label is later determined when getting each object.</p>
<p>If there is something not clear please let me know.</p>
<p>But it might help if you share the relevant part of your code or a reproducible example of what you want.</p>