Tip: sampling withour replacement from my custom dataloader

username: Matt2
cooked: <p>I have a custom dataloader where my available ids for picking samples of my dataset are stored during the initialization of the dataloader as follows:</p>
<p><code>self.tileIds = [4, 56, 78, 10, 23]</code>,</p>
<p>and in the <code>__get_item()__</code> function i sample elements as follows:</p>
<pre><code class="lang-auto">def __getitem__(self, idx):
        dataId = self.tileIds[idx]
        img = self.getRawSample(dataId)
        meta = self.meta[idx]
</code></pre>
<p>the problem is that I can draw the same id from tileIds in the same epoch and I wonder if there is a smart method to avoid this behavior with some sort of callbacks for example. I would like to use pure Pytorch without lightning.</p>

username: Matias_Vasquez
cooked: <p>hi,</p>
<p>You might want to look into the sampler/batch_sampler parameters of the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc">Dataloader</a>.</p>
<p><a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/sampler.html#RandomSampler" rel="noopener nofollow ugc">Here</a> are different strategies to create samplers. You could create your own, using your available ids.</p>
<p>If you post a little more of your code I might be able to help you a little more.</p>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>

username: ejguan
cooked: <p>You can sort your <code>self.tileIds</code> to mimic sampling without replacement</p>

username: Matt2
cooked: <p>I found these samplers but I didn’t get how to integrate the inside the dataloader <strong>getitem</strong>() method!</p>
<p>What part of the code do you need to understand better my problem?</p>

username: Matt2
cooked: <p>The list is already sorted, however the <strong>getitem()</strong> will sample at random from this list.</p>

username: Matias_Vasquez
cooked: <p>Maybe I’m not understanding right, so please correct me if I’m wrong.</p>
<p>What I think you have is a custom <strong>Dataset</strong> defined, where you have a list with the possible indexes that can be used.</p>
<p>Maybe something like this:</p>
<pre><code class="lang-python">class MyCustomDataset(Dataset):
    def __init__(self, tileIds):
        super().__init__()
        self.tileIds = tileIds
        self.data

    def __getitem__(self, idx):
        dataId = self.tileIds[idx]
        img = self.getRawSample(dataId)
        meta = self.meta[idx]
        return img, meta

    def __len__(self):
        return len(self.samples) # Whatever the actual length is
    
    def getRawSample(self, idx):
        return random.random() # Whatever your RawSample is
</code></pre>
<p>Then I am assuming that you create your actual <strong>DataLoader</strong></p>
<pre><code class="lang-python">ds = MyCustomDataset(tileIds=[4, 56, 78, 10, 23])
dl = DataLoader(ds, batch_size=8)
</code></pre>
<p>What I meant, is that when you create your <strong>DataLoader</strong>, you can specify the Sampler, to something custom that you create on your own, or something that is already defined, like the <strong>SubsetRandomSampler</strong>. This will take a list of indices, and will select values from this list at a random order, which will then be used with your <strong>Dataset</strong>, as the index that is passed to <code>__getitem__()</code>.</p>
<p>With this, you only need to take care of the indices in the <strong>DataLoader</strong> and not the <strong>Dataset</strong>.</p>
<pre><code class="lang-python">from torch.utils.data import SubsetRandomSampler


indices=[4, 56, 78, 10, 23]
ds = MyCustomDataset() # No need to give the indices to the dataset
dl = DataLoader(ds, batch_size=8, sampler=SubsetRandomSampler(indices)
</code></pre>
<p>Now every time you iterate through the dataloader, you will only receive indices defined in the list, and they will not repeat. In this example, the list has only 5 elements and the batch is supposed to have 8 elements, this means you will only get the 5 idx from the list.</p>
<p>They will NOT repeat during the batch.</p>
<p>However, they WILL repeat during new epochs. (this is normal behavior)</p>
<p>But if you ABSOLUTELY do not want the values to repeat during new epochs, you can redefine the dataloader in each new epoch, by changing the <code>indices</code> from where it can get items.</p>
<p>Then you can do something like this:</p>
<pre><code class="lang-python">from torch.utils.data import Dataset

class MyCustomDataset(Dataset):
    def __init__(self):
        super().__init__()

    def __getitem__(self, idx):
        print(idx) # Print the index when the dataloader fetches something
        return idx
</code></pre>
<pre><code class="lang-python"># Create the dataset
ds = MyCustomDataset()
</code></pre>
<pre><code class="lang-python">epochs = 3
all_items = random.sample(list(range(15)), 15)

for e in range(epochs):
    print(f"Epoch: {e}")
    dl = torch.utils.data.DataLoader(ds, batch_size=1, sampler=SubsetRandomSampler(all_items[e*5:e*5+5]))
    for batch in dl:
        pass
</code></pre>
<pre><code class="lang-python"># Output:
Epoch: 0
7
13
3
2
6
Epoch: 1
11
14
9
10
0
Epoch: 2
4
5
8
1
12
</code></pre>
<p>But I would <strong>NOT</strong> recommend doing this.</p>
<p>Hope this helps,</p>
<p>If something is not clear, please let me know <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

username: Matt2
cooked: <p>Thanks for your reply. However this solution is not right, as you pass <code>indices</code> to the random sampler (which is nothing but <code>tileIds=[4, 56, 78, 10, 23]</code>). However the <code>idx</code> of <code>__getitem__</code> method are the number of samples not the indices in my opinion. So for example if <code>idx=2</code> I will get 78 from my list. So probabily the olsution is just to pass a list of integers of the size of the dataset like <code>indices=[0,1,2,3,4,5,6,7,8,9,10]</code> if my dataset has just 10 elements.</p>
<p>This Dataloder seems weird but <code>tileIds=[4, 56, 78, 10, 23]</code> represent the id of the images of interest given a unique grid of images stored in memory essentially. Bacause I don’t have the images but a single file which I access with byte shifting.</p>

username: Matias_Vasquez
cooked: <p>The Sampler give back numbers from the list that you give as input.</p>
<p>In this example I passed a list from 0 to 14 (<code>range(15)</code>), that is why the outputs range from 0 to 14.</p>
<aside class="quote no-group" data-username="Matias_Vasquez" data-post="7" data-topic="148346">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matias_vasquez/48/47847_2.png" class="avatar"> Matias_Vasquez:</div>
<blockquote>
<pre><code class="lang-auto">epochs = 3
all_items = random.sample(list(range(15)), 15)

for e in range(epochs):
    print(f"Epoch: {e}")
    dl = torch.utils.data.DataLoader(ds, batch_size=1, sampler=SubsetRandomSampler(all_items[e*5:e*5+5]))
    for batch in dl:
        pass
</code></pre>
</blockquote>
</aside>
<p>If you pass a list with the numbers such as <code>[4, 56, 78, 10, 23]</code>, then you will only get these values back, which will be passed to <code>__getitem__()</code>.</p>
<p>You can try this yourself and see what actually happens.</p>
<p>Also, that example was only to show you how you could change what values the <code>DataLoader</code> can draw values from.</p>
<p>This example was with your data.</p>
<aside class="quote no-group" data-username="Matias_Vasquez" data-post="7" data-topic="148346">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matias_vasquez/48/47847_2.png" class="avatar"> Matias_Vasquez:</div>
<blockquote>
<pre><code class="lang-auto">from torch.utils.data import SubsetRandomSampler


indices=[4, 56, 78, 10, 23]
ds = MyCustomDataset() # No need to give the indices to the dataset
dl = DataLoader(ds, batch_size=8, sampler=SubsetRandomSampler(indices)
</code></pre>
</blockquote>
</aside>
<p>So, the numbers will not repeat.<br>
For each batch you will get numbers from this list.</p>
<p>I will try to explain again in the next post to make it more simple</p>

username: Matias_Vasquez
cooked: <p>With this simple example you can see that the indices are taken from this list I have given the sampler.</p>
<p>In each epoch, there appears to be each value just once.<br>
They do not repeat.</p>
<pre><code class="lang-python">from torch.utils.data import Dataset

class MyCustomDataset(Dataset):
    def __init__(self):
        super().__init__()

    def __getitem__(self, idx):
        return idx
</code></pre>
<pre><code class="lang-python">ds = MyCustomDataset()

all_items = [4, 56, 78, 10, 23, 6, 7, 105, 90, 38, 76, 49, 123, 321, 0]
dl = torch.utils.data.DataLoader(ds, batch_size=5, sampler=SubsetRandomSampler(all_items))

epochs = 3
for e in range(epochs):
    print(f"\nEpoch: {e}")
    for batch in dl:
        print(batch)
</code></pre>
<pre><code class="lang-python"># Output:
Epoch: 0
tensor([  0, 321,  10,  76,   7])
tensor([ 56, 123,   6,   4,  49])
tensor([105,  23,  90,  78,  38])

Epoch: 1
tensor([ 7, 78,  6, 23, 49])
tensor([ 76,   0,  10,  90, 123])
tensor([105,  56,   4, 321,  38])

Epoch: 2
tensor([ 76,  56,  38,  10, 321])
tensor([ 0,  4, 78,  7,  6])
tensor([ 49,  90, 123, 105,  23])
</code></pre>

username: Matt2
cooked: <p>Hi! Thanks I get the point with your first answer, the solution is to keep these indices inside my dataloader as before and creating a list of samples ids from 0 to the length of the dataset in order to obtain the correct idxs for my batches!</p>