How to use BatchSampler with __getitem__ dataset

dvirginz: <p>I am trying to call the <code>\_\_getitem__</code> function of my dataset once per batch due to the cost of each dataset query (on remote).</p>
<pre><code class="lang-auto">class Dataset(Dataset):

    def __init__(self):
       ...

    def __len__(self):
        ...

    def __getitem__(self, batch_idx):  ------&gt; here I get only one index
        return self.wiki_df.loc[batch_idx]


loader = DataLoader(
                dataset=dataset,
                batch_sampler=BatchSampler(
                    SequentialSampler(dataset), batch_size=self.hparams.batch_size, drop_last=False),
                num_workers=self.hparams.num_data_workers,
            )
</code></pre>
<p>This is the current implementation, which does not work.<br>
Is there a way to get the list of indices in the <strong>getitem</strong> function of the dataset</p>

ptrblck: <p>You could disable automatic batching as described <a href="https://pytorch.org/docs/stable/data.html#disable-automatic-batching">here</a> and use a <code>BatchSampler</code>.<br>
Let me know, if that works for you.</p>

dvirginz: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="78788">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>You could disable automatic batching as described <a href="https://pytorch.org/docs/stable/data.html#disable-automatic-batching" rel="noopener nofollow ugc">here</a> and use a  <code>BatchSampler</code> .<br>
Let me know, if that works for you.</p>
</blockquote>
</aside>
<p>Well conceptually yes, But practically I just can’t get my hands around the documentation.<br>
If I set both batchsampler and batchsize to none (in order to turn off automatic batching) how does the system knows my batchsize? how does the <code>__getitem__</code> gets triggered?</p>

ptrblck: <p>Maybe this code snippet might be helpful:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.arange(100).view(100, 1).float()
        
    def __getitem__(self, index):
        x = self.data[index]
        return x
    
    def __len__(self):
        return len(self.data)

dataset = MyDataset()        
sampler = torch.utils.data.sampler.BatchSampler(
    torch.utils.data.sampler.RandomSampler(dataset),
    batch_size=10,
    drop_last=False)

loader = DataLoader(
    dataset,
    sampler=sampler)

for data in loader:
    print(data)
</code></pre>
<p>The <code>index</code> inside <code>__getitem__</code> will contain 10 random indices, which are used to create the batch in:</p>
<pre><code class="lang-python">x = self.data[index]
</code></pre>
<p>You could replace this with your expensive data loading operation.</p>
<p>Let me know, if that helps.</p>

dvirginz: <aside class="quote no-group quote-modified" data-username="ptrblck" data-post="4" data-topic="78788">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote></blockquote>
</aside>
<p>Yes, it does:)<br>
So not intuitive that <code>batchsampler</code> goes into the sampler parameter.<br>
maybe it’s just me</p>

amirhf: <p>Can we do the same thing with <code>batch_sampler</code>?</p>

ptrblck: <p>Could you explain your question and use case a bit?<br>
In my code snippet I’m already using a <code>BatchSampler</code>.</p>