Truncate an iterable dataset for efficient debugging

am_ar: <p>Hello,<br>
Here is the problem. I am looking for a  concise solution to truncate an iterable dataset to an iterable with a set of limited elements (in order to mock this dataset with the real one, and iterate rapidly for debugging).</p>
<p>In the case of “map_style” dataset, we can the method “torch.utils.data.Subset”, to shorten the dataset, which doesn’t seem to work on the other case.</p>
<p>The solution , I thought of was to transform an “Iterable-style” dataset into a “map-style” dataset using  the following code before using the “torch.utils.data.Subset” command :</p>
<p>def convert_to_map_dataset(my_dataset):<br>
class MyDataset(torch.utils.data.Dataset):<br>
def <strong>init</strong>(self):<br>
self.dataset = list(my_dataset)<br>
def <strong>getitem</strong>(self,idx):<br>
return self.dataset[idx]<br>
def <strong>len</strong>(self):<br>
return len(self.dataset)<br>
my_new_dataset = MyDataset()<br>
return my_new_dataset</p>
<p>I was wondering , if there was any more straightforward solution to the problem.</p>
<p>To be more specific, the data is loaded using the following commands:</p>
<p>from torchtext.datasets import Multi30k<br>
train_dataset, val_dataset, test_dataset = Multi30k(language_pair=(“en”, “de”), split=(‘train’, ‘valid’, ‘test’))</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> !</p>

ptrblck: <p>Your idea to process a few samples and to wrap them in a map-style <code>Dataset</code> sounds like a good idea.<br>
However, this line of code:</p>
<pre><code class="lang-python">self.dataset = list(my_dataset)
</code></pre>
<p>would try to create <em>all</em> samples, if I’m not mistaken, and could be quite expensive.<br>
Maybe iterating <code>my_dataset</code> for a new steps would work instead which would allow you to wrap the samples into a <code>TensorDataset</code>.<br>
I’m thinking about something like this:</p>
<pre><code class="lang-python">data = []
target = []
dataset_iter = iter(my_dataset)
for _ in range(nb_samples):
    x, y = next(dataset_iter)
    data.append(x)
    target.append(y)
data = torch.stack(data)
target = torch.stack(target)
self.dataset = TensorDataset(data, target)
</code></pre>

am_ar: <p>Hello,<br>
Thank you for your reply, it’s a good remark indeed!<br>
I replaced my current solution by adding the following code:</p>
<pre><code class="lang-auto">from itertools import islice
m = 5 # number of samples of my new  dataset
dataset = list(islice(my_dataset,m))
</code></pre>
<p>but I am still not satisfied by my solution, which doesn’t seem clean.</p>
<p>Concerning your proposed solution, although it is cleaner, it works for regular dataset but not for the specified one, because the dataset I am working with outputs a pairs of sentences , of variable length.</p>
<p>If you have any other suggestion, to resolve the problem, I’ll be glad to hear about it.<br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <aside class="quote no-group" data-username="am_ar" data-post="3" data-topic="171472">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/am_ar/48/56907_2.png" class="avatar"> am_ar:</div>
<blockquote>
<p>it works for regular dataset but not for the specified one, because the dataset I am working with outputs a pairs of sentences , of variable length.</p>
</blockquote>
</aside>
<p>This would also mean that you are not able to create batches using these samples unless you pad them or return a <code>list</code>, right?<br>
In this case the cleanest approach might be to create a custom <code>IterableDataset</code> using an <code>end</code> counter as shown in e.g. <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset">the example from the docs</a>.</p>