IterableDataset how to implemtent __iter__

DoubleRedX: <p>Hey guy,code like this</p>
<pre><code class="lang-auto">class RSNA_volume(IterableDataset):
    def __init__(self, stage2_train=None, instance_dir=None):
        super(RSNA_volume, self).__init__()
        # self.instances = glob.glob(str(Path(stage2_train) / "**" / "*.nii.gz"))
        self.instances = stage2_train
        self.instance_label = pd.read_csv(filepath_or_buffer=instance_dir, usecols=['InstanceID', 'Label'])
        self.number_of_instances = len(self.instances)
        LOGGER.info(f"Instances number in RSNA: {self.number_of_instances}")
        LOGGER.info(f"Instances label shape: {self.instance_label.shape}")

    def __iter__(self):
        self.count = 0
        return self

    def __next__(self):
        if self.count == self.number_of_instances:
            raise StopIteration
        instance = self.instances[self.count]
        data = tio.ScalarImage(instance)

        instance_name = Path(instance).stem.split('.')[0]
        label = self.instance_label[self.instance_label['InstanceID'] == instance_name]['Label'].tolist()
        label = eval(label[0])
        # print(label)
        self.count += 1

        return data, torch.tensor(data=label)

    def __len__(self):
        return self.number_of_instances
</code></pre>
<p>and this is the torch turtorial example</p>
<pre><code class="lang-auto">&gt;&gt;&gt; class MyIterableDataset(torch.utils.data.IterableDataset):
...     def __init__(self, start, end):
...         super(MyIterableDataset).__init__()
...         assert end &gt; start, "this example code only works with end &gt;= start"
...         self.start = start
...         self.end = end
...
...     def __iter__(self):
...         worker_info = torch.utils.data.get_worker_info()
...         if worker_info is None:  # single-process data loading, return the full iterator
...             iter_start = self.start
...             iter_end = self.end
...         else:  # in a worker process
...             # split workload
...             per_worker = int(math.ceil((self.end - self.start) / float(worker_info.num_workers)))
...             worker_id = worker_info.id
...             iter_start = self.start + worker_id * per_worker
...             iter_end = min(iter_start + per_worker, self.end)
...         return iter(range(iter_start, iter_end))
</code></pre>
<p>My question is how to change my code to be like the example.<br>
For the reason I change it,cause I think my <strong>next</strong> is redundant,i check the code of dataloader,and find that <strong>next</strong> (iterator) has been implemented,so i think maye the code in <strong>next</strong> should be in <strong>iter</strong>,but i don’t know how to change it.I don;t know how to use yield. <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_tear.png?v=12" title=":smiling_face_with_tear:" class="emoji" alt=":smiling_face_with_tear:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_tear.png?v=12" title=":smiling_face_with_tear:" class="emoji" alt=":smiling_face_with_tear:" loading="lazy" width="20" height="20"><br>
Thanks.</p>

Matias_Vasquez: <p>Hi,</p>
<p>May I ask why you <strong>need</strong> an <code>IterableDataset</code>. I think this should work fine with a normal <code>Dataset</code> object since you are just reading a <code>csv</code> file and then reading images when you have to give the next item.</p>
<p>The tutorial in <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#" rel="noopener nofollow ugc">this link</a> does the same thing, reading a file and then giving an image that corresponds to some path specified in the <code>csv</code> file.</p>
<p>The <code>DataLoader</code> will then take care to iterate through the dataset.</p>
<p>This is the code for the <code>Dataset</code> from the tutorial</p>
<pre><code class="lang-python">class FaceLandmarksDataset(Dataset):
    def __init__(self, csv_file, root_dir, transform=None):
        self.landmarks_frame = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.landmarks_frame)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        img_name = os.path.join(self.root_dir,
                                self.landmarks_frame.iloc[idx, 0])
        image = io.imread(img_name)
        landmarks = self.landmarks_frame.iloc[idx, 1:]
        landmarks = np.array([landmarks])
        landmarks = landmarks.astype('float').reshape(-1, 2)
        sample = {'image': image, 'landmarks': landmarks}

        if self.transform:
            sample = self.transform(sample)

        return sample
</code></pre>
<p>This would be then for the <code>DataLoader</code></p>
<pre><code class="lang-python">dataloader = DataLoader(transformed_dataset, batch_size=4,
                        shuffle=True, num_workers=0)

for i_batch, sample_batched in enumerate(dataloader):
    print(i_batch, sample_batched['image'].size(),
          sample_batched['landmarks'].size())

    # observe 4th batch and stop.
    if i_batch == 3:
        plt.figure()
        show_landmarks_batch(sample_batched)
        plt.axis('off')
        plt.ioff()
        plt.show()
        break
</code></pre>

DoubleRedX: <p>Thanks for your reply.<br>
I know how to custome dataset in map-style.I’d like to implement it in iterable-style. <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20"><br>
And i don’t know to implement it.<img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20"></p>

Matias_Vasquez: <p>Oh ok,</p>
<p>then here is a blog with multiple examples going step by step into how to build one. This might be helpful.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://medium.com/speechmatics/how-to-build-a-streaming-dataloader-with-pytorch-a66dd891d9dd">
  <header class="source">
      <img src="https://miro.medium.com/1*m-R_BkNf1Qjr1YbyOIJY2w.png" class="site-icon" width="32" height="32">

      <a href="https://medium.com/speechmatics/how-to-build-a-streaming-dataloader-with-pytorch-a66dd891d9dd" target="_blank" rel="noopener nofollow ugc" title="10:31AM - 31 January 2022">Medium – 31 Jan 22</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/388;"><img src="https://miro.medium.com/max/1200/1*MZxE59ZXQfktTDzA66leUg.png" class="thumbnail" width="690" height="388"></div>

<h3><a href="https://medium.com/speechmatics/how-to-build-a-streaming-dataloader-with-pytorch-a66dd891d9dd" target="_blank" rel="noopener nofollow ugc">How to Build a Streaming DataLoader with PyTorch</a></h3>

  <p>Learn how the new PyTorch 1.2 dataset class `torch.utils.data.IterableDataset` can be used to implement a parallel streaming DataLoader.</p>

  <p>
    <span class="label1">Reading time: 12 min read</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>