Example for torch.utils.data.IterableDataset

username: Rabeeh_Karimi
cooked: <p>Hi<br>
I have an iterable dataset, then I want to write a dataloader for it, in tutorial, I only find this example:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/data.html#multi-process-data-loading" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/data.html#multi-process-data-loading" target="_blank" rel="noopener nofollow ugc">torch.utils.data — PyTorch 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>which is not clear how to expand it for a real dataset. could you provide me an example where you are given an iterable dataset, and you can write a dataloader for it. thanks</p>

username: Shisho_Sama
cooked: <p>What it is that you want? please provide a bit of information concerning your usecase</p>

username: Rabeeh_Karimi
cooked: <p>Hi<br>
I have a tf.data.Dataset dataset, I’d like to write an iterable Dataloader for pytroch on it, and I’d like to make sure this works with multiple workers, like this example <a href="https://pytorch.org/docs/stable/data.html#multi-process-data-loading" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/data.html#multi-process-data-loading</a> . Also, I need to use it with pytroch lightening so it needs to work with PyTorch lightening as well.</p>

username: Rabeeh_Karimi
cooked: <p>This is example from pytorch tutorial, in this example  in <strong>iter</strong> they write " iter(range(iter_start, iter_end))", but in the real scenario, one input an iterable to this class, and wants to iterate over it, could you tell me please how I can change this example when having an iterable dataset? in a way which allow it to work well with pytorch lightening and in a distributed way on tpu/gpu. thanks</p>
<pre><code class="lang-auto">&gt;&gt;&gt; class MyIterableDataset(torch.utils.data.IterableDataset):
...     def __init__(self, start, end):
...         super(MyIterableDataset).__init__()
...         assert end &gt; start, "this example code only works with end &gt;= start"
...         self.start = start
...         self.end = end
...
...     def __iter__(self):
...         worker_info = torch.utils.data.get_worker_info()
...         if worker_info is None:  # single-process data loading, return the full iterator
...             iter_start = self.start
...             iter_end = self.end
...         else:  # in a worker process
...             # split workload
...             per_worker = int(math.ceil((self.end - self.start) / float(worker_info.num_workers)))
...             worker_id = worker_info.id
...             iter_start = self.start + worker_id * per_worker
...             iter_end = min(iter_start + per_worker, self.end)
...         return iter(range(iter_start, iter_end))
...
&gt;&gt;&gt; # should give same set of data as range(3, 7), i.e., [3, 4, 5, 6].
&gt;&gt;&gt; ds = MyIterableDataset(start=3, end=7)

&gt;&gt;&gt; # Single-process loading
&gt;&gt;&gt; print(list(torch.utils.data.DataLoader(ds, num_workers=0)))
[3, 4, 5, 6]

&gt;&gt;&gt; # Mult-process loading with two worker processes
&gt;&gt;&gt; # Worker 0 fetched [3, 4].  Worker 1 fetched [5, 6].
&gt;&gt;&gt; print(list(torch.utils.data.DataLoader(ds, num_workers=2)))
[3, 5, 4, 6]
</code></pre>

username: ivan-jgr
cooked: <aside class="quote no-group" data-username="Rabeeh_Karimi" data-post="4" data-topic="101175">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rabeeh_karimi/48/30472_2.png" class="avatar"> Rabeeh_Karimi:</div>
<blockquote>
<p>an iterable to this class</p>
</blockquote>
</aside>
<p>Maybe you could use <code>yield</code>?</p>
<pre><code class="lang-auto">class MyIterableDataset(torch.utils.data.IterableDataset):
...     def __init__(self, your_args):
...         super(MyIterableDataset).__init__()
...         # this depends on your dataset, suppose your dataset contains 
...         # images whose path you save in this list
...         self.dataset = # something for load the path of images 
...
...     def __iter__(self):
...         for image_path in self.dataset:
...             sample, label = read_single_example(image_path) # read an individual sample and its label
...             yield sample, label  
</code></pre>

username: Rabeeh_Karimi
cooked: <p>Hi Ivan<br>
Thank you, my question is when we have multiple workers, then based on example given in pytorch tutorial, each worker specify the start and end part of the data, and the data got split between multiple workers. Please have a look at <code>__iter__</code> function, where iter_start and iter_end are specified for the workers. then the worker needs to iterate in this range, in the pytorch example given they wrote <code>iter(range(iter_start, iter_end))</code> because they have a simple list as input, I am not sure how to do it with iterable datasets, thank you.</p>
<p>Here is the link to the tutorial also  <a href="https://pytorch.org/docs/stable/data.html#multi-process-data-loading" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/data.html#multi-process-data-loading</a>, where they explain that each worker needs to access different chunk of data.</p>
<p>Also, in case of pytorch lightening, does the user still need to specify worker_info ? thanks</p>

username: Shisho_Sama
cooked: <p>Why don’t you simply turn your tensorflow dataset to a list (since its a iterable, you should be able to do so in a one liner) and then solve problem from there. That is simply do :</p>
<pre><code class="lang-auto">tf_lst = list(tf_dataset)
</code></pre>
<p>now you have a list which you can simply incorporate into a new pytorch dataset and do as you wish!</p>

username: Rabeeh_Karimi
cooked: <p>Hi<br>
This is a large dataset, it is not possible to load it in memory, so this is an iterable one.</p>

username: Shisho_Sama
cooked: <p>Cant  you use the <code>range()</code> method of the dataset to get around  this issue? give each owrker a chunk of data with different steps maybe?</p>

username: Rabeeh_Karimi
cooked: <p>Hi,<br>
how would you write it in case of iterable datasets?</p>

username: Shisho_Sama
cooked: <p>Your tensorflow dataset has a range method where you can specify the start, stop and step,<br>
your pytorch dataset, requires you to give a chunk of data to each worker. and you can identify each worker using workerinfo if I’m not mistaken!</p>
<p>consider the following example :</p>
<pre><code class="lang-auto">import tensorflow as tf 
dataset = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
dataset = dataset.map(lambda x: x*2)
worker1_data =dataset.range(0, 3, 1)
worker2_data = dataset.range(4, 8, 1)
...
</code></pre>
<p>and based on each worker you can use one of them.<br>
you may also have a look at <code>tf.data.Dataset.shard()</code> method which was created for this very reason.</p>

username: Rabeeh_Karimi
cooked: <p>Hi,<br>
dataset.range does not work on a tf.data.datasets and replaces the dataset with range values.</p>

username: Rabeeh_Karimi
cooked: <p>Hi<br>
Could you provide me with a full example showing how to use iterative datasets with multiple workers? thanks</p>

username: Rabeeh_Karimi
cooked: <p>here is the minimalist example to show what I mean, could you help me please complete it with different workers.</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset, DataLoader
import torch
import tensorflow_datasets as tfds
import tensorflow as tf
import itertools
from itertools import cycle, islice



def get_dummy_dataset():
  inputs = ["input 1",
        "input 2",
        "input 3",
        "input 4"]
  target = ["target 1",
            "target 2",
            "target 3",
            "target 4"]
  features = {"inputs": inputs, "targets": target}
  def my_fn(features):
    ret = {}
    for k, v in features.items():
          ret[f'{k}_plaintext'] = v
    return ret
  dataset = tf.data.Dataset.from_tensor_slices(features)
  dataset = dataset.map(my_fn, num_parallel_calls=tf.data.experimental.AUTOTUNE)
  return dataset


class WMTDataset(torch.utils.data.IterableDataset):
    def __init__(self, batch_size):
        super(WMTDataset).__init__()
        dataset = get_dummy_dataset()
        self.dataset_size = 4
        self.batch_size = batch_size
        self.dataset = self.create_dataset(dataset)

    def __len__(self):
      return self.dataset_size

    def __iter__(self):
      return self.dataset

    def create_dataset(self, dataset):
      dataset = dataset.batch(self.batch_size, drop_remainder=False)
      return itertools.cycle(dataset)



iterable_dataset = WMTDataset(batch_size=2)
loader = DataLoader(iterable_dataset, batch_size=None)
for batch in islice(loader, 2):
    print("#########batch ", batch)
</code></pre>

username: ivan-jgr
cooked: <p>I’m not sure if I understand what you want but you couldn’t generate a csv file with the path of all the samples in your dataset, then create a dataframe and assign 1/n samples to each worker?</p>

username: Etienne_Perot
cooked: <aside class="quote no-group" data-username="Rabeeh_Karimi" data-post="6" data-topic="101175">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rabeeh_karimi/48/30472_2.png" class="avatar"> Rabeeh_Karimi:</div>
<blockquote>
<p>iter_start</p>
</blockquote>
</aside>
<p>here is an example of using iterative dataset with multiple workers, the trick is to use multiple workers: <a href="https://medium.com/speechmatics/how-to-build-a-streaming-dataloader-with-pytorch-a66dd891d9dd" class="inline-onebox" rel="noopener nofollow ugc">How to Build a Streaming DataLoader with PyTorch | by David MacLeod | Speechmatics | Medium</a>  <a href="https://github.com/etienne87/pytorch-streamloader/blob/master/pytorch_iterable.py" rel="noopener nofollow ugc">https://github.com/etienne87/pytorch-streamloader/blob/master/pytorch_iterable.py</a></p>

username: root20
cooked: <p>Hi, did you get the answer of this problem?</p>