DDP on iterable Dataset?

Ravi_Teja2: <p>I’m bit new to using Iterable Datasets. I have been using torchdata to build my dataloaders but they seem to deprecate and even delete this functionality now <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_tear.png?v=12" title=":smiling_face_with_tear:" class="emoji" alt=":smiling_face_with_tear:" loading="lazy" width="20" height="20">. So, I want to build Iterable Dataset but it seems DistributedSampler cannot be used on iterable dataset. Any suggestions on how to use DDP on iterable Datasets?</p>
<p>I’m aware of this large issue: <a href="https://github.com/pytorch/pytorch/pull/26547" class="inline-onebox" rel="noopener nofollow ugc">ChunkDataset API proposal by thiagocrepaldi · Pull Request #26547 · pytorch/pytorch · GitHub</a><br>
But I don’t think this functionality is added yet.<br>
And also this post: <a href="https://discuss.pytorch.org/t/using-iterabledataset-with-distributeddataparallel/92589" class="inline-onebox">Using IterableDataset with DistributedDataParallel</a><br>
Here I don’t find any workable process or examples</p>

602389789: <p>Maybe you can use something like the IterableDatasetShard to wrap your IterableDataset</p><aside class="onebox githubblob" data-onebox-src="https://github.com/huggingface/transformers/blob/main/src/transformers/trainer_pt_utils.py#L798">
  <header class="source">

      <a href="https://github.com/huggingface/transformers/blob/main/src/transformers/trainer_pt_utils.py#L798" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/huggingface/transformers/blob/main/src/transformers/trainer_pt_utils.py#L798" target="_blank" rel="noopener nofollow ugc">huggingface/transformers/blob/main/src/transformers/trainer_pt_utils.py#L798</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="788" style="counter-reset: li-counter 787 ;">
          <li>        for batch_start in range(self.batch_size * self.process_index, self.total_num_samples, self.total_batch_size):</li>
          <li>            result += indices[batch_start : batch_start + self.batch_size]</li>
          <li></li>
          <li>        return iter(result)</li>
          <li></li>
          <li>    def __len__(self):</li>
          <li>        # Each shard only sees a fraction of total_num_samples.</li>
          <li>        return self.total_num_samples // self.num_processes</li>
          <li></li>
          <li></li>
          <li class="selected">class IterableDatasetShard(IterableDataset):</li>
          <li>    """</li>
          <li>    Wraps a PyTorch `IterableDataset` to generate samples for one of the processes only. Instances of this class will</li>
          <li>    always yield a number of samples that is a round multiple of the actual batch size (which is `batch_size x</li>
          <li>    num_processes`). Depending on the value of the `drop_last` attribute, it will either stop the iteration at the</li>
          <li>    first batch that would be too small or loop with indices from the beginning.</li>
          <li></li>
          <li>    On two processes with an iterable dataset yielding of `[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]` with a batch size of</li>
          <li>    2:</li>
          <li></li>
          <li>    - the shard on process 0 will yield `[0, 1, 4, 5, 8, 9]` so will see batches `[0, 1]`, `[4, 5]`, `[8, 9]`</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>