Data missing with torchdata + ddp?

tree33: <p>I’m trying to test my model with torchdata and ddp (2 gpus or sometimes 6 gpus), however, the final number of samples is always less than the actural number here.</p>
<p>My datadir has 17 packed tar files, number 0~15 has 5120 entries, while the last one has only 2730 entries. If I add shardingfilter after filelister, it seems that all of datas in the last tar file will be ignored by the dataloader. I also tried to adding shardingfilter after the map function, however, there are still servel samples missing  .</p>
<p>Is there anyway I can make sure I can have the correct number of files? Do I have to repack my dataset so they all have the same number of entries?<br>
Many thanks in advance!</p>
<pre><code class="lang-auto">    import torchdata
    from torchdata.datapipes.iter import FileLister, FileOpener

    rank, world_size = get_dist_info()
    rootdir = "/data/test/"
    dataset = FileLister(rootdir, "*.tar")

    # if dist: dataset = dataset.sharding_filter()
    dataset = FileOpener(dataset, mode="rb")
    dataset = dataset.load_from_tar(length=length)
    dataset = dataset.webdataset().map(postprocess_func)
    if dist: dataset = dataset.sharding_filter()

    data_loader = DataLoader(
        dataset,
        batch_size=batch_size,
        num_workers=num_workers,
        pin_memory=pin_memory,
        shuffle=shuffle,
        drop_last=False,
        **kwargs)
    
    cnts = 0
    for ind, x in enumerate(data_loader):
        cnts += len(x)
    # cnts doesnt match here
</code></pre>

ejguan: <aside class="quote no-group" data-username="tree33" data-post="1" data-topic="161475">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tree33/48/10609_2.png" class="avatar"> tree33:</div>
<blockquote>
<p>If I add shardingfilter after filelister, it seems that all of datas in the last tar file will be ignored by the dataloader.</p>
</blockquote>
</aside>
<p>For distributed training, the data needs to be balanced across distributed processes. When you shard data over files (with 2 gpus), there will be one rank having more data from the last tar file than the another rank. Technical speaking, I think your script should be hanging at the end of epoch caused by the unbalanced data.</p>
<aside class="quote no-group" data-username="tree33" data-post="1" data-topic="161475">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tree33/48/10609_2.png" class="avatar"> tree33:</div>
<blockquote>
<p>I also tried to adding shardingfilter after the map function, however, there are still servel samples missing .</p>
</blockquote>
</aside>
<p>It depends on your <code>batch_size</code>. The total amount of data is  <code>5120 * 16 + 2730 = 84650</code>, then the total batches would be <code>84650 / batch_size</code>. If this number can not be divided by the number of GPUs without remainders, then a few samples will be ignored. And, your distributed process should be hanging as well.</p>
<p>BTW, to prevent hanging problem, we do provide a DataPipe operation called <code>fullsync</code> in this PR <a href="https://github.com/pytorch/data/pull/713" class="inline-onebox" rel="noopener nofollow ugc">Implement FullSyncIterDataPipe by ejguan · Pull Request #713 · pytorch/data · GitHub</a>. You can attach it at the end of the pipeline. To get it, you need to use torchdata nightly release via the following two options:</p>
<ul>
<li><code>pip3 install --pre torch torchdata --extra-index-url https://download.pytorch.org/whl/nightly/cpu</code></li>
<li><code>conda install pytorch torchdata -c pytorch-nightly</code></li>
</ul>