How to get DistributedRandomSampler to use a new random order in every epoch?

username: alexrw
cooked: <p>Hi everyone,</p>
<p>I am trying to use <code>DistributedRandomSampler</code> in the C++ frontend. However, it is not behaving as I would expect: it uses the same random order of examples in every epoch (despite <code>set_epoch()</code>). Probably I am doing something wrong, but I was not able to find what it is. The Python equivalent of the C++ code does what I expect. Can you help me find the problem?</p>
<p>Example:</p>
<pre><code class="lang-cpp">#include &lt;torch/torch.h&gt;
#include &lt;stdio.h&gt;
#include &lt;stdlib.h&gt;

int main(int argc, char *argv[]) {
  int N = 8;
  auto inputs = torch::arange(N).view({N, 1});
  auto dataset = torch::data::datasets::TensorDataset({inputs});

  // this works as expected: new random order in every epoch
  // torch::data::samplers::RandomSampler sampler (dataset.size().value());

  // this does not: same random order in every epoch
  torch::data::samplers::DistributedRandomSampler sampler (dataset.size().value(), /*num_replicas=*/2, /*rank=*/0);

  auto loader = torch::data::make_data_loader(
                  dataset,
                  sampler,
                  torch::data::DataLoaderOptions().batch_size(2));


  for (unsigned int epoch=0; epoch!=3; ++epoch) {
    std::cout &lt;&lt; "====== epoch " &lt;&lt; epoch &lt;&lt; "\n";

    sampler.set_epoch(epoch);
    // sampler.reset(); // also tried to manually reset the sampler here, but this did not help

    unsigned long batch_idx = 0;
    for (auto&amp; batch : *loader) {
      std::cout &lt;&lt; "batch " &lt;&lt; batch_idx &lt;&lt; ": ";
      for (auto&amp; example : batch) {
        std::cout &lt;&lt; example.data[0].item&lt;float&gt;() &lt;&lt; " ";
      }
      std::cout &lt;&lt; "\n";
      ++batch_idx;
    }
  }

  return 0;
}
</code></pre>
<p>This produces batches with the same random order in every epoch:</p>
<pre><code class="lang-auto">====== epoch 0
batch 0: 0 2 
batch 1: 1 5 
====== epoch 1
batch 0: 0 2 
batch 1: 1 5 
====== epoch 2
batch 0: 0 2 
batch 1: 1 5 
</code></pre>
<p>I am trying to have different orders in every epoch. I.e., I would expect the output to look like this:</p>
<pre><code class="lang-auto">====== epoch 0
batch 0: 4 7
batch 1: 2 1
====== epoch 1
batch 0: 5 2
batch 1: 7 1
====== epoch 2
batch 0: 0 7
batch 1: 6 1
</code></pre>
<p>The following Python code produces the output that I expect:</p>
<pre><code class="lang-python">import torch

N = 8
inputs = torch.arange(N, dtype=torch.float32).view(N, 1)
dataset = torch.utils.data.TensorDataset(inputs)

sampler = torch.utils.data.DistributedSampler(dataset, num_replicas=2, rank=0, shuffle=True)
loader = torch.utils.data.DataLoader(dataset, batch_size=2, sampler=sampler)

for epoch in range(3):
    sampler.set_epoch(epoch)
    print("====== epoch " + str(epoch))
    for batch_idx, (input,) in enumerate(loader):
        print("batch " + str(batch_idx) + ": " + " ".join([str(int(x)) for x in input.squeeze().tolist()]))
</code></pre>
<p>I am using PyTorch version 1.10.0.</p>
<p>Thanks in advance for any help!</p>
<p>Cheers<br>
Alexander</p>

username: ptrblck
cooked: <p>That’s indeed weird as I would expect <code>set_epoch</code> would behave the same in <code>libtorch</code>.<br>
Would you mind creating an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> so that we could track it, please?</p>

username: alexrw
cooked: <p>Thanks for the quick reply! See <a href="https://github.com/pytorch/pytorch/issues/73141" class="inline-onebox" rel="noopener nofollow ugc">libtorch: `DistributedRandomSampler` uses the same random order in every epoch · Issue #73141 · pytorch/pytorch · GitHub</a>.</p>