Can Pytorch Dataset be used for generating instances?

gemsanyou: <p>I have a problem instance class which take <em>batch_size</em> and <em>data_property</em> to randomly generate a number of <em>batch_size</em> problem instances based on the given <em>data_property</em>, as shown below:</p>
<pre><code class="lang-auto">class Problem(object):
  def __init__(self, batch_size, data_property):
    super(Problem, self).__init__()
    self.batch_size = batch_size
    self.fields_A = self.generate_A(batch_size, data_property)
    self.fields_B = self.generate_B(batch_size, data_property)
</code></pre>
<p>How can I make this class / adjust this class so it can be a Pytorch Dataset?</p>
<pre><code class="lang-auto">import torch

class ProblemDataset(torch.utils.data.Dataset):
  'Characterizes a dataset for PyTorch'
  def __init__(self, data_properties):
        'Initialization'
        self.data_properties = data_properties

  def __len__(self):
        # What to do here?

  def __getitem__(self, index):
        # What to do here?
</code></pre>

ptrblck: <p>If I understand your use case correctly, your <code>Problem</code> object yields a batch of samples without using any index, which makes sense since you are randomly generating the data.<br>
In that case you could check, if an <code>IterableDataset</code> could make more sense, which would return an <code>iterator</code> it its <code>__iter__</code> method.</p>

gemsanyou: <p>Yes. And because we are producing one <code>Problem</code> object at a time with <code>IterableDataset</code>, is there any benefit of making this object into <code>IterableDataset</code> ? If I may add to the question, how will the object be scattered if we are using <code>DistributedDataParallel</code>?</p>

ptrblck: <p>You could wrap it into a <code>DataLoader</code> which would use multiple workers etc. once you use the PyTorch <code>data</code> classes.<br>
In <code>DistributedDataParallel</code> you would use a <code>DistributedSampler</code> (probably not possible or necessary in your case as you are randomly generating the data). Besides that each process will create the <code>Dataset</code> or <code>DataLoader</code>. In the latter case each worker will create a copy of the passed <code>Dataset</code>.</p>

gemsanyou: <p>I see, the latter case is using RPC so each process can solve separate problem and synchronize after every epoch.<br>
Bu what if I just generate 1 batch at a time and return the fields (which is <code>torch.Tensor</code>)?</p>
<pre><code class="lang-auto">import torch

class ProblemDataset(torch.utils.data.Dataset):
  'Characterizes a dataset for PyTorch'
  def __init__(self, data_properties, batch_size):
        'Initialization'
        self.data_properties = data_properties
        self.data_property_idx = 0
        self.batch_size = batch_size

  def next_data_property(self):
        self.data_property_idx += 1

  def __len__(self):
       return self.batch_size

  def __getitem__(self, index):
        data_property = self.data_properties[self.data_property_idx]
        problem = Problem(batch_size=1, data_property)
        fields_A = problem.fields_A
        fields_B = problem.fields_B
        return {"fields_A": fields_A, "fields_B": fields_B}
</code></pre>
<p>Is this a proper Dataset? and can this be utilized with <code>DistributedSampler</code> and <code>DistributedDataParallel</code>? If I can use it this way, what should be returned by _<em>len</em>_ ?</p>