Multiprocessing - torch.multiprocessing.spawn

TommyLeeJones: <p>Hi,<br>
Can somebody answer pls the following questions</p>
<ul>
<li>can I create in a model and custom data iterator inside the main_method</li>
<li>will there be 4 data sets loaded into the RAM / CPU memory?</li>
<li>will each “for batch_data in…” iterate independently</li>
<li>will the model be updated e.g. every independed batch operation. Obviously I don’t want to have four independed models. What’s the process flow in this case… when gradients are updated etc?</li>
</ul>
<p>I have seen this solution but it uses DataLoader (not a custom iterator) and the model is instantiated before the train method is called. - <a href="https://github.com/pytorch/examples/tree/master/mnist_hogwild" rel="nofollow noopener">https://github.com/pytorch/examples/tree/master/mnist_hogwild</a></p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.l1 = nn.Linear(100, 50)
        self.l2 = nn.Linear(50, 2)

    def forward(self, x):
        return self.l2(self.l1(x))

class CustomDataClassIterator():
    def __init__(self):
        self.data = None
        self.batch_size = 10

    def __iter__(self):
        while True:
            yield


def main_method(i, args):
    print(i, datetime.datetime.now())
    model = Net()
    data = CustomDataClassIterator()
    for epoch in args.epoch_n:
        for batch_data in data:
            pass # some stuff

if __name__ == '__main__':
    args = {'test': 10}
    torch.multiprocessing.spawn(fn=main_method, args=(args), nprocs=4)

</code></pre>

mrshenli: <blockquote>
<p>can I create in a model and custom data iterator inside the main_method?</p>
</blockquote>
<p>Given the above example, you created a generator to produce input data? If that is the case, yes, sure you can do that.</p>
<blockquote>
<p>will there be 4 data sets loaded into the RAM / CPU memory?</p>
</blockquote>
<p>I am assumingg <code>pass # some stuff</code> statement will be replaced by actual forward-backward-step functions? If that is the case, then the 4 date sets won’t be loaded into memory at the same time. Instead, each data set will no longer be needed and can be gc-ed at the end of every iteration.</p>
<blockquote>
<p>will each “for batch_data in…” iterate independently</p>
</blockquote>
<p>Yes. It will have it’s own forward pass (building autograd graph), backward pass (generating grads and sync them if necessary), and step function (updating params)</p>
<blockquote>
<p>will the model be updated e.g. every independed batch operation. Obviously I don’t want to have four independed models. What’s the process flow in this case… when gradients are updated etc?</p>
</blockquote>
<p>When you call backward, the gradient will be accumulated into <code>Tensor.grad</code>, and it is up to you regarding when to call <code>Optimizer.step()</code> to apply those grads into the parameter.</p>

mrshenli: <p>I saw you had a pointer to the hogwild training example. Could you please elaborate your use case? Are you looking for distributed data parallel training (like nn.parallel.DistributedDataParallel) or specifically asking for hogwild?</p>