Training Multiple Models Simultaneously

username: semperDM
cooked: <p>Hello,</p>
<p>I am trying to train n-models.  Each model has the same structure, same inputs, but is learning a different output.  Since the models are distinct from one another, I would like to parallelize them.  My attempt so far, is putting models on different GPUs.  This is done through:</p>
<pre><code class="lang-auto">def try_gpu(i=0):  #@save
    """Return gpu(i) if exists, otherwise return cpu()."""
    if torch.cuda.device_count() &gt;= i + 1:
        return torch.device(f'cuda:{i}')
    return torch.device('cpu')

model = [LinearNet(net_num).double().to(try_gpu(i%num_gpu) for i in range(net_num)]
</code></pre>
<p>So for example, if I have 6 models and num_gpu = 2, then model[0] is on cuda:0, model[1] is on cuda:1, etc.</p>
<p>I have also made sure to put the inputs onto the different GPUs as well.  However, I’m stuck at actually making this parallel - I’m still just looping over models and running them sequentially.  How would I actually run model[0] and model[1] in parallel?</p>
<p>Thanks for the help!</p>

username: zetyquickly
cooked: <p>Hi <a class="mention" href="/u/semperdm">@semperDM</a></p>
<p>In your case I’d propose to give a try to Hydra parallel multirun (<a href="https://hydra.cc/docs/plugins/joblib_launcher/" class="inline-onebox" rel="noopener nofollow ugc">Joblib Launcher plugin | Hydra</a>) or simply to parameterize your script with command line arguments and launch multiple Python script instances using Bash scripting</p>

username: semperDM
cooked: <p>Hello,</p>
<p>I’m trying to avoid doing multiple script launches, because I’m trying to avoid writing data and then reading it back in.  Currently, I have another script that creates data and then that data is immediately fed into the neural network script.  If I do multiple launches with Bash, then I either have to keep rerunning the data making script, or keep reading in the data, neither of which seems optimal.  As things scale up in terms of data and number of networks being trained, it would become slower and slower to read it in and to recalculate it, hence why I’m trying to avoid that.</p>

username: zetyquickly
cooked: <p>I don’t understand why it would write and read the data back in. If you say that the experiments have the same inputs.</p>
<p>Decouple data creation script from the model, and it won’t affect the training</p>

username: semperDM
cooked: <p>Hello,</p>
<p>If I decouple the data creation script from the model, then I must write all of that data somewhere.  Then, every time I launch a new model, I will have to read in all of this data.  Once our data reaches the millions, this will become rather inefficient, so I’d rather keep them coupled so that the data is just passed to the model script, and then the model script itself is parallelized to run the n models on different GPUs.</p>

username: zetyquickly
cooked: <p>In that case, you probably should look at in memory datasets and Joblib launcher of model training in parallel. Once you created in memory dataset you can instantiate N dataloaders that is pased to a N different models on start of the job</p>

username: ptrblck
cooked: <p>Your workflow can launch the model execution on different devices and they will be executed asynchronously unless you are synchronizing explicitly (via <code>torch.cuda.synchronize()</code>) ir implicitly e.g. via <code>tensor.nonzero()</code>, <code>tensor.item()</code> etc.<br>
If you are not seeing overlapping execution in e.g. Nsight Systems, your CPU (and the overall dispatching) might be too slow compared to the GPU execution time.</p>

username: semperDM
cooked: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>So what I’ve done so far, is that I send model1 to gpu 1 and model2 to gpu 2, and I define</p>
<pre><code class="lang-auto">models = [model1, model2]
</code></pre>
<p>In my training model then, I have</p>
<pre><code class="lang-auto">loss_fn = nn.MSELoss()
optimizer = [optim.Adam(_.parameters(), lr=lr for _ in models]
</code></pre>
<p>After sending the input tensors (X,y) to the appropriate GPUs as well, I then have</p>
<pre><code class="lang-auto">for _ in models:
    _.train()

pred = []
for i in range(len(models)):
    pred.append(model[i](X[i])
</code></pre>
<p>And it is at this point that despite the models being on different GPUs, that I’m worried I’m still doing things in a serial fashion.</p>
<p>Thanks for the help!</p>

username: ptrblck
cooked: <p>The CPU will sequentially launch the model execution, but as long as no model synchronizes internally, both would be executed in parallel on both GPUs.<br>
You could verify it with a profiler as mentioned before. If you do this and don’t see an overlap, your training might be CPU bound and the CPU might not be fast enough in the work scheduling compared to the runtime of the models on the GPU.</p>

username: viciopoli
cooked: <p>Hello everyone, I’m trying to do something similar but on the same GPU. In my case, I believe I need to use the <code>Process</code> from <code>torch.multiprocessing</code>. However, I see that there is not a real boost in terms of performance with respect to the sequential case. I’m doing something like:</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp
from torch.multiprocessing import Process

if __name__ == '__main__':
    mp.set_start_method('spawn')
    ...
    procs = []
    for i in range(N):
        procs.append(Process(target=train, args=(models[i],data[i],)))

    for p in procs:
        p.start()
    
    for p in procs:
        p.join()
</code></pre>
<p>Is there something wrong I’m doing? I’m not sure about how the GPU memory is then shared/used by the processes. I’d like the models to be trained as completely separate entities.<br>
Thanks for your answers!</p>