Dataparallel, training with multiple gpus

pytorchnewbie: <p>Hi! So I do this:</p>
<p>net = torch.nn.DataParallel(AutoEncoder(config), device_ids = list_of_gpus)</p>
<p>But then when I print:  for elem in net.parameters(): print(elem.device)</p>
<p>I get a list of “cpu” for most of hte parameters, and at the end I get cuda:0. Why is this the case? Thanks</p>

ptrblck: <p>Could you post your model definition, so that we could have a look at it, please?</p>

pytorchnewbie: <p>I’m using this open-source code: <a href="https://github.com/sweetcocoa/ddsp-pytorch/blob/master/train/train.py" rel="noopener nofollow ugc">https://github.com/sweetcocoa/ddsp-pytorch/blob/master/train/train.py</a></p>
<p>Except I don’t want to use just one gpu, so I modified up until line 78 in the file linked above to be:</p>
<p>config = setup(default_config="…/configs/violin.yaml")</p>
<p>print(OmegaConf.create(config.<strong>dict</strong>).pretty())</p>
<p>set_seeds(config.seed)</p>
<p>Trainer.set_experiment_name(config.experiment_name)</p>
<p>list_of_gpus = [int(s) for s in config.gpu_list.split(’,’)]</p>
<p>torch.cuda.set_device(0)</p>
<p>net = AutoEncoder(config)</p>
<p>net.to(0)</p>
<p>net = torch.nn.DataParallel(net, device_ids = list_of_gpus)</p>

ptrblck: <p>Thanks for the link. Which parameters are still on the CPU after pushing the model to the GPU?</p>

pytorchnewbie: <p>Whats the best way for me to figure that out?</p>

ptrblck: <p>You can print then via:</p>
<pre><code class="lang-python">for name, param in model.named_parameters():
    print(name, param.device)
</code></pre>

pytorchnewbie: <p>It seems like everything is on cuda:0, but the error says something is on cuda:1 :</p>
<p>Printed from code above:</p>
<pre><code class="lang-auto">module.decoder.mlp_f0.mlp_layer1.0.weight cuda:0
module.decoder.mlp_f0.mlp_layer1.0.bias cuda:0
module.decoder.mlp_f0.mlp_layer1.1.weight cuda:0
module.decoder.mlp_f0.mlp_layer1.1.bias cuda:0
module.decoder.mlp_f0.mlp_layer2.0.weight cuda:0
module.decoder.mlp_f0.mlp_layer2.0.bias cuda:0
module.decoder.mlp_f0.mlp_layer2.1.weight cuda:0
module.decoder.mlp_f0.mlp_layer2.1.bias cuda:0
module.decoder.mlp_f0.mlp_layer3.0.weight cuda:0
module.decoder.mlp_f0.mlp_layer3.0.bias cuda:0
module.decoder.mlp_f0.mlp_layer3.1.weight cuda:0
module.decoder.mlp_f0.mlp_layer3.1.bias cuda:0
module.decoder.mlp_loudness.mlp_layer1.0.weight cuda:0
module.decoder.mlp_loudness.mlp_layer1.0.bias cuda:0
module.decoder.mlp_loudness.mlp_layer1.1.weight cuda:0
module.decoder.mlp_loudness.mlp_layer1.1.bias cuda:0
module.decoder.mlp_loudness.mlp_layer2.0.weight cuda:0
module.decoder.mlp_loudness.mlp_layer2.0.bias cuda:0
module.decoder.mlp_loudness.mlp_layer2.1.weight cuda:0
module.decoder.mlp_loudness.mlp_layer2.1.bias cuda:0
module.decoder.mlp_loudness.mlp_layer3.0.weight cuda:0
module.decoder.mlp_loudness.mlp_layer3.0.bias cuda:0
module.decoder.mlp_loudness.mlp_layer3.1.weight cuda:0
module.decoder.mlp_loudness.mlp_layer3.1.bias cuda:0
module.decoder.gru.weight_ih_l0 cuda:0
module.decoder.gru.weight_hh_l0 cuda:0
module.decoder.gru.bias_ih_l0 cuda:0
module.decoder.gru.bias_hh_l0 cuda:0
module.decoder.mlp_gru.mlp_layer1.0.weight cuda:0
module.decoder.mlp_gru.mlp_layer1.0.bias cuda:0
module.decoder.mlp_gru.mlp_layer1.1.weight cuda:0
module.decoder.mlp_gru.mlp_layer1.1.bias cuda:0
module.decoder.mlp_gru.mlp_layer2.0.weight cuda:0
module.decoder.mlp_gru.mlp_layer2.0.bias cuda:0
module.decoder.mlp_gru.mlp_layer2.1.weight cuda:0
module.decoder.mlp_gru.mlp_layer2.1.bias cuda:0
module.decoder.mlp_gru.mlp_layer3.0.weight cuda:0
module.decoder.mlp_gru.mlp_layer3.0.bias cuda:0
module.decoder.mlp_gru.mlp_layer3.1.weight cuda:0
module.decoder.mlp_gru.mlp_layer3.1.bias cuda:0
module.decoder.dense_harmonic.weight cuda:0
module.decoder.dense_harmonic.bias cuda:0
module.decoder.dense_filter.weight cuda:0
module.decoder.dense_filter.bias cuda:0
module.reverb.fir cuda:0
module.reverb.drywet cuda:0
module.reverb.decay cuda:0
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto"> File "/ddsp-pytorch/train/trainer/trainer.py", line 246, in train
    phase, num_steps=num_steps
  File "/ddsp-pytorch/train/trainer/trainer.py", line 377, in _train
    results = self._step(phase=phase, iterator=step_iterator)
  File "/ddsp-pytorch/train/trainer/trainer.py", line 336, in _step
    outputs = self.net(batch_dict)
  File ".local/share/virtualenvs/ddsp-pytorch-BwyVRuzu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/.local/share/virtualenvs/ddsp-pytorch-BwyVRuzu/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 161, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/.local/share/virtualenvs/ddsp-pytorch-BwyVRuzu/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 171, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/.local/share/virtualenvs/ddsp-pytorch-BwyVRuzu/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/.local/share/virtualenvs/ddsp-pytorch-BwyVRuzu/lib/python3.7/site-packages/torch/_utils.py", line 428, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 1 on device 1.
Original Traceback (most recent call last):
  File "/.local/share/virtualenvs/ddsp-pytorch-BwyVRuzu/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/.local/share/virtualenvs/ddsp-pytorch-BwyVRuzu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/ddsp-pytorch/train/network/autoencoder/autoencoder.py", line 64, in forward
    batch = self.encoder(batch)
  File /.local/share/virtualenvs/ddsp-pytorch-BwyVRuzu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "ddsp-pytorch/train/network/autoencoder/encoder.py", line 101, in forward
    batch["loudness"] = self.loudness_extractor(batch)
  File "/.local/share/virtualenvs/ddsp-pytorch-BwyVRuzu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/ddsp-pytorch/train/../components/loudness_extractor.py", line 79, in forward
    sliced_windowed_signal = sliced_signal * self.smoothing_window
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!
</code></pre>

ptrblck: <p>Are you using any <code>.cuda()</code> or <code>to('cuda')</code> operations inside the model’s <code>forward</code> method?<br>
If so, you would have to remove them and use e.g. the <code>.device</code> attribute of an internal parameter or the input tensor.<br>
<code>nn.DataParallel</code> will create model replicas on all devices, which would yield a device mismatch if you create tensors in the <code>forward</code> on the default device.</p>

pytorchnewbie: <p>I don’t see .cuda() or (‘cuda’) operations anywhere in the source code… Do you? In trainer.py i do see this though:</p>
<p>def get_model_device(net):<br>
device = torch.device(“cuda:0”)<br>
<span class="hashtag">#for</span> param in net.parameters():<br>
#    device = param.device<br>
#    break<br>
return device</p>

ptrblck: <p>I haven’t checked the complete repository, but guess that the returned device might be used somewhere, where it shouldn’t. Could you check, where this method is called and if the <code>device</code> is used to push some activations or parameters to <code>cuda:0</code>?</p>

pytorchnewbie: <p>So on line 163 in <a href="https://github.com/sweetcocoa/ddsp-pytorch/blob/5d0247e8b1c108662f7ac2507625103471d27e49/train/trainer/trainer.py" rel="noopener nofollow ugc">https://github.com/sweetcocoa/ddsp-pytorch/blob/5d0247e8b1c108662f7ac2507625103471d27e49/train/trainer/trainer.py</a> they do that map to self.device, and self.device is earlier initialized to the output of get_model_device function above. Firstly, I’m not sure why the model being set to cuda:0 is a problem since if i remember correctly the model actually stays on the default device no? Even if this is wrong, i tried removing the map_location argument and got the same error…</p>
<p>However! I bet line 316-327 in the same link above in trainer.py really matters. They send the <em>batch</em> to the device, and I’m not sure what I shoudl replace to self.device with</p>

ptrblck: <p>Sending the <code>nn.DataParallel</code> model and the data to the default device is the right approach.<br>
If you haven’t found any device calls inside the model (e.g. in the <code>forward</code> method), could you try to narrow down the code and post an executable code snippet, so that we could debug it?</p>

pytorchnewbie: <p>I’m pretty sure the _step() function is the forward() function though in this case, and they do send the input batch to the default device. Though you say thats okay, but so since step() is forward() I’m not quite sure what you mean.</p>

pytorchnewbie: <p>In the source code I linked, the actual net gets sent to cuda in the train.py file, but as I mentioned before, I removed that and replaced it with:</p>
<pre><code class="lang-auto">torch.cuda.set_device(0)
net = AutoEncoder(config)
net.to(0)
net = torch.nn.DataParallel(net, device_ids = list_of_gpus)
</code></pre>

pytorchnewbie: <p>The error first comes In “loudness_extractor.py”, which containers the first part of the encoder,  so I set a pdb.set_trace() in there:</p>
<pre><code>   ```
</code></pre>
<p>“”"<br>
def forward(self, z):<br>
Compute A-weighted Loudness Extraction<br>
Input:<br>
z[‘audio’] : batch of time-domain signals<br>
Output:<br>
output_signal : batch of reverberated signals<br>
“”"</p>
<pre><code>    input_signal = z['audio']
    paded_input_signal = nn.functional.pad(input_signal, (self.frame_length * 2, self.frame_length * 2))
    sliced_signal = paded_input_signal.unfold(1, self.n_fft, self.frame_length)
    pdb.set_trace()
    sliced_windowed_signal = sliced_signal * self.smoothing_window
</code></pre>
<pre><code class="lang-auto">I set the pdb.set_trace() and did self.smoothing_window.cuda and got: device(type='cuda', index=0) and then i also did sliced_signal.get_device() and got 0.</code></pre>

pytorchnewbie: <p>Hi! Just wondering if you have any next steps I can take?</p>

ptrblck: <p>If you get stuck, you could still try to remove all unnecessary code and post a minimal code snippet which would reproduce this issue so that we could debug it.</p>