nn.DataParallel: TypeError: expected sequence object with len >= 0 or a single integer

pytorching: <p>In my <code>forward</code> function:</p>
<pre><code class="lang-auto">def __call__(self, train=True):
    if train:
        predicted = self.forward(...)
        loss = ....
        return loss # return a single value that's fine
        # loss.size() = the number of my GPUs.
    else:
        predicted = self.forward(...)
        return predicted # expected sequence object with len &gt;= 0 or a single integer
        # In validation step I want to return the whole predict labels for other purpose
        # predicted with shape [16, 1] on each device and I have 4 GPU
</code></pre>
<p>My code works before <code>model=nn.DataParallel(model)</code>.</p>

albanD: <p>Hi,</p>
<p>This is hard to say without more context.<br>
Can you share the stack trace for your functoin as well as where this call function is defined?</p>

pytorching: <p>The code is here. But the code is not quite organized. It has other problems prevent me from using <code>nn.DataParallel</code> only this one I cannot solve.<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/lifanchen-simm/transformerCPI/blob/f9301880740975ddc1d56ce19f9eb52a6ad75933/Kinase/model.py#L315" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/lifanchen-simm/transformerCPI/blob/f9301880740975ddc1d56ce19f9eb52a6ad75933/Kinase/model.py#L315" target="_blank" rel="nofollow noopener">lifanchen-simm/transformerCPI/blob/f9301880740975ddc1d56ce19f9eb52a6ad75933/Kinase/model.py#L315</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="305" style="counter-reset: li-counter 304 ;">
<li>    # protein = torch.unsqueeze(protein, dim=0)</li>
<li>    # protein =[ batch size=1,protein len, protein_dim]</li>
<li>    enc_src = self.encoder(protein)</li>
<li>    # enc_src = [batch size, protein len, hid dim]</li>
<li>
</li><li>    out = self.decoder(compound, enc_src, compound_mask, protein_mask)</li>
<li>    # out = [batch size, 2]</li>
<li>    # out = torch.squeeze(out, dim=0)</li>
<li>    return out</li>
<li>
</li><li class="selected">def __call__(self, data, train=True):</li>
<li>
</li><li>    compound, adj, protein, correct_interaction ,atom_num,protein_num = data</li>
<li>    # compound = compound.to(self.device)</li>
<li>    # adj = adj.to(self.device)</li>
<li>    # protein = protein.to(self.device)</li>
<li>    # correct_interaction = correct_interaction.to(self.device)</li>
<li>    #scale = torch.tensor([1.0,4.0], device=self.device)</li>
<li>    Loss = nn.CrossEntropyLoss()</li>
<li>
</li><li>    if train:</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


albanD: <p>The first issue is that you should <em>never</em> redefine the <code>__call__</code> method on a Module. Just the forward. This is going to provent it from working nicely with other parts of pytorch.</p>
<p>More generally, the error most likely refers to the creation of the DataParallel where the device argument does not have the right type.</p>

pytorching: <p>I’m trying to remove the <code>__call__()</code>. But I don’t understand which part of the <code>device</code> is not right? To be honest, I don’t know which <code>device</code> should <code>input.to(device)</code> and <code>model.to(device)</code> use, when using <code>nn.DataParallel</code>. I just use <code>device_ids[0]</code>.<br>
You mean the bug is here? Thank you very much!<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/lifanchen-simm/transformerCPI/blob/f9301880740975ddc1d56ce19f9eb52a6ad75933/Kinase/main.py#L68" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/lifanchen-simm/transformerCPI/blob/f9301880740975ddc1d56ce19f9eb52a6ad75933/Kinase/main.py#L68" target="_blank" rel="nofollow noopener">lifanchen-simm/transformerCPI/blob/f9301880740975ddc1d56ce19f9eb52a6ad75933/Kinase/main.py#L68</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="58" style="counter-reset: li-counter 57 ;">
<li>batch = 64</li>
<li>lr = 1e-4</li>
<li>weight_decay = 1e-4</li>
<li>iteration = 300</li>
<li>kernel_size = 9</li>
<li>
</li><li>encoder = Encoder(protein_dim, hid_dim, n_layers, kernel_size, dropout, device)</li>
<li>decoder = Decoder(atom_dim, hid_dim, n_layers, n_heads, pf_dim, DecoderLayer, SelfAttention, PositionwiseFeedforward, dropout, device)</li>
<li>model = Predictor(encoder, decoder, device)</li>
<li># model.load_state_dict(torch.load("output/model/lr=0.001,dropout=0.1,lr_decay=0.5"))</li>
<li class="selected">model.to(device)</li>
<li>trainer = Trainer(model, lr, weight_decay, batch)</li>
<li>tester = Tester(model)</li>
<li>
</li><li>"""Output files."""</li>
<li>file_AUCs = 'output/result/AUCs--lr=1e-4,dropout=0.1,weight_decay=1e-4,kernel=9,n_layer=3,batch=64,balance,lookaheadradam'+ '.txt'</li>
<li>file_model = 'output/model/' + 'lr=1e-4,dropout=0.1,weight_decay=1e-4,kernel=9,n_layer=3,batch=64,balance,lookaheadradam'</li>
<li>AUC = ('Epoch\tTime(sec)\tLoss_train\tAUC_dev\tPRC_dev')</li>
<li>with open(file_AUCs, 'w') as f:</li>
<li>    f.write(AUC + '\n')</li>
<li>
</li></ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


albanD: <p>As mentioned in the DataParallel <a href="https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html#torch.nn.DataParallel">doc</a>: " The parallelized <code>module</code> must have its parameters and buffers on <code>device_ids[0]</code> before running this <a href="https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html#torch.nn.DataParallel"> <code>DataParallel</code> </a> module."</p>
<p>I can’t find any reference to DataParallel in the repo so not sure where you do that. By I was talking about the place where you wrap your module in DataParallel.</p>

pytorching: <p>Sorry, just the line before referenced.</p>
<pre><code class="lang-auto">model = Predictor(encoder, decoder, device)
# model.load_state_dict(torch.load("output/model/lr=0.001,dropout=0.1,lr_decay=0.5"))
model = nn.DataParallel(model, device_ids=[0,1,2,3])  # I add the code here
model.to(device)
</code></pre>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/lifanchen-simm/transformerCPI/blob/f9301880740975ddc1d56ce19f9eb52a6ad75933/Kinase/main.py#L68" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/lifanchen-simm/transformerCPI/blob/f9301880740975ddc1d56ce19f9eb52a6ad75933/Kinase/main.py#L68" target="_blank" rel="nofollow noopener">lifanchen-simm/transformerCPI/blob/f9301880740975ddc1d56ce19f9eb52a6ad75933/Kinase/main.py#L68</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="58" style="counter-reset: li-counter 57 ;">
<li>batch = 64</li>
<li>lr = 1e-4</li>
<li>weight_decay = 1e-4</li>
<li>iteration = 300</li>
<li>kernel_size = 9</li>
<li>
</li><li>encoder = Encoder(protein_dim, hid_dim, n_layers, kernel_size, dropout, device)</li>
<li>decoder = Decoder(atom_dim, hid_dim, n_layers, n_heads, pf_dim, DecoderLayer, SelfAttention, PositionwiseFeedforward, dropout, device)</li>
<li>model = Predictor(encoder, decoder, device)</li>
<li># model.load_state_dict(torch.load("output/model/lr=0.001,dropout=0.1,lr_decay=0.5"))</li>
<li class="selected">model.to(device)</li>
<li>trainer = Trainer(model, lr, weight_decay, batch)</li>
<li>tester = Tester(model)</li>
<li>
</li><li>"""Output files."""</li>
<li>file_AUCs = 'output/result/AUCs--lr=1e-4,dropout=0.1,weight_decay=1e-4,kernel=9,n_layer=3,batch=64,balance,lookaheadradam'+ '.txt'</li>
<li>file_model = 'output/model/' + 'lr=1e-4,dropout=0.1,weight_decay=1e-4,kernel=9,n_layer=3,batch=64,balance,lookaheadradam'</li>
<li>AUC = ('Epoch\tTime(sec)\tLoss_train\tAUC_dev\tPRC_dev')</li>
<li>with open(file_AUCs, 'w') as f:</li>
<li>    f.write(AUC + '\n')</li>
<li>
</li></ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


pytorching: <p>Another question,(sorry I’m new to pytorch)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d276a4adbca6e9b373b579e675d76a9f2295121.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5d276a4adbca6e9b373b579e675d76a9f2295121" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d276a4adbca6e9b373b579e675d76a9f2295121_2_690x225.jpeg" alt="image" data-base62-sha1="di4QMVD15gtZDplbD4aF3Be0MVj" width="690" height="225" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d276a4adbca6e9b373b579e675d76a9f2295121_2_690x225.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d276a4adbca6e9b373b579e675d76a9f2295121_2_1035x337.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d276a4adbca6e9b373b579e675d76a9f2295121.jpeg 2x" data-dominant-color="DBE3EA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1080×353 130 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
By this image in some blog about <code>nn.DataParallel</code>.<br>
The first step in Backward(Compute loss gradient on GPU-1) results in imbalanced GPU usage.<br>
Is that means in <code>DataParallel</code> <code>loss.backward()</code> only happens in GPU-1 not other GPUs, but <code>optimizer.step(),optimizer.zero_grad()</code> are parallel(step 2,3,4 in backward)?<br>
Thank you very much.</p>

albanD: <p>What DataParallel does is more version 3 of this image: split the input on each GPUs and run on each of them independently. Then accumulate.<br>
Note that the backward will run on the same device as the forward. whatever the device of the Tensor on which you call .backward().</p>

pytorching: <p>Then where does the imbalanced GPU usage come from?<br>
You means <code>loss.backward()</code> is also parallel right?<br>
I’m a little confused.</p>

albanD: <p>It depends if the loss is inside the DataParallel or not.<br>
If it is, then there won’t be any imbalance.<br>
If it is outside and just computed on one GPU, then this GPU will do a bit more work indeed.</p>

pytorching: <blockquote>
<p>It depends if the loss is inside the DataParallel or not.</p>
</blockquote>
<p>In <code>DataParallel</code> you mean inside the <code>forward</code> function? But most time the <code>forward</code> function won’t contain <code>loss</code> computation right ?<br>
I’m also confused about <code>imbalance</code> come from <code>loss.backward()</code> or from <code>loss=criterion(ture, pred)</code> ?<br>
Thank you for your patience!</p>

albanD: <p>The DataParallel takes a Module as input so it can contain anything you want <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
And yes what is executed is what is in the forward function of your Module.</p>
<p>The imbalance won’t come from the <code>loss.backward()</code> because it runs at the same place as the forward. So if the forward is balanced, the backward will be as well.</p>

pytorching: <p>Another weird problem in <code>nn.DataParallel</code><br>
in my <code>main.py</code> I put the model to device</p>
<pre><code class="lang-auto">encoder = Encoder(protein_dim, hid_dim, n_layers, kernel_size, dropout)
decoder = Decoder(atom_dim, hid_dim, n_layers, n_heads, pf_dim, DecoderLayer, SelfAttention,
                  PositionwiseFeedforward, dropout)
model = Predictor(encoder, decoder)
# model.load_state_dict(torch.load("output/model/lr=0.001,dropout=0.1,lr_decay=0.5"))
model = nn.DataParallel(model)
model.to(device)

trainer = Trainer(model, lr, weight_decay, scaler)
tester = Tester(model)
loss_train = trainer.train(train_dl, device=device)  # This line throw errors
</code></pre>
<p>But I got the following error</p>
<pre><code class="lang-auto">assert all(map(lambda i: i.is_cuda, inputs))
AssertionError
</code></pre>
<p>I have test all <code>model.parameters()</code> and <code>inputs</code> in <code>train()</code>:</p>
<pre><code class="lang-auto">def train(self, dataloader, device):
    self.model.train()

    if self.scaler is None:
        for i, data_pack in enumerate(dataloader):
            data_pack = to_cuda(data_pack, device=device)

            assert (all(map(lambda i: i.is_cuda, self.model.parameters())))
            assert (all(map(lambda i: i.is_cuda, data_pack)))
            loss, _, _ = self.model(data_pack)  # This line throw errors

            self.optimizer.zero_grad()
            loss.sum().backward()
            self.optimizer.step()
</code></pre>
<p>The results are all <code>True</code>. But I still get this error in the third line <code>loss, _, _ = self.model(data_pack)</code><br>
What happened?<br>
This is my forward function:</p>
<pre><code class="lang-auto">def forward(self, data):
    compound, adj, protein, correct_interaction, atom_num, protein_num = data
    # compound = [batch,atom_num, atom_dim]
    # adj = [batch,atom_num, atom_num]
    # protein = [batch,protein len, 100]

    compound_max_len = compound.shape[1]
    protein_max_len = protein.shape[1]
    compound_mask, protein_mask = self.make_masks(atom_num, protein_num, compound_max_len, protein_max_len)
    compound = self.gcn(compound, adj)
    # compound = torch.unsqueeze(compound, dim=0)
    # compound = [batch size=1 ,atom_num, atom_dim]

    # protein = torch.unsqueeze(protein, dim=0)
    # protein =[ batch size=1,protein len, protein_dim]
    enc_src = self.encoder(protein)
    # enc_src = [batch size, protein len, hid dim]

    predicted_interaction = self.decoder(compound, enc_src, compound_mask, protein_mask)
    # out = [batch size, 2]
    # out = torch.squeeze(out, dim=0)
    loss = self.Loss(predicted_interaction, correct_interaction.view(-1, 1))
    return torch.unsqueeze(loss, 0), predicted_interaction.cpu().detach().view(-1, 1), correct_interaction.cpu().detach().view(-1, 1)
</code></pre>
<p>Thank you very much !!!</p>

albanD: <p>From the DataParallel doc, you should send your model to the device <strong>before</strong> wrapping it in DataParallel!</p>

pytorching: <p>You mean</p>
<pre><code class="lang-auto">model = nn.DataParallel(model)  # The order is wrong?
model.to(device)
</code></pre>
<pre><code class="lang-auto">model.to(device)  # This is right?
model = nn.DataParallel(model)
</code></pre>
<p>But in the doc<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/0/90f420258da03fe8f4410974379798abcd1dd39c.png" data-download-href="https://discuss.pytorch.org/uploads/default/90f420258da03fe8f4410974379798abcd1dd39c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/90f420258da03fe8f4410974379798abcd1dd39c_2_690x264.png" alt="image" data-base62-sha1="kGjN7SAgfH1zbl9jkMw2SXuCH8M" width="690" height="264" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/90f420258da03fe8f4410974379798abcd1dd39c_2_690x264.png, https://discuss.pytorch.org/uploads/default/original/3X/9/0/90f420258da03fe8f4410974379798abcd1dd39c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/0/90f420258da03fe8f4410974379798abcd1dd39c.png 2x" data-dominant-color="F0F0F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">891×342 35.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>BTW, where is the complete doc of <code>nn.DataParallel</code>??</p>

albanD: <p>Here: <a href="https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html#torch.nn.DataParallel">https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html#torch.nn.DataParallel</a></p>

pytorching: <p>I have tested both the orders. It doesn’t help.<br>
Can you plz look at the code?</p>

albanD: <p>Is the error just with your own assert that checks if things are on the GPU?</p>

pytorching: <p>No. I just use the assert to verify but all the inputs and parameters are on cuda. So I’m confused.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b8d3c4df1dd6e7550b64d8df721a50aebd6c4c4.png" data-download-href="https://discuss.pytorch.org/uploads/default/0b8d3c4df1dd6e7550b64d8df721a50aebd6c4c4" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b8d3c4df1dd6e7550b64d8df721a50aebd6c4c4.png" alt="image" data-base62-sha1="1EbQzCq73ngAsKseMt9jZtfLm3q" width="690" height="70" data-dominant-color="E4E4E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">695×71 11.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cf94413a67e3a6372834ba3652e2f0b9f26c6ba.png" data-download-href="https://discuss.pytorch.org/uploads/default/6cf94413a67e3a6372834ba3652e2f0b9f26c6ba" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6cf94413a67e3a6372834ba3652e2f0b9f26c6ba_2_690x237.png" alt="image" data-base62-sha1="fy1Ceax0dkJt73Y38tbP27y1QE2" width="690" height="237" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6cf94413a67e3a6372834ba3652e2f0b9f26c6ba_2_690x237.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6cf94413a67e3a6372834ba3652e2f0b9f26c6ba_2_1035x355.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cf94413a67e3a6372834ba3652e2f0b9f26c6ba.png 2x" data-dominant-color="0B343E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1256×433 76.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>