Error in torch.nn.DataParallel

aturahc13: <p>Hi. I would like to use “DataParallel” in DNN training in Pytorch but get some errors.<br>
Before I use “DataParallel”, the code is;</p>
<p>&lt;Code 1&gt;</p>
<pre><code class="lang-auto">for epoch in range(epochs):
    train_loss = 0.0
    val_loss = 0.0
    
    train_loader2 = MakeDataset(file_x_train, file_y_mask_train, tmpbatch_size, shuffle=False)
    test_loader2 = MakeDataset(file_x_test, file_y_mask_test, batch_size)
    # Training the model
    net2.train()
    counter = 0
    i = 0
    for data in train_loader2:
        inputs, labels = data
        
        if i ==0:
            conv_inputs = inputs
            conv_labels = labels
            #print(i,conv_inputs.shape)
        else:
            conv_inputs = torch.cat([conv_inputs, inputs],dim=0)
            conv_labels = torch.cat([conv_labels, labels],dim=0)
            #print(i,conv_inputs.shape)
            
        i += 1
            
        if i == nconv:
            #print(i,conv_inputs.shape)
            if cuda:
                conv_inputs, conv_labels = Variable(conv_inputs).cuda(), Variable(conv_labels).cuda()

            net_optimizer.zero_grad()
            outputs = net2.forward(conv_inputs)

            loss = criterion(outputs, conv_labels) # ###############

            loss.backward()
            net_optimizer.step()
            train_loss += loss.item()*conv_inputs.size(0)
            counter += 1
            print("\r{0}".format(counter), end="")
            
            i = 0
            
    # Evaluating the model
    net2.eval()
    
    
    counter = 0
    # Tell torch not to calcualte gradients
    with torch.no_grad():
        for data in test_loader2:
            # Move to device
            inputs, labels = data
            if cuda:
                inputs, labels = Variable(inputs).cuda(), Variable(labels).cuda()
            
            # Forward pass
            output = net2.forward(inputs)
            
            # Calculate Loss  
            valloss = criterion(output, labels) # ###############

            # Add loss to the validation set's running loss
            val_loss += valloss.item()*inputs.size(0)
                        
                
    # Get the average loss for the entire epoch
    train_loss = train_loss/train_len
    valid_loss = val_loss/test_len
    train_loss_list.append(train_loss)
    test_loss_list.append(valid_loss)

   
    # print out the information
    print('[%d] Training Loss: %.6f, Validation Loss: %.6f'  % (epoch + 1, train_loss, valid_loss))
    torch.save(net2.state_dict(), file_nnstate)
print('Finished Training')
</code></pre>
<p>The function " MakeDataset" is loading the training data from certain files(made by me). The DNN model is named “net2”.<br>
I would like to use “DataParallel” and add the code below, before this.<br>
&lt;Code 2&gt;</p>
<pre><code class="lang-auto">device_ids = range(torch.cuda.device_count())
print(device_ids)
net2 = torch.nn.DataParallel(net, device_ids=device_ids)
</code></pre>
<p>This is from <em><a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html</a></em><br>
The output corresponding this is</p>
<pre><code class="lang-auto">range(0, 8)
</code></pre>
<p>However, when I run &lt;Code 2&gt;, I get error on &lt;Code 1&gt;. The error is like this;<br>
&lt;Error on Code 1&gt;</p>
<pre><code class="lang-auto">ValueError: Caught ValueError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/_utils.py", line 385, in reraise
    raise self.exc_type(msg)
ValueError: Caught ValueError in replica 7 on device 7.
Original Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "&lt;ipython-input-16-0dd5899e55f0&gt;", line 14, in forward
    x = self.fc(input)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 81, in forward
    exponential_average_factor, self.eps)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/functional.py", line 1666, in batch_norm
    raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))
ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 220])
</code></pre>
<p>When I write</p>
<pre><code class="lang-auto">device_ids = range(1)
</code></pre>
<p>in &lt;Code 2&gt;, &lt;Code 1&gt; runs without errors.<br>
What should I do to run &lt;Code 1&gt; ? Is &lt;Code 2&gt; not enough?<br>
Thank you very much.</p>

ptrblck: <p>The error points to a batchnorm layer, which wants to calculate the running statistics, but fails due to a single provided sample.<br>
This is usually the case, if you feed a batch to the model containing one sample.<br>
Since you are using <code>nn.DataParallel</code> this could happen, if the batch size equals the number of GPUs.<br>
Also, the last batch might be smaller, which could trigger this effect. In that case, you can skip the last (smaller) batch using <code>drop_last=True</code> in your <code>DataLoader</code>.</p>

aturahc13: <p>Dear ptrblck,<br>
Thank you very much for your reply !<br>
As in your reply, I tried to set batch size 100.(bigger than 8 = the number of GPUs) and added <code>drop_last = True</code> in the <code>DataLoader</code> but I get the same error…<br>
Are there anything else to be checked?<br>
Thank you very much.</p>

ptrblck: <p>I assume the code is working using a single GPU?<br>
Could you print the shape of the input data inside <code>forward</code> for the sake of debugging?</p>

aturahc13: <p>input data shape is <code>torch.Size([10, 396]) torch.Size([10, 396]) torch.Size([10, 396]) torch.Size([10, 396]) torch.Size([10, 396]) torch.Size([10, 396]) torch.Size([10, 396]) torch.Size([10, 396]) torch.Size([10, 396]) torch.Size([10, 396])</code></p>
<p>I get this code by</p>
<pre><code class="lang-auto">#############################################(2019/11/5変更)


for epoch in range(epochs):
    train_loss = 0.0
    val_loss = 0.0
    
    train_loader2 = MakeDataset(file_x_train, file_y_mask_train, tmpbatch_size, shuffle=False)
    test_loader2 = MakeDataset(file_x_test, file_y_mask_test, batch_size)
    # Training the model
    net.train()
    counter = 0
    i = 0
    for data in train_loader2:
        inputs, labels = data
        
        ### This is new ###
        print(inputs.shape)
        ### This is new ###
        
        if i ==0:
            conv_inputs = inputs
            conv_labels = labels
            #print(i,conv_inputs.shape)
        else:
            conv_inputs = torch.cat([conv_inputs, inputs],dim=0)
            conv_labels = torch.cat([conv_labels, labels],dim=0)
            #print(i,conv_inputs.shape)
            
        i += 1
            
        if i == nconv:
            #print(i,conv_inputs.shape)
            if cuda:
                conv_inputs, conv_labels = Variable(conv_inputs).cuda(), Variable(conv_labels).cuda()

            net_optimizer.zero_grad()
            outputs = net.forward(conv_inputs)

            loss = criterion(outputs, conv_labels) # ###############

            loss.backward()
            net_optimizer.step()
            train_loss += loss.item()*conv_inputs.size(0)
            counter += 1
            print("\r{0}".format(counter), end="")
            
            i = 0
            
    # Evaluating the model
    net.eval()
    
    
    counter = 0
    # Tell torch not to calcualte gradients
    with torch.no_grad():
        for data in test_loader2:
            # Move to device
            inputs, labels = data
            if cuda:
                inputs, labels = Variable(inputs).cuda(), Variable(labels).cuda()
            
            # Forward pass
            output = net.forward(inputs)
            
            # Calculate Loss  
            valloss = criterion(output, labels) # ###############

            # Add loss to the validation set's running loss
            val_loss += valloss.item()*inputs.size(0)
                        
                
    # Get the average loss for the entire epoch
    train_loss = train_loss/train_len
    valid_loss = val_loss/test_len
    train_loss_list.append(train_loss)
    test_loss_list.append(valid_loss)

   
    # print out the information
    print('[%d] Training Loss: %.6f, Validation Loss: %.6f'  % (epoch + 1, train_loss, valid_loss))
    torch.save(net.state_dict(), file_nnstate)
print('Finished Training')

</code></pre>

ptrblck: <p>Is your code constantly printing out this shape? Even when the error is thrown?</p>

aturahc13: <p>Yes, it is. This is printed out when I use single GPU.</p>

aturahc13: <p>When I run <code>device_ids = range(torch.cuda.device_count()) print(device_ids) net2 = torch.nn.DataParallel(net, device_ids=device_ids)</code> before this (the output for this is <code>range(0, 8)</code>), it prints</p>
<pre><code class="lang-auto">torch.Size([10, 396])
torch.Size([10, 396])
torch.Size([10, 396])
torch.Size([10, 396])
torch.Size([10, 396])
torch.Size([10, 396])
torch.Size([10, 396])
torch.Size([10, 396])
torch.Size([10, 396])
torch.Size([10, 396])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-77-fb8be4f5e216&gt; in &lt;module&gt;()
     35 
     36             net_optimizer.zero_grad()
---&gt; 37             outputs = net2.forward(conv_inputs)
     38 
     39             loss = criterion(outputs, conv_labels) # ###############

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
    150             return self.module(*inputs[0], **kwargs[0])
    151         replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
--&gt; 152         outputs = self.parallel_apply(replicas, inputs, kwargs)
    153         return self.gather(outputs, self.output_device)
    154 

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py in parallel_apply(self, replicas, inputs, kwargs)
    160 
    161     def parallel_apply(self, replicas, inputs, kwargs):
--&gt; 162         return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
    163 
    164     def gather(self, outputs, output_device):

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py in parallel_apply(modules, inputs, kwargs_tup, devices)
     83         output = results[i]
     84         if isinstance(output, ExceptionWrapper):
---&gt; 85             output.reraise()
     86         outputs.append(output)
     87     return outputs

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/_utils.py in reraise(self)
    383             # (https://bugs.python.org/issue2651), so we work around it.
    384             msg = KeyErrorMessage(msg)
--&gt; 385         raise self.exc_type(msg)

ValueError: Caught ValueError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/_utils.py", line 385, in reraise
    raise self.exc_type(msg)
ValueError: Caught ValueError in replica 6 on device 6.
Original Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "&lt;ipython-input-16-0dd5899e55f0&gt;", line 14, in forward
    x = self.fc(input)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 81, in forward
    exponential_average_factor, self.eps)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/functional.py", line 1666, in batch_norm
    raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))
ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 220])```</code></pre>

ptrblck: <p>So are you multiplying the batch size by the number of GPUs (9)?<br>
<code>nn.DataParallel</code> will chunk the batch in dim0 and send each piece to a GPU.<br>
Since you get <code>[10, 396]</code> inside the <code>forward</code> method for a single GPU as well as for multiple GPUs using <code>nn.DataParallel</code>, your provided batch should have the shape <code>[90, 396]</code> before feeding it into the <code>nn.DataParallel</code> model.<br>
Is my assumption correct?</p>

aturahc13: <p>Dear ptrblck,<br>
I am sorry to be late even though you replied quickly yesterday.<br>
I did not multiple (9) and I have just tried now.</p>
<p>And it starts working !!!</p>
<p>The only remaining concern is if it is training correctly, so I will confirm that the performance is as same as the one I trained before, after some epochs.<br>
Thank you very much again for your easy-to-understand and polite explanation.</p>

PantherYan: <p>Yes, I also faced this problem.<br>
Traceback (most recent call last):                                                                 | 0/765 [00:00&lt;?, ?it/s]</p>
<pre><code>outputs = model(**inputs)
</code></pre>
<p>File “/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/usr/local/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py”, line 152, in forward<br>
outputs = self.parallel_apply(replicas, inputs, kwargs)<br>
File “/usr/local/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py”, line 162, in parallel_apply<br>
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])<br>
File “/usr/local/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py”, line 85, in parallel_apply<br>
output.reraise()<br>
File “/usr/local/lib/python3.6/site-packages/torch/_utils.py”, line 394, in reraise<br>
raise self.exc_type(msg)<br>
ValueError: Caught ValueError in replica 0 on device 0.<br>
Original Traceback (most recent call last):<br>
File “/usr/local/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py”, line 60, in _worker<br>
output = module(*input, **kwargs)<br>
File “/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/workspace/git/transformers/src/transformers/modeling_bert.py”, line 1522, in forward<br>
loss = loss_fct(logits, labels)<br>
File “/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/usr/local/lib/python3.6/site-packages/torch/nn/modules/loss.py”, line 916, in forward<br>
ignore_index=self.ignore_index, reduction=self.reduction)<br>
File “/usr/local/lib/python3.6/site-packages/apex/amp/wrap.py”, line 28, in wrapper<br>
return orig_fn(*new_args, **kwargs)<br>
File “/usr/local/lib/python3.6/site-packages/torch/nn/functional.py”, line 2021, in cross_entropy<br>
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)<br>
File “/usr/local/lib/python3.6/site-packages/apex/amp/wrap.py”, line 28, in wrapper<br>
return orig_fn(*new_args, **kwargs)<br>
File “/usr/local/lib/python3.6/site-packages/torch/nn/functional.py”, line 1836, in nll_loss<br>
.format(input.size(0), target.size(0)))<br>
ValueError: Expected input batch_size (2) to match target batch_size (4).</p>
<hr>
<p>After modified the batch data, but forgot to multiple * max(1, args.n_gpu) on the sample number dimension.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a>   So many thanks!</p>