Loss increasing dramatically when running with multiple GPU

Truong_Khang: <p>Hello,<br>
I run my model on a single GPU, the result of the test set is normal. But when I change to run on multiple GPU, the metrics evaluated on the test set (loss, RMSE,…) are not stable, they increase dramatically over time. I wonder why this problem happens, I thought the result must be the same in both cases.</p>
<p>Pytorch version: 0.4.1.</p>
<p>Best,<br>
Khang Truong.</p>

ptrblck: <p>Depending on the batch size (and potentially other hyperparameters you’ve changed), the output might not be exactly the same.<br>
I would recommend to update to the latest stable PyTorch release (<code>1.5.0</code>) and rerun the script again.<br>
<code>0.4.1</code> is quite old by now and a lot of fixes went into the code since then. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Truong_Khang: <p>Thanks for your quick response. I’ll try to test with a newer version version. Can I install pytorch version 1.5.0 along with cuda version 9.0?</p>

ptrblck: <p>You would have to build from source for CUDA9.0, as the <code>1.5</code> binaries ship with CUDA9.2, 10.1, and 10.2.</p>

Truong_Khang: <p>I tried to upgrade pytorch to 1.1.0 and re-run my code. But the problem is still happening. I don’t change any hyperparameters, only change running from a single GPU to multiple GPU by using torch.DataParallel(). Is there any reason for this?</p>

ptrblck: <p>If you didn’t change any hyperparameters, I assume the batch size is also the same as for the single GPU run.<br>
If that’s the case, your training might suffer, if you are using batchnorm layers, as their running estimates might be off due to the lower batch size on each device.<br>
<code>DistributedDataParallel</code> provides <code>SyncBatchNorm</code>, which synchronizes these stats between the processes.</p>
<p>PyTorch <code>1.1.0</code> is also not really new. Are you having trouble updating to the latest version?<br>
If so, could you post the issues, so that we can help with it?</p>

Truong_Khang: <p>My model does not use any BatchNorm layers. However, I implemented a normalized convolution like below, and my network only utilizes this operation</p>
<pre><code class="lang-auto"># The proposed Normalized Convolution Layer
class NConv2d(_ConvNd):
    def __init__(self, in_channels, out_channels, kernel_size, pos_fn='softplus',
                 init_method='k', stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros'):
        kernel_size = _pair(kernel_size)
        stride = _pair(stride)
        padding = _pair(padding)
        dilation = _pair(dilation)
        # Call _ConvNd constructor
        super(NConv2d, self).__init__(in_channels, out_channels, kernel_size,
                                      stride, padding, dilation, False, _pair(0), groups, bias, padding_mode)
        
        self.eps = 1e-20
        self.pos_fn = pos_fn
        self.init_method = init_method
        
        # Initialize weights and bias
        self.init_parameters()
        
        if self.pos_fn is not None:
            EnforcePos.apply(self, 'weight', pos_fn)
        
    def forward(self, data, conf):
        
        # Normalized Convolution
        denom = F.conv2d(conf, self.weight, None, self.stride,
                        self.padding, self.dilation, self.groups)        
        nomin = F.conv2d(data*conf, self.weight, None, self.stride,
                        self.padding, self.dilation, self.groups)        
        nconv = nomin / (denom+self.eps)

        # Add bias
        b = self.bias
        sz = b.size(0)
        b = b.view(1,sz,1,1)
        b = b.expand_as(nconv)
        nconv += b
        
        # Propagate confidence
        cout = denom
        sz = cout.size()
        cout = cout.view(sz[0], sz[1], -1)
        
        k = self.weight
        k_sz = k.size()
        k = k.view(k_sz[0], -1)
        s = torch.sum(k, dim=-1, keepdim=True)        

        cout = cout / s
        cout = cout.view(sz)
        
        return nconv, cout
        
        
# Non-negativity enforcement class        
class EnforcePos(object):
    def __init__(self, pos_fn, name):
        self.name = name
        self.pos_fn = pos_fn


    @staticmethod
    def apply(module, name, pos_fn):
        fn = EnforcePos(pos_fn, name)
        
        module.register_forward_pre_hook(fn)                    

        return fn

    def __call__(self, module, inputs):
       if module.training:
            weight = getattr(module, self.name)
            weight.data = self._pos(weight).data
       else:
            pass

    def _pos(self, p):
        pos_fn = self.pos_fn.lower()
        if pos_fn == 'softmax':
            p_sz = p.size()
            p = p.view(p_sz[0],p_sz[1], -1)
            p = F.softmax(p, -1)
            return p.view(p_sz)
        elif pos_fn == 'exp':
            return torch.exp(p)
        elif pos_fn == 'softplus':
            return F.softplus(p, beta=10)
        elif pos_fn == 'sigmoid':
            return F.sigmoid(p)
        else:
            print('Undefined positive function!')
            return
</code></pre>
<p>You can see that this implementation used class <code>EnforcePos</code> to force the weights non-negative. I wonder that this implementation might cause the problem as I mentioned in this post, but I’m not sure. Do you have any suggestions for implementing non-negative constraint?</p>
<p>I have some problems when building pytorch 1.5.0 from source because my CUDA version is 9.0. Anyway, I’ll try to update to lastest version and re-test again!</p>

ptrblck: <p>Using the <code>.data</code> attribute is not recommended and might yield unwanted side effects, so this line of code might be dangerous:</p>
<pre><code class="lang-python">weight.data = self._pos(weight).data
</code></pre>
<p>You could probably use <code>.copy_</code> and wrap it in a <code>with torch.no_grad()</code> block, if Autograd raises an issue.</p>
<p>Besides that I cannot see any obvious errors.</p>

Truong_Khang: <p>I did as you suggested. It raised this error<br>
<code>RuntimeError: diff_view_meta-&gt;output_nr_ == 0 ASSERT FAILED at /opt/conda/conda-bld/pytorch_1556653183467/work/torch/csrc/autograd/variable.cpp:209, please report a bug to PyTorch.</code><br>
How to fix this?</p>

ptrblck: <p>Could you post an executable code snippet to reproduce this error?</p>

Truong_Khang: <p>In the <code>EnforcePos</code> above, I changed the line <code>weight.data = self._pos(weight).data)</code> to:</p>
<pre><code class="lang-auto">with torch.no_grad():
    weight.copy_(self._pos(weight))
</code></pre>
<p>Next I made a simple network as follows:</p>
<pre><code class="lang-auto">class NormCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.nconv1 = NConv2d(1, 2, (5, 5), pos_fn='softplus', padding=2)
        self.nconv2 = Nconv2d(2, 2, (5, 5), pos_fn='softplus', padding=2)

    def forward(self, x, c):
        xout, cout = self.nconv1(x, c)
        xout, cout = self.nconv2(xout, cout)
        return xout, cout
</code></pre>
<p>The testing code is given by:</p>
<pre><code class="lang-auto">model = NormCNN()
model = torch.DataParallel(model)
model = model.cuda()

x = torch.rand(4, 1, 480, 752).cuda()
c = (x &gt; 0.5).float().cuda()
xo, co = model(x, c)
</code></pre>
<p>And then, I got the error as above!</p>

ptrblck: <p>Thanks for the code.<br>
It seems your approach is similar to <code>weight_norm</code>, which uses the <code>setattr</code> and <code>getattr</code> to modify the <code>weight</code> parameter so you could try to adapt <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/utils/weight_norm.py"><code>WeightNorm</code></a> to your use case.<br>
Alternatively, it might be easier to use the functional API, which would define a <code>self.weight</code> parameter, manipulate it, and use it via <code>F.conv2d</code>.</p>
<p>Let me know, if this would work.</p>

Truong_Khang: <p>I followed the implementation of <code>weight_norm</code> to correct my function. It works well now. I can train on multiple GPUs without increasing loss anymore.</p>
<p>Thanks for your help!</p>