Convert_syncbn_model causes gradient overflow with apex mixed precision

username: hanzCV
cooked: <p>Hi,<br>
I was trying to train my network using apex mixed precision. I’ve tried DenseNet and Resnet as backbones for a segmentation task using CityScapes. Unfortunately, when I try to synchronize the batch norm using <code>convert_syncbn_model</code>, the <code>scale_loss</code> ends up being zero after a few iterations because of gradient overflow. This does not happen if I remove the batch normalization.<br>
The snipet of my code is the following:</p>
<pre><code class="lang-auto">model.cuda(gpu)
model = apex.parallel.convert_syncbn_model(model)
optimizer = optim.Adam(model.parameters())
model, optimizer = apex.amp.initialize(model, optimizer)
#model = apex.parallel.convert_syncbn_model(model) #I also tried to put it here
net = DDP(model, delay_allreduce=True)
</code></pre>
<p>…</p>
<pre><code class="lang-auto">loss = Cross_entropy(y_pred, y_gt)
with apex.amp.scale_loss(loss, optimizer) as scaled_loss:
                    scaled_loss.backward()
optimizer.step()
optimizer.zero_grad()
</code></pre>
<p>System:<br>
OS: Ubuntu 16.04 and 18.04<br>
Pytorch: tried with 1.4, 1.5 and .1.6<br>
apex: 0.1</p>
<p>Did anyone experience the same?<br>
Thanks</p>

username: ptrblck
cooked: <p>Mixed precision training utilities as well as synchronized batchnorm layers are now available in PyTorch directly, so you don’t need <code>apex</code> anymore.<br>
We recommend to use these native implementations now. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Could you try them and see, if you encounter any issues?</p>

username: hanzCV
cooked: <p>I’ve replaced the code with the following cuda.amp code:</p>
<pre><code class="lang-auto">scaler = torch.cuda.amp.GradScaler()

for epoch in range(n_epochs):
    for i, (X_batch, y_gt) in enumerate(batches):
        X_batch = X_batch.cuda()
        y_gt = y_gt.cuda()
        optimizer.zero_grad()
        with torch.cuda.amp.autocast():
            y_pred = model(X_batch)
            loss = Cross_entropy(y_pred, y_gt)

        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()

</code></pre>
<p>It is training stably so far but it is really slow. I’m using Densenet121 backbone and some convolutions to output 19 segmentation classes with a batch size of 2 with 2 GPUs GTX1080Ti. The input size is 512x512. When I don’t use cuda.amp the training speed per iteration is ~2 seconds involving forward and backward passes. However, if I use cuda.amp, the speed per iteration is ~6.6 seconds.</p>
<p>System:<br>
OS: 18.04<br>
Pytorch: 1.6 (1.6.0+cu101)<br>
Cuda: 10.1 (cannot get cuda 10.2 because I don’t have superuser access)<br>
GPU: 2 GTX 1080Ti</p>

username: ptrblck
cooked: <p>Your 1080Ti GPUs do not have TensorCores, so that you shouldn’t expect a speedup from the computations. However, how did you measure the performance? Did you synchronize the code properly using <code>torch.cuda.synchronize()</code> before starting and stopping the timers?</p>

username: hanzCV
cooked: <p>I measure the performance following the post bellow. Instead of <code>while</code> I have the forward, backward, optimizer step and zeroing the parameter gradients, then I print the time each iteration takes.</p>
<p>I can confirm that the training did not face any problems yet, apart from the slow training <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20">.<br>
Do you think the cuda version might be causing this? Is there any verbose I can enable for debugging, or if it can print something like apex when there is gradient overflow and is adjusting the scale loss?</p>
<aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="20137">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"><a href="https://discuss.pytorch.org/t/is-there-any-difference-between-x-to-cuda-vs-x-cuda-which-one-should-i-use/20137/5">Is there any difference between x.to('cuda') vs x.cuda()? Which one should I use?</a></div>
<blockquote>
<pre><code class="lang-auto">torch.cuda.synchronize()
t1 = time.time()
while i&lt; 500:
    a += 1
    a -= 1
    i+=1
torch.cuda.synchronize()
t2 = time.time()
print('cuda string', t2-t1)
</code></pre>
</blockquote>
</aside>
<p>Thanks for the help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">.</p>

username: ptrblck
cooked: <p>Thanks for the information.<br>
I assume the slowdown can be observed on a single device and would thus be unrelated to <code>SyncBatchnorm</code>.<br>
Could you install the latest stable PyTorch release (or the nightly binaries) with CUDA10.2 and cudnn7.6.5.32?<br>
Note that the binaries ship with their own CUDA and cudnn rumtimes. Your local installations won’t be used so you don’t need to update the local CUDA version.</p>

username: hanzCV
cooked: <p>Thanks,<br>
I have downloaded the latest PyTorch with CUDA 10.2 using (pip install torch torchvision). To reduce possible errors, I tested it in a machine with CUDA 10.2 as well. Unfortunately, the training time remains slow when using <code>torch.cuda.amp</code>.</p>
<p>System:<br>
Ubuntu 18.04<br>
Cuda: 10.2<br>
Pytorch: 1.6<br>
GPU: 2 GTX 1080Ti</p>

username: ptrblck
cooked: <p>Thanks for the check.<br>
Are you seeing the slowdown using the <code>torchvision.models.densenet121</code> or are you changing it somehow?<br>
I would like to reproduce it with cudnn7 as well as cudnn8 and check, why amp is slowing down the model.</p>

username: hanzCV
cooked: <p>Hi, I modified the original PyTorch densenet121 to extract features at different scales. <a href="https://drive.google.com/file/d/1ZCGB2_lytxRyel2cO1EjZdxmzMJZvtDm/view?usp=sharing" rel="nofollow noopener">Here</a> is the modified densenet; it still can load the weights provided by PyTorch. I cannot send you our exact decoder, but I guess any SegNet-like architecture for pixel-wise segmentation should show the same behavior.</p>
<p>To check if the error was caused by my code, I used this <a href="https://github.com/nyoki-mtl/pytorch-segmentation" rel="nofollow noopener">repo</a>, which is a deeplab v3 implementation with pretrained weights for cityscapes. I tested <code>torch.cuda.amp</code>, <code>O1</code> from <code>apex.amp</code>, and no mixed precision (the last one was tested with half of the size of the input image used by the mixed precision tests).</p>
<p>I’ve run some epochs and found that <code>torch.cuda.amp</code> is twice as slow as <code>apex.amp</code>, and no mixed precision. In numbers, 1 iteration (forward and backward) using <code>torch.cuda.amp</code> takes ~2.50 seconds, the other two take around ~1.20 seconds.</p>
<p>Configuration:<br>
batch size: 2 per GPU<br>
Image size: 512x512 (mixed precision), 256x512 (no mixed precision)<br>
Batch sync: Yes</p>
<p>Dataset: Cityscapes semantic segmentation<br>
Code: <a href="https://github.com/nyoki-mtl/pytorch-segmentation" rel="nofollow noopener">https://github.com/nyoki-mtl/pytorch-segmentation</a><br>
optimizer: adam<br>
loss: cross entropy</p>
<p>System:<br>
Ubuntu 18.04<br>
Cuda: 10.2<br>
CudNN: 8.0.3<br>
Pytorch: 1.6<br>
GPU: 2 GTX 1080Ti</p>
<p>If you need more details please let me know. Thanks for the help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">.</p>
<hr>
<p>p.s.<br>
Just to update more information about Apex. I’m still getting gradient overflow and <code>scale_loss</code> being reduce to zero when using <code>apex.parallel.convert_syncbn_model()</code> after few epochs.</p>

username: hanzCV
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for your help. I manage to get access to a pair of RTX2080Ti. Here the speed of <code>torch.amp</code> is similar to <code>apex</code>. I guess <code>torch.amp</code> is optimized to use TensorCores to boost the speed, given that GTX GPUs do not have them, the computation time is slower. Could you confirm this?</p>

username: ptrblck
cooked: <p>The model execution shouldn’t be slower on Pascal GPUs but you also shouldn’t expect significant speedup. The model execution should benefit from the reduced memory transfers, which could show a speedup even on older GPUs.</p>
<p>Are you seeing the slowdown on the 1080Ti even with <code>torch.backends.cudnn.benchmark=True</code>?</p>

username: hanzCV
cooked: <p>Yes, even with <code>torch.backends.cudnn.benchmark=True</code> placed at the beginning of the script it is slow using a GTX1080Ti. On the other hand, It is around 3 days of training using <code>torch.cuda.amp</code> with a pair of RTX2080Ti’s without any problems. This reinforces more my theory about how <code>torch.cuda.amp</code> is optimized.</p>