`CUDA_LAUNCH_BLOCKING=1` freeze

acgtyrant: <p>I assume you have downloaded <code>gtFine_trainvaltest.zip</code> and <code>leftImg8bit_trainvaltest.zip</code> from <a href="https://www.cityscapes-dataset.com/downloads/" class="inline-onebox" rel="noopener nofollow ugc">Login – Cityscapes Dataset</a> and extract to <code>~/BigDatas/cityscapes</code> which has <code>gtFine</code> and <code>leftImage8bit</code>.</p>
<p>Reproduce steps:</p>
<pre><code>git clone https://github.com/fyu/drn.git
cd drn
python3 datasets/cityscapes/prepare_data.py ~/BigDatas/cityscapes/gtFine
cp datasets/cityscapes/create_lists.sh ~/BigDatas/cityscapes
cp datasets/cityscapes/info.json ~/BigDatas/cityscapes
cd ~/BigDatas/cityscapes
sh create_lists.sh
chmod u+x segment.py
</code></pre>
<p>Then <code>CUDA_VISIBLE_DEVICES=0,1  python3 segment.py train -d ~/BigDatas/cityscapes -c 19 -s 896 --arch drn_d_22 --batch-size 32 --epochs 250 --lr 0.01 --momentum 0.9 --step 100</code> works very well. However, <code>CUDA_VISIBLE_DEVICES=0,1 CUDA_LAUNCH_BLOCKING=1 python3 segment.py train -d ~/BigDatas/cityscapes -c 19 -s 896 --arch drn_d_22 --batch-size 32 --epochs 250 --lr 0.01 --momentum 0.9 --step 100</code> will freeze like below:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/26ace31ebc92846f05455078d50dd30ce4f85e79.jpg" data-download-href="https://discuss.pytorch.org/uploads/default/26ace31ebc92846f05455078d50dd30ce4f85e79" title="2018-07-04-105307_1917x1053_scrot"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/26ace31ebc92846f05455078d50dd30ce4f85e79_2_690x379.jpg" alt="2018-07-04-105307_1917x1053_scrot" data-base62-sha1="5w8yhBSBQIxlCVinEsnaz3DPLiF" width="690" height="379" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/26ace31ebc92846f05455078d50dd30ce4f85e79_2_690x379.jpg, https://discuss.pytorch.org/uploads/default/optimized/2X/2/26ace31ebc92846f05455078d50dd30ce4f85e79_2_1035x568.jpg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/2/26ace31ebc92846f05455078d50dd30ce4f85e79_2_1380x758.jpg 2x" data-dominant-color="2B2C2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2018-07-04-105307_1917x1053_scrot</span><span class="informations">1917×1053 991 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am doubting that I am the first man who meet the problem of “train with CUDA_LAUNCH_BLOCKING=1 will freeze” in the world, I am helpless~</p>
<p>Ubuntu 16.04, four GTX 1080, PyTorch 0.4 which is installeld from <code>pip install torch</code>.</p>

acgtyrant: <p>I debug by myself now, I will try my best to reduce the range of the bug or the problem.</p>

acgtyrant: <p>I simplify my project to a file <code>segment.py</code>:</p>
<pre><code>#!/usr/bin/env python

import torch
from torch import nn
import torch.utils.data


class DRN(nn.Module):
    def __init__(self):
        super(DRN, self).__init__()
        self.a = nn.Conv2d(3, 16, kernel_size=7)

    def forward(self, x):
        print('before DRN forward')
        return x


if __name__ == '__main__':
    model = torch.nn.DataParallel(DRN()).cuda().train()
    input_ = torch.rand(2).cuda()
    print('before input')
    model(input_)
    print('end')
</code></pre>
<p>If I run <code>CUDA_LAUNCH_BLOCKING=1 CUDA_VISIBLE_DEVICES=0,1 ./segment.py</code>, It will stucks after print <code>before input</code>.</p>
<p>However, if I change <code>rand(2)</code> to <code>rand(1)</code>, it does not stuck again.</p>

acgtyrant: <p>I report the bug here: <a href="https://github.com/pytorch/pytorch/issues/9163" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/9163</a></p>

acgtyrant: <p><code>CUDA_LAUNCH_BLOCKING=1</code> does not work with DataParallel actually.</p>

tk0320: <p>what does <code>CUDA_LAUNCH_BLOCKING=1 </code> means?</p>

ptrblck: <p>This env variable will synchronize the kernel launches, so that the stacktrace would point to the right line of code in case a kernel is hitting an internal assert. Otherwise, due to the async execution of CUDA kernels, the errors might be reported in another line of code since the CPU could run ahead.</p>

youli_li: <p>I encountered the same problem. Finally, I replaced “CUDA_VISIBLE_DEVICES=2,3” with “CUDA_VISIBLE_DEVICES=3” to fix the problem.</p>

zdundjkl: <p>Me too.I set “2”,It worked.</p>