Do I need add torch.cuda.stream in such forward function?

weikunhan: <p>Hello everyone,</p>
<p>I am currently implementing my research idea in the following picture:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/4/041efffdd88c3908c02d52d5d1f3bfe3b8e8782c.png" data-download-href="https://discuss.pytorch.org/uploads/default/041efffdd88c3908c02d52d5d1f3bfe3b8e8782c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/041efffdd88c3908c02d52d5d1f3bfe3b8e8782c_2_690x240.png" alt="image" data-base62-sha1="Askh0sFcDuKmYe5Lp5WLj93el6" width="690" height="240" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/041efffdd88c3908c02d52d5d1f3bfe3b8e8782c_2_690x240.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/041efffdd88c3908c02d52d5d1f3bfe3b8e8782c_2_1035x360.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/041efffdd88c3908c02d52d5d1f3bfe3b8e8782c_2_1380x480.png 2x" data-dominant-color="EBEEEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1630×569 88.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Based on the information on <a href="https://pytorch.org/docs/master/notes/cuda.html#asynchronous-execution" rel="noopener nofollow ugc">asynchronous execution</a>, I know I don’t need to worry too much about the “forward” function. However, is it that possible to use " torch.cuda.stream" could be faster than without using it? I hope someone can answer my question. Thank you very much.<br>
H</p>
<p>To clarify my question, I display two sample implementations (incomplete code to clarifying my question).</p>
<ol>
<li>without " torch.cuda.stream"</li>
</ol>
<pre data-code-wrap="python"><code class="lang-python">class PointConvResNet(nn.Module):

    def __init__(self):
        self.spixelnet = SpixelNet()
        self.layer1 = ResNet().layer1 
        self.layer2 = self._make_layer(block, 128, layers[1])
        self.layer3 = self._make_layer(block, 256, layers[2])
        self.layer4 = self._make_layer(block, 512, layers[3])
        self.avgpool = nn.AdaptiveAvgPool1d((1,))
        self.fc = nn.Linear(512 * block.expansion, num_classes)

    def forward(self, images):
        superpixels = self.spixelnet(images)
        x = self.conv1(images)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.layer1(x)
        xy = convert_xy(superpixels, self.imsize[0], self.imsize[1])
        points = convert_points(x, superpixels, self.imsize[0], self.imsize[1])
        xy, points = self._forward_impl(xy, points, self.wsize[0], self.layer2)
        xy, points = self._forward_impl(xy, points, self.wsize[1], self.layer3)
        xy, points = self._forward_impl(xy, points, self.wsize[2], self.layer4)
        x = self.avgpool(points)
        x = torch.flatten(x, 1)
        x = self.fc(x)

        return x
</code></pre>
<p>2 use " torch.cuda.stream"</p>
<pre data-code-wrap="python"><code class="lang-python">class PointConvResNet(nn.Module):

    def __init__(self):
        self.spixelnet = SpixelNet()
        self.layer1 = ResNet().layer1 
        self.layer2 = self._make_layer(block, 128, layers[1])
        self.layer3 = self._make_layer(block, 256, layers[2])
        self.layer4 = self._make_layer(block, 512, layers[3])
        self.avgpool = nn.AdaptiveAvgPool1d((1,))
        self.fc = nn.Linear(512 * block.expansion, num_classes)
        self.stream1 = torch.cuda.Stream()
        self.stream2 = torch.cuda.Stream()

    def forward(self, images):
        with torch.cuda.stream(stream1):
            superpixels = self.spixelnet(images)

        with torch.cuda.stream(stream2):
            x = self.conv1(images)
            x = self.bn1(x)
            x = self.relu(x)
            x = self.layer1(x)
        xy = convert_xy(superpixels, self.imsize[0], self.imsize[1])
        points = convert_points(x, superpixels, self.imsize[0], self.imsize[1])
        xy, points = self._forward_impl(xy, points, self.wsize[0], self.layer2)
        xy, points = self._forward_impl(xy, points, self.wsize[1], self.layer3)
        xy, points = self._forward_impl(xy, points, self.wsize[2], self.layer4)
        x = self.avgpool(points)
        x = torch.flatten(x, 1)
        x = self.fc(x)

        return x
</code></pre>

weikunhan: <p>Hope @<a class="mention" href="/u/ptrblck">@ptrblck</a> can help me. Thank you so much!</p>

mrshenli: <aside class="quote no-group" data-username="weikunhan" data-post="1" data-topic="106573">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/weikunhan/48/27369_2.png" class="avatar"> weikunhan:</div>
<blockquote>
<p>However, is it that possible to use " torch.cuda.stream" could be faster than without using it?</p>
</blockquote>
</aside>
<p>Yep, using multiple streams could be faster, if operators from one stream cannot fully utilize all computing resources on the GPU. To check if it is the case for your application, you can measure the total execution time using either CUDA events <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.Event.elapsed_time" rel="noopener nofollow ugc"><code>elapsed_time</code></a> or <a href="https://pytorch.org/docs/master/cuda.html#torch.cuda.synchronize" rel="noopener nofollow ugc"><code>torch.cuda.synchronize()</code></a> with <code>time.time()</code>.</p>
<p>However, in your second <code>forward </code> function, there are potential unsafe usage of streams. The line below consumes the value of <code>superpixels</code> on the default stream, but the preceding assignment <code>superpixels = self.spixelnet(images)</code> was on <code>stream1</code>. In this case, its possible that the default stream reads <code>superpixels</code> before <code>stream1</code> writes it.</p>
<pre data-code-wrap="python"><code class="lang-python">convert_xy(superpixels, self.imsize[0], self.imsize[1])
</code></pre>
<p>The same thing also applies to <code>x</code>. You can use <a href="https://pytorch.org/docs/master/cuda.html#torch.cuda.Stream.wait_stream" rel="noopener nofollow ugc"><code>wait_stream</code></a> to synchronize streams.</p>

weikunhan: <aside class="quote no-group" data-username="mrshenli" data-post="3" data-topic="106573">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p><code>_xy(superpixels, self.imsize[0], self.imsize[1])</code></p>
</blockquote>
</aside>
<p>Thanks for helping. <a href="https://pytorch.org/docs/master/cuda.html#torch.cuda.Stream.wait_stream" rel="noopener nofollow ugc"><code>wait_stream</code></a> is necessary in my case.  I could use <a href="https://pytorch.org/docs/master/cuda.html#torch.cuda.synchronize" rel="noopener nofollow ugc"><code>torch.cuda.synchronize()</code></a> to check which way is better for implementation.</p>