Slower Backward performance on CUSTOM C++ EXTENSIONS

eduardo4jesus: <p>I am following this tutorial on <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">CUSTOM C++ AND CUDA EXTENSIONS</a>.</p>
<p>I pretty much took the code available in the page and ran in a <a href="https://colab.research.google.com/gist/eduardo4jesus/d3880eff95cb54b54d5586fc3141abfa/custom-c-extensions.ipynb" rel="noopener nofollow ugc">Google Colab environment</a>.</p>
<h1><a name="expectation-1" class="anchor" href="#expectation-1"></a>Expectation</h1>
<p><strong>From the tutorial page we have the following comparisons</strong></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a2e158c259683cc244e8592757046bdb9d3a9d4.png" data-download-href="https://discuss.pytorch.org/uploads/default/5a2e158c259683cc244e8592757046bdb9d3a9d4" title="Screen Shot 2021-01-07 at 12.07.10 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a2e158c259683cc244e8592757046bdb9d3a9d4_2_690x297.png" alt="Screen Shot 2021-01-07 at 12.07.10 PM" data-base62-sha1="cRLHQy0RpTMc51uoxWfnMzFn2o4" width="690" height="297" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a2e158c259683cc244e8592757046bdb9d3a9d4_2_690x297.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a2e158c259683cc244e8592757046bdb9d3a9d4_2_1035x445.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a2e158c259683cc244e8592757046bdb9d3a9d4_2_1380x594.png 2x" data-dominant-color="F4F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-01-07 at 12.07.10 PM</span><span class="informations">1472×634 45.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/eef468bc457401716175362eb632e8cdd12ccdfe.png" data-download-href="https://discuss.pytorch.org/uploads/default/eef468bc457401716175362eb632e8cdd12ccdfe" title="Screen Shot 2021-01-07 at 12.07.25 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eef468bc457401716175362eb632e8cdd12ccdfe_2_690x370.png" alt="Screen Shot 2021-01-07 at 12.07.25 PM" data-base62-sha1="y5ThunIuKj2G5U0zW0t4p5vqzjo" width="690" height="370" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eef468bc457401716175362eb632e8cdd12ccdfe_2_690x370.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eef468bc457401716175362eb632e8cdd12ccdfe_2_1035x555.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eef468bc457401716175362eb632e8cdd12ccdfe_2_1380x740.png 2x" data-dominant-color="F2F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-01-07 at 12.07.25 PM</span><span class="informations">1478×794 67.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<h1><a name="reality-2" class="anchor" href="#reality-2"></a>Reality</h1>
<p>But, when running on the notebook, the backward pass gets worse. Is there any explanation on why this would happen?</p>
<pre><code class="lang-auto">====== CPU Default
Forward: 216.003 us | Backward 237.442 us
====== CPU Cpp Custom Extension
Forward: 171.469 us | Backward 437.926 us
</code></pre>
<pre><code class="lang-auto">====== GPU Default
Forward: 309.739 us | Backward 523.746 us
====== GPU Cpp Custom Extension
Forward: 250.254 us | Backward 879.118 us
</code></pre>

albanD: <p>Hi,</p>
<p>This is surprising indeed.<br>
It is possible though that the backward of rnn was heavily optimized since this post was done. And so the manual implementation is not as optimized anymore <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>