Why does Winograd algorithm speedup convolution, given that MUL and ADD cost the same clock cycles on GPU?

zfzhang: <p>Note: this is a general (algorithmic &amp; CUDA) question, not related to Pytorch.</p>
<p>The <a href="https://en.wikipedia.org/wiki/Coppersmith%E2%80%93Winograd_algorithm" rel="nofollow noopener">Winograd algorithm</a> transforms normal convolution computation by reducing <strong>MUL</strong> and at the same time increasing <strong>ADD</strong> operations. The number of <strong>MUL</strong> reduced is less than the number of <strong>ADD</strong> increased. However,  <a href="https://developer.download.nvidia.com/compute/cuda/1.0/NVIDIA_CUDA_Programming_Guide_1.0.pdf" rel="nofollow noopener">CUDA programming guide</a> says both floating point <strong>MUL</strong> &amp; <strong>ADD</strong> <strong>equally costs 4 clock cycles</strong>.</p>
<p>Question: How does Winograd speedup <strong>rather than slowdown</strong> convolution computation?</p>

KFrank: <p>Hello zfzhang!</p>
<aside class="quote no-group quote-modified" data-username="zfzhang" data-post="1" data-topic="89993" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/ecc23a/48.png" class="avatar"> zfzhang:</div>
<blockquote>
<p>Then how does Winograd speedup rather than slowdown convolution computation?</p>
</blockquote>
</aside>
<p>The Winograd matrix-multiplication algorithm does <em>not</em> speed up<br>
convolution.  I am not aware that the Winograd algorithm is ever<br>
used in practice.</p>
<p>Note, the Winograd algorithm <em>does</em> reduce the <em>total</em> number of<br>
additions needed.  The confusion might be that the algorithm is<br>
recursive, and, at a given step in the recursion, the number of<br>
<em>matrix</em> additions is, indeed, larger than in the straightforward<br>
algorithm, but the number of <em>matrix</em> multiplications is reduced,<br>
more than compensating as you recurse down to the bottom.</p>
<p>(Some notes:  The straightforward algorithm for multiplying <code>n x n</code><br>
matrices has time-complexity O(n^3).  The Winograd algorithm is<br>
one of a series of sub-O(n^3) matrix-multiplication algorithms, but<br>
they are all primarily of purely theoretical interest, and are used<br>
rarely (if ever) in practice.  They only become competitive with the<br>
“naive” algorithm when <code>n</code> is <em>quite</em> large, because, even when<br>
only counting floating-point operations, they have much larger<br>
pre-factors.  Furthermore, they break the benefits of floating-point<br>
pipelines, so, in practice, they make use of significantly fewer<br>
floating-point operations per unit of time.  Nonetheless, this whole<br>
story is quite interesting (if you care about such things).  I believe<br>
it all started with Strassen’s algorithm.)</p>
<p>Best.</p>
<p>K. Frank</p>

zfzhang: <p>Hi KFrank, thanks for the prompt reply <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Current research does indicate that Winograd speeds up convolution, see:</p>
<p><a href="https://arxiv.org/abs/1509.09308" rel="nofollow noopener">https://arxiv.org/abs/1509.09308</a></p>
<p><a href="http://cs231n.stanford.edu/reports/2016/pdfs/117_Report.pdf" rel="nofollow noopener">http://cs231n.stanford.edu/reports/2016/pdfs/117_Report.pdf</a></p>

KFrank: <p>Hi zfzhang!</p>
<aside class="quote no-group" data-username="zfzhang" data-post="3" data-topic="89993" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/ecc23a/48.png" class="avatar"> zfzhang:</div>
<blockquote>
<p>Current research does indicate that Winograd speeds up convolution, see:</p>
</blockquote>
</aside>
<p>These two papers you cite:</p>
<p><a href="https://arxiv.org/abs/1509.09308" rel="noopener nofollow ugc">https://arxiv.org/abs/1509.09308</a><br>
and<br>
<a href="http://cs231n.stanford.edu/reports/2016/pdfs/117_Report.pdf" rel="noopener nofollow ugc">http://cs231n.stanford.edu/reports/2016/pdfs/117_Report.pdf</a></p>
<p>refer to a “minimal filtering algorithm” published by Winograd in 1980.</p>
<p>The Wikipedia link you give for the “Winograd algorithm” in your original<br>
post:</p>
<aside class="onebox wikipedia" data-onebox-src="https://en.wikipedia.org/wiki/Coppersmith%E2%80%93Winograd_algorithm">
  <header class="source">

      <a href="https://en.wikipedia.org/wiki/Coppersmith%E2%80%93Winograd_algorithm" target="_blank" rel="noopener nofollow ugc">en.wikipedia.org</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://en.wikipedia.org/wiki/Coppersmith%E2%80%93Winograd_algorithm" target="_blank" rel="noopener nofollow ugc">Computational complexity of matrix multiplication</a></h3>

<p>What is the fastest algorithm for matrix multiplication? In theoretical computer science, the computational complexity of matrix multiplication dictates how quickly the operation of matrix multiplication can be performed. Matrix multiplication algorithms are a central subroutine in theoretical and numerical algorithms for numerical linear algebra and optimization, so finding the fastest algorithm for matrix multiplication is of major practical relevance.
 Directly applying the mathematical defini...</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>is for the Strassen-like Coppersmith-Winograd matrix-multiplication<br>
algorithm published by them in 1990.  Same Winograd, two different<br>
algorithms.</p>
<p>I’m willing to believe that Winograd’s minimal-filtering algorithm can<br>
be used in practice to speed up CNNs.</p>
<p>Best.</p>
<p>K. Frank</p>

ptrblck: <aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="89993">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>I’m willing to believe that Winograd’s minimal-filtering algorithm can<br>
be used in practice to speed up CNNs.</p>
</blockquote>
</aside>
<p>For the sake of completeness: winograd algos can be picked by <code>cudnnFind</code> (via <code>torch.backends.cudnn.benchmark = True</code>), if they are the fastest algo for this particular workload.<br>
E.g. the cudnn docs mention it <a href="https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#tensor-ops-conv-functions-supported-algos">here</a>.</p>

zfzhang: <p>Hi ptrblck,</p>
<p>my point is: the algorithm makes the assumption that MUL is more expensive than ADD, but this assumption doesn’t seem to hold in the CUDA computation model, where they both cost 4 clock cycles. (side note: on CPU the cost ratio is typically 4:1)</p>

ptrblck: <p>That’s a valid concern from a theoretical point of view.<br>
Note however, that a lot of operations are not compute bound, but often memory bandwidth bound.<br>
I.e. the GPU computation would be “fast enough”, but the overall speed is limited by the reads and writes of the data.<br>
Even if the MUL and ADD are more expensive on GPUs, which could theoretically mean that there wouldn’t be any expected speedup using Winograd, in reality the kernel performance might still be better than other (less compute-intensive) kernels.</p>
<p>cudnn in particular uses the benchmark mode to profile different algorithms and should only select the Winograd kernel, if it’s the fastest one (and also meets other requirements specified by the user such as determinism).</p>