Why does pytorch prefer using NCHW?

johnestar: <p>Numpy uses NHWC, pytorch uses NCHW, all the conversion seems a bit confusing at times, why does Pytorch use NCHW at the very beginning?</p>

ptrblck: <p>I think it was the default format in LuaTorch and I don’t know, why this format was preferred over NHWC.<br>
However, note that PyTorch has now <a href="https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html">experimental channels-last support</a>. By using this, you would still create and index the tensors as NCHW to guarantee backwards compatibility.</p>

Nikronic: <p>Hi,</p>
<p>About the ordering, I think <code>NCHW</code> is much more intuitive rather than latter choice. It is like going from high level to low level view (batch_size &gt; patch_size &gt; channel_size).</p>
<p>PyTorch uses a <code>Storage</code> for each tensor that follows a particular <a href="https://pytorch.org/docs/stable/tensor_attributes.html#torch-layout" rel="noopener nofollow ugc">layout</a>. As PyTorch uses <code>strided</code> layout for mapping logical view to the physical location of data in the memory, there should not be any difference in performance as it is only a swap of channels in <code>stride</code> value.  There are some information about this in a talk by <a class="mention" href="/u/ezyang">@ezyang</a> in his blog post <a href="http://blog.ezyang.com/2019/05/pytorch-internals/" rel="noopener nofollow ugc">PyTorch internals : Inside 245-5D</a>).</p>
<p>In a part of it, this image has been provided and a caption that says: <em>By the way, we’re interested in making this picture not true; instead of having a separate concept of storage, just define a view to be a tensor that is backed by a base tensor. This is a little more complicated, but it has the benefit that contiguous tensors get a much more direct representation without the Storage indirection. A change like this would make PyTorch’s internal representation a bit more like Numpy’s.</em></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/6/76cc1a74536d5397a4aae5b7c7cdef78f9c0b44c.png" data-download-href="https://discuss.pytorch.org/uploads/default/76cc1a74536d5397a4aae5b7c7cdef78f9c0b44c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76cc1a74536d5397a4aae5b7c7cdef78f9c0b44c_2_666x500.png" alt="image" data-base62-sha1="gWVD7andC4Ln9w9mr26O3Q2vSW0" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76cc1a74536d5397a4aae5b7c7cdef78f9c0b44c_2_666x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76cc1a74536d5397a4aae5b7c7cdef78f9c0b44c_2_999x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76cc1a74536d5397a4aae5b7c7cdef78f9c0b44c_2_1332x1000.png 2x" data-dominant-color="F3F3F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1600×1200 84.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But in the link you have provided, it has been shown that there is a perf gain in almost all prominent architectures. What could be the possible explanation for this?</p>
<p>Bests</p>

albanD: <p>IIRC, NHWC allows you much better implementation for convolution layers that gives a nice boost in perf (because the value for all channels is accessed for every pixel, so data locality is much better in that format).<br>
The problem is that batchnorm-like layer are so much faster in NCHW, that vision models nowadays do not perform that much better in NHWC (for these layers, you reduce over the HW dimensions. And so having C last breaks the data locality).</p>
<p>But for the original question: I would agree with <a class="mention" href="/u/ptrblck">@ptrblck</a>: for historical reasons mostly and because there is no clear reason that the other format is better in general (at least not worth rewriting all the TH/THC libraries!).</p>

ptrblck: <aside class="quote no-group" data-username="Nikronic" data-post="3" data-topic="83637">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar"> Nikronic:</div>
<blockquote>
<p>What could be the possible explanation for this?</p>
</blockquote>
</aside>
<p>As <a class="mention" href="/u/alband">@albanD</a> pointed out, the memory access pattern is beneficial in NHWC for (some) convolution implementations. Also, TensorCores natively use NHWC tensors, so you are saving a permutation inside the cudnn kernels.<br>
The last slide in <a href="https://developer.download.nvidia.com/video/gputechconf/gtc/2019/presentation/s9926-tensor-core-performance-the-ultimate-guide.pdf">this presentation</a> shows some numbers.</p>

fxmarty: <p>I don’t understand why NHWC layout helps for convolution layers in the im2col + GEMM implementation. In my opinion to fill the result matrix below, we rather need to iterate over HW first, no?</p>
<p>Reference image:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e946fa13dd593d9e27e9eb8b82a99d86ed6b05b6.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e946fa13dd593d9e27e9eb8b82a99d86ed6b05b6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e946fa13dd593d9e27e9eb8b82a99d86ed6b05b6_2_690x487.jpeg" alt="image" data-base62-sha1="xhFjHO2Y4qHfAVciryJwl4VRe3s" width="690" height="487" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e946fa13dd593d9e27e9eb8b82a99d86ed6b05b6_2_690x487.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e946fa13dd593d9e27e9eb8b82a99d86ed6b05b6_2_1035x730.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/9/e946fa13dd593d9e27e9eb8b82a99d86ed6b05b6.jpeg 2x" data-dominant-color="B2B1AC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1280×904 116 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

llseek: <p>Your reference image is one example, another possible layout after im2col looks like (only show 1st column):</p>
<p>1<br>
17<br>
33<br>
2<br>
18<br>
34<br>
5<br>
21<br>
37<br>
6<br>
22<br>
38</p>
<p>as long as filter has the same layout after im2col.<br>
With cuDnn’s terms, this is called the ‘CRS’ dimension. (C: Channel, R: kernel height, S: kernel width)</p>