Channels last question

michaelklachko: <p>I’m looking at <a href="https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html</a></p>
<p>Specifically, this example:</p>
<pre><code class="lang-auto">    input = torch.randint(1, 10, (2, 8, 4, 4), dtype=torch.float32, device="cuda", requires_grad=True)
    model = torch.nn.Conv2d(8, 5, 3).cuda().float()

    input = input.contiguous(memory_format=torch.channels_last)
    model = model.to(memory_format=torch.channels_last) # Module parameters need to be Channels Last

    out = model(input)
    print(out.is_contiguous(memory_format=torch.channels_last)) # Outputs: True
</code></pre>
<p>If I comment out the line that converts <code>model</code> to <code>channels_last</code> format, I expect it to fail, but it does not.<br>
How come <code>input</code> which are in <code>channel_last</code> format can be convolved with weight filters in <code>channel_first</code> format without dim mismatch error:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; model.weight.shape
torch.Size([5, 8, 3, 3])
&gt;&gt;&gt; model.weight.is_contiguous(memory_format=torch.channels_last)
False
&gt;&gt;&gt; input.shape
torch.Size([2, 8, 4, 4])
&gt;&gt;&gt; input.is_contiguous(memory_format=torch.channels_last)
True
</code></pre>
<p>My understanding is <code>input</code> actual shape in this case is <code>[2, 4, 4, 8]</code>, and therefore the weight input channels dim (8) should not match the <code>input</code> channels dim (4).</p>
<p><a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> ?</p>

ptrblck: <p>I think the input would be transformed to <code>channels_last</code> internally (I asked the same question recently and discussed it with one of the original authors), but let’s see if Vitaly can confirm it.</p>

michaelklachko: <p>This is my understanding as well. It looks like a bug to me, because if <code>input</code> is being permuted to <code>channels_last</code>, but <code>weight</code> remains <code>channels_first</code> then the code effectively becomes:</p>
<pre><code class="lang-auto">input = torch.randint(1, 10, (2, 4, 4, 8), dtype=torch.float32, device="cuda", requires_grad=True)
model = torch.nn.Conv2d(8, 5, 3).cuda().float()
out = model(input)
</code></pre>
<p>And it obviously fails because of the dim mismatch, but my example above runs fine even when <code>weight</code> remains in <code>channels_first</code> format. Something is not right here.</p>

ptrblck: <p>No, I don’t think it’s a bug, if PyTorch internally makes sure that the input and parameter uses the same memory layout.<br>
Also, the result is the same:</p>
<pre><code class="lang-python">input = torch.randint(1, 10, (2, 8, 4, 4), dtype=torch.float32, device="cuda", requires_grad=True)
model = torch.nn.Conv2d(8, 5, 3).cuda().float()

input = input.contiguous(memory_format=torch.channels_last)
model = model.to(memory_format=torch.channels_last) # Module parameters need to be Channels Last

out_ref = model(input)
print(out_ref.is_contiguous(memory_format=torch.channels_last))
&gt; True

model.to(memory_format=torch.contiguous_format)
out = model(input)

print(out.is_contiguous(memory_format=torch.channels_last))
&gt; True

print((out_ref - out).abs().max())
&gt; tensor(0., device='cuda:0', grad_fn=&lt;MaxBackward1&gt;)
</code></pre>

michaelklachko: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="106279">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>PyTorch internally makes sure that the input and parameter uses the same memory layout.</p>
</blockquote>
</aside>
<p>What does it mean? To me it seems when the layout is different, the op should fail with a “memory layout mismatch” error message. Otherwise, how does the Pytorch know which layout is the correct one? Does it convert <code>weight</code> to <code>channel_last</code>, to match <code>input</code>, or does it convert <code>input</code> to <code>channel_first</code> to match weight? If it’s always the former, what’s the point of <code>model=model.to(memory_format=torch.channels_last)</code> line?</p>

ptrblck: <p>This might have been one possible approach, but would potentially break ambiguous memory layout checks, such as <code>1x1</code> kernels.<br>
The current implementations checks, if the suggested memory format of the <code>input</code> or <code>weight</code> is <code>channels_last</code> (<a href="https://github.com/pytorch/pytorch/blob/22a34bcf4e5eaa348f0117c414c3dd760ec64b13/aten/src/ATen/native/ConvUtils.h#L92-L94">code</a>) and should use it, if applicable.</p>

michaelklachko: <p>I see. So does this mean that, in the tutorial example, it’s <em>not</em> necessary to set <code>input</code> to <code>channels_last</code> - it’s redundant because setting the model to <code>channels_last</code> forces it anyway. Is there any situation where we would want to set the inputs in addition to setting the model?</p>

ptrblck: <p>I would always set the memory layout explicitly, not rely on the internal workflow of fixing ambiguous memory layouts, and would stick to the tutorial.</p>

michaelklachko: <p>Makes sense, thank you Piotr. I have to say, this behavior will cause confusion, because people will forget to set one or the other, see that it still works, and will wonder whether it actually worked as intended. At the very least I’d mention something about this in the tutorial.</p>

Etienne_Perot: <p>Is it possible to run a Conv2d on a NHWC tensor (not NCHW)? it seems that if I put channels explicitely at the end of <a class="mention" href="/u/michaelklachko">@michaelklachko</a> example and keep “channels_last” behavior it does not work.</p>

michaelklachko: <p>Not sure what you mean by “put channels explicitly”. If you mean you manually transposed the tensor so that channels dim is the last one, then yes, it will fail, because “channels_last” feature is designed to work with tensors in their default shapes. The transformation happens internally.</p>
<p>Like I said, this is going to be confusing to people until the docs and error messages are improved.</p>

Etienne_Perot: <p>yes i agree and yes this is what i meant. It would be easier to allow conv2d operator to run on NHWC tensors.</p>

emilyfy: <p>hello, I know this discussion is more than a year old but I just came across this as I’m facing an issue that I think is relevant to this.</p>
<p>I couldn’t reproduce some output and when I dug into it found out that the reason was because in one case the model.to(memory_format=torch.channels_last) is set but in another case it’s not. Now, according to this discussion it shouldn’t matter and pytorch should internally take care of it, but that’s not what I’m observing.</p>
<p>Here’s a sample code</p>
<pre><code class="lang-auto">input = torch.randn(1, 1, 100, 100)
conv = torch.nn.Conv2d(1, 64, kernel_size=(3, 3), padding=(1, 1), bias=False)
conv.load_state_dict({"weight": torch.randn(64, 1, 3, 3)})
with torch.no_grad():
    out_ref = conv(input)

conv.to(memory_format=torch.channels_last)
with torch.no_grad():
    out = conv(input)

print(torch.mean(torch.abs(out - out_ref)))
</code></pre>
<p>it should print some non-zero value.</p>
<p>Interestingly, if the input channel is 3 instead of 1, it will be correct. Just wondering if this could be some bug? I know the proper way is to set the input memory format to also be channels_last, but this is some online repo that I’m looking at instead of my own code, so I’m just wondering if this behavior is intended by pytorch / the author or if it’s a bug that the author isn’t aware of, in which case I should raise it with them.</p>

michaelklachko: <p><a class="mention" href="/u/emilyfy">@emilyfy</a> how large are the differences?</p>

emilyfy: <p><a class="mention" href="/u/michaelklachko">@michaelklachko</a> it’s pretty significant, as in it’s definitely not a precision issue.<br>
error mean: tensor(3.1527), and<br>
error sum: tensor(2017749.7500)</p>

emilyfy: <p>also, I noticed this only happens in CPU, if I add .cuda() to both input and conv, the error is 0.</p>

michaelklachko: <p>Looks like a bug. Can you please create an issue on github?</p>

emilyfy: <p>done, <a href="https://github.com/pytorch/pytorch/issues/82060" class="inline-onebox" rel="noopener nofollow ugc">Wrong output of single-channel channels_last convolution with channels_first input · Issue #82060 · pytorch/pytorch · GitHub</a> thanks!</p>

deJQK: <p>I tried to use channels last memory format but seems it will give a different result. I am using cudnn 8.6.0, cuda 11.7, python 3.8.16, and pytorch 2.0.1+cu117. Here are the results:</p>
<pre><code class="lang-auto">import copy

import torch
import torch.nn as nn


x = torch.randn(64, 4, 64, 64).cuda()
layer = nn.Conv2d(4, 128, 3).cuda()

y = layer(x)

x_mem = (x + 0).to(memory_format=torch.channels_last)
layer_mem = copy.deepcopy(layer).to(memory_format=torch.channels_last)
y_mem = layer_mem(x_mem)

x_half = (x + 0).half()
layer_half = copy.deepcopy(layer).half()
y_half = layer_half(x_half)

x_half_mem = (x_half + 0).to(memory_format=torch.channels_last)
layer_half_mem = copy.deepcopy(layer_half).to(memory_format=torch.channels_last)
y_half_mem = layer_half_mem(x_half_mem)

print('x - x_half: ', (x - x_half).abs().max().item())
print('x - x_mem: ', (x - x_mem).abs().max().item())
print('x - x_half_mem: ', (x - x_half_mem).abs().max().item())
print('y - y_half: ', (y - y_half).abs().max().item())
print('y - y_mem: ', (y - y_mem).abs().max().item())
print('y_half - y_half_mem: ', (y_half - y_half_mem).abs().max().item())

</code></pre>
<p>Output:</p>
<blockquote>
<p>x - x_half:  0.0018982887268066406<br>
x - x_mem:  0.0<br>
x_half - x_half_mem:  0.0<br>
y - y_half:  0.0024781227111816406<br>
y - y_mem:  0.0012029409408569336<br>
y_half - y_half_mem:  0.001953125</p>
</blockquote>
<p>The differences between <code>y</code> and <code>y_mem</code> or <code>y_half</code> and <code>y_half_mem</code> are not negligible.</p>

ptrblck: <aside class="quote no-group" data-username="deJQK" data-post="19" data-topic="106279">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dejqk/48/4965_2.png" class="avatar"> deJQK:</div>
<blockquote>
<p>The differences between <code>y</code> and <code>y_mem</code> or <code>y_half</code> and <code>y_half_mem</code> are not negligible.</p>
</blockquote>
</aside>
<p>The posted differences are expected.<br>
In your code you are directly calling <code>half()</code> on the inputs and layer to compare <code>y_half</code> against <code>y_half_mem</code>, which shows a mismatch in the expected range for <code>float16</code>.<br>
Also, you are not disallowing TF32 and are then comparing <code>y</code> vs. <code>y_mem</code>.<br>
Disable it via <code>torch.backends.cudnn.allow_tf32 = False</code> and the numerical mismatch will reduce.</p>