Why pytorch changes strides of tensor after inference?

username: tejal567
cooked: <p>I have observed that strides of input and output tensors (just before and after network inference) are different. This behavior can be observed both in python and C++. I am not sure whether this is an inherent feature or a bug.</p>
<p>Example:<br>
input_Tensor size: [1, 3, 256, 256]<br>
input_Tensor strides: [256x256x3, 256x256, 256, 1]</p>
<p>output_Tensor size: [1, 3, 256, 256]<br>
output_Tensor strides: [256x256x3, 1 , 256x3 , 3]</p>
<p>This behavior is specially problematic when in C++ (libtorch) we try to set input Tensor data from a buffer like this:</p>
<pre><code class="lang-auto">torch::Tensor input_Tensor = torch::from_blob(input_buffer,{1,3,256,256},torch::kFloat);
</code></pre>
<p>and after inference try to copy output Tensor data to a buffer like this:</p>
<pre><code class="lang-auto">memcpy(output_buffer, (float*)output_Tensor.data_ptr(), 256*256*3*sizeof(float));
</code></pre>
<p>So, if strides of input_Tensor and output_Tensor differs like in above example, we will get wrong data filled in output_buffer. One way to resolve this is to make output_Tensor contiguous before doing memcpy.</p>

username: tejal567
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Please comment on this</p>

username: ptrblck
cooked: <p>I guess you are using the <code>channels_last</code> memory format, which would change the meta data as described <a href="https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html#beta-channels-last-memory-format-in-pytorch">here</a>.</p>

username: tejal567
cooked: <p>Yes, output_Tensor is in <code>channels_last</code> format but I have not explicitly defined anywhere to use <code>channels_last</code> memory format. It’s just that the input_Tensor is contiguous but output_Tensor is not. Pytorch (libtorch) automatically did it.</p>

username: ptrblck
cooked: <p>That shouldn’t happen. Could you post an executable code snippet to reproduce this issue?</p>

username: tejal567
cooked: <p>Please find below exact steps and code to reproduce this issue. In my original code, I have transferred my model weights from a caffe2 network, so below python code is simulating this thing. First we will create our model file from this python code:<br>
My execution environment:<br>
Python torch version: 1.7.0+cu101<br>
Libtorch version: 1.7.0 cuda 10.1</p>
<pre><code class="lang-auto">class TestNet(nn.Module):
    def __init__(self):
        super(TestNet, self).__init__()
        self.conv1 = nn.Conv2d(2,10,3,padding=1)

    def forward(self,x):
        x = self.conv1(x)
        return x

def get_arr():
    arr = np.random.rand(10,3,3,2)
    arr = np.transpose(arr,(0,3,1,2))
    return arr

model = network.TestNet()
arr = get_arr()
model._modules["conv1"].weight.data = torch.from_numpy(arr.astype(np.float32))
print (arr.shape)
print (arr.strides)
print (arr.data.contiguous)
model.eval()
arr = np.zeros((1,2,256,256)).astype(np.float32)
inp_T = torch.from_numpy(arr)
traced_script_module = torch.jit.trace(model, inp_T)
traced_script_module.save("TestNet.pt")
</code></pre>
<p>We have created our model file from above code, now we will execute below C++ code:</p>
<pre><code class="lang-auto">torch::jit::script::Module model = torch::jit::load("TestNet.pt");
model.to(at::kCUDA);
model.eval();
torch::NoGradGuard no_grad;
torch::Tensor tensor_in;
float* in_data = new float[2 * 256 * 256];
tensor_in = torch::from_blob(in_data, { 1, 2, 256, 256 }, torch::kFloat);
tensor_in = tensor_in.to(at::kCUDA);
tensor_in.set_requires_grad(0);
cout &lt;&lt; tensor_in.is_contiguous() &lt;&lt; endl;
cout &lt;&lt; tensor_in.strides() &lt;&lt; endl;
cout &lt;&lt; tensor_in.sizes() &lt;&lt; endl;
std::vector&lt;torch::jit::IValue&gt; inputs;
inputs.push_back(tensor_in);
torch::Tensor pred_out = model.forward(inputs).toTensor();
cout &lt;&lt; pred_out.is_contiguous() &lt;&lt; endl;
cout &lt;&lt; pred_out.strides() &lt;&lt; endl;
cout &lt;&lt; pred_out.sizes() &lt;&lt; endl;
</code></pre>
<p>Output of python code:</p>
<pre><code class="lang-auto">(10, 2, 3, 3)
(144, 8, 48, 16)
False
</code></pre>
<p>Output of C++ code:</p>
<pre><code class="lang-auto">1
131072,65536,256,1
1,2,256,256
0
655360,1,2560,10
1,10,256,256
</code></pre>
<p>We can see from above C++ output that although our input tensor is contiguous, output tensor is not. Interestingly, I found that get_arr() function in python code is responsible for all this. For example, if we replace above get_arr() function with below one, we see that now our output tensor is also contiguous.<br>
Replace above get_arr() with this:</p>
<pre><code class="lang-auto">def get_arr():
    arr = np.random.rand(10,3,3,2)
    arr = np.transpose(arr,(0,3,1,2))
    arr = np.ascontiguousarray(arr)
    return arr
</code></pre>
<p>Now. output of python code:</p>
<pre><code class="lang-auto">(10, 2, 3, 3)
(144, 72, 24, 8)
True
</code></pre>
<p>Output of C++ code:</p>
<pre><code class="lang-auto">1
131072,65536,256,1
1,2,256,256
1
655360,65536,256,1
1,10,256,256
</code></pre>
<p>Observe that our output tensor is contiguous now. How is this behaviour connected to our model weights? We can infer that if our model weights are not contiguous then output tensor is also not contiguous. But, how do we explain below behaviour then:<br>
If get_arr() is like this:</p>
<pre><code class="lang-auto">def get_arr():
    arr = np.random.rand(10,2,3,3)
    arr = np.transpose(arr,(0,1,3,2))
    return arr
</code></pre>
<p>Output of python code:</p>
<pre><code class="lang-auto">(10, 2, 3, 3)
(144, 72, 8, 24)
False
</code></pre>
<p>Output of C++ code:</p>
<pre><code class="lang-auto">1
131072,65536,256,1
1,2,256,256
1
655360,65536,256,1
1,10,256,256
</code></pre>
<p>So, we can see that in this case model weights are not contiguous but output tensor is contiguous. It would be great if someone can explain this weird behaviour.</p>

username: ptrblck
cooked: <p>Thanks for the update.<br>
I haven’t reproduced the complete issue in <code>libtorch</code>, since you are already manually manipulating the <code>.data</code> attribute (which is not recommended, as it could yield unwanted issues, so wrap the code in a <code>with torch.no_grad()</code> block and assign the new <code>nn.Parameter</code> directly if necessary) and set it into a channels-last format.<br>
As your Python script shows, <code>arr.data.contiguous</code> returns <code>False</code> and also:</p>
<pre><code class="lang-python">print(model.conv1.weight.stride())
&gt; (18, 1, 6, 2)
print(model.conv1.weight.is_contiguous())
&gt; False
print(model.conv1.weight.is_contiguous(memory_format=torch.channels_last))
&gt; True
</code></pre>
<p>shows that your manual assignment is setting the <code>weight</code> parameter to <code>channels_last</code>.</p>

username: tejal567
cooked: <p>Yes, it looks like so. Strides and contiguity of output tensor depends on model weights. Anyway, thanks Piotr for your help.</p>