Torch::jit::load -> C++ vs Python

BiPaulEr: <p>I used torch.jit.load with python and  torch::jit::load with C++ (each 1.9.1 cu11.1).<br>
The model that is transcripted is TransfomerNet as decribed in the following.<br>
The results from python and C++ are differents as described by the images.</p>
<p>Python Result<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96dd8e07a119b1e82efa459973af4c85146c7cb7.jpeg" alt="Transformer" data-base62-sha1="lwCizs5EESle4RVw7YI6bn3UmJF" width="641" height="363"><br>
C++ Result<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b7758778f75f62f4475938a9edc1ba6f156401e.jpeg" alt="Transformer_v6" data-base62-sha1="1DqWPwKvCEVdLuLYDU7eE2DOHV4" width="641" height="363"></p>
<blockquote>
<p>Blockquote<br>
class TransformerNetwork(nn.Module):</p>
</blockquote>
<pre><code>"""Feedforward Transformation Network without Tanh

reference: https://arxiv.org/abs/1603.08155 

exact architecture: https://cs.stanford.edu/people/jcjohns/papers/fast-style/fast-style-supp.pdf

"""

def __init__(self, ConvLayer, ResidualLayer, DeconvLayer):

    super(TransformerNetwork, self).__init__()

    self.ConvBlock = nn.Sequential(

        ConvLayer(3, 32, 9, 1),

        nn.ReLU(),

        ConvLayer(32, 64, 3, 2),

        nn.ReLU(),

        ConvLayer(64, 128, 3, 2),

        nn.ReLU()

    )

    self.ResidualBlock = nn.Sequential(

        ResidualLayer(128, 3), 

        ResidualLayer(128, 3), 

        ResidualLayer(128, 3), 

        ResidualLayer(128, 3), 

        ResidualLayer(128, 3)

    )

    self.DeconvBlock = nn.Sequential(

        DeconvLayer(128, 64, 3, 2, 1),

        nn.ReLU(),

        DeconvLayer(64, 32, 3, 2, 1),

        nn.ReLU(),

        ConvLayer(32, 3, 9, 1, norm="None")

    )

def forward(self, x):

    x = self.ConvBlock(x)

    x = self.ResidualBlock(x)

    out = self.DeconvBlock(x)

    return out
</code></pre>
<p>class ConvLayer(nn.Module):</p>
<pre><code>def __init__(self, in_channels, out_channels, kernel_size, stride, norm="instance"):

    super(ConvLayer, self).__init__()

    # Padding Layers

    padding_size = kernel_size // 2

    self.reflection_pad = nn.ReflectionPad2d(padding_size)

    # Convolution Layer

    self.conv_layer = nn.Conv2d(in_channels, out_channels, kernel_size, stride)

    # Normalization Layers

    self.norm_type = norm

    if (norm=="instance"):

        self.norm_layer = nn.InstanceNorm2d(out_channels, affine=True)

    elif (norm=="batch"):

        self.norm_layer = nn.BatchNorm2d(out_channels, affine=True)

def forward(self, x):

    x = self.reflection_pad(x)

    x = self.conv_layer(x)

    if (self.norm_type=="None"):

        out = x

    else:

        out = self.norm_layer(x)

    return out
</code></pre>
<p>class ResidualLayer(nn.Module):</p>
<pre><code>"""

Deep Residual Learning for Image Recognition

https://arxiv.org/abs/1512.03385

"""

def __init__(self, channels=128, kernel_size=3):

    super(ResidualLayer, self).__init__()

    self.conv1 = ConvLayer(channels, channels, kernel_size, stride=1)

    self.relu = nn.ReLU()

    self.conv2 = ConvLayer(channels, channels, kernel_size, stride=1)

def forward(self, x):

    identity = x                     # preserve residual

    out = self.relu(self.conv1(x))   # 1st conv layer + activation

    out = self.conv2(out)            # 2nd conv layer

    out = out + identity             # add residual

    return out
</code></pre>
<p>class DeconvLayer(nn.Module):</p>
<pre><code>def __init__(self, in_channels, out_channels, kernel_size, stride, output_padding, norm="instance"):

    super(DeconvLayer, self).__init__()

    # Transposed Convolution 

    padding_size = kernel_size // 2

    self.conv_transpose = nn.ConvTranspose2d(in_channels, out_channels, kernel_size, stride, padding_size, output_padding)

    # Normalization Layers

    self.norm_type = norm

    if (norm=="instance"):

        self.norm_layer = nn.InstanceNorm2d(out_channels, affine=True)

    elif (norm=="batch"):

        self.norm_layer = nn.BatchNorm2d(out_channels, affine=True)

def forward(self, x):

    x = self.conv_transpose(x)

    if (self.norm_type=="None"):

        out = x

    else:

        out = self.norm_layer(x)

    return out
</code></pre>
<blockquote>
<p>Blockquote</p>
</blockquote>

ptrblck: <p>The C++ result looks interleaved, which is often caused by a wrongly used <code>view</code> or <code>reshape</code> where a <code>permute</code> would be needed.<br>
I don’t see any data processing code posted, but I would guess that you are using <code>PIL</code> in Python to load the image (channels-first by default) and OpenCV in C++ (channels-last). In C++ you might have been running into a shape mismatch error and tried to use <code>view/reshape</code> to create an input tensor of <code>[batch_size, channels, height, width]</code>. If so, use <code>permute</code> to permute the axes to create a channels-first tensor.</p>

BiPaulEr: <p>Thanks for your reply.<br>
Unfortunately, the input input tensor is <code>[batch_size, channels, height, width]</code> and I used <code>permute</code> to achieve it.<br>
I used the same pipeline with other C++ network and it’s worked well. That’s why I asked this question. I can’t understand where the issue comes from. I supposed it may come from the transcription from the python network to c++ network but it works when I used torch.jit.load.<br>
For the transcription, I used only trace. The <code>.graph</code> and the .<code>code</code> do not help me well.<br>
It may come from <code>dtype</code> issue. I’ll investigate on this.</p>

ptrblck: <p>That’s weird, as I’m pretty sure you are displaying an interleaved image.<br>
You can easily reproduce it by loading the first (properly displayed) image and by running this code:</p>
<pre><code class="lang-python">import PIL
import numpy as np
import matplotlib.pyplot as plt

img = PIL.Image.open("./discuss01.jpeg")
arr = np.array(img)
shape = arr.shape
print(shape)
plt.imshow(arr)

arr_interleaved = arr.transpose(2, 0, 1).reshape(shape)
plt.imshow(arr_interleaved)
</code></pre>
<p>which will result exactly in the interleaved second output you’ve posted.</p>

BiPaulEr: <p>Thanks, I run your code and it seems to be like I have. So my pipeline has an issue :<br>
<code> tensor = torch::from_blob(open_cv.data, { 1, height, width, channels = 3 }, at::kByte).to(at::kFloat).div(255).permute({ 0, 3, 1, 2 }))</code> gives a Tensor of dimension <code>{1, 3, height, width}</code>.</p>
<p>Then I do :<br>
<code>model.forward({ tensor }).toTensor().to(at::kByte).squeeze().permute({ 1,2,0 })</code> to have a tensor of dimension  <code>{height, width, 3}</code> for OpenCV displaying.<br>
When I do not do <code>model.forward</code> the result is normal.</p>

BiPaulEr: <p>With your help I investigate in the right direction. The problem is solved when I use <code>contiguous()</code> on the net result. I need to understand more the methods that modifies the memory and the others that is just a “view” of it.<br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/muscle.png?v=10" title=":muscle:" class="emoji" alt=":muscle:"></p>