Values after softmax in torchscript in c++ is not same as python torch

username: Aditya_Kumar
cooked: <pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;torch/script.h&gt; // One-stop header.
#include &lt;iostream&gt;
#include &lt;memory&gt;
#include &lt;opencv2/core/core.hpp&gt;
#include &lt;opencv2/imgproc/imgproc.hpp&gt;
#include &lt;opencv2/highgui/highgui.hpp&gt;

using namespace cv;
using namespace std;

int main() {
    
    std::cout &lt;&lt; std::fixed &lt;&lt; std::setprecision(4);
    torch::jit::script::Module module = torch::jit::load("/data2/aditya/classifiers/LHSRHS/lhsrhs_script.zip");
    torch::data::transforms::Normalize&lt;&gt; normalize_transform({0.485, 0.456, 0.406}, {0.229, 0.224, 0.225});
    
        
    Mat image_bgr, image, image1;
    image_bgr = imread("/data2/aditya/classifiers/LHSRHS/new_data801/RHS/100_5751_rhs_front_door_assy.png");
    cvtColor(image_bgr, image, COLOR_BGR2RGB);
    resize(image, image1, {448, 448});
    
//     cout&lt;&lt;image1&lt;&lt;'\n';
    
    torch::Tensor tensor_image = torch::from_blob(image1.data, {3, image1.rows, image1.cols}, at::kByte);
    tensor_image = tensor_image.to(torch::kFloat32).div_(255);
    cout&lt;&lt;tensor_image.sizes()&lt;&lt;'\n';
    
    module.eval();
    torch::Tensor tensor_image1 = normalize_transform(tensor_image).unsqueeze_(0);

    std::vector&lt;torch::jit::IValue&gt; input;
    input.push_back(tensor_image1);
    
    at::Tensor output = module.forward(tensor_image1).toTensor();
    
    at::Tensor output1 = torch::softmax(output, 1);
    
    std::cout&lt;&lt;output1&lt;&lt;'\n';
    
}
    
</code></pre>
<p>I trained a binary classifier in fastai and then using <code>torch.jit.trace</code>, I converted the model to torchscript and then I am loading the image and model in c++. But the predictions and the values which I am getting after softmax in c++ are very different and inaccurate as compared to torch in python. And I am not getting what is the problem. So, please check where I am going wrong.</p>

username: dambo
cooked: <aside class="quote no-group" data-username="Aditya_Kumar" data-post="1" data-topic="67947">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aditya_kumar/48/6715_2.png" class="avatar"> Aditya_Kumar:</div>
<blockquote>
<p>normalize_transform</p>
</blockquote>
</aside>
<p>Where is the python code?<br>
Are you sure you are using the same transforms in both cases?</p>

username: Aditya_Kumar
cooked: <p><a class="mention" href="/u/dambo">@dambo</a> thank you for your response.<br>
Yes, I am sure. The code for inference which I am using in python is:-</p>
<pre><code class="lang-auto">learn = load_learner("/data2/aditya/classifiers/LHSRHS/new_data801/", "final_lhs_rhs_model.pkl")
my_model = learn.model.cpu()
softmaxer = torch.nn.Softmax(dim=1)
my_model.eval()
image = Image.open("/data2/aditya/classifiers/LHSRHS/new_data801/RHS/100_5751_rhs_front_door_assy.png")
image = image.resize((448, 448))
x = TTF.to_tensor(image)
x = TTF.normalize(x, [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
x.unsqueeze_(0)
print(x.size())
raw_out = my_model(x)
print(softmaxer(raw_out))

traced_cell = torch.jit.trace(my_model, x)
traced_cell.save('lhsrhs_script.zip')
</code></pre>
<p>I think there is some problem with normalize_transform function in c++ but I don’t know what it is because when I am printing <code>tensor_image1</code> in c++ the values are not coming as expected.</p>

username: ptrblck
cooked: <p>If I’m not mistaken, OpenCV reads the image as <code>[height, width, channels]</code>, while <code>PIL.Image</code> returns the array as <code>[channels, height, width]</code>.<br>
The <code>torch::from_blob</code> call might thus interleave the pixels. Have you checked the outputs?</p>
<p>Also, <code>resize</code> should use the linear interpolation by default in OpenCV while PIL uses nearest neighbors by default.</p>
<p>Could you check the results separately?</p>

username: dambo
cooked: <p>Try reading your image like this:</p><aside class="quote quote-modified" data-post="6" data-topic="47701">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dambo/48/29088_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/libtorch-c-convert-a-tensor-to-cv-mat-single-channel/47701/6">Libtorch C++ convert a Tensor to cv:Mat (single channel</a> <a class="badge-category__wrapper " href="/c/c/11"><span data-category-id="11" data-drop-close="true" class="badge-category " title="Topics related to the C++ Frontend, C++ API or C++ Extensions"><span class="badge-category__name">C++</span></span></a>
  </div>
  <blockquote>
    From cv2 to TORCH: 
 cv::cvtColor(frame, frame, CV_BGR2RGB);
 frame.convertTo(frame, CV_32FC3, 1.0f / 255.0f);
 auto input_tensor = torch::from_blob(frame.data, {1, frame_h, frame_w, kCHANNELS});
 input_tensor = input_tensor.permute({0, 3, 1, 2});

From TORCh to cv2: 
input_tensor = input_tensor.to(at::kCUDA);
                torch::Tensor out_tensor = module-&gt;forward({input_tensor}).toTensor();

                out_tensor = out_tensor.squeeze().detach().permute({1, 2, 0});
                out_t…
  </blockquote>
</aside>

<p>For the sake of testing, <strong>DO NOT</strong> use normalisation either in Python nor C++ and see if you get teh same results.</p>

username: Aditya_Kumar
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/dambo">@dambo</a> Thanks for your answer. Difference in values after softmax was due to error in image loading. Now, I am getting  approximately same values after softmax in python and C++ both.</p>

username: solarflarefx
cooked: <p><a class="mention" href="/u/aditya_kumar">@Aditya_Kumar</a> what was the specific source of the problem?  Were you able to get the same output with normalization?  I am having a problem getting the same output and I am trying to find the source of my problem.</p>

username: Aditya_Kumar
cooked: <p><a class="mention" href="/u/solarflarefx">@solarflarefx</a> There was some problem with image loading, I was loading image using OpenCV and OpenCV reads the image as  <code>[height, width, channels]</code> and but for <code>torch::from_blob</code> it should be <code>[channels, height, width]</code>. But after correcting this I was getting, approximately same result. Yes, I was using normalization and the getting same answer without normalization I couldn’t get. If you can post your code then I think I can help you with resolving your query.</p>

username: solarflarefx
cooked: <p><a class="mention" href="/u/aditya_kumar">@Aditya_Kumar</a>  I see.  I thought I accounted for this in my code.  But I am putting it below just to make sure.  I am simply loading the ResNet18 model, and loading a sample image: <a href="https://raw.githubusercontent.com/pytorch/hub/master/dog.jpg" rel="nofollow noopener">https://raw.githubusercontent.com/pytorch/hub/master/dog.jpg</a></p>
<p>Python Code:</p>
<pre><code class="lang-auto">import torch
import torchvision.models as models
import urllib.request
from PIL import Image
from torchvision import transforms

def norm_chan(chan, mean, std):
    b = (chan - mean) / std
    return b

# Load resnet18 model
model = models.resnet18(pretrained = True)

# Download an example image from the pytorch website
url, filename = ("https://github.com/pytorch/hub/raw/master/dog.jpg", "dog.jpg")
urllib.request.urlretrieve(url, filename)

# sample execution (requires torchvision)
input_image = Image.open(filename)
preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

input_tensor = preprocess(input_image)

import numpy as np
input_tensor_numpy = input_tensor.numpy()
input_batch = input_tensor.unsqueeze(0) # create a mini-batch as expected by the model


# move the input and model to GPU for speed if available
if torch.cuda.is_available():
    input_batch = input_batch.to('cuda')
    model.to('cuda')

with torch.no_grad():
    output = model(input_batch)

import numpy as np
print()
output_cpu = output[0].cpu()
output_numpy = output_cpu.numpy()
print(output_numpy[0:9])
</code></pre>
<p>C++ Code:</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt;
#include &lt;torch/torch.h&gt;
#include &lt;ATen/Tensor.h&gt;
#include &lt;opencv2/opencv.hpp&gt;
#include &lt;opencv2/imgproc/imgproc.hpp&gt;
#include &lt;opencv2/core/core.hpp&gt;
#include &lt;opencv2/highgui/highgui.hpp&gt;
#include &lt;iostream&gt;
#include &lt;memory&gt;

int main(int argc, const char* argv[]) {


    // Load up GPU stuff
    torch::DeviceType device_type;
    if (torch::cuda::is_available()) {
        std::cout &lt;&lt; "CUDA available! Training on GPU." &lt;&lt; std::endl;
        device_type = torch::kCUDA;
    }
    else {
        std::cout &lt;&lt; "Training on CPU." &lt;&lt; std::endl;
        device_type = torch::kCPU;
    }
    torch::Device device(device_type);

    torch::jit::script::Module module;
    std::cout &lt;&lt; "Attempting to load resnet model.." &lt;&lt; std::endl;
    try {
        // Deserialize the ScriptModule from a file using torch::jit::load().
        module = torch::jit::load("C:\\PyTorchPictureTest\\Model\\traced_resnet_model.pt");
        std::cout &lt;&lt; "Successfully loaded resnet model" &lt;&lt; std::endl;

        module.to(at::kCUDA);
        std::cout &lt;&lt; "Moved model to gpu" &lt;&lt; std::endl;

        // load image and transform
        cv::Mat image;
        image = cv::imread("C:\\PyTorchPictureTest\\dog.jpg", 1);

        cv::cvtColor(image, image, cv::COLOR_BGR2RGB); 
        cv::Mat img_float;
        image.convertTo(img_float, CV_32F, 1.0 / 255);
        cv::resize(img_float, img_float, cv::Size(224, 224), cv::INTER_NEAREST);

        auto img_tensor = torch::from_blob(img_float.data, { 1, 224, 224, 3 }).to(torch::kCUDA);
        img_tensor = img_tensor.permute({ 0, 3, 1, 2 });
        img_tensor[0][0] = img_tensor[0][0].sub(0.485).div(0.229);
        img_tensor[0][1] = img_tensor[0][1].sub(0.456).div(0.224);
        img_tensor[0][2] = img_tensor[0][2].sub(0.406).div(0.225);
        auto img_var = torch::autograd::make_variable(img_tensor, false);

        std::vector&lt;torch::jit::IValue&gt; inputs;
        inputs.push_back(img_var);
        torch::Tensor out_tensor = module.forward(inputs).toTensor();
        std::cout &lt;&lt; out_tensor.slice(1, 0, 10) &lt;&lt; '\n';
    }
    catch (const c10::Error &amp; e) {
        std::cerr &lt;&lt; "error loading the model\n";
        return -1;
    }

    std::cout &lt;&lt; "ok\n";
}
</code></pre>
<p>Is there a good way to probe intermediate points in the code?  It’s easy to print tensor slices in Python.  How would I do this in C++?  Would it be using the narrow method?  Maybe I could do it on my img_tensor in C++:</p>
<pre><code class="lang-auto">auto img_tensor_cpu = img_tensor.to(torch::kCPU);
auto img_tensor_cpu_sliced = img_tensor_cpu.narrow(2, 0, 1);
std::cout &lt;&lt; '\n';
std::cout &lt;&lt; img_tensor_cpu_sliced &lt;&lt; '\n';
</code></pre>
<p>I believe the equivalent in my Python code would be:</p>
<pre><code class="lang-auto">print(torch.narrow(input_tensor, 2, 0, 1))
</code></pre>
<p>I don’t be getting the same values here.</p>

username: Aditya_Kumar
cooked: <p>I think you are doing wrong in this line <code>auto img_tensor = torch::from_blob(img_float.data, { 1, 224, 224, 3 }).to(torch::kCUDA);</code>. Because before this dimensions of your image are still [height, width, channels] then how you added an extra dimension in  { 1, 224, 224, 3 },  so first load your image by replacing it with only {224, 224, 3} and then before passing image to model just use this <code>unsqueeze_(0)</code>. For further reference, you can see my code <a href="https://github.com/adityak2920/Traning-and-Inference/blob/master/classifier.cpp" rel="nofollow noopener">here</a>.</p>
<p>You can also torchscript transform template also in c++.</p>

username: solarflarefx
cooked: <p><a class="mention" href="/u/aditya_kumar">@Aditya_Kumar</a><br>
Thanks for your reply.  So I tried following your code structure:</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt;
#include &lt;torch/torch.h&gt;
#include &lt;ATen/Tensor.h&gt;
#include &lt;opencv2/opencv.hpp&gt;
#include &lt;opencv2/imgproc/imgproc.hpp&gt;
#include &lt;opencv2/core/core.hpp&gt;
#include &lt;opencv2/highgui/highgui.hpp&gt;
#include &lt;iostream&gt;
#include &lt;memory&gt;
#include &lt;iomanip&gt;

std::string type2str(int type) {
    std::string r;

    uchar depth = type &amp; CV_MAT_DEPTH_MASK;
    uchar chans = 1 + (type &gt;&gt; CV_CN_SHIFT);

    switch (depth) {
    case CV_8U:  r = "8U"; break;
    case CV_8S:  r = "8S"; break;
    case CV_16U: r = "16U"; break;
    case CV_16S: r = "16S"; break;
    case CV_32S: r = "32S"; break;
    case CV_32F: r = "32F"; break;
    case CV_64F: r = "64F"; break;
    default:     r = "User"; break;
    }

    r += "C";
    r += (chans + '0');

    return r;
}

int main(int argc, const char* argv[]) {


    // Load up GPU stuff
    torch::DeviceType device_type;
    if (torch::cuda::is_available()) {
        std::cout &lt;&lt; "CUDA available! Training on GPU." &lt;&lt; std::endl;
        device_type = torch::kCUDA;
    }
    else {
        std::cout &lt;&lt; "Training on CPU." &lt;&lt; std::endl;
        device_type = torch::kCPU;
    }
    torch::Device device(device_type);
    torch::data::transforms::Normalize&lt;&gt; normalize_transform({ 0.485, 0.456, 0.406 }, { 0.229, 0.224, 0.225 });
    torch::jit::script::Module module;
    std::cout &lt;&lt; "Attempting to load resnet model.." &lt;&lt; std::endl;
    try {
        // Deserialize the ScriptModule from a file using torch::jit::load().
        module = torch::jit::load("C:\\PyTorchPictureTest\\Model\\traced_resnet_model.pt");
        std::cout &lt;&lt; "Successfully loaded resnet model" &lt;&lt; std::endl;

        module.to(at::kCUDA);
        std::cout &lt;&lt; "Moved model to gpu" &lt;&lt; std::endl;

        // load image and transform
        cv::Mat image;
        image = cv::imread("C:\\PyTorchPictureTest\\dog.jpg", cv::IMREAD_COLOR);

        std::cout &lt;&lt; image.size &lt;&lt; "\n" &lt;&lt; std::endl;
        std::cout &lt;&lt; image.cols &lt;&lt; "\n" &lt;&lt; std::endl;
        std::cout &lt;&lt; image.rows &lt;&lt; "\n" &lt;&lt; std::endl;
        std::cout &lt;&lt; type2str(image.type()) &lt;&lt; "\n" &lt;&lt; std::endl;

        cv::cvtColor(image, image, cv::COLOR_BGR2RGB);
        cv::Mat img_float;
        image.convertTo(img_float, CV_32FC3, 1.0f / 255.0f);
        cv::resize(img_float, img_float, cv::Size(224, 224), cv::INTER_NEAREST);

        auto img_tensor = torch::from_blob(img_float.data, { img_float.rows, img_float.cols, 3 }).to(torch::kCUDA);
        img_tensor = img_tensor.permute({ 2, 0, 1 });
        std::cout &lt;&lt; img_tensor.sizes() &lt;&lt; '\n';
        
        torch::Tensor img_tensor_norm = normalize_transform(img_tensor).unsqueeze_(0);

        std::vector&lt;torch::jit::IValue&gt; inputs;
        inputs.push_back(img_tensor_norm.to(at::kCUDA));
        torch::Tensor out_tensor = module.forward(inputs).toTensor();
        std::cout &lt;&lt; out_tensor &lt;&lt; '\n';
    }
    catch (const c10::Error &amp; e) {
        std::cerr &lt;&lt; "error loading the model\n";
        return -1;
    }

    std::cout &lt;&lt; "ok\n";
}
</code></pre>
<p>I seem to be getting an error on this line:</p>
<pre><code class="lang-auto">torch::Tensor img_tensor_norm = normalize_transform(img_tensor).unsqueeze_(0);
</code></pre>
<p>Error:<br>
&lt;Information not available, no symbols loaded for c10.dll&gt;<br>
“expected device cuda:0 but got device cpu”</p>

username: solarflarefx
cooked: <p>Sorry my fault.  The problem line was:</p>
<pre><code class="lang-auto">auto img_tensor = torch::from_blob(img_float.data, { img_float.rows, img_float.cols, 3 }).to(torch::kCUDA);
</code></pre>
<p>I changed this to:</p>
<pre><code class="lang-auto">auto img_tensor = torch::from_blob(img_float.data, { img_float.rows, img_float.cols, 3 });
</code></pre>
<p>This prevented the error from happening.  However, I still do not get the same output.</p>
<p>Python:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/963e7aac1ecdf4c1eee1d8db3717ebad3d2b5c1d.png" data-download-href="https://discuss.pytorch.org/uploads/default/963e7aac1ecdf4c1eee1d8db3717ebad3d2b5c1d" title="Python"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/6/963e7aac1ecdf4c1eee1d8db3717ebad3d2b5c1d.png" alt="Python" data-base62-sha1="lr7tRjEgCQoDFNxfqC2c5f04GWF" width="690" height="342" data-dominant-color="1C1C1C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Python</span><span class="informations">973×483 17.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Cpp:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1f4598ba31d634ce0459b4a4dbb387be1da3dd30.png" data-download-href="https://discuss.pytorch.org/uploads/default/1f4598ba31d634ce0459b4a4dbb387be1da3dd30" title="Cpp"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1f4598ba31d634ce0459b4a4dbb387be1da3dd30.png" alt="Cpp" data-base62-sha1="4sDUwa3FC0wlAOIXl86p2Rpb12o" width="690" height="360" data-dominant-color="232323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Cpp</span><span class="informations">979×512 14.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I suppose at this point it could be that my input tensors are not the same, or that I have made an error exporting to Torchscript (which I doubt, since I simply followed the example in the official documentation).</p>

username: solarflarefx
cooked: <p>Ok, so I believe I found my source of error.  In Python before I serialized and saved the model, I forgot to specify to load the <em>pretrained</em> network.</p>
<p>Updated Python Code:</p>
<pre><code class="lang-auto">import torch
import torchvision.models as models
import urllib.request
from PIL import Image
from torchvision import transforms
import numpy as np

def norm_chan(chan, mean, std):
    b = (chan - mean) / std
    return b

# Load resnet18 model
model = models.resnet18(pretrained = True)

serialize_model = True
if (serialize_model):
    example = torch.rand(1, 3, 224, 224)
    traced_script_module = torch.jit.trace(model, example)
    traced_script_module.save("traced_resnet_model_pretrained.pt")

# sample execution (requires torchvision)
input_image = Image.open("C:\\PyTorchPictureTest\\dog.jpg")
np_im = np.array(input_image)

preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

input_tensor = preprocess(input_image)
input_batch = input_tensor.unsqueeze(0) # create a mini-batch as expected by the model

# move the input and model to GPU for speed if available
if torch.cuda.is_available():
    input_batch = input_batch.to('cuda')
    model.to('cuda')

with torch.no_grad():
    output = model(input_batch)

import numpy as np
print()
output_cpu = output[0].cpu()
output_numpy = output_cpu.numpy()
for i in range(0, len(output_numpy)):
    if ( i%10 == 0):
        print("\n")
    print(str(output_numpy[i]), end = " ")
print()
</code></pre>
<p>Cpp Code:</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt;
#include &lt;torch/torch.h&gt;
#include &lt;opencv2/opencv.hpp&gt;
#include &lt;opencv2/imgproc/imgproc.hpp&gt;
#include &lt;opencv2/core/core.hpp&gt;
#include &lt;opencv2/highgui/highgui.hpp&gt;
#include &lt;iostream&gt;
#include &lt;memory&gt;
#include &lt;iomanip&gt;

std::string type2str(int type) {
    std::string r;

    uchar depth = type &amp; CV_MAT_DEPTH_MASK;
    uchar chans = 1 + (type &gt;&gt; CV_CN_SHIFT);

    switch (depth) {
    case CV_8U:  r = "8U"; break;
    case CV_8S:  r = "8S"; break;
    case CV_16U: r = "16U"; break;
    case CV_16S: r = "16S"; break;
    case CV_32S: r = "32S"; break;
    case CV_32F: r = "32F"; break;
    case CV_64F: r = "64F"; break;
    default:     r = "User"; break;
    }

    r += "C";
    r += (chans + '0');

    return r;
}

int main(int argc, const char* argv[]) {


    // Load up GPU stuff
    torch::DeviceType device_type;
    if (torch::cuda::is_available()) {
        std::cout &lt;&lt; "CUDA available! Training on GPU." &lt;&lt; std::endl;
        device_type = torch::kCUDA;
    }
    else {
        std::cout &lt;&lt; "Training on CPU." &lt;&lt; std::endl;
        device_type = torch::kCPU;
    }
    torch::Device device(device_type);
    torch::data::transforms::Normalize&lt;&gt; normalize_transform({ 0.485, 0.456, 0.406 }, { 0.229, 0.224, 0.225 });
    torch::jit::script::Module module;
    std::cout &lt;&lt; "Attempting to load resnet model.." &lt;&lt; std::endl;
    try {
        // Deserialize the ScriptModule from a file using torch::jit::load().
        module = torch::jit::load("C:\\PyTorchPictureTest\\Model\\traced_resnet_model_pretrained.pt");
        std::cout &lt;&lt; "Successfully loaded resnet model" &lt;&lt; std::endl;

        module.to(at::kCUDA);
        std::cout &lt;&lt; "Moved model to gpu" &lt;&lt; std::endl;

        // load image and transform
        cv::Mat image;
        image = cv::imread("C:\\PyTorchPictureTest\\dog.jpg", cv::IMREAD_COLOR);

        std::cout &lt;&lt; image.size &lt;&lt; "\n" &lt;&lt; std::endl;
        std::cout &lt;&lt; image.cols &lt;&lt; "\n" &lt;&lt; std::endl;
        std::cout &lt;&lt; image.rows &lt;&lt; "\n" &lt;&lt; std::endl;
        std::cout &lt;&lt; type2str(image.type()) &lt;&lt; "\n" &lt;&lt; std::endl;

        cv::cvtColor(image, image, cv::COLOR_BGR2RGB);
        cv::Mat img_float;
        image.convertTo(img_float, CV_32FC3, 1.0f / 255.0f);
        cv::resize(img_float, img_float, cv::Size(224, 224), cv::INTER_NEAREST);

        auto img_tensor = torch::from_blob(img_float.data, { img_float.rows, img_float.cols, 3 });
        img_tensor = img_tensor.permute({ 2, 0, 1 });
        std::cout &lt;&lt; img_tensor.sizes() &lt;&lt; '\n';
        
        torch::Tensor img_tensor_norm = normalize_transform(img_tensor).unsqueeze_(0);

        std::vector&lt;torch::jit::IValue&gt; inputs;
        inputs.push_back(img_tensor_norm.to(at::kCUDA));
        torch::Tensor out_tensor = module.forward(inputs).toTensor();
        
        std::cout &lt;&lt; out_tensor &lt;&lt; '\n';

    }
    catch (const c10::Error &amp; e) {
        std::cerr &lt;&lt; "error loading the model\n";
        return -1;
    }

    std::cout &lt;&lt; "ok\n";
}
</code></pre>
<p>Python Output:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2f5ceba191598a714e6a04dc17a24988debc8ff.png" data-download-href="https://discuss.pytorch.org/uploads/default/a2f5ceba191598a714e6a04dc17a24988debc8ff" title="Python"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a2f5ceba191598a714e6a04dc17a24988debc8ff_2_690x357.png" alt="Python" data-base62-sha1="nfBZp8CwgDjSGFd7Ymsr9KPgJLV" width="690" height="357" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a2f5ceba191598a714e6a04dc17a24988debc8ff_2_690x357.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a2f5ceba191598a714e6a04dc17a24988debc8ff_2_1035x535.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a2f5ceba191598a714e6a04dc17a24988debc8ff_2_1380x714.png 2x" data-dominant-color="242424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Python</span><span class="informations">2325×1206 116 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Cpp Output:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44ce4af5cb582fcb5d45ce792fec93749f5df5cd.png" data-download-href="https://discuss.pytorch.org/uploads/default/44ce4af5cb582fcb5d45ce792fec93749f5df5cd" title="Cpp"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44ce4af5cb582fcb5d45ce792fec93749f5df5cd_2_690x377.png" alt="Cpp" data-base62-sha1="9OGrNhpesVqKPmQPhUymxm5DyI5" width="690" height="377" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44ce4af5cb582fcb5d45ce792fec93749f5df5cd_2_690x377.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44ce4af5cb582fcb5d45ce792fec93749f5df5cd_2_1035x565.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44ce4af5cb582fcb5d45ce792fec93749f5df5cd_2_1380x754.png 2x" data-dominant-color="212121"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Cpp</span><span class="informations">2317×1268 106 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So the outputs are similar, but you can see the differences.  Is this what you would expect?  Though the first point seems to be quite off (-0.7060 in Cpp, -0.575 in Python).  What is the source of the difference?  Does it lie in the serialization of the model?  Or does it have to do with pillow versus opencv?</p>
<p>By the way, is there an easy way to compare corresponding slices of the input tensor in C++ and Python?</p>

username: Aditya_Kumar
cooked: <p>I think first you should be doing predictions in evaluation mode that is <code>model.eval()</code> which will deactivate batchnorm and dropout layers in both c++ and python. In your python code, you are using a transform <code>transforms.CenterCrop</code> which you are not doing in c++. And there should not be large difference in answers between cpp and python. For Example, if cpp gives 0.75 then python should be in range 0.74-0.76.<br>
Why do you want to compare input tensor?</p>

username: solarflarefx
cooked: <p><a class="mention" href="/u/aditya_kumar">@Aditya_Kumar</a><br>
Thanks for your reply.  Would the evaluation mode cause a difference in the output, or does it have to do with speed and efficiency?</p>
<p>So you are right about the cropping.  Since I could not find an equivalent of transforms.CenterCrop(224), I attempted to do the equivalent of this in OpenCV.</p>
<p>Python Code (same as before, haven’t implemented your eval mode suggestion yet):</p>
<pre><code class="lang-auto">import torch
import torchvision.models as models
import urllib.request
from PIL import Image
from torchvision import transforms
import numpy as np

def norm_chan(chan, mean, std):
    b = (chan - mean) / std
    return b

# Load resnet18 model
model = models.resnet18(pretrained = True)

serialize_model = True
if (serialize_model):
    example = torch.rand(1, 3, 224, 224)
    traced_script_module = torch.jit.trace(model, example)
    traced_script_module.save("traced_resnet_model_pretrained.pt")

# sample execution (requires torchvision)
input_image = Image.open("C:\\PyTorchPictureTest\\dog.jpg")
np_im = np.array(input_image)

preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

input_tensor = preprocess(input_image)
input_batch = input_tensor.unsqueeze(0) # create a mini-batch as expected by the model

# move the input and model to GPU for speed if available
if torch.cuda.is_available():
    input_batch = input_batch.to('cuda')
    model.to('cuda')

with torch.no_grad():
    output = model(input_batch)

import numpy as np
print()
output_cpu = output[0].cpu()
output_numpy = output_cpu.numpy()
for i in range(0, len(output_numpy)):
    if ( i%10 == 0):
        print("\n")
    print(str(output_numpy[i]), end = " ")
print()
</code></pre>
<p>C++ code:</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt;
#include &lt;torch/torch.h&gt;
#include &lt;opencv2/opencv.hpp&gt;
#include &lt;opencv2/imgproc/imgproc.hpp&gt;
#include &lt;opencv2/core/core.hpp&gt;
#include &lt;opencv2/highgui/highgui.hpp&gt;
#include &lt;iostream&gt;
#include &lt;memory&gt;
#include &lt;iomanip&gt;

std::string type2str(int type) {
    std::string r;

    uchar depth = type &amp; CV_MAT_DEPTH_MASK;
    uchar chans = 1 + (type &gt;&gt; CV_CN_SHIFT);

    switch (depth) {
    case CV_8U:  r = "8U"; break;
    case CV_8S:  r = "8S"; break;
    case CV_16U: r = "16U"; break;
    case CV_16S: r = "16S"; break;
    case CV_32S: r = "32S"; break;
    case CV_32F: r = "32F"; break;
    case CV_64F: r = "64F"; break;
    default:     r = "User"; break;
    }

    r += "C";
    r += (chans + '0');

    return r;
}

int main(int argc, const char* argv[]) {


    // Load up GPU stuff
    torch::DeviceType device_type;
    if (torch::cuda::is_available()) {
        std::cout &lt;&lt; "CUDA available! Training on GPU." &lt;&lt; std::endl;
        device_type = torch::kCUDA;
    }
    else {
        std::cout &lt;&lt; "Training on CPU." &lt;&lt; std::endl;
        device_type = torch::kCPU;
    }
    torch::Device device(device_type);
    torch::data::transforms::Normalize&lt;&gt; normalize_transform({ 0.485, 0.456, 0.406 }, { 0.229, 0.224, 0.225 });
    torch::jit::script::Module module;
    std::cout &lt;&lt; "Attempting to load resnet model.." &lt;&lt; std::endl;
    try {
        // Deserialize the ScriptModule from a file using torch::jit::load().
        module = torch::jit::load("C:\\PyTorchPictureTest\\Model\\traced_resnet_model_pretrained.pt");
        std::cout &lt;&lt; "Successfully loaded resnet model" &lt;&lt; std::endl;

        module.to(at::kCUDA);
        std::cout &lt;&lt; "Moved model to gpu" &lt;&lt; std::endl;

        // load image and transform
        cv::Mat image;
        image = cv::imread("C:\\PyTorchPictureTest\\dog.jpg", cv::IMREAD_COLOR);
        cv::cvtColor(image, image, cv::COLOR_BGR2RGB);
        cv::Mat img_float;
        image.convertTo(img_float, CV_32FC3, 1.0f / 255.0f);

        // Scale image down
        int scaledown_factor = 256;
        cv::resize(img_float, img_float, cv::Size(img_float.cols / (img_float.rows / (float)scaledown_factor), scaledown_factor), cv::INTER_NEAREST);

        // Emulate transforms.CenterCrop(224)
        cv::Rect roi;
        int new_width = 224;
        int new_height = 224;
        roi.x = img_float.size().width / 2 - new_width / 2;
        roi.width = new_width;
        roi.y = img_float.size().height / 2 - new_height / 2;
        roi.height = new_height;
        cv::Mat img_cropped = img_float(roi);

        // Convert to tensor
        auto img_tensor = torch::from_blob(img_cropped.data, { img_cropped.rows, img_cropped.cols, 3 });
        img_tensor = img_tensor.permute({ 2, 0, 1 });
        std::cout &lt;&lt; img_tensor.sizes() &lt;&lt; '\n';
        
        // normalize
        torch::Tensor img_tensor_norm = normalize_transform(img_tensor).unsqueeze_(0);

        std::vector&lt;torch::jit::IValue&gt; inputs;
        inputs.push_back(img_tensor_norm.to(at::kCUDA));

        // forward pass
        torch::Tensor out_tensor = module.forward(inputs).toTensor();
        
        // print output
        std::cout &lt;&lt; out_tensor &lt;&lt; '\n';

    }
    catch (const c10::Error &amp; e) {
        std::cerr &lt;&lt; "error loading the model\n";
        return -1;
    }

    std::cout &lt;&lt; "ok\n";
}
</code></pre>
<p>However, even with my changes, there is still a larger different in my outputs than you have suggested should be the case.</p>
<p>Python Output:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/d/bd3185c8d740473871de9a28e9ca1fa386417e65.png" data-download-href="https://discuss.pytorch.org/uploads/default/bd3185c8d740473871de9a28e9ca1fa386417e65" title="Python"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/d/bd3185c8d740473871de9a28e9ca1fa386417e65.png" alt="Python" data-base62-sha1="qZGliKqHeMZvO5rKIgvKmvbzW5f" width="690" height="361" data-dominant-color="262626"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Python</span><span class="informations">979×513 29.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>C++ Output:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/0/20a405f4c0e6e9538247e902e11dca8faa0e7cd2.png" data-download-href="https://discuss.pytorch.org/uploads/default/20a405f4c0e6e9538247e902e11dca8faa0e7cd2" title="Cpp"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/0/20a405f4c0e6e9538247e902e11dca8faa0e7cd2.png" alt="Cpp" data-base62-sha1="4EKHelxGVO5CuLQLG0PmQW2JTuW" width="690" height="363" data-dominant-color="242424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Cpp</span><span class="informations">980×516 16.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Regarding the comparing of input tensors, I more wanted to debug the cause of the difference in my outputs: whether it was due to there being a difference of the data going into the model, or whether the model itself produces a different output.</p>
<p>Perhaps I am overcomplicating things a bit for myself.  I am relatively new to torchscript and I was trying to experiment with it to ensure that the output in C++ is equivalent to the output in Python.  I chose to use ResNet18 since that is the example that was shown on PyTorch’s documentation.  Would I be better off doing this exercise with a different network?</p>

username: Ashkan
cooked: <p>Hi,</p>
<p>I ran to the same problem! Did you find any solution? please.</p>