C++ Inference using OpenCV Data

username: thorsten
cooked: <p>Hi,</p>
<p>I’m trying to use a model which I trained in Python in C++ for inference. While being in Python, the model produces reasonable outputs. However, when using it for inference in C++, the output does not make sense. My feeling is that the input/output formats are somehow wrong (type, size, etc.). In the following, I will detail the steps for:</p>
<ul>
<li>Exporting the model via tracing (in Python)</li>
<li>Loading the model (in C++)</li>
<li>Preparing the input data for the model (from OpenCV data structures in C++)</li>
<li>Forward pass</li>
<li>Converting the output data of the network back to OpenCV</li>
</ul>
<p>Any idea or help would be greatly appreciated. I already checked a lot of resources both here and on Stack Overflow, and other sources. According to what I found, I cannot identify the error.</p>
<p>Thanks in advance,<br>
Thorsten</p>
<h1>Exporting the model</h1>
<pre><code class="lang-python">model = model.to('cpu')
model.train(False)
example = torch.rand(1, 3, height, width)
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save(save_filename)
</code></pre>
<p>Moving model to CPU as this is were inference will be done (does it matter?).<br>
Disabling training mode as I am using “training-only” features like batch normalization.</p>
<h1>Loading model</h1>
<pre><code class="lang-auto">std::ifstream is(network_file_path, std::ifstream::binary);
std::shared_ptr&lt;torch::jit::script::Module&gt; network = torch::jit::load(is);
</code></pre>
<h1>Input data preparation</h1>
<p>There are three data channels which are stacked like a three layer image. Except that the channels do not represent RGB or the channels of another color format but our own data.</p>
<p>data0, data1 and data2 contain 8_bit data (value range 0 to 255). For the training, this was converted to float and scaled to the range 0 to 1.0. All channels have the same size.</p>
<p>So, converting to float and scaling:</p>
<pre><code class="lang-auto">cv::Mat channel0(data0.size(), CV_32FC1);
cv::Mat channel1(data1.size(), CV_32FC1);
cv::Mat channel2(data2.size(), CV_32FC1);

float scale_factor = 1 / 255.0;

data0.convertTo(channel0, CV_32F, scale_factor);
data1.convertTo(channel1, CV_32F, scale_factor);
data2.convertTo(channel2, CV_32F, scale_factor);
</code></pre>
<p>Now, the three channels are combined to one cv::Mat with three channels:</p>
<pre><code class="lang-auto">std::vector&lt;cv::Mat&gt; network_input_channels;
network_input_channels.push_back(channel0);
network_input_channels.push_back(channel1);
network_input_channels.push_back(channel2);

cv::Mat network_input(channel0.size(), CV_32FC3);
cv::merge(network_input_channels, network_input);    
</code></pre>
<p>The input is converted to a tensor:</p>
<pre><code class="lang-auto">std::vector&lt;int64_t&gt; sizes = {1, 3, network_input.rows, network_input.cols};
at::TensorOptions options(at::kFloat);
at::Tensor input_tensor = torch::from_blob(network_input.data, at::IntList(sizes), options);

std::vector&lt;torch::jit::IValue&gt; inputs;
inputs.push_back(input_tensor);
</code></pre>
<p>When I examine the values of the input tensor via <code>std::cout</code> and compare the values with the expected values for my data from the channels above, the values are different. Therefore, I suspect that the error has already happened until this point. However, I cannot rule out that the data is just interpretetet differently by the tensor and everything is still fine. Therefore, for the sake of completeness, the next steps in the pipeline.</p>
<h1>Forward pass</h1>
<pre><code class="lang-auto">auto output_tensor = network-&gt;forward(inputs).toTensor();
</code></pre>
<h1>Converting data back to OpenCV data structures</h1>
<pre><code class="lang-auto">cv::Mat output_tmp(cv::Size(width, height) , CV_32FC1, output_tensor.data&lt;float&gt;());
cv::Mat network_output(output_tmp.size(), CV_8UC1);
float scale_factor_up = 255.0;
output_tmp.convertTo(network_output, CV_8U, scale_factor_up);
</code></pre>
<p>As expected, the values in the output do not make sense. Most of the output is saturated (maxed out or zero). I blame the wrong input for that and not the model because in Python I get fine results.</p>
<p>If anybody could give me any hints I would be very thankful for your help.</p>
<p>Best,<br>
Thorsten</p>

username: thorsten
cooked: <p>Follow-up remarks:<br>
Pytorch version: 1.0<br>
Python installation of Pytorch: Using Anaconda<br>
CUDA version: 9.0 (only used for training)<br>
Libtorch was compiled from source. It is not possible to use the downloadable version because our project also relies on boost. The boost/Pytorch issue is discussed e.g. here: <a href="https://discuss.pytorch.org/t/libtorch-does-not-link-together-with-boost/27956/7" class="inline-onebox">Libtorch does not link together with boost</a></p>

username: tom
cooked: <p>OpenCV is typically HWC (channel last) and BRG at that. Print out a small (eg 3x3x3) subtensor of the C++ and Python inputs, they must match.<br>
If you can’t match those check preprocessing - eg print mean and std. Those should match, too. You might be able to absorb preprocessing into the model (by reimplementing it).</p>
<p>Best regards</p>
<p>Thomas</p>

username: thorsten
cooked: <p>Thanks for your reply. Yes, that seems to be the issue. Inspecting the values confirms your thought that the dimensions from openCV do not match.</p>
<p>I see two possible solutions:</p>
<ul>
<li>Create three 2D tensors and merging them (instead of merging in OpenCV and creating a tensor afterwards). How can I merge the tensors?</li>
<li>Reshaping/Sorting the existing tensor which was created using <code>torch::from_blob</code>. How can I reshape the tensor?</li>
</ul>
<p>I guess that I can find the answer to both questions here: <a href="https://pytorch.org/cppdocs/api/namespace_at.html#namespace-at" rel="nofollow noopener">https://pytorch.org/cppdocs/api/namespace_at.html#namespace-at</a></p>
<p>However, the documentation is rather limited. I could therefore need a hand pointing me to the correct function. If possible, maybe a little example <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: tom
cooked: <p>I’d first check the Python documentation. You probably want permute and reshape, pass the dimensions / shapes as <code>t.permute({1, 2, 0})</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: thorsten
cooked: <p>Dear Thomas,</p>
<p>Thanks for the input.</p>
<p>After studying what torch::from_blob really does (interpreting some allocated memory in a designated way - in my case as if if contains floating point numbers), I decided to simply provide a memory portion with the correct data stored in it. I chose the classic “C-Style” way using a pointer to an array. Although I would usually prefer a more C+±ish solution, it works for now.</p>
<h1>Solution</h1>
<p>In case anybody finds this thread and has the same problem, here is what I did to solve my problem:</p>
<pre><code class="lang-auto">float manual_input[channel_number][row_number][column_number] // You have to allocate the memory dynamically, using this line only for the sake of shortness. 
for (int y = 0; y &lt; 64; y++) {
    for (int x = 0; x &lt; 96; x++) {
        manual_input[0][y][x] = channel0.at&lt;float&gt;(cv::Point(x,y));
        std::cout &lt;&lt; manual_input[0][y][x]; // Verifying my input data
    }
    std::cout &lt;&lt; std::endl; // Verifying my input data
}
// Same loop for the other two channels

std::vector&lt;int64_t&gt; sizes = {1, channel_number, row_number, column_number};
at::TensorOptions options(at::kFloat);
at::Tensor input_tensor = torch::from_blob(manual_input, at::IntList(sizes), options);
</code></pre>
<p><code>channel0</code> as in the original post, <code>channel_number</code>, <code>row_number</code> and <code>column_number</code> according to the input size.</p>
<p>Inference with <code>input_tensor</code> as in the original post.</p>
<p>Thanks,<br>
Thorsten</p>

username: tom
cooked: <p>Glad you solved it!</p>
<p>If anyone is searching for a more compact and efficient solution, you might try something like</p>
<pre><code class="lang-auto">std::vector&lt;int64_t&gt; sizes = {1, network_input.rows, network_input.cols, 3};
torch::TensorOptions options(at::kFloat);
torch::Tensor input_tensor = torch::from_blob(network_input.data, at::IntList(sizes), options);

std::vector&lt;torch::jit::IValue&gt; inputs;
inputs.push_back(input_tensor.permute({0, 3, 1, 2}));
</code></pre>
<p>Note that input_tensor and thus also the permutation don’t own the memory and you need to keep <code>network_input</code> around or clone after from_blob.</p>
<p>Best regards</p>
<p>Thomas</p>

username: thorsten
cooked: <p>Nice! I confirm that the solution of Thomas works as well as mine. Thomas solution is much more elegant, so I will choose it. That was the function I was looking for.</p>
<p>Thanks,<br>
Thorsten</p>

username: yf225
cooked: <p>A tangential follow-up: we now have new ABI binaries for libtorch that works with boost. They can be found on<br>
<a href="http://pytorch.org/" rel="nofollow noopener">http://pytorch.org</a>, or at <a href="https://github.com/pytorch/pytorch/issues/17492#issuecomment-524692441" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/17492#issuecomment-524692441</a>.</p>