Weird time measurement outputs and correct way of C++ inference

username: m3tobom_M
cooked: <p>Hello. I wrote a prediction code for a classifier in C++. I am not experienced with C++ and new to libtorch. When model prediction print line exist timing output is like below:</p>
<p>Timing output when model prediction print line exits.</p>
<pre><code class="lang-auto">Time passed outside of function in seconds: 0.052
Time passed inside of function in seconds: 0.006
</code></pre>
<p>When model prediction print line does not exist timing output is like below:</p>
<p>Timing output when model prediction print line does not exists.</p>
<pre><code class="lang-auto">Time passed inside of function in seconds: 0.051
Time passed outside of function in seconds: 0.051
</code></pre>
<p>model prediction print line</p>
<pre><code class="lang-auto">std::cout &lt;&lt; "Prediction is: " &lt;&lt; prediction &lt;&lt; std::endl;
</code></pre>
<p>My full code:</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include &lt;torch/torch.h&gt; 
#include &lt;torch/script.h&gt;
#include &lt;opencv2/highgui/highgui.hpp&gt;
#include &lt;opencv2/imgproc.hpp&gt;
#include &lt;zmq.hpp&gt; 
#include "metolib.hpp"
#include &lt;chrono&gt;
#include &lt;unistd.h&gt; // timelib



torch::Tensor classifier_predict_image(cv::Mat image, torch::jit::script::Module model)
{
    auto func_t0 = std::chrono::high_resolution_clock::now();

    torch::Tensor tensor_image = metolib::mat2tensor(image, image.size().height, image.size().width, image.channels()); 
    std::vector&lt;torch::jit::IValue&gt; input;
    input.push_back(tensor_image.to(torch::kCUDA));
    torch::Tensor output = model.forward(input).toTensor();
    output = torch::argmax(output);

    auto func_t1 = std::chrono::high_resolution_clock::now();
    auto duration = std::chrono::duration_cast&lt;std::chrono::milliseconds&gt;(func_t1 - func_t0);
    std::cout &lt;&lt; "Time passed inside of function in seconds: " &lt;&lt; (float)duration.count() / 1000 &lt;&lt; std::endl;

    return output;
} 


int main(int argc, const char *argv[])
{     
    // Inits
    torch::init_num_threads();
    cv::Mat frame;
    torch::Tensor prediction;

    // Load model
    torch::jit::script::Module model = torch::jit::load(argv[1]);
    model.eval(); 
    model.to(torch::kCUDA);

    // Video input
    cv::VideoCapture cap(0);

    while(cap.isOpened())
    {
        // read  frame
        cap &gt;&gt; frame;  



        // predict
        auto main_loop_t0 = std::chrono::high_resolution_clock::now();
        
        prediction = classifier_predict_image(frame, model);
        std::cout &lt;&lt; "Prediction is: " &lt;&lt; prediction &lt;&lt; std::endl;

        auto main_loop_t1 = std::chrono::high_resolution_clock::now();
        auto duration = std::chrono::duration_cast&lt;std::chrono::milliseconds&gt;(main_loop_t1 - main_loop_t0);
        std::cout &lt;&lt; "Time passed outside of function in seconds: " &lt;&lt; (float)duration.count() / 1000 &lt;&lt; std::endl;

        cv::imshow("Output", frame);
        char key = cv::waitKey(1);
        if((int)key == 'q') {std::cout &lt;&lt; "Q pressed. Exiting..." &lt;&lt; std::endl; cv::destroyAllWindows(); break;}
    }

}

</code></pre>
<p>mat2tensor function:</p>
<pre><code class="lang-auto">torch::Tensor mat2tensor(cv::Mat image, int h, int w, int channels)
    {
        torch::Tensor out_tensor = torch::from_blob(image.data, {1, h, w, channels}).permute({0, 3, 1, 2});
        return out_tensor; 
    }
</code></pre>
<p>I followed the below tutorial for exporting Python trained model to C++.<br>
<a href="https://pytorch.org/tutorials/advanced/cpp_export.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/cpp_export.html</a></p>
<p>My CMakeLists.txt:</p>
<pre><code class="lang-auto">cmake_minimum_required (VERSION 2.8)
project(example_project)
set( CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++14" )
set( OpenCV_DIR "/usr/share/OpenCV" )
set( PKG_CONFIG_PATH "/usr/lib/pkgconfig" )
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

find_package(Torch REQUIRED)
find_package(OpenCV REQUIRED) 


add_executable(../bin/app ../sources/app.cpp)
include_directories(/home/m3/meto_ws/c_cpp_libs/libtorch/include/torch/csrc/api/include)
include_directories(../libs)
target_link_libraries(../bin/app "${TORCH_LIBRARIES}"  "${OpenCV_LIBS}")

 

set_property(TARGET ../bin/app PROPERTY CXX_STANDARD 14) 

</code></pre>
<p>I downloaded libtorch from <a href="http://pytorch.org" rel="noopener nofollow ugc">pytorch.org</a> using the below link.<br>
<a href="https://download.pytorch.org/libtorch/cu111/libtorch-cxx11-abi-shared-with-deps-1.8.1%2Bcu111.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cu111/libtorch-cxx11-abi-shared-with-deps-1.8.1%2Bcu111.zip</a></p>
<p>My Cuda version is 11.1.</p>
<p>My questions:<br>
1 - Any idea why timing outputs are differs so much?<br>
2 - Is my CMakeLists.txt correct?<br>
3 - Is my code correct?</p>

username: ptrblck
cooked: <p>CUDA operations are executed asynchronously, so you would need to synchronize the code before starting and stopping the timers to get a valid profile of the runtime. The <code>std::cout</code> operation is most likely a synchronization point, since the result is needed so that it can be printed in the terminal.</p>

username: m3tobom_M
cooked: <p>Can you show me samples or a source to read about synchronization? I have no idea how to do it.</p>

username: ptrblck
cooked: <p>You can synchronize your code in <code>libtorch</code> via:</p>
<pre><code class="lang-auto">auto stream = c10::cuda::getCurrentCUDAStream(device.index());
C10_CUDA_CHECK(cudaStreamSynchronize(stream));
</code></pre>

username: m3tobom_M
cooked: <p>I added two lines to my code but no change.</p>