Why is the const time with FP32 and FP16 almost the same in libtorch's forward?

cyanM: <p>I trained a simple refinedet-512 model to test the predictive time, and found that it took about 60ms to compute with float and also 60ms to compute with half single, why?</p>
<p>env: GTX1660<br>
libtorch: 1.1 with cuda 10</p>

ptrblck: <p>Could you share the code you’ve used to time your model?<br>
The GTX 1660 should accelerate FP16 operations as far as I know.</p>

cyanM: <p>I convert the entire model to FP16 with code<br>
network-&gt;to(torch::kCUDA).to(torch::kHalf),<br>
and also the input data (kfloat to kHalf)<br>
input = input.to(torch::kCUDA).to(torch::kHalf).</p>
<p>Then I package data preprocessing, forward() and following analysis processing into a predict function, and use std::chrono to measure the elapsed time of this predict function.</p>

ptrblck: <p>Are you profiling your “complete” code, i.e. including the data loading?<br>
Could it be that the data loading might be the bottleneck in your application or did you make sure to just time the <code>forward</code> and <code>backward</code> functions?</p>

cyanM: <p>like this?</p>
<p>// matImg is a cv::Mat loaded with opencv,and convert to CV_32FC3<br>
// m_model is already loaded</p>
<p>at::Tensor input = torch::from_blob(matImg.data, { 1, 512,512,3 }, torch::kFloat).to(torch::kCUDA);<br>
input = input.permute({ 0,3,1,2 });</p>
<p>input = input.to(torch::kHalf);<br>
m_model-&gt;to(torch::kHalf);</p>
<p>auto output = m_model-&gt;forward({ input }).toTuple();</p>
<p>at::Tensor arm_loc = output-&gt;elements()[0].toTensor().to(torch::kFloat).to(torch::kCUDA);<br>
at::Tensor arm_conf = output-&gt;elements()[1].toTensor().to(torch::kFloat).to(torch::kCUDA);<br>
at::Tensor odm_loc = output-&gt;elements()[2].toTensor().to(torch::kFloat).to(torch::kCUDA);<br>
at::Tensor odm_conf = output-&gt;elements()[3].toTensor().to(torch::kFloat).to(torch::kCUDA);</p>
<p>and then following with post-processing( such as nms) to get the final result of detection.<br>
The amount of time I count includes putting data into cuda, forward(),and post-processing, where loading data and post-processing occpuied about 10ms in 60ms</p>

ptrblck: <p>Thanks for the code.<br>
Could you add synchronization points before starting and stopping the timer for the forward and backward pass?</p>
<pre><code class="lang-auto">at::cuda::CUDAStream stream = at::cuda::getCurrentCUDAStream();
AT_CUDA_CHECK(cudaStreamSynchronize(stream));
</code></pre>
<p>It would be interesting to see just the time for the actual GPU workload.</p>

cyanM: <p>Thanks.<br>
After adding synchronization, the partical code is as follows</p>
<blockquote>
<p>std::chrono::steady_clock::time_point t1, t2;<br>
std::chrono::duration time_used;</p>
<p>at::cuda::CUDAStream stream = at::cuda::getCurrentCUDAStream();<br>
AT_CUDA_CHECK(cudaStreamSynchronize(stream));<br>
t1 = std::chrono::steady_clock::now();</p>
<p>auto output = m_model-&gt;forward({ input }).toTuple();</p>
<p>AT_CUDA_CHECK(cudaStreamSynchronize(stream));<br>
t2= std::chrono::steady_clock::now();<br>
time_used = std::chrono::duration_cast&lt;std::chrono::duration&gt;(t2 - t1) * 1000;</p>
<p>printf(“forward cost time %f ms \n”, time_used);</p>
</blockquote>
<p>The test results are as follows, and find that fp16 is even slightly slower than fp32<br>
FP16<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/b/b685d8e5588aaf995f63c56319f934ce7bd05d67.png" alt="%E6%8D%95%E8%8E%B7FP16" data-base62-sha1="q2FFidUuOhylt4kW1iqAyH97MxN" width="290" height="98"><br>
FP32<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/6/64b15b1803e5fbf46430d79aeb8d5e801065c4f7.png" alt="%E6%8D%95%E8%8E%B7fp32" data-base62-sha1="emLJgLNyyKIzKcZ8ef2tPrB2haD" width="307" height="101"></p>