Timing forward call in C++ frontend using libtorch

tanmay2592: <p>I am trying to measure the time spent in model inference (forward() call on torch::jit::script::Module).</p>
<pre><code class="lang-auto">torch::NoGradGuard no_grad;
model_outputs = torch_model.forward(input_tensors);
</code></pre>
<p>However, the timing is coming out to be really small. I found the reason is because of the default Asynchronous mode for GPU operations described here:<br>
<a href="https://pytorch.org/docs/master/notes/cuda.html#asynchronous-execution" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/master/notes/cuda.html#asynchronous-execution</a></p>
<p>The description provides a solution for python using <code>torch.cuda.Event</code> abstraction, but I could not find analogous abstraction in C++.</p>
<p>Coming back to the example above, I believe the synchronization occurs when data is read out of the <code>model_outputs</code>. Is this understanding correct?</p>
<p>I tried the following solution to capture the correct timestamp on <code>c10::cuda::getCurrentCUDAStream().stream()</code></p>
<pre><code class="lang-auto">cudaLaunchHostFunc(
        c10::cuda::getCurrentCUDAStream().stream(), TimestampCaptureCallback,
        reinterpret_cast&lt;void*&gt;(&amp;compute_end_ns));
</code></pre>
<p>This gives me seemingly accurate results but is this solution generic and work for all kind of models? There is a risk in using internals which are not properly documented.</p>

SplitInfinity: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> probably knows more about CUDA and might be able to help.</p>

ptrblck: <p>I’m not sure about the <code>TimestampCaptureCallback</code> approach, but in any case you could add manual synchronizations via:</p>
<pre><code class="lang-python">auto stream = at::cuda::getCurrentCUDAStream(); // or at::cuda::getDefaultCUDAStream())
AT_CUDA_CHECK(cudaStreamSynchronize(stream));
</code></pre>
<p>to make sure your timings are correct. Alternatively you should also be able to use <code>cudaDeviceSynchronize</code> directly:</p>
<pre><code class="lang-auto">cudaError_t err = cudaDeviceSynchronize();
bool isEQ = err == cudaSuccess;
ASSERT_TRUE(isEQ);
</code></pre>

tanmay2592: <p>The first approach does sound good but won’t it have an adverse affect on the performance? I thought <code>cudaLaunchHostFunc</code> method to be less intrusive. What are your reservations about this approach?</p>
<p>The second alternative will not work for us because we might  have multiple models deployed and being served simultaneously on the GPU including models from TensorRT, TensorFlow, ONNX and pyTorch. Synchronizing the whole device will affect all these models negatively. See <a href="https://github.com/triton-inference-server/server" rel="noopener nofollow ugc">Triton Inference Server</a> for more reference.</p>
<p>If there are no other ways then our best bet would be to go by the first method.</p>

ptrblck: <p>I’m not familiar with your use case and don’t know what exactly you would like to profile, but based on the <a href="https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__EXEC.html#group__CUDA__EXEC_1gab95a78143bae7f21eebb978f91e7f3f">docs</a>:</p>
<blockquote>
<p>Host functions without a mandated order (such as in independent streams) execute in undefined order and may be serialized.</p>
</blockquote>
<p>you would need to check, if it fits your use case.</p>
<aside class="quote no-group" data-username="tanmay2592" data-post="4" data-topic="120569">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tanmay2592/48/38025_2.png" class="avatar"> tanmay2592:</div>
<blockquote>
<p>but won’t it have an adverse affect on the performance?</p>
</blockquote>
</aside>
<p>If you are synchronizing on a custom or the default stream used in PyTorch it will impact the performance, but again I might not understand your use case and how you would like to time kernel executions without synchronizations/events.</p>

tanmay2592: <blockquote>
<p>I’m not familiar with your use case</p>
</blockquote>
<p>I understand. I will try to simplify my use case.</p>
<p>I am trying to time the duration between <strong>from when the model execution started till when the outputs are first  ready to be read</strong> with minimal impact to performance. I am not interested in profiling individual kernel executions. With our discussion so far this is a possible solution:</p>
<pre><code class="lang-auto">1. Record start timestamp
2. model_outputs = torch_model.forward(input_tensors)
3. 
auto stream = at::cuda::getCurrentCUDAStream(); // or at::cuda::getDefaultCUDAStream())
AT_CUDA_CHECK(cudaStreamSynchronize(stream));
4. Record end timestamp
5. Read output tensor from `model_outputs`.
</code></pre>
<p>The alternative I propose is to attach a <strong>single</strong> host function to just capture the end of the execution on the <strong>default</strong> stream. The only assumption being the execution on all the streams being used by pytorch was over when default stream ends execution. The description of default stream being the one doing most of the work adds some credibility to above assumption. Hence, the warning should not apply here.<br>
Something like below:</p>
<pre><code class="lang-auto">1. Record start timestamp
2. model_outputs = torch_model.forward(input_tensors)
3. Record end timestamp in a callback on default stream.
4. Read output tensor from `model_outputs`.
</code></pre>
<p>Description of default stream:</p>
<blockquote>
<p>The default stream is where most computation occurs when you aren’t explicitly using streams.<br>
<a href="https://github.com/pytorch/pytorch/blob/master/c10/cuda/CUDAStream.h#L199" rel="noopener nofollow ugc">Link</a></p>
</blockquote>