How to control tensor size inside of functional files in C++

Serif_Inanir: <p>Hello,</p>
<p>When I tried to get result from two separated datasets created with 128x128 in first and 256x226 resolutions in second, I obtained to same prediction durations for each sets. However this only applies to GPU usage. In CPU, the durations are more rational.</p>
<p>I have investigate the source code written just Python, not C++. And I saw that each layer results of a model is expected sizes. So PyTorch doesn’t change input sizes with a constant size amount from beginning of a prediction process to its end or layers. By reason of this, I think that input sizes can be changed in C++ side, for example in Convolution progress.</p>
<p>I wonder that how to break this case, and starting to get rational (or expected) prediction durations which are changes in accordance with input sizes?</p>
<p>Thanks for your attentions, have a good day!</p>

Serif_Inanir: <p>Or it can be nice to know that why I get same prediction durations for different sized datasets, using GPU?</p>

ptrblck: <p>Could you post the code you’ve used to profile the workload, please?</p>

Serif_Inanir: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  thanks for your reply.</p>
<p>I guess you don’t need to a code to understand. However, probably, I didn’t explain myself clearly.<br>
I have prepared a code which will show you what’s going on. Data set is the major difference between my project and the code which I have prepared for you. On the other hand, the general concept is same.</p>
<p>You can find it here:</p>
<blockquote>
<p><a href="https://github.com/Serif-NNR/Same-Prediction-Durations-with-Different-Imput-Sizes/blob/main/Same_Prediction_Durations_with_Different_Input_Sizes.ipynb" rel="noopener nofollow ugc">github/Same-Prediction-Durations-with-Different-Input-Sizes</a></p>
</blockquote>
<p>As you can see, we obtain same prediction duration using GPU, but not with CPU. I wonder reasons of same prediction duration for GPU using different input sizes.</p>

ptrblck: <p>Thanks for the code!<br>
Your profiling is unfortunately wrong, since GPU operations are executed asynchronously and you would thus need to synchronize the device before starting and stopping the timers:</p>
<pre><code class="lang-python">    for i in range(loop_count):
      outset = time.time()
      model(tensor.to(device))
      durations.append(time.time()-outset)
</code></pre>
<p>via <code>torch.cuda.synchronize()</code>.<br>
Otherwise you would profile the dispatching, kernel launches, etc. unless you are (accidentally) synchronizing the code already.</p>

Serif_Inanir: <p>Great! I overlooked whether a prediction process can work asynchronously and it can be stopped with a signal.  However, it doesn’t give the wanted result I have expressed. Maybe you can control the last calculations and suggest a different way. Because I expect that the calculation for 128x128 resolution should be completed in about 1/4 (fairly roughly) duration according to 256x256 resolution.</p>
<p>Here the code with your last suggestion:</p>
<p><a href="https://colab.research.google.com/drive/1brpoYD-ekHTi1ojJfA9ukeeBsFuMd37g?usp=sharing" rel="noopener nofollow ugc">&gt; Colab/Same Prediction Durations with Different Input Sizes - with cuda.synchroniz.ipynb</a></p>

ptrblck: <p>Great! Your first synchronization is still wrong, since you would need to synchronize the code before starting the timer, not afterwards.<br>
In any case, you should be careful with your expectation of seeing a perfect 1/4 of the time for a smaller input image since your overall use case is small and I doubt you are saturating your GPU at all. Thus your workload might already be CPU-limited which you could verify by profiling the code.<br>
The actual kernel times might be significantly faster, but your timeline might just see more idle times in case the CPU isn’t fast enough in scheduling the work.</p>

Serif_Inanir: <p>I understood. Thanks your your clues and the starting point related to profile. I believe that I can proceed more with your suggestions after my personal researches.<br>
Have a nice day!</p>

ptrblck: <p>Sure! Let me know how it goes or if you get stuck somewhere.</p>