Libtorch spends lots of time once changing the input size

username: cfanyyx
cooked: <p>Env: C++, cuda9, self compiled libtorch from souce(1.1.0a0+deadf3b), tracing model<br>
Problem: libtorch spends lots of time in reloading the model if I change the input size during inference(In fact, it seems the same in python). If I don’t change the input size, it will get fast inference after the first time-consuming input. What can I do to accelerate the inference if I need to change the input size all the time?</p>

username: mhubii
cooked: <p>what do you mean by changing the input size?</p>

username: cfanyyx
cooked: <p>When I do inference, the size of each test input is different. For example, first one is [1, 3, 512, 960], then second one may be [1, 3, 1440, 1000]…</p>

username: cfanyyx
cooked: <p>I tried to use <code>c10::cuda::CUDACachingAllocator::emptyCache();</code> to see whether it may make some differences. But it did not help.</p>

username: cfanyyx
cooked: <p>I also found it would occupy more gpu memory using c++ than using python. Hoping to know the reason.</p>

username: mhubii
cooked: <p>hm that seems unlikely as the c++ api is really just that. The code it interfaces with is just the same that the python api uses. Does the python equivalent of your code also slow down once you change the input sizes?</p>

username: cfanyyx
cooked: <p>Yes, it is! And I found c++ would use more gpu memory and inference time than python.</p>

username: ngimel
cooked: <p>do you have cudnn benchmarking mode turned on? (torch.backends.cudnn.benchmark=True in python, not sure about C++ apis). If so, pytorch would search for the fastest convolution algorithm for each input size and that can slow you down.</p>

username: cfanyyx
cooked: <p>Thanks for your reply. I have tried <code>torch.backends.cudnn.benchmark=True/False</code> on both c++ and python. I find it makes no difference whether the benchmark value is set to True or False.</p>
<p>I also find that if an input image with a typical size has been inferred once, then the model can make fast inference when it encounters the input image with the same size.</p>
<blockquote>
<p>–debug-- input_concat shape: torch.Size([1, 3, 2272, 1280])<br>
cost:  4.270411252975464<br>
–debug-- input_concat shape: torch.Size([1, 3, 2272, 1280])<br>
cost:  0.027519702911376953<br>
–debug-- input_concat shape: torch.Size([1, 3, 2272, 1280])<br>
cost:  0.058194875717163086<br>
–debug-- input_concat shape: torch.Size([1, 3, 2272, 1280])<br>
cost:  0.04706835746765137<br>
–debug-- input_concat shape: torch.Size([1, 3, 2272, 1280])<br>
cost:  0.048264265060424805<br>
–debug-- input_concat shape: torch.Size([1, 3, 2272, 1280])<br>
cost:  0.04921388626098633<br>
–debug-- input_concat shape: torch.Size([1, 3, 1696, 1280])<br>
cost:  3.20564603805542<br>
–debug-- input_concat shape: torch.Size([1, 3, 1712, 1280])<br>
cost:  3.278776168823242<br>
–debug-- input_concat shape: torch.Size([1, 3, 1696, 1280])<br>
cost:  0.03748798370361328<br>
–debug-- input_concat shape: torch.Size([1, 3, 1280, 1696])<br>
cost:  3.295097589492798<br>
–debug-- input_concat shape: torch.Size([1, 3, 1712, 1280])<br>
cost:  0.032279014587402344<br>
–debug-- input_concat shape: torch.Size([1, 3, 1712, 1280])<br>
cost:  0.05348610877990723<br>
–debug-- input_concat shape: torch.Size([1, 3, 1712, 1280])<br>
cost:  0.04692506790161133<br>
–debug-- input_concat shape: torch.Size([1, 3, 1712, 1280])<br>
cost:  0.03901052474975586<br>
–debug-- input_concat shape: torch.Size([1, 3, 2272, 1280])<br>
cost:  0.05834245681762695<br>
–debug-- input_concat shape: torch.Size([1, 3, 1696, 1280])<br>
cost:  0.034003496170043945<br>
–debug-- input_concat shape: torch.Size([1, 3, 1696, 1280])<br>
cost:  0.04744458198547363<br>
–debug-- input_concat shape: torch.Size([1, 3, 1712, 1280])<br>
cost:  0.04431772232055664<br>
–debug-- input_concat shape: torch.Size([1, 3, 2272, 1280])<br>
cost:  0.06284832954406738<br>
–debug-- input_concat shape: torch.Size([1, 3, 1712, 1280])<br>
cost:  0.04384565353393555<br>
–debug-- input_concat shape: torch.Size([1, 3, 1280, 1712])<br>
cost:  2.316293716430664<br>
–debug-- input_concat shape: torch.Size([1, 3, 1712, 1280])<br>
cost:  0.027036666870117188<br>
–debug-- input_concat shape: torch.Size([1, 3, 1712, 1280])<br>
cost:  0.050749778747558594<br>
–debug-- input_concat shape: torch.Size([1, 3, 1280, 1696])<br>
cost:  0.047841787338256836<br>
–debug-- input_concat shape: torch.Size([1, 3, 1280, 1952])<br>
cost:  3.9861865043640137<br>
–debug-- input_concat shape: torch.Size([1, 3, 1712, 1280])<br>
cost:  0.027935028076171875<br>
–debug-- input_concat shape: torch.Size([1, 3, 1280, 1280])<br>
cost:  2.379218101501465<br>
–debug-- input_concat shape: torch.Size([1, 3, 1696, 1280])<br>
cost:  0.02846550941467285<br>
–debug-- input_concat shape: torch.Size([1, 3, 1712, 1280])<br>
cost:  0.04407143592834473<br>
–debug-- input_concat shape: torch.Size([1, 3, 2272, 1280])<br>
cost:  0.06075263023376465<br>
–debug-- input_concat shape: torch.Size([1, 3, 1920, 1280])<br>
cost:  3.509601593017578<br>
–debug-- input_concat shape: torch.Size([1, 3, 1712, 1280])<br>
cost:  0.021055936813354492<br>
–debug-- input_concat shape: torch.Size([1, 3, 1696, 1280])<br>
cost:  0.03338050842285156<br>
–debug-- input_concat shape: torch.Size([1, 3, 1712, 1280])<br>
cost:  0.04442191123962402<br>
–debug-- input_concat shape: torch.Size([1, 3, 1712, 1280])<br>
cost:  0.04460453987121582<br>
–debug-- input_concat shape: torch.Size([1, 3, 1712, 1280])<br>
cost:  0.03949451446533203<br>
–debug-- input_concat shape: torch.Size([1, 3, 1712, 1280])<br>
cost:  0.052010297775268555<br>
–debug-- input_concat shape: torch.Size([1, 3, 1280, 1696])<br>
cost:  0.050638675689697266<br>
–debug-- input_concat shape: torch.Size([1, 3, 1712, 1280])<br>
cost:  0.058187246322631836<br>
–debug-- input_concat shape: torch.Size([1, 3, 1712, 1280])<br>
cost:  0.054537296295166016<br>
avg cost: 0.6914166688919068</p>
</blockquote>

username: Riddick_Gao
cooked: <p>I have the same problem that libtorch uses much  more GPU memory than python with the same image size. Do you have deal with  it?  Thank you.</p>

username: cfanyyx
cooked: <p>Actually, I have not figured it out yet.</p>

username: Aeroxander
cooked: <p>Putting torch::NoGradGuard no_grad; in the scope is the first solution, haven’t found much more that works yet tho… Would love to be able to easily quantisize the model when tracing the model to C++</p>