Speed of Custom RNN is SUPER SLOW

senmao: <p>Hi,</p>
<p>Based on code here</p>
<blockquote>
<p><a href="https://github.com/pytorch/pytorch/blob/master/benchmarks/fastrnns/custom_lstms.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/benchmarks/fastrnns/custom_lstms.py at main · pytorch/pytorch · GitHub</a></p>
</blockquote>
<p>I write an example to compare the cumputation capability of native lstm and custom lstm.<br>
But I found that the speed of custom lstm is 100 times slower than native lstm class.</p>
<p>here is my test code:</p>
<pre><code class="lang-auto">import torch.nn as nn
import time
from models.custom_lstm import LSTMLayer, LSTMCell, script_lstm, LSTMState

input_size = 1024
cell_size =2048
batch_size =20 
seq_len = 200 

native_lstm=nn.LSTM(input_size, cell_size,1).cuda()
custom_lstm=script_lstm(input_size, cell_size,1).cuda()
inp = torch.randn(seq_len,batch_size,input_size).cuda()
hx = inp.new_zeros(batch_size, cell_size, requires_grad=False)
cx = inp.new_zeros(batch_size, cell_size, requires_grad=False)

t1 = time.time()
out, hid = native_lstm(inp)
t2 = time.time()
out2, hid2 = custom_lstm(inp, [(hx, cx)])
t3 = time.time()

print ('lstm:{}\ncustom lstm:{}\n'.format(t2-t1, t3-t2))
</code></pre>
<p>And here is the result:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1dff809f5f6394ed9589a23c54f714bba13b0d3.png" alt="image" data-base62-sha1="webhu2ZTMHAw2zaJf77AFtkWdHB" width="359" height="54"></p>
<pre><code class="lang-auto">lstm:0.015676498413085938
custom lstm:1.0338680744171143
</code></pre>
<p>My torch version is 1.3.1, GPU is TITANV with Cuda10 and cuDNN7.4.1<br>
I also tried on pytorch 1.1, GPU TITAN XP with CUDA9.1, which has same ratio of speed.</p>
<p>Any idea?<br>
Thanks so much</p>

driazati: <p>The TorchScript runtime does some optimizations on the first pass (it assumes you will be running your compiled model’s inference many times), so this is likely why it looks much slower. Could you try running <code>custom_lstm</code> a couple times before you benchmark it and comparing?</p>

senmao: <p>I retest both class for 1000 times, and the result seems more reasonable.</p>
<blockquote>
<p>lstm:49.758071184158325<br>
custom lstm:55.80940389633179</p>
</blockquote>
<p>Thanks for you answer.</p>

huoge: <p>I met the same problem. is there any way to disable the optimization or choose the optimization level or after optimization we can save the model.because when I load the torchscript model in C++, the first pass takes about 20s while the others’ infer time is about 0.5s.</p>

G.M: <p>I think there’s a parameter called <code>optimize</code> for scripting in scripting using touchscript.</p>

huoge: <p>in the source code "<code>optimize</code> is deprecated and has no effect."<br>
<a href="https://pytorch.org/docs/stable/_modules/torch/jit.html#script" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/jit.html#script</a></p>

G.M: <p>The code suggested an alternative: <code>warnings.warn("`optimize` is deprecated and has no effect. Use `with torch.jit.optimized_execution() instead")</code></p>

villedepommes: <p>you could try setting <code>torch._C._jit_set_profiling_mode()</code> to <code>True</code> and <code>torch._C._jit_set_profiling_mode</code> to <code>False</code><br>
This mode was specifically added for speeding up compilation times for inference.<br>
You could also indeed try <code>with torch.jit.optimized_execution()</code> if compilation times are still high for you. The latter runs even fewer optimizations.</p>

huoge: <p>ok, thx. I will try this method</p>

huoge: <p>great, thanks. I will try to use these methods</p>

huoge: <p>when I use python the torch.jit.optimized_execution() could solve the problem, thanks<br>
however, how should I solve this problem in C++?<br>
thanks in andvanve</p>

villedepommes: <p>you could try this.</p>
<pre><code class="lang-auto">#include &lt;torch/csrc/jit/update_graph_executor_opt.h&gt;
//...
setGraphExecutorOptimize(false);
</code></pre>

huoge: <p>Great, thanks.I solved the problem by setGraphExecutorOptimize(false);</p>

KENG: <p>Hi, I still have some questions about the custom RNN:</p>
<ol>
<li>
<p>I am able to reproduce senmao’s results that lstm and custom lstm have similar performance in 1000 times, but this is partly due to the original lstm becomes worse. This can be seen in senmao’s results. The first run of the original lstm is 0.015. If the performance is consistent, 1000 runs would take 15 seconds instead of 49.758 as reported (I have verified this myself).</p>
</li>
<li>
<p>Although I have no idea why the original lstm becomes worse, I get rid of the problem by changing the hyper parameters to:<br>
input_size = 37<br>
cell_size =256<br>
batch_size =128<br>
seq_len = 60<br>
Now the original lstm becomes stable. In this case, the custom lstm is 10 times slower than the original lstm. Here are the results of 1000 runs: lstm:1.54s, custom lstm:19.75s. Can anyone please suggest how the custom lstm can be modified to have comparable performance with the original lstm?</p>
</li>
</ol>
<p>Thanks so much!</p>

David_Lorca: <p>Hi senmao! I’m writting you because I tried to get your results by my own… but, I cound’t get them. I have the same as you posted and I’m getting the time by this code:</p>
<p>for i in range(1000):</p>
<p>t1 = time.time()</p>
<p>out, hid = native_lstm(inp)</p>
<p>t2 = time.time()</p>
<p>t_native += t2-t1</p>
<p>t3 = time.time()</p>
<p>out2, hid2 = custom_lstm(inp, [(hx, cx)])</p>
<p>t4 = time.time()</p>
<p>t_custom += t4-t3</p>
<p>Thanks for all!</p>
<p>Regards,<br>
David.</p>