Backward is too slow

mmmartin: <p>Hi<br>
I have installed cuda8 pytorch 0.4.1 with conda.<br>
the back-prop looks too slow.<br>
here is the report using util.bottleneck:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/986722bb240c79c4082be41fb3e57a2f2f294f42.png" data-download-href="https://discuss.pytorch.org/uploads/default/986722bb240c79c4082be41fb3e57a2f2f294f42" title="WX20180911-170102%402x"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/986722bb240c79c4082be41fb3e57a2f2f294f42_2_683x500.png" alt="WX20180911-170102%402x" data-base62-sha1="lKdxDYA9K9cU4SfBchYE2RJMfAu" width="683" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/986722bb240c79c4082be41fb3e57a2f2f294f42_2_683x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/9/986722bb240c79c4082be41fb3e57a2f2f294f42_2_1024x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/9/986722bb240c79c4082be41fb3e57a2f2f294f42_2_1366x1000.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WX20180911-170102%402x</span><span class="informations">1968×1440 326 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
here is my test code for main training process in debug mode</p>
<pre><code class="lang-auto">start1 = time.time()
for _ in range(100):
    word_in1 = torch.cuda.LongTensor(word_in)
    word_out1 = torch.cuda.LongTensor(word_out)
    label = torch.cuda.DoubleTensor(train_label)
    emb_u =  nn.functional.embedding(word_in1,syn0)
    emb_v =  nn.functional.embedding(word_out1,syn1)
    outs = torch.sigmoid(torch.sum(torch.mul(emb_u, emb_v), dim=-1))
    loss = Lossfunc.cuda()(outs, label)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
print (time.time() - start1)
</code></pre>
<pre><code class="lang-auto">    optimizer = optim.SGD([syn0, syn1], lr=alpha)
    Lossfunc = nn.BCELoss(reduction='sum')
</code></pre>
<p>and I found the last three lines(.zero_grad(),.backward(),.step()) occupy most of the time.<br>
So what should i do next?</p>

albanD: <p>Hi,</p>
<p>Why do you think it is too slow?<br>
Running the backward should be between 1 and 2x the forward pass.<br>
Then the gradient step depends on the size of your weights.</p>
<p>If your Embedding layers are very large compared to the rest of the net, you can do sparse updates by using <code>sparse=True</code> for it. See <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.functional.embedding">doc</a>. And by using an optimizer that supports sparse updates like <code>SGD</code> or <code>SparseAdam</code>.</p>

mmmartin: <p>Thank you for your reply.<br>
The reason I think it’s slow is that the same training process costs two minutes using numpy+CPU but  an hour using pytorch+GPU.<br>
here is my numpy code:</p>
<pre><code class="lang-auto">                    z = np.dot(syn0[context_word],syn1[word_out].T)
                    p = expit(z)
                    g = alpha * (label-p)
                    neu1e = syn1[x_]
                    syn1[x_] += np.outer(g,syn0[context_word])
                    syn0[context_word] += np.dot(g,neu1e)
</code></pre>
<p>There must be something wrong with my pytorch code.</p>

mmmartin: <p>According to my test, The time of backward is about 170x that of forward pass, which is 5.8s vs 0.035s in 100 cycles.</p>

albanD: <p>Hi,</p>
<p>Could you send me a full script that runs with all the sizes being the ones you use. Replacing your data with random tensors.<br>
I guess two possible things here:</p>
<ul>
<li>The graph expands accross iterations and thus the traversal for the backward becomes dead slow. I need the full running code to check that.</li>
<li>Give the simplicity of your graph and the fact that it’s mainly Embedding from what I see, you might be hitting a very bad worst case. Here again, try <code>sparse=true</code> for the embedding, it was made for that purpose.</li>
</ul>

mmmartin: <aside class="quote no-group" data-username="albanD" data-post="5" data-topic="24909" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>Hi,</p>
<p>Could you send me a full script that runs with all the sizes being the ones you use. Replacing your data with random tensors.<br>
I guess two possible things here:</p>
<ul>
<li>The graph expands accross iterations and thus the traversal for the backward becomes dead slow. I need the full running code to check that.</li>
<li>Give the simplicity of your graph and the fact that it’s mainly Embedding from what I see, you might be hitting a very bad worst case. Here again, try <code>sparse=true</code> for the embedding, it was made for that purpose.</li>
</ul>
</blockquote>
</aside>
<p>Thank you very much!<br>
Here is my code:</p>
<pre><code class="lang-auto">import torch.optim as optim
import torch
import torch.nn as nn
import numpy as np
import os
import time
if __name__ == '__main__':
    os.environ['CUDA_VISIBLE_DEVICES'] = '0'
    syn0 = torch.randn((2829,100),requires_grad=True,device='cuda')
    syn1 = torch.randn((2829,100),requires_grad=True,device='cuda')
    optimizer = optim.SGD([syn0, syn1], lr=0.025)
    Lossfunc = nn.BCELoss(reduction='sum').cuda()
    start1 = time.time()
    for index,_ in enumerate(range(40000)):
        word_in = np.random.randint(low=0, high=2829, size=32)
        word_out = np.random.randint(low=0, high=2829, size=32)
        if index%10000 == 0:
            print(('%d of 40000 (%.2f%%)')%(index,index / 400.0))
        word_in1 = torch.cuda.LongTensor(word_in)
        word_out1 = torch.cuda.LongTensor(word_out)
        label = torch.cuda.FloatTensor([1]+ [0]*31)
        emb_u = nn.functional.embedding(word_in1,syn0,sparse=True)
        emb_v = nn.functional.embedding(word_out1,syn1,sparse=True)
        outs = torch.sigmoid(torch.sum(torch.mul(emb_u, emb_v), dim=-1))
        loss = Lossfunc(outs,label)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print (time.time()-start1)
</code></pre>
<p>When i replace my data with random tensor,  the backward become about 2x slowly than the forward pass just like you said.</p>
<p>But it’s still very slow, it took about 1 min to run 40000 rounds in a Tesla M40. Is there any mistake in my code writing? If not, can you give me some suggestion to speed up it? Should i use torch.set_num_thread() or torch.multiprocessing?</p>
<p>Thanks again!</p>

albanD: <p>Hi,</p>
<p>Some observations:</p>
<ul>
<li>My first point does not happen, so no problem on that side.</li>
<li>Sparse=True make the perf slightly worst here. This is expected as your embedding is not that big.</li>
<li>In my tests, I moved all the data generation out of the loop but that does not change much.</li>
<li>Here each forward-backward-update takes &lt;1ms on my machine. I don’t think you can expect it to be much faster. It’s just that the outer loop is very large. Try removing the python <code>if</code> statement in your loop, you actually see the difference in runtime.</li>
<li>The gpu usage is actually quite low, increasing the batch size to 128 still gives me a runtime of &lt;1ms per iterations. So If you want this to run faster, increase the batch size.</li>
<li>torch.set_num_thread will only change cpu core usage for heavy operations. But you don’t do any such operation of cpu here.</li>
<li>torch.multiprocessing would allow you to do multiCPU-multiGPU. But you can’t use the one you have already fully, so there is little hope you can improve on that side.</li>
</ul>

mmmartin: <p>Thank you! Your advice is very helpful to me.</p>
<p>The GPU usage is indeed slow, but when i want to use CPU and remove all “.cuda”, it becomes about 50x slower, why?</p>

albanD: <p>Well because even low GPU usage is much faster than CPU. Especially for such ops.</p>

mmmartin: <p>But the following code based on numpy could run much faster for the same training process, just using CPU(4s VS 65s):</p>
<pre><code class="lang-auto">        syn0 = np.random.uniform(low=-0.5/100, high=0.5/100, size=(2829, 100))
        syn1 = np.zeros(shape=(2829,100))
        start1 = time.time()
        for index,_ in enumerate(range(40000)):
            x_ = np.random.randint(low=0, high=2829, size=32)
            if index%10000 == 0:
                print(('%d of 40000 (%.2f%%)')%(index,index / 400.0))
            context_word = np.random.randint(low=0, high=2829, size=1)[0]
            label = np.array([1] + [0]*5)
            z = np.dot(syn0[context_word], syn1[x_].T)
            p = expit(z)
            g = 0.025 * (label - p)
            neu1e = syn1[x_]
            syn1[x_] += np.outer(g, syn0[context_word])
            syn0[context_word] += np.dot(g, neu1e)
        print(time.time() - start1)
</code></pre>
<p>I want to use autograd function provided by pytorch, but it slow down severely.<br>
I tried increase the batch size, and the performance will drop correspondingly.</p>

albanD: <p>Hi,</p>
<p>It is expected that there is some overead from the autograd engine especially for such a small graph. But it looks a bit too much in this case.<br>
I’m not super fluent in numpy code but it looks like:</p>
<ul>
<li>Your context word is of size 1, while in the pytorch code it’s of size batch_size=32</li>
<li>Your label is of size 6 while in the pytorch code it’s of size 32</li>
<li>What is the <code>expit</code> function doing?</li>
<li>Have you tried replacing each op from your numpy code with the torch counterpart ? This should give similar runtime on cpu and speedup on gpu if the ops are big enough.</li>
</ul>

SkyAndCloud: <p>Hi~ I also encountered this problem. My code is like this:</p>
<pre><code class="lang-auto"># fd_prob: [batch_size, tgt_len, vocab_size], means words probability distribution
# bd_hyp: [batch_size, infer_len], means another output labels
# fd_bd_attn: [batch_size, tgt_len, infer_len], means edit probability distribution
# fd_p_gen: [batch_size, tgt_len, 1], means copy mode probability
batch_size, tgt_len, _ = fd_prob.size()
_, infer_len = bd_hyp.size()
# incorporate copy mode
for i in range(batch_size):
    for j in range(tgt_len):
        for k in range(infer_len):
            fd_prob[i][j][bd_hyp[i][k]] += (1 - fd_p_gen[i][j][0]) * fd_bd_attn[i][j][k]
loss = criterion(fd_prob, ground_truth)
loss.backward()
</code></pre>
<p>I modified output distribution by a 3-layer loop which contained lots of index, and I found that forward cost less than 1s while backward cost more than 1.5min, which is unacceptable. I think it’s due to lots of index. Have you found the solution or is there an elegant way to do this?<br>
Also see <a href="https://discuss.pytorch.org/t/indexing-is-very-slow-for-backpropagation/10752">Indexing is very slow for backpropagation</a></p>

albanD: <p>Hi,</p>
<p>This ia expected. Each operations that you do add an operation in the computational graph. You’re creating a huge graph here so the backward pass is going to be very slow. You will need to parallelize your operations using builtin functions and/or masking.</p>

SkyAndCloud: <p>Thank you for your quick reply. I fixed this problem since I used <code>scatter_add_</code> to do this operation.</p>

gzy-gxq: <p>Could you provide your modified code?  I meet same problem</p>

Yongjie_Shi: <aside class="quote no-group" data-username="SkyAndCloud" data-post="14" data-topic="24909" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/f475e1/48.png" class="avatar"> SkyAndCloud:</div>
<blockquote>
<p>Thank you for your quick reply. I fixed this problem since I used <code>scatter_add_</code> to do this operation.</p>
</blockquote>
</aside>
<p>I meet the same problem, could you please provide the modified code?</p>

neoncube: <p><a class="mention" href="/u/skyandcloud">@SkyAndCloud</a> I wanted to say thank you for pointing me towards the <code>scatter</code> functions <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>For anyone else who runs across this, here is an example of converting a loop to use <code>scatter_()</code></p>
<p><strong>Original code</strong></p>
<pre data-code-wrap="python"><code class="lang-python">for i in range(len(next_model_states)):
    target[i][actions[i]] = next_model_states[i]
</code></pre>
<p><strong>With scatter_()</strong></p>
<pre data-code-wrap="python"><code class="lang-python">target.scatter_(
  1,
  actions.view(len(next_model_states), 1),
  torch.unsqueeze(next_model_states, 0).view(len(next_model_states), 1)
)
</code></pre>
<p>Your <code>scatter_()</code> code will probably look different, but that’s the general idea! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I found this tutorial very helpful for understanding how <code>scatter_()</code> works: <a href="https://yuyangyy.medium.com/understand-torch-scatter-b0fd6275331c" class="inline-onebox" rel="noopener nofollow ugc">Understand torch.scatter_(). First, note that scatter_() is an… | by Yu Yang | Medium</a></p>