How to implement accumulated gradient in pytorch (i.e. iter_size in caffe prototxt)

username: Hengck
cooked: <p>how to can i accumulate gradient during gradient descent in pytorch (i.e. iter_size in caffe prototxt).<br>
Currently, my code is:</p>
<pre><code class="lang-auto">     for iter, (images, labels, indices) in enumerate(train_loader, 0):
 
            optimizer.zero_grad() 
            outputs = net(Variable(images.cuda()))
            loss    = criterion(outputs, Variable(labels.cuda()))
            loss.backward()
            optimizer.step()
</code></pre>
<p>Do i do this?</p>
<pre><code class="lang-auto">     for iter in range(N):
 
            optimizer.zero_grad() 
            loss = 0

            for i in range(M):
                  (images, labels, indices)=train_loader.next():
                   outputs = net(Variable(images.cuda()))
                   loss    +=  criterion(outputs, Variable(labels.cuda()))
                   loss.backward()

            optimizer.step()
            loss = loss/M
</code></pre>

username: smth
cooked: <p>yes, you are on point. not zeroing the grad will keep accumulating the gradient in</p>

username: zhuyi490
cooked: <p><a class="mention" href="/u/hengck">@Hengck</a> <a class="mention" href="/u/smth">@smth</a> Hi, I have a quick question. As mentioned in <a href="https://discuss.pytorch.org/t/cuda-memory-continuously-increases-when-net-images-called-in-every-iteration/501/8">here</a>,</p>
<blockquote>
<p>loss    +=  criterion(outputs, Variable(labels.cuda()))</p>
</blockquote>
<p>this will build the graph again and again inside the loop, which may increase memory usage. So should I just write</p>
<blockquote>
<p>loss    =  criterion(outputs, Variable(labels.cuda()))</p>
</blockquote>
<p>This will also accumulate the gradients, right? I am confusing about which one to use, “=” or “+=”? I just want to have the effect of “iter_size” in Caffe to train large models. Thanks.</p>

username: Hengck
cooked: <p>Here is the corrected code</p>
<pre><code class="lang-auto"> for i in range(num_iters):
 
            optimizer.zero_grad() 
            batch_loss_value = 0

            for m in range(M):
                  (images, labels, indices) = train_loader.next():
                   outputs = net(Variable(images.cuda()))
                   loss    = criterion(outputs, Variable(labels.cuda()))
                   loss.backward()
                  
                  batch_loss_value += loss.cpu().numpy()[0]

            optimizer.step()
            batch_loss_value = batch_loss_value/M
</code></pre>

username: Zijun_Wei
cooked: <p>A follow up question: how is the result via this way different from feed a batch-size to M GPUs?<br>
Is this only a matter of speed?</p>

username: will_soon
cooked: <p>Have you tried it, does it perform better than smaller batch_size?</p>

username: kenfehling
cooked: <p>What do you do then with batch_loss_value, call backward() on it?</p>

username: riccardosamperna
cooked: <p>Hey Heng,</p>
<p>I am having a similar problem.</p>
<p>I am trying to build a recurrent neural network that accumulates the gradient over each sequence and performs backpropagation through time. Do you think I can achieve that with the code you posted? If you prefer I can show you what I have so far in my training.</p>

username: Mata_Fu
cooked: <p>Hi, have you solved this problem?</p>

username: saransh_karira
cooked: <p>Also be sure to either scale down the gradients before update or decrease the learning rate</p>

username: Ray_Wong
cooked: <p>Hello, riccard.  Do you solve the problem please? I have a similar problem. If following the code example here, you make it, please let me know. Thanks</p>

username: singleroc
cooked: <p>One issue of the iter_size is the BatchNorm. A bigger BS (for example bs=128, iter_size=1) doesn’t give the same result of bs=64 and iter_size=2.</p>

username: peter_wills
cooked: <p>I’m guessing that he plots it.</p>

username: DTCancri
cooked: <p>I’m having the same issue. I use this method to increase batch_size by 100.</p>
<p>I plotted the training loss with 3 different settings (batch_size 200, batch_size 1 * 200 multiplier, batch_size 10 * 20 multiplier) in the following picture:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/1/61108cb5a8427d1391965f7fd21792b734bd253a.png" data-download-href="https://discuss.pytorch.org/uploads/default/61108cb5a8427d1391965f7fd21792b734bd253a" title="Screenshot%20from%202019-09-18%2014-38-28"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61108cb5a8427d1391965f7fd21792b734bd253a_2_690x211.png" alt="Screenshot%20from%202019-09-18%2014-38-28" data-base62-sha1="dQFLRUilaIdCjKgxdjFSOjy5o7g" width="690" height="211" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61108cb5a8427d1391965f7fd21792b734bd253a_2_690x211.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61108cb5a8427d1391965f7fd21792b734bd253a_2_1035x316.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/1/61108cb5a8427d1391965f7fd21792b734bd253a.png 2x" data-dominant-color="F4F0F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-09-18%2014-38-28</span><span class="informations">1309×401 113 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As you see, these 3 loss curves are almost equivalent. But I don’t know why the test accuracy is much worse when I use this large batch multiplier. Thanks!</p>

username: MaxT123
cooked: <p>Hi Heng,</p>
<p>I am dealing with a similar problem and you post helps. Thanks!<br>
There is one thing I would like to point out: You probably want to divide gradient by M, if you intended to average gradients for M iterations</p>