Subtle difference of RMSprop between PyTorch and Tensorflow

ultmaster: <p>I think this is a follow-up discussion of <a href="https://github.com/pytorch/pytorch/issues/23796" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/23796</a>.</p>
<p>It’s a bit of cliche, but when trying to reproduce a repo of Tensorflow using PyTorch, RMSprop again failed to align. Apart from eps inside/outside problem, I spotted another two differences, which I think should be open to discussion. (I could be wrong since I know little about all those optimization tricks.)</p>
<ol>
<li>Momentum is not multiplied by another lr in <a href="https://github.com/tensorflow/tensorflow/blob/a592eff24bdae523970ec636d38d8c6b221de40a/tensorflow/core/kernels/training_ops_gpu.cu.cc#L428" rel="nofollow noopener">tensorflow’s implementation</a>, compared to <a href="https://github.com/pytorch/pytorch/blob/33430cf0946650058118096f78741440676e33c0/torch/optim/rmsprop.py#L104" rel="nofollow noopener">PyTorch implementation</a>. To align, I think RMSprop should be</li>
</ol>
<pre><code class="lang-auto">buf.mul_(group['momentum']).addcmul_(grad, group['lr'] / avg)
p.data.add_(-buf)
</code></pre>
<ol start="2">
<li>Weight decay should be multiplied by 2 if I’m using PyTorch. As far as I understand, in <a href="https://github.com/tensorflow/tensorflow/blob/a592eff24bdae523970ec636d38d8c6b221de40a/tensorflow/python/keras/regularizers.py#L221" rel="nofollow noopener">Tensorflow/keras</a>, the l2 loss is not multiplied by 1/2, the derivative of which should be multiplied by 2. However this 2 is missing in <a href="https://github.com/pytorch/pytorch/blob/33430cf0946650058118096f78741440676e33c0/torch/optim/rmsprop.py#L90" rel="nofollow noopener">PyTorch’s implementation</a>.</li>
</ol>
<p>Correct me if I’m wrong. I’m also interested in why these differences exist if they are by design.</p>

leengsmile: <p>I guess the idea behind the update rule for <code>buf</code> here(rmsprop) is to stay consistent with that for <strong>sgd with momentum</strong>.</p>
<pre><code class="lang-auto">buf.mul_(momentum).add_(1 - dampening, d_p)
</code></pre>
<p><a href="https://github.com/pytorch/pytorch/blob/33430cf0946650058118096f78741440676e33c0/torch/optim/sgd.py#L100" rel="noopener nofollow ugc">SGD Momentum implementation</a></p>
<p>In pytoch implementation, the <code>momentum</code> is slightly changed, refer to <a href="https://pytorch.org/docs/stable/optim.html?highlight=sgd#torch.optim.SGD" rel="noopener nofollow ugc">sgd doc</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4b2cd1fbad3f8b0013157c3bcd8c2f87ce659f4.png" data-download-href="https://discuss.pytorch.org/uploads/default/e4b2cd1fbad3f8b0013157c3bcd8c2f87ce659f4" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4b2cd1fbad3f8b0013157c3bcd8c2f87ce659f4.png" alt="image" data-base62-sha1="wD9WtGAeroc5fd591PtsfnZEOrO" width="690" height="405" data-dominant-color="ECEDF0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">929×546 39.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>They are the same for Normal SGD and pytorch SGD when learning rate is fixed.</p>
<p>Hope this will be helpful.</p>

amorphous: <p>On <span class="hashtag">#1</span> - both equations end up having the same result…, if momentum_buffer=0 at t=0 (which is what pytorch uses as initialization, but am not sure TF)</p>