Suboptimal convergence when compared with TensorFlow model

christianperone: <p>I ported a simple model (using dilated convolutions) from TensorFlow (written in Keras) to pytorch (last stable version) and the convergence is very different on pytorch, leading to results that are good but not even close of the results I got with TensorFlow. So I wonder if there are differences on optimizers in Pytorch, what I already checked is:</p>
<ul>
<li>Same parameters for optimizer (Adam)</li>
<li>Same loss function</li>
<li>Same initialization</li>
<li>Same learning rate</li>
<li>Same architecture</li>
<li>Same amount of parameters</li>
<li>Same data augmentation / batch size</li>
</ul>
<p>So I wonder, what else should I check ? It seems that everything was covered already. Any ideas ?</p>
<p>Example of convergence of the loss (unfortunately the two series are the same color, but pytorch is the one below), for this loss, higher is better:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/ea369adc57732e9acf2c536800ad58f7a746547a.png" data-download-href="https://discuss.pytorch.org/uploads/default/ea369adc57732e9acf2c536800ad58f7a746547a" title="Screen Shot 2017-07-21 at 11.16.44 AM.png"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ea369adc57732e9acf2c536800ad58f7a746547a_2_690x379.png" width="690" height="379" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ea369adc57732e9acf2c536800ad58f7a746547a_2_690x379.png, https://discuss.pytorch.org/uploads/default/original/2X/e/ea369adc57732e9acf2c536800ad58f7a746547a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/e/ea369adc57732e9acf2c536800ad58f7a746547a.png 2x" data-dominant-color="FCF9F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2017-07-21 at 11.16.44 AM.png</span><span class="informations">884×486 20.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thank you !</p>

recastrodiaz: <p>I have had similar issues with Pytorch vs Keras, but while I haven’t found a simple answer, these are other things I would check:</p>
<ul>
<li>Is Keras using any regularizers or constraints?</li>
<li>Is Keras using biases whilst PyTorch is not?</li>
<li>Are you computing the loss the exact same way?</li>
</ul>

christianperone: <p>Thanks for the answer Rodrigo ! I’m not using any constraint or regularizer and biases are also the same. I’ll try to double check everything again, but this is really weird.</p>

tom: <p>Sometimes subtle differences in the definitions can make comparisons difficult, e.g. when a mean is taken in the loss function when your comparison uses a sum, you would need to compensate for that.<br>
Also there are quite a few parameters to the optimizer that have differing defaults between implementations.<br>
I’d probably try to start both with the same weights and see where it differs.</p>
<p>Best regards</p>
<p>Thomas</p>

christianperone: <p>Thanks Thomas ! I’ll certainly check that !</p>

christianperone: <p>I have some news regarding this issue:</p>
<ul>
<li>
<p>I initialized the model in Pytorch with the same weights of a model trained on Keras (using TensorFlow backend) and suprisingly, the results of this new model with the same weights yield the <strong>SAME</strong> results from Keras model;</p>
</li>
<li>
<p>However, if I train this same model on Pytorch (even using the same initialization weights), Pytorch <strong>always yields suboptimal results</strong>, like the one saw on the image above;</p>
</li>
</ul>
<p>My opinion is that something is really weird with the Adam optimizer in Pytorch, yielding poor results when compared to Keras/TensorFlow. So my question is: how were these optimizers tested ? It is consistently yielding poor results to me. Does anyone saw that before ? Is there any workaround for that ?</p>
<p>Thanks !</p>

tom: <p>Did you check the initialization and the loss, too?<br>
I think keras uses Xavier Glorot’s method by default for some layers.<br>
You probably could extend your comparison to the gradient and possible regularizer and then look at the optimizer step itself.<br>
The good news is that you can probably just “bisect” the backward pass if you find that the gradients differ.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>(who is also trying to stare down a model where he cannot make sense of the apparent training deficiency)</p>

christianperone: <p>Thanks <a class="mention" href="/u/tom">@tom</a>, actually I used the same initial weights (not only the same initialization method) to train it, so it really seems to be something fundamentally wrong with the Adam optimizer itself (I also checked the loss multiple times). I think that I’ll wait for Pytorch to stabilize because I don’t have so much time to invest in debugging it, unfortunately. Dissecting every aspect of the model takes a <strong>lot of time</strong>. Good luck with your model by the way !!</p>

Kaixhin: <p>This is peculiar, but thanks for testing this thoroughly. I’ve had a quick look at <a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/adam.py" rel="nofollow noopener">PyTorch Adam</a> (seems fine) vs. <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/training/adam.py" rel="nofollow noopener">TensorFlow Adam</a> (complex, but also seems fine) vs. <a href="https://github.com/fchollet/keras/blob/master/keras/optimizers.py#L376" rel="nofollow noopener">Keras Adam</a> (also seems fine), and can’t spot any issues, but perhaps someone more observant will.</p>

christianperone: <p>Just to complement, I also tried this change: <a href="https://github.com/pytorch/pytorch/issues/2060" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/2060</a> but it didn’t change anything in the convergence.</p>

dgriff: <p>at first glance. it looks like tensor flow is using a slightly different version of episilon definition in their Adam . They are using the the “epsilon hat” version</p>
<p>they replace these three lines of algorithm:</p>
<pre><code class="lang-auto">m t ←mt/(1−β1t)(Computebias-correctedfirstmomentestimate)
v t ← vt /(1 − β2t ) (Compute bias-corrected second raw moment estimate) √
θt ←θt−1 −α·m t/( v t +ε)(Updateparameters)
</code></pre>
<p>with these two lines:</p>
<pre><code class="lang-auto">αt =α· 1−β2t/(1−β1t)
θt ←θt−1 −αt ·mt/(√vt +εˆ).
</code></pre>
<p>and just took a glance at keras and seems they are too</p>
<p><strong>EDIT:</strong>  Scratch that. We are using the same here as well. We use the bottom two lines as well</p>

dgriff: <p>Tensorflow Adam –   “Momentum decay (beta1) is also applied to the entire momentum<br>
accumulator. This means that the sparse behavior is equivalent to the dense<br>
behavior (in contrast to some momentum implementations which ignore momentum<br>
unless a variable slice was actually used).”</p>
<p>Yeah this will cause performance differences.</p>
<p>But it’s not that pytorch is using a weird Adam optimizer in contrary it’s looks like pytorch just has the plain vanilla version. You guys see the same thing?</p>

Kaixhin: <p>The paper says:</p>
<blockquote>
<p>Note that the efficiency of algorithm 1 can, at the expense of clarity, be improved upon by changing the order of computation, e.g. by replacing the last three lines in the loop with the following lines…</p>
</blockquote>
<p>So I’m surprised that it should make a noticeable difference, but maybe that is the case. <a class="mention" href="/u/christianperone">@christianperone</a> would you mind trying the altered version of Adam on your problem? Fingers crossed this might be the solution.</p>
<p>The bit about sparse updates with TensorFlow Adam I would assume don’t matter in this case. I don’t know how PyTorch deals with sparse modules wrt gradient updates, but what TF claims to do sounds like the correct approach.</p>

dgriff: <p>It was this part that made me think it could lead to noticeable difference–</p>
<pre><code class="lang-auto">Tensorflow Adam – “    The sparse implementation of this algorithm (used when the gradient is an
    IndexedSlices object, typically because of `tf.gather` or an embedding
    lookup in the forward pass) does apply momentum to variable slices even if
    they were not used in the forward pass (meaning they have a gradient equal to zero. Momentum decay (beta1) is also applied to the entire momentum
accumulator. This means that the sparse behavior is equivalent to the dense
behavior (in contrast to some momentum implementations which ignore momentum
unless a variable slice was actually used).”
</code></pre>
<p>As I see a lot of training embedding models in pytorch and would be comparing to tensorflow I bet a lot of these performance differences stem from that as it auto applies the momentum decay and we would have default not too.</p>
<p>Anyways I have always been able to get just as good or better than tensorflow performance but I usually use custom stuff most the time but the underlying framework has shown no insuffiency in performance for me and usually find quite the opposite</p>

dgriff: <p>Small minute differences in hyperparameters do often show unproportional performance differences in my experience</p>

christianperone: <p>I’ll test it. If someone has the code change in hands that would help a lot, otherwise I’ll have to come back to this in near future due to my time constraints. Thanks for the help !</p>

Pawel_Subko: <p>I also experienced suboptimal behaviour with Adam compared to SGD in PyTorch. Similar code in Tensorflow performed the other way around, i.e. optimizing with Adam was much easier. I have also used an Embedding layer.</p>

stefanonardo: <p>I thought I was the only one! Same problem here: <a href="https://discuss.pytorch.org/t/rnn-and-adam-slower-convergence-than-keras/11278/2">RNN and Adam: slower convergence than Keras</a></p>
<p>When I’ll have time I’ll try with other optimizers.</p>
<p>EDIT: same situation with RMSProp.</p>

stefanonardo: <p>Up… Shouldn’t this problem be investigated?</p>

Re3aa: <p>Same issue here, same model architecture in Keras that is trained using Adam, gives better result comparing with Pytorch.</p>