Poor Convergence on PyTorch compared to TensorFlow using Adam Optimizer

username: Pritish_Yuvraj
cooked: <p>I am <strong>providing</strong> a <strong>reproducible problem</strong> to compare poor convergence of PyTorch model compared to Tensorflow for Adam optimizer with learning rate = 0.001, beta (0.9, 0.999), and elipson=1e-8.</p>
<p>The model is:</p>
<pre><code class="lang-auto">convLayer = nn.Conv1d(in_channel=24, out_channel=128, kernel_size=15,stride=1,padding=7)
convLayerGate = nn.Conv1d(in_channel=24, out_channel=128, kernel_size=15,stride=1,padding=7)
GLU = convLayer * Torch.sigmoid(convLayerGate)
</code></pre>
<p>Input to both PyTorch and Tensorflow (for comparison and reproducibility):</p>
<pre><code class="lang-auto">np.random.seed(0)
input = np.random.randn(1, 24, 128)
output = np.random.randn(1, 128, 128)
</code></pre>
<p><strong>Loss in Tensorflow:</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/56dc4b8bf903540e4367f8d4cc40abc55367d9e9.png" data-download-href="https://discuss.pytorch.org/uploads/default/56dc4b8bf903540e4367f8d4cc40abc55367d9e9" title="TensorFlow"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/56dc4b8bf903540e4367f8d4cc40abc55367d9e9_2_690x343.png" alt="TensorFlow" data-base62-sha1="cop2DxhxPqfuAa6GdYm7Bh4jLol" width="690" height="343" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/56dc4b8bf903540e4367f8d4cc40abc55367d9e9_2_690x343.png, https://discuss.pytorch.org/uploads/default/optimized/2X/5/56dc4b8bf903540e4367f8d4cc40abc55367d9e9_2_1035x514.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/5/56dc4b8bf903540e4367f8d4cc40abc55367d9e9_2_1380x686.png 2x" data-dominant-color="FEFEFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">TensorFlow</span><span class="informations">1920×955 28.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Loss in PyTorch</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3cf3156b3f2946aa833f99698625a1bcf7256258.png" data-download-href="https://discuss.pytorch.org/uploads/default/3cf3156b3f2946aa833f99698625a1bcf7256258" title="PyTorch"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3cf3156b3f2946aa833f99698625a1bcf7256258_2_690x343.png" alt="PyTorch" data-base62-sha1="8Hbs3JdCvB5cb8YrpwN1NUvwvpu" width="690" height="343" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3cf3156b3f2946aa833f99698625a1bcf7256258_2_690x343.png, https://discuss.pytorch.org/uploads/default/optimized/2X/3/3cf3156b3f2946aa833f99698625a1bcf7256258_2_1035x514.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/3/3cf3156b3f2946aa833f99698625a1bcf7256258_2_1380x686.png 2x" data-dominant-color="FEFEFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">PyTorch</span><span class="informations">1920×955 28.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Tensorflow converges Loss to → 0.0030888948<br>
PyTorch Converges Loss to → 0.012638479471</p>
<p>If one implements the model in Tensorflow one can spot the difference.</p>

username: SimonW
cooked: <p>Did you make sure that you initialized the layers <strong>identically</strong> in pytorch and tf?</p>

username: Pritish_Yuvraj
cooked: <p>I used the default initialization for both PyTorch and Tensorflow. Even if the initialization scheme is not same, it seems unreasonable that Tensorflow converges to a loss which is 4 times better than what PyTorch converges to for such a simple model.</p>

username: SimonW
cooked: <p>Initialization matters a lot… please try with same initial value.</p>

username: SimonW
cooked: <p>Initialization matters a lot… please try with same initial value.</p>

username: Pritish_Yuvraj
cooked: <p>To level the <strong>playing fields</strong> between both Tensorflow and PyTorch:</p>
<ol>
<li>Both are now using <strong>“Glorot Uniform” Initializer.</strong></li>
<li>Bias is turned off for both the models.</li>
</ol>
<p>New Tensorlfow model loss function:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/68099c58af649f29a2bb0f6f14f82de417e44b96.png" data-download-href="https://discuss.pytorch.org/uploads/default/68099c58af649f29a2bb0f6f14f82de417e44b96" title="Tensorflow_no_bias"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/6/68099c58af649f29a2bb0f6f14f82de417e44b96_2_690x343.png" alt="Tensorflow_no_bias" data-base62-sha1="eQmfmBIK59Xe7Wcg1HcuidoLB4y" width="690" height="343" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/6/68099c58af649f29a2bb0f6f14f82de417e44b96_2_690x343.png, https://discuss.pytorch.org/uploads/default/optimized/2X/6/68099c58af649f29a2bb0f6f14f82de417e44b96_2_1035x514.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/6/68099c58af649f29a2bb0f6f14f82de417e44b96_2_1380x686.png 2x" data-dominant-color="FEFEFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Tensorflow_no_bias</span><span class="informations">1920×955 29.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>New PyTorch model loss function:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0b2587a90bd1a1dd237e43ea11810784d5ba09c4.png" data-download-href="https://discuss.pytorch.org/uploads/default/0b2587a90bd1a1dd237e43ea11810784d5ba09c4" title="PyTorch_no_bias"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0b2587a90bd1a1dd237e43ea11810784d5ba09c4_2_690x343.png" alt="PyTorch_no_bias" data-base62-sha1="1ABETKwRIYvFKlpJC0u0UbgKae8" width="690" height="343" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0b2587a90bd1a1dd237e43ea11810784d5ba09c4_2_690x343.png, https://discuss.pytorch.org/uploads/default/optimized/2X/0/0b2587a90bd1a1dd237e43ea11810784d5ba09c4_2_1035x514.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/0/0b2587a90bd1a1dd237e43ea11810784d5ba09c4_2_1380x686.png 2x" data-dominant-color="FEFEFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">PyTorch_no_bias</span><span class="informations">1920×955 29 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Tensorflow minimum loss: 0.00354829<br>
PyTorch minimum loss: 0.011800370</p>

username: RyanS
cooked: <p>Has any further progress been made on this issue as of yet? I am struggling with a problem very similar in nature to this.</p>