Loss first decreased and then increased in the training set

fan_zhimin: <p>I don’t know why. I tried to visualize the parameters and the gradients. I checked the distribution of some parameters, as shown in the figure below：<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/4/24bfc4eae89e92305c3708b9b95f6eac4adf8771.png" data-download-href="https://discuss.pytorch.org/uploads/default/24bfc4eae89e92305c3708b9b95f6eac4adf8771" title="微信图片_20210523205859"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/24bfc4eae89e92305c3708b9b95f6eac4adf8771_2_689x231.png" alt="微信图片_20210523205859" data-base62-sha1="5f63g5utjeiCCblNN13drsejRzr" width="689" height="231" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/24bfc4eae89e92305c3708b9b95f6eac4adf8771_2_689x231.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/24bfc4eae89e92305c3708b9b95f6eac4adf8771_2_1033x346.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/4/24bfc4eae89e92305c3708b9b95f6eac4adf8771.png 2x" data-dominant-color="FCFAF9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">微信图片_20210523205859</span><span class="informations">1344×451 102 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>one layer parameters is as follows:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/1/21f75f34e4a2b635ec901bf3e1fb4a9e13bc1a92.png" alt="image" data-base62-sha1="4QtKoLrGtJ0ffZTgOFX2Yu4cQro" width="669" height="439"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1a48f9dbc1916f6027b238de9f5a98c414403331.jpeg" alt="image" data-base62-sha1="3KwLpab68bGLwCY0msi4OJ4cQvv" width="686" height="473"></p>
<p>The graph of gradient is as follows：<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1ecb0c4cf983d0f9622630ab1f1dbaa6691ce224.png" alt="image" data-base62-sha1="4oplRdjKxTYdzYV02FjD0WI6r7S" width="658" height="464"></p>
<p>But I don’t know where to start analyzing parameters and gradients。<br>
I read some articles saying that it was because of the gradient explosion, but from the gradient and other information, I don’t know how to find out the reason。</p>
<p>And you can see that the parameters have changed, but my gradient image is still 0. What’s wrong with me?， In the visualization, I get the code of parameters and gradient information as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/0/60e38b7943fe80e06bd9c59b2f1def6853b228d2.png" data-download-href="https://discuss.pytorch.org/uploads/default/60e38b7943fe80e06bd9c59b2f1def6853b228d2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/0/60e38b7943fe80e06bd9c59b2f1def6853b228d2.png" alt="image" data-base62-sha1="dP7lGcQeFYwua8aKSph92ZtTIvE" width="690" height="73" data-dominant-color="323334"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">760×81 3.02 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Who can help me? Thanks</p>