Entropy loss decrease sharply when training a drl agent

username: 007
cooked: <p>When I am training a drl agent with discrete action space using ppo-based algorithm, the training process is not stable and when training to 1800 episode around, the entropy loss deceases sharply and the agent got a really bad average reward. I have used nn.LayerNorm and state, reward is constrained to around (-1,1),  Does anyone has some good suggestions?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/55f9e37641b91888097025f71a026d997bb8baab.png" data-download-href="https://discuss.pytorch.org/uploads/default/55f9e37641b91888097025f71a026d997bb8baab" title="newplot(1)"><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/55f9e37641b91888097025f71a026d997bb8baab.png" alt="newplot(1)" data-base62-sha1="cgzY3ByK6C6nXD8I3kqxAlOP0y7" width="690" height="443" data-dominant-color="EEF4F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">newplot(1)</span><span class="informations">700×450 9.25 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/a7f5a7000c0cefccb32313edf94a2a05d6dc0670.png" data-download-href="https://discuss.pytorch.org/uploads/default/a7f5a7000c0cefccb32313edf94a2a05d6dc0670" title="newplot(2)"><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a7f5a7000c0cefccb32313edf94a2a05d6dc0670.png" alt="newplot(2)" data-base62-sha1="nXQ2O4QRGLCM2vzfc0MbxT9EccE" width="690" height="443" data-dominant-color="F5F9FB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">newplot(2)</span><span class="informations">700×450 15 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/5034087c9c559cafd7c19029d804532341c1f27d.png" data-download-href="https://discuss.pytorch.org/uploads/default/5034087c9c559cafd7c19029d804532341c1f27d" title="newplot(3)"><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5034087c9c559cafd7c19029d804532341c1f27d.png" alt="newplot(3)" data-base62-sha1="brvG2NOEKAhvaoP35KhJf0pvDkN" width="690" height="443" data-dominant-color="F7FAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">newplot(3)</span><span class="informations">700×450 14.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/08e9611461ee5aa5c6d013d39403331f626f08b3.png" data-download-href="https://discuss.pytorch.org/uploads/default/08e9611461ee5aa5c6d013d39403331f626f08b3" title="newplot(4)"><img src="https://discuss.pytorch.org/uploads/default/original/2X/0/08e9611461ee5aa5c6d013d39403331f626f08b3.png" alt="newplot(4)" data-base62-sha1="1gPPBRvqf6QzcjC8XcpbyGmKHqb" width="690" height="443" data-dominant-color="EEF4F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">newplot(4)</span><span class="informations">700×450 16.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>