REINFORCE not able to learn policy

username: gabrieldlm
cooked: <p>Hi everyone,</p>
<p>I´m trying to implement REINFORCE as described in Sutton<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/19894deaed71d75103199abec41ccdda15ab8888.png" data-download-href="https://discuss.pytorch.org/uploads/default/19894deaed71d75103199abec41ccdda15ab8888" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19894deaed71d75103199abec41ccdda15ab8888_2_690x248.png" alt="image" data-base62-sha1="3DU6OS7LKJmqhfhz1fsv9R5F0P6" width="690" height="248" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19894deaed71d75103199abec41ccdda15ab8888_2_690x248.png, https://discuss.pytorch.org/uploads/default/original/3X/1/9/19894deaed71d75103199abec41ccdda15ab8888.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/9/19894deaed71d75103199abec41ccdda15ab8888.png 2x" data-dominant-color="D7D7D7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">776×279 41.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However it doesn´t seem to be learning. Bellow is the code for the train function (using Gymnasium CartPole-v1)</p>
<p>´´´<br>
def train(ep, policy, optimizer, gamma=1):<br>
G = 0<br>
for t in range(len(ep)-1):<br>
st, at, rt = ep[t]<br>
G = rt + gamma * G<br>
pi = torch.log(policy(st)[at])  # My NN last layer is a softmax<br>
loss = -G*pi<br>
optimizer.zero_grad()<br>
loss.backward()<br>
torch.nn.utils.clip_grad_norm_(policy.parameters(), 1.)<br>
optimizer.step()<br>
´´´</p>
<p>What it does is basicaly choose to ways, go for a real bad policy, averaging 10 for the rewards (while a random agente usually scores in the ~20s) or learn a little bit and stay arround 80 avg reward.</p>
<p>First I thought that the policy was changing to abruptly so I inserted the clip_grad_norm_, which increased the amount of time the agent got stucked at the 80ish but didn´t did much else (maybe got stucked in some local minima?)</p>
<p>I tried to play with weight initialization, net archetecture but to no avail. I would appreciate if someone could shed light in what maybe the issue.</p>

username: vmoens
cooked: <p>Not sure this naive version will do the job if the task is hard-ish… You should perhaps use some more evolved algorithm like PPO or a better advantage estimation like GAE.<br>
One thing I spotted in the code is that you seem to be computing the advange starting from t=0 and ending at t=H, but it should be</p>
<pre><code class="lang-auto">for t in range(len(traj), -1, -1):
    G = r[t] + gamma * G
</code></pre>

username: gabrieldlm
cooked: <blockquote>
<p>Not sure this naive version will do the job if the task is hard-ish…</p>
</blockquote>
<p>Yeah, I was wondering about that, however I was expecting some sort of “asymptotically” learning instead of the plateau that is happening<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5df68921c82c2267158e6df5524fddb15207cb3e.png" alt="train_reinforce" data-base62-sha1="dpeBvACQNawFBZXQC8vFVdqI8Mu" width="640" height="480"><br>
This Is a “good” run (blue lines the reward for each episode the orange is the moving average)</p>
<blockquote>
<p>You should perhaps use some more evolved algorithm like PPO or a better advantage estimation like GAE.</p>
</blockquote>
<p>I´m trying to implement the algorithms my self to get a better fealling for how things work, PPO is the next one.</p>