Help with PyTorch Policy Gradient agent that learns actions resulting in consistent negative rewards

username: jkking
cooked: <p><strong>EDIT</strong>:  Lowering the learning rate all the way down to 0.0001 seems to make a big difference here. And changing the reinforce part away from pytorch’s example to another implementation seems to help but haven’t figured out all the details yet.</p>
<hr>
<p>I have a toy RL project implementing he REINFORCE algorithm with a policy gradient agent that seems to be consistently learning to choose actions that reliably generate negative rewards. Are there any likely reasons this could be so?</p>
<p>Additional details below. The rules of the environment are of my own design. Since that makes it obviously hard to diagnose, I’ll take whatever high or low granularity suggestions I can get.</p>
<hr>
<p>My environment is a 25x25 grid upon which a bot can move and attempt to acquire a target.</p>
<p>Bots can choose the following 17 actions:</p>
<ul>
<li>
<p>Do nothing</p>
</li>
<li>
<p>Move 1 space (in any of the 8 directions surrounding it)</p>
</li>
<li>
<p>Attempt to acquire a target 1 space away (in any of the 8 directions surrounding it)</p>
</li>
</ul>
<p>The reward system is such that:</p>
<ul>
<li>
<p>+1 for moving closer to any target</p>
</li>
<li>
<p>-1 for moving farther away from the targets</p>
</li>
<li>
<p>+10 for acquiring a target</p>
</li>
<li>
<p>-0.5 for any invalid action (i.e. moving outside the grid, aiming where a target doesn’t exist)</p>
</li>
</ul>
<p>Somehow, the bots almost always seem to learn the behavior of constantly heading in the direction of a wall. You can see this in the attached image.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f04da528cd4d1a7d2d8be0ccedd61a2bef1557af.png" data-download-href="https://discuss.pytorch.org/uploads/default/f04da528cd4d1a7d2d8be0ccedd61a2bef1557af" title="grid_agents"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f04da528cd4d1a7d2d8be0ccedd61a2bef1557af.png" alt="grid_agents" data-base62-sha1="yhOWHOEgzBqrXyNhJEJZuG8dgLR" width="484" height="500" data-dominant-color="F5F9F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grid_agents</span><span class="informations">577×596 6.37 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>There are two bots, each on “teams” (red and blue), but this is just for future implementation and doesn’t have any effect on their policy. Green are the targets.</p>
<p>Here is the link to the full code: <a href="https://colab.research.google.com/drive/1FehDqc291HOAGAkua4Q8eAn5BNUZqr2m?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
<p>but since that’s long I’ll try and highlight what I imagine are the relevant bits. Here’s the forward/action pass of the network:</p>
<pre><code class="lang-auto">    def forward(self, x):

      # Run through all of our layers defined above
      x = ... # cut for brevity
      x = F.softmax(self.linear_final(x), dim=1)

      return x

    def decide_action(self, x):

      # Create the prob distribution from output and return the action/logprob
      probs = self.forward(x)
      prob_dist = Categorical(probs)
      action = prob_dist.sample()
      action_item = action.item()
      logp = prob_dist.log_prob(action)

      return action_item, logp
</code></pre>
<p>And here’s the reinforce part:</p>
<pre><code class="lang-auto"># Function to calculate loss and update bot network
def reinforce_bot(b, debug):

    # Setup lists and vars to work off
    discounted_reward = 0
    l_returns = []
    l_policy_loss = []

    # Work through the bot's episode rewards backwards
    # The net effect of this will be such that we built rewards for only actions and their following rewards
    # (i.e. action for step n only gets rewards for steps &gt; n, never steps &lt; n)
    # Additionally we'll build in our reward discounting (where future steps contribute less to overall reward)
    for reward in b.l_episode_rewards[::-1]:
        discounted_reward = reward + gamma * discounted_reward
        l_returns.insert(0, discounted_reward) # but insert back at the beginning to get correct order
    
    # Now turn the rewards into a tensor for working with gradient
    t_returns = torch.tensor(l_returns)
    # But standardize the rewards to stabilize training
    t_returns = (t_returns - t_returns.mean()) / (t_returns.std())

    # Now build up our actual policy loss by multiplying it by our logprobs
    for logp, discounted_reward in zip(b.l_episode_log_probs, l_returns):
        l_policy_loss.append(-logp * discounted_reward)

    # Zero our gradient to get ready for backprop
    b.optimizer.zero_grad()

    # Technically our l_policy_loss is a list of tensors, so smoosh those together
    # Then sum to get the total loss
    policy_loss = torch.cat(l_policy_loss).sum()

    # Now run our optimizer
    policy_loss.backward()
    b.optimizer.step()

    # Text to print some helpful debugging
    if debug:
        print(f'{b.team} had awards array of {b.l_episode_rewards}')
        print(f'{b.team} had l_returns of {l_returns}')
        #print(f'{b.team} had l_policy_loss of {l_policy_loss}')
        print(f'{b.team} had a policy loss of  {policy_loss}')

    # And cleanup our epsiode tracking lists now since we don't need them
    del b.l_episode_rewards[:]
    del b.l_episode_log_probs[:]
</code></pre>
<p>Various debugging notes:</p>
<ul>
<li>
<p>The mapping of reward and actions within the environment is correct as best I can tell. If I comment out the network updating lines, the bots take all random actions and receive the appropriate rewards.</p>
</li>
<li>
<p>Swapping the sign of the rewards very strangely doesn’t help (the bots still move to the edges). It seems this reliably yields some reward, and it’s as if the bots are just trying to maximize the absolute value of reward received.</p>
</li>
</ul>
<p>Very grateful for any wisdom I can get. This one has me plain stuck.</p>