What's the right way of implementing policy gradient?

Eddie_Li: <p>Hello everyone!<br>
I’m trying to perform this gradient update directly, without computing loss.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/1X/9d03fd1985406773f4ae4e494d4555f356f3080f.png" width="382" height="39"><br>
But I simply haven’t seen any ways I can achieve this. Looks like first I need some function to compute the gradient of policy, and then somehow feed it to the backward function.</p>

alexis-jacq: <p>I think you should look at this topic explaining the tensor.reinforce method:</p><aside class="quote quote-modified" data-post="1" data-topic="1294">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yunjey/48/1386_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-is-action-reinforce-r-doing-actually/1294">What is action.reinforce(r) doing actually?</a> <a class="badge-category__wrapper " href="/c/reinforcement-learning/6"><span data-category-id="6" data-drop-close="true" class="badge-category " title="A section to discuss RL implementations, research, problems"><span class="badge-category__name">reinforcement-learning</span></span></a>
  </div>
  <blockquote>
    Hi, 
I am studying RL with reinforcement/reinforce.py in pytorch/examples. I have some questions about it. 


What does <a href="https://github.com/pytorch/examples/blob/master/reinforcement_learning/reinforce.py#L68" rel="nofollow noopener">action.reinforce®</a> internally do ? 


Below is REINFORCE update rule where v_t is a return. We need to do gradient “ascent” as below but if we use <a href="https://github.com/pytorch/examples/blob/master/reinforcement_learning/reinforce.py#L71" rel="nofollow noopener">optimizer.step</a>, it is gradient “descent”. Is [action.reinforce®]((<a href="https://github.com/pytorch/examples/blob/master/reinforcement_learning/reinforce.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/reinforcement_learning/reinforce.py</a>) multiplying log probability by -r? Then, it makes sense. 
[image] 


In <a href="https://github.com/pytorch/examples/blob/master/reinforcement_learning/reinforce.py#L70" rel="nofollow noopener">autograd.ba…</a>
  </blockquote>
</aside>
<p>
If the action is the result of a sampling, calling <code>action.reinforce(r)</code> acts as a policy gradient.<br>
You can find a code example of implementation here:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/examples/blob/main/reinforcement_learning/reinforce.py">
  <header class="source">

      <a href="https://github.com/pytorch/examples/blob/main/reinforcement_learning/reinforce.py" target="_blank" rel="noopener">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/examples/blob/main/reinforcement_learning/reinforce.py" target="_blank" rel="noopener">pytorch/examples/blob/main/reinforcement_learning/reinforce.py</a></h4>


      <pre><code class="lang-py">import argparse
import gym
import numpy as np
from itertools import count
from collections import deque
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.distributions import Categorical


parser = argparse.ArgumentParser(description='PyTorch REINFORCE example')
parser.add_argument('--gamma', type=float, default=0.99, metavar='G',
                    help='discount factor (default: 0.99)')
parser.add_argument('--seed', type=int, default=543, metavar='N',
                    help='random seed (default: 543)')
parser.add_argument('--render', action='store_true',
                    help='render the environment')
parser.add_argument('--log-interval', type=int, default=10, metavar='N',
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/examples/blob/main/reinforcement_learning/reinforce.py" target="_blank" rel="noopener">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ruotianluo: <p>Or you can just have<br>
loss = samples_logprob * (reward - baseline)</p>
<p>(logprob is the output of the network, and reward and baseline are leaf variables)</p>
<p>If you do backward, the gradient backpropogated is equivalent to the policy gradient.</p>

11118: <p>hi everyone, I am new to pytorch, my question is a litter different from <a class="mention" href="/u/eddie_li">@Eddie_Li</a>,<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/7d4f429dc060210bd980e721a2bbdb275ec2fe04.png" data-download-href="https://discuss.pytorch.org/uploads/default/7d4f429dc060210bd980e721a2bbdb275ec2fe04" title="07"><img src="https://discuss.pytorch.org/uploads/default/original/2X/7/7d4f429dc060210bd980e721a2bbdb275ec2fe04.png" alt="07" width="690" height="119" data-dominant-color="F5F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">07</span><span class="informations">740×128 8.89 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
is there any way to change grad after backward?<br>
I mean loss = right side of the red line<br>
After backward, can I do pai(theta) * loss.grad, and then optimizer.step() ?<br>
Is that way work?</p>

alexis-jacq: <p>the right side of your red line looks rather the gradient of your loss?</p>
<p>Your loss would be something like</p>
<pre><code class="lang-auto">loss = - torch.sum(torch.log( policy(state) * (reward - baseline)))
</code></pre>
<p>then you compute the gradient of this loss with respect to all the parameters/variables that requires a gradient in your code by calling:</p>
<pre><code class="lang-auto">loss.backward()
</code></pre>
<p>and if you created, before the training loop, an optimizer associated to your policy like this:</p>
<pre><code class="lang-auto">optim_policy = optim.MyOptimizer( policy.parameters(), lr = wathever)
</code></pre>
<p>you can update your optimizer simply like this, after each backward call of your loss:</p>
<pre><code class="lang-auto">optim_policy.step()
</code></pre>
<p>the parameters of your policy (theta) will be updated with the gradient of your loss in order to minimize your loss.</p>

11118: <p>Thanks for your help, but my question is a little different<br>
In my formula A* is a set of all possible episodes<br>
when I update parameters , I want something like<br>
policy * grad of loss<br>
or in other words<br>
(p1<em>loss(p1) + …+pN</em>loss(pN)) , where p1+…+pN = 1<br>
your solution is<br>
(loss(p1)+…+loss(pN)) / N<br>
I want to change grad between loss.backward() and  optim_policy.step()<br>
I tried loss.grad() but it is None  <img src="//discuss.pytorch.org/images/emoji/apple/frowning.png?v=5" title=":frowning:" class="emoji" alt=":frowning:"></p>

alexis-jacq: <p>Then, I am not sure to understand your question… You want to use the policy as weights for the gradient update ? that way, you should re-write your loss:</p>
<pre><code class="lang-auto">weighted_loss = p1*loss(p1) + .. + pn*loss(pn)
weighted_loss.backward()
optimizer.step()
</code></pre>

11118: <p>Emm, here is the full formula<br>
policy is the weight of loss.grad, not the weight of loss itself.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0d9f7dad16012975eaac5907212d848d3f12160c.png" data-download-href="https://discuss.pytorch.org/uploads/default/0d9f7dad16012975eaac5907212d848d3f12160c" title="42"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0d9f7dad16012975eaac5907212d848d3f12160c_2_690x161.png" alt="42" width="690" height="161" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0d9f7dad16012975eaac5907212d848d3f12160c_2_690x161.png, https://discuss.pytorch.org/uploads/default/optimized/2X/0/0d9f7dad16012975eaac5907212d848d3f12160c_2_1035x241.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/0/0d9f7dad16012975eaac5907212d848d3f12160c.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">42</span><span class="informations">1348×316 27.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

alexis-jacq: <blockquote>
<p>policy is the weight of loss.grad, not the weight of loss itself.</p>
</blockquote>
<p>taken as a scalar quantity (that’s what I mean by weight) it’s just the same: <code>grad(w*x) = w*grad(x) </code><br>
you just have to make sure you are not using it as a variable of the tree (using pi.detach() should do it)</p>

11118: <p>Thanks a lot, it looks like detach() is what I want:blush:</p>

JMPF: <p>I am doing a similar example to this and I am having trouble defining the “baseline” variable. I tried</p>
<pre><code class="lang-auto">base_line=Variable(Variable(torch.ones(2), requires_grad=True))
base_line=base_line[0]
</code></pre>
<p>and get the error “RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn”. I only have one baseline value, could you suggest a way to define this varieble?</p>

alexis-jacq: <p>Generally, the baseline is an approximation of the expected reward, that does not depend on the policy parameters (so it does not affect the direction of the gradient).</p>
<p>This approximation can be the output of another network that takes the state as input and returns a value, and you minimize the distance between the observed rewards and the predicted values. Then, over a rollout, you predict the values of all states with this model, and remove the values from the rewards:</p>
<pre><code class="lang-python"># policy = Policy() # the network that computes the prob of your actions
# value = Value() # the network that predicts the rewards
loss = - torch.sum( torch.log(policy(states)) * (rewards - values(states) )
</code></pre>
<p>That’s the idea of the actor critic for policy gradient. You have a full example here: <a href="https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py">https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py</a></p>
<p>If your rollout is long enough, the sum of rewards approximates Q and the sum of values approximates V, so the term (rewards - values) approximate the advantage (Q-V) which is a non-biased estimator of the temporal difference (hence the name advantage-actor-critic).</p>

ymeng: <p>Just a quick question. The log probabilities are negative. If the rewards are positive, then the product with high rewards will make negative log prob values even lower (and loss even bigger)? Isn’t it opposite to what we need? Or the rewards should be negative? Did I miss anything?</p>

alexis-jacq: <p>the <code>log_prob*advantage</code> quantity is the objective to maximize. If a reward is negative, then having a large negative log_probs gives a large objective. At the opposite, if a reward is positive, then having small negative log_probs (close to 0) maximizes the objective.</p>
<p>With Pytorch, we need a loss to minimize, so we take <code>loss=-objective</code>.</p>

mansur: <aside class="quote no-group" data-username="ruotianluo" data-post="3" data-topic="4003">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ruotianluo/48/383_2.png" class="avatar"> ruotianluo:</div>
<blockquote>
<p>backward, the gradient backpropogated</p>
</blockquote>
</aside>
<p>Hello, I implemented the loss like you said, and it works.<br>
However, why doesn’t it have the opposite sign? Because the given gradient is actually for gradient ascent.</p>

RylanSchaeffer: <p><a class="mention" href="/u/alexis-jacq">@alexis-jacq</a> I think your line is wrong</p>
<p><code>loss = - torch.sum(torch.log( policy(state) * (reward - baseline)))</code></p>
<p>Shouldn’t the <code>reward - baseline</code> be outside the <code>log(policy(state))</code>?</p>

Baboon: <p>I am thinking the same thing… if <em>reward - baseline = 0</em> then this will give log(0)</p>

Muktan_Patel: <p>I had similar question, with 0 reward the loss will be equal to 0 and if the reward its 1 or greater the loss will be some positive value. shouldn’t it be opposite, getting reward 0 should have higher loss and getting reward 1 should have lower loss when inplementing REINFORCE in pytorch?</p>

vmoens: <p>This “loss” is actually not a proper loss.<br>
The negative loss (the objective) is actually<br>
<code>int p(x) r(x) dx</code><br>
where <code>p(x)</code> is the prob distribution and <code>r(x)</code> the reward function.</p>
<p>So if the expectation of <code>r(x)</code> gets higher, your objective gets higher (this is what you want).<br>
Now, we derive the gradient of that wrt some parameter theta (that i omitted for simplicity) of the distribution and we obtain that the gradient is:</p>
<p><code>grad obj(theta) = grad int p(x;theta) r(x) dx</code><br>
which we simplify into<br>
<code>grad obj(theta) = int grad (log p(x;theta)) p(x;theta) r(x;theta) dx</code><br>
(this comes from the fact that <code>grad f(x) = grad (log f(x)) f(x)</code> if <code>f(x) &gt; 0</code>).</p>
<p>If you take monte carlo samples, you get that<br>
<code>grad log p(x;theta) r</code><br>
if an estimator (i.e. converges with large number of samples) to <code>grad obj(theta)</code> when <code>x</code> is sampled according to <code>p(x;theta)</code>.</p>
<p>From there, people do the hack of saying that <code>log p(x;theta) r</code> is your objective BUT it’s not true (programatically it is how we will write it but you should not look at its value).</p>
<p>If you want something to log your loss, simply look at your reward <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> that’s what your objective (negative loss) truly is.</p>

Muktan_Patel: <p>I didn’t get it <a class="mention" href="/u/vmoens">@vmoens</a> , lets say we use vanilla loss, without subtraction of the b</p>
<p>for eg.<br>
current_loss = -( reward * log(p(action)))</p>
<p>if reward == 0:<br>
loss = 0<br>
if reward == 1:<br>
loss will be in range of [0.05, 1] if probablity is in range of [0.9, 0.1]<br>
if reward == 2:<br>
loss will be in range of [0.1, 2] if probablity is in range of [0.9, 0.1]<br>
if reward == 3:<br>
loss will be in range of [0.15, 3] if probablity is in range of [0.9, 0.1]</p>
<p>higher the reward higher the loss, lower the reward lower the loss.</p>