Confusion about computing policy gradient with automatic differentiation ( material from Berkeley CS285)

username: Cloud_Huang
cooked: <p>I am taking Berkeley’s CS285 via self-study. On this particular lecture regarding Policy Gradient,  I am very confused about the inconsistency between the concept explanation and the demonstration of code snippet. I am new to RL<br>
and hope someone could clarify this for me.</p>
<h2><a name="context-1" class="anchor" href="#context-1"></a>Context</h2>
<p>1.The lecture defines policy gradient as follow:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/9/c97ddd97ba5bad136fee795cc91d8c3147acc18b.png" data-download-href="https://discuss.pytorch.org/uploads/default/c97ddd97ba5bad136fee795cc91d8c3147acc18b" title="Screen Shot 2021-02-11 at 2.33.53 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c97ddd97ba5bad136fee795cc91d8c3147acc18b_2_690x120.png" alt="Screen Shot 2021-02-11 at 2.33.53 PM" data-base62-sha1="sKtDw00eY2b3dsHkwsLBckaJkFl" width="690" height="120" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c97ddd97ba5bad136fee795cc91d8c3147acc18b_2_690x120.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c97ddd97ba5bad136fee795cc91d8c3147acc18b_2_1035x180.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c97ddd97ba5bad136fee795cc91d8c3147acc18b_2_1380x240.png 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-02-11 at 2.33.53 PM</span><span class="informations">1560×272 22.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>log(pi_theta(a | s)) denotes the log probability of action given state under policy parameterized by theta</p>
<p>gradient log(pi_theta(a |s)) denotes the gradient of parameter theta with respect to the predicted log probability of action</p>
<p>2.The lecture defines a pseudo-loss.By auto differentiate the pseudo-loss, we recovery the policy gradient.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b569797673d8d0ea2db62b597bfcb472cf540a55.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b569797673d8d0ea2db62b597bfcb472cf540a55" title="Screen Shot 2021-02-11 at 2.36.22 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b569797673d8d0ea2db62b597bfcb472cf540a55_2_690x386.jpeg" alt="Screen Shot 2021-02-11 at 2.36.22 PM" data-base62-sha1="pSQoOamuGhfgtMRyHzt94PpxWbr" width="690" height="386" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b569797673d8d0ea2db62b597bfcb472cf540a55_2_690x386.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b569797673d8d0ea2db62b597bfcb472cf540a55_2_1035x579.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b569797673d8d0ea2db62b597bfcb472cf540a55_2_1380x772.jpeg 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-02-11 at 2.36.22 PM</span><span class="informations">2142×1200 210 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here  Q_hat is short hand of sum of r(s_i_t, a_i_t) in the Policy gradient equation under 1)</p>
<ol start="3">
<li>The lecture then proceeds to gives a pseudo-code implementation of 2)</li>
</ol>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6e47811238cc88d921e2f9bc88b2767da12c97ae.png" data-download-href="https://discuss.pytorch.org/uploads/default/6e47811238cc88d921e2f9bc88b2767da12c97ae" title="Screen Shot 2021-02-11 at 2.37.44 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e47811238cc88d921e2f9bc88b2767da12c97ae_2_690x361.png" alt="Screen Shot 2021-02-11 at 2.37.44 PM" data-base62-sha1="fJzIyU5oz4xxUPFvO6Lpad0p8F8" width="690" height="361" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e47811238cc88d921e2f9bc88b2767da12c97ae_2_690x361.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e47811238cc88d921e2f9bc88b2767da12c97ae_2_1035x541.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e47811238cc88d921e2f9bc88b2767da12c97ae_2_1380x722.png 2x" data-dominant-color="F6F4F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-02-11 at 2.37.44 PM</span><span class="informations">2344×1228 268 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<h3><a name="my-confusion-2" class="anchor" href="#my-confusion-2"></a>My confusion</h3>
<ol>
<li></li>
</ol>
<p>From 1) above,</p>
<p>gradient log(pi_theta(a |s)) denotes the gradient of parameter theta with respect to the predicted log probability of action , not a loss value calculated from a label action and predicted action.</p>
<p>Why does the below in 2) implies that gradient log(pi_theta(a |s)) just morph into output of loss function instead of just predicted action probability as defined in 1) ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/8/989e3750d30770dcfbca8f49c1af1ee5ec426a99.png" data-download-href="https://discuss.pytorch.org/uploads/default/989e3750d30770dcfbca8f49c1af1ee5ec426a99" title="Screen Shot 2021-02-11 at 2.39.23 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/989e3750d30770dcfbca8f49c1af1ee5ec426a99_2_690x129.png" alt="Screen Shot 2021-02-11 at 2.39.23 PM" data-base62-sha1="lM7ydsPo8CmgQ2kQ6iGVnZJnF0Z" width="690" height="129" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/989e3750d30770dcfbca8f49c1af1ee5ec426a99_2_690x129.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/989e3750d30770dcfbca8f49c1af1ee5ec426a99_2_1035x193.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/989e3750d30770dcfbca8f49c1af1ee5ec426a99_2_1380x258.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-02-11 at 2.39.23 PM</span><span class="informations">2038×382 42.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<ol start="2">
<li></li>
</ol>
<p>In this pseudo-code implementation,</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c5db7ee14cc817126c4ca0b87572cba949b8ef57.png" data-download-href="https://discuss.pytorch.org/uploads/default/c5db7ee14cc817126c4ca0b87572cba949b8ef57" title="Screen Shot 2021-02-11 at 2.41.29 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5db7ee14cc817126c4ca0b87572cba949b8ef57_2_690x179.png" alt="Screen Shot 2021-02-11 at 2.41.29 PM" data-base62-sha1="sekkoeWhqGVuMlZ7V7QJQtdWyc7" width="690" height="179" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5db7ee14cc817126c4ca0b87572cba949b8ef57_2_690x179.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5db7ee14cc817126c4ca0b87572cba949b8ef57_2_1035x268.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5db7ee14cc817126c4ca0b87572cba949b8ef57_2_1380x358.png 2x" data-dominant-color="F6F2F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-02-11 at 2.41.29 PM</span><span class="informations">2292×596 93.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Particularly, this line below.</p>
<pre><code class="lang-auto">negative_likelihoods = tf.nn.softmax_cross_entrophy_with_logis(labels=actions, logits=logits)

</code></pre>
<p>Where does the actions even coming from ? If it comes from collected trajectory,  aren’t the actions result of <code>logits = policy.predictions(states)</code> to begin with ?<br>
Then won’t  <code>tf.nn.softmax_cross_entrophy_with_logis(labels=actions, logits=logits)</code> always return 0 ?</p>
<ol start="3">
<li></li>
</ol>
<p>Based on the definition of policy gradient in 1), shouldn’t the implementation of pseudo-loss be like below ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/6/862450835c24603435c0be3b312ea721c73f484b.png" data-download-href="https://discuss.pytorch.org/uploads/default/862450835c24603435c0be3b312ea721c73f484b" title="Screen Shot 2021-02-11 at 2.48.38 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/862450835c24603435c0be3b312ea721c73f484b_2_690x130.png" alt="Screen Shot 2021-02-11 at 2.48.38 PM" data-base62-sha1="j8FMDDZb6H27RmiqQAsRZe94d6P" width="690" height="130" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/862450835c24603435c0be3b312ea721c73f484b_2_690x130.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/862450835c24603435c0be3b312ea721c73f484b_2_1035x195.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/6/862450835c24603435c0be3b312ea721c73f484b.png 2x" data-dominant-color="F8F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-02-11 at 2.48.38 PM</span><span class="informations">1254×238 27.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">
# Given:
# actions - (N*T) x Da tensor of actions
# states - (N*T) x Ds tensor of states
# q_values – (N*T) x 1 tensor of estimated state-action values
# Build the graph:
logits = policy.predictions(states) # This should return (N*T) x Da tensor of action logits 
weighted_predicted_probability = tf.multiply(torch.softmax(logits), q_values)
loss = tf.reduce_mean(weighted_predicted_probability )
gradients = loss.gradients(loss, variables)

</code></pre>