Backpropagation rule for REINFORCE weight updates using a Multinomial distribution

Bick95: <p>Hello,</p>
<p>I am implementing a REINFORCE algorithm (as superficially explained over <a href="https://pytorch.org/docs/stable/distributions.html" rel="noopener nofollow ugc">here</a>) using a <a href="https://pytorch.org/docs/stable/distributions.html#categorical" rel="noopener nofollow ugc">Categorical</a> distribution (or <a href="https://pytorch.org/docs/stable/generated/torch.multinomial.html#torch.multinomial" rel="noopener nofollow ugc">Multinomial</a> distribution as argued in the explanation of the Categorical distribution) for sampling actions.</p>
<p>I would like to know what the backpropagation rule looks like when plugging a Multinomial distribution into the REINFORCE update procedure.</p>
<p>More specifically, to say it in <a href="https://link.springer.com/content/pdf/10.1007/BF00992696.pdf" rel="noopener nofollow ugc">Williams</a>’ words, I would like to know what <strong>characteristic eligibility</strong> looks like when using a Multinomial distribution during the training of a REINFORCE algorithm. I saw in the aforementioned paper (by Williams) how the eligibility associated with a Bernoulli distribution looks like, but I don’t know how to generalize that further.<br>
From the explanation of <a href="https://pytorch.org/docs/stable/distributions.html" rel="noopener nofollow ugc">distributions</a> in PyTorch, I know that all this is implemented by default in PyTorch. However, I would still like to know how this theoretically works or if there are any papers describing the backpropagation procedure I am interested in.</p>
<p>Thank you very much in advance!</p>

Bick95: <p>More practically speaking, my problem is the following. Consider the minimal code snippet below:</p>
<pre><code class="lang-auto">import torch

print("Forward pass:")

# Probability vector for parameterizing Categorical distribution
probs = torch.tensor([[0.69, 0.1, 0.2, 0.01]], requires_grad=True)

# Define distribution
dist = torch.distributions.Categorical(probs=probs)

# Sampled category
category = torch.tensor([0])

# Get log prob for previously sampled category
log_prob = dist.log_prob(category)
log_prob.retain_grad()
print("log_prob:", log_prob)

# Get prob from log prob
prob = torch.exp(log_prob)
prob.retain_grad()
print("prob:", prob)

prob.backward()

print("Backward pass - Grads:")
print("probs-grad:", probs.grad)
print("log_prob-grad:", log_prob.grad)
print("prob-grad:", prob.grad)
</code></pre>
<p>Running the code results in the following output being generated:</p>
<pre><code class="lang-auto">Forward pass:
log_prob: tensor([-0.3711], grad_fn=&lt;SqueezeBackward1&gt;)
prob: tensor([0.6900], grad_fn=&lt;ExpBackward&gt;)
Backward pass - Grads:
probs-grad: tensor([[ 0.3100, -0.6900, -0.6900, -0.6900]])
log_prob-grad: tensor([0.6900])
prob-grad: tensor([1.])
</code></pre>
<p>Given the above, I am wondering how the gradient associated with the <code>probs</code> tensor is calculated. Trying to work out the maths, I would have expected the components of corresponding gradient to be <code>0</code> for all categories that have not been sampled. Instead, these gradient components are uniformly <code>-0.6900</code>.</p>
<p>Does anyone know by means of which rule the gradient vector associated with the probability vector <code>probs</code> is calculated?</p>

foshea: <p>This happens because the probabilities in the Categorical distribution are <em>relative</em> probabilities.  So, the probability of category zero being chosen is probs_0 / sum(probs_i).  Since the probs_i show up in the probability of drawing category 0, they will have a gradient.  An easier way to see this (for me) is to convert probs to have the same relative probabilities but be much larger:</p>
<pre><code class="lang-auto">probs = torch.tensor([[69., 10., 20., 1.]], requires_grad=True)
</code></pre>
<p>You should see that none of the printed values change, except for probs-grad, which are now 100x smaller.</p>