Backpropagation through sampling a normal distribution

username: vvanirudh
cooked: <p>In my model, I predict the mean and variance of a normal distribution. To get the input at the next time-step, I need to sample from the normal distribution predicted previously. If I use <code>torch.normal</code>, when I call <code>backward</code> does it backprop through this operation?</p>

username: tom
cooked: <p>Hello <a class="mention" href="/u/vvanirudh">@vvanirudh</a>,</p>
<p>no (unless you are doing reinforce &amp; co). That is why the standard thing is to sample standard normal (<code>torch.randn</code>), multiply with the standard deviation and add the mean.<br>
This is sometimes labeled the <em>reparametrisation trick</em>, see <a href="https://arxiv.org/abs/1312.6114" rel="nofollow noopener">Kingma &amp; Welling, Section 2.4</a>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: zuoxingdong
cooked: <p>Hi <a class="mention" href="/u/tom">@tom</a></p>
<p>In case of using <code>torch.normal</code>, are there reasonable values to set <code>.reinforce(reward)</code> ? e.g. the gradients (backpropagated) in the next time step as the reward for each sample in current time step.</p>
<p>Concretely, say we have 50 initial samples, through NN, we get 50 outputs, then we compute mean and std, then generate 50 new samples, and feed this new samples through NN again, then we compute a loss.</p>

username: beginner
cooked: <p>Can anyone give a quick explanation as to why we can’t backpropagate through a sample?</p>

username: tom
cooked: <p>If you consider loss minimization, you can consider the (negative) gradient as a hint “go this way with your parameters a tiny bit to reduce the loss”. Now if you can apply the reparametrisation trick, you can say that tweaking the parameters a little bit also moves the output by small amount. This relationship enables you to backpropagate.<br>
However, if you have, say a bernoulli random variable (i.e. 1 with probability p, 0 with probablity (1-p)), you cannot “wiggle” with the outcomes - they’ll be 0 or 1.<br>
The basic trick there is to recognize that you are interested in the expectation rather than individual sample and then to figure out how to exchange expectation and derivative for backpropagation and make an expression with an expectation from the result - this is a key to the REINFORCE-type methods.</p>
<p>Best regards</p>
<p>Thomas</p>

username: haku
cooked: <p><a class="mention" href="/u/tom">@tom</a> I feel like it’s kind of off-topic,<br>
but could you elaborate on the notion that</p>
<aside class="quote no-group" data-username="tom" data-post="5" data-topic="3164">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>this is a key to the REINFORCE-type methods.</p>
</blockquote>
</aside>
<p>Is it because RL usually object to find action/policy/or-some-sort to maximize expected reward?</p>

username: tom
cooked: <p>Yes.<br>
So in gross simplification, we want to optimize <code>E(R(a))</code>, the expected (E) reward (<code>R</code>) that is a function of our action (a). This is similar to wishing to optimize the loss (except it is minimization) where we want to optimize <code>E(L(p))</code> the expected (E) loss (L) of a prediction (<code>p</code>) in supervised learning.<br>
Now in supervised learning, the loss and prediction is typically framed as something continuous, either a distance (in regression) or a score/probability (in classification) and you can differentiate <code>E(L(p))</code> by <code>p</code>. In the RL methods tackled with REINFORCE-type methods, you typically have discrete actions are discrete and thus are non-differentiable. But you do have the ability to move the distribution of the actions, thus changing the “density” implicit in the expectation <code>E</code>. That is then differentiable.</p>
<p>Similar things are done elsewhere, too, e.g. in Mathematical Finance if you do Monte-Carlo pricing and want to estimate sensitivities, you can sometimes do path-wise derivatives (i.e. differentiate the integrand in the expectation) and sometimes you have to use the “Likelihood ratio method” (which is differentiating the density to get a derivative of the expectation). From the top of my head Glasserman, Monte Carlo Methods in Financial Engineering, treats this in chapter 7.</p>
<p>Best regards</p>
<p>Thomas</p>