Activation function and distribution for action space between 0 and 1?

username: joel
cooked: <p>Hi,</p>
<p>I am building my first REINFORCE (policy gradient) model with a continuous action space between 0 and 1. Right now, I use the following code for this:</p>
<pre><code class="lang-auto">        self.loc_layer = nn.Sequential(
            nn.Linear(size, n_params),
            nn.Sigmoid()
        )
        self.scale_layer = nn.Sequential(
            nn.Linear(size, n_params),
            nn.Sigmoid()
        )

    	# ...

        loc = self.loc_layer(x)
        if self.training:
            scale = self.scale_layer(x)
            dist = Normal(loc=loc, scale=scale)
            params = dist.sample()
            log_prob = dist.log_prob(params).sum(dim=1)
            params = params.clamp(min=0, max=1)
        else:
            params = loc
            log_prob = None
</code></pre>
<p>However, this does not seem right to me. For example, I do not like that I have to clamp to ensure that the actions (<code>params</code>) are between 0 and 1. Therefore, I consider using a Beta or Logit-Normal distribution instead. In other examples, I have seen that Softplus is often used as the activation function for the scale argument. But this does not seem right here because I want to limit the action space.</p>
<p>Can anyone with some experience in this area give me a recommendation to which activation function and distribution to use in this case?</p>

username: iffiX
cooked: <p>Three solutions:</p>
<ol>
<li>
<p>use a normal distribution, use tanh as <code>mu</code> activation (to keep the center in range, prevent shifting too much) and then clamp, but you should do clamping only on the action sent to the environment, and not actions stored in buffers. In this way, you are not changing the pdf of your action, but changing the reward distribution.</p>
</li>
<li>
<p>use a normal distribution, use tanh to clamp your output right after sampling, but in order to calculate the log probability of your sample, you need to do this step, which is used in the <a href="https://arxiv.org/pdf/1801.01290.pdf" rel="noopener nofollow ugc">Soft Actor Critic algorithm</a>.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/d/bd5629b56fefadc5c22e2b775f6f69e9a7acda0d.png" alt="image" data-base62-sha1="r0WQoG5NPGkCIm1d1dJuCuJjzLD" width="332" height="51"><br>
It should be relatively easy to deduce this equation</p>
</li>
<li>
<p>use a different distribution, but normally, the normal distribution performs the best in all algorithms, I have not tried Beta or Logit-Normal distribution and don’t know their performance. This method is theoretically possible, but result is not garanteed.</p>
</li>
</ol>

username: joel
cooked: <p>Thanks a lot for your answer! Do you think it’s better to use a transformed tanh (i.e. <code>(tanh + 1)/2</code>) instead of sigmoid for actions in the range of 0 to 1? If so, because of the stronger gradient or is there another reason?</p>
<p>I have switched to softplus for the activation function of the scale parameter (since this seems be commonly used) and it seems to work. Is this what you would have recommended as well?</p>

username: iffiX
cooked: <p>tanh and sigmoid does not quite affect the performance.</p>
<p>Anything that will output a positive value can be used as your sigma (scale) activation, softplus is commonly used because its smooth, close to linear (y=x) when x &gt;&gt; 0 and works on the negative region.</p>

username: dbsx
cooked: <p>Thanks for your advice, and I have a math question. I know how to transfer a sampled value of a specific normal distribution to a specifc range. 1.using tanh to  bound the sampled_value in range [-1,1], 2. use the formula  <a href="https://stackoverflow.com/questions/49911206/how-to-restrict-output-of-a-neural-net-to-a-specific-range" rel="noopener nofollow ugc">here</a> to transfer it to specific range. But what about bounding the log_prob value of the sampled value to specific range?      The formula you referred of SAC, is only suitable when the target range is in [-1,1].</p>

username: X3N4
cooked: <p><strong>Short answer</strong>:</p>
<ul>
<li>A transformed Beta distribution policy is an option. See for example in <a href="https://arxiv.org/pdf/1805.09613.pdf" rel="noopener nofollow ugc">this paper</a>. However, I haven’t personally gotten Beta distributions to work. The training is always very unstable.</li>
<li>Have the network output actions in [-1, 1] and transform to the desired range as part of the environment. See <a href="https://stable-baselines.readthedocs.io/en/master/guide/rl_tips.html" rel="noopener nofollow ugc">here</a>.</li>
</ul>
<p><strong>Long answer regarding the transformation of sampled actions</strong>:<br>
In general, doing a location-scale transformation of your clamped actions in [-1, 1] should not affect the SAC entropy correction formula. To see why, consider your transformation of the type l + c*a, where a = tanh(u) just like in the SAC paper (l being a location term, c being a scaling factor).<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/d/bd5629b56fefadc5c22e2b775f6f69e9a7acda0d.png" alt="" role="presentation" width="332" height="51"><br>
The term in the log behind the sum comes from the derivate of your tanh transformation. If you instead take the derivative of l + c*tanh(u), the term l will immediately fall away and you will have<br>
<img src="https://user-images.githubusercontent.com/36668911/135235521-986a4d26-5f9b-4be2-9f01-f466c03841fe.png" alt="image" width="280" height="58"><br>
as the derivative of the tanh-function applied element-wise to your actions.<br>
You then want to take the log of the following density<br>
<img src="https://user-images.githubusercontent.com/36668911/135235890-b473f643-65b1-4778-81b5-761038fbc836.png" alt="image" width="238" height="56"><br>
The scaling term c will persist, but due to the log you can move it out of the sum.<br>
<img src="https://user-images.githubusercontent.com/36668911/135236026-10a0d9d1-d3cf-42b8-bf30-edb036b4abd9.png" alt="image" width="485" height="274"><br>
Here D is the dimensionality of your action space.<br>
Now when you want to perform backprop, the term D log c only depends on the constants D and c. These are irrelevant for the derivative.<br>
<strong>What does this mean for you?</strong><br>
You can use 6+4*tanh(u) to transform your actions between e.g. [2, 10] <em>without having to change the entropy correction formula</em>.</p>
<p>Source: This has been part of my master’s thesis.</p>