Categorical distribution returning breaking

Mkruger: <p>Hey, I am not too sure what is going wrong with my code, however, I am using a categorical distribution and getting a fairly weird error and am uncertain as to why. I did look around for a while online but nothing I found seemed to explain what exactly this issue was and how to get around it.</p>
<p>The error I am getting is as follows:</p>
<pre><code class="lang-auto">&gt; ValueError: Expected parameter logits (Tensor of shape (1024, 6)) of distribution Categorical(logits: torch.Size([1024, 6])) to satisfy the constraint IndependentConstraint(Real(), 1), but found invalid values:
&gt; tensor([[nan, nan, nan, nan, nan, nan],
&gt;         [nan, nan, nan, nan, nan, nan],
&gt;         [nan, nan, nan, nan, nan, nan],
&gt;         ...,
&gt;         [nan, nan, nan, nan, nan, nan],
&gt;         [nan, nan, nan, nan, nan, nan],
&gt;         [nan, nan, nan, nan, nan, nan]], device='cuda:0',
&gt;        grad_fn=&lt;SubBackward0&gt;)
</code></pre>
<p>The code I am using to generate this is just a simple feed forward network.</p>
<pre><code class="lang-auto">class actor(nn.Module):
    def __init__(self, input_size, n_actions):
        super(actor, self).__init__()

        self.base = layer_init(nn.Linear(input_size, 512))
        self.actor = layer_init(nn.Linear(512, n_actions), std=0.01)

    def forward(self, x):
        x = x.clone()
        x = self.base(x)
        x = torch.tanh(x)
        x = self.actor(x)
        return x
</code></pre>
<p>The output of this just gets put into a categorical dist: probs = Categorical(logits=logits)<br>
This seems to be where the error is occurring. The code does not break on the first run, it takes a couple hundred thousand steps before it breaks.</p>
<p>If anyone knows what the problem is and how to fix it, would appreciate it immensely.</p>

ptrblck: <p>Based on the error message it seems the <code>actor</code> is creating NaN outputs after a few iterations of training. Are you seeing an increase in the value range of its output during training, which could then overflow after a while?</p>

Mkruger: <p>Hi, yeah when I was attempting to debug it yesterday I noticed that the augmentation I had in my loss function started to give extremely large values. I did change this to clip the loss function if it goes out of a certain range and it seems like the error was fixed. So, I think you are correct and overflow is causing the issue.</p>
<p>Thank you for the assistance.</p>

zuoj: <p>Hi, I have encountered a similar issue when training a PPO agent for discrete actions. Can you give a hint on how to set the clipped loss range as you said here? Thank you!</p>

Mkruger: <p>Hey yeah not an issue.</p>
<p>The values were going way out and into the range of a couple hundreds which slowly exploded to thousands and so forth… Basically, as the probability of an action got really small it caused the logs to get out of control. So, I just clipped on 2 and 20 but in particular on the absolute value, however in this case given that the value should all be negative if it is a log of a probability that you are dealing with then you can probably just clip on the neg. However, clip on both just for robustness sake I suppose.</p>

zuoj: <p>Thank you very much for the explanation!</p>

OkabeSan: <p>I have the same issue and still need to figure out how to solve it. The main question is that when I run my model on my laptop, there is no error, and it works fine, but when I run this on an HPC server, I face the error. Most vector arrays are not big, but one of them gets a nan value. Do you know what the issue can be here?</p>

gaina54: <p>thanks for sharing the information.<a href="https://brevardcountypropertyappraiser.website/" rel="noopener nofollow ugc">.</a></p>