How does torch.normal work with a negative standard deviation?

jwillette: <p>I saw some code in a repo that was using <code>torch.normal()</code> with a possibly negative standard deviation. I thought the code was wrong and that <code>torch.normal()</code> would surely give an error with a negative standard deviation, but when I tried it, it just worked with the resulting sample being positive…</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/generated/torch.normal.html#torch.normal" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/generated/torch.normal.html#torch.normal" target="_blank" rel="noopener nofollow ugc">torch.normal — PyTorch 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I had trouble finding the exact implementation in the pytorch library so I was unable to see the actual code to see what it is doing, it seems like it just takes the absolute value, but I am not sure. How does this work?</p>

Nikronic: <p>Hi</p>
<p>As far as I can remember you cannot use negative value for <code>std</code> and you should not be able to use as it does not make any sense mathematically.</p>
<p>To just verify, I ran it and it gives a proper error:</p>
<pre><code class="lang-auto">torch.manual_seed(8)
torch.normal(0, -1, (10, ))

# output
# RuntimeError                              Traceback (most recent call last)
# &lt;ipython-input-6-f3bb1e6c1119&gt; in &lt;module&gt;()
    #   1 torch.manual_seed(8)
# ----&gt; 2 torch.normal(0, -1, (10, ))

# RuntimeError: normal_ expects std &gt; 0.0, but found std=-1
</code></pre>
<p>Bests</p>

jwillette: <p>hmm… I see. That is what I would expect to see. Your method does give an error, but I was using a slightly different argument.</p>
<p>I was doing it this way without error…</p>
<pre><code class="lang-auto">mu = torch.tensor(0.0)
sigma = torch.tensor(-3.0)
torch.normal(mu, sigma)

# ...output
&gt;&gt;&gt; torch.normal(mu, sigma)
tensor(-3.6977)
&gt;&gt;&gt; torch.normal(mu, sigma)
tensor(-3.0448)
&gt;&gt;&gt; torch.normal(mu, sigma)
tensor(-0.2327)
</code></pre>
<p>Any idea why the arguments set like this are valid? If I add a size argument to the above code it fails with an error. But without the size argument, it seems to work.</p>

KFrank: <p>Hi Jeff (and Doosti)!</p>
<aside class="quote no-group" data-username="jwillette" data-post="3" data-topic="107571" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jwillette/48/17366_2.png" class="avatar"> jwillette:</div>
<blockquote>
<p>If I add a size argument to the above code it fails with an error. But without the size argument, it seems to work.</p>
</blockquote>
</aside>
<p>For what it’s worth, I can reproduce your and Doosti’s results on both<br>
pytorch 1.6.0 and on 1.8.0.dev20201203.  That is, I get the “normal_<br>
expects std &gt; 0.0” error when I call <code>torch.normal()</code> with the third (size)<br>
argument, but not when I don’t.</p>
<p>I suppose you could say that it’s not a bug when software fails to issue<br>
an error when the user does something wrong.  However, pytorch is<br>
generally pretty liberal about issuing invalid-argument errors, and here<br>
we have the inconsistent reporting of the error based on exactly how<br>
the function is called.  So I would call it a bug.</p>
<p>Not that is matters, but as to what <code>torch.normal()</code> actually does:</p>
<p>It appears that <code>torch.normal()</code>, if effect, generates a <em>normal deviate</em><br>
(a sample from a normal distribution with mean zero and standard<br>
deviation one), multiplies it by the passed-in <code>sigma</code>, and then shifts<br>
it by the passed in <code>mu</code>.  This is <em>statistically</em> the same as just using<br>
<code>abs (sigma)</code>, but gives specific values that differ.  Here is a quick<br>
example:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.__version__
'1.6.0'
&gt;&gt;&gt; torch.random.manual_seed (2020)
&lt;torch._C.Generator object at 0x7f0d2dc95910&gt;
&gt;&gt;&gt; torch.normal (torch.tensor([0.0]), torch.tensor ([10.0]))
tensor([12.3723])
&gt;&gt;&gt; torch.random.manual_seed (2020)
&lt;torch._C.Generator object at 0x7f0d2dc95910&gt;
&gt;&gt;&gt; torch.normal (torch.tensor([0.0]), torch.tensor ([-10.0]))
tensor([-12.3723])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

ptrblck: <p>Thanks for the detailed explanation <a class="mention" href="/u/kfrank">@KFrank</a> and checking the code <a class="mention" href="/u/nikronic">@Nikronic</a>!<br>
<a class="mention" href="/u/jwillette">@jwillette</a> could you create a GitHub issue so that we could make sure the methods raise the same error?</p>

jwillette: <p>I already started one here <a href="https://github.com/pytorch/pytorch/issues/49998#issuecomment-754366391" class="inline-onebox" rel="noopener nofollow ugc">torch.normal only issues error for negative sigma when shape argument is given · Issue #49998 · pytorch/pytorch · GitHub</a></p>