How does torch.randn generate variates?

ph14: <p>How does the function <a href="https://pytorch.org/docs/stable/generated/torch.randn.html" rel="noopener nofollow ugc">torch.randn</a> generate variates? What specific method does it use, and where can I find a reference (if there is one)? Also, does the method change based on CPU/GPU usage?</p>

KFrank: <p>Hi Piers!</p>
<aside class="quote no-group" data-username="ph14" data-post="1" data-topic="128905" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f04885/48.png" class="avatar"> ph14:</div>
<blockquote>
<p>How does the function <a href="https://pytorch.org/docs/stable/generated/torch.randn.html" rel="noopener nofollow ugc">torch.randn</a> generate variates? What specific method does it use</p>
</blockquote>
</aside>
<p>As near as I can tell, <code>randn()</code> uses the <a href="https://en.wikipedia.org/wiki/Box%E2%80%93Muller_transform" rel="noopener nofollow ugc">Box-Muller</a> method, both on<br>
the cpu and gpu.</p>
<aside class="quote no-group">
<blockquote>
<p>where can I find a reference (if there is one)?</p>
</blockquote>
</aside>
<p>I am not aware that the algorithm used is documented anywhere<br>
but in the code.  It’s also quite opaque to me how any given pytorch<br>
python call gets dispatched down to the code that actually does the<br>
work.</p>
<p>But my best guess for the cpu is:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/cpu/DistributionTemplates.h#L84">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/cpu/DistributionTemplates.h#L84" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/cpu/DistributionTemplates.h#L84" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/main/aten/src/ATen/native/cpu/DistributionTemplates.h#L84</a></h4>



    <pre class="onebox"><code class="lang-h">
      <ol class="start lines" start="74" style="counter-reset: li-counter 73 ;">
          <li>      return random(generator);</li>
          <li>    });</li>
          <li>  });</li>
          <li>}</li>
          <li></li>
          <li>template&lt;typename RNG&gt;</li>
          <li>struct RandomKernel {</li>
          <li>  void operator()(TensorIteratorBase&amp; iter, c10::optional&lt;Generator&gt; gen) {</li>
          <li>    random_kernel(iter, check_generator&lt;RNG&gt;(gen));</li>
          <li>  }</li>
          <li class="selected">};</li>
          <li></li>
          <li>// ==================================================== Normal ========================================================</li>
          <li></li>
          <li>#ifdef CPU_CAPABILITY_AVX2</li>
          <li>static void normal_fill_16_AVX2(float *data,</li>
          <li>                         const __m256* two_pi,</li>
          <li>                         const __m256* one,</li>
          <li>                         const __m256* minus_two,</li>
          <li>                         const __m256* mean,</li>
          <li>                         const __m256* std_v) {</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>and for the gpu is:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/cuda/DistributionTemplates.h#L414">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/cuda/DistributionTemplates.h#L414" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/cuda/DistributionTemplates.h#L414" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/main/aten/src/ATen/native/cuda/DistributionTemplates.h#L414</a></h4>



    <pre class="onebox"><code class="lang-h">
      <ol class="start lines" start="404" style="counter-reset: li-counter 403 ;">
          <li>    distribution_nullary_kernel&lt;scalar_t, accscalar_t, curand4_engine_calls/2&gt;(iter,</li>
          <li>      gen,</li>
          <li>      [] __device__ (curandStatePhilox4_32_10_t* state) { return curand_uniform2_double(state); },</li>
          <li>      transform);</li>
          <li>  } else {</li>
          <li>    distribution_nullary_kernel&lt;scalar_t, accscalar_t, curand4_engine_calls&gt;(iter,</li>
          <li>      gen,</li>
          <li>      [] __device__ (curandStatePhilox4_32_10_t* state) { return curand_uniform4(state); },</li>
          <li>      transform);</li>
          <li>  }</li>
          <li class="selected">}</li>
          <li></li>
          <li>template&lt;typename scalar_t, typename accscalar_t, size_t curand4_engine_calls, typename RNG, typename transform_t&gt;</li>
          <li>void normal_and_transform(TensorIteratorBase&amp; iter, RNG gen, transform_t transform) {</li>
          <li>  if (std::is_same&lt;scalar_t, double&gt;::value) {</li>
          <li>    distribution_nullary_kernel&lt;scalar_t, accscalar_t, curand4_engine_calls/2&gt;(iter,</li>
          <li>      gen,</li>
          <li>      [] __device__ (curandStatePhilox4_32_10_t* state) { return curand_normal2_double(state); },</li>
          <li>      transform);</li>
          <li>  } else {</li>
          <li>    distribution_nullary_kernel&lt;scalar_t, accscalar_t, curand4_engine_calls&gt;(iter,</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>The gpu code calls down into the <code>curand_normal()</code> set of functions<br>
in NVIDIA’s <a href="https://docs.nvidia.com/cuda/curand/device-api-overview.html" rel="noopener nofollow ugc">cuRAND</a> library.</p>
<p>Quoting from the cuRAND documentation:</p>
<pre data-code-wrap="C"><code class="lang-C">__device__ float2
curand_normal2 (curandState_t *state)
...
</code></pre>
<blockquote>
<p>The above functions generate two normally or log normally distributed pseudorandom results with each call. Because the underlying implementation uses the Box-Muller transform, this is generally more efficient than generating a single result with each call.</p>
</blockquote>
<aside class="quote no-group">
<blockquote>
<p>Also, does the method change based on CPU/GPU usage?</p>
</blockquote>
</aside>
<p>As described above the cpu and gpu versions are implemented<br>
entirely independently.  It appears, however, that they both end up<br>
using the same underlying Box-Muller algorithm (near as I can tell …).</p>
<p>Best.</p>
<p>K. Frank</p>