How does pytorch use c++?

Nick_Halden: <p>Hey there,</p>
<p>I started to use pytorch two days ago and I was trying to figure out how it all works.</p>
<p>I was trying to see where the implementation of ReLU is, but, after a certain point of backtracking in python code, I did not find the implementation.</p>
<p>Then I check on the github and found the implementation, but where exactly is that binary on linux? Couldn’t find it. And how/where is the python code calling it?</p>
<p>Thank you.</p>

ranman: <p>Slightly out of date but still quite useful blog post: <a href="http://blog.ezyang.com/2019/05/pytorch-internals/" rel="noopener nofollow ugc">http://blog.ezyang.com/2019/05/pytorch-internals/</a></p>
<p>I’m still learning myself but my understanding is that PyTorch uses a dispatcher mechanism to select an appropriate c++ implementation of various operators. This is covered in detail in another great blog post from Ed: <a href="http://blog.ezyang.com/2020/09/lets-talk-about-the-pytorch-dispatcher/" rel="noopener nofollow ugc">http://blog.ezyang.com/2020/09/lets-talk-about-the-pytorch-dispatcher/</a></p>
<p>Since PyTorch operators work on various hardware platforms, tensor types, and data types - the underlying operator implementation can be optimized in C++ by humans, codegen, and compilers.</p>
<p>For relu in particular it looks like the c++ version just calls threshold:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Activation.cpp#L368" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Activation.cpp#L368" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/aten/src/ATen/native/Activation.cpp#L368</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="358" style="counter-reset: li-counter 357 ;">
<li>    Scalar threshold,</li>
<li>    const Tensor&amp; output) {</li>
<li>  Tensor grad_input;</li>
<li>  auto iter = TensorIterator::binary_op(grad_input, grad_output, output);</li>
<li>  softplus_backward_stub(iter.device_type(), iter, beta, threshold);</li>
<li>  return iter.output();</li>
<li>}</li>
<li>
</li><li>// computes `result = self &lt;= threshold ? value : other`</li>
<li>// other is `self` in threshold() and `grad` in threshold_backward()</li>
<li class="selected">static Tensor threshold_out(</li>
<li>    optional&lt;Tensor&gt; opt_result,</li>
<li>    const Tensor&amp; self,</li>
<li>    Scalar threshold,</li>
<li>    Scalar value,</li>
<li>    const Tensor&amp; other) {</li>
<li>  Tensor result = opt_result.value_or(Tensor());</li>
<li>  auto iter = TensorIteratorConfig()</li>
<li>    .set_check_mem_overlap(false)  // threshold is idempotent, so overlap is okay</li>
<li>    .add_output(result)</li>
<li>    .add_input(self)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Here’s the CPU implementation for threshold:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cpu/Activation.cpp#L86" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cpu/Activation.cpp#L86" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/aten/src/ATen/native/cpu/Activation.cpp#L86</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="76" style="counter-reset: li-counter 75 ;">
<li>      },</li>
<li>      [=](Vec a, Vec b, Vec c) -&gt; Vec {</li>
<li>        auto mask = a &lt; zero_vec;</li>
<li>        auto max_deriv_vec = Vec::blendv(zero_vec, one_vec.neg(), mask);</li>
<li>        auto sign_vec = Vec::blendv(one_vec.neg(), one_vec, mask);</li>
<li>        return (max_deriv_vec + sign_vec * ((b - one_vec) / b)).neg() * c;</li>
<li>      });</li>
<li>  });</li>
<li>}</li>
<li>
</li><li class="selected">static void threshold_kernel(</li>
<li>    TensorIterator&amp; iter,</li>
<li>    Scalar threshold_scalar,</li>
<li>    Scalar value_scalar) {</li>
<li>  AT_DISPATCH_ALL_TYPES(iter.dtype(), "threshold_cpu", [&amp;] {</li>
<li>    using Vec = Vec256&lt;scalar_t&gt;;</li>
<li>    scalar_t threshold = threshold_scalar.to&lt;scalar_t&gt;();</li>
<li>    Vec threshold_v = Vec(threshold);</li>
<li>    scalar_t value = value_scalar.to&lt;scalar_t&gt;();</li>
<li>    Vec value_v = Vec(value);</li>
<li>    cpu_kernel_vec(</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Hope that helps!</p>