What is the difference between torch.relu, torch.nn.ReLU and torch.nn.functional.relu?

username: erfan_mhi
cooked: <p>I was reading about different implementations of the ReLU activation function in Pytorch, and I discovered that there are three different ReLU functions in Pytorch. As I read <a href="https://discuss.pytorch.org/t/whats-the-difference-between-nn-relu-vs-f-relu/27599">this post</a>, I realized that the difference between <code>torch.nn.ReLU</code> and <code>torch.nn.functional.relu</code> is more about the coding style. However, there is a third function, <code>torch.relu</code>, which has the same functionality as <code>torch.nn.functional.relu</code>. Considering this, my main question here is that what is the difference between <code>torch.relu</code> and <code>torch.nn.functional.relu</code>, and can we use them interchangeably?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The short answer is none.<br>
The longer answer is that our binding code to cpp is set up so that most low level optimized functions (like relu) get bound to the <code>torch.foo</code> namespace.<br>
In this case, you can use <code>torch.relu</code> and <code>torch.nn.functional.relu</code> interchangeably yes.</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>You may want to know about some tips here:</p><aside class="quote" data-post="1" data-topic="90934">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/i_h_yoo/48/19022_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/whats-difference-of-nn-softmax-nn-softmax-nn-functional-softmax/90934">What's difference of nn.Softmax(), nn.softmax(), nn.functional.softmax()?</a> 
  </div>
  <blockquote>
    Hi. I guess there are three way to use activation function in a custom module: 

Use nn.Softmax() in the initializer in the custom model.
Use nn.softmax() in the forward function in the custom model.
Use nn.functional.softmax() in the forward function in the custom model.

What are the differences among them and what is the proper way to use an activation function in my custom model?
  </blockquote>
</aside>

<aside class="quote quote-modified" data-post="2" data-topic="90554">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/why-there-isnt-a-method-named-torch-softmax/90554/2">Why there isn't a method named torch.softmax</a> 
  </div>
  <blockquote>
    So the idea is to put more deep-learning-oriented functions in torch.nn.functional and keep general-purpose functions in under torch directly. softmax was deemed to fall into the former, sigmoid in the latter category. 
While there is torch.softmax, this is by accident (which is why it is not documented) rather than as a design (previous version of PyTorch didn’t have the fancy torch._C._nn module to put the C++-implementations of torch.nn.functional-functions). I would advise to only use the do…
  </blockquote>
</aside>

<p>Bests</p>