Facing problem to identify which RELU activation function is used in pytorch Recurrent neural network module

username: Digvijay_mishra
cooked: <p>Hello everyone, let me explain you a little background of my project and then I will tell you what problem I am facing so you get a clear picture of my problem.</p>
<p>so using pytroch.nn.RNN I trained neural network with 4 input neuron, 2 hidden layers , each have 8 neurons and 2 output neurons. so I trained my RNN model and I choose relu in 'nonlinearity ’ option and everything is fine there , my results are also ok. here is the picture of my model.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/5748b889ab2085ab5301f137328bbc5646355044.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5748b889ab2085ab5301f137328bbc5646355044" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/5748b889ab2085ab5301f137328bbc5646355044_2_577x500.jpeg" alt="1" data-base62-sha1="cs9liVnbuDutQ7oPnXLUie4n5kg" width="577" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/5748b889ab2085ab5301f137328bbc5646355044_2_577x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/5/7/5748b889ab2085ab5301f137328bbc5646355044.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/7/5748b889ab2085ab5301f137328bbc5646355044.jpeg 2x" data-dominant-color="EDF0EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">777×673 84.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>(its just half code because I am not allowed to post 2 pictures)</p>
<p>now as per my project requirements, I had to make same RNN structure from scratch using trained weights of the above-inbuilt model and I did that but my results are not matching when I apply relu activation function as ( relu(x) = max(0,x) ). but when I use ‘Tanh’ activation in inbuilt model and use that trained weights, then my model which I made from scratch is giving similar results as inbuilt model. i know there are several types of ‘relu’ activation functions and I tried a lot to find out what relu is exactly used by inbuilt model but I couldn’t find any answer so please please anyone can tell me exactly what relu RNN module is using so that I can also use the same and get results. It will be great help as I am in last semester of my masters and I am stuck in relu.</p>
<p>i also gone through it several times and I am not able to run torch._VF.rnn_relu. which is given in source code and I tried my best to make it run but libraries which are mentioned in source code , that I am not able to import in my google colab notebook even after installing all of them in colab.<br>
It’s documented within the source file,<br>
<a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/rnn.html#RNN" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/_modules/torch/nn/modules/rnn.html#RNN</a></p>
<p><code>torch._VF.rnn_relu</code></p>
<p>can someone please help me in this… thank you in advance</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/digvijay_mishra">@Digvijay_mishra</a></p>
<pre><code class="lang-auto">Parameters
...
        num_layers – Number of recurrent layers. E.g., setting num_layers=2 would mean stacking two RNNs together to form a stacked RNN, with the second RNN taking in outputs of the first RNN and computing the final results. Default: 1

        nonlinearity – The non-linearity to use. Can be either 'tanh' or 'relu'. Default: 'tanh'
...
</code></pre>
<p>As per the documentaton, the default non linearity in the RNN module is tanh. This might be the reason why your model performs well with inbuilt model with tanh</p>
<p>Both tanh and relu have their own advantages and disadvantages, so at a general level you will have to understand your data and model and apply accordingly</p>

username: Digvijay_mishra
cooked: <p>i think you didn’t understand my problem. activation functions mathematics we all know right. it doesn’t matter weather I use relu or tanh as activation function when I m using trained weights by Pytorch RNN module and that is giving fine results then why m self coded RNN is not giving similar results as pytroch module. and that is only when I use relu as a activation function , when I use tanh in pytorch module and match the ans with my self coded RNN both are matching that means there is no problem in my codes. i just want to know what type of relu exactly they used while building RNN.</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/digvijay_mishra">@Digvijay_mishra</a> Got it</p>
<ul>
<li>
<p>Before you compare the results, you have to ensure that both are using the same random seed</p>
</li>
<li>
<p>Also, when you see that your results are not matching with the inbuilt function’s output, are you comparing the output of self.rnn or are you checking the output after the self.fc ( fully connected layers )<br>
The reason is ask this is because for RNN we have just two activations inbuilt ( TANH and RELU ) but for a general layer there are lots of activation functions like LEAKYRELU etc</p>
</li>
</ul>
<p>Can you share the forward code of the above module, so that we can help further</p>

username: Digvijay_mishra
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b201784ac9a8b6d84b3e52a9cac2ccec273d49f.png" data-download-href="https://discuss.pytorch.org/uploads/default/4b201784ac9a8b6d84b3e52a9cac2ccec273d49f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b201784ac9a8b6d84b3e52a9cac2ccec273d49f.png" alt="image" data-base62-sha1="aIAz8KVq8O42p9sX9teDAfFGk5V" width="535" height="500" data-dominant-color="ECEFEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">641×598 15.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>here is the 2nd part of code</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/digvijay_mishra">@Digvijay_mishra</a> The code looks good</p>
<p>Just compare the results of out_h2 between your implementation and inbuilt model. And use a common seed.</p>
<pre><code class="lang-auto">torch.manual_seed(42)
</code></pre>
<p>RNN by default uses tanh and if you mention ‘relu’ then it uses the default Relu</p>

username: my3bikaht
cooked: <p>_vf is a mapping to underlying cpp ATen library<br>
I think in the end you’ll need to trace every operation inside RNN to find the origin of the problem</p>
<p>Meanwhile, you can replace your max(0, x) with torch.nn.functional.relu() but I doubt it will help</p>

username: Digvijay_mishra
cooked: <p>is there any way to run such library into python?</p>

username: Digvijay_mishra
cooked: <p>can i share my codes which i made from scratch because your answers are somewhere out of the point until unless you see both the things.may be after that I can get proper solution.</p>

username: my3bikaht
cooked: <p>I looked where this rabbit hole leads and it is way deeper, up to libcudnn.so</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/master/torch/csrc/cuda/shared/cudnn.cpp">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/cuda/shared/cudnn.cpp" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/cuda/shared/cudnn.cpp" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/torch/csrc/cuda/shared/cudnn.cpp</a></h4>


      <pre><code class="lang-cpp">// The clang-tidy job seems to complain that it can't find cudnn.h without this.
// This file should only be compiled if this condition holds, so it should be safe.
#if defined(USE_CUDNN) || defined(USE_ROCM)
#include &lt;torch/csrc/utils/pybind.h&gt;

#include &lt;array&gt;
#include &lt;tuple&gt;

namespace {
using version_tuple = std::tuple&lt;size_t, size_t, size_t&gt;;
}

#ifdef USE_CUDNN
#include &lt;cudnn.h&gt;

namespace {

version_tuple getCompileVersion() {
  return version_tuple(CUDNN_MAJOR, CUDNN_MINOR, CUDNN_PATCHLEVEL);
}
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/cuda/shared/cudnn.cpp" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>maybe someone from pytorch team will help here, but I think that It is much easier to trace where your calculations divert. For example, when you run single block with relu, have you checked that all positive values of result tensors match? Since, you know, ReLU only zeroing up negative values, so your positive values should match.</p>