In RNN module 'ReLU' activation function (which relu is used in development of RNN module?)

username: Digvijay_mishra
cooked: <p>Hello everyone, let me explain you a little background of my project and then I will tell you what problem I am facing so you get a clear picture of my problem.</p>
<p>so using pytroch.nn.RNN I trained neural network with 4 input neuron, 2 hidden layers , each have 8 neurons and 2 output neurons. so I trained my RNN model and I choose relu in 'nonlinearity ’ option and everything is fine there , my results are also ok. here is the picture of my model.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/5748b889ab2085ab5301f137328bbc5646355044.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5748b889ab2085ab5301f137328bbc5646355044" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/5748b889ab2085ab5301f137328bbc5646355044_2_577x500.jpeg" alt="1" data-base62-sha1="cs9liVnbuDutQ7oPnXLUie4n5kg" width="577" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/5748b889ab2085ab5301f137328bbc5646355044_2_577x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/5/7/5748b889ab2085ab5301f137328bbc5646355044.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/7/5748b889ab2085ab5301f137328bbc5646355044.jpeg 2x" data-dominant-color="EDF0EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">777×673 84.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
(its just half code because I am not allowed to post 2 pictures)</p>
<p>now as per my project requirements, I had to make same RNN structure from scratch using trained weights of the above-inbuilt model and I did that but my results are not matching when I apply relu activation function as (    relu(x) = max(0,x)   ). but when I use ‘Tanh’ activation in inbuilt model and use that trained weights, then my model which I made from scratch is giving similar results as inbuilt model. i know there are several types of ‘relu’ activation functions and I tried a lot to find out what relu is exactly used by inbuilt model but I couldn’t find any answer so please please anyone can tell me exactly what relu RNN module is using so that I can also use the same and get results. It will be great help as I am in last semester of my masters and I am stuck in relu.</p>

username: AlphaBetaGamma96
cooked: <p>It’s documented within the source file,<br>
<a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/rnn.html#RNN" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/_modules/torch/nn/modules/rnn.html#RNN</a></p>
<p><code>torch._VF.rnn_relu</code></p>

username: Digvijay_mishra
cooked: <p>I already seen it many times but i dont know why I am not able to run a code torch._VF.rnn_relu. even Installed many useful libraries as per suggestions by google colab but still i was getting error everytime i run this codes</p>

username: AlphaBetaGamma96
cooked: <p>See if you can follow the implementation in the example above,</p>
<p>if you look in the RNNBase class, it’s defined in the forward method,</p>
<pre><code class="lang-auto">        _impl = _rnn_impls[self.mode]
        if batch_sizes is None:
            result = _impl(input, hx, self._flat_weights, self.bias, self.num_layers,
                           self.dropout, self.training, self.bidirectional, self.batch_first)
        else:
            result = _impl(input, batch_sizes, hx, self._flat_weights, self.bias,
                           self.num_layers, self.dropout, self.training, self.bidirectional)
</code></pre>
<p>see if you can follow this and get that to work, you may need to wrap <code>torch._VF.rnn_relu</code> within a <code>torch.nn.Module</code> maybe? Also, given the VF package is prefaced with an underscore it’s probably not meant to be used within the python API.</p>
<p>Also, if you have an error can you share the error with a minimal reproducible example?</p>

username: Digvijay_mishra
cooked: <p>torch._vf i guess uses in C++. also I just want some simple things but I guess you didn’t understand my problem. there is no problem in coding part nor mathematics nor implementation. its just all about I want to know what relu developers had used while building RNN module. nothing else.</p>