where to find the implementation of 'torch._C._VariableFunctions.rnn_tanh'?

username: claire
cooked: <p>Could someone show me how to find where torch._C._VariableFunctions.rnn_tanh is implemented?<br>
(Any general hint or advice is appreciated)</p>
<p>My motivation is that I implemented my own rnn (lstm) using nn.Linear() as components of an nn module (I know I can call pytorch’s lstm, but I wanted to learn), and found out that my lstm is very slow on gpu most likely because i don’t run operation for all 4 gates in parallel.  So I wanted to learn how pytorch’s rnn (lstm) library does it.</p>
<p>I started from RNNBase::forward() where it calls <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py#L204" rel="nofollow noopener">_rnn_impls[‘rnn_tanh’]</a>, which is torch._C._VariableFunctions.rnn_tanh.</p>
<p>For torch._C._VariableFunctions, I found where it is added as c++ extension <a href="https://github.com/pytorch/pytorch/blob/master/tools/autograd/templates/python_torch_functions.cpp#L518" rel="nofollow noopener">here</a>.</p>
<p>I tried to see if any of its <a href="https://github.com/pytorch/pytorch/blob/master/tools/autograd/templates/python_torch_functions.cpp#L453" rel="nofollow noopener">tp_methods</a> are about rnn, but could not figure out any further.</p>
<p>(Before this, by following RNNBase::init(), I saw that the 4 gates are placed consecutively in memory <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cudnn/RNN.cpp#L622" rel="nofollow noopener">here</a>)</p>
<p>I did find two related, very helpful threads, <a href="https://discuss.pytorch.org/t/where-does-torch-c-come-from/2015">link1</a> <a href="https://discuss.pytorch.org/t/where-to-find-torch-c-variablefunctions-module/41305">link2</a>.<br>
But still could not figure out.</p>

username: watteo
cooked: <p>Bump because I’m looking for a similar thing – can’t find the definition of ‘torch._C._VariableFunctions.gru’ anywhere in the source code.</p>
<p>All I know is that nn.GRU implements a different forward computation from that defined in one of the <a href="https://github.com/pytorch/pytorch/blob/bfeff1eb8f90aa1ff7e4f6bafe9945ad409e2d97/benchmarks/fastrnns/cells.py#L96" rel="nofollow noopener">gru_cell</a> functions, because with exactly the same parameters they give substantially different activities.</p>
<p>Found the same links and c++ files as OP, but no luck either. All I wanted was to access my GRU update and reset gate activations … <img src="https://discuss.pytorch.org/images/emoji/apple/disappointed.png?v=9" title=":disappointed:" class="emoji" alt=":disappointed:"></p>

username: wenleix
cooked: <p>Not 100% sure, but it looks to me <code>rnn_tanh</code> is defined by macro expansion in <a href="https://github.com/pytorch/pytorch/blob/8822c7e052e8f5b6d60098e7d2acf6cc191a4254/aten/src/ATen/native/RNN.cpp#L1436" rel="noopener nofollow ugc">Aten/native/RNN.cpp</a></p>
<p>Where the macro is defined <a href="https://github.com/pytorch/pytorch/blob/8822c7e052e8f5b6d60098e7d2acf6cc191a4254/aten/src/ATen/native/RNN.cpp#L1185-L1221" rel="noopener nofollow ugc">here</a> . It essentially dispatch to different implementations based on availability .  For the cuDNN implementation, it should be in Aten/native/cudnn/RNN.cpp</p>