GRU calculations for gates and output in testing mode

username: dinesh_mareedu
cooked: <p>Hi,</p>
<p>I tried in many ways but I could not get the solution. I am implementing GRU layer in a model in c.<br>
I have extracted weights and biases. Can you give me the equations to compute the gates and outputs.</p>
<p>I am using hidden size = 32<br>
input_size =32<br>
num_layers = 1<br>
batch_first  = true</p>
<p>I followed the equations at <a href="https://pytorch.org/docs/stable/nn.html" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html</a> at GRU section.<br>
My expected output is 151x32<br>
Output is matched with just 1x32 rest 150x32 is not matching.</p>
<p>I am not able to know, what is called the time step in implementation time.<br>
If possible suggest me any document about implementing GRU in testing mode.</p>

username: tom
cooked: <aside class="quote no-group" data-username="dinesh_mareedu" data-post="1" data-topic="49606">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dinesh_mareedu/48/12060_2.png" class="avatar"> dinesh_mareedu:</div>
<blockquote>
<p>I am not able to know, what is called the time step in implementation time.</p>
</blockquote>
</aside>
<p>GRU (forward) behaves the the same (and as in the documentation you linked) between training and testing. The time step is the first dimension in the 3d input and output <code>seq_len x batch x features</code>. Maybe considering the (unit, use <code>unsqueeze</code>) batch size explicitly can help you (just in case you accidentally fed a single-time-step, 151 batch size into the GRU when you wanted 151 time step, batch size 1.</p>
<p>Best regards</p>
<p>Thomas</p>

username: dinesh_mareedu
cooked: <pre><code class="lang-auto">Thanks for the quick reply, can you fix my issue one more step. I am giving my code which I stuck for few days, please correct me.
Input dimension is 1x151x32
k = crnn_model.rnn.weight_ih_l0
p  = crnn_model.rnn.weight_hh_l0
m  = crnn_model.rnn.bias_hh_l0
l  = crnn_model.rnn.bias_ih_l0

As mentioned, gru layer has 1 layer, I am feeding zeros as state. And updating with the new state after 
calculating output. 

This is the code giving me wrong results as i mentioned. Please solve my issue to go ahead, I am stuck here for days

ht = np.zeros(32)
for j in range(151):
    for i in range(32):
        sum_r = np.sum(k[i]*input[0][j]) + l[i+0]  +  m[i+0] + (np.sum(p[i]))*ht[i]
        r  =   sum_r 
        r  = 1 / (1 + math.exp(-r))
        
        sum_z = np.sum(k[i+32]*input[0][j]) + l[i+32] +  m[i+32] + np.sum(p[i+32])*ht[i]
        z  =   sum_z
        z  = 1 / (1 + math.exp(-z))
        
        n = np.sum(k[i+64]*input[0][j]) + l[i+64] +(m[i+64]+np.sum(p[i+64])*ht[i])*r
        n = math.tanh(n)
        out   = (1-z)*n +  z*(ht[i])
        ht_past[i] =  out        
    ht = ht_past 	

</code></pre>

username: dinesh_mareedu
cooked: <p>My Input to the GRU is 1x151x32.<br>
Hope I get to go forward.</p>

username: tom
cooked: <ul>
<li>I’m not sure why this is a libtorch question?</li>
<li>I would suggest rewriting it in PyTorch first <code>using torch.nn.functional</code> (with <code>linear</code>, <code>sigmoid</code>, and <code>tanh</code>). If you have to, you can refine that afterwards.</li>
<li>np.sum seems wrong, as you want to do reduction over one axis only.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

username: dinesh_mareedu
cooked: <p>May be I expressed the question in a wrong way. It is not libtorch. just checking the outputs by loading the model and extraction weights in python itself.</p>
<ul>
<li>
<p>I converted them into numpy, then only I am using np.add. I skipped those steps in post.</p>
</li>
<li>
<p>Is that procedure correct that I followed in the view of time steps. Taking 32 features at a time.</p>
</li>
</ul>

username: dinesh_mareedu
cooked: <p>I used this to take care of axis</p>
<ul>
<li>out = out.permute(0, 2, 1, 3)</li>
<li>out = out.view(out.shape[0], out.shape[1], out.shape[2])</li>
</ul>

username: dinesh_mareedu
cooked: <p>I am using batch_first is true<br>
so My input dimension is</p>
<ul>
<li>batch x seq_len x features</li>
</ul>

username: tom
cooked: <p>Well, yeah, the summation for the matmuls are still wrong.</p>

username: dinesh_mareedu
cooked: <p>Actually, I wanted to know which operation is going wrong.can you point me, which matmul calculation is wrong.</p>
<ol>
<li>Should not use numpy.sum()? Is this the one…</li>
</ol>
<p>Sorry, still i am not getting which line is wrong.<img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"></p>

username: tom
cooked: <p>Do print your the result of your sum and then look what’s wrong.</p>

username: dinesh_mareedu
cooked: <p>Ok, those are multiplications, right?(except ht) Not point wise multiplications.</p>

username: dinesh_mareedu
cooked: <p>Hi,</p>
<pre><code>Thank you for your answers. Problem is resolved. But precision is missing by  0.003. It is not desirable, right. Can you give me suggestions on this.</code></pre>

username: tom
cooked: <p>Depending on the size of the arguments, this can be an effect of machine precision (~relative 1e-7 for float, ~1e-14-15 for double).</p>

username: dinesh_mareedu
cooked: <p>Hi,</p>
<pre><code>   Yes, the precision is (~relative 1e-7 ) for the first time step. For the second time step precision is going at (~relative 1e-3) .
</code></pre>
<p>Rest of the time steps also precision loss is getting accumulated as the current time step is depend on the previous time step.</p>
<p>Is there any way that I can solve the issue?</p>
<p>Thanks,<br>
Dinesh</p>

username: tom
cooked: <p>Other than using double?</p>

username: dinesh_mareedu
cooked: <p>I tried in C++ as long, long double and float also. May be if 1st time step output is perfectly match without any precision loss, it works. But i am not able to know how to do that.  Because every time steps ’ output is depend on previous time steps. Is it ?</p>

username: dinesh_mareedu
cooked: <p>shall I send you the code in c which i have implemented?</p>

username: tom
cooked: <p>To be perfectly honest, I think we’ve strayed quite far from using libtorch already and for me I’d not take this any further.</p>
<p>Best regards and all the best for your project</p>
<p>Thomas</p>

username: dinesh_mareedu
cooked: <p>Problem resolved. Thank you very much for your answers and they helped me to get to know about time steps.</p>