GRU vs Bidirectional GRU

shwe87: <p>Hello,</p>
<p>I created this model to adapt both GRU and bidrectional GRU, would it be the correct way? Because I don’t understand Bidirectional GRU completely…</p>
<p>Here are the snippets where I change according to if it is bidirectional or not:</p>
<pre><code class="lang-auto">class MySpeechRecognition(nn.Module):
    """
    The RNN model that will be used to perform Sentiment analysis.
    """

    def __init__(self, input_size, output_size, hidden_dim, n_layers, n_feats, drop_prob=0.5, bidir=False):
        super(MySpeechRecognition, self).__init__()
        #output_dim = will be the alphabet + '' and space = 28 chars
        self.input_size = input_size
        self.hidden_dim = hidden_dim
        self.n_layers = n_layers
        self.drop_prob = drop_prob
        self.output_dim = output_size
        self.bidir = bidir
        

        # GRU Layer --&gt; input (batch, channel*features, time)
        # Input size = number of features
        self.gru = nn.GRU(input_size, hidden_dim, n_layers, batch_first=True, dropout=drop_prob, bidirectional=bidir)
        # shape output (batch, channel*features, time * hidden_size)
        self.layer_norm = nn.LayerNorm(n_feats)
        # (batch, channel, features, time)
        #Fully Connected 
        **if self.bidir:**
**            self.fc1 = nn.Linear(self.hidden_dim*2,512)**
**        else:**
**            self.fc1 = nn.Linear(self.hidden_dim,512)**
        self.fc2 = nn.Linear(512, self.output_dim)
        self.dropout = nn.Dropout(0.2)

        
    def forward(self, x, hidden):
       # Forward function is same for both!
        
    
    def init_hidden(self, batch_size):
        ''' Initializes hidden state '''
        # Create two new tensors with sizes n_layers x batch_size x hidden_dim,
        # initialized to zero, for hidden state and cell state of LSTM
        weight = next(self.parameters()).data
        **if (self.bidir):**
**            self.n_layers = self.n_layers*2**
            
        if (train_on_gpu):
            hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_()).cuda()
        else:
            hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())
        
        return hidden
</code></pre>
<p>Additionally to that question, if I have a tensor of (32, 1, 128, 250) [It is melspectrogram], which I understand is → [Batch, Channel, Height, Width]. In this case, which would be the x0, x1, x2 (shown in picture)  in the GRU? The Height or the Width?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/e7496a33d835f085d800ee17c0ade05895a89551.png" data-download-href="https://discuss.pytorch.org/uploads/default/e7496a33d835f085d800ee17c0ade05895a89551" title="lstm-output-vs-hidden"><img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e7496a33d835f085d800ee17c0ade05895a89551.png" alt="lstm-output-vs-hidden" data-base62-sha1="x03AlD3QYEjqGqTmkDe4za9w3df" width="583" height="500" data-dominant-color="EDEFF2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">lstm-output-vs-hidden</span><span class="informations">640×548 20.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

vdw: <p>Here’s <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/models/text/classifier/rnn.py" rel="nofollow noopener">my code</a> for an RNN-based classifier. It can configured to use GRU or LSTM, both uni- or bidirectional. This might help for your first question.</p>
<p>Regarding your data, given your shape of  [Batch, Channel, Height, Width], where is your time dimension? Or do you have a series of these spectrograms?</p>

shwe87: <p>So, I am working with audio,first I convert those audios to melspectrograms using librosa.<br>
The ouput using librosa is --&gt; shape=(n_mels, t). I gues t is the time</p>
<p>Then the batch looks like:<br>
[batch, channel=1, n_mel=128, t=variable].</p>
<p>That’s why its confusing for me. I configured the input of GRU for 128 inputs, number of features because alll audios are time variant…Not sure if my implementation is correct…</p>
<p>I added a layer normalization layer before GRU to have the data normalized and to improve network’s performance. Do you think it’s good idea? I have been thinking about it and not sure if it’s better to put the layer normalization before or after GRU.</p>
<p>Thanks so much <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

shwe87: <p>I looked at your code and I see that the difference between using GRU/LSTM and bidirectiornal is the hidden dimension, which should be multiplied by the number of direction (1 or 2).</p>
<p>I am also doing the same in my code but not sure why it’s not working. Will have to check again I guess…</p>

shwe87: <p>I also see that you use Xavier Normalization to initialize weights. Is Xavier Normalization good for RNN networks?<br>
Is there any paper to know what type of weight initliazition is best for type of network? I don’t seem to find any…<br>
Thanks.</p>