System freezes on a GRU forward pass

username: tanvir_siddique
cooked: <p>Hello. I am having trouble over a small piece of code.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import math

class Segmentation(torch.nn.Module):
    def __init__(self, input, hidden, output, layers, bidirectional, kernal, dropout_prob):
        super().__init__()
        
        assert(kernal[0]%2==1 and kernal[1]%2==1), 'Convolution kernal sizes should be odd for keeping same dimensions.'
        padding = (math.floor(kernal[0]/2), math.floor(kernal[1]/2))

        # cnn input = [N * C * H * W]   H=time, W=Frequency

        self.conv_1 = nn.Conv2d(1, 1, kernel_size=kernal, padding=padding)
        self.mp_1 = nn.MaxPool2d(kernel_size=kernal, stride=1, padding=padding)
        self.conv_2 = nn.Conv2d(1, 1, kernel_size=kernal, padding=padding)
        self.mp_2 = nn.MaxPool2d(kernel_size=kernal, stride=1, padding=padding)

        self.dropout = nn.Dropout(p=dropout_prob)

        self.gru = nn.GRU(input, hidden, num_layers=layers, 
                    dropout=dropout_prob, bidirectional=bidirectional, batch_first=True)
        in_features = hidden*2 if bidirectional else hidden
        self.linear = nn.Linear(in_features=in_features, out_features=output)
        self.log_softmax = nn.LogSoftmax(dim=-1)

    def forward(self, input, input_lens):
        # inputs must be length wise sorted
        # input is already padded
        input = self.mp_1(self.conv_1(input))
        input = self.mp_2(self.conv_2(input))
        input = self.dropout(input)
        # now pack the sequences.
        # input -&gt; [batch, ch=1, height=time, width=frequency]
        input = input.squeeze(1)
        packed_input = nn.utils.rnn.pack_padded_sequence(input, input_lens, batch_first=True)
        packed_output, _ = self.gru(packed_input)
        output, _ = nn.utils.rnn.pad_packed_sequence(packed_output, batch_first=True)

        output = self.linear(output)
        
        return output
</code></pre>
<p>The problem occurs on line 39 <code>packed_output, _ = self.gru(packed_input)</code>. After the tesnor is passed to the gru layer, the system just freezes. I didn’t understand what was happening at first. Then I opened the debugger and system monitor and found that after coming to this line, the RAM consumption starts increasing. My RAM is 8GB and the programme alone consumes more than 7 GB, which is really weird, and therefore the system crushes. Every time I run this script, I had to force restart my machine. I tried googling the issue, NO LUCK!<br>
the input to the <code>self.gru(packed_input</code> is a packed sequence.<br>
The only reason I can think of this kind of behavior is an infinite for loop. But How can that happen on the forward layer of a built in library?<br>
Or can it?</p>

username: omarfoq
cooked: <p>Hello,</p>
<p>The fact that you are using 7GB is not necessarily weird. Can you try decreasing the batch size, and see if you still have the same problem? Also if you train this model without GPU it will take you forever.</p>

username: tanvir_siddique
cooked: <p>You are right. Memory seems to be not enough. I am actually working with speech audio data (MFCC features) so the input to the gru has a shape of [128 * 1500(aprx) * 20], 128:-batch size, 1500- sequence length in time, 20:- feature vector size. the corresponding audio files are anywhere from 2-10 seconds. This shape is actually huge for the GRU. I tried batch size of 1, and it still crashes. Any idea about a solution?</p>

username: omarfoq
cooked: <p>Hi,</p>
<p>In fact you are applying GRU to the output of the second convolution if I am not wrong. And I think this is maybe a heavy operation (depending on the number of kernels in you convolution).</p>
<p>If you can’t fit this load in memory, the only possibilities for you are to have better hardware (you can use colab, it may provide you with a GPU of 15 GB), or you can change the architecture, or reduce the input size, (if you change the parameters that you use to compute MFCC features, you can get denser features)</p>

username: tanvir_siddique
cooked: <p>I understand what you are saying. Thank you for your reply!</p>