How to use RNN with variable audio signals?

username: shwe87
cooked: <p>Hi everyone,</p>
<p>I have created a few models using Deep Learning but mostly have worked with images. Unlike with audio, images can be cropped, resized, etc.</p>
<p>So, I was in the process of creating a Speech Recogniton model (for Spanish) as my thesis and I got to do these steps:</p>
<ol>
<li>
<p>This is my dataset, the dataset are of different lengths.</p>
</li>
<li>
<p>I created spectrograms for each one of them with 128 features, and of course each one has different numbers of frames due to their duration variation.</p>
</li>
<li>
<p>I created a custom dataset where each data is spectrogram + label:</p>
</li>
</ol>
<pre><code class="lang-auto">train_dataset = SpanishSpeechDataSet(csv_file='../input/spanish-single-speaker-speech-dataset/transcript.txt', root_dir='../input/spanish-single-speaker-speech-dataset/', data_type='train')
</code></pre>
<p>I did a test and show the spectrogram:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ecf17bdbfdf93e83b8f521b9e5c5e4b1ef57bff8.png" data-download-href="https://discuss.pytorch.org/uploads/default/ecf17bdbfdf93e83b8f521b9e5c5e4b1ef57bff8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ecf17bdbfdf93e83b8f521b9e5c5e4b1ef57bff8_2_690x240.png" alt="image" data-base62-sha1="xO63HCN738q2OyRJOZxRvJQAdNC" width="690" height="240" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ecf17bdbfdf93e83b8f521b9e5c5e4b1ef57bff8_2_690x240.png, https://discuss.pytorch.org/uploads/default/original/3X/e/c/ecf17bdbfdf93e83b8f521b9e5c5e4b1ef57bff8.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/c/ecf17bdbfdf93e83b8f521b9e5c5e4b1ef57bff8.png 2x" data-dominant-color="B79BAF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">802×280 59.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<ol start="4">
<li>I create a data processing function:</li>
</ol>
<pre><code class="lang-auto">def data_processing(audio_data):

    spectrograms = []
    labels = []
    input_lengths = []
    label_lengths = []
    
    for (spec,label) in audio_data:
        #The spectrogram is in (128, 407) and (128, 355) for example but later on for padding the function expects (407, 128) and (355, 128). So we need to transpose the matrices.
        spectrograms.append(torch.Tensor(spec.transpose()))
        t = TextProcessing()
        label = torch.Tensor(t.text2int(text=label))
        labels.append(label)
        input_lengths.append(spec.shape[0]//2)
        label_lengths.append(len(label))
    print(len(spectrograms))
    spec_pad = torch.nn.utils.rnn.pad_sequence(spectrograms, batch_first=True).unsqueeze(1).transpose(2,3)
    label_pad = torch.nn.utils.rnn.pad_sequence(labels, batch_first=True)
    return spec_pad, label_pad, input_lengths, label_lengths
</code></pre>
<ol start="5">
<li>DataLoader:</li>
</ol>
<pre><code class="lang-auto">data_loader = DataLoader(dataset=train_dataset, batch_size=32,shuffle=True,collate_fn=lambda x: data_processing(x))
</code></pre>
<ol start="6">
<li>Test the size of each:</li>
</ol>
<pre><code class="lang-auto">it = iter(data_loader)
first = next(it)
second = next(it)
print(first[0].shape)
print(second[0].shape)
</code></pre>
<pre><code class="lang-auto">32
32
torch.Size([32, 1, 128, 594])
torch.Size([32, 1, 128, 558])
</code></pre>
<p>For each batch, the max size of the batch is different.</p>
<ol start="7">
<li>Now, my question is if I want to pass these to a RNN, the input size would be 128 (number of features of the spectrogram) or would it be the frames (above case 594 or 558)? I am really confused!</li>
</ol>
<p>Thanks.<br>
BR,<br>
Shweta.</p>

username: ptrblck
cooked: <p>In the default setup <code>nn.RNN</code> (as well as e.g. <code>nn.LSTM</code>) expect an input in the shape <code>[seq_len, batch_size, features]</code>.<br>
I assume you are working with 128 frequencies and a variable sequence length in the last dimension.<br>
If that’s the case, you could remove dim1, which seems to be an unnecessary channel dimension via <code>input = input.squeeze(1)</code> and permute the inputs afterwards via:</p>
<pre><code class="lang-python">input = input.permute(1, 0, 2)
</code></pre>
<p>to get the desired shape of <code>[seq_len, batch_size, features]</code>.</p>
<p>You could also specify <code>batch_first=True</code> in the RNN, which would then expect the input in the shape <code>[batch_size, seq_len, features]</code>.</p>

username: shwe87
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thank you so much. Your reply was very helpful <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I tried it yesterday but now I am facing issues working with GRU and CTC Loss function, I created another post about this…</p>
<p>Have a nice day!</p>