Loading varying length data for an Sequence to Sequence Model

knoriy: <p>I’m creating an incremental TTS model, the model is based on <a href="https://arxiv.org/pdf/2011.02126.pdf" rel="noopener nofollow ugc"><em>Incremental Machine Speech Chain</em><br>
<em>Towards Enabling Listening while Speaking in Real-time</em></a></p>
<p>In this paper, they describe passing a small section of audio (windowing) for training as shown in the diagram below:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/5/0574cd36e06fb823abe546484e59a475ec06dc45.png" data-download-href="https://discuss.pytorch.org/uploads/default/0574cd36e06fb823abe546484e59a475ec06dc45" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/0574cd36e06fb823abe546484e59a475ec06dc45_2_646x500.png" alt="image" data-base62-sha1="MgDhOlnECwFC47AJde00ghGkhn" width="646" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/0574cd36e06fb823abe546484e59a475ec06dc45_2_646x500.png, https://discuss.pytorch.org/uploads/default/original/3X/0/5/0574cd36e06fb823abe546484e59a475ec06dc45.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/5/0574cd36e06fb823abe546484e59a475ec06dc45.png 2x" data-dominant-color="F4F5F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">718×555 48.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What is the best way to load my data for training? (The dataset consists of Pulse-Code Modulation (PCM) and labels )</p>
<h2><a name="what-ive-tried-so-far-1" class="anchor" href="#what-ive-tried-so-far-1"></a>What I’ve tried so far?</h2>
<p>For each tensor in a batch, I create a new tenor using the method shown <a href="https://stackoverflow.com/questions/60157188/how-can-i-resize-a-pytorch-tensor-with-a-sliding-window" rel="noopener nofollow ugc">Here</a>. and this new tensor is used for training.</p>
<pre><code class="lang-auto">for i, batch in enumerate(train_loader):
    x, y = ttsModel.parse_batch(batch) # send data to GPU
    for tensor in x:
        windowed_datas = window_split(tensor)
        for windowed_data in windowed_datas:
            tts_output, hd0, he0 = ttsModel(windowed_data, hd0, he0)
</code></pre>
<p>I already see an issue, the model is no longer training on batches but single tensors.</p>
<p>I’ve tried this approach in the <code>torch.utils.data.Dataloader</code> using the <code>collate_fn</code> <a href="https://github.com/NVIDIA/tacotron2/blob/master/data_utils.py" rel="noopener nofollow ugc">Nvidia: Tacotron 2</a>, but I’m not sure if this is correct for the model.</p>
<pre><code class="lang-auto">class TextAudioCollate():
    """ Zero-pads model inputs and targets based on number of frames per setep """
    def __init__(self, n_frames_per_step):
        self.n_frames_per_step = n_frames_per_step

    def __call__(self, batch):
        """Collate's training batch from normalized text and mel-spectrogram
        PARAMS
        ------
        batch: [text_normalized, mel_normalized]
        """
        # Right zero-pad all one-hot text sequences to max input length
        input_lengths, ids_sorted_decreasing = torch.sort( torch.LongTensor([len(batch[i][1]) for i in range(len(batch))]), dim=0, descending=True)
        max_input_len = input_lengths[0]
        # max_input_len = 512


        text_padded = torch.LongTensor(len(batch), max_input_len)
        text_padded.zero_()

        for i in range(len(ids_sorted_decreasing)):
            text = batch[ids_sorted_decreasing[i].numpy()][1]
            text_padded[i, :text.size(0)] = text

        # Right zero-pad mel-spec
        num_mels = batch[0][0].size(0)
        max_target_len = max([batch[i][0].size(1) for i in range(len(batch))])


        if max_target_len % self.n_frames_per_step != 0:
            max_target_len += self.n_frames_per_step - max_target_len % self.n_frames_per_step
            assert max_target_len % self.n_frames_per_step == 0

        # include mel padded and gate padded
        mel_padded = torch.FloatTensor(len(batch), num_mels, max_target_len)
        mel_padded.zero_()
        gate_padded = torch.FloatTensor(len(batch), max_target_len)
        gate_padded.zero_()
        output_lengths = torch.LongTensor(len(batch))
        for i in range(len(ids_sorted_decreasing)):
            mel = batch[ids_sorted_decreasing[i].numpy()][0]
            mel_padded[i, :, :mel.size(1)] = mel
            gate_padded[i, mel.size(1)-1:] = 1
            output_lengths[i] = mel.size(1)
        
        # Windowing
        text_windowed = []
        for text in text_padded:
            text_windowed.append(window_split(text))
        audio_windowed = []
        for audio in mel_padded:
            audio_windowed.append(window_split(audio))

        return (text_windowed, audio_windowed)
</code></pre>