VQVAE doesn't learn on audio data

username: Flock1
cooked: <p>Hi,</p>
<p>I am trying to train a VQVAE. I used <a href="https://colab.research.google.com/github/zalandoresearch/pytorch-vq-vae/blob/master/vq-vae.ipynb" rel="noopener nofollow ugc">this</a> implementation.</p>
<p>As for the input, I tried various types of input, either taking some part of the original audio or taking that audio and converting it to a 2D matrix and applying minmax normalization. I tried various input types and even different loss functions but the model isn’t able to reconstruct the original audio. I have tried MSE and L1 loss. I have even tried UNet, but none of them seem to work. This is how I load the data:</p>
<pre><code class="lang-auto">class data_gen(torch.utils.data.Dataset):
    def __init__(self, files):
        
        self.data = files
        
    def __getitem__(self, i):

        tmp = self.data[i]
        x = np.load('/aud_images/'+tmp)
        noise = np.random.normal(0.3,0.01,(256,256))
        x_n = x+noise
        x = np.reshape(x,(1,x.shape[0],x.shape[1]))
        x = torch.from_numpy(x).float()
        x_n = np.reshape(x_n,(1,x_n.shape[0],x_n.shape[1]))
        x_n = torch.from_numpy(x_n).float()

        
        return x_n, x

    def __len__(self): 
        
        return len(self.data)
</code></pre>
<p>What should I do since almost everywhere I read that MSE loss works fine for audio data. Let me know.</p>

username: shivammehta007
cooked: <p>As I work with Speech Synthesis, from what I have seen people generally have a hard time reconstructing the raw waveforms and I have not seen a lot of people trying to do it as you lose a lot of information in the frequency domain which could be useful for the properties of the Audio data, I think a good idea will be to turn it into more audio friendly representation’s first like <strong>Spectrogram</strong> or <strong>MFCC</strong> or <strong>Mel-Spectrogram</strong> if it is speech.</p>
<p>More information could be found regarding this in <code>torchaudio.transforms.Spectrogram</code> and <code>torchaudio.transforms.MFCC</code>.</p>
<p>Then you can use any vocoder to reconstruct it into the raw waveforms.</p>
<p>I hope it helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Flock1
cooked: <p>But constructing audio from data like Spectrogram, I face the problem of output size. I have audio data of different sizes. So the input and output (which will be a fully connected layer) will be of different sizes. So how do I deal with that?</p>
<p>Also, why is it so hard to reconstruct raw audio data?</p>

username: Flock1
cooked: <p>So then how is Jukebox able to work so well? I don’t think so it uses any frequency domain data for audio reconstruction.</p>

username: shivammehta007
cooked: <p>I am not familiar with it but a high level, understanding from what I see is that they do use some other feature representation to reconstruct like they used Transformers for text representations and conditioned the synthesis on it as well.</p>

username: Flock1
cooked: <p>So <a href="https://youtu.be/1aO-uHXbzmQ?t=1127" rel="noopener nofollow ugc">this</a> is a good explanation of Jukebox and you can see in the architecture that they are not using any spectral information to recreate the original signal.</p>

username: shivammehta007
cooked: <p>I don’t know! If it is still of interest to you, another major challenge that I forgot to mention regarding why people often avoid synthesising waveforms is the sampling rate of the raw audio waveform. Generally, it is &gt;= 16000 Hz which means for one second of audio you need to synthesise 16000 data points, which unless done by some parallel process is a lot of datapoints to synthesise. Therefore people often try to synthesise spectrograms and then transform into waveforms with help of some vocoders.</p>

username: 111493
cooked: <p>For audio, an autoregressive decoder should be used, or it cannot be trained with MSE loss. Spectral loss can be used without autoregressive structure.</p>