Audio frame wise training

username: oceanchad
cooked: <p>I’m trying to build MIR 1K dataset and train with DNN frame wise based.</p>
<p>Given the spectrogram with feature data (40, 150) as below and label (150). eg. frame index 0 has feature (40, ) with MIDI label 68.</p>
<p>The above Mel spec and label is obtained as below.</p>
<pre><code class="lang-auto">class MIR1K(Dataset):
  """
  offical document
  frame size of 40ms -&gt; 640 sample points
  hop size of 20 ms -&gt; 320 sample points
  each 10ms hop size has one pitch vector

  Args:
      Dataset (_type_): return mel, pitch
  """

  def __init__(self, transform=None, segment_len=150):
  # def __init__(self, val_ratio:float):
    self.audio_dir = DBPATH + MIRAUD + '/Wavfile/'
    self.label_dir = DBPATH + MIRAUD + '/PitchLabel'
    self.total_song = os.listdir(self.audio_dir)
    self._transform = transform
    self.o_sr = 0
    self.segment_len = segment_len
    # self.val_ratio = val_ratio

  def __len__(self):
    'return the fold song number'
    return len(self.total_song)

  def __getitem__(self, index):
    songname = self._get_song(self.audio_dir, index)
    print(songname)
    audio_path = self._get_audio(songname)
    pitch = self._get_label(songname)

    audio, self.o_sr = librosa.load(audio_path, sr=None, mono=True)
    if self._transform is None:
      return torch.FloatTensor(audio),  torch.FloatTensor(pitch)
    else:
      audio = self._transform(audio, self.o_sr)

    # rescale rescale audio and pitch
    shape_diff = audio.shape[1] - pitch.shape[0]
    if (shape_diff) == 1:
      audio = audio[:,1:]
    elif (shape_diff) == 2:
      audio = audio[:,1:-1]
    else:
      assert(shape_diff &gt; 2)


    # Segmemt mel-spectrogram into "segment_len" frames
    if (audio.shape[1] &gt; self.segment_len):
      start = random.randint(0, audio.shape[1] - self.segment_len)
      mel = torch.FloatTensor(audio[:,start:start+self.segment_len])
      pitch = torch.FloatTensor(pitch[start:start+self.segment_len])
    else:
      mel = torch.FloatTensor(audio)
      pitch = torch.FloatTensor(pitch)

    return mel, pitch
    # return audio, pitch

  def _get_song(self, audio_dir, index):
    return os.listdir(audio_dir)[index].split('.')[0]

  def _get_audio(self, song):
    if ".wav" in song:
      return os.path.join(self.audio_dir, song)
    return os.path.join(self.audio_dir, song + '.wav')

  def _get_label(self, song):
    pitch_label = np.loadtxt(os.path.join(self.label_dir, song + '.pv'))
    # convert midi to frequency
    pitch = 2**( (pitch_label - 69)/12) * 440
    return pitch
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/9/99e9a5d556d5a0d7e8c0ad730330529e77957061.png" data-download-href="https://discuss.pytorch.org/uploads/default/99e9a5d556d5a0d7e8c0ad730330529e77957061" title="Figure_1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/9/99e9a5d556d5a0d7e8c0ad730330529e77957061.png" alt="Figure_1" data-base62-sha1="lXzDKcO2ahZ9dhxIQhoF2kaOHpn" width="517" height="372" data-dominant-color="89C89B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Figure_1</span><span class="informations">836×601 26.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there any tutorial to use the data and train frame wise ?</p>