Repository: pytorch/audio

Issue:
amplitude normalization in create_fb_matrix
## 🐛 Not necessarily a Bug

Not exactly a bug, but I think it makes more sense if the triangular Mel filterbanks are area-normalized, as opposed to the current behavior where they are height-normalized.

## To Reproduce / Expected behavior

Steps to reproduce the behavior:

```python
import matplotlib.pyplot as plt
import torchaudio
import librosa

plt.figure(figsize=(16, 8))
plt.subplot(2, 1, 1)
plt.plot(torchaudio.transforms.MelScale(n_stft=1025, sample_rate=16000).fb.detach().cpu().numpy())
plt.title('torchaudio Mel filterbanks')

plt.subplot(2, 1, 2)
plt.plot(librosa.filters.mel(sr=16000, n_fft=2048).transpose())
plt.title('librosa Mel filterbanks')
plt.show()
```

![image](https://user-images.githubusercontent.com/266841/65094873-abe6b200-d973-11e9-9464-fd652f3a5faa.png)

The former has an advantage of preserving the energy similar to the STFT, while the latter keeps the overall spectral envelope unchanged.

If this is an intended behavior, feel free to close this issue. Thanks!

## Environment

 - What commands did you used to install torchaudio (conda/pip/build from source)?

I did `pip install torchaudio -f https://download.pytorch.org/whl/torch_stable.html`, following `README.md`

 - What does `torchaudio.__version__` print? (If applicable)

'0.3.0'

Please copy and paste the output from our [environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py).

```
Collecting environment information...
PyTorch version: 1.2.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 16.04.5 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
CMake version: version 3.11.1

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration:
GPU 0: Tesla V100-SXM2-16GB
GPU 1: Tesla V100-SXM2-16GB
GPU 2: Tesla V100-SXM2-16GB
GPU 3: Tesla V100-SXM2-16GB
GPU 4: Tesla V100-SXM2-16GB
GPU 5: Tesla V100-SXM2-16GB
GPU 6: Tesla V100-SXM2-16GB
GPU 7: Tesla V100-SXM2-16GB

Nvidia driver version: 410.79
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.2

Versions of relevant libraries:
[pip3] numpy==1.16.5
[pip3] pytorch-memlab==0.0.3
[pip3] torch==1.2.0
[pip3] torch-bst==0.0.0
[pip3] torch-data==0.0.1
[pip3] torchaudio==0.3.0
[pip3] torchvision==0.4.0
[conda] blas                      1.0                         mkl
[conda] faiss-gpu                 1.5.3            py37h1a5d453_0    pytorch
[conda] mkl                       2019.4                      243
[conda] mkl-service               2.3.0            py37he904b0f_0
[conda] mkl_fft                   1.0.14           py37ha843d7b_0
[conda] mkl_random                1.0.2            py37hd81dba3_0
[conda] pytorch-memlab            0.0.3                    pypi_0    pypi
[conda] torch                     1.2.0                    pypi_0    pypi
[conda] torch-bst                 0.0.0                     dev_0    <develop>
[conda] torch-data                0.0.1                     dev_0    <develop>
[conda] torchaudio                0.3.0                    pypi_0    pypi
[conda] torchvision               0.4.0                    pypi_0    pypi
```



Discussion:
Thanks for pointing this out :) Do we have a comparison with kaldi for this graph?
I can still reproduce this PR after #292.
This would be breaking backward compatibility of course, so we'd need strong demand to change this.

* Is there a simple way to convert from one to the other as a post-processing?
* I'd be open to having an option to switch too. Please feel free to open a pull request for this :)
To answer the first, I was able to convert from one to the other by adding a linear function of Mel frequency, with minimal numerical differences.

I haven't used Kaldi, but I agree that breaking backward compatibility is a bad idea. I think it'd still be helpful to the community if we can make the output exactly match librosa's outputs, as it is very popular for Music processing in Python, and [after NVIDIA's Tacotron 2 implementation adopted the same filterbank](https://github.com/NVIDIA/tacotron2/blob/1ec0e5e8cd1152f3a551027201a26de0389e1235/layers.py#L50-L51) many TTS systems started to use Librosa's Mel spectrograms as a common mid-level representation for different vocoder implementations, for instance.

I'll be interested in working on a PR for this, in the following ~4 weeks timeline. If you feel that's okay please feel free to assign this to me!
> To answer the first, I was able to convert from one to the other by adding a linear function of Mel frequency, with minimal numerical differences.

If this is a general transformation that can be used elsewhere, it could be a solution. Otherwise, offering a toggle/switch is more appropriate.

> I haven't used Kaldi, but I agree that breaking backward compatibility is a bad idea. I think it'd still be helpful to the community if we can make the output exactly match librosa's outputs, as it is very popular for Music processing in Python, and [after NVIDIA's Tacotron 2 implementation adopted the same filterbank](https://github.com/NVIDIA/tacotron2/blob/1ec0e5e8cd1152f3a551027201a26de0389e1235/layers.py#L50-L51) many TTS systems started to use Librosa's Mel spectrograms as a common mid-level representation for different vocoder implementations, for instance.
> 
> I'll be interested in working on a PR for this, in the following ~4 weeks timeline. If you feel that's okay please feel free to assign this to me!

Thanks! That would be great :)

+1 for this! 
 > To answer the first, I was able to convert from one to the other by adding a linear function of Mel frequency, with minimal numerical differences.

@jongwook do you have a gist for this?
@delip I did, and unfortunately I no longer have access to it. 

I'm feeling bad about not getting my hands on this over the break, but I should be able to soon. I will keep you posted!
https://gist.github.com/jongwook/d6fdf57a6fd06d56a2bfd6b505e7804c

The notebook above demonstrates that Torchaudio mel spectrogram and librosa mel spectrogram differ by a linear function of Mel frequency, when `htk=True` is specified in librosa.

I'm thinking of adding a keyword argument `normalized=False` to `MelScale` and `MelSpectrogram`, and use librosa-style filterbank when `normalized=True` is given.

Regarding `htk=True`, since the librosa default Slaney filterbank is also quite widely used (ultimately from MATLAB auditory toolbox), I'd be also keen to add another optional kwargs `filterbank="htk" or "slaney"`.

Ultimately, it'd be nice to have module `torchaudio.compliance.librosa`, but first things first!

Resolving patch:
diff --git a/torchaudio/functional.py b/torchaudio/functional.py
--- a/torchaudio/functional.py
+++ b/torchaudio/functional.py
@@ -424,7 +424,8 @@ def create_fb_matrix(
         f_min: float,
         f_max: float,
         n_mels: int,
-        sample_rate: int
+        sample_rate: int,
+        norm: str = "",
 ) -> Tensor:
     r"""Create a frequency bin conversion matrix.
 
@@ -434,6 +435,8 @@ def create_fb_matrix(
         f_max (float): Maximum frequency (Hz)
         n_mels (int): Number of mel filterbanks
         sample_rate (int): Sample rate of the audio waveform
+        norm (str): If 'slaney', divide the triangular mel weights by the width of the mel band
+        (area normalization). (Default: '')
 
     Returns:
         Tensor: Triangular filter banks (fb matrix) of size (``n_freqs``, ``n_mels``)
@@ -461,6 +464,12 @@ def create_fb_matrix(
     down_slopes = (-1.0 * slopes[:, :-2]) / f_diff[:-1]  # (n_freqs, n_mels)
     up_slopes = slopes[:, 2:] / f_diff[1:]  # (n_freqs, n_mels)
     fb = torch.max(zero, torch.min(down_slopes, up_slopes))
+
+    if norm == "slaney":
+        # Slaney-style mel is scaled to be approx constant energy per channel
+        enorm = 2.0 / (f_pts[2:n_mels + 2] - f_pts[:n_mels])
+        fb *= enorm.unsqueeze(0)
+
     return fb
 
 


Test patch:
diff --git a/test/test_librosa_compatibility.py b/test/test_librosa_compatibility.py
--- a/test/test_librosa_compatibility.py
+++ b/test/test_librosa_compatibility.py
@@ -53,19 +53,20 @@ def test_griffinlim(self):
 
         torch.testing.assert_allclose(ta_out, lr_out, atol=5e-5, rtol=1e-5)
 
-    def _test_create_fb(self, n_mels=40, sample_rate=22050, n_fft=2048, fmin=0.0, fmax=8000.0):
+    def _test_create_fb(self, n_mels=40, sample_rate=22050, n_fft=2048, fmin=0.0, fmax=8000.0, norm=None):
         librosa_fb = librosa.filters.mel(sr=sample_rate,
                                          n_fft=n_fft,
                                          n_mels=n_mels,
                                          fmax=fmax,
                                          fmin=fmin,
                                          htk=True,
-                                         norm=None)
+                                         norm=norm)
         fb = F.create_fb_matrix(sample_rate=sample_rate,
                                 n_mels=n_mels,
                                 f_max=fmax,
                                 f_min=fmin,
-                                n_freqs=(n_fft // 2 + 1))
+                                n_freqs=(n_fft // 2 + 1),
+                                norm=norm)
 
         for i_mel_bank in range(n_mels):
             torch.testing.assert_allclose(fb[:, i_mel_bank], torch.tensor(librosa_fb[i_mel_bank]),
@@ -79,6 +80,12 @@ def test_create_fb(self):
         self._test_create_fb(n_mels=56, fmin=800.0, fmax=900.0)
         self._test_create_fb(n_mels=56, fmin=1900.0, fmax=900.0)
         self._test_create_fb(n_mels=10, fmin=1900.0, fmax=900.0)
+        self._test_create_fb(n_mels=128, sample_rate=44100, norm="slaney")
+        self._test_create_fb(n_mels=128, fmin=2000.0, fmax=5000.0, norm="slaney")
+        self._test_create_fb(n_mels=56, fmin=100.0, fmax=9000.0, norm="slaney")
+        self._test_create_fb(n_mels=56, fmin=800.0, fmax=900.0, norm="slaney")
+        self._test_create_fb(n_mels=56, fmin=1900.0, fmax=900.0, norm="slaney")
+        self._test_create_fb(n_mels=10, fmin=1900.0, fmax=900.0, norm="slaney")
 
     def test_amplitude_to_DB(self):
         spec = torch.rand((6, 201))
diff --git a/test/test_torchscript_consistency.py b/test/test_torchscript_consistency.py
--- a/test/test_torchscript_consistency.py
+++ b/test/test_torchscript_consistency.py
@@ -99,7 +99,8 @@ def func(_):
             f_max = 20.0
             n_mels = 10
             sample_rate = 16000
-            return F.create_fb_matrix(n_stft, f_min, f_max, n_mels, sample_rate)
+            norm = ""
+            return F.create_fb_matrix(n_stft, f_min, f_max, n_mels, sample_rate, norm)
 
         dummy = torch.zeros(1, 1)
         self._assert_consistency(func, dummy)