Repository: librosa/librosa

Issue:
librosa.stft 0.10.0-dev from github has numerical issue
**Describe the bug**

librosa.stft 0.10.0-dev from github has numerical issue. The current version is https://github.com/librosa/librosa/commit/dc0ed20f128d881ba5bcccd6687ac25fa38b7e27. I installed the github version with
```
pip install git+https://github.com/librosa/librosa
```

Results from pypi 0.9.2 are good.

**To Reproduce**
```python
import torch
import numpy as np
import librosa

librosa.version.show_versions()

def test1():
    sizes = (10, 4000)
    n_fft = 1024
    hop_length = 512
    center = True

    x = torch.randn(*sizes, dtype=torch.float64)
    res_torch = torch.stft(x,
        n_fft=n_fft, hop_length=hop_length,
        win_length=n_fft, center=center, return_complex=True)

    result = []
    window = np.ones(n_fft)
    res_librosa = librosa.stft(x.cpu().numpy(),
        n_fft=n_fft, hop_length=hop_length, win_length=n_fft,
        window=window, center=center, pad_mode='reflect')

    np.testing.assert_allclose(res_torch[:, :, 1:], res_librosa[:, :, 1:], atol=1e-12)  # this matches!
    np.testing.assert_allclose(res_torch, res_librosa, atol=1e-12)  # this doesn't match

test1()
```

librosa pypi 0.9.2
```
INSTALLED VERSIONS
------------------
python: 3.10.4 (main, Mar 23 2022, 23:05:40) [GCC 11.2.0]

librosa: 0.9.2

audioread: 3.0.0
numpy: 1.23.1
scipy: 1.8.0
sklearn: 1.1.1
joblib: 1.1.0
decorator: 5.1.1
soundfile: 0.10.3
resampy: 0.4.0
numba: 0.56.2
pooch: v1.6.0
packaging: 20.9

numpydoc: None
sphinx: 4.5.0
sphinx_rtd_theme: None
sphinx_multiversion: None
sphinx_gallery: None
mir_eval: None
ipython: None
sphinxcontrib-svg2pdfconverter: None
pytest: 7.1.2
pytest-mpl: None
pytest-cov: None
matplotlib: 3.5.1
samplerate: None
soxr: None
contextlib2: installed, no version number available
presets: None
```

librosa 0.10.0-dev0
```
INSTALLED VERSIONS
------------------
python: 3.10.4 (main, Mar 23 2022, 23:05:40) [GCC 11.2.0]

librosa: 0.10.0.dev0

audioread: 3.0.0
numpy: 1.23.1
scipy: 1.8.0
sklearn: 1.1.1
joblib: 1.1.0
decorator: 5.1.1
soundfile: 0.10.3
resampy: 0.4.0
numba: 0.56.2
pooch: v1.6.0
packaging: 20.9

numpydoc: None
sphinx: 4.5.0
sphinx_rtd_theme: None
sphinx_multiversion: None
sphinx_gallery: None
mir_eval: None
ipython: None
sphinxcontrib-svg2pdfconverter: None
pytest: 7.1.2
pytest-mpl: None
pytest-cov: None
matplotlib: 3.5.1
samplerate: None
soxr: None
contextlib2: installed, no version number available
presets: None
Traceback (most recent call last):
  File "/home/xwang/Developer/pytorch-test/librosa-0.10.0.py", line 27, in <module>
    test1()
  File "/home/xwang/Developer/pytorch-test/librosa-0.10.0.py", line 25, in test1
    np.testing.assert_allclose(res_torch, res_librosa, atol=1e-12)
  File "/home/xwang/.local/lib/python3.10/site-packages/numpy/testing/_private/utils.py", line 1527, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/home/xwang/.local/lib/python3.10/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-07, atol=1e-12

Mismatched elements: 5130 / 41040 (12.5%)
Max absolute difference: 3.09427497
Max relative difference: 112.4026424
 x: array([[[ 42.760336+0.000000e+00j,  67.242289+0.000000e+00j,
          62.596586+0.000000e+00j, ..., -37.036895+0.000000e+00j,
          35.28637 +0.000000e+00j,  58.455984+0.000000e+00j],...
 y: array([[[ 44.884661+0.000000e+00j,  67.242289+0.000000e+00j,
          62.596586+0.000000e+00j, ..., -37.036895+0.000000e+00j,
          35.28637 +0.000000e+00j,  58.455984+0.000000e+00j],...
```

**Expected behavior**
Results should be the same.

**Screenshots**
N/A

**Software versions**
<!--
Please run the following Python code snippet and paste the output below.
```
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import librosa; print("librosa", librosa.__version__)

librosa.show_versions()
```
-->

See above.


**Additional context**
pytorch is from the latest master on github, though I guess it's not the problem here, since I get matched results with librosa 0.9.2




Discussion:
Thanks for the thorough report - this is interesting. 


I'm able to reproduce your example on my local install (with torch 1.11.0).

The main PR in the 0.10 series that would affect this code is #1514, wherein we factored out the edge handling.  It's possible that we have a bug here, but it didn't show up in any of my testing during implementation of 1514.

It's worth noting that setting `center=False` in your example does allow the test to pass.  Also, setting `n_fft` to any value other than `2*hop_length` (1024, 1025, 1026, ...) produces a passing test.  It's possible that we have an off-by-one in our array slicing logic.

As a side note, I also tried swapping out float64 for float32, and we do get much closer to numerical precision:

```
AssertionError: 
Not equal to tolerance rtol=1e-07, atol=1e-12

Mismatched elements: 20357 / 35910 (56.7%)
Max absolute difference: 1.5377536e-05
Max relative difference: 1.9009907e-05
 x: array([[[-23.94077  +0.j      ,  20.457932 +0.j      ,
          18.15456  +0.j      , ...,  80.080505 +0.j      ,
          81.312065 +0.j      ,  51.30126  +0.j      ],...
 y: array([[[-23.94077  +0.j      ,  20.457933 +0.j      ,
          18.154564 +0.j      , ...,  80.080505 +0.j      ,
          81.312065 +0.j      ,  51.301258 +0.j      ],...
```
I would consider this "close enough", and this is probably how an error could have slipped through in the first place.

So it looks like just the 0th bin (dc term) in the first frame, which
suggests something in dc offset treatment.

But can it really be that float32 and float64 give different answers far
beyond their resolution differences?

  DAn.

On Tue, Sep 6, 2022 at 20:03 Brian McFee ***@***.***> wrote:

> Thanks for the thorough report - this is interesting.
>
> I'm able to reproduce your example on my local install (with torch 1.11.0).
>
> The main PR in the 0.10 series that would affect this code is #1514
> <https://github.com/librosa/librosa/pull/1514>, wherein we factored out
> the edge handling. It's possible that we have a bug here, but it didn't
> show up in any of my testing during implementation of 1514.
>
> It's worth noting that setting center=False in your example does allow
> the test to pass. Also, setting n_fft to any value other than 2*hop_length
> (1024, 1025, 1026, ...) produces a passing test. It's possible that we have
> an off-by-one in our array slicing logic.
>
> As a side note, I also tried swapping out float64 for float32, and we do
> get much closer to numerical precision:
>
> AssertionError:
> Not equal to tolerance rtol=1e-07, atol=1e-12
>
> Mismatched elements: 20357 / 35910 (56.7%)
> Max absolute difference: 1.5377536e-05
> Max relative difference: 1.9009907e-05
>  x: array([[[-23.94077  +0.j      ,  20.457932 +0.j      ,
>           18.15456  +0.j      , ...,  80.080505 +0.j      ,
>           81.312065 +0.j      ,  51.30126  +0.j      ],...
>  y: array([[[-23.94077  +0.j      ,  20.457933 +0.j      ,
>           18.154564 +0.j      , ...,  80.080505 +0.j      ,
>           81.312065 +0.j      ,  51.301258 +0.j      ],...
>
> I would consider this "close enough", and this is probably how an error
> could have slipped through in the first place.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/librosa/librosa/issues/1567#issuecomment-1238763964>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAEGZUJAWAY43NR2WMIY2WDV47LU5ANCNFSM6AAAAAAQGIRLQY>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>

> So it looks like just the 0th bin (dc term) in the first frame, which suggests something in dc offset treatment.

In the example above, it looks that way, but testing everything past the DC bin also triggers an error:
```python
...:     np.testing.assert_allclose(res_torch[..., 1:], res_librosa[..., 1:], atol=1e-12)  # this matches!
...:     np.testing.assert_allclose(res_torch[..., 1:, :], res_librosa[..., 1:, :], atol=1e-12)
```
```
AssertionError: 
Not equal to tolerance rtol=1e-07, atol=1e-12

Mismatched elements: 5120 / 40960 (12.5%)
Max absolute difference: 2.55303805
Max relative difference: 183.62055503
 x: array([[[-54.860028+7.105427e-15j,   0.874121-5.332900e+00j,
          54.760726-3.649396e+01j, ..., -13.672482-2.277587e+01j,
          31.414315+4.429429e+01j, -11.343588-3.101224e+01j],...
 y: array([[[-54.765506-2.775558e-17j,   0.874121-5.332900e+00j,
          54.760726-3.649396e+01j, ..., -13.672482-2.277587e+01j,
          31.414315+4.429429e+01j, -11.343588-3.101224e+01j],...
```



> But can it really be that float32 and float64 give different answers far beyond their resolution differences?

My first thought was that there's some sneaky dtype promotion happening under the hood that only distinguishes the results at float64.  I ruled this out after the tests with different frame lengths passed - if it was a precision issue, it should fail in those cases too.
The most likely relevant bit of code is the following:
https://github.com/librosa/librosa/blob/dc0ed20f128d881ba5bcccd6687ac25fa38b7e27/librosa/core/spectrum.py#L259-L263
where we identify how many frames at the beginning (start) and end (tail) will depend on padding.  This gets used subsequently here:
https://github.com/librosa/librosa/blob/dc0ed20f128d881ba5bcccd6687ac25fa38b7e27/librosa/core/spectrum.py#L272-L284

A few quick spot checks with the frame and hop length seem to produce the correct results for me, so I'm not entirely sure if/where there's a bug, but this is the place to start looking.
After a bit of environment hacking, I have a setup with torch, librosa-dev and librosa-092 all living together.

I am indeed seeing a discrepancy between 0.9.2 and dev, but not when using the default Hann window.  Differences show up for ones or hamming, and I expect generally it will happen for windows that are strictly positive.  Hann is zero at the zeroth sample, and I think this is where the trouble is starting.

The strange thing is that it appears to be a constant real-valued difference to all frequencies in the zeroth frame.  The only way to get the same real-valued constant offset in all frequencies is if it appears at the zeroth sample of the (padded) frame, which is the only place where all DFT bases are simultaneously `1+0j`.  With reflection padding of `n_fft//2`, the zeroth sample should correspond to the (unpadded) input sample `y[n_fft//2]`.

Hacking the two STFT implementations to print out the first few samples of the padded array confirms this:
```python
sr = 22050

y = librosa.chirp(fmin=20, fmax=1000, duration=2)

S1 = librosa.stft(y, n_fft=1024, hop_length=512, center=True, pad_mode='reflect', window='ones')

S2 = librosa092.stft(y, n_fft=1024, hop_length=512, center=True, pad_mode='reflect', window='ones')

np.testing.assert_allclose(S1, S2, atol=1e-12)
```

```
# librosa-dev
[0.1675417  0.16165999 0.1675417  0.17341693 0.17928547 0.18514712
 0.19100167 0.19684892 0.20268865 0.20852067]

# librosa092
[0.15577202 0.16165999 0.1675417  0.17341693 0.17928547 0.18514712
 0.19100167 0.19684892 0.20268865 0.20852067]
```

The really confusing part at this point is that everything past the first sample is correct!  It's not as though the entire buffer is shifted by one sample or anything.

As an aside, this is good news in at least two respects: 1) the issue will not occur with any window that zeros the first sample, especially including the default Hann; 2) the issue does not occur with zeros or edge-padding.  These are important to keep in mind because it means that results computed with the default settings of the dev branch are still "correct".  (This is also almost certainly how the bug slipped through in the first place - insufficient testing on my part!)

Coming back to the issue above, it appears to be a simple off-by-one error with a non-simple outcome.  Notice that the first and third samples in -dev are identical: this is because `reflect` padding is ping-ponging off the edge of the array, which was too short to provide a full half-frame of samples.  This slice happens here: 
https://github.com/librosa/librosa/blob/dc0ed20f128d881ba5bcccd6687ac25fa38b7e27/librosa/core/spectrum.py#L279-L283
and adding a `+1` so that it becomes
```python
y_pre = np.pad(
    y[..., : (start_k - 1) * hop_length - n_fft // 2 + n_fft + 1],
    padding,
    mode=pad_mode,
)
```
seems to fix things and ensures numerical equivalence between versions.

Now that I have an environment set up to properly regression-test these things, I'll set up a full battery across configurations.  Once I've confirmed that the fix is correct, I'll put up a PR.

Thanks again @xwang233 for finding this!

---

<details>
<summary> :skull_and_crossbones:  How to install multiple versions of a package in one environment </summary>

1. Check out the version you want to install: `$ git checkout 0.9.2`
2. Install without dependencies to an isolated folder: `$ python -m pip install --no-deps --target=/home/bmcfee/librosa092`
3. Use the following code block:

```python
import importlib

spec = importlib.util.spec_from_file_location('librosa092',
                                              location='/home/bmcfee/librosa092/librosa/__init__.py')

librosa092 = importlib.util.module_from_spec(spec)

sys.modules["librosa092"] = librosa092

spec.loader.exec_module(librosa092)
```


</details>
The following now passes successfully with the proposed fix:
```python
y = np.random.randn(10, 4000)
for n_fft in tqdm([512, 1024, 1023, 1025, 2048], leave=False):
    for win_length in tqdm([n_fft, n_fft//2], leave=False):
        for hop_length in tqdm([256, 512, 1024], leave=False):
            for window in tqdm(['ones', 'hann', 'hamming'], leave=False):
                for pad_mode in tqdm(['reflect', 'edge', 'constant'], leave=False):
                    S1 = librosa.stft(y, n_fft=n_fft, hop_length=hop_length, win_length=win_length, pad_mode=pad_mode)
                    S2 = librosa092.stft(y, n_fft=n_fft, hop_length=hop_length, win_length=win_length, pad_mode=pad_mode)
                    np.testing.assert_allclose(S1, S2)
```

Resolving patch:
diff --git a/librosa/core/spectrum.py b/librosa/core/spectrum.py
--- a/librosa/core/spectrum.py
+++ b/librosa/core/spectrum.py
@@ -276,12 +276,16 @@ def stft(
             start = start_k * hop_length - n_fft // 2
             padding[-1] = (n_fft // 2, 0)
 
+            # +1 here is to ensure enough samples to fill the window
+            # fixes bug #1567
             y_pre = np.pad(
-                y[..., : (start_k - 1) * hop_length - n_fft // 2 + n_fft],
+                y[..., : (start_k - 1) * hop_length - n_fft // 2 + n_fft + 1],
                 padding,
                 mode=pad_mode,
             )
             y_frames_pre = util.frame(y_pre, frame_length=n_fft, hop_length=hop_length)
+            # Trim this down to the exact number of frames we should have
+            y_frames_pre = y_frames_pre[..., :start_k]
 
             # How many extra frames do we have from the head?
             extra = y_frames_pre.shape[-1]


Test patch: