Repository: pytorch/audio

Issue:
transforms.AmplitudeToDB does not handle cut-off correctly for multi-channel or batched data
## 🐛 Bug

From my understanding (based on e.g. #337), all transforms should be able to operate on tensors with dimensions `(batch, channels, ...)` or `(channels, ...)`, with `...` being dependent on the type of data being processed, e.g.`time` for waveform data and `freq, time` for spectrograms. In this way, we can pass multiple chunks of data (waveforms, spectrograms…) at once and expect to get the same results as if we would pass them one by one.

However, this is not the case `transforms.AmplitudeToDB`: As easily traceable in the [source code of the corresponding functional](https://pytorch.org/audio/_modules/torchaudio/functional.html#amplitude_to_DB), this transform blindly operates on the passed tensor without taking its dimensionality and the related semantics into account in any way.

This becomes a problem in the calculation of the cut-off. The purpose of this step is to clamp low dB values to a minimum value some fixed amount of decibels below the maximum value _in the respective spectrogram_. However, `amplitude_to_db` uses the single _global_ maximum in the passed tensor to calculate the cut-off for _all_ contained spectrograms. Thus, when passing batched data, the results for one spectrogram are dependent on all the other spectrograms in the same batch, which to my understanding is not the correct behavior. My conclusion is that `AmplitudeToDB` silently outputs wrong (in the sense of `transforms` general interface contract) data for batched or multi-channel data, which I would consider really dangerous for applications.

Ideally, this should be fixed directly in `functional.amplitude_to_DB`, so that we can also pass batched data there.

## Environment
TorchAudio 0.7.0



Discussion:
Thanks for opening the issue :) Are you referring to the use of `x_db.max()` on [this line](https://github.com/pytorch/audio/blob/master/torchaudio/functional/functional.py#L225) when `top_db is not None`? Do you have an example that would give an incorrect result? 
> Thanks for opening the issue :) Are you referring to the use of `x_db.max()` on [this line](https://github.com/pytorch/audio/blob/master/torchaudio/functional.py#L238) when `top_db is not None`?

Exactly.

> Do you have an example that would give an incorrect result?

Sure. Using a batch of two 2*2 example "spectrograms":

```
In: spec_batch = torch.tensor([[[2.0, 1.0], [4.0, 0.0]], [[10000000000.0, 5000000000.0], [8000000000.0, 9000000000.0]]])

In: F.amplitude_to_DB(spec_batch[0], 10.0, 0.0, 1.0, 80.0)  # Processing the first spectrogram on its own yields expected result.
Out: 
tensor([[ -6.9897, -10.0000],
        [ -3.9794, -83.9794]])

In: F.amplitude_to_DB(spec_batch, 10.0, 0.0, 1.0, 80.0)  # Processing the whole batch at once yields different (incorrect) results for the first spectrogram.
Out: 
tensor([[[10.0000, 10.0000],
         [10.0000, 10.0000]],
        [[90.0000, 86.9897],
         [89.0309, 89.5424]]])
```

The key point is that results for individual items should be the same regardless whether they are processed individually or in a batch.
That's a very good catch, thanks! This should be replaced by a min/max, e.g. [here](https://github.com/pytorch/pytorch/issues/2793#issuecomment-428784713). Do you want to open a pull request for this?
Thanks for your feedback!

As I'm not familiar with the Torch (Audio) Codebase and especially test infrastructure, I'm afraid I won't be able to open a pull request myself.
I've got a local patch for this, didn't realize there was already a ticket open. Would you like me to open a PR?
I think in order to upstream this I think I need to clarify some things: 

1. My understanding is that the objective here is to retain the clamping behaviour, but apply a separate clamp per-spectrogram. Is this correct?
2. What input shape should `amplitude_to_DB` expect? Is it always a spectrogram with N channels, requiring batches to be packed? Should it be capable of receiving an unpacked batch? Are there always two trailing dimensions, freq & time, or might the data be in a different format?
3. Should the clamping be applied per-spectrogram, or per-channel? If it's the former, that makes things more complicated as the different MFCCs are packed - so there's a single combined dimension for the batch & channel.

My patch currently clamps per-channel (not per spectrogram), and assumes a packed batch of two-dimensional spectrogram channels (so three dimensions, `[channels, freq, time]`).
> I've got a local patch for this, didn't realize there was already a ticket open. Would you like me to open a PR?

That would great, thanks! :)

> 1. My understanding is that the objective here is to retain the clamping behaviour

Yes, we should retain the clamping behavior. 

> but apply a separate clamp per-spectrogram. Is this correct?

That's an interesting point: per channel or spectrogram? Reading at the code, I'd say the intended behavior is likely to have been per spectrogram, since the batching came later. So let's keep this behavior.

> 2. What input shape should `amplitude_to_DB` expect? Is it always a spectrogram with N channels, requiring batches to be packed? Should it be capable of receiving an unpacked batch? 

Batches and channels were packed together to add batching support, so this does need to be fixed. In particular, this means the transform should likely not fold batches into channels to apply the transform.

> Are there always two trailing dimensions, freq & time, or might the data be in a different format?

The format should be `(..., freq, time)`. The function does not apply to complex tensor (which would have had a shape (..., freq, time, 2).

> 3. Should the clamping be applied per-spectrogram, or per-channel? If it's the former, that makes things more complicated as the different MFCCs are packed - so there's a single combined dimension for the batch & channel.

Yes, indeed, though [comment](https://github.com/pytorch/audio/issues/994#issuecomment-724382436) above would help take care of replacing clamp by min/max.


Awesome!

>Reading at the code, I'd say the intended behavior is likely to have been per spectrogram, since the batching came later. So let's keep this behavior.

This also matches [librosa's](https://github.com/librosa/librosa/blob/a53fa56bdb6695a994008d5b6ccd0100870a6036/librosa/core/spectrum.py#L1565) behavior.
Working through this now - I need to clarify the intended structure.

The current [test](https://github.com/pytorch/audio/blob/master/test/torchaudio_unittest/functional/functional_cpu_test.py#L84) expects `amplitude_to_DB` to work on both waves and spectrograms. This means we can't infer the indexes of the input tensor unless we either assume the data is always batched, or always a spectrogram. Would you like two methods, `amplitude_to_DB` and `amplitude_to_DB_batched`? Or maybe `amplitude_to_DB_wave` and `amplitude_to_DB_spec`? Alternatively, it could always expect batched data or wave support could be dropped.

Edit: I'm assuming flags aren't the correct way to do this in PyTorch, e.g. [mask_along_axis_iid](https://github.com/pytorch/audio/blob/master/torchaudio/functional/functional.py#L527)

(The same applies to the `AmplitudeToDB` transform.)
@vincentqb any thoughts? I have the PR ready to go, I just need to know the structure. I'm thinking following the lead of `mask_along_axis` and having two methods is sensible.

(Side note, but what does `iid` stand for specifically?)
HI @jcaw 

You can go ahead and open a PR. That way it is easier to keep the discussion going.
I would like to update our test infrastructure to catch this kind of bugs, so it will help to think of a way to fix the tests.
Great, will do. 

Resolving patch:
diff --git a/torchaudio/functional/functional.py b/torchaudio/functional/functional.py
--- a/torchaudio/functional/functional.py
+++ b/torchaudio/functional/functional.py
@@ -237,14 +237,16 @@ def amplitude_to_DB(
         db_multiplier: float,
         top_db: Optional[float] = None
 ) -> Tensor:
-    r"""Turn a tensor from the power/amplitude scale to the decibel scale.
+    r"""Turn a spectrogram from the power/amplitude scale to the decibel scale.
 
-    This output depends on the maximum value in the input tensor, and so
-    may return different values for an audio clip split into snippets vs. a
-    full clip.
+    The output of each tensor in a batch depends on the maximum value of that tensor,
+    and so may return different values for an audio clip split into snippets vs. a full clip.
 
     Args:
-        x (Tensor): Input tensor before being converted to decibel scale
+
+        x (Tensor): Input spectrogram(s) before being converted to decibel scale. Input should take
+          the form `(..., freq, time)`. Batched inputs should include a channel dimension and
+          have the form `(batch, channel, freq, time)`.
         multiplier (float): Use 10. for power and 20. for amplitude
         amin (float): Number to clamp ``x``
         db_multiplier (float): Log10(max(reference value and amin))
@@ -258,7 +260,15 @@ def amplitude_to_DB(
     x_db -= multiplier * db_multiplier
 
     if top_db is not None:
-        x_db = x_db.clamp(min=x_db.max().item() - top_db)
+        # Expand batch
+        shape = x_db.size()
+        packed_channels = shape[-3] if x_db.dim() > 2 else 1
+        x_db = x_db.reshape(-1, packed_channels, shape[-2], shape[-1])
+
+        x_db = torch.max(x_db, (x_db.amax(dim=(-3, -2, -1)) - top_db).view(-1, 1, 1, 1))
+
+        # Repack batch
+        x_db = x_db.reshape(shape)
 
     return x_db
 
diff --git a/torchaudio/transforms.py b/torchaudio/transforms.py
--- a/torchaudio/transforms.py
+++ b/torchaudio/transforms.py
@@ -530,24 +530,16 @@ def forward(self, waveform: Tensor) -> Tensor:
         Returns:
             Tensor: specgram_mel_db of size (..., ``n_mfcc``, time).
         """
-
-        # pack batch
-        shape = waveform.size()
-        waveform = waveform.reshape(-1, shape[-1])
-
         mel_specgram = self.MelSpectrogram(waveform)
         if self.log_mels:
             log_offset = 1e-6
             mel_specgram = torch.log(mel_specgram + log_offset)
         else:
             mel_specgram = self.amplitude_to_DB(mel_specgram)
-        # (channel, n_mels, time).tranpose(...) dot (n_mels, n_mfcc)
-        # -> (channel, time, n_mfcc).tranpose(...)
-        mfcc = torch.matmul(mel_specgram.transpose(1, 2), self.dct_mat).transpose(1, 2)
-
-        # unpack batch
-        mfcc = mfcc.reshape(shape[:-1] + mfcc.shape[-2:])
 
+        # (..., channel, n_mels, time).tranpose(...) dot (n_mels, n_mfcc)
+        # -> (..., channel, time, n_mfcc).tranpose(...)
+        mfcc = torch.matmul(mel_specgram.transpose(-2, -1), self.dct_mat).transpose(-2, -1)
         return mfcc
 
 


Test patch:
diff --git a/test/torchaudio_unittest/batch_consistency_test.py b/test/torchaudio_unittest/batch_consistency_test.py
--- a/test/torchaudio_unittest/batch_consistency_test.py
+++ b/test/torchaudio_unittest/batch_consistency_test.py
@@ -3,6 +3,8 @@
 import itertools
 from parameterized import parameterized
 
+import math
+
 import torch
 import torchaudio
 import torchaudio.functional as F
@@ -59,6 +61,78 @@ def test_detect_pitch_frequency(self, frequency, sample_rate, n_channels):
                                              n_channels=n_channels, duration=5)
         self.assert_batch_consistencies(F.detect_pitch_frequency, waveform, sample_rate)
 
+    def test_amplitude_to_DB(self):
+        torch.manual_seed(0)
+        spec = torch.rand(2, 100, 100) * 200
+
+        amplitude_mult = 20.
+        amin = 1e-10
+        ref = 1.0
+        db_mult = math.log10(max(amin, ref))
+
+        # Test with & without a `top_db` clamp
+        self.assert_batch_consistencies(F.amplitude_to_DB, spec, amplitude_mult,
+                                        amin, db_mult, top_db=None)
+        self.assert_batch_consistencies(F.amplitude_to_DB, spec, amplitude_mult,
+                                        amin, db_mult, top_db=40.)
+
+    def test_amplitude_to_DB_itemwise_clamps(self):
+        """Ensure that the clamps are separate for each spectrogram in a batch.
+
+        The clamp was determined per-batch in a prior implementation, which
+        meant it was determined by the loudest item, thus items weren't
+        independent. See:
+
+        https://github.com/pytorch/audio/issues/994
+
+        """
+        amplitude_mult = 20.
+        amin = 1e-10
+        ref = 1.0
+        db_mult = math.log10(max(amin, ref))
+        top_db = 20.
+
+        # Make a batch of noise
+        torch.manual_seed(0)
+        spec = torch.rand([2, 2, 100, 100]) * 200
+        # Make one item blow out the other
+        spec[0] += 50
+
+        batchwise_dbs = F.amplitude_to_DB(spec, amplitude_mult, amin,
+                                          db_mult, top_db=top_db)
+        itemwise_dbs = torch.stack([
+            F.amplitude_to_DB(item, amplitude_mult, amin,
+                              db_mult, top_db=top_db)
+            for item in spec
+        ])
+
+        self.assertEqual(batchwise_dbs, itemwise_dbs)
+
+    def test_amplitude_to_DB_not_channelwise_clamps(self):
+        """Check that clamps are applied per-item, not per channel."""
+        amplitude_mult = 20.
+        amin = 1e-10
+        ref = 1.0
+        db_mult = math.log10(max(amin, ref))
+        top_db = 40.
+
+        torch.manual_seed(0)
+        spec = torch.rand([1, 2, 100, 100]) * 200
+        # Make one channel blow out the other
+        spec[:, 0] += 50
+
+        specwise_dbs = F.amplitude_to_DB(spec, amplitude_mult, amin,
+                                         db_mult, top_db=top_db)
+        channelwise_dbs = torch.stack([
+            F.amplitude_to_DB(spec[:, i], amplitude_mult, amin,
+                              db_mult, top_db=top_db)
+            for i in range(spec.size(-3))
+        ])
+
+        # Just check channelwise gives a different answer.
+        difference = (specwise_dbs - channelwise_dbs).abs()
+        assert (difference >= 1e-5).any()
+
     def test_contrast(self):
         waveform = torch.rand(2, 100) - 0.5
         self.assert_batch_consistencies(F.contrast, waveform, enhancement_amount=80.)
@@ -103,7 +177,7 @@ class TestTransforms(common_utils.TorchaudioTestCase):
 
     """Test suite for classes defined in `transforms` module"""
     def test_batch_AmplitudeToDB(self):
-        spec = torch.rand((6, 201))
+        spec = torch.rand((2, 6, 201))
 
         # Single then transform then batch
         expected = torchaudio.transforms.AmplitudeToDB()(spec).repeat(3, 1, 1)
diff --git a/test/torchaudio_unittest/functional/functional_cpu_test.py b/test/torchaudio_unittest/functional/functional_cpu_test.py
--- a/test/torchaudio_unittest/functional/functional_cpu_test.py
+++ b/test/torchaudio_unittest/functional/functional_cpu_test.py
@@ -83,46 +83,78 @@ def test_pitch(self, frequency):
         self.assertFalse(s)
 
 
-class TestDB_to_amplitude(common_utils.TorchaudioTestCase):
-    def test_DB_to_amplitude(self):
-        # Make some noise
-        x = torch.rand(1000)
-        spectrogram = torchaudio.transforms.Spectrogram()
-        spec = spectrogram(x)
-
+class Testamplitude_to_DB(common_utils.TorchaudioTestCase):
+    @parameterized.expand([
+        ([100, 100],),
+        ([2, 100, 100],),
+        ([2, 2, 100, 100],),
+    ])
+    def test_reversible(self, shape):
+        """Round trip between amplitude and db should return the original for various shape
+
+        This implicitly also tests `DB_to_amplitude`.
+
+        """
+        amplitude_mult = 20.
+        power_mult = 10.
         amin = 1e-10
         ref = 1.0
-        db_multiplier = math.log10(max(amin, ref))
-
-        # Waveform amplitude -> DB -> amplitude
-        multiplier = 20.
-        power = 0.5
-
-        db = F.amplitude_to_DB(torch.abs(x), multiplier, amin, db_multiplier, top_db=None)
-        x2 = F.DB_to_amplitude(db, ref, power)
+        db_mult = math.log10(max(amin, ref))
 
-        self.assertEqual(x2, torch.abs(x), atol=5e-5, rtol=1e-5)
+        torch.manual_seed(0)
+        spec = torch.rand(*shape) * 200
 
         # Spectrogram amplitude -> DB -> amplitude
-        db = F.amplitude_to_DB(spec, multiplier, amin, db_multiplier, top_db=None)
-        x2 = F.DB_to_amplitude(db, ref, power)
+        db = F.amplitude_to_DB(spec, amplitude_mult, amin, db_mult, top_db=None)
+        x2 = F.DB_to_amplitude(db, ref, 0.5)
 
         self.assertEqual(x2, spec, atol=5e-5, rtol=1e-5)
 
-        # Waveform power -> DB -> power
-        multiplier = 10.
-        power = 1.
-
-        db = F.amplitude_to_DB(x, multiplier, amin, db_multiplier, top_db=None)
-        x2 = F.DB_to_amplitude(db, ref, power)
-
-        self.assertEqual(x2, torch.abs(x), atol=5e-5, rtol=1e-5)
-
         # Spectrogram power -> DB -> power
-        db = F.amplitude_to_DB(spec, multiplier, amin, db_multiplier, top_db=None)
-        x2 = F.DB_to_amplitude(db, ref, power)
-
-        self.assertEqual(x2, spec, atol=5e-5, rtol=1e-5)
+        db = F.amplitude_to_DB(spec, power_mult, amin, db_mult, top_db=None)
+        x2 = F.DB_to_amplitude(db, ref, 1.)
+
+        self.assertEqual(x2, spec)
+
+    @parameterized.expand([
+        ([100, 100],),
+        ([2, 100, 100],),
+        ([2, 2, 100, 100],),
+    ])
+    def test_top_db_clamp(self, shape):
+        """Ensure values are properly clamped when `top_db` is supplied."""
+        amplitude_mult = 20.
+        amin = 1e-10
+        ref = 1.0
+        db_mult = math.log10(max(amin, ref))
+        top_db = 40.
+
+        torch.manual_seed(0)
+        # A random tensor is used for increased entropy, but the max and min for
+        # each spectrogram still need to be predictable. The max determines the
+        # decibel cutoff, and the distance from the min must be large enough
+        # that it triggers a clamp.
+        spec = torch.rand(*shape)
+        # Ensure each spectrogram has a min of 0 and a max of 1.
+        spec -= spec.amin([-2, -1])[..., None, None]
+        spec /= spec.amax([-2, -1])[..., None, None]
+        # Expand the range to (0, 200) - wide enough to properly test clamping.
+        spec *= 200
+
+        decibels = F.amplitude_to_DB(spec, amplitude_mult, amin,
+                                     db_mult, top_db=top_db)
+        # Ensure the clamp was applied
+        below_limit = decibels < 6.0205
+        assert not below_limit.any(), (
+            "{} decibel values were below the expected cutoff:\n{}".format(
+                below_limit.sum().item(), decibels
+            )
+        )
+        # Ensure it didn't over-clamp
+        close_to_limit = decibels < 6.0207
+        assert close_to_limit.any(), (
+            f"No values were close to the limit. Did it over-clamp?\n{decibels}"
+        )
 
 
 class TestComplexNorm(common_utils.TorchaudioTestCase):