Repository: lhotse-speech/lhotse

Issue:
Sometimes start_frame + num_frames > T in K2SpeechRecognitionIterableDataset
Here is a failure example :

```
feature shape (B,T,F) : torch.Size([10, 3017, 40])
start_frame + num_frames: tensor([1460, 3018, 1476, 2968, 1472, 2968, 1532, 2836, 1491, 2809, 1536, 2688,
        1545, 2681, 1557, 2593, 1594, 2085, 2580, 1640], dtype=torch.int32)
```
As you can see, there is `3018 > T = 3017`

I comput (and save) feature with
 https://github.com/k2-fsa/snowfall/blob/811e5333281a279e27e3008f5d025111d19cc487/egs/librispeech/asr/simple_v1/prepare.py#L42-L48,
then load `train-clean-100` with https://github.com/k2-fsa/snowfall/blob/811e5333281a279e27e3008f5d025111d19cc487/egs/librispeech/asr/simple_v1/train.py#L177-L181

Noted here is a warning about torch version, should I upgrade PyTorch version?
```
/ceph-hw/lhotse/lhotse/augmentation/torchaudio.py:13: UserWarning: Torchaudio SoX effects chains are only introduced in version 0.7 - please upgrade your PyTorch to 1.7+ and torchaudio to 0.7+ to use them.
  warnings.warn('Torchaudio SoX effects chains are only introduced in version 0.7 - '
```




Discussion:
Thanks for reporting, I will look into it today.

You don't have to update unless you plan to use torchaudio instead of
WavAugment for data augmentation. I'd suggest to make this transition when
the recipe is otherwise working and you start tuning which augmentations to
use.

pt., 13 lis 2020, 04:36 użytkownik Haowen Qiu <notifications@github.com>
napisał:

> Here is a failure example :
>
> feature shape (B,T,F) : torch.Size([10, 3017, 40])
> start_frame + num_frames: tensor([1460, 3018, 1476, 2968, 1472, 2968, 1532, 2836, 1491, 2809, 1536, 2688,
>         1545, 2681, 1557, 2593, 1594, 2085, 2580, 1640], dtype=torch.int32)
>
> As you can see, there is 3018 > T = 3017
>
> I comput (and save) feature with
>
> https://github.com/k2-fsa/snowfall/blob/811e5333281a279e27e3008f5d025111d19cc487/egs/librispeech/asr/simple_v1/prepare.py#L42-L48
> ,
> then load train-clean-100 with
> https://github.com/k2-fsa/snowfall/blob/811e5333281a279e27e3008f5d025111d19cc487/egs/librispeech/asr/simple_v1/train.py#L177-L181
>
> Noted here is a warning about torch version, should I upgrade PyTorch
> version?
>
> /ceph-hw/lhotse/lhotse/augmentation/torchaudio.py:13: UserWarning: Torchaudio SoX effects chains are only introduced in version 0.7 - please upgrade your PyTorch to 1.7+ and torchaudio to 0.7+ to use them.
>   warnings.warn('Torchaudio SoX effects chains are only introduced in version 0.7 - '
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/lhotse-speech/lhotse/issues/125>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ADZRKQDGFHWNXJQH5O6PSD3SPT4ZNANCNFSM4TULO2MQ>
> .
>

thanks!

Resolving patch:
diff --git a/lhotse/dataset/speech_recognition.py b/lhotse/dataset/speech_recognition.py
--- a/lhotse/dataset/speech_recognition.py
+++ b/lhotse/dataset/speech_recognition.py
@@ -7,7 +7,7 @@
 from torch.utils.data.dataloader import DataLoader
 
 from lhotse.cut import AnyCut, CutSet
-from lhotse.utils import Seconds
+from lhotse.utils import Seconds, compute_num_frames
 
 EPS = 1e-8
 
@@ -206,6 +206,24 @@ def __next__(self) -> Dict[str, Union[torch.Tensor, List[str]]]:
         # Get a tensor with batched feature matrices, shape (B, T, F)
         features = _collate_features(cuts)
 
+        def asserted_num_frames(start: Seconds, duration: Seconds, frame_shift: Seconds) -> int:
+            """
+            This closure with compute the num_frames, correct off-by-one errors in edge cases,
+            and assert that the supervision does not exceed the feature matrix temporal dimension.
+            """
+            offset = compute_num_frames(start, frame_shift=frame_shift)
+            num_frames = compute_num_frames(duration, frame_shift=frame_shift)
+            diff = features.shape[1] - (offset + num_frames)
+            # Note: we tolerate off-by-ones because some mixed cuts could have one frame more
+            # than their duration suggests (we will try to change this eventually).
+            if diff == -1:
+                num_frames -= 1
+            assert offset + num_frames <= features.shape[1], \
+                f"Unexpected num_frames ({offset + num_frames}) exceeding features time dimension for a supervision " \
+                f"({features.shape[1]}) when constructing a batch; please report this in Lhotse's GitHub issues, " \
+                "ideally providing the Cut data that triggered this."
+            return num_frames
+
         return {
             'features': features,
             'supervisions': default_collate([
@@ -213,8 +231,8 @@ def __next__(self) -> Dict[str, Union[torch.Tensor, List[str]]]:
                     'cut_id': cut.id,
                     'sequence_idx': sequence_idx,
                     'text': supervision.text,
-                    'start_frame': round(supervision.start / cut.frame_shift),
-                    'num_frames': round(supervision.duration / cut.frame_shift),
+                    'start_frame': compute_num_frames(supervision.start, cut.frame_shift),
+                    'num_frames': asserted_num_frames(supervision.start, supervision.duration, cut.frame_shift),
                 }
                 for sequence_idx, cut in enumerate(cuts)
                 for supervision in cut.supervisions


Test patch: