Timestamps in AutomaticSpeechRecognitionPipeline not aligned in sample space
Author: lemswasabi
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.17.0.dev0
- Platform: Linux-5.10.98-1-MANJARO-x86_64-with-glibc2.33
- Python version: 3.9.8
- PyTorch version (GPU?): 1.10.2+cpu (False)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.4.0 (cpu)
- Jax version: 0.3.1
- JaxLib version: 0.3.0
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@Narsil
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- T5, BART, Marian, Pegasus, EncoderDecoder: @patrickvonplaten
- Blenderbot, MBART: @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet, BigBird: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder, UniSpeech, UniSpeechSAT, SEW, SEW-D, Speech2Text: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten @narsil
- Tokenizers: @SaulLu
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Issue

Generating timestamps in the `AutomaticSpeechRecognitionPipeline` does not match the timestamps generated from `Wav2Vec2CTCTokenizer.decode()`. The timestamps from the pipeline are exceeding the duration of the audio signal because of the strides.

## Expected behavior

Generating timestamps using the pipeline gives the following prediction IDs and offsets:

```python
pred_ids=array([ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0, 12, 14, 14,  0,  0,
        0, 22, 11,  0,  7,  4, 15, 16,  0,  0,  4, 17,  5,  7,  7,  4,  4,
       14, 14, 18, 18, 15,  4,  4,  0,  7,  5,  0,  0, 13,  0,  9,  0,  0,
        0,  8,  0, 11, 11,  0,  0,  0, 27,  4,  4,  0, 23,  0, 16,  0,  5,
        7,  7,  0, 25, 25,  0,  0, 22,  7,  7, 11,  0,  0,  0, 10, 10,  0,
        0,  8,  0,  5,  5,  0,  0, 19, 19,  0,  4,  4,  0,  0,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  5,  0,  6,  4,  4,  0, 25,  0,
       14, 14, 16,  0,  0,  0,  0, 10,  9,  9,  0,  0,  0,  0, 25,  0,  0,
        0,  0,  0, 26, 26, 16, 12, 12,  0,  0,  0, 19,  0,  5,  0,  8,  8,
        4,  4, 27,  0, 16,  0,  0,  4,  4,  0,  3,  3,  0,  0,  0,  0,  0,
        0,  0,  0,  4,  4, 17, 11, 11, 13,  0, 13, 11, 14, 16, 16,  0,  6,
        5,  6,  6,  4,  0,  5,  5, 16, 16,  0,  0,  7, 14,  0,  0,  4,  4,
       12,  5,  0,  0,  0, 26,  0, 13, 14,  0,  0,  0,  0, 23,  0,  0, 21,
       11, 11, 11,  0,  5,  5,  5,  7,  7,  0,  8,  0,  4,  4,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
        0, 12, 21, 11, 11, 11,  0,  4,  4,  0,  0, 12, 11, 11,  0,  0,  0,
        0,  0,  7,  7,  5,  5,  0,  0,  0,  0, 23,  0,  0,  8,  0,  0,  4,
        4,  0,  0,  0,  0, 12,  5,  0,  0,  4,  4, 10,  0,  0, 24, 14, 14,
        0,  7,  7,  0,  8,  8, 10, 10,  0,  0,  0,  0,  0,  0,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0, 26,  0,  5,  0,  0,  0, 20,  0,  5,  0,
        0,  5,  5,  0,  0, 19, 16, 16,  0,  6,  6, 19,  0,  5,  6,  6,  4,
        4,  0, 14, 18, 18, 15,  4,  4,  0,  0, 25, 25,  0,  5,  4,  4, 19,
       16,  8,  8,  0,  8,  8,  4,  4,  0, 23, 23,  0, 14, 17, 17,  0,  0,
       17,  0,  5,  6,  6,  4,  4])
       
decoded['word_offsets']=[{'word': 'dofir', 'start_offset': 12, 'end_offset': 22}, {'word': 'hu', 'start_offset': 23, 'end_offset': 25}, {'word': 'mer', 'start_offset': 28, 'end_offset': 32
}, {'word': 'och', 'start_offset': 34, 'end_offset': 39}, {'word': 'relativ', 'start_offset': 42, 'end_offset': 60}, {'word': 'kuerzfristeg', 'start_offset': 63, 'end_offset': 94}, {'word'
: 'en', 'start_offset': 128, 'end_offset': 131}, {'word': 'zousazbudget', 'start_offset': 134, 'end_offset': 170}, {'word': 'vu', 'start_offset': 172, 'end_offset': 175}, {'word': '<unk>',
 'start_offset': 180, 'end_offset': 182}, {'word': 'milliounen', 'start_offset': 192, 'end_offset': 207}, {'word': 'euro', 'start_offset': 209, 'end_offset': 217}, {'word': 'deblokéiert', 
'start_offset': 221, 'end_offset': 249}, {'word': 'déi', 'start_offset': 273, 'end_offset': 278}, {'word': 'direkt', 'start_offset': 283, 'end_offset': 303}, {'word': 'de', 'start_offset': 311, 'end_offset': 313}, 
{'word': 'sportsbeweegungen', 'start_offset': 317, 'end_offset': 492}, {'word': 'och', 'start_offset': 495, 'end_offset': 499}, {'word': 'ze', 'start_offset': 503, 'end_offset': 507}, {'word': 'gutt', 'start_offset': 509, 'end_offset': 516}, 
{'word': 'kommen', 'start_offset': 519, 'end_offset': 532}]
```

However, the following is computed using `Wav2Vec2CTCTokenizer.decode()` and these offsets are expected:

```python
pred_ids=tensor([ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0, 12, 14, 14,  0,  0,  0,
        22, 11,  0,  7,  4, 15, 16,  0,  0,  4, 17,  5,  7,  7,  4,  0, 14, 14,
        18, 18, 15,  4,  4,  0,  7,  5,  0,  0, 13,  0,  9,  0,  0,  0,  8,  0,
        11, 11,  0,  0,  0, 27,  4,  4,  0, 23,  0, 16,  0,  5,  7,  7,  0, 25,
        25,  0,  0, 22,  7,  7, 11,  0,  0,  0, 10, 10,  0,  0,  8,  0,  5,  5,
         0,  0, 19, 19,  0,  4,  4,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
         0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
         0,  0,  5,  0,  6,  4,  4,  0, 25,  0, 14, 14, 16,  0,  0,  0,  0, 10,
         9,  9,  0,  0,  0,  0, 25,  0,  0,  0,  0,  0, 26, 26, 16, 12, 12,  0,
         0,  0, 19,  0,  5,  0,  8,  8,  4,  4, 27,  0, 16,  0,  0,  4,  4,  0,
         3,  3,  0,  0,  0,  0,  0,  0,  0,  0,  4,  4, 17, 11, 11, 13,  0, 13,
        11, 14, 16, 16,  0,  6,  5,  6,  6,  4,  0,  5,  5, 16, 16,  0,  0,  7,
        14,  0,  0,  4,  4, 12,  5,  0,  0,  0, 26,  0, 13, 14,  0,  0,  0,  0,
        23,  0,  0, 21, 11, 11, 11,  0,  5,  5,  5,  7,  7,  0,  8,  0,  4,  4,
         0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
         0,  0,  0, 12, 21, 11, 11, 11,  0,  4,  4,  0,  0, 12, 11, 11,  0,  0,
         0,  0,  0,  7,  7,  5,  5,  0,  0,  0,  0, 23,  0,  0,  8,  0,  0,  4,
         4,  0,  0,  0,  0, 12,  5,  0,  0,  4,  4, 10,  0,  0, 24, 14, 14,  0,
         7,  7,  0,  8,  8, 10, 10,  0,  0,  0, 26,  5,  5,  0,  0,  0, 20,  5,
         0,  0,  0,  5,  0,  0, 19,  0, 16, 16,  6,  6, 19, 19,  5,  5,  6,  6,
         4,  0, 14,  0, 18, 15, 15,  4,  4,  0,  0, 25,  0, 16,  0,  4,  4, 19,
        16,  8,  0,  0,  8,  0,  4,  0,  0, 23,  0, 14,  0, 17,  0,  0,  0, 17,
         5,  5,  6,  6,  4,  4])
         
word_offsets=[{'word': 'dofir', 'start_offset': 12, 'end_offset': 22}, {'word': 'hu', 'start_offset': 23, 'end_offset': 25}, {'word': 'mer', 'start_offset': 28, 'end_offset': 32},  {'word': 'och', 'start_offset': 34, 'end_offset': 39}, {'word': 'relativ', 'start_offset': 42, 'end_offset': 60}, {'word': 'kuerzfristeg', 'start_offset': 63, 'end_offset': 94},  {'word': 'en', 'start_offset': 128, 'end_offset': 131}, {'word': 'zousazbudget', 'start_offset': 134, 'end_offset': 170},  {'word': 'vu', 'start_offset': 172, 'end_offset': 175}, {'word': '<unk>', 'start_offset': 180, 'end_offset': 182}, {'word': 'milliounen', 'start_offset': 192, 'end_offset': 207},  {'word': 'euro', 'start_offset': 209, 'end_offset': 217}, {'word': 'deblokéiert', 'start_offset': 221, 'end_offset': 249},  {'word': 'déi', 'start_offset': 273, 'end_offset': 278}, {'word': 'direkt', 'start_offset': 283, 'end_offset': 303},  {'word': 'de', 'start_offset': 311, 'end_offset': 313}, {'word': 'sportsbeweegungen', 'start_offset': 317, 'end_offset': 360}, {'word': 'och', 'start_offset': 362, 'end_offset': 367},  {'word': 'zu', 'start_offset': 371, 'end_offset': 374}, {'word': 'gutt', 'start_offset': 377, 'end_offset': 383},  {'word': 'kommen', 'start_offset': 387, 'end_offset': 400}]
```

## Potential Fix

A fix could be removing the strides instead of filling them with `first_letter` and `last_letter` in `apply_stride()`:

```python
def apply_stride(tokens, stride):

    input_n, left, right = rescale_stride(tokens, stride)[0]
    left_token = left
    right_token = input_n - right

    return tokens[:, left_token:right_token]
```

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Hi @lemswasabi ,

Thanks for the report. We unfortunately cannot drop the stride as when there's batching involved, the tensors cannot be of different shapes. We can however keep track of the stride and fix the timestamps.

I'll submit a patch tomorrow probably.

Hi @Narsil,

Thanks for having a look at it.

@lemswasabi , in the end I aligned most of the code closer to what `ctc_with_lm` did (dropping the stride in  the postprocess method as I said, we can't in _forward).

There could still be some slight issue with timings, so the PR is not necessarily finished

@Narsil, I understand that dropping the stride has to be done in the postprocess method. I tested your patch and it works as expected now.

Great ! Please let us know if there's any further issue, ctc_with_lm coming right up :D