Seamless M4T-v2 Inference bug when using chunk_length_s parameter
Author: asusdisciple
Pull request: no

### System Info

Ubuntu 22
Python 3.12
Latest Transformers

### Who can help?

@Narsil 
@SunMarc 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I know the chunk parameter is experimental, but it completely falls apart in case of m4t-v2. It works FAR better in
Whisper. Would it be possible to disable it or print a big fat warning when it is used? To reproduce:

1. Use seamless m4t-v2 model in a transformers pipeline.
2. Use task transcribe and ASR for any language you like.
3. Use an Audio which is longer than 30 seconds.
4. Set the `chunk_length_s` parameter to any value, e.g. 30
5. Compare score to transcription without setting `chunk_length_s`.

The scores in terms of WER % error are usually 4-5 times worse than you would expect. It is basically unusable at the moment, which make m4t-v2 not usable for ASR, except your files are very short.

### Expected behavior

Just chunk it and get approxximateky the same score if possible. I dont know why it fails so hard in comparison to whisper which works pretty good.

Comments:
cc @ylacombe 

Hey @asusdisciple, thanks for opening this issue. I'll look into it, but first, could you send a snippet which show how you use the pipeline ?

Sure. Here is how I initialize the pipeline:
```

obj = pipeline(task="automatic-speech-recognition",
               model="facebook/seamless-m4t-v2-large",
               device=device,
               chunk_length_s=30
               batch_size=64,
               generate_kwargs={"num_beams": 1,
                                "temperature": 0,
                                "task": "transcribe",
                                "do_sample": True
                                }
             )
  
```

And this is how I transcribe it with the given language:

```
tmp = model(audiopath,
                    generate_kwargs={"tgt_lang": lang}
                    )
```

Just remember the audiofile should be longer than 10s. I already see this issue with files which are 20s long but 1min probably works better. I am even unsure why it happens with 20s audiofiles because setting chunking to 30s should not affect those right?

Thanks @asusdisciple, note that the chunk parameter for seq2seq models is highly experimental as pointed out in #20104.

@Narsil, it could be good to have your opinion on this since you added the algorithm. Could you give an explanation of how it works and how to improve it ? Many thanks

Could you share an actual file that triggers it, the expected output, and the actual output with chunking ? Having a fully complete reproducer really helps narrow down these kind of things.

Chunking works like the following. Chunk every audio into chunks of length `chunk_length_s` with overlapping (1/6 * chunk_length_s by default). We run the model on each of those, and then stitch the results by finding the maximum overlapping output, and fusing those strings together.

Many things could be at play, it would be easier to get good hints with the actual issue.

-> seamless m4t might not be trained on long audio, and having position_ids or something, making it harder for it to run on those extra chunks. Shouldn't affect chunked with chunks < trained length tho.
-> There could be an issue with the chunking and/or stitching code.
-> On whisper, without issue, the fact that the chunking ahppens arbitrarily in audio means that the model is hallucinating more, on both ends of the chunks, meaning the stitching will include the hallucination (since it's a maximally overlapping string). No good solution for that one, either try to precut audio along silences if they exist, or find a way to fine-tune the model to handle those. If it happens for whisper, it can happen here (although the degradation for whisper is a fraction of a percent of WER, as it also sometimes improves baseline whisper). Maybe S4t is more sensitive to it.


For whisper I can state that on my FLEURS benchmark, where I concatenated files to longer version (5 and 30min) WER dropped by 10% for all HF implementations. Faster-Whisper (Repo) however performed exactly the same as on small files when I used the parameter `condition_on_previous_text=False`. If I used Faster-Whisper with this parameter on `True` the WER skyrocketed to 200%. So maybe this plays a role since you can not turn it off in HF.

As for seemless m4t-v2 here is a minimal reproducer with ground_truth text and the audio file, which was concatenated from a few sentences in FLEURS. WER 66%. English Language. Repetitions were cleaned. The audio file is in the uploads: 
[17837018222808398366.wav.zip](https://github.com/huggingface/transformers/files/13920168/17837018222808398366.wav.zip)

```
from transformers import pipeline
import jiwer

ground_truth = "Police superintendent Chandra Shekhar Solanki said the accused appeared in court with covered faces. During the struggle for independence organised by the Mau movement, a peaceful gathering in the town resulted in the killing of the paramount chief Tupua Tamasese Lealofi III. Martelly swore in a new Provisional Electoral Council (CEP) of nine members yesterday. Europe is a continent that is relatively small but with many independent countries. Under normal circumstances, travelling through multiple countries would mean having to go through visa applications and passport control multiple times. Their thermal behavior is not as steady as large caves on Earth that often maintain a fairly constant temperature, but it is consistent with these being deep holes in the ground,' said Glen Cushing of the United States Geological Survey (USGS) Astrogeology Team and of Northern Arizona University located in Flagstaff, Arizona. Sleep interruption is the process of purposefully awakening during your normal sleep period and falling asleep a short time later (10–60 minutes). Most of the buildings on the edges of the complex have been rebuilt in order to give tourists a better idea of how they originally appeared. Angel (2006), explains the Continuum approach as a method being used to help organizations reach a higher level of performance. Due to the underwater topology the return flow is concentrated at a few deeper sections, and a fast current to deep water may form there. towards the end of the Middle Ages western Europe began to develop their own style. one of the biggest developments of the time as a result of the crusades people began to use buttons to fasten clothing. NextGen is a system the FAA claims would allow aircraft to fly shorter routes and save millions of gallons of fuel each year and cut carbon emissions. A full 20 percent of the water that pours out of the planet's rivers into the oceans comes from the Amazon. So, it is likely that the notation was added simply as a label. He was greeted by Singapore's Deputy Prime Minister Wong Kan Seng and discussed trade and terrorism issues with the Singapore Prime Minister Lee Hsien Loong. The disease is carried by pigs, which then migrates to humans through mosquitos. Elements like calcium and potassium are considered metals. Of course, there are also metals like silver and gold. Field trips are a large part of any classroom. Quite often a teacher would love to take her students places to which a bus trip is not an option. People now write messages on computer screens, never having to come close to a sharpener. One of the most common methods used to illustrate the importance of socialization is to draw upon the few unfortunate cases of children who were, through neglect, misfortune, or wilful abuse, not socialized by adults while they were growing up. He added that 'they should not, however, be asked to take on obligations that go beyond their development stage, responsibility and capabilities.' Ironing damp clothes can help them dry. Many hotels have an iron and ironing board available for loan, even if one is not present in the room. Apia is the capital of Samoa. The town is on the island of Upolu and has a population of just under 40,000. A curry is a dish based on herbs and spices, together with either meat or vegetables. Four skiers in the women's sitting group failed to finish their runs, and 45 of the 117 total skiers in the Giant Slalom failed to rank in the race. The debate was sparked by controversy over spending on relief and reconstruction in the wake Hurricane Katrina; which some fiscal conservatives have humorously labeled 'Bush's New Orleans Deal.' In just two weeks the Americans and Free French forces had liberated southern France and were turning towards Germany. The Great Pyramid at Giza is the only one of the seven wonders that is still standing today. The two compounds react with one another to form crystals that may block kidney function, researchers at the university said. Most interpretations of technological determinism share two general ideas: that the development of technology itself follows a path largely beyond cultural or political influence, and that technology in turn has 'effects' on societies that are inherent, rather than socially conditioned."


obj = pipeline(task="automatic-speech-recognition",
               model="facebook/seamless-m4t-v2-large",
               device="cuda:0",
               chunk_length_s=30,  # NEVER USE THIS PARAMETER, DEGRADES SCORE FOR LONG AUDIO
               batch_size=4,
               generate_kwargs={"num_beams": 1,
                                "temperature": 0,
                                "task": "transcribe",
                                "do_sample": True
                                }
               )


audiopath = "..."

pred = obj(audiopath,
          generate_kwargs={"tgt_lang": "eng"}
                    )


err = round(jiwer.process_words(ground_truth, pred["text"]).wer, 2)
print(err)

```






First of all, there is a BIG warning that this is experimental:

https://github.com/huggingface/transformers/pull/20104/files#diff-ccc8e98fcaa81fdf6317a652438a309bcede0bbe336774288c2fcf91d9f11082R291-R296

I played around a bit, I found a bug in ASR pipeline, although it won't fix your issues, since it really seems that it's the model that has a lot of issues with hallucinations.

[simple.webm](https://github.com/huggingface/transformers/assets/204321/a5f111dc-cfb8-4b62-bf9e-6484cd162135)

Here is the cut version (30s).

Without chunking the output is
```
'Police Superintendent Shandra Shankar Solanki said the accused appeared in court with their faces covered during th
e independence struggle organized by the Maoist movement, which resulted in the killing of the Supreme Leader Tupa T
upa Tupa Tsai.'
```

With chunking (it will chunk  at 25s and rerun the missing 5 with overlap)

```
'Police Superintendent Shandra Shankar Solanki said the accused appeared in court with their faces covered during the independence struggle organized by the Maoist movement, which resulted in the killing of the Supreme Leader Tupa Tupa Tupa Tsai. The newly elected president of the United States, Martin Luther King, Jr., was sworn in on Tuesday, July 9, at the age of 93.'
```

The second sentence seems like pretty bad hallucination from the actual audio. 

@ylacombe is this model known for hallucinating so much ?



Thanks for investigating @Narsil ! So this is the result after fixing the bug you found ?

> @ylacombe is this model known for hallucinating so much?

It is, the model principal usage is translation. Using it in an ASR settings is likely to produce big hallucinations. Moreover, from my own usage, it is not really good with short audios.



> So this is the result after fixing the bug you found ?

The bugfix doesn't really help that much tbh (so no the example doesn't include it, the bug is that we don't reinclude the `stride` parameter when the model is `seq2seq` withing the forward method, meaning we don't use the custom stitching for seq2seq, but given the high hallucination rate, it doesn't really help, it's actually worse on the entire file in this particular instance).

Are there any benchmarks on hallucinations in m4t and benchmarks in general? In the paper the performance for ASR and M4T-v2 is almost double as good as Whisper-v2. In my own reproduction on Fleurs I was able to reproduce the paper results. 

Also I know this feature is experimental, just wanted to let you guys now! I really appreciate the work you do and I know more than a few people who use huggingface implementations just for the sake of chunking. So I think its an important feature and even if the behaviour can't be changed, knowledge about things like this is key right? :) 

![Screenshot from 2024-01-19 10-57-10](https://github.com/huggingface/transformers/assets/138434950/b797a31e-a4f1-4dfd-a50d-4c4afcc17fe0)


Hello @asusdisciple and @Narsil, circling back on this issue, I've used jiwer visualisation tools to pinpoint where the transcription fails. (`print(jiwer.visualize_alignment(jiwer.process_words(ground_truth.lower(), pred["text"].lower())))`)

Turns out M4T doesn't hallucinate that much


```python
sentence 1
REF: police superintendent chandra shekhar  solanki said the accused appeared in court with covered faces. during the struggle for independence organised by the mau movement, a peaceful gathering in the town resulted in the killing of the paramount chief ***** ******** *** ******** ***** ** * *** *********** ********* ******* tupua tamasese lealofi      iii. martelly swore in a new provisional electoral council (cep) of nine members yesterday. europe is a continent that is relatively small but with many independent countries. under normal circumstances, travelling through multiple countries would mean having to go through visa applications and passport ********* ****** ************* ********** ******* ******** ********* ***** **** ****** ** ** ******* **** ************ *** ******** control multiple times. their thermal  behavior is not as steady as large caves on earth that often maintain a fairly constant temperature, but it is consistent with these being deep holes in the ground,' said glen cushing of the united states geological survey *** **** ********** **** ***** ***** **** ***** ** *** ******* **** ***** ******* ** *** ****** (usgs) astrogeology   team and  of northern arizona university located in flagstaff, arizona. sleep interruption is the process of ************ purposefully awakening during your normal sleep period and falling asleep a short time  later (10–60 minutes). most of the buildings  on the edges   of the complex have been rebuilt in order to give tourists a better idea of how they originally appeared. angel (2006), explains the continuum approach as a method being used to help organizations   reach a higher  level of performance. due to the underwater topology the return flow is concentrated at a    few deeper sections, and a fast current to deep water may form there. towards the end of the middle ages western europe began to develop their own style. one of the biggest developments of the time as a result of the crusades people began to use buttons to fasten clothing. nextgen is a system the faa claims would allow aircraft to fly shorter routes and save millions of gallons of fuel each year and cut carbon emissions. a full 20 percent of the water   that pours out of the planet's rivers into the oceans comes from the amazon. so, it is likely that the notation  was added   simply as   a  label.     he was greeted       by singapore's  deputy prime minister wong  kan seng and discussed  trade        and terrorism issues with the singapore prime minister lee hsien loong. the disease is carried by pigs, which then migrates to humans through  mosquitos. elements like calcium     and potassium are considered metals.         of course, there are also metals like silver and gold. field trips are a large part of           any classroom. quite often a teacher would love to take her students places to which a bus trip is not an option. people now write messages on computer screens, never having to come close to a sharpener. one of the most common methods used to illustrate the importance of socialization is to draw upon the few unfortunate cases of children who were, through neglect, misfortune, or  wilful abuse, not socialized by ** adults while they were growing up.  he added that 'they should not, however, be asked to take on obligations that go beyond their development stage, responsibility and capabilities.' ironing damp clothes can help them dry. many hotels have an iron and ironing board available for loan, even if one is not present in the room. apia is the capital of samoa. the town is  on the island of upolu and has a population of just under 40,000.       a curry is a dish based on herbs and spices, together with either meat or vegetables. four skiers in the women's sitting group failed to finish their runs, and 45 of the 117 total skiers in the giant slalom failed to rank in the race. the debate was sparked by controversy over spending on relief and reconstruction in the wake ** hurricane katrina; which some fiscal conservatives have humorously labeled 'bush's new orleans deal.'  in just two weeks the americans and free french forces had liberated southern france and were turning towards germany. the great pyramid at giza is the only one of the seven wonders that is still standing today. the two compounds react with one another to form crystals that may block kidney function, researchers at the university said. most interpretations of technological determinism share two general ***** **** *** *********** ** ********** ****** ******* * **** ******* ****** ******** ** ********* ********* **** *************** ** ************* *********** ***** *** ******* ideas: that the development of technology itself follows a path largely beyond cultural or political influence, and that technology in turn has 'effects' on societies that are inherent, rather than socially conditioned.
HYP: police superintendent shandra shankar solansky said the accused appeared in court with covered  faces during the struggle for independence organized by the mao  movement a peaceful gathering in the town resulted in the killing of *** paramount chief tuppa tupaczai iii martelli swore in a new provisional electoral council    of     nine members yesterday martelli swore in a new provisional electoral council ***** of nine members yesterday. europe is a continent that is relatively small but with many independent countries. under normal circumstances, travelling through multiple countries would mean having to go through visa applications and passport controls. normal circumstances travelling through multiple countries would mean having to go through visa applications and passport control multiple  times their thermal behaviour is not as steady as large caves on earth that often maintain a fairly constant  temperature but ** is consistent with these being deep holes in the   ground said glen cushing of the united states geological survey but it's consistent with these being deep holes in the ground, said glenn cushing of the united states   geological survey and the northern arizona university located in flagstaff, arizona. ***** ************ ** the process of deliberately       waking        up during your normal sleep period and falling asleep a short time later. ****** ********* **** ** the      best way  to    do this  is ******* **** **** ******* ** ***** to  use ******** a ****** **** ** *** **** ********** ********* ***** ******* ******** *** continuum approach ** * ****** ***** **** to help organizations achieve * higher levels of performance. *** ** the       next      gen *** ****** **** is ************ ** a system   that ********* *** * **** ******* ** **** ***** *** **** ****** ******* the *** ** *** ****** **** ******* ****** ***** ** ******* ***** *** ****** *** ** *** ******* ************ ** *** **** ** * ****** ** *** ******** ****** ***** ** *** ******* ** ****** ********* ******* ** * ****** *** faa claims would allow aircraft to fly shorter routes and save millions of gallons of fuel each year and cut carbon emissions. * **** ** ******* ** the   faa claims ***** *** ** *** ******** ****** **** *** ****** ***** **** *** ******* *** ** ** ****** that the   system will allow aircraft to fly shorter routes and    save millions          of gallons    of     fuel each year **** and       cut carbon emissions. ********* ****** **** the ********* ***** ******** *** ***** ****** *** disease is carried by pigs, which then migrates to humans through mosquitoes.      the most  common methods      used  to illustrate     the importance ******* ***** *** **** ****** **** ****** *** ***** ***** ***** *** * ***** **** of socialization       are: ***** ***** * ******* ***** **** ** **** *** ******** ****** ** ***** * *** **** ** *** ** ******* ****** *** ***** ******** ** ******** ******** ***** ****** ** **** ***** ** * ********** one of the most common methods used to illustrate the importance of socialization is to draw   on the *** unfortunate cases of children who  were through neglect,  misfortune or willful  abuse not socialized by an  adult while they were growing up. the  city   of ***** ****** **** ******** ** ***** ** **** ** *********** **** ** ****** ***** *********** ****** ************** *** ************** ******* **** ******* *** **** **** **** **** ****** **** ** **** *** ******* ***** ********* *** ***** **** ** *** ** *** ******* ** *** ***** apia is the capital of samoa. the city of upu  is   home to ***** *** *** a population of just under  40,000 people. ***** ** * **** ***** ** ***** *** ******* ******** **** ****** **** ** *********** **** ****** ** the ******* ******* ***** ****** ** ****** ***** ***** *** ** ** *** *** ***** ****** ** *** ***** ****** ****** ** **** ** *** ***** *** debate was sparked by controversy over spending on relief and reconstruction in the wake of hurricane katrina, which some fiscal conservatives have humorously labeled  bush's new orleans  deal. its **** *** ***** *** ********* *** **** ****** forces had liberated southern france and were turning towards  germany the great pyramid at giza is *** **** one of the seven wonders that is still standing  today the two compounds react with one another to form crystals that may block kidney  function researchers at the university  said most interpretations of technological determinism share two general ideas that the development of technology itself follows a path largely beyond cultural or political influence most interpretations of technological determinism share two general ideas: that the development of technology itself follows a path largely beyond cultural or political  influence and that technology in turn has   effects on   society that are  inherent rather than socially conditioned.
                                 S       S        S                                                      S                                              S          S         S                                                               D                     I        I   I        I     I  I I   I           I         I       I     S        S       S         S        S                                                  D                                                                                                                                                                                                                                                         I      I             I          I       I        I         I     I    I      I  I  I       I    I            I   I        I                       S                       S                                                                                           S      D                                                         S                                                            I    I          I    I     I     I    I     I  I   I       I    I     I       I  I   I      I      S            S      S       S                                                                D            D  D                           I            S         S                                                                      S      D         D    D  D             S   S   S     S    S   S       D    D    D       D  D     D       S        D        D    D  D   D    D          D         D     D       D        D   D                     D D      D     D    D                             S D             S                   D  D              S        S   D      D    D               D  D        S      S         D   D D    D       D  D    D     D   D    D      D       D       D  D   D      D    D       D      D     D  D       D     D   D      D   D  D   D       D            D  D   D    D  D D      D  D   D        D      D     D  D   D       D  D      D         D       D  D D      D   D                                                                                                                                D    D  D       D  D         S      S     D   D  D   D        D      D    D   D      D     D    D   D       D   D  D  D      D                 S    S     S        S  S   S       S      S   S       S        S           S       S     S        S    S    S    D             S      S          S         D      D    D             D     D        D   D     D      D   D                                                                             S        S    S       S       S         S   S          S       S          S       D     D   D    D      D    D      D   D     D     D     D   D D     D    D                S          S     D     D D       D     D    D  D    D   D        D      D  D     D D   D    D  D   D  D       D      D   D     D        D  D        D        D     D      D  D    D     D  D D          D                                                                                                 S       D                                       S                            S          S      S                    I      S                               S     S    S     D      D    D        D  D     D  D    D  D           D    D  D      D     D           D      D              D   D              D       D    D       D   D    D    D    D    D      D    D  D    D   D       D     D         D   D     D    D  D   D  D   D       D  D   D     D                                      S  S   S   S      S  S     D   D   D                                  S       S     D  D D    D     D  D     D   D       D        D    D      D    D  D           D    D      D  D           D       D     D      D  D      D     D     D   D  D  D   D   D     D      D  D   D     D      D      D  D    D  D   D     D   D                                                                                           I                  S                                                               S                  S   S    D   D     D   D         D   D    D      D                                                                      S                                D    D                                                      S                                                                                         S                                   S                                                                         I    I   I           I  I          I      I       I I    I       I      I        I  I         I         I    I               I  I             I           I     I   I       I                                                                                                               S                                         S            S                  S                                  

number of sentences: 1
substitutions=99 deletions=280 insertions=72 hits=333

mer=57.53%
wil=69.10%
wip=30.90%
wer=63.34%
```

It seems that the transcription works quite well but repeats a lot of sequence -> indicate that it should work better with a low `stride_length_s`.

I've tried with `stride_length_s=1`, and I already get a much better WER (45%).

```python
sentence 1
REF: police superintendent chandra shekhar  solanki said the accused appeared in court with covered faces. during the struggle for independence organised by the mau movement, a peaceful gathering in the town resulted in the killing of the paramount chief tupua tamasese lealofi     iii. martelly swore in a new provisional electoral council (cep) of nine members yesterday. europe is a continent that is relatively small but with many independent countries. under normal circumstances, travelling through multiple countries would mean having to go through visa applications and passport control multiple times. their thermal  behavior is not as steady as large caves on ****** *** ***** ** *** ******* earth that often **** ** ***** **** maintain          a fairly constant temperature, but it is consistent with these being deep holes in the ground,' said  glen cushing of the united states geological survey (usgs) astrogeology team and  of northern arizona university located in flagstaff, arizona. sleep interruption is the process of ************ purposefully awakening during your normal sleep period and falling asleep a short time  later (10–60 minutes). most of the buildings on the edges of the complex have been rebuilt in order to give tourists a better idea of how they originally appeared. angel (2006), explains the continuum approach as a method being used to help organizations reach a higher level of performance. due to the underwater topology the return flow is concentrated at a few deeper sections, and a fast current to deep water may form there. towards the end of the middle ages western europe began to develop their own style. one of the biggest developments of  the time as a result of the crusades ***** people began to use buttons to fasten ***** ******* clothing. nextgen is a system the *** faa claims **** *** ****** would allow aircraft to fly shorter routes and save millions of gallons of fuel each year and cut carbon emissions.       a full 20 percent of the water that pours out of the planet's rivers into the oceans comes from the amazon. so, it is likely that the notation was added simply as a label. he was greeted by singapore's deputy prime minister wong kan seng and discussed trade and terrorism issues with the singapore prime minister lee hsien loong. the disease is carried by pigs, which then migrates to humans through mosquitos. elements like calcium and potassium are considered metals. of course, there are also metals like silver and gold. field trips are a large part of any classroom. quite often a teacher would love to take her students places to which a bus trip is not an option. people now write messages on computer screens, never having to come close to a sharpener. one of the most common methods used to illustrate the importance of socialization is to draw upon the few unfortunate cases of children who were, through neglect, misfortune, or  wilful abuse, not socialized by ** adults while they were growing up. he added that 'they should not, however, be asked to take on obligations that go beyond their development stage, responsibility and capabilities.' ironing damp clothes can help them dry. many hotels have an iron and ironing board available for loan, even if one is not present in the  room. apia is the capital of samoa. the town is   on the island of upolu and has a population of just under 40,000.       a curry is a dish based on herbs and spices, together with either meat or vegetables. four skiers in the women's sitting group failed to finish their runs, and 45 of the 117 total skiers in the giant slalom failed to rank in the race. the debate was sparked by controversy over spending on relief and reconstruction in the wake ** hurricane katrina; which some fiscal conservatives have humorously labeled 'bush's new orleans deal.'  in just two weeks the americans and free french forces had liberated southern france and were turning towards germany. the great pyramid at giza is the only one of the seven wonders that is still standing today. the two compounds react with one another to form crystals that may block kidney function, researchers at the university said. most interpretations of technological determinism share two general ideas: that the development of technology itself follows a path largely beyond cultural or political ********* ******** ** ********* influence, and that technology in turn has 'effects' on societies that are inherent, rather than socially conditioned.
HYP: police superintendent shandra shankar solansky said the accused appeared in court with covered  faces during the struggle for independence organized by the mao  movement a peaceful gathering in the town resulted in the killing of *** paramount chief tuppa tupaczai     iii martelli ******** swore in a new provisional electoral council ***** of nine members  yesterday europe is a continent that is relatively small but with many independent countries. under normal circumstances, travelling through multiple countries would mean having to go through visa applications and passport control multiple times. their thermal behaviour is not as steady as large caves on earth. the study of the earth's crust   is often done in caves that      are relatively stable       in temperature, but it is consistent with these being deep holes in the  ground, said glenn cushing of the united states geological survey ****** ************ **** and the northern arizona university located in flagstaff, arizona. ***** ************ ** the process of deliberately       waking        up during your normal sleep period and falling asleep a short time later. ****** ********* **** ** *** ********* ** *** ***** ** *** ******* **** **** ******* ** ***** ** **** ******** * ****** **** ** *** **** ********** ********* ***** ******* ******** *** ********* ******** ** * ****** ***** **** ** **** ************* ***** * ****** ***** ** ************ due to the underwater topology the return flow is concentrated at a few deeper  sections and a fast current to deep water may form  there towards the end of the middle ages western europe began to develop their own  style one of the biggest developments of that time as a result of the crusades began people began to use buttons to fasten their clothes      next     gen is a system the the faa claims that the system  will allow aircraft to fly shorter routes and save millions of gallons of fuel each year and cut carbon emissions. discuss **** ** ******* ** *** ***** **** ***** *** ** *** ******** ****** **** *** ****** ***** **** *** ******* *** ** ** ****** **** *** ******** *** ***** ****** ** * ****** ** *** ******* ** *********** ****** ***** ******** **** *** **** *** ********* trade and terrorism issues with the singapore prime minister lee hsien   long the disease is carried by  pigs which then migrates to humans through mosquitoes elements like calcium and potassium are considered  metals of  course there are also metals like silver and  gold field trips are a large part of any  classroom quite often a teacher would love to take her students places to which a bus **** ** *** ** ******* ****** *** ***** ******** ** ******** ******** ***** ****** ** **** ***** ** * ********** one of the most common methods used to illustrate the importance of socialization is to draw   on the *** unfortunate cases of children who  were through neglect,  misfortune or willful  abuse not socialized by an  adult while they were growing up. ** ***** **** ***** ****** **** ******** ** ***** ** **** ** *********** **** ** ****** ***** *********** ****** ************** *** ************** ******* **** ******* *** **** **** **** **** ****** **** ** **** *** ******* ***** ********* *** ***** **** ** *** ** *** ******* ** the island **** ** *** ******* of    upu *** **** is home  to ****** ** ***** *** *** a population of just under  40,000 people. ***** ** * **** ***** ** ***** *** ******* ******** **** ****** **** ** *********** **** ****** ** the ******* ******* ***** ****** ** ****** ***** ***** *** ** ** *** *** ***** ****** ** *** ***** ****** ****** ** **** ** *** ***** *** debate was sparked by controversy over spending on relief and reconstruction in the wake of hurricane katrina, which some fiscal conservatives have humorously labeled  bush's new orleans  deal. its **** *** ***** *** ********* *** **** ****** forces had liberated southern france and were turning towards  germany the great pyramid at giza is *** **** one of the seven wonders that is still standing  today the two compounds react with one another to form crystals that may block kidney  function researchers at the university  said most interpretations of technological determinism share two general  ideas that the development of technology itself follows a path largely beyond cultural or political influence cultural or political  influence and that technology in turn has   effects on   society that are  inherent rather than socially conditioned.
                                 S       S        S                                                      S                                              S          S         S                                                               D                     S        S       S        S        D                                                  D                          S                                                                                                                                                                                                                                                                    S                                         I   I     I  I   I       I     S    S          I  I     I    I        S          S      S        S                                                                             S          S                                                     D            D    D       S                                                                D            D  D                           I            S         S                                                                      S      D         D    D  D   D         D  D   D     D  D   D       D    D    D       D  D     D  D    D        D D      D    D  D   D    D          D         D     D       D        D   D         D        D  D D      D     D    D  D    D             D     D D      D     D  D            D                                                                                        S                                                S                                                                                   S                                       S                                      I                                           I       I         S       S                   I               I   I      I     S                                                                                                                     S    D  D       D  D   D     D    D     D   D  D   D        D      D    D   D      D     D    D   D       D   D  D  D      D    D   D        D   D     D      D  D D      D  D   D       D  D           D      D     D        D    D   D    D   D         D                                                                             S                               S                                                S                                                          S          S                                           S                                              S                                                                                D  D   D  D       D      D   D     D        D  D        D        D     D      D  D    D     D  D D          D                                                                                                 S       D                                       S                            S          S      S                    I      S                              D     D    D     D      D    D        D  D     D  D    D  D           D    D  D      D     D           D      D              D   D              D       D    D       D   D    D    D    D    D      D    D  D    D   D       D     D         D   D     D    D  D   D  D   D       D  D          S    D  D   D       D         S   D    D       S   S      D  D     D   D   D                                  S       S     D  D D    D     D  D     D   D       D        D    D      D    D  D           D    D      D  D           D       D     D      D  D      D     D     D   D  D  D   D   D     D      D  D   D     D      D      D  D    D  D   D     D   D                                                                                           I                  S                                                               S                  S   S    D   D     D   D         D   D    D      D                                                                      S                                D    D                                                      S                                                                                         S                                   S                                                                          S                                                                                                       I        I  I         I          S                                         S            S                  S                                  

number of sentences: 1
substitutions=65 deletions=236 insertions=24 hits=411

mer=44.16%
wil=52.55%
wip=47.45%
wer=45.65%
```

~~It also seems that the quality of the transcription degrades the longer the audio is, despite using chunking. I'm indeed under the impression that there are more deletions as we reach the end of the audio.~~

This could maybe prevented by modifying a bit how the transcription algorithm `(_find_longest_common_sequence`) work.
At the moment, for each chunk, it checks the longest common sequence between the already concatened sequence (composed of the concatenation of each previous chunks) and the newest chunk, and add the newest chunk where it doesn't match.

As @Narsil indicates, some ways to improve it could maybe be linked to the stride parameters and the fact that we have some heuristics on how to stitch the different chunks


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@ylacombe May I ask where to set the parameter `stride_length_s` ? Is this part of the pipeline implementation or the model? I sometimes find it really confusing where to find and set parameter in hf haha

No worries, it get confusing at times, it's [here](https://huggingface.co/docs/transformers/v4.38.1/en/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline.chunk_length_s). 
Note that we generally decouple the underlying model methods which usually work on a single chunk,  from the post-processing that enables chunks of audio to be cut and reattached.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.