noamaanMulla-03/transformers-issues huggingface dataset:
Streaming support in automatic-speech-recognition pipeline
Author: CoderHam
Pull request: no

### Feature request

I'd like to request for the ability to stream back chunks of audio transcripts instead of having to wait for the entire audio to be processed. For real time use cases, it helps to be able to have small chunks of audio (10 seconds) and have their transcriptions returned as soon as they are available. Think live closed captioning with whisper.

This will open up the real world applications that we can use whisper and similar models for.

### Motivation

When I was working on transcribing long form videos of lectures/talks for a personal project I noticed that I need to wait for the entire audio to be transcribed before I can pass the same down to a downstream application.

**Details of my use case / project:**
* Take a video as input
* Play the video with closed captions using whisper
* During the video if the user asks a question, pause the lecture and use an LLM (plus some custom logic) to answer the users question.
* Resume the video after answering the user's question

I am concerned that breaking the audio into 10 second pieces will have lower throughput and underutilize the GPU. Is there something I have not considered here?

### Your contribution

I have dug through the code and narrowed the location of the bulk of the changes down to the [following lines](https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/automatic_speech_recognition.py#L681-L693) in the pipeline for "automatic-speech-recognition"

Even though the outputs are accumulated much earlier and appended to the `chunks` object, we only get the results once the audio is completely processed. Using a yield operator or a similar solution will allow us to return the processed chunks as soon as they are available.

I would be happy to make a PR for this is functionality folks are interested. For now I have a messy hack that updates the ASR pipeline to fit my needs.

Comments:
Hey! This seems to already be supported see this: #21196 does this not work for you problem? 

Shoot I wasn't aware this is already a feature. Yes I can make it work for me with the changes from that PR

Closing this ticket but if there are official docs for the above I'd appreciate a pointer to it. Thank you for your quick response 😁
batch_sampler with trainer.py would not set the epoch 
Author: rabeehk
Pull request: no
Labels: name: wontfix

Dear Huggingface team
If one uses batch_sampler instead of sampler then in trainer.py the part you set_epoch for the dataloader sampler it would not work, also in case one define custom sampler in certain application, like what I do, again in trainer.py it would not be called.
I was wondering if the codes can be more general to include these cases.

thanks.

Best
Rabeeh 

Comments:
Hi there.
To use custom sampling (either through a sampler or a batch sampler), users are expected to subsclass `Trainer` and override the `get_train_dataloader`/`get_eval_dataloader` methods to suit their needs.

Note that those changes might then not be compatible with distributed training/TPU training.

Hi there
yes, thats correct, but looking into train() method of trainer.py class,
the user needs to overwrite the whole train() function for such cases, and
this is just for setting the epoch for other type of sampler, it would be
very nice if the train() method allowed custom sampler. thanks.

On Sun, Dec 6, 2020 at 11:56 PM Sylvain Gugger <notifications@github.com>
wrote:

> Hi there.
> To use custom sampling (either through a sampler or a batch sampler),
> users are expected to subsclass Trainer and override the
> get_train_dataloader/get_eval_dataloader methods to suit their needs.
>
> Note that those changes might then not be compatible with distributed
> training/TPU training.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/8952#issuecomment-739579294>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABP4ZCGAIHTYBBBJI75TGKLSTQDYXANCNFSM4UPTHQNQ>
> .
>


This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

This seems to work for setting the epoch for the train_dataloader, without overwriting the train() function:

class SetEpochCallback(TrainerCallback):
&nbsp;&nbsp;&nbsp;&nbsp;def on_epoch_begin(self, args, state, control, **kwargs):
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;kwargs['train_dataloader'].batch_sampler.set_epoch(int(state.epoch))
translation main-class files to chinese
Author: jiaqiw09
Pull request: yes

# What does this PR do?

Part of  #26803 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? _not necessary_


## Who can review?
@stevhliu

Comments:
@stevhliu @statelesshz 

Hi, here is remaining translation work of main-class folder.

For these translation, I just keep many title or sub-title in origin format as they are just class attributes.

You can double check of these files.

Best

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27588). All of your documentation changes will be reflected on that endpoint.

@stevhliu 

I just update the review. Sorry that I forget to change these two line during checking.

Best
Translate  `en/model_doc` to JP
Author: rajveer43
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->


translate files

<!-- Remove if not applicable -->

Fixes #27556




## Who can review?

Documentation: @stevhliu and @MKhalusova



Comments:
Sure

> Hi, thanks for your PR and apologies for the delay! Several things:
> 
> 1. There is some overlap in models (`albert.md` to `autoformer.md`) here with [Translating `en/model_doc` docs to Japanese. #27401](https://github.com/huggingface/transformers/pull/27401) so lets remove those from this PR. You should have some way of claiming models from the list in [[i18n-JP] Translating `en/model_doc` docs to Japanese #27392](https://github.com/huggingface/transformers/issues/27392) to avoid future duplicate work.
> 2. This is also slightly out-of-date with the recent standardization to the model API docs introduced in [[Docs] Model_doc structure/clarity improvements #26876](https://github.com/huggingface/transformers/pull/26876). Would you please mind updating?
> 3. As you mentioned [here](https://github.com/huggingface/transformers/issues/27392#issuecomment-1803669747), try to work on ~10 at a time please. 🙏
> 4. You'll need to add the docs you're working on to the `toctree` to get the `build pr` CI test to pass.

I had already work on it weeks ago. so It slipped. Next PR would contain no more than 10 docs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27264). All of your documentation changes will be reflected on that endpoint.
fix roformer prepare_inputs_for_generation not return model_kwargs
Author: ylzz1997
Pull request: yes

Fixes Roformer prepare_inputs_for_generation not return model_kwargs

### Motivation
This bug causes the parameters passed into the **generate** function to be unable to be received by the model's **forward** function.
This PR is aimed at fixing this issue.

Comments:
> Thanks for opening a PR, the goal of this function is basically to expose the inputs that are used by the model. Which inputs in particular are you trying to unlock here?

I'll be using RoFormer with a multi-encoder, single-decoder structure to accommodate MultiModal Learning. I hope to include the `encoder_hidden_states` and `encoder_attention_mask` parameters in the model's inputs to enable **cross-attention** during generation.

The best way to do this would be to just overload this function in your codebase! Transformers is meant to be easy to work on top of, and this change is really not aligned with most models were we only return the args needed for generation, otherwise it's very messy! 

> The best way to do this would be to just overload this function in your codebase! Transformers is meant to be easy to work on top of, and this change is really not aligned with most models were we only return the args needed for generation, otherwise it's very messy!

However, when using RoFormer to form a common encoder-decoder structure, cross-attention is also necessary for feature input. I think the input of `encoder_hidden_states` features is essential.

Hey! I understand your frustration but I believe that if you are using your custom code you should be able to overwrite the `perpare_inputs_for_generation` function. Also I recommend you to checkout the `EncoderDecoderModel` which supports this I believe! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
deepspeed Integration error
Author: SingL3
Pull request: no

### System Info

N/A
See Bug report for DS [4194](https://github.com/microsoft/DeepSpeed/issues/4194) and [4533](https://github.com/microsoft/DeepSpeed/issues/4533)

### Who can help?

@pac

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

N/A

### Expected behavior

Deepspeed training successfully.

Comments:
hey. If you want use to help you according to the contribution guidelines could you write what is your issue here, with small reproducible snippet N 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
fix a typo in README_ru.md
Author: megavaz
Pull request: yes

# What does this PR do?

- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

change to correct Russian spelling

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add BeitBackbone
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR is part one of the bigger #25799. It adds the `BeitBackbone` class, making it possible to use this vision transformer as a backbone for downstream tasks, like depth estimation. 

To do:

- [x] make `out_indices` backwards compatible
- [x] check whether this class is compatible with Mask R-CNN

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I've added the conversion script to convert DPT + BEiT checkpoints (which is known as [DPT 3.1](https://github.com/isl-org/MiDaS/tree/master#setup)), as `BeitBackbone` now allows this. Was initially part of #25799

@amyeroberts all comments are addressed, thanks for the review!
Trigger corresponding pipeline tests if `tests/utils/tiny_model_summary.json` is modified
Author: ydshieh
Pull request: yes

# What does this PR do?

Trigger corresponding pipeline tests if `tests/utils/tiny_model_summary.json` is modified.

It happened several times that we merged a PR and pipeline testing failed, because the CI triggered on those PRs didn't cover all files (as `tests/utils/tiny_model_summary.json` is not a python file)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[docs] Quantization
Author: stevhliu
Pull request: no

I was going through the [Quantization](https://huggingface.co/docs/transformers/main/en/main_classes/quantization) docs and noticed there is a ton of cool content in here that I wouldn't have necessarily discovered if I wasn't reviewing another PR that was translating this page. 

The current API page feels cluttered, making it a bit difficult to lookup the API references (parameters, docstrings, etc.) of the different configurations. I think it'd be nicer and cleaner if we moved all of this content into its own separate doc in the Performance and scalability section and leave the API page like this:

```md
# Quantization

add link to quantization guide here

## AWQ

[[autodoc]] AwqConfig

## AutoGPTQ

[[autodoc]] GPTQConfig

## bitsandbytes

[[autodoc]] BitsAndBytesConfig
```

This way, the quantization docs would also be more visible on their own and we can avoid blurring the lines between our guides and API docs.

Would love to hear what you think @younesbelkada @SunMarc @ArthurZucker ! 🙂

Comments:
Sounds good to me 😉 

Thanks for the proposal @stevhliu ! That sounds great !
[docs] Quantization
Author: stevhliu
Pull request: yes

This PR resolves #27575 to clean up the Quantization API docs and create a new section for the existing content in the Performance and scalability section for more visibility.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I just saw this today @stevhliu, but I think that we table of content is not displayed correctly. It would be great to fix this since the doc is quite big. Thanks ! Feel free to merge the PR after that ! 

<img width="1287" alt="Screenshot 2023-11-27 at 12 18 43 PM" src="https://github.com/huggingface/transformers/assets/57196510/13e4248e-dd1b-43c2-874d-ed276e84244a">


Thanks y'all, pinging @ArthurZucker for a final review! The PR preview is still building I think, but the side table of contents should be fixed now which was caused by starting with the wrong section level in `## Quantization`
Refactoring Trainer, adds `save_only_model` arg and simplifying FSDP integration
Author: pacman100
Pull request: yes

# What does this PR do?
1. Bumps up the minimum Accelerate version to 0.21.0
2. Add `save_only_model` arg - This enables the feature request https://github.com/huggingface/transformers/issues/26706
3. Simplifies a lot of logic in FSDP:
     a. Currently, FSDP-XLA logic is custom in Trainer and normal FSDP is using the Accelerate's integration. There were many zombie code snippets related to normal FSDP. Cleaned those.
     b. Made it easier to train with FSDP. When using `FULL_STATE_DICT` setting, it should now save the model in transformers format using the default safetensors sharded format. This reduces the burden on users to later load, shard and save in safetensors format.
     c. Should fix https://github.com/huggingface/transformers/issues/27432 but don't have access to TPUs to test this.
     d. Fixes https://github.com/huggingface/transformers/issues/27166
     e. This is built upon the PR in Accelerate to simplify FSDP integration https://github.com/huggingface/accelerate/pull/2177. It should be merged first.



Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi, 

The (Circle)CI did run all the tests: for example, in `torch_job`, we can see

```
python -m pytest --junitxml=test-results/junit.xml -n 6 --max-worker-restart=0 --dist=loadfile --make-reports=tests_torch tests/benchmark tests/bettertransformer tests/deepspeed tests/extended tests/fixtures tests/fsdp tests/generation tests/models tests/optimization tests/peft_integration tests/quantization tests/sagemaker tests/test_backbone_common.py tests/test_configuration_common.py tests/test_configuration_utils.py tests/test_feature_extraction_common.py tests/test_feature_extraction_utils.py tests/test_image_processing_common.py tests/test_image_processing_utils.py tests/test_image_transforms.py tests/test_modeling_common.py tests/test_modeling_flax_common.py tests/test_modeling_flax_utils.py tests/test_modeling_tf_common.py tests/test_modeling_tf_utils.py tests/test_modeling_utils.py tests/test_pipeline_mixin.py tests/test_sequence_feature_extraction_common.py tests/test_tokenization_common.py tests/test_tokenization_utils.py tests/tokenization tests/tools tests/trainer tests/utils || true
```

The CI uses the latest `main` branch of accelerate (so newer than the latest released version)
```
"pip install -U --upgrade-strategy eager -e git+https://github.com/huggingface/accelerate@main#egg=accelerate"
````

Nothing  extra to check.

save_only_model is a nice feature indeed, but it does not work together with load_best_model_at_end (at least with deepspeed enabled), since the final model cannot be loaded from the checkpoint.
Add LLaVA model to transformers
Author: mattmazzola
Pull request: yes

# What does this PR do?

Add LLaVA model to transformers
There are 4 pretrained models
- [liuhaotian/llava-v1.5-7b](https://huggingface.co/liuhaotian/llava-v1.5-7b)
- [liuhaotian/llava-v1.5-13b](https://huggingface.co/liuhaotian/llava-v1.5-13b)
- [liuhaotian/llava-v1.5-7b-lora](https://huggingface.co/liuhaotian/llava-v1.5-7b-lora)
- [liuhaotian/llava-v1.5-13b-lora](https://huggingface.co/liuhaotian/llava-v1.5-13b-lora)

Fixes # (issue)

#22848

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts


Comments:
Hi @mattmazzola 
Thank you very much for your great work! I think that there is already a contributor working on adding Llava in transformers: https://github.com/huggingface/transformers/pull/25789 - perhaps you can follow up with them and check the current status on that PR and see how you can collaborate? 🙏
Docs: Fix broken cross-references, i.e. `~transformer.` -> `~transformers.`
Author: tomaarsen
Pull request: yes

# What does this PR do?

I noticed that some of the docs use incorrect cross-references, e.g. here: https://huggingface.co/docs/transformers/v4.35.2/en/main_classes/trainer#transformers.Trainer.add_callback

Then I noticed these use `~transformer.foo` rather than `~transformers.foo`. I've replaced all `~transformer.` with `~transformers.` in this PR.

## Before submitting
- [x] This PR fixes a typo or improves the docs

## Who can review?

Documentation: @stevhliu and @MKhalusova

- Tom Aarsen

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Gladly! Thank you for your work on the docs! It's always an underappreciated part of any OS project.
TECO - Temporally Consistent Transformers for Video Generation
Author: amyeroberts
Pull request: no
Labels: name: New model

### Model description

TECO is a vector-quantized latent dynamics video prediction model that learns compressed representations to efficiently condition on long videos of hundreds of frames during both training and generation.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Github: https://github.com/wilson1yan/teco
Website: https://wilson1yan.github.io/teco/index.html
Paper: https://arxiv.org/pdf/2210.02396.pdf

Comments: