noamaanMulla-03/transformers-issues huggingface dataset:
deploy whisper by passing last transcribed sentences to decoder's past_key values
Author: hannan72
Pull request: no

I'm working on using whisper model for real time live transcription. I have to deploy audio chucks on model to have a sense of real time transcription, i.e. every 1 second I deploy audio of last 5 seconds.
For such a task, I have to merge transcribed text since the audio has 4 seconds of overlapping with previous samples. Thus, the output transcription at each 1-second time-step has some words in common with previous ones.
There are several solutions for merging such transcribed texts such as using a language model or dynamic programming.
But I have an Idea to use whisper model itself for merging text while it has a language model.
I want to pass the previous transcriptions to its decoder's past key and do the generation based on the initial texts generated at previous time-steps.
Do you have any idea that how can I implement such idea?
@sanchit-gandhi 

Comments:
Hi @hannan72, thanks for raising an issue!

Questions like this should be asked in the [forum](https://discuss.huggingface.co/) as we try to reserve github issues for bugs and specific feature requests. 

> Hi @hannan72, thanks for raising an issue!
> 
> Questions like this should be asked in the [forum](https://discuss.huggingface.co/) as we try to reserve github issues for bugs and specific feature requests.

Thanks to inform me. I posted there. 

Hey @hannan72 - do you have the link to the forum post? I can reply directly there :) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Pre-training language model with Translation Language Modelling (TLM) objective 
Author: aloka-fernando
Pull request: no

### Feature request

For large-scale pre-training MLM objective is heavily used. eg: BERT (Devlin et al., 2018), XLM-R (Connaue et al.,2019). The resources available from huggingface is to train a language model (LM) using MLM or CLM objectives. (ie. https://github.com/huggingface/notebooks/blob/main/examples/language_modeling.ipynb)

To recreate XLM (Lample and Connaue, 2019) I wish to pre-train my own language model using both MLM and TLM objectives. Please advice on how to do this using the huggingface transformers.

Thank you.

### Motivation

MLM + TLM is a common pre-training objective during language modelling training. Therefore for further improvement to these models, we need to first train using MLM + TLM objectives.  In addition to this, if I need to customize the masking to be done for noun terms only, or verb terms only, please let me know whether this is allowed.

### Your contribution

I would appreciate if the pre-training on TLM objective is also provided.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

If this is supported in the codebase please provide the steps or refer to an available resource.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Documentation] Possible mistake in model_doc LLaMA
Author: habaneraa
Pull request: no

### System Info

Maybe a small mistake in the documentation. Here: https://github.com/huggingface/transformers/blob/ef42c2c487260c2a0111fa9d17f2507d84ddedea/docs/source/en/model_doc/llama.mdx?plain=1#L17

The title "LLaMA: Open and Efficient Foundation Language Models" is repeated. Does it mean [this arxiv link](https://arxiv.org/pdf/2302.13971.pdf)?

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce:

1. Open docs https://huggingface.co/docs/transformers/main/en/model_doc/llama#overview
2. See the first line

### Expected behavior

It should be a link to https://arxiv.org/pdf/2302.13971.pdf

Comments:
@habaneraa You're right! This seems to be a mistake in the docs. I will open a PR to fix this.
Update LLaMA docs with arxiv link
Author: awinml
Pull request: yes

# What does this PR do?
Fixes #23186 

Adds arxiv link for "LLaMA: Open and Efficient Foundation Language Models" (https://arxiv.org/abs/2302.13971) to LLaMA model docs.

## Who can review?
@sgugger, @stevhliu and @MKhalusova

Comments:
_The documentation is not available anymore as the PR was closed or merged._
fix random attention for pytorch's bigbird/pegasus_bigbird
Author: Bearnardd
Pull request: yes

Fixes # (issue)
https://github.com/huggingface/transformers/issues/23055

# What does this PR do?
Add control over usage of random attention of `BigBird` based on current mode (training/eval)

## Who can review?

@sanchit-gandhi @ydshieh 



Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @Bearnardd Thank you for the PR.

I have one question: why `def _bigbird_block_rand_mask_with_head` is not modified for this pytorch BigBird file ..?

Hi @sanchit-gandhi! I have removed the static method as I think it is the best approach. 

> Hi @Bearnardd Thank you for the PR.
> 
> I have one question: why `def _bigbird_block_rand_mask_with_head` is not modified for this pytorch BigBird file ..?

Thanks for the comment! To be honest I am not sure If I understand you correctly, since from what I can see this function is updated. Could you elaborate what exactly is missing?

> > Hi @Bearnardd Thank you for the PR.
> > I have one question: why `def _bigbird_block_rand_mask_with_head` is not modified for this pytorch BigBird file ..?
> 
> Thanks for the comment! To be honest I am not sure If I understand you correctly, since from what I can see this function is updated. Could you elaborate what exactly is missing?

Sorry, my bad. You are right :-)

cc @sgugger 

I have pushed the changes @sgugger :)
examples/run_speech_recognition_ctc: UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 725: character maps to <undefined>
Author: XekriRedmane
Pull request: no

### System Info

- `transformers` version: 4.28.1
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.11.2
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Create a json file corresponding to the [first example in speech recognition for pytorch](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition#single-gpu-ctc). See attached.

Run `python run_speech_recognition_ctc.py train.json`

Get error:

```
Traceback (most recent call last):
  File "F:\eo-reco\run_speech_recognition_ctc.py", line 775, in <module>
    main()
  File "F:\eo-reco\run_speech_recognition_ctc.py", line 378, in main
    model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\eo-reco\.env\Lib\site-packages\transformers\hf_argparser.py", line 391, in parse_json_file
    data = json.loads(open_json_file.read())
                      ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\rober\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 725: character maps to <undefined>
```

[train.json.zip](https://github.com/huggingface/transformers/files/11415631/train.json.zip)




### Expected behavior

No error.

Comments:
Fix hf_argparser.parse_json_file to open file with utf-8 encoding, close file when finished
Author: XekriRedmane
Pull request: yes

# What does this PR do?

Fixes #23193


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Making Simple whitespace tokenizer and then using that tokenizer to make a language model from scratch?
Author: ishaansharma
Pull request: no
Labels: name: wontfix

# ❓ How can I make a whitespace tokenizer and use it to build a language model from scratch using transformers.

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details

I am trying to make a language model usingtransformer from scratch , For that I want to build a tokenizer that tokenize a text data using whitespace only, nothing else. It should generate a vocab file that does not have any special character , just the words seperated by whitespace and than I want to use that tokenizer to build a language model from scratch using https://huggingface.co/blog/how-to-train. 

I don't want my tokenizer to generate vocabs that have any kind of special characters viz "##" in front of words and any accents in my vocab. 

I know there are tokenizers that give good results for language model like bpe and word peice. but I have a requirement where I just want to use whitespace tokenizer only for training a language model. 

Thanks and Regards
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
I am also looking for a solution to this problem.

I want the tokenizer to do something like this one
https://github.com/huggingface/transformers/issues/1036#issuecomment-522201118

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


It has been three years since this was marked stale, has hugging face or others implemented something that I could use for this use case?
Generate: starcoder 🤜 🤛 assisted generation
Author: gante
Pull request: yes

# What does this PR do?

Starcoder (GPTBigCode) has a unique cache format, and assisted generation is heavy on cache-related ops. This PR adds the GPTBigCode special case.

All slow tests for assisted generation are passing after these changes.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Arg. Would love to be able to change those parts directly in the model files in the future, instead of hacking special keys like this :-/

@sgugger I'm going to add a common test for the cache format, to ensure we don't do this again for future models :)
ImportError: cannot import name 'OpenLlamaForCausalLM' from 'transformers'
Author: MoaazZaki
Pull request: no

### System Info

- `transformers` version: 4.29.0.dev0
- Platform: Linux-5.15.0-1035-aws-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.0+cu117 (False)
- Tensorflow version (GPU?): 2.11.1 (False)
- Flax version (CPU?/GPU?/TPU?): 0.5.3 (cpu)
- Jax version: 0.3.6
- JaxLib version: 0.3.5
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. pip install git+https://github.com/huggingface/transformers.git#egg=transformers or clone & pip install -e .
2. `from transformers import OpenLlamaForCausalLM`



### Expected behavior

The module `OpenLlamaForCausalLM` is imported successfully, as I tried the latest version.

am I doing something wrong here?

Comments:
This works for me on main. Are you sure you have pulled the latest changes if installing from the repo, or that you execute the code in the same environment you installed Transformers from source in?

It seems like it was an issue with pip failed to replace the current installed version with the main. Doing the following solve everthing:

1- `pip uninstall transformers`
2- Cloning the repo & `pip install -e .`

Thanks for the help 🙌
In one-node multi-GPU setups, the ImageProcessor module in Detr and other object detection models may cause an error in the Trainer.
Author: alexw994
Pull request: no

### System Info

When training object detection models using multiple GPUs in a single node, I encountered an issue with nn.DataParallel. The nn.DataParallel module splits Tensor data across each GPU before model inference, but it does not split List data. Since all object detection ImageProcessor modules return List objects, the targets will not be split, but the inputs will have already been split. Additionally, inputs are of type Tensor, while targets are of type List.

During loss computation, the shapes of inputs and targets may not match along the 0th dimension.

Is this the best practice for single-node multi-GPU setup? Should I avoid using nn.DataParallel?

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Due to confidentiality reasons, I cannot submit the complete code. I will provide a example later.

### Expected behavior

Training object detection model with multiple GPUs on a single machine as usual.

Comments:
You should use DistributedDataParallel for multi-GPU setup, which is better supported and the recommended way from the PyTorch team. It is supported by the Trainer, but just launch your training with `torchrun` instead of `python`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Export pix2struct models to ONNX
Author: arvisioncode
Pull request: no

How can the pix2struct models be exported to ONNX?

I think these models are not available yet to make the transformation with:
!python -m transformers.onnx --model=google/pix2struct-docvqa-base --feature=vision2seq-lm scratch/onnx --atol 1e-3


Comments:
ONNX conversion is now fully handled by the `optimum` library, so you should open your issue there :-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
FastTokenizer for LLaMa
Author: theblackcat102
Pull request: no

### Feature request

FastTokenizer support for LLaMa sentencepiece tokenizer.

### Motivation

The offset_mapping is only available in FastTokenizer, it would be useful if there's support for this.

### Your contribution

I have tried using existing sentencepiece based model as replacement. However hf conversation code means we are missing the byte fallback support 

```
The sentencepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers
```

Which means out of vocabulary tokens are simply mapped to <unk> instead of using the byte mapping inside the vocab.

Comments:
Let's maybe wait for the LLaMa PR to be merged first?

it is fix on tokenizers

https://github.com/huggingface/tokenizers/pull/1183

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Unit tests for hf_argparser
Author: XekriRedmane
Pull request: yes

# What does this PR do?

Adds unit tests for hf_argparser.py

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Oh, I didn't see that file. Will retract request.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23212). All of your documentation changes will be reflected on that endpoint.
[BigBird] random attention in FlaxBigBird is not random
Author: thevasudevgupta
Pull request: no
Labels: name: Good Second Issue

name: Good Difficult Issue

name: bug

### System Info

```shell
NOT NEEDED
```


### Who can help?

@patrickvonplaten 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`numpy.random` is getting used everywhere in `FlaxBigBird` for fetching random indices for computing attention over random tokens. This is wrong as these indices would be cached during jit compilation. Hence, bigbird would attend similar random tokens (i.e same indices) in every step
This can be fixed if random indices are prepared in datacollator and passed to `model.__call__` during training.

https://github.com/huggingface/transformers/blob/ee393c009a243bbb86fa11d2efa771a1704d85cf/src/transformers/models/big_bird/modeling_flax_big_bird.py#L975



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I see, thanks for putting up this issue here! 

Think the easiest fix in this case would be to use transition from `numpy.random` to `jax.random`: https://jax.readthedocs.io/en/latest/jax.random.html no? This way the user can always pass a PRNG key to the forward which would define the random attention mask?

What do you think @vasudevgupta7 ?

Hello @patrickvonplaten,

sorry for the late reply. I missed your comment somehow.

> Think the easiest fix in this case would be to use transition from numpy.random to jax.random: https://jax.readthedocs.io/en/latest/jax.random.html no? This way the user can always pass a PRNG key to the forward which would define the random attention mask?

Yes, this should be the best way to go. But then Jax implementation will diverge from PyTorch implementation in case of block sparse attention right?? and some tests would fail ig.

Thanks for the reply @vasudevgupta7!

Leaving this as a "second good issue" now as I won't find the time to dig into it in the near future. @community Please ping me here if you'd like this to be fixed soon

Hi @patrickvonplaten @thevasudevgupta - is this still a problem? If so I would love to pick it up to have some work on the Christmas break :)

This would be great @Bearnardd :-) 

@thevasudevgupta this issue can be closed as the fix is merged :)

Very cool to know that!
[SAM] Add resources
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds links to 2 demo notebooks I made regarding SAM.

It also fixes a hyperlink which didn't render properly in the docs.

cc @younesbelkada 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23224). All of your documentation changes will be reflected on that endpoint.