noamaanMulla-03/transformers-issues huggingface dataset:
Tokenizer behavior is not aligned when initialized from vocab_file and from_pretrained.
Author: KawaiiNotHawaii
Pull request: no

### System Info

transformers 4.31.0
python 3.10

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Load the llama tokenizer using `transformers.AutoTokenizer.from_pretrained()` and specify the pad_token='<pad>'
2. The pad_token_id returns as 32000 (same as '<unk>'), and the total number of tokens is 32001
3. Load the llama tokenizer by specifying the vocab_file and pad_token='<pad>'
4. The pad_token_id returns as 0 (same as '<unk>'), and the total number of tokens is 32000

### Expected behavior

Two different loading methods should result in the same number of tokens.

Comments:
cc @ArthurZucker 

Hey! Thanks for reporting, this is a duplicate of #23930, and will be fixed by #23909 this week! 

Pr is just waiting for a final review! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Remove usage of warnings.warn
Author: osanseviero
Pull request: no

Usage of `warnings.warn` has been deprecated in favor of `logging.warning`, which is managed by the `transformers.logging` utility. 

* https://github.com/search?q=repo%3Ahuggingface%2Ftransformers+%22warnings.warn%22&type=code
* https://github.com/search?q=repo%3Ahuggingface%2Ftransformers%20UserWarning&type=code

`warnings.warn` may still be leveraged for situations where a single warning per runtime is favored, but this will be deprecated in the future as well.

Comments:
Hi @osanseviero I want to work on this issue. Can you please this issue to me so that I can start working on it?


If I understand correctly, we should use `logger.warning` where `logger = logging.get_logger(__name__)` ? 

I can work on this!!


hey @osanseviero, I have started solving the issue. Please assign this issue to me.

Hello, Can I work on this issue? Can you please assign me this?

Hello, we have worked on a different approach with Omar that will not require us to change all `warnings` to `logging` statements.

I'm opening a PR here implementing it: https://github.com/huggingface/transformers/pull/26527

However, in order to honor the contributions you have wanted to make (or have already started making!), I have put you all five as co-authors of the commit, @sahilbhosale63, @Adithya4720, @sachinSingh16-09, @riiyaa24.

I have added the `HACKTOBERFEST-ACCEPTED` tag in case you're participating.

Thanks once again for your contribution, and we're looking forward to the next one :hugs: 

Thanks!! 

Is this still open? 

It will be closed by https://github.com/huggingface/transformers/pull/26527 @AVAniketh0905
Add many missing spaces in adjacent strings
Author: tomaarsen
Pull request: yes

Hello!

# What does this PR do?
This PR resolves numerous occurrences of the following:
```python
        raise ValueError(
            "`training_args.block_size` needs to be a multiple of the global train/eval batch size."
            f"Got {training_args.block_size}, {train_batch_size} and {eval_batch_size} respectively instead."
        )
```
Which results in errors like:
```
`training_args.block_size` needs to be a multiple of the global train/eval batch size.Got 4, 6 and 12 respectively instead."
                                                                                     ^^
```

## How did I go about it?
I created a simple regex: `(["'][^"'\n]*[^ n])(["']\n *f?r?["'][^ ])`, which matches full strings that don't end with spaces or `n` (due to `\n`), followed by a newline, some spaces, and then another string that also doesn't start with a space. I manually went over all cases where this pattern matched through the codebase, and replaced the pattern with `$1 $2` if it was indeed a real mistake.


## Before submitting
- [x] This PR fixes a typo or improves the docs 

## Who can review?

@ArthurZucker 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
CI error is superfluous, merging!

Awesome! Thanks for getting to this so quickly, these kinds of PRs can become a merge conflict mess if they're left for too long.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26751). All of your documentation changes will be reflected on that endpoint.
[docs] LLM prompting guide
Author: MKhalusova
Pull request: yes

# What does this PR do?

This PR addresses part 2.2 ("Prompting" ) of the issue [#24575](https://github.com/huggingface/transformers/issues/24575)  

It adds an LLM Prompting Guide to the docs that covers the following topics:
* basics of prompting,
* encoder-decoder models vs decoder-only models,
* base vs instruct models,
* basic prompts to solve common NLP tasks,
* best practices for prompting,
* advanced techniques like few-shot learning and chain-of-thought
* prompting vs fine-tuning

Let me know, if there's anything missing that has to be included.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The first draft of the LLM prompting guide is ready for review, let me know if anything major is missing. cc @patrickvonplaten 

Feel free to merge when satisfied with it!

@LysandreJik I'm happy with it, but I think we should wait for @gante to review it once he's back from vacation. 

Gently pinging @gante for a review :)
[docstring] Fix docstrings for `UniSpeechConfig`, `UniSpeechForCTC`, `UniSpeechSatConfig`, `UniSpeechSatForCTC` and `Wav2Vec2ForCTC`
Author: gizemt
Pull request: yes

Fixes #26638 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ydshieh


Comments:
Hi @gizemt I think the commit history is messed up when you merged the branch into this PR. You can see irrelevant changes to this PR shown up in `Files changed`.

Could you resolve this please?

oops sorry, messed up rebase. fixed it now. @ydshieh 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26664). All of your documentation changes will be reflected on that endpoint.

good to go @ydshieh thank you!
[DOCS] Update docstrings for  GPT2 and  Whisper tokenizer
Author: McDonnellJoseph
Pull request: yes

# What does this PR do?

- [x ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?







Comments:
@ydshieh Does this look good for you?

@ydshieh I tried troubleshooting on my side I don't understanding the difference between `vocab_file` and `merges_file`, both [merges.txt](https://huggingface.co/gpt2/blob/main/merges.txt) and [vocab.json](https://huggingface.co/gpt2/blob/main/vocab.json) from the GPT2 model card don't seem to work. Maybe something needs to be updated? I'd be happy to take a look 

Hi @McDonnellJoseph It's fine. No need to dive into those 2 files. I requests 2 tiny changes. Once you commit them, we are good to merge the PR.

@ydshieh  No problem I pushed the requested changes :smile: 

Hi, thanks for the commit. May I know why there is irrelevant changes other than pad token id in the last commit pushed?

Sorry my linter formatted automatically and I dind't notice it should be fixed now

We are very close: just need to `make style` and push the change. `tokenization_whisper.py` has some format issue



Ok I thing we're good now :smile: 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26642). All of your documentation changes will be reflected on that endpoint.
[docstring] Fix docstring for 'BertGenerationConfig'
Author: AdwaitSalankar
Pull request: yes

Fixes #26638
## Before Submitting

- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x]  Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [x]  Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [x]  Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can Review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.


Comments:
@ydshieh Can you please review this?

Hi @ydshieh, Does this PR look good?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26661). All of your documentation changes will be reflected on that endpoint.
Fix `PerceiverModelIntegrationTest::test_inference_masked_lm`
Author: ydshieh
Pull request: yes

# What does this PR do?

The PR #23909 changed the result of `vocab_size` of 

```
tokenizer = PerceiverTokenizer.from_pretrained("deepmind/language-perceiver")
```
from `262` to `256`, but the logit has shape `[1, 2048, 262]`.

Let's use `len` here.


## To reproduce:
```
from transformers import PerceiverTokenizer

tokenizer = PerceiverTokenizer.from_pretrained("deepmind/language-perceiver")
# 256 on `2da88537` but `262` on one commit before (`835b0a05`)
print(tokenizer.vocab_size)
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Skip `TrainerIntegrationFSDP::test_basic_run_with_cpu_offload` if `torch < 2.1`
Author: ydshieh
Pull request: yes

# What does this PR do?

Skip `TrainerIntegrationFSDP::test_basic_run_with_cpu_offload` if `torch < 2.1` as this takes 4 hours on `torch 2.0`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Tesnsor size issue
Author: J-JEMINA
Pull request: no

Sorry! I am new here and didn't know where to post my query.

I am using PhageRBP detection which uses bioembeddings tool. In that, I am trying to create embeddings for a genome of e.coli phage of size 108485kb but I get this error immediately. I am working in Google Colab since I am a complete beginner and only comfortable working in that

The expanded size of the tensor (108485) must match the existing size (40000) at non-singleton dimension 1. Target sizes: [1, 108485]. Tensor sizes: [1, 40000]. This most likely means that you don't have enough GPU RAM to embed a protein this long.

As I tried to navigate, I realized it originates from the transformer tool that they are trying to use. I tried seeing if chatgpt can be of any use but it just says the expand function cannot expand beyond 40000. I am willing to pay for Colab Pro or Pro+ as well if that would solve this issue. I have 100s of genomes of similar size to run as well. I do not know how to solve this.

Can someone help me with how to resolve this, please?

Any help would be appreciated.

Thank you!

Comments:
Hi @J-JEMINA, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

Thank you so much for the information!
But when I post there the bot classifies it as spam and hides it. Can anyone help me to get an answer on this?

Hi @J-JEMINA, 

Sorry to hear that. If there's a link to the post, I can see if I can clear it as a real human's question. Another alternative is asking on our discord: https://discord.com/invite/hugging-face-879548962464493619

https://discuss.huggingface.co/t/facing-tensor-size-issue-in-tranformer-tool/54510

This is the link to the query I raised in the forum! I would be happy if you can help

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
model.generate source code
Author: yaof20
Pull request: no
Labels: name: wontfix

Hi there,

I am trying to use BART to do NLG task. During my reading the BART tutorial on the website, I couldn't find the definition of 'model.generate()" function. Could you please add some explaination on that? thanks in advance

Comments:
`generate()` is defined in the `PretrainedModel` class here https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_utils.py#L872. Given that `BartModel`, `BartForConditionalGeneration`, etc,  inherit from `PretrainedModel` they all have access to this generate function. 

Hope this help? 

We actually just moved all of the `generate` functions to their own file:
https://github.com/huggingface/transformers/blob/c4d4e8bdbd25d9463d41de6398940329c89b7fb6/src/transformers/generation_utils.py#L101


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> We actually just moved all of the `generate` functions to their own file:
> 
> https://github.com/huggingface/transformers/blob/c4d4e8bdbd25d9463d41de6398940329c89b7fb6/src/transformers/generation_utils.py#L101

Hi, it seems that .generate() can only take input_ids as source input. I wonder whether input_embs can be used as input.

in case anyone's still looking for the function, it has moved again and can currently be found here:

https://github.com/huggingface/transformers/blob/3e93dd295b5343557a83bc07b0b2ea64c926f9b4/src/transformers/generation/utils.py#L1342
CoaT
Author: sushmanthreddy
Pull request: no
Labels: name: New model

### Model description

Co-scale conv-attentional image Transformers (CoaT), a Transformer-based image classifier equipped with co-scale and conv-attentional mechanisms. First, the co-scale mechanism maintains the integrity of Transformers’ encoder branches at individual scales, while allowing representations learned at different scales to ef- fectively communicate with each other; we design a series of serial and parallel blocks to realize the co-scale mecha- nism. Second, we devise a conv-attentional mechanism by realizing a relative position embedding formulation in the factorized attention module with an efficient convolution-like implementation. CoaT empowers image Transformers with enriched multi-scale and contextual modeling capabilities. On ImageNet, relatively small CoaT models attain superior classification results compared with similar-sized convolu- tional neural networks and image/vision Transformers.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://arxiv.org/pdf/2104.06399.pdf
https://github.com/mlpc-ucsd/CoaT

Comments:
Update expect outputs of `IdeficsProcessorTest.test_tokenizer_padding`
Author: ydshieh
Pull request: yes

# What does this PR do?

There is a change on this Hub repo. 

https://huggingface.co/HuggingFaceM4/tiny-random-idefics/commit/cb18d7776a2e725eb20a2b9f8addf0991b9194b6

that is used for testing.

cc @leot13: you and @ArthurZucker  knows better than me on this

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[docstring] Fix docstring for `RwkvConfig`
Author: Bojun-Feng
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #26638


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
> Spuer clean, thank you @Bojun-Feng again!
> 
> (Could you remove the Draft status - I think it's ready to be merged?)

Of course!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26782). All of your documentation changes will be reflected on that endpoint.
Fix num. of minimal calls to the Hub with peft for pipeline 
Author: ydshieh
Pull request: yes

# What does this PR do?

This is a continuation of #25715 where @sgugger fix `test_cached_model_has_minimum_calls_to_head` but not `test_cached_pipeline_has_minimum_calls_to_head` (as I didn't mention this to him).

This PR mostly copies the logic from #25715.

The current failing test and the error are

```bash
tests/pipelines/test_pipelines_common.py::CustomPipelineTest::test_cached_pipeline_has_minimum_calls_to_head
(line 905)  AssertionError: 2 != 1
```



Comments:
_The documentation is not available anymore as the PR was closed or merged._

> This seems okay to me generally, would like another review as it's touching critical code.
> 
> Not too sure how we could do it, but it would also be nice to have tests for this, as it's the kind of thing that will continue to break unless we actively test it.

We do have tests: 4 tests named `test_cached_xxx_has_minimum_calls_to_head`. This is why I found the failure to fix. 

Merge now.