noamaanMulla-03/transformers-issues huggingface dataset:
Shouldn't GPT2 use Linear instead of Conv1D?
Author: spolu
Pull request: no

Conv1D seems to be inherited from GPT but does not seem to serve any special purpose in GPT2 (BERT uses Linear).

Should GPT2's model be moved to using Linear (which is easier to grasp obvioulsy)?

Comments:
Maybe this would break pre-trained weights loading? Interested to understand if that's the only reason?

Possibility, feel free to test the modification and submit a PR @spolu!

Hi guys,
I also wondered whether anyone modified the gpt2 model to have nn.Linear instead of Conv1D layers (using the pre-trained weights).
Did any of you success or found such implementation?

Hi folks,

I am working on a use case which takes only nn.Linear layers (so the code worked fine for BERT) but it didn't for GPT-2 because of the same reason?

Is there a way to change the Conv1D() to linear layers? Below is the actual gpt2 I was getting when using standard script form HF - 

```
GPT2Model(
  (wte): Embedding(50257, 768)
  (wpe): Embedding(1024, 768)
  (drop): Dropout(p=0.1, inplace=False)
  (h): ModuleList(
    (0-11): 12 x GPT2Block(
      (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
      (attn): GPT2Attention(
        (c_attn): Conv1D()
        (c_proj): Conv1D()
        (attn_dropout): Dropout(p=0.1, inplace=False)
        (resid_dropout): Dropout(p=0.1, inplace=False)
      )
      (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
      (mlp): GPT2MLP(
        (c_fc): Conv1D()
        (c_proj): Conv1D()
        (act): NewGELUActivation()
        (dropout): Dropout(p=0.1, inplace=False)
      )
    )
  )
  (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
```

@NamburiSrinath @IdoAmit198

The difference between Convo1D and Linear layers is,
- In Linear layers weight matrix is transposed when doing the calculation
- In Convo1D no transpose occurs

So what you can do is,  get the state dict from the pre trained weights, get Convo1d layer entries and transpose those tensors.

After this just replace convo1D with linear layers and load the state dict to this new model.
ImportError: cannot import name 'logging' from 'huggingface_hub'
Author: zyc1128
Pull request: no

### System Info

from transformers import AutoTokenizer,AutoModelForCausalLM
all dependencies have the latest versions

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1

### Expected behavior

1

Comments:
Hi @zyc1128, thanks for opening an issue! 

I'm unable to replicate this problem and can run `from transformers import AutoTokenizer,AutoModelForCausalLM` without issue. 

> all dependencies have the latest versions

This is underspecified. For example, is "latest" nightly builds or stable release? Which dependencies (it depends how you are using the library). The best way to get this information is by running `transformers-cli env` in the terminal.

Could you share a full traceback of the error? 

Hi @zyc1128 please try upgrading `huggingface-hub` with `pip install --upgrade huggingface-hub` and also restart your kernel or session after doing so

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
avoid padding for `num_frames` in `AutomaticSpeechRecognitionPipeline`
Author: faaany
Pull request: yes

## What does this PR do?
The following example in the batch size part of [this](https://huggingface.co/docs/transformers/en/pipeline_tutorial) tutorial return an error.
```python
from transformers import pipeline

transcriber = pipeline(model="openai/whisper-large-v2", device=0, batch_size=2)
audio_filenames = [f"https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/{i}.flac" for i in range(1, 5)]
texts = transcriber(audio_filenames)
print(texts)
```
```bash
Traceback (most recent call last):
  File "/workspace/DOC/tutorials.py", line 5, in <module>
    texts = transcriber(audio_filenames)
  File "/workspace/transformers/src/transformers/pipelines/automatic_speech_recognition.py", line 284, in __call__
    return super().__call__(inputs, **kwargs)
  File "/workspace/transformers/src/transformers/pipelines/base.py", line 1235, in __call__
    outputs = list(final_iterator)
  File "/workspace/transformers/src/transformers/pipelines/pt_utils.py", line 124, in __next__
    item = next(self.iterator)
  File "/workspace/transformers/src/transformers/pipelines/pt_utils.py", line 269, in __next__
    processed = self.infer(next(self.iterator), **self.params)
  File "/root/miniforge3/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 631, in __next__
    data = self._next_data()
  File "/root/miniforge3/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 675, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/root/miniforge3/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 42, in fetch
    return self.collate_fn(data)
  File "/workspace/transformers/src/transformers/pipelines/base.py", line 194, in inner
    padded[key] = _pad(items, key, _padding_value, padding_side)
  File "/workspace/transformers/src/transformers/pipelines/base.py", line 100, in _pad
    max_length = max(item[key].shape[1] for item in items)
  File "/workspace/transformers/src/transformers/pipelines/base.py", line 100, in <genexpr>
    max_length = max(item[key].shape[1] for item in items)
IndexError: tuple index out of range
```
This PR fixes this error. Below is the results after fix:
```bash
[{'text': ' He hoped there would be stew for dinner, turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick, peppered, flour-fattened sauce.'}, {'text': ' Stuff it into you, his belly counselled him.'}, {'text': ' After early nightfall the yellow lamps would light up here and there the squalid quarter of the brothels.'}, {'text': ' Y en las ramas medio sumergidas revoloteaban algunos pájaros de quimérico y legendario plumaje.'}]
```


Comments:
@kamilakesbi 

@faaany thanks for working on this! 

This should be solved by PR #31776 ! let's wait for it to be merged :) 

good, thanks for fixing it! close this one
🚨 Bloom support for cache class
Author: zucchini-nlp
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Follow-up from #31421, adds support for Cache class in Bloom models. This is a breaking change as the cache class currently has its keys in shape=`(bs*heads, seq_len, dim)` as opposed to prev shape=`(bs*heads, dim, seq_len)`

This change was needed because cache class assumes that concatenation is always performed along "dim=-2"


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31445). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Exactly, I was also thinking of that but didn't want to break Bloom more. Will add full compatibility with Cache then

(pls ping when it's ready for a re-review!)

@gante ready for review

- added cache position
- deprecation warning for legacy cache until then next release
- bloom has standard format cache and some bloom tests aren't skipped anymore

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@gante ping

Couldn't add copy statements because Bloom is different from Llama, and accepts some other kwargs. Addressed all other comments

Maybe a final review from @ArthurZucker would be nice here

Yes, this one and the other PR that deprecates cache will be waiting for @ArthurZucker :)

Will merge it today evening, got approve from one of the core maintainers. @amyeroberts , let me know if you want to review once more, so I won't merge yet :)
Upload new model failure report to Hub
Author: ydshieh
Pull request: yes

# What does this PR do?

Upload new model failure report to Hub, and add a button pointing to it on the Slack report channel.

<img width="590" alt="Screenshot 2024-07-29 093428" src="https://github.com/user-attachments/assets/95866526-22d2-419e-ab1a-176d5f56be11">

and


<img width="967" alt="Screenshot 2024-07-29 093550" src="https://github.com/user-attachments/assets/c54c86eb-88ee-4144-a6eb-b73df0ef9c85">


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32264). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Optimize t5 tokenize logic to avoid redundant calls
Author: leejet
Pull request: yes

# What does this PR do?

Optimize t5 tokenize logic to avoid redundant calls.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
error when convert llama1 ckpts to hf formath
Author: mabingqi
Pull request: no

### System Info

- `transformers` version: 4.41.0.dev0
- Platform: Linux-4.18.0-425.3.1.el8.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.12
- Huggingface_hub version: 0.21.4
- Safetensors version: 0.4.2
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

`python transformers/models/llama/convert_llama_weights_to_hf.py --input_dir path-to-llama1-7b-source --model_size 7B --output_dir path-to-llama1-7b-target --llama_version 1`

### Expected behavior

error: RuntimeError: shape '[32, 2, 2, 4096]' is invalid for input of size 16777216
In line https://github.com/huggingface/transformers/blob/f26e4073707189c93915227779a4f6ea3c40d43b/src/transformers/models/llama/convert_llama_weights_to_hf.py#L181
the k_proj in llama1 7b is 4096 by 4096, but the dim1 here is 128.
Maybe this is a bug when converting llama1 ckpt

Comments:
cc @ArthurZucker 

Same question!

Haha that's annoying, we might have broken conversion for llama1 when adding llama3. 
Could you test on `transformers==4.38` or `4.39`? 

I encountered the same error when I was converting the Llama 2 model. Using ``transformers==4.38`` solved this problem.

Yep, it's not expected. I'll open a PR to fix conversion on all models 🤗 

Any update on this issue, please.

If you just need to convert the old Llama models, you can quickly revert to a previous version of transformers (prior to 4.39.0) 😉 

The problem is not solved by changing the transformer version. I'm trying to convert llama-2 and it gives me the same error

Sorry about that, I'll do this today!


Hey @ArthurZucker. Any updates? I am running into the same issue when trying to convert llama-2 (and downgrading transformers to 4.38 also doesn't fix it). 

Hey! I am downloading everything now to fix it ! 🤗  there is #30734 as well 

Sorry for the delay! 


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Resize embeds with DeepSpeed 
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Continues #32192. I used the provided script and found why it was being set to 0, we needed to gather parameters once more when assisgning the `vocab_size` because in the current version we are basically using the old embedding class, which was sharded with Deepspeed. In the prev impl it worked without gathering as we used a newly initialized Embedding class which apparently doesn't require gather to access its weights. 

Also I ecountered another error that in DeepSpeed the weight's shape didn't change even after assigning `weight.data`. To solve that we have to assign the `Parameter` and not tensors within parameter

cc @iamseokhyun, can you verify this works for you?

Comments:
I confirmed that this PR works for both the test script in [#32192](https://github.com/huggingface/transformers/pull/32192) and the reported scripts in [#32170](https://github.com/huggingface/transformers/issues/32170). I believe this patch should be released quickly, considering that this method is widely used in fine-tuning scripts. Thanks for your help!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32214). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: Fixed wrong argument passed to `convert_blip_checkpoint` function call
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
The definition of `convert_blip_checkpoint` function from the same file:
https://github.com/huggingface/transformers/blob/44f6fdd74f84744b159fa919474fd3108311a906/src/transformers/models/blip/convert_blip_original_pytorch_to_hf.py#L82

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ArthurZucker @amyeroberts 

Comments:
make `p_mask` a numpy array before passing to `select_starts_ends`
Author: faaany
Pull request: yes

# What does this PR do?
The multimodal example in [this](https://huggingface.co/docs/transformers/en/pipeline_tutorial#multimodal-pipeline) tutorial fails
```bash
from transformers import pipeline

vqa = pipeline(model="impira/layoutlm-document-qa")
output = vqa(
    image="https://huggingface.co/spaces/impira/docquery/resolve/2359223c1837a7587402bda0f2643382a6eefeab/invoice.png",
    question="What is the invoice number?",
)
output[0]["score"] = round(output[0]["score"], 3)
print(output)
```
```bash
Traceback (most recent call last):
  File "/workspace/DOC/tutorials.py", line 4, in <module>
    output = vqa(
  File "/workspace/transformers/src/transformers/pipelines/document_question_answering.py", line 271, in __call__
    return super().__call__(inputs, **kwargs)
  File "/workspace/transformers/src/transformers/pipelines/base.py", line 1247, in __call__
    return next(
  File "/workspace/transformers/src/transformers/pipelines/pt_utils.py", line 125, in __next__
    processed = self.infer(item, **self.params)
  File "/workspace/transformers/src/transformers/pipelines/document_question_answering.py", line 448, in postprocess
    answers = self.postprocess_extractive_qa(model_outputs, top_k=top_k, **kwargs)
  File "/workspace/transformers/src/transformers/pipelines/document_question_answering.py", line 477, in postprocess_extractive_qa
    starts, ends, scores, min_null_score = select_starts_ends(
  File "/workspace/transformers/src/transformers/pipelines/question_answering.py", line 121, in select_starts_ends
    undesired_tokens = np.abs(p_mask.numpy() - 1)
AttributeError: 'list' object has no attribute 'numpy'

```
This PR fixes this. 

Comments:
Hi @faaany, thanks for digging into this! 

Instead of needing to do `np.array` conversions, we could instead construct a [numpy array directly here](https://github.com/huggingface/transformers/blob/fe008d6ebea1f5770b740991daeefd9322fa434a/src/transformers/pipelines/document_question_answering.py#L381) and just remove the `.numpy` cast 

@faaany Could you rebase on main to include upstream changes? This should resolve the failing quality tests. 

There's some failing pipeline tests related to this change which will need to be resolved before we can merged. 

> @faaany Could you rebase on main to include upstream changes? This should resolve the failing quality tests.
> 
> There's some failing pipeline tests related to this change which will need to be resolved before we can merged.

done

Thanks again, @faaany!
Repo: remove exceptions in `check_docstrings`
Author: gante
Pull request: yes

# What does this PR do?

Remove exceptions from the objects to ignore list that no longer need to be exceptions, in `check_docstrings`.

(I'm aiming at doing one "repo quality control"-related PR a day throughout the summer, so I can learn the ropes of our current quality controls and try to improve on them 🤗 ) 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32259). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Woah that's super nice!
fix(docs): Fixed a link in docs
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
Just fixes a link in docs.

Fixes # (issue)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts @ArthurZucker 

Comments:
Whisper tokenizer word level timestamps
Author: kamilakesbi
Pull request: yes

# What does this PR do?

This PR aims at solving issue #31778. 

When using the `_decode_asr` of Whisper Tokenizer to decode with word-level timestamps, a weird behavior is observed with tokens and timestamps going back in time. 

@xenova proposed a simple solution to solve this issue in transformers.js (see this [PR](https://github.com/xenova/transformers.js/pull/545/commits/9ac7ceb408900d4f0ab7699ef96fc498cbd4f49d)): when finding the common tokens between successive chunks, we need to add an additional step where we make sure matching tokens also have timestamps that are order. 

This solves the observed error. 

# Who can review: 

@sanchit-gandhi 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32197). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@sanchit-gandhi thanks for your review! 

This should be ready for final review @amyeroberts