noamaanMulla-03/transformers-issues huggingface dataset:
QWenLMHeadModel does not support Flash Attention 2.0 yet.
Author: zhangfan-algo
Pull request: no

![image](https://github.com/huggingface/transformers/assets/47747764/63db1736-d049-4ab0-8187-df43d3afb75e)


Comments:
+1

Hi @zhangfan-algo, thanks for creating this feature request. 

Is this model code on the hub? If so, the request to add FA2 support should be done on the discussion page for that model. 

In the meantime - we should update this message on our end to route users to the correct place. 

> Hi @zhangfan-algo, thanks for creating this feature request.
> 
> Is this model code on the hub? If so, the request to add FA2 support should be done on the discussion page for that model.
> 
> In the meantime - we should update this message on our end to route users to the correct place.

魔塔下载的模型，今天微调的时候出现这个错误：
```
Traceback (most recent call last):
  File "/mnt/pfs-guan-ssai/nlu/data/tianxy/LLaMA-Factory-2/src/train_bash.py", line 15, in <module>
    main()
  File "/mnt/pfs-guan-ssai/nlu/data/tianxy/LLaMA-Factory-2/src/train_bash.py", line 5, in main
    run_exp()
  File "/mnt/pfs-guan-ssai/nlu/data/tianxy/LLaMA-Factory-2/src/llmtuner/train/tuner.py", line 26, in run_exp
    run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
  File "/mnt/pfs-guan-ssai/nlu/data/tianxy/LLaMA-Factory-2/src/llmtuner/train/sft/workflow.py", line 29, in run_sft
    model, tokenizer = load_model_and_tokenizer(model_args, finetuning_args, training_args.do_train)
  File "/mnt/pfs-guan-ssai/nlu/data/tianxy/LLaMA-Factory-2/src/llmtuner/model/loader.py", line 67, in load_model_and_tokenizer
    model = AutoModelForCausalLM.from_pretrained(
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 561, in from_pretrained
    return model_class.from_pretrained(
  File "/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3456, in from_pretrained
    config = cls._autoset_attn_implementation(
  File "/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1302, in _autoset_attn_implementation
    cls._check_and_enable_flash_attn_2(
  File "/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1382, in _check_and_enable_flash_attn_2
    raise ValueError(
ValueError: QWenLMHeadModel does not support Flash Attention 2.0 yet. Please open an issue on GitHub to request support for this architecture: https://github.com/huggingface/transformers/issues/new
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add Depth Anything
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds an alternative design to #28643, which adds a standalone separate model.

Pros:

- [x] does not clutter the existing modeling_dpt.py
- [x] is more in line with the [philosophy](https://huggingface.co/blog/transformers-design-philosophy)

Cons:

- [x] actually, not a lot :)

Comments:
The tests_pr_documentation_tests is failing due to the checkpoint not yet being available on the TikTok organization.

@amyeroberts thanks for your review, addressed all comments. CI is failing due to unrelated tests
Keep relevant weights in fp32 when `model._keep_in_fp32_modules` is set even when `accelerate` is not installed
Author: fxmarty
Pull request: yes

As per title, aligns the behavior of `PreTrainedModel.from_pretrained(..., torch_dtype=torch.float16)` when accelerate is installed and when it is not.

Previously,

```python
import torch
from transformers import AutoModelForSeq2SeqLM

model = AutoModelForSeq2SeqLM.from_pretrained("t5-small", torch_dtype=torch.float16)

print(model.encoder.block[0].layer[1].DenseReluDense.wo.weight.dtype)
```
would print `torch.float16` when accelerate was not installed, and `torch.float32` when installed. Having different dtype depending on an external package being installed or not is bug prone. [As `accelerate` is a hard requirement](https://github.com/huggingface/transformers/blob/e4e55af79c9b3dfd15cc2224f8f5b80680d83f03/setup.py#L260), it could be also reasonable to simply raise an error in `from_pretrained` if using pytorch & accelerate is not installed.

Note:
```python
for name, param in model.named_parameters():
    param = param.to(torch.float32)
```
does nothing out of the loop scope.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

sounds good

I fixed the original test that was not actually testing the case where `accelerate` is not available.

nice!
Improve Backbone API docs
Author: merveenoyan
Pull request: yes

I improved the wording of the Backbone API docs and added a new illustration.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28666). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts I don't have write access in this repository, can you merge this? 

@merveenoyan Yep!
Fix lr_scheduler in no_trainer training scripts
Author: bofenghuang
Pull request: yes

# What does this PR do?

Hi!

I ran into unexpected behaviors with the LR scheduler when using gradient accumulation in a distributed environment within this script. I managed to fix it on my end with the following changes:

1. `AcceleratedScheduler` currently steps only when gradient accumulation is performed. As a result, there's no need to multiply `gradient_accumulation_steps` when setting up `num_warmup_steps` and `num_training_steps` for `get_scheduler`, since it will be handled inside `AcceleratedScheduler`.

https://github.com/huggingface/accelerate/blob/6a4857fec2bbae83880014cfd834d8a3e22de68b/src/accelerate/scheduler.py#L60-L64

2. In distributed settings, the scheduler steps for `accelerator.num_processes` each time, requiring multiplying `num_warmup_steps` by `accelerator.num_processes` here to guarantee proper warm-up.

3. When handling distributed training and setting `max_train_steps` for total training steps instead of `num_train_epochs`, `num_training_steps` for the scheduler should be multiplied by `accelerator.num_processes` as well, to ensure the correct number of updating steps.

I could also update other scripts facing a similar issue, if this modification is validated :)

Thanks in advance!

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

cc @muellerzr @pacman100

Comments:
Kindly ping @muellerzr 

Hi @muellerzr, thanks for the review! Just updated other no_trainer scripts which handle gradient accumulation by `with accelerator.accumulate(model)`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ImageToTextPipeline does not support InstructBlip Models
Author: elena-soare20
Pull request: no
Labels: name: Good Second Issue

name: Feature request

### System Info


- `transformers` version: 4.36.0.dev0
- Platform: Linux-generic-x86_64
- Python version: 3.8.12
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 1.10.0a0+0aef44c (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@Narsil @amyeroberts

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

processor = InstructBlipProcessor.from_pretrained("Salesforce/instructblip-flan-t5-xl")
pipe = pipeline("image-to-text", model="Salesforce/instructblip-flan-t5-xl", processor=processor.image_processor, tokenizer=processor.tokenizer, device=0)
prompt = "describe te following image"
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

pipe(images=image, prompt=prompt)

### Expected behavior

returns a textual description of the image. 
Instead, I get an error: 
`TypeError: ones_like(): argument 'input' (position 1) must be Tensor, not NoneType`

I suspect this is caused by the `ImageToTextPipeline.preprocess()`, where we should ave custom behaviour for InstructBlip models to process the image and text in one go: `inputs = processor(images=image, text=prompt, return_tensors="pt")`


Comments:
Hi @elena-soare20, thanks for raising this issue! 

Yes, at the moment InstructBLIP isn't compatible with the pipeline because of the specific processing it does - which is different from many other models. Specifically, it has two tokenizers to create `qformer_input_ids` and `input_ids` to be passed to the model. There's some ongoing work to unify our processors so that hopefully more models like these can be quickly integrated. 

Happy to review any PRs for anyone in the community who would like to enable this. See also: #21110 


hey @amyeroberts I would be happy to work on this


@nakranivaibhav Awesome! Feel free to ping me for review when you have a PR ready 🤗 

@amyeroberts Give me some time on this. The models are very large to reproduce the error. I am figuring out where to reproduce the error to start working on it.

@nakranivaibhav If all you need is a model to test functionality i.e. a randomly initialized model that outputs nonsense is fine, then the small model used during tests might help here. The config to build the model and test inputs can be [found here](https://github.com/huggingface/transformers/blob/f40b87de0ca234df61f76928956c4a2118c0b548/tests/models/instructblip/test_modeling_instructblip.py#L410).

@amyeroberts Yes that i what I need. Thank you for pointing it out.
Update docs related to generate() method in transformers/pipelines
Author: coolyashas
Pull request: yes

# What does this PR do?

Fixes #28224

In the issue, it is suggested to change the link to [this](https://huggingface.co/docs/transformers/generation_strategies), but I find [this](https://huggingface.co/docs/transformers/v4.36.1/en/main_classes/text_generation) to be a better fit.

@stevhliu and @MKhalusova

Comments:
I believe all necessary changes have been made @stevhliu 

Hey, can you also apply the changes to the `text2text_generation.py` and `text_generation.py` files please?

> Hey, can you also apply the changes to the `text2text_generation.py` and `text_generation.py` files please?

![image](https://github.com/huggingface/transformers/assets/32161167/92b28c41-8fa3-41f8-bdad-e0a15a513872)

Hey, the changes have already been made in the files you mentioned in this PR itself [here](https://github.com/huggingface/transformers/pull/28231/files). Are there any other changes you wanted implemented?


@stevhliu I deleted this repo mistakenly. Could you pls reopen it in order to finish the PR?

@coolyashas Unfortunately we can't reopen this PR, as the repo through which this PR was submitted has been deleted. If you reopen another PR with the same changes I'll be happy to do a quick review. 

> @coolyashas Unfortunately we can't reopen this PR, as the repo through which this PR was submitted has been deleted. If you reopen another PR with the same changes I'll be happy to do a quick review.

Sure will do, thanks for the reply!
Don't fail when `LocalEntryNotFoundError` during `processor_config.json` loading
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix #28697.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28709). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
bug fix: fix vocab_size being 0 for deepspeed zero3
Author: circlecrystal
Pull request: yes

# What does this PR do?

This PR fixes the error encountered during model training with DeepSpeed Zero-3.

@pacman100

Comments:
> Hello, please give a minimal reproducer of the issue first that this PR is meant to resolve.

Sorry but I encountered a bug related to these lines of codes in my company project. Since the code is proprietary, I cannot share it. Despite hoping to build a minimal example, I cannot find a simple way to achieve it. I encountered this error when running loss computation at this line:
https://github.com/huggingface/transformers/blob/4edffda636fb2bf673282b31163e598b5872994e/src/transformers/models/llama/modeling_llama.py#L1209
When using DeepSpeed Zero-3, without the provided fix, this line will report error (because self.config.vocab_size will be zero without the fix).

The reported error is not related to the provided fix. I think it's a mis-report.
![image](https://github.com/huggingface/transformers/assets/5665980/43c75a89-dcf8-443d-ba70-3c6aeebb7656)
![image](https://github.com/huggingface/transformers/assets/5665980/652df546-6296-485c-a140-01f92dca883b)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
support PeftMixedModel signature inspect
Author: Facico
Pull request: yes

support PeftMixedModel signature inspect

Use model.base_model.model to get the base model(PeftMixedModel don't have "get_base_model" attribute)

Comments:
cc @younesbelkada 

@younesbelkada Thanks for the suggestion, I've pushed the changes!

@Facico can you confirm the latest commit on this branch fixes your issue?

Yes, They all work. @younesbelkada 

@younesbelkada How do I pass the tests_exotic_models?

Rebasing on main should help! 

Yes @Facico please merge your branch with upstream main and the test should be fixed. Afrer that we'll be able to merge your PR

@younesbelkada @ArthurZucker Thanks! That helped.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28321). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix `weights_only`
Author: ydshieh
Pull request: yes

# What does this PR do?

The changes in #28506 is incorrect, and after that we still have issue with torch < 1.13. This PR fixes the issue in a correct way.

Fix #28720

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28725). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Current version 4.37.1 only match torch>=1.13.0, not torch > 1.11
Author: StrivedTye
Pull request: no

when using torch<1.13.0, current version (4.37.1) will report a OSError, because `torch.load()` in torch==1.12 does not has the keyword of `weights_only`. This issue occurs in the file `modeling_utils, Line 533`.

Comments:
cc @ydshieh 

Sorry, indeed, it's my bad

#28506 didn't fix #27282  properly, I will open a PR

@StrivedTye Fix is merged. I tried it and it works. Let me know if you have further issue with this. Thanks for reporting the issue.
Stop confusing the TF compiler with ModelOutput objects
Author: Rocketknight1
Pull request: yes

The `test_saved_model_creation` test was failing for BLIP with the rather unusual symptom that the `loss` key of one of the intermediate outputs had transformed into a strange generator object. I still don't know **why** this happened, but it requires the following:

1) TF compilation (doesn't happen in eager mode)
2) `ModelOutput` dicts with `loss` as the first, optional key
3) The `ModelOutput` dicts have to be returned internally and then used in a subsequent step, rather than returned as the last step of the outermost model

Since this is a nightmare zone, I'm going to work around the issue by just setting `return_dict` to `False` when calling the sub-model and get the tensors we need from the output tuple instead. This should be invisible for our users!

I also slipped a quick fix into the loss calculation to avoid potential NaNs from passing negative labels to one of the built-in TF loss functions. Even though all the negative-label positions should be masked, NaNs tend to persist (because `nan * 0 == nan`).

Comments:
@amyeroberts sorry, let me explain what I meant with the model output being 'used'!

The issue occurs when the main `BlipModel` calls the `BlipText` sub-model, and the sub-model returns a `ModelOutput` which the main `BlipModel` reads values from. As far as I can tell, the issue does **not** arise if the `BlipText` model is called directly. So this seems to be some issue with the TF compiler not really understanding what kind of object a `ModelOutput` is when it has to compile through one inside a model `call()`.

I ran the slow tests and all passed!
Add w2v2bert to pipeline
Author: ylacombe
Pull request: yes

# What does this PR do?
https://github.com/huggingface/transformers/pull/28165 introduced a new W2V2-based model that uses a different feature extractor than classic CTC-based models.
In particular, it takes mel-spectrograms as `input_features`, instead of raw waveform as `input_values`.

The pipeline only takes `input_values` for this kind of models, which requires a bit of workaround.

Note that I've also run every slow tests from the ASR pipeline, just to make sure.

cc @amyeroberts @sanchit-gandhi 

Also cc @ydshieh, it would be nice to add the model architecture to the tiny model repo !

Comments:
> Also cc @ydshieh, it would be nice to add the model architecture to the tiny model repo !

OK, will update next week (if it succeed to create the tiny model though)

> second approval

nothing big from my side other than 2 tiny comments.

> all slow ASR tests pass

yes, confirmed
Unpin pydantic
Author: ydshieh
Pull request: yes

# What does this PR do?

Unpin pydantic as no failure anymore (CircleCI, docker image build).

Fix #27933

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28728). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Initialize _tqdm_active with hf_hub_utils.are_progress_bars_disabled(…
Author: ShukantPal
Pull request: yes

# What does this PR do?

…) to respect HF_HUB_DISABLE_PROGRESS_BARS

It seems like enable_progress_bar() and disable_progress_bar() sync up with huggingface_hub, but the initial value is always True. This changes will make sure the user's preference is respected implicity on initialization.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28717). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] Fix datasets in guides
Author: stevhliu
Pull request: yes

An issue was raised at https://github.com/huggingface/datasets/issues/6605 that the ELI5 dataset is no longer accessible, impacting the causal/masked language modeling guides. This PR replaces it with the [ELI5-Category](https://huggingface.co/datasets/eli5_category) dataset, which should work fine as a drop-in replacement.

Comments:
[Flax] Update no init test for Flax v0.7.1
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

PR to unblock new model additions in Flax. As of version 0.7.1, Flax defaults to returning regular dictionaries with the methods .init and .apply, not frozen dictionaries as was the case before: https://github.com/google/flax/discussions/3191.

This means our "no automatic init" method returns regular dicts, instead of frozen dicts. Until we merge a bigger change to bring ourselves in-line with Flax (_c.f._ https://github.com/huggingface/transformers/issues/28368#issue-2068557686), we need to update our "no automatic init" test to account for both possible dicts, otherwise we'll have a red CI.


Comments:
cc @ArthurZucker for possible up-coming Flax model additions

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28735). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.