noamaanMulla-03/transformers-issues huggingface dataset:
GPUA10：QWenLMHeadModel does not support Flash Attention 2.0 yet
Author: PolarPeak
Pull request: no

ValueError: QWenLMHeadModel does not support Flash Attention 2.0 yet. Please open an issue on GitHub to request support for this architecture: https://github.com/huggingface/transformers/issues/new

my GPU is A10


transformers>=4.36.2

Comments:
Hey, I don't think Qwen was merged into transformers it's a code on the hub, you should open the issue directly on the model discussion page : https://huggingface.co/Qwen/Qwen-72B/discussions
Data collator does not pass inputs to tokenizer
Author: EricLBuehler
Pull request: no

Hello all,

While attempting to train a model using `Trainer` and `DataCollatorForSeq2Seq`, I get the following error:
`ValueError: You should supply an encoding or a list of encodings to this method that includes input_ids, but you provided []`.

I have written my own model class (this is necessary for the design), which wraps the base model (a `PreTrainedModel`) and has only the `forward`, `generate`, `prepare_inputs_for_generation` and some utility methods.

Am I missing a method to let `transformers` know what inputs I expect? I would greatly appreciate any tips!

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Thanks!

Ok, thanks for the recommendation!
loss did not change in expert loss of mixtral model
Author: LinB203
Pull request: no

### System Info

FROM THIS: https://github.com/huggingface/transformers/issues/28093

If I understand correctly, the loss consists of two parts, the autoregressive categorical loss on the one hand, and balancing the loss of each expert on the other.
I print each of the above two losses before adding up the final one, however, only the first one has a `grad_fn` of `NllLossBackward0`, the second one is just a tensor without `grad_fn` . That's why the `grad_fn` of the final loss is `NllLossBackward0` instead of `AddBackward`.
And the balancing expert's loss doesn't change during the training process. Maybe the changes you see are just changes due to the autoregressive categorical loss.

### Who can help?

@ArthurZucker

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

print each of the above two losses before adding up the final one

### Expected behavior

The expert balanced loss should converge and have `grad_f`.

Comments:
have you fix this problem, i just met the same issue

> have you fix this problem, i just met the same issue

I change to use the deepspeed implementation of moe. I don't know how to fix it in HF. HF's loss is always a constant, while deepspeed's converges to change.

@LinB203 could you point me to the deepspeed implementation of moe? Is it this one : [deepspeed/moe/layer.py](https://github.com/microsoft/DeepSpeed/blob/1787673edc7e45cd79fe10b95f92a02d3eb91505/deepspeed/moe/layer.py#L16) ?

> @LinB203 could you point me to the deepspeed implementation of moe? Is it this one : [deepspeed/moe/layer.py](https://github.com/microsoft/DeepSpeed/blob/1787673edc7e45cd79fe10b95f92a02d3eb91505/deepspeed/moe/layer.py#L16) ?

Yes.

@LinB203  when i run demo ,cannot find preprocessor_config.json on LanguageBind_Video_merge and LanguageBind_Video_Image huggingface.co, can you offen it ?

Traceback (most recent call last):
  File "/home/guowl/.local/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 270, in hf_raise_for_status
    response.raise_for_status()
  File "/home/guowl/.local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/LanguageBind/LanguageBind_Video_merge/resolve/main/preprocessor_config.json


Alright, I'll test again for the masked language modeling loss which should no be None. 
The expert loss is gonna be fixed by #28256
[AttentionMaskConverter] fix sdpa unmask unattended
Author: zspo
Pull request: yes

# What does this PR do?

Keep tensor devices consistent

@ArthurZucker @amyeroberts 

Comments:
> Thanks, can you update the PR name to "[`AttentionMaskConverter`] fix sdpa unmask unattended"

Hi, nailed it！ Thanks

Thanks 🤗
How to use an efficient encoder as shared EncoderDecoderModel?
Author: Bachstelze
Pull request: no
Labels: name: Feature request

### Feature request

Efficient encoder like destilBERT, ALBERT or ELECTRA aren't supported as decoder of the EncoderDecoderModel and so they can't be shared as encoder and decoder.

### Motivation

Warm-starting shared models is a powerful way to build transformer models. Yet the efficient models can't be used.

### Your contribution

We could implement the support for destilBERT, ALBERT or ELECTRA. They shouldn't be that different from other encoders.

Comments:
Deberta as shared encoder and decoder would be also good: https://github.com/huggingface/transformers/issues/12436
MobileVitV2 doesn't have optimizer and loss argument in `model.compile()`?
Author: adhiiisetiawan
Pull request: no

### System Info

- `transformers` version: 4.35.2
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): 2.15.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.5 (cpu)
- Jax version: 0.4.23
- JaxLib version: 0.4.23
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

vision models: @amyeroberts 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Run this official tensorflow [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/transformers_doc/en/tensorflow/image_classification.ipynb#scrollTo=-Pi8UI110qKL) from huggingface and change the model to `apple/mobilevitv2-1.0-imagenet1k-256`. When try to `model.compile()` will result an error `TypeError: compile() got an unexpected keyword argument 'optimizer'`. But, when we delete the arguments, code will run without problem, but of course, we need define optimizer there.

Also when we not pass the arguments, `model.fit` give an error too `AttributeError: 'MobileViTV2Model' object has no attribute 'fit'`

Try with my own notebook for better reproducibility [here](https://colab.research.google.com/drive/1-TFfsun98fKchJRY-ZCV5_tYOwPYLtuX?usp=sharing).

### Expected behavior

MobileViTV2 can running

Comments:
cc @Rocketknight1 

Hi @adhiiisetiawan, the issue here is that when you changed the code, you switched from TF classes to PyTorch classes. Your code uses `MobileViTV2Model`, but this is a PyTorch class. Therefore, the reason that the `compile()` method is behaving incorrectly for you is that you're actually loading a PyTorch model and calling PyTorch's `compile()` method, which does something very different to TensorFlow's `model.compile()`!

Unfortunately, `MobileVitV2` is not yet supported in TensorFlow in `transformers`, so if you want to use this model for now you'll need to do it in PyTorch. You can either use another model (and change the code back to use `TFAutoModelForImageClassification`) if you want to stick with TensorFlow, or you can switch to the PyTorch notebook instead if you want to fine-tune `apple/mobilevitv2-1.0-imagenet1k-256`.

ohh i see, okay thank you @Rocketknight1 for explanation
fix pipeline to support tuple model output
Author: jiqing-feng
Pull request: yes

Hi @Narsil @amyeroberts .

This PR considers pipeline.model may output a tuple (if return_dict=False). Would you please help to review it? Thx!

Problem can be reproduced by
```python
from transformers import pipeline, AutoModel, AutoTokenizer

sentences = ["This is an example sentence", "Each sentence is converted"]
model = AutoModel.from_pretrained('sentence-transformers/all-mpnet-base-v2', return_dict=False)
tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/all-mpnet-base-v2')
extractor = pipeline("feature-extraction", model=model, tokenizer=tokenizer)

extractor(sentences, return_tensors=True, batch_size=2)
```

Comments:
Hi @amyeroberts . Thanks for your review. The default dataclass works well as you said. However, sometimes we would like to enable other optimization like `model = torch.jit.trace(model, inputs)` (which will get a significant acceleration in both GPU and CPU), and it can't recognize the dataclass, so I set `return_dict=False` to return tuple outputs. 

I think considering different types of model outputs would be better since the models support return tuples.

I agree with @amyeroberts in general with this PR.

IMO pipeline shouldn't have any excessive amount of code to support use cases like this. If you really want to wrap a jitted model, wrap you own jitted model to return a dict instead of a tuple.

```python

class Wrapper:
    def forward(self, ...):
        tuple = self.inner.forward(..)
        return ModelOutputXXX(tuple)
```

Also why using `jit.trace` instead of `torch.compile`  ? I though this was the new way to do it.
What kind of speedups are you seeing ?

Hi @amyeroberts @Narsil . Thanks for your review, I propose that since the transformers model supports return tuple, we should support tuple outputs in any other APIs that are related to the model's outputs.

But you were right, I can wrap the model in my script.

Thanks again for your review. Feel free to close this PR if you don't have any questions.

@jiqing-feng Thanks for your time working on this. Following from the discussion, I'm closing this PR.
update warning for image processor loading
Author: ydshieh
Pull request: yes

# What does this PR do?

Similar to #28108, there is not much we/user can do with the files on the Hub.

For this image processor loading, IMO, the attribute `image_processor_class` / `feature_extractor_class` are super unlikely specified by users (they are created automatically during saving for auto class to work). So `logger.info` is fine.

Comments:
cc @NielsRogge 

Thanks for opening this @ydshieh! 

As mentioned in the internal slack channel - I'd rather we had a transition with this: at least two releases instructing the users how to update before downgrading the warnings. Unlike the text_config for CLIP, loading feature extractors for vision is  deprecated and it's necessary for users to change their configs to ensure future compatibility. 

Thanks @amyeroberts Sound fair.

Do you want me to change the warning to include the instructions and the version number involved (i.e. 2 release after since the current one)?



@amyeroberts I'm not sure it's doable to expect users to update their config? That would upset a lot of users.

We have 8000+ `preprocessor_config.json` [files on the hub](https://huggingface.co/models?pipeline_tag=image-classification&library=transformers&sort=trending) => we can't update all of these right?

So my suggestion would be to gracefully handle this internally in the `from_pretrained` method without informing users.

@ydshieh Yes please! 

> We have 8000+ preprocessor_config.json [files on the hub](https://huggingface.co/models?pipeline_tag=image-classification&library=transformers&sort=trending) => we can't update all of these right?

@NielsRogge Certainly not! :) This is why we will downgrade the warning after a cycle. However, we should give users the opportunity to correctly update their configs as necessary by giving them the necessary information as they will not be supported in the future. 

I have to change a bit of the warning message - if you would like to take another look.
Training for ZeroShotImageClassification
Author: hxydxn
Pull request: no

### System Info

- `transformers` version: 4.35.2
- Platform: Linux-5.15.0-83-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.19.1
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@amyeroberts @pacman100 @muellerz 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

[https://colab.research.google.com/drive/1ugxSI63fQd7YvO4IX5H9YrxyY7NDu2il?usp=sharing](https://colab.research.google.com/drive/1ugxSI63fQd7YvO4IX5H9YrxyY7NDu2il?usp=sharing)

### Expected behavior

I am trying to finetune a ZeroShotImageClassification model, particularly `geolocal/StreetCLIP`. I'm encountering the following error:

![err](https://github.com/huggingface/transformers/assets/64323922/4776936b-1297-4c02-9e76-b18bcdbc3bbd)


I'm restricted to a batch size of 1 for train and eval set, otherwise I will get an error:
```Attention mask should be of size (1, 1, 8, 8), but is torch.Size([8, 1, 8, 8])```

I'm currently finetuning on 1M images across test and train sets and hope to train with 8 A40 GPUs.

I based the flow very heavily from [https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb](https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb)
and
[https://stackoverflow.com/questions/75802931/i-cant-fine-tune-clip-model-from-huggingface](https://stackoverflow.com/questions/75802931/i-cant-fine-tune-clip-model-from-huggingface)

Thank you for the help! Any sample code or advice is appreciated!






Comments:
Hi,

The  example you refer to is not meant for zero-shot image classification. It's meant for supervised image classification, when you have a dataset of (image, label) pairs.

CLIP can be fine-tuned using this script: https://github.com/huggingface/transformers/tree/main/examples/pytorch/contrastive-image-text if you want to improve zero-shot image classification on a certain domain.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Apparent Problem deploying to AWS SageMaker Endpoint with VQA Models | Version Incompatability?
Author: buyblvd-ryan
Pull request: no

### System Info

Transformers: 4.26.0
PyTorch: 1.13.
Python: 3.9

Model: Salesforce/blip-vqa-capfilt-large

Deployment code (default from Hugging Face):
```python
import sagemaker
import boto3
from sagemaker.huggingface import HuggingFaceModel

try:
	role = sagemaker.get_execution_role()
except ValueError:
	iam = boto3.client('iam')
	role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']

# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'Salesforce/blip-vqa-capfilt-large',
	'HF_TASK':'visual-question-answering'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.26.0',
	pytorch_version='1.13.1',
	py_version='py39',
	env=hub,
	role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.m5.xlarge' # ec2 instance type
)
``` 

### Who can help?

@amyeroberts @Narsil @gante 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Spin up JupyterLab notebook on AWS Sagemaker
2. Run default deployment code above
3. In SageMaker Studio, navigate to Inference > Endpoints, select the endpoint created and Test Inference with the JSON below
```
{
  "inputs": {
    "question": "[question]"
    "image": "[url to remote image]"
}
``` 

These steps have been tested repeatedly with numerous different inputs.  I was unable to test with a B64 encoded image due to another issue already reported.  The models, methods, questions, and image URLs used to test work without issue locally and in a JupyterLab notebook, but using up-to-date versions of Python (3.11), Torch (2.1.1), and Transformers (4.35.2).  SageMaker doesn't support configurations besides those specified in the default deployment script (3.9/1.13.1/4.26).

### Expected behavior

It shouldn't error out.  Best guess is the error relates to image_processing and feature_extraction in the VisualQuestionAnsweringPipeline in the versions required by SageMaker.


![CleanShot 2023-12-01 at 14 08 22@2x](https://github.com/huggingface/transformers/assets/117312895/5b86ede0-a68f-4245-ac08-1f2e5b526ba6)


Comments:
Hi @buyblvd-ryan 👋 

To be able to help, we would need a reproducible script using `transformers` (and not sagemaker, to rule out that it is not a sagemaker issue) :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[PatchTST] make tests more robust
Author: kashif
Pull request: yes

# What does this PR do?

fix the fragile Slow CI test

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
A100 makes too large check point and slow learning
Author: alexxony
Pull request: no

### System Info

transformers 4.36.0
Python 3.8.10
Driver Version: 535.104.12   
CUDA Version: 12.2

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

best_state_dict = fit(
    model=model,
    train_loader=train_loader,
    valid_loader=valid_loader,
    max_epoch=config.max_epoch,
    lr=config.lr,
    gradient_accumulate_step=config.gradient_accumulate_step,
    early_stop_patience=config.early_stop_patience,
)

model.load_state_dict(best_state_dict)
model.save_pretrained(os.path.join(config.save_dir, "best_model"))
feature_extractor.save_pretrained(os.path.join(config.save_dir, "best_model"))

### Expected behavior


I've run a Audio classification code in 4090, H100, A100. 
In 4090, H100 Code ran well and not large check point , also speed was fast

 When it comes to run in 4*A100,
It made too large check point and was too slow ; I thouth  it is due to **No** pararell process code
So , I  made a container with a A100 GPU
But it caused also same problems

Here is my code and data link 

Can you help me using A100?

Data
https://drive.google.com/file/d/1tKNgHiy-b9_oL8hWG4vpDKePqAv8GHNc/view?usp=drive_link
Code
https://drive.google.com/file/d/1zU0UziwtI8SN7PJD35E7-Tg1NSKnYbSr/view?usp=drive_link

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Thanks!

> Hey 🤗 thanks for opening an issue! We try to keep the github issues for bugs/feature requests. Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!
> 
> Thanks!

I uploaded, but can you help me with your code running?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
`Failed to build tokenizers` when installing 2.5.1 version.
Author: xieyu0214
Pull request: no

`Failed to build tokenizers` when i try to install transformers==2.5.1

`Failed to build tokenizers` when i try to install tokenizers == 0.5.2

so, i want to know `tokenizers == 0.5.2` is must ?

thanks~

Comments:
`tokenizers` is only required when you wish to use the new, fast [tokenizers](https://github.com/huggingface/tokenizers). By default, though, the standard (slower) tokenizers are used. So you do not actually need the `tokenizers` library to run the `transformers` library.

Related:
https://github.com/huggingface/transformers/issues/2980
https://github.com/huggingface/transformers/issues/2831


Can you please open an issue over at https://github.com/huggingface/tokenizers with your OS/Python details?

while I am installing transformer==4.4, I am facing issue: Failed to build tokenizers.