noamaanMulla-03/transformers-issues huggingface dataset:
Populate torch_dtype from a model to a pipeline
Author: B-Step62
Pull request: no

### Feature request

When constructing a pipeline object from a model and a tokenizer, the pipeline doesn't inherit the `torch_dtype` field from the underlying model.

```
model = AutoModelForCausalLM.from_pretrained("t5-small", torch_dtype = torch.bfloat16)
pipeline = pipeline(model=model, task="text-generation", tokenizer=...)

print(pipeline.torch_dtype)
=> None
```

However, it would be more convenient if the constructor extract the dtype from the model and populate it to pipeline's `torch_dtype` field. I think it's safe to assume the store model's dtype as pipeline's `torch_dtype` based on the documentation.

> Sent directly as model_kwargs (just a simpler shortcut) to use the available precision for this model (torch.float16, torch.bfloat16, … or "auto").

We should be able to determine model's dtype either from `model.config.torch_dtype` or `next(model.parameters()).dtype`.



### Motivation

I'm a maintainer of [MLflow](https://github.com/mlflow/mlflow/tree/master) and we have a logic to save metadata of Transformers pipeline, such as torch_dtype, task, etc. Since the pipeline doesn't populate `torch_dtype` field from the model, we need to check the underlying model's parameters. While we've implemented [a custom extraction logic](https://github.com/mlflow/mlflow/pull/10979) in our code base, I think this capability could be beneficial for other users of Transformers as well.

### Your contribution

I can submit a PR.

Comments:
cc @Rocketknight1 WDYT? Sounds good to me 

This sounds like a safe assumption to me too, though obviously I'd like to confirm that with some tests! I'm in favour of the PR if you're happy to open it @B-Step62 

@ArthurZucker @Rocketknight1  Great! I will open a PR soon, in the meantime could you assign the issue to me?

@B-Step62 Done!

cc @Rocketknight1 we usually don't assign issues, and rather let the code talk: if a PR is open and pinned then that means someone is working on something and we can check the progress 😉 

Hi @Rocketknight1 @ArthurZucker! I just opened a PR ^, please take a look whenever you have time, thanks!
[Regression] Yi 200K models won't load in latest release
Author: Mihaiii
Pull request: no

### System Info

- `transformers` version: 4.38.1
- Platform: Linux-5.4.0-167-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no (but it also doesn't work if I do)
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Simply try to load a 200K context length Yi model in latest transformers ([v4.38.1](https://github.com/huggingface/transformers/releases/tag/v4.38.1)) and it will crash:

I just retried "01-ai/Yi-6B-200K" to be 100% sure:

```python
model = AutoModelForCausalLM.from_pretrained("01-ai/Yi-6B-200K")
```

Everything is ok when trying to load [01-ai/Yi-6B](https://huggingface.co/01-ai/Yi-6B).
I did not also test [01-ai/Yi-34B-200K](https://huggingface.co/01-ai/Yi-34B-200K), but I tried finetunes of it and they failed so I'm assuming Yi-34B-200K fails too.

This is a regression because loading a 200K model works ok with the release of transformers from ~3weeks ago ([v4.37.2](https://github.com/huggingface/transformers/releases/tag/v4.37.2)).


### Expected behavior

The expected behavior is for the model to load, like it does in previous released versions of the transformers library.

Comments:
Hey! 
I am running this and so far I do not have any issues. I think you don't have enough memory to run it. That is a regression indeed, and a fix is coming to better cater with caching the `attention_mask`.
TLDR; you don't have enough ram to use 200K anyways (as this would always require creating a 200K causal mask) 

Related to #29263, I think we'll default a smaller value to make sure this does not take too much ram, but also using bool will help see #29241 that should have helped

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

This was fixed by #29753

Sorry for the inconvenience
Unexpected warning of SlowTokenizer
Author: hiyouga
Pull request: no

### System Info

- `transformers` version: 4.38.1
- Platform: Linux-5.15.0-91-generic-x86_64-with-glibc2.35
- Python version: 3.10.10
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: 0.26.1
- Tokenizers version: 0.15.2
- PyTorch version (GPU?): 2.1.1+cu121 (True)

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
import transformers
from transformers import AutoTokenizer
transformers.utils.logging.set_verbosity(transformers.logging.INFO)
transformers.utils.logging.enable_default_handler()
transformers.utils.logging.enable_explicit_format()
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf", use_fast=False)
# [INFO|tokenization_utils_base.py:2044] 2024-02-23 17:26:31,129 >> loading file tokenizer.model
# [INFO|tokenization_utils_base.py:2044] 2024-02-23 17:26:31,130 >> loading file added_tokens.json
# [INFO|tokenization_utils_base.py:2044] 2024-02-23 17:26:31,130 >> loading file special_tokens_map.json
# [INFO|tokenization_utils_base.py:2044] 2024-02-23 17:26:31,130 >> loading file tokenizer_config.json
# [INFO|tokenization_utils_base.py:2044] 2024-02-23 17:26:31,130 >> loading file tokenizer.json
tokenizer.encode("hello")
# [WARNING|tokenization_utils.py:562] 2024-02-23 17:26:41,845 >> Keyword arguments {'add_special_tokens': False} not recognized.
# [1, 22172]
tokenizer.encode("hello", add_special_tokens=False)
# [WARNING|tokenization_utils.py:562] 2024-02-23 17:26:58,903 >> Keyword arguments {'add_special_tokens': False} not recognized.
# [22172]
```


### Expected behavior

Do not show redundant warnings like the fast one.


Comments:
Thanks! That is indeed an issue and should be fixed! I tracked it but could not reproduced 🤗 Thanks

The same issue

@ArthurZucker This issue is a duplicate of #29237 and can be fixed in #29278 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

For context, #29346 fixed this, thanks @hiyouga for your initial PR
Add LLaVa 1.6
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds the new LLaVa 1.6 model.

To do:

- [x] not sure how batched generation works => is supported in case `image_sizes` are the same for all images
- [x] make `image_sizes` a tensor instead of a list
- [x] make sure llava 1.5 still works

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29012). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks, I've addressed all comments!

Will update the conversion script to use cuda:0 when all checkpoints are uploaded (34b models still need to be uploaded)

Ok thanks for your review.

> 🔴 changes to the LlavaForCausalLM are just to big to say it's still Llava. We can either have a new forward, or we do what we always do, add a new "Architecture" to properly seperate both. I don't mind having it in the same file, but the expected input is different from normal Llava.

I feel like adding a whole new architecture is a bit of an overkill, as the only change of llava 1.6 is that pixel values are now of shape 5 instead of 4 (batch_size, num_patches, num_channels, height, width). Llava 1.6 is also still part of the [same paper](https://static.hliu.cc/files/llava/improved_llava.pdf), they just extend it to work on arbitrary image resolutions.

Could you clarify what you mean by a new forward, since the model LlavaForConditionalGeneration can only have a single forward method defined, I assume. Do you mean defining a whole new class as well?

> 🔴 changes are not explained in the slightest, so coming from the llava code I wrote I have no idea what is happening, why we need to iterate of the images etc. This needs to be explained and needs a separate code-path

I'll add some more comments to explain what's happening.

Closing this PR in favor of #29586
Handle padding warning in generation when using `inputs_embeds`
Author: zrthxn
Pull request: yes

# What does this PR do?

Fixes: #23042
if `input_ids` was given, check if the last id in any sequence is `pad_token_id`
if `inputs_embeds` was given, check if the last embed in any sequence is *all* zeros

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? [Please add a link
      to it if that's the case.](https://github.com/huggingface/transformers/issues/23042)
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
cc @gante

_The documentation is not available anymore as the PR was closed or merged._

> Why not simply add `and len(inputs_tensor.shape) == 2` to the `if`? Short, clear code is easier to maintain 🙌

Just adding `and len(inputs_tensor.shape) == 2` wouldn't print the warning if using `inputs_embeds`. Would you prefer that behaviour? Also, this logic would fail if the pad token for some embedding is **not** a tensor full of the `pad_token_id`.

> Just adding and len(inputs_tensor.shape) == 2 wouldn't print the warning if using inputs_embeds. Would you prefer that behaviour? Also, this logic would fail if the pad token for some embedding is not a tensor full of the pad_token_id.

Yeah, I'd prefer the shorter version with the `len(inputs_tensor.shape) == 2` check only. The reason being that although it is less precise, using the embeddings as an input is an advanced use case, for which we tend to be more hands-off. It also makes the code shorter and, therefore, more readable :)

(if we were to make complete checks at all points, the code would quickly become unmaintainable)

Alright, I've changed the logic to be very simple now. It just doesn't check this condition if `inputs_embeds` was passed.

@zrthxn this PR needs to be rebased with `main` -- we fixed a dependency issue that's showing up on the CI down here 😬 apologies for the extra work!

@gante no problem! Just did that.
Security Awareness for users on Security Policy
Author: diogoteles08
Pull request: no

### Feature request

Hi, I'm Diogo from [GOSST](https://github.com/diogoteles08#about-gosst-ghost) and I'd like to suggest that Transformers adopt a security policy that not only allows security researchers to privately report security vulnerabilities in Transformers, but also informs users of common security practices they should consider when using it.

### Motivation

This information will benefit:

1. the user, that will have guidelines on how to safely run a model for their application
2. the project, that can avoid receiving false positive vulnerability reports

### Your contribution

I'll send a PR along with this issue with a initial version of a Security Policy that will keep the current guidelines on how to report vulnerabilities, but also expose security guidance on how to deal with data privacy, untrusted models, untrusted inputs, etc.

Thanks!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
add model `mae_st` masked spatiotemporal
Author: innat
Pull request: no
Labels: name: New model

### Model description

[Paper 2022](https://arxiv.org/abs/2205.09113) abstract 

Cited by 171

> This paper studies a conceptually simple extension of Masked Autoencoders (MAE) to spatiotemporal representation learning from videos. We randomly mask out spacetime patches in videos and learn an autoencoder to reconstruct them in pixels. Interestingly, we show that our MAE method can learn strong representations with almost no inductive bias on spacetime (only except for patch and positional embeddings), and spacetime-agnostic random masking performs the best. We observe that the optimal masking ratio is as high as 90% (vs. 75% on images), supporting the hypothesis that this ratio is related to information redundancy of the data. A high masking ratio leads to a large speedup, e.g., > 4x in wall-clock time or even more. We report competitive results on several challenging video datasets using vanilla Vision Transformers. We observe that MAE can outperform supervised pre-training by large margins. We further report encouraging results of training on real-world, uncurated Instagram data. Our study suggests that the general framework of masked autoencoding (BERT, MAE, etc.) can be a unified methodology for representation learning with minimal domain knowledge.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://github.com/facebookresearch/mae_st

Comments:
Hi @innat, thanks for opening this new model request! 

We have recently been trying to push for `model on the hub` and have as much support as we can there. It will also be easier to integrate it! Here is a [tutorial](https://huggingface.co/docs/transformers/custom_models) if that sound good to you or anyone who wishes to contribute this model! 

Hi, I start working on this issue. 

@Natyren check [this](https://github.com/facebookresearch/SlowFast/issues/668) in case you face the same.
Add Arabic translation for README
Author: strikoder
Pull request: yes

# What does this PR do?
This PR introduces the Arabic translation of the README file.

## Before submitting
- [x] This PR improves the docs.

## Fixes #29045 

 @stevhliu 
@MKhalusova 

Comments:
I've addressed the duplicates and verified the Markdown formatting; all links are functioning correctly. Screenshots documenting each issue you highlighted have been attached for reference. 

Regarding the Arabic langauge, I don't really know who speaks Arabic at Hugging Face, could you please help me with that?

Great, thanks!

I would join our [Discord](https://discord.gg/hugging-face-879548962464493619) and see if any community members there would be interested in reviewing the translation. You can also try asking on our [forums](https://discuss.huggingface.co/) as well :)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29052). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

LGTM 💯
I am Egyptian and I speak Arabic.
Everything is correct  
the work of @Strikoder is right and important for Arabic Language.
thanks to you 👍🏻 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[pipeline][video-classification] a question about the use of decord in VideoClassificationPipeline
Author: Tyx-main
Pull request: no
Labels: name: Feature request

name: Vision

### System Info

transformers:4.37.2
platform: aarch


### Who can help?

@Narsil@sanchit-gandhi

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

It seems that when we use pipeline for VideoClassification, an issue can not be avoided that a third party named decord can only be used in x86/arm instead of aarch, but my machine only supports aarch, is there any method to solve this problem?

### Expected behavior

VideoClassificationPipeline can be used in aarch platform

Comments:
@Narsil @sanchit-gandhi

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @Tyx-main, apologies for the delay in response.

Indeed, decord can cause some issues. In fact, we removed it from a bunch of our examples in https://github.com/huggingface/transformers/pull/21572. The fix would be to use pyav in the pipelines as well. Would you like to open a PR to add this?

> Hi @Tyx-main, apologies for the delay in response.
> 
> Indeed, decord can cause some issues. In fact, we removed it from a bunch of our examples in #21572. The fix would be to use pyav in the pipelines as well. Would you like to open a PR to add this?

Thanks！I will do it soon！

Closing as resolved in #29747
Gemma optimizations for finetuning and infernece
Author: rombodawg
Pull request: no
Labels: name: Feature request

### System Info

Latest transformers version, most platforms. 

### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Finetuning, inference

### Expected behavior

Recently it was discovered that poor optimization in transformers has been taken place, and there are ways to fix it. Bellow is a link on the findings
https://www.reddit.com/r/LocalLLaMA/comments/1bd18y8/gemma_finetuning_should_be_much_better_now/

Comments:
When i run with transformers==4.36.2 for LLama2, Mistral => Inference very ok
But, when run with upgrade version (4.38.2) => Very slow

Please check issues

Hi @phanxuanphucnd, could you open a new issue? This helps us keep better track of what has been resolved and addressed. The original issue post is a feature request, whereas you appear to be reporting a possible bug or regression

In the issue, can you make sure to include: 
* a minimal code snippet to reproduce the issue, or specifically the benchmarking 
* Information about the issue, in this case numbers about the running times between the two versions

Hi @phanxuanphucnd I meet the same problem. Has it been resolved? Thanks!

> Hi @phanxuanphucnd I meet the same problem. Has it been resolved? Thanks!

Down version transformers back to 4.36.2. :< 

@phanxuanphucnd Wait are you saying Llama and Mistral themselves are slower now? That's very interesting - the Gemma bug fixes I did do make finetuning itself somewhat slower, but the results become accurate. I would say 5% at maximum. I also fixed the Llama RoPE issue, so another 5% hit.

However, for inference, recently the smart HF team made inference fully compile-able with `torch.compile`, enabling 4x faster inference - this might cause non compiled codepaths to be slower.

However, to my knowledge, Mistral itself was not touched, nor did I include changes to update Mistral - this leads me to suggest some other issue is causing the slowdown.

So from the back of the envelope (ie just guess) how much % slower is Mistral, Llama and or Gemma?

Hey all! #29753 this should have fixed the issues no?  cc @phanxuanphucnd

> Hey all! #29753 this should have fixed the issues no? cc @phanxuanphucnd

Thank you @ArthurZucker  . The problem has been fixed @danielhanchen 

Closing then! Thanks all for the input
Fix header in IFE task guide
Author: merveenoyan
Pull request: yes

Zero idea on why it happened but it didn't render properly
Also when you approve please feel free to merge as I can't merge

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29859). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks
Support download of models from Torrent 
Author: filopedraz
Pull request: no

### Feature request

### Description

Support models download from Torrent passing as a parameter in the `AutoTokenizer` and `AutoModel` class `magnet_url`.

### Example Usage

```python
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("magnet:?xt=urn:btih:XXXXXX")
model = AutoModel.from_pretrained("magnet:?xt=urn:btih:XXXXXX")

inputs = tokenizer("Hello world!", return_tensors="pt")
outputs = model(**inputs)
```

### Motivation

Models could be censored and removed from HF. In order to keep them alive, Torrent could be a solution.

### Your contribution

I can do a POC of the implementation using a model mirrored from HF to Torrent. 

Comments:
Hi! I'm new to open source, can you please help me understand the issue better? I understand I have to show a POC for this, but has this functionality already been implemented, or would I have to make it myself? 
Thank You in Advance!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey! This is currently not planned 🤗 

That would be much better than the current `git lfs clone` way.

Today I've spent several hours downloading a model that had about 64 GiB of pytorch weights, and after 72 GiB of GIT LFS downloading it still was on 14%.

And all of that downloading is lost after an accidental disconnect because the files are written to the directory only when downloaded completely. And this process could not be resumed from cache, AFAIK.

This is just unusable.

Hey @tsilvs, for a while we've been recommending the HTTP way to download models rather than the `git lfs clone`, which is, as you have seen, much slower. Read about it here: https://huggingface.co/docs/huggingface_hub/concepts/git_vs_http#what-should-i-use--and-when-

Additionnally, you may be interested in [this guide](https://huggingface.co/docs/huggingface_hub/v0.22.0/en/guides/download#faster-downloads) which goes over optimizing downloads for speeds using the `hf-transfer` utility to take advantage of your full bandwidth.