noamaanMulla-03/transformers-issues huggingface dataset:
Llama model not generating mentioned "max_tokens" number of tokens as output 
Author: vineel96
Pull request: no
Labels: name: Usage

name: bug

name: Generation

### System Info

- `transformers` version: 4.36.0
- Platform: Linux-5.14.0-427.18.1.el9_4.aarch64-aarch64-with-glibc2.34
- Python version: 3.9.18
- Huggingface_hub version: 0.25.1
- Safetensors version: 0.4.5
- Accelerate version: 0.34.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker,  @zucchini-nlp , @Rocketknight1 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Code:

import transformers
import torch

model_id = "meta-llama/Meta-Llama-3-8B"
pipeline = transformers.pipeline("text-generation", model=model_id, model_kwargs={"torch_dtype": torch.float16}, device_map="auto")
output = pipeline("There are two persons named ram and krishna", max_new_tokens=10000, do_sample=True)
final_op = output[0]["generated_text"]
print(final_op)


### Expected behavior

Expected output: generated output length should be 10000 tokens 
Real output: generated output length is less than 10000 tokens

How to get full 10000 tokens as output?

Comments:
Hey! Parameter `max_new_tokens` controls only the upper bound when generating. If you want to generate until the max possible tokens, you can either set `eos_token_id` in `model.config` and `model.generation_config`  to `None` or pass a `min_new_tokens=1000`. But note that in the first case there is a risk of going in infinite loop in no upper bound max tokens are set 

@zucchini-nlp Can we set a default value of around 1000 for max_new_tokens? This will prevent the model from running into an infinite loop when we set eos_token_id in both model.config and model.generation_config to None, and no upper limit for max tokens is defined.

Sure, if the default is set through model's generation config, you don't have to set it manually. I just wanted to note the possibility ofan infinite loop but setting eos-token to `None` is okay as long as you do it intentionally

@zucchini-nlp , Thanks
Arrow files too large when training SegFormer
Author: MRNOBODY-ZST
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.42.4
- Platform: Windows-11-10.0.22631-SP0
- Python version: 3.12.4
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.3.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Fine_tune_SegFormer_on_custom_dataset.ipynb
- GPU type: NVIDIA GeForce RTX 4090

### Who can help?

@amyeroberts

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Make a custon dataset to train SegFormer
I'm using a dataset around 33000 images, 26000 for training 7000 for validation
When training started, the transformer will continuously writing cache files (with speed around 50mb/s) to C:/Users/<Username>/.cache/huggingface/metrics/mean_io_u/default
Which name is default_experiment-1-0.arrow
When it reaches 50 epochs, the file size is about 707GB.
I don't know why.

### Expected behavior

Disk space drainage, the training can't be continued.

Comments:
Hi @MRNOBODY-ZST Thanks for reporting! Let me check this!

@qubvel Really Appreciate This!!!
By the way, i'm using the training script in this [repo](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SegFormer).


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Don`t have the bandwidth to look into this, the alternative way might be to use [torchmetrics for mIoU calculation](https://lightning.ai/docs/torchmetrics/stable/segmentation/mean_iou.html)
Mixtral manual `head_dim`
Author: wavy-jung
Pull request: no
Labels: name: Usage

name: Feature request

### Feature request

https://github.com/huggingface/transformers/blob/816f4424964c1a1631e303b663fc3d68f731e923/src/transformers/models/mixtral/modeling_mixtral.py#L284
`head_dim` in `mixtral` model is forced to have the value of `hidden_size // num_heads`. However, this it not the case in [`llama` model](https://github.com/huggingface/transformers/blob/e95ea479eebb6e01679907db910b5dc5eb64b3c7/src/transformers/models/llama/modeling_llama.py#L290) or even in [`mistral` model](https://github.com/huggingface/transformers/blob/816f4424964c1a1631e303b663fc3d68f731e923/src/transformers/models/mistral/modeling_mistral.py#L194). So, it will be a good minor feature to support manual `head_dim` setting for `mixtral` model as well!

### Motivation

- manual `head_dim` in llama or mistral model

### Your contribution

PR

Comments:
manual `head_dim` for `mixtral` model
Author: wavy-jung
Pull request: yes

# What does this PR do?
- manual `head_dim` setting for `mixtral` model
- same as `llama` and `mistral` implementation
- worked well when I tested with sparse upcycled llama model

Fixes #34261 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ArthurZucker @gante

Comments:
@ArthurZucker I literally have no idea why the CI check fails at glm modeling code. I did not make any changes to the glm models. Can you help me with this or can you give me a brief guideline?

`make fix-copies` might be needed as mistral is copied here and there

@ArthurZucker I did `fix-copies` & `rebase`. It seems all the checks are passed.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34281). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Use unidic-lite instead of ipadic for Japanese tokenization
Author: KanTakahiro
Pull request: no
Labels: name: Core: Tokenization

name: bug

### System Info

- `transformers` version: 4.44.0
- Platform: Linux-6.9.3-76060903-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.23.3
- Safetensors version: 0.4.3
- Accelerate version: 0.31.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.4.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA GeForce RTX 2080 Ti

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

In line 383 of this file `src/transformers/models/bert_japanese/tokenization_bert_japanese.py`, the default dictionary is set to be `ipadic` and I have to install `ipadic-py`. But `ipadic-py`'s [GitHub page](https://github.com/polm/ipadic-py) said "You Shouldn't Use This" and recommend using UniDic.
However, although I installed `unidic-lite` only, transformers still need `ipadic`. I have to modify the transformers source code to use `unidic-lite`. I changed the  line 383 of `src/transformers/models/bert_japanese/tokenization_bert_japanese.py`:
```
- mecab_dic: Optional[str] = "ipadic",
+ mecab_dic: Optional[str] = "unidic-lite",
```
I think the official version should also be updated to use `unidic-lite` for a modern Japanese tokenization.

My script:
```
import random
import glob
from tqdm import tqdm

import torch
from torch.utils.data import DataLoader
from transformers import BertJapaneseTokenizer, BertForSequenceClassification
import pytorch_lightning as pl

MODEL_NAME = 'cl-tohoku/bert-base-japanese-whole-word-masking'

tokenizer = BertJapaneseTokenizer.from_pretrained(MODEL_NAME)
bert_sc = BertForSequenceClassification.from_pretrained(
    MODEL_NAME, num_labels=10
)
# bert_sc = bert_sc.cuda()

tokenizer = BertJapaneseTokenizer.from_pretrained(MODEL_NAME)
```

### Expected behavior

Change the default dictionary for Japanese tokenization from `ipadic` to `unidic-lite`.

Comments:
Hey! Sorry for the delay, would you like to open a PR for a fix? As long as the outputs are unaffected this would be nice indeed! 

Hello! I have just open a [PR](https://github.com/huggingface/transformers/pull/33372) for this issue. Please check it and told me if there is anything I need to adjust or improve.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing this because #33372 has been merged!
Fix onnx non-exportable inplace aten op
Author: IlyasMoutawwakil
Pull request: yes

# What does this PR do?
All of these changes were introduced in `4.46` via #33586, which broke them


Comments:
@ArthurZucker 

same as https://github.com/huggingface/transformers/pull/31696

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34376). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Could you run make fix-copies? 

neat ! love this command 😄

feel free to merge!
'drop_path` argument for SwinStage class is unused. 
Author: deo-abhijit
Pull request: no
Labels: name: bug

name: Vision

### System Info

Nothing related to system 

### Who can help?

 @amyeroberts, @qubvel

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

-

### Expected behavior

I feel like i have found some minor errors in swin transformer code.

The SwinEncoder class has layers attribute which is a ModuleList of SwinStage objects.  https://github.com/huggingface/transformers/blob/38f9f10dd9240619ea17fb6c7acb51b3bc592232/src/transformers/models/swin/modeling_swin.py#L821-L840

SwinStage class takes in `drop_path` as input argument, but in SwinStage class constructor defination, this argument is not being used anywhere.   

https://github.com/huggingface/transformers/blob/38f9f10dd9240619ea17fb6c7acb51b3bc592232/src/transformers/models/swin/modeling_swin.py#L762-L780

Probability for DropPath is picked from the config  in DropPath layer.

https://github.com/huggingface/transformers/blob/38f9f10dd9240619ea17fb6c7acb51b3bc592232/src/transformers/models/swin/modeling_swin.py#L637-L750



Comments:
In general this is quite ugly indeed, the SwinLayer should only take: config and layer_ids. It can then do `config.num_heads[i_layer]` inside the `__init__` reducing the complexity. 

Would you like to open a PR to fix this? 🤗 


Sure. I would love to!
Expand AcceleratorConfig to accommodate other features such as NCCL timeout etc
Author: kmehant
Pull request: no
Labels: name: Feature request

### Feature request

Expand `AcceleratorConfig` and corresponding transformers trainer args to allow transformer users to use full feature set of accelerate through the config arguments supported by `Accelerator()`. The args are being materialized here for use - https://github.com/huggingface/transformers/blob/a769ed45e17c44fd17b85c025863c4e4f2f73634/src/transformers/trainer.py#L5000


### Motivation

When using `HF/transformers` or `HF/trl SFTTrainer` with accelerate under the hood, its sad that only a limited set of arguments are exposed in `AcceleratorConfig` thereby not having control over using other features of the `Accelerator` such modifying the NCCL timeout. 

### Your contribution

I will be glad to raise a PR to expand AcceleratorConfig to enable wide array of arguments supported by `Accelerator`. 

Comments:
cc @muellerzr I think! (Zach I'm new to being on watch so feel free to yell at me if this is someone else's problem)

Certain parameters are exposed via the `TrainingArguments` class, for example `--ddp_timeout` configures the `timeout` argument of `torch.distributed.init_process_group()` method which in turn controls the NCCL timeout.

Ref: https://github.com/huggingface/transformers/blob/e9ad46049411624bb1b6e830fbc1138991c0135e/src/transformers/training_args.py#L736-L740

Thanks @VassilisVassiliadis 

Adding to this, Since some of the args are outside the AccelerateConfig, is this because the accelerate CLI interface does not support certain args like timeout?
[Demo][ExecuTorch] Lower and run native Gemma e2e in ExecuTorch
Author: guangy10
Pull request: yes
Labels: name: WIP

name: ExecuTorch

This PR is a prototype to showcase the minimal changes required to lower Gemma-2b to ExecuTorch w/ static kv cache and run it directly in [llama runner](https://github.com/pytorch/executorch/tree/main/examples/models/llama2) w/o single line of code change in the ExecuTorch runtime.

By standardizing on the contract between HuggingFace modeling and ExecuTorch runtime, any LLM in HuggingFace could utilize llama runner as a universal runtime for a given backend.

Instructions to run the demo:

To run the demo, you need follow this [guide](https://pytorch.org/executorch/main/getting-started-setup.html) to install ExecuTorch, patch [PR#4088](https://github.com/pytorch/executorch/pull/4088) to include the script export_hf_model.py there to export and lower the model to XNNPACK backend. From there, you can:

1. Run the export_hf_model.py to lower gemma-2b to ExecuTorch:
```
cd executorch # In the root dir in executorch
python -m examples.models.export_hf_model -hfm "google/gemma-2b" --export  # The model is exported statical dims with static KV cache
```
2. Run the tokenizer.py to generate the binary format for ExecuTorch runtime:
```
python -m examples.models.llama2.tokenizer.tokenizer -t <path_to_downloaded_gemma_checkpoint_dir>/tokenizer.model -o <your_out_dir>/tokenizer.bin
```
3. Build and run the lowered model wiht llama runner by following this guide [step 4](https://github.com/pytorch/executorch/tree/main/examples/models/llama2#step-4-run-on-your-computer-to-validate)

![Screenshot 2024-07-23 at 2 09 56 PM](https://github.com/user-attachments/assets/d3930556-d900-4cc4-803e-f5ca77d31cf5)


NOTE: This prototype is to demonstrate the feasibility of exporting and running native HF model in ExecuTorch by reusing llama runner. The demo shown in the screenshot is using [XNNPACK delegation](https://pytorch.org/executorch/main/native-delegates-executorch-xnnpack-delegate.html) running the fp32 model on a Linux host. 

Comments:
Gemma and Gemma2 have been enabled already. This demo PR can be closed.
MobileBERT is ExecuTorch compatible
Author: guangy10
Pull request: no
Labels: name: Feature request

name: ExecuTorch

### Feature request

Enable MobileBERT to ["Export to ExecuTorch"](https://github.com/huggingface/transformers/issues/32253) workflow


### Motivation

See details in #32253

### Your contribution

Enable MobileBERT

Comments:
fix incorrect warning
Author: yonigozlan
Pull request: yes

# What does this PR do?
Follows this PR https://github.com/huggingface/transformers/pull/34285
The warnings are displayed for all kwargs not in common, even if they are used in other modalities.
This PR only changes the logic of the warning, the kwargs handling logic should already be correct.


<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->



## Who can review?

Cc @molbap , sorry I think you may have hinted at this problem previously but I did not pick up on it.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34416). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Tests are failing for unrelated (I think) reasons
Un-deprecate timeout arg in pipelines
Author: Rocketknight1
Pull request: yes

As part of the pipeline sync project, this PR allows some arguments in `transformers` that don't have to be in the Hub spec. This is limited to arguments that do not actually affect the core pipeline, like `timeout`, which controls the timeout for downloading images passed as path arguments.

Comments:
cc @LysandreJik @Wauplin for review!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34382). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

(x-linking https://github.com/huggingface/transformers/pull/34290#issuecomment-2432768995 where this PR was discussed)
Albert is executorch compatible
Author: tibor-reiss
Pull request: yes

Albert is executorch compatible, see #33836 

Testing
`RUN_SLOW=1 pytest tests/models/roberta/test_modeling_roberta.py -k test_export -v`

E2E test in ExecuTorch
Needs adjusment [here](https://github.com/pytorch/executorch/pull/6509/files#diff-6cd1385d8080276c875ed72e3fb5f07fa545644273aeea4e3eef6795b20bf337R46) because `use_cache` is not recognized by Albert.
Suggestion: make `use_cache` part of args.


- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. #33836 
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


cc @guangy10 

Comments:
cc @ArthurZucker @qubvel who looked at the other executorch PRs!
solve issue #32575
Author: YOUSEFNANIS
Pull request: yes

The issue in #32575 is caused by the replace function, which replaces " 's" with "'s". To resolve this, we can add a space after " 's", changing it to " 's ". This ensures that the replace function changes it to "'s ". By doing this, we can ensure that "'s" is not mistakenly identified as the end of a word. If "'s " appears at the end of a sentence, followed by a period, the replace function will automatically remove the space before the period

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32670). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.