noamaanMulla-03/transformers-issues huggingface dataset:
Register a custom tokenizer with AutoTokenizer
Author: rahular
Pull request: no

### System Info

(Possible duplicate: #10256)
I have written a custom tokenizer that builds on top of `BertTokenizer` (returns one extra list of ids that will later be embedded in a custom model). I have pushed it to Hub as well. Now, how can I allow others to use it? The code for the tokenizer is uploaded to Hub along with the code for the model (they are in the same file), but since I cannot register the tokenizer with `AutoTokenizer` like I can do for models (`CustomModel.register_for_auto_class("AutoModel")`), others cannot load this tokenizer, and hence use the model.

Is there a workaround for this?

Version: 4.27.4

### Who can help?

@ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The code for both the tokenizer and model can be found here: https://huggingface.co/mcgill-babylm/bert_ds10M_np512_nh2_nl2_hs128_postags_ungrouped/blob/main/pos_bert.py

I am able to load the model with no problems since I push it after registering it as follows
```
BertForMaskedLMWithPOSEmb.register_for_auto_class("AutoModel")
BertForMaskedLMWithPOSEmb.register_for_auto_class("AutoModelForMaskedLM")
```

### Expected behavior

I should be able to register custom tokenizers with `AutoTokenizer` (which might be a new feature request) or work around it somehow to allow other users to use a custom tokenizer.

Comments:
You can have a look at the [documentation here](https://huggingface.co/docs/transformers/custom_models) but this is already supported. Just do `CustomTokenizer.register_for_auto_class()` like for the models.

Duh! I was doing this for the models but didn't make the connection to the tokenizer. Thanks @sgugger!

For someone looking for the complete answer in the future:
```
CustomTokenizer.register_for_auto_class("AutoTokenizer")
```

@rahular  Hi . I am doing the same thing but I am unable to do this can you help me by giving the code of how to integrate the tokenizer with huggingface?

Same 
https://github.com/huggingface/transformers/issues/23072#issuecomment-1819028435

could you explain what you are unable to do with a reproducer and a repo on the hub? 🤗 

I've figure it out. I'm trying to use InternLM2 from AutoClass.
For 
transformers    4.34.1
tokenizers        0.14.1
`CustomTokenizer.register_for_auto_class("AutoTokenizer") `  doesn't work. 
However, the following works:
`
AutoConfig.register("internlm2", InternLM2Config)
AutoTokenizer.register(InternLM2Config, InternLM2Tokenizer, InternLM2TokenizerFast)
AutoModel.register(InternLM2Config, InternLM2Model)
AutoModelForCausalLM.register(InternLM2Config, InternLM2ForCausalLM)
`

Then I'm able to finish
`
tokenizer = AutoTokenizer.from_pretrained("path")
`

Great that you were able to debug. Do you want to open a PR to add more details about registering the tokenzier to help others in the community? 🤗
[`Refactor Attention mask handling`] Moves attention mask processing to the Attention class
Author: ArthurZucker
Pull request: yes

# What does this PR do?
This is more aligned with our philosophy, but also simplifies and will simplify things.
Will help a lot with the static cache.

The only way to share the mask is to call `LlamaAttention` but if you have a better way I'll update it! 
This makes the attention class self contained, which is also pretty convenient for testing. 
Ran the slow test without fa2 will run them again on dgx once approved. 

cc @patrickvonplaten for visibility 

Comments:
We added support for 4d attention mask inside the converter so should be alright but yeah will check related issues! 

Can you specify how this helps with the static cache?

The static cache should also work with the attention_mask being passed at every forward call (it'll always have the same shape). I don't think it's a good idea to have the `attention_mask` be a class variable. 

It will not be a class variable forgot to update but I'll follow what we do with jax.
This will help as the cache length is different from the number of tokens that are seen which you get when you are in the attention layer. 

Can give more details but basically new cache + attention was not behaving properly. This is gonna be my priority this week anyway! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
The model has parameters that do not require training, causing training to be interrupted.
Author: xmy0916
Pull request: no

### System Info

- `transformers` version: 4.31.0
- Platform: Linux-5.4.143.bsk.8-amd64-x86_64-with-glibc2.28
- Python version: 3.10.13
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.4.0
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
def build_vision_projector(config, delay_load=False, **kwargs):
    return nn.ModuleDict({
        "image": nn.Linear(config.mm_hidden_size, config.hidden_size),
        "video": nn.Linear(config.mm_hidden_size, config.hidden_size)
    })
```

i have a projector layer which will match the input tensor by the key image or video. above code will cause the training to get stuck.

when i modify the code like:
```python
def build_vision_projector(config, delay_load=False, **kwargs):
    proj = nn.Linear(config.mm_hidden_size, config.hidden_size)
    return nn.ModuleDict({
        "image": proj,
        "video": proj
    })
```

it means that all inputs use the same projector layer. This way can train normally.

### Expected behavior

The training error is:
![image](https://github.com/huggingface/transformers/assets/43675899/0c2163b2-d9f4-4702-ae19-e4b1eef7830f)

Some batches of my training data are a mixture of videos and images, some have only videos, and some have only images. This will result in the projector layer of some batches of video projector layer or image projector layer not being forwarded. It seems that this is the reason why the training is stuck.

Is there any solution?

Comments:
Hey! No idea as we don't have access to the full custom code, but would recommend you to use the latest version of transformers, and share a full snippet + traceback otherwise we can't help you 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix `FastSpeech2ConformerModelTest` and skip it on CPU
Author: ydshieh
Pull request: yes

# What does this PR do?

- There is device issue on GPU.
- There is low level (C++) issue with torch 2.2 (we have to report to torch team)
  - Skip this  `FastSpeech2ConformerModelTest` on CPU. 
  - (It is still tested on GPU, and it pass)


Comments:
use device directly in `torch.ones`. Let me know if you insist to use torch full

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28888). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
unpin torch
Author: ydshieh
Pull request: yes

# What does this PR do?

So we can use torch 2.2

Comments:
Let's see what CI says.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28892). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

The failing tests are already on `main`. I have no idea, but it's not from the torch 2.2.

They were failing already 15 hours ago

https://app.circleci.com/pipelines/github/huggingface/transformers/83829/workflows/a864e1cf-46bc-4643-9a2f-9ba65ee7074e


Let's move on and check those tf/flax vs torch equivalence tests later

(also, `test_save_load_fast_init_from_base` failing is known and we are waiting @ArthurZucker to take a look) 

We are good to merge. The TF/Flax vs torch job is addressed

https://github.com/huggingface/transformers/pull/28898

> Thanks for handling all the version compatibility issues!

Except `ydshieh.__version__` ⬇️ 😫
tokenizer_kwargs in text-generation pipeline __call__()
Author: farzadsbakhtiar
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #25994 for `text-generation` pipelines. The primary use case is to have truncation in the pipeline. This is useful in e.g. RAG when the relevant documents are too long.


## Who can review?

@nmcahill @Narsil @ArthurZucker 

<!--

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
@ArthurZucker what do you think about adding a `truncation` (and `max_length`) arg to the pipeline call/constructor instead of this? For all text pipelines. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

(I answered on the linked PR! 🤗 )

What is the solution? Nothing seems to be working for me.

It's in #28362. Use something like this:
```
text_generator_pipeline(
                test_str,
                do_sample=False,
                return_full_text=False,
                truncation=True,
                max_length=3,
            )
```
Explicit server error on gated model
Author: Wauplin
Pull request: yes

When a `GatedRepoError` error is raised while accessing a file, `transformers` catches the error to set a custom message inside a `EnvironmentError` (i.e. OSError) exception. This can be confusing since the underlying error (the `GatedRepoError`) contains more information on why the model cannot be accessed. Even though the `GatedRepoError` is thrown in the stacktrace, the message that could help the user is too hidden which can be misleading. This PR fixes that by forwarding the server message inside the `EnvironmentError`.

cc @coyotte508 who suggested this PR [in slack](https://huggingface.slack.com/archives/C02EMARJ65P/p1706866127367819?thread_ts=1706783851.655979&cid=C02EMARJ65P) (private) cc @meganriley as well

Error on gated repo if no auth:
```
Cannot access gated repo for url https://huggingface.co/nvidia/nemotron-3-8b-base-4k/resolve/main/.gitattributes.
Repo model nvidia/nemotron-3-8b-base-4k is gated. You must be authenticated to access it.
```

Error on gated repo if auth but not requested yet
```
Cannot access gated repo for url https://huggingface.co/baseten/docs-example-gated-model/resolve/main/.gitattributes.
Access to model baseten/docs-example-gated-model is restricted and you are not in the authorized list. Visit https://huggingface.co/baseten/docs-example-gated-model to ask for access.
```

Error on gated repo if auth but pending request:
```
Cannot access gated repo for url https://huggingface.co/nvidia/nemotron-3-8b-base-4k/resolve/main/.gitattributes.
Your request to access model nvidia/nemotron-3-8b-base-4k is awaiting a review from the repo authors.
```

Comments:
sounds like a good idea to more clearly throw `huggingface_hub` specific errors

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28894). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the review @amyeroberts. CI seems to be failing but I don't think it's related to my changes. Feel free to merge the PR if you think it's fine or let me know what I can do for it. Thanks in advance!

@Wauplin Yes, unfortunately there's been a lot of issues with compatible libraries the past few weeks 😫  torchaudio been the current culprit. 

Merging as these failures are unrelated :)
[Docs] Fix spelling and grammar mistakes
Author: khipp
Pull request: yes

# What does this PR do?

This PR fixes various spelling and grammar mistakes in the documentation and docstrings within the source code.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28825). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix input data file extension in examples
Author: khipp
Pull request: yes

Ensure that the input data file extension is set correctly when running example scripts without specifying a training data file.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28741). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Docs] Fix backticks in inline code and documentation links
Author: khipp
Pull request: yes

# What does this PR do?

This PR adds missing backticks to inline code segments and documentation links that were not wrapped properly.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Documentation: @stevhliu


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28875). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Docs] Update project names and links in awesome-transformers
Author: khipp
Pull request: yes

# What does this PR do?

This PR updates the linked projects in awesome-transformers:

* "Lama Cleaner" has been renamed to "IOPaint" ([Commit](https://github.com/Sanster/IOPaint/commit/a73e2a531f54c693763c61f611f453add42e5c8d))
* "LLaMA Efficient Tuning" has been renamed to "LLaMA Factory" ([Commit](https://github.com/hiyouga/LLaMA-Factory/commit/197c754d731d495330f33bbf962f8bbc7a10c0cc))
* "adapter-transformers" has been replaced by the "Adapters" library ([Commit](https://github.com/adapter-hub/adapter-transformers-legacy/commit/59ca38464956ac9d100f02638ae5977227556b79))


Comments:
Hotfix - make `torchaudio` get the correct version in `torch_and_flax_job`
Author: ydshieh
Pull request: yes

# What does this PR do?

Apply one more hot fix to make the torchaudio get's the correct version .

(failing job run [here](https://app.circleci.com/pipelines/github/huggingface/transformers/83910/workflows/023ecc92-20eb-44b5-8f09-98f3d9459f53/jobs/1083262))

(not identified on the previous PR, because of the cache restoring mechanism on CircleCI)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28899). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Docs] Add missing language options and fix broken links
Author: khipp
Pull request: yes

# What does this PR do?

This PR adds missing entries to the language selection menu, as well as links to the Colab and AWS Studio notebooks for the ONNX examples.
It also fixes various hyperlinks that were broken due to spaces within the URL or spaces between the link text and the URL, and updates the links to OpenAI research articles.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28852). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Rebased to resolve merge conflicts with the main branch.
Trainer logging_first_step not evaluate on first step as it is documented
Author: RmZeta2718
Pull request: no
Labels: name: Good First Issue

name: Good First Documentation Issue

### System Info

`transformers` version: 4.35.2

### Who can help?

trainer: @muellerzr and @pacman100

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

document says that [logging_first_step](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.logging_first_step) will evaluate on the first global_step. But it only logs on the first step, not evaluate.

Related code: [link](https://github.com/huggingface/transformers/blob/633215ba58fe5114d8c8d32e415a04600e010701/src/transformers/trainer_callback.py#L435)

### Expected behavior

Either fix the document (remove "evaluate") or add evaluate feature to `logging_first_step` (I would prefer the latter)

Or if it's confusing for `logging_first_step` to evaluate, maybe we can add a `evaluate_first_step` argument.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Gently pinging @muellerzr and @pacman100 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Marking as good first issue, doc can be updated 😉 

Hey, @pacman100. Can I work on this issue?

Sure feel free to open a PR to update the doc 😉
Revert "[WIP] Hard error when ignoring tensors."
Author: ydshieh
Pull request: yes

Reverts huggingface/transformers#27484

Causing failing in TF/Flax vs torch equivalence tests

https://app.circleci.com/pipelines/github/huggingface/transformers/83829/workflows/a864e1cf-46bc-4643-9a2f-9ba65ee7074e



Comments:
The failing will disappear after the unpin torch PR is merged

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28898). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @Narsil and @ArthurZucker for visibility 

Going to merge to keep CI green



Just FMI who will take over the PR again? 

> Just FMI who will take over the PR again?

I assume it would be @Narsil ...?