noamaanMulla-03/transformers-issues huggingface dataset:
Unable to load from pretrained by inputting state_dict directly
Author: mikeybellissimo
Pull request: no

### System Info

In the documentation it says that if you put the pretrained_model_name_or_path == None then you can directly input the state_dict for the model. I needed to do this for a specific use case where it doesn't make sense for me to save and then load the checkpoints like usual.  Attached is the screenshot of the documentation that says this. 

I was able to implement this functionality with very minimal changes to the code and without interfering with anything else so I would love to push this to the repository I just need the proper permissions.


![State_dict](https://github.com/huggingface/transformers/assets/13650673/856d0a3a-9a52-4db9-8c15-3aba605e9587)



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When loading from_pretrained and inputting the config and state_dict but setting pretrained_model_name_or_path == None you get a None object does not have an 'endswith' method or something along those lines. 

### Expected behavior

The expected behavior is that it would load the state_dict that you input. My fixed version does this.

Comments:
Hi, after reading some more online, I don't believe that this is the proper way to do a pull request. This is my first time contributing to open source. Sorry about that. I would delete this all if I was able. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add accelerate support - vision MAE models
Author: amyeroberts
Pull request: yes

# What does this PR do?

Adds accelerate support to VideoMAE and ViTMAE following the changes made in the [equivalent ViT PR](https://github.com/huggingface/transformers/pull/20174)

Fixes #23086 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23114). All of your documentation changes will be reflected on that endpoint.

@sgugger I checked with 2 GPUs, I'll run with just one to make sure it still works 👍
Whisper: fix prompted max length
Author: gante
Pull request: yes

# What does this PR do?

Fixes #24600 

#23724 Added the ability to guide generation with Whiper through `prompt_ids`. It was increasing the generation length by the length of the prompt -- these tokens were being hardcoded, and thus "not generated".

However, in the default case, we were already setting the generation length to the maximum allowed model length (see [model config](https://huggingface.co/openai/whisper-large-v2/blob/main/config.json#L42)). This increment was forcing us to go behind the maximum length and, because the model uses a `nn.Embedding` for the position embedding, indexing exceptions started popping up on long audio inputs :D

This PR modifies the length extension to what I believe was the author's original goal: only increment the length if `max_new_tokens` is passed. By default, this argument is not set and should correspond to the "new" (=non-prompt) generated tokens.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@amyeroberts @sanchit-gandhi 

After the latest changes, a warning is emitted when we cross `config.max_position_embeddings` for the first time.

For instance, if you now run 

```py
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
model = AutoModelForCausalLM.from_pretrained("distilgpt2").to("cuda")

inputs = tokenizer(["The quick brown"], return_tensors="pt").to("cuda")
# distilgpt2 has a maximum length of 1024
gen_out = model.generate(**inputs, do_sample=True, eos_token_id=-1, max_length=1025)
```

You'll see

```
This is a friendly reminder - the current text generation call will exceed the model's predefined maximum length (1024). Depending on the model, you may observe exceptions, performance degradation, or nothing at all.
```

(And if you set `max_length=1026`, you'll see the warning right before the exceptions. This is because we can technically generate `config.max_position_embeddings + 1` tokens even with restrictive position embeddings, although we shouldn't!)
Not able to load T5 tokenizer
Author: adithyaan-creator
Pull request: no

Transformers==4.0.0
torch == 1.7.0+cu101
tensorflow == 2.3.0
Platform = Colab notebook

@julien-c  @patrickvonplaten

Not able to load T5 tokenizer using
 `tokenizer = T5Tokenizer.from_pretrained('t5-base')'

Getting error - 
![image](https://user-images.githubusercontent.com/54103522/102049719-6c717480-3e07-11eb-8671-9c10c3169b17.png)

I am able to download the pre-trained model though.

Comments:
Hey @adithyaan-creator, 

as the error message says you need to install the sentence piece library :-) 

If you run:

```
pip install sentencepiece==0.1.91
```

before, it should work.

Thanks @patrickvonplaten . 

Hi @patrickvonplaten 
I installed sentencepiece, but it still doesnt seem to be working for me. Please see the snapshot below. Please help.

![image](https://user-images.githubusercontent.com/48218899/103457616-d1452e00-4d26-11eb-9cc3-7bb81b807750.png)


@DesiKeki try sentencepiece version 0.1.94.

Thanks @adithyaan-creator , it worked!

Hello @patrickvonplaten 
I have gone through the issue and the suggestions given above. However, I am facing the same issue and for some reason, none of the above solutions are proving fruitful.

The issue I am facing is exactly the same as the one stated above:
`from transformers import T5Tokenizer,T5ForConditionalGeneration,Adafactor`
`!pip install sentencepiece==0.1.91`
`tokenizer = T5Tokenizer.from_pretrained("t5-base")`
`print(tokenizer)`

The output of the above code is: None. 
I tried using other versions of sentencepiece as well (as the one suggested above 0.1.94 and others as well). But it is still not working.

![ImportingT5BaseFails](https://user-images.githubusercontent.com/48618975/111974487-74873180-8b25-11eb-8b75-63cd31750711.PNG)


Did you restart your kernel after installing `sentencepiece`? See conversation in https://github.com/huggingface/transformers/issues/10797

> Did you restart your kernel after installing `sentencepiece`? See conversation in #10797

it works for me, thank you

> Did you restart your kernel after installing `sentencepiece`? See conversation in #10797

It works for me. Thanks a lot.
Adding GPTNeoX (Tensorflow version)
Author: kanpuriyanawab
Pull request: no
Labels: name: New model

### Model description

Huggingface has GPTNeoX model by ElutherAI. It's a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license.

However Huggingface currently has only PyTorch implementation of the model.
I would like to contribute it's corresponding TensorFlow implementation.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
cc: @sgugger @NielsRogge 

How would you run this model in Tensorflow though? I don't think it offers the same flexibility as PyTorch wrt to distributing layers on several devices.

cc @Rocketknight1 

I believe a TF implementation with distribution of layers across devices is possible using [DTensor](https://www.tensorflow.org/guide/dtensor_overview). We've been in communication with the TF team about DTensor, but we haven't implemented a TF model with it yet. This could be a good model to try it with!

Note that implementing our first DTensor model will probably be quite challenging @shivance - we'll support you if you try it, but you should expect that the PR will need changes to some of our tests or internal functions to support DTensor, and there'll be several rounds of iteration before it's ready.

@sgugger @Rocketknight1 Check this out
https://github.com/huggingface/transformers/blob/17a55534f5e5df10ac4804d4270bf6b8cc24998d/src/transformers/models/esm/modeling_tf_esm.py#L102

 I've seen the use of dtensor at KerasNLP deeply and I think I can start working on it if you and @sgugger give me an initial roadmap!

Hi @shivance, the first task would be to make a port of GPT-NeoX to TF, and then we can start adding `DTensor` layout code to it. You can look at this [ongoing LLAMA PR](https://github.com/huggingface/transformers/pull/24375) to see what you need to do. The biggest thing you have to do is to make a conversion of the existing `modeling_gpt_neox.py` file to `modeling_tf_gpt_neox.py`, then import the classes from it and run some tests to see that it gives (approximately) the same outputs. Once that's done, we can talk about next steps!

Hey @Rocketknight1 !
I've worked on same as my Google Summer of Code project, check Tf version of GPT Neo X [here](https://github.com/keras-team/keras-nlp/pull/1056), I think what's needed is to do it in Huggingface Design.
Felix
Author: patrickvonplaten
Pull request: no
Labels: name: New model

# 🌟 New model addition

## Model description

New model released by [google](https://github.com/google-research/google-research/tree/master/felix)

## Open source status

* [x] the model implementation is available: https://github.com/google-research/google-research/tree/master/felix
* [x] the model weights are available: https://github.com/google-research/google-research/tree/master/felix (pretrained `bert-base-cased` can be used)
* [x] who are the authors: @Jmallins


Comments:
Can I help with this, @patrickvonplaten ?

Hey @mrm8488,

It would be amazing if you're interested in adding this model. It looks like a difficult one, but I'd definitely try to help you as much as I can if you want to add it :-)

Looking forward ! 

@ArthurZucker @younesbelkada (the ML engineers currently in charge of the text models). This has been collecting dust for a while. I still think it's relevant because it's the only encoder-only model that I'm aware of that can do text editing. Encoder-only models are faster than encoder-decoder and decoder-only models for inference.

This might be of interest for @fxmarty 👀 

(I mean, I suppose LaserTagger is also encoder-only, but FELIX came later and outperforms LaserTagger in both "%exact match with the reference sentence" and latency.)

Is anyone working on this? If not, I might try to take a shot at it...but first I'd ask if anyone has implemented already anything related to this model or this remained self-contained in the issues cathegory.

I think you can take this on! 

Is there any new update on this? This approach for text editing with PointerNet reordering feels like a very important step for text editing and faster text generation inference. Is text editing, in general, something the HuggingFace team is interested in pursuing?

@afonso-sousa so far I didn't have time to work on this. I will eventually find the time, I hope this soon enough, but I would do it in my free time so I have no guarantees. But I'm not jealous: if anyone finds time to do it before me please do!
Add Epsilon and Eta sampling
Author: shermansiu
Pull request: no

### Feature request

I would like to add Epsilon and Eta sampling from [Truncation Sampling as Language Model Desmoothing](https://arxiv.org/abs/2210.15191). They are novel truncation sampling decoding algorithms that have led to better human judgement scores and less repetition than nucleus sampling.

- [Official repository](https://github.com/john-hewitt/truncation-sampling)

### Motivation

I would like to generate more human-like and less repetitious text with Huggingface models.

### Your contribution

I am able to submit a PR for this.

Comments:
cc @gante 

Yesterday's ACL 2023 tutorial on "Generating Text from Large Language Models" covers eta-sampling and more! John Hewitt, the first author of the eta-sampling paper, was one of the presenters for that tutorial!

Site: https://rycolab.io/classes/acl-2023-tutorial/
Slides: https://drive.google.com/file/d/1UHbGcjzBURG1n2DufC7iDTmGNjIz5Dp_/view
Add HyenaDNA model
Author: tanaymeh
Pull request: no
Labels: name: New model

### Model description

HyenaDNA is a long-range genomic foundation model pretrained on context lengths of up to 1 million tokens at single nucleotide resolution.

I would like to add this model to the transformers.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Code: https://github.com/HazyResearch/hyena-dna
Weights: https://huggingface.co/LongSafari
Paper: https://arxiv.org/abs/2306.15794

cc @exnx

Comments:
Hi @heytanay, thanks for opening this issue! 

The easiest and recommended way to make a model available in `transformers` is to add the modeling code directly on the hub: https://huggingface.co/docs/transformers/custom_models

This means, once working, the model can be found and used immediately without having to go through the PR process. We find this is a lot quicker as the bar for adding code into the library is high due to the maintenance cost of every new model, and so reviews take quite a while.

Let us know if you have any questions about how to add a model using this process. Looking forward to seeing this model in action! 

Thanks for this @amyeroberts! I will proceed with that!

Hi, @heytanay we are also working on adding hyena models to transformers, how far along are you ?

@djaym7 As Amy mentioned, I won't be implementing the model directly in transformers and instead will be adding it directly to the hub. If you are doing it / already have done it, please go ahead!
Update conditional logic and -> or in SAM postprocessing
Author: hwuebben
Pull request: yes

should be ors here

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@younesbelkada exactly but with the current implementation it could also be a list of arrays and the exception would not be thrown

FYI, this PR triggered 3 test failures on the CI.

(The third one is GPU OOM, which is likely caused by the other 2 failures).

cc @younesbelkada 


```bash
tests/models/sam/test_modeling_sam.py::SamModelIntegrationTest::test_inference_mask_generation_one_point_one_bb
(line 235)  ValueError: Input boxes must be a list of list of list of floating integers.
tests/models/sam/test_modeling_sam.py::SamModelIntegrationTest::test_inference_mask_generation_one_point_one_bb_zero
(line 235)  ValueError: Input boxes must be a list of list of list of floating integers.
tests/models/sam/test_modeling_sam.py::SamModelIntegrationTest::test_inference_mask_generation_two_points_batched
(line 808)  torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB (GPU 0; 14.76 GiB total capacity; 11.65 GiB already allocated; 792.75 MiB free; 12.74 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```

 ValueError: Input boxes must be a list of list of list of floating integers. I AM Still getting this error, when even run the example notebooks, what do I do? I just have a list of bounding box coordinated(flatenned)

Hi @karthikdatta98 
thanks for reporting, in https://github.com/huggingface/notebooks/pull/409 I modified the notebook accordingly to show how to correctly pass bounding boxes
[i18n-<languageCode>] Translating docs to <languageName>
Author: Everton-12
Pull request: no
Labels: name: WIP

<!--
Note: Please search to see if an issue already exists for the language you are trying to translate.
-->

Hi!

Let's bring the documentation to all the <languageName>-speaking community 🌐 (currently 0 out of 267 complete)

Who would want to translate? Please follow the 🤗 [TRANSLATING guide](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md). Here is a list of the files ready for translation. Let us know in this issue if you'd like to translate any, and we'll add your name to the list.

Some notes:

* Please translate using an informal tone (imagine you are talking with a friend about transformers 🤗).
* Please translate in a gender-neutral way.
* Add your translations to the folder called `<languageCode>` inside the [source folder](https://github.com/huggingface/transformers/tree/main/docs/source).
* Register your translation in `<languageCode>/_toctree.yml`; please follow the order of the [English version](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml).
* Once you're finished, open a pull request and tag this issue by including #issue-number in the description, where issue-number is the number of this issue. Please ping @ArthurZucker, @sgugger for review.
* 🙋 If you'd like others to help you with the translation, you can also post in the 🤗 [forums](https://discuss.huggingface.co/).

## Get Started section

- [ ] [index.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/index.md) https://github.com/huggingface/transformers/pull/20180
- [ ] [quicktour.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/quicktour.md) (waiting for initial PR to go through)
- [ ] [installation.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/installation.md).

## Tutorial section
- [ ] [pipeline_tutorial.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/pipeline_tutorial.md)
- [ ]  [autoclass_tutorial.md](https://github.com/huggingface/transformers/blob/master/docs/source/autoclass_tutorial.md)
- [ ]  [preprocessing.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/preprocessing.md)
- [ ]  [training.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/training.md)
- [ ]  [accelerate.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/accelerate.md)
- [ ]  [model_sharing.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_sharing.md)
- [ ]  [multilingual.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/multilingual.md)

<!--
Keep on adding more as you go 🔥
-->


Comments:
Hi @Everton-12, Could you make sure to fill in the template for this issue? At the moment there is no specified language.
[Patch-t5-tokenizer] Patches the changes on T5 to make sure previous behaviour is still valide for beginning of words
Author: ArthurZucker
Pull request: yes

# What does this PR do?
There was a small typo that modified the behaviour in #24565, the test were not able to catch it.  #24569
When a sentence does not start with a space, a space was added.
Before:
```python
>>>tokenizer.tokenize("Hello <extra_id_0>")
['_', '_Hello', '<extra_id_0>']
```
After:
```python 
>>>tokenizer.tokenize("Hello <extra_id_0>")
['_Hello', '<extra_id_0>']
```
# Big bug 35 models involved
Not only punctuation but anything after a special token is basically wrong... Let's ignore the fact that we also split when it's the beginning of word less important
<img width="1020" alt="image" src="https://github.com/huggingface/transformers/assets/48595927/d805bd21-4f2a-411b-ad2b-754f4f69517c">


Tests were added as they were green before merging

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Ran all the tests with `RUN_SLOW`, switch-ci is fixed

Until the added tokens are fixed, this will break the slow version that use extra ids, because by default we strip left and right..... So buggy 

The bug has always been in T5, but since some models were trained with the bugged T5, we will let the user decide whether or not they incorparate the change
Docs: add `kwargs` type to fix formatting
Author: gante
Pull request: yes

# What does this PR do?

As the title indicates: in several places our docs `kwargs` did not include its type, which made our doc builder treat it like a continuation of the previous parameter 💔 

Example of currently broken docs ([this function](https://huggingface.co/docs/transformers/v4.30.0/en/main_classes/processors#transformers.ProcessorMixin.save_pretrained)):
<img width="915" alt="Screenshot 2023-07-10 at 18 20 47" src="https://github.com/huggingface/transformers/assets/12240844/d027950f-7082-416a-b2da-e4f3712bd27c">

This PR is a result of CMD+F on the broken pattern, and applying the fix :)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

After the PR (as seen in the doc preview):
<img width="926" alt="Screenshot 2023-07-10 at 18 55 21" src="https://github.com/huggingface/transformers/assets/12240844/9329186d-86ee-4091-a5c5-bbee77ab83f7">


Happy to push a `doc-builder` side change if needed  -- and to do the opposite of this PR: remove `Dict[str, Any]` from `kwargs` whenever it is present.

Just let me know your preference :) If it's neutral for you, I think having the explicit type is friendly for Python newbies.

@amyeroberts if you don't oppose, I'll merge this PR 🤗 

@gante Go for it!
add gradient checkpointing for distilbert
Author: jordane95
Pull request: yes

# What does this PR do?

Fixes #9113 and #23219 

I just added the gradient checkpointint feature for DistilBert following the implementation in BERT.

This should be useful if one wants to train a relatively small model with extremly large batch size for better performance in application scenarios such as text retrieval or embeddings.

@ArthurZucker @sgugger

Comments:
Hi @jordane95, thanks for opening this PR. 

Overall changes look OK to me. I can see from the issue discussion that gradient checkpointing was deliberately not added to DistilBert. In general, we try to avoid adding complexity to existing models, in particular their forward pass. Let's get @sgugger second opinion on whether this should be merged into main. 

For the quality checks, you'll need to run `make style` and push any changes made to this branch.

_The documentation is not available anymore as the PR was closed or merged._
Skip keys not in the state dict when finding mismatched weights
Author: sgugger
Pull request: yes

# What does this PR do?

When looping through the keys in `find_mismatched_weights`, we loop through all the `loaded_keys` which are all the keys in the checkpoint. If the checkpoint is sharded, the `state_dict` passed won't contain all those keys, only a subset of them, so we need to skip the keys not present in the `state_dict`.

Fixes #24704 

Comments:
_The documentation is not available anymore as the PR was closed or merged._