noamaanMulla-03/transformers-issues huggingface dataset:
Token indices sequence length is longer than the specified maximum sequence length for this model (11261 > 1024). Running this sequence through the model will result in indexing errors
Author: FurkanGozukara
Pull request: no

### System Info

I tested 2 models (sshleifer/distilbart-cnn-12-6 , facebook/bart-large-cnn) and they both have very small 1024 max token length

So which model supports full length or the most token count?





### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I tested 2 models (sshleifer/distilbart-cnn-12-6 , facebook/bart-large-cnn) and they both have very small 1024 max token length

So which model supports full length or the most token count?





### Expected behavior

I tested 2 models (sshleifer/distilbart-cnn-12-6 , facebook/bart-large-cnn) and they both have very small 1024 max token length

So which model supports full length or the most token count?





Comments:
Please use the [forums](https://discuss.huggingface.co/) for questions like this as we keep the issues for bugs (in the library) and feature requests only.

@sgugger you are right i am closing this thread

could you answer there: https://discuss.huggingface.co/t/which-summarization-model-of-huggingface-supports-more-than-1024-tokens-which-model-is-more-suitable-for-programming-related-articles/25095
Can run_translation.py support nllb model fine-tuning ?
Author: cokuehuang
Pull request: no

### Feature request

Can run_translation.py support nllb model fine-tuning ? As run_translation.py is much easier to fine-tuning a model.

### Motivation

Want to an easy way to fine-tuning nllb model, as it is so difficult to fine-tuning a nllb model from its docs.

### Your contribution

star the repository 

Comments:
The script support NLLB models, if you run any issue you can use the [forums](https://discuss.huggingface.co/) to ask the community for help.
About the `head_mask` of the Bert model `forward` really speed up?
Author: CaffreyR
Pull request: no

### System Info

- `transformers` version: 4.20.1
- Platform: macOS-12.4-arm64-arm-64bit
- Python version: 3.9.10
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.13.0.dev20220709 (False)
- Tensorflow version (GPU?): 2.8.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@LysandreJik Many thanks!

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi, in `bert` model or other model, when we try to get the output, we usually use `output = model(**batch)`. And in the source code of bert model, we actually have a parameter called `head_mask`. So when we give them different head_mask of the model, like `outputs = model(head_mask=head_mask, **batch)`, the time will be different?

So I try to run the two codes, and the time of the code that have more-zero `head_mask` do not speed up

```
head_mask = torch.ones(12, 12)
print(head_mask)
for batch in dataloader:
    for k, v in batch.items():
        batch[k] = v
    import time
    start=time.perf_counter()
    outputs = model(head_mask=head_mask, **batch)
    end = time.perf_counter()
    print(end-start)
```

```
head_mask = torch.zeros(12, 12)
for i  in range (12):
    head_mask[i][0] = 1
print(head_mask)
for batch in dataloader:
    for k, v in batch.items():
        batch[k] = v
    import time
    start=time.perf_counter()
    outputs = model(head_mask=head_mask, **batch)
    end = time.perf_counter()
    print(end-start)
```

### Expected behavior

So I try to run the two codes, and the time of the code that have more-zero `head_mask` do not speed up

Many thanks!

Comments:
Hey @CaffreyR, the head mask isn't there to speed things up.

You can read a bit more about it in the [bertology](https://huggingface.co/docs/transformers/v4.23.1/en/bertology) documentation. It's mostly to see which heads impact your prediction, it's not made to speed things up.

So you means that only if we prune the head according to the `head mask`, we can speed up our model?

I mean that looking at `head_mask` as a way to speed up the model doesn't work :)
If you'd like to speed up your model, you can look at changing the precision, quantizing it, distillating it; but removing heads isn't going to speed it up or very very marginally.

Great! Many thanks!
Create dummy models
Author: ydshieh
Pull request: yes

# What does this PR do?

This is a new script based on [a previous one](https://gist.github.com/LysandreJik/39058fe6fa8771f74dda7e789a6f63ea#file-create_dummy_models-py)

In the comment, links to 3 reports are provided.

### (Probably) To Do:
- As we shrink the tokenizer vocab size, the special tokens (bos/eos/pad etc.) might change too. I think we should also try to update the attributes in `tiny_config` whose names end with `__token_id`.
- We should probably provide an option to upload to Hub.

### Remark

Currently, if we can not shrink the tokenizer's vocab size for a model type, we still build models for it but give a warning in the report. We should not use them for pipeline testing though (which is what our pipeline testing does so far)

### Current states

- #### These need to be treated specially
    - EncoderDecoder
    - VisionEncoderDecoder
    - SpeechEncoderDecoderModel
    - VisionTextDualEncoder

- #### Some of the following need to check, but others are expected not to work
    - BertGeneration
    - Camembert
    - DecisionTransformer [This model doesn't require any processor -> need to allow this case]
    - ~~DonutSwin~~
    - Esm
    - MarianForCausalLM
    - MT5
    - ~~PegasusX~~
    - QDQBert
    - ReformerModelWithLMHead
    - Speech2Text2ForCausalLM
    - TimeSeriesTransformer [This model doesn't require any processor -> need to allow this case]
    - TrajectoryTransformer
    - TrOCRForCausalLM
    - ~~ViTMSN~~
    - ~~Wav2Vec2Conformer~~
    - XLMProphetNet
    - XLMRoberta

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Here are the 3 formats of the reports

[simple_report.txt](https://github.com/huggingface/transformers/files/9872354/simple_report.txt)
[failed_report.json](https://github.com/huggingface/transformers/files/9872355/failed_report.txt)
[tiny_model_creation_report.json](https://github.com/huggingface/transformers/files/9872356/tiny_model_creation_report.txt)




Nice, thanks @ydshieh! I'll take it for a spin tomorrow.

I will take care of the quality check (don't want to push more commits at this moment) 🙏 

> The description of which models succeeded and which didn't could be slimmer, for example in a TQDM bar where we would print the models that didn't succeed, for example for:

Do you mean we only print the failed ones?

@LysandreJik Could you take a final look regarding my 2 comment above? Also [this one](https://github.com/huggingface/transformers/pull/19901#issuecomment-1293736263) 🙏 

Thank you for the review 💯 

Final remark: those progress bar are not downloading, but the training of the tokenizers (to reduce the vocab size). I will ask @Narsil if we can disable showing those 😃 

Close for now in order to fix a few really edge cases (not to run CI).

You can disable the progress bar indeed `Trainer(... show_progress=False)`
Use self._trial  to generate trial_name for Trainer.
Author: reyoung
Pull request: yes

# What does this PR do?

Generate trial name unless the trial is not None, and use `(self._trial or trial)` to generate trial name. 
Because [currently the `optuna` backend give a None trial when using DDP and rank != 0](https://github.com/huggingface/transformers/blob/v4.23.1/src/transformers/integrations.py#L193)

Related code:
https://github.com/huggingface/transformers/blob/bd469c40659ce76c81f69c7726759d249b4aef49/src/transformers/integrations.py#L160-L208


Or maybe the documentation should be changed.

https://github.com/huggingface/transformers/blob/bd469c40659ce76c81f69c7726759d249b4aef49/src/transformers/trainer.py#L2318-L2319

Who can review: 

* Trainer: @sgugger  
* optuna HPO: @sywangyi


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger Please take a review
Add Onnx Config for ImageGPT 
Author: RaghavPrabhakar66
Pull request: yes

# What does this PR do?
Fixes #16308

Add changes to make ImageGPT models available for Onnx conversion.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ChainYo

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@lewtun With latest changes, all the 4 tests are now passing.

![image](https://user-images.githubusercontent.com/52318784/198269069-0fa809f5-ae25-4b7e-8276-c34d7a941017.png)


> @lewtun With the latest changes, all the 4 tests are now passing.

Thanks for iterating so fast, @RaghavPrabhakar66. Good work!

If you have time, you can try to upload an ONNX ImageGPT model to the ONNX organization on the hub if you want.

@ChainYo Sure.

@sgugger fixed it.

Thanks!
'FlaubertTokenizer' object has no attribute 'do_lowercase'
Author: davidavdav
Pull request: no

### System Info

- `transformers` version: 4.23.1
- Platform: Linux-5.13.0-30-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: not essential
- Using distributed or parallel set-up in script?: no


### Who can help?

@SaulLu 

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("flaubert/flaubert_base_uncased")
tokenizer.tokenize("bonjour")
AttributeError: 'FlaubertTokenizer' object has no attribute 'do_lowercase'
```

This was not the case back in the days of transformers-2

I can fix it by saying
```python
tokenizer.do_lowercase = True
```

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("flaubert/flaubert_base_uncased")
tokenizer.tokenize("bonjour")
AttributeError: 'FlaubertTokenizer' object has no attribute 'do_lowercase'
```


### Expected behavior

```
['bonjour</w>']
```


Comments:
Thanks for reporting! I believe this has been fixed on the main branch. While waiting for the next release, could you do a source install?

Yes, when installing from main 
```
pip install git+https://github.com/huggingface/transformers.git
```

things work again as expected
Activation checkpointing for TFGPT2DoubleHeadsModel
Author: visionscaper
Pull request: no

### Feature request

Activation checkpointing is implemented for the PyTorch GPT2 model (and different head variants), however, this is not the case for the Tensorflow implementation of GPT2,

### Motivation

A lot of GPU memory is required to finetune GPT2. This is especially the case for TFGPT2DoubleHeadsModel, because different choices (represented by different sequences) are combined in one sample. I think [`tf.recompte_grad`](https://www.tensorflow.org/api_docs/python/tf/recompute_grad) can play a role here.

### Your contribution

I've more experience with PyTorch then Tensorflow, but I could investigate possible solution directions. If it turns out to be easy I could spent time to create a PR, however, help by others is appreciated.

Comments:
cc @Rocketknight1 @gante

Hi @visionscaper - this seems like something that could work! We haven't experimented with `tf.recompute_grad` in detail but the core code for training our Keras models is in the `train_step` and `test_step` methods [here](https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_tf_utils.py#L1380), so you could try adding it there. Anything you add there will control `model.fit()` for all of our models.

Bear in mind I don't know if this will work - I don't know the exact semantics of `recompute_grad` or if it plays nicely with Graph mode, but if you discover anything or you have any questions feel free to post them here!



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add BPE Wav2Vec2CTCTokenizer
Author: OllieBroadhurst
Pull request: no

### Feature request

Hi there!

Is there scope for a BPE (SentencePiece) CTC tokenizer? Using a trained SentencePiece vocabulary in a CTC model is pretty straightforward - all we need now is a tokenizer than can bridge the missing step between grouping CTC ids and decoding them with SentencePiece.

### Motivation

Grouping characters that occur commonly together is a cool way of sometimes helping with spelling mistakes (like a mini LM) and introduces dependence between characters which can help hold the model's hand.

As far as I know this is fully supported by pyctcdecode so no pipelines/LM processors, etc. need to change.

### Your contribution

The few changes would be something like

```python
def _tokenize(self, text):
        return self.sp_model.encode(text, out_type=str)
```
```python
def _convert_token_to_id(self, token):
    spm_id = self.sp_model.PieceToId(token)
    return spm_id
```

Additional SentenciePiece args would be similar to [other SentencePiece tokenizers](https://huggingface.co/docs/transformers/model_doc/xlm-roberta#transformers.XLMRobertaTokenizer), as would be the [saving of the tokenizer](https://github.com/huggingface/transformers/blob/2c8b508ccabea6638aa463a137852ff3b64be036/src/transformers/models/deberta_v2/tokenization_deberta_v2.py#L474).

Comments:
cc @SaulLu @ArthurZucker 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Map `RealmBertModel` for `AutoModel`
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR maps `RealmModel` to be able to use it with `AutoModel` - for consistency with `BertModel` 

Why this PR? I wanted to automate tests for `BetterTransformers` integration in `optimum` without having to import manually the class, see here: https://github.com/younesbelkada/optimum/blob/49575c5b016392383f0c2ebc1565ef56747b87e6/tests/bettertransformers/test_bettertransformers.py#L68 
Since `Realm` should be supported by `BetterTransformers`, this PR would help me design easier-to-implement tests

cc @sgugger @ydshieh 

https://github.com/huggingface/optimum/pull/423

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @younesbelkada 
We should also work on `src/transformers/__init__.py`, but otherwise LGTM, thanks!

Ah yes, great catch! Will add it now

Thanks a lot for the explanation!
Yes let's stay pragmatic, I will probably just remove it from the `BetterTransformers` test 

@sgugger But we usually expose the base model, no, like `BertModel`?

Yes, but I'm very unsure that this is the base REALM model. It's more of a building block towards it.
CLIPTokenizer behaves inconsistently depending on whether ftfy is installed or not
Author: kjsman
Pull request: no
Labels: name: bug

### System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.22.1
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.14
- Huggingface_hub version: 0.9.1
- PyTorch version (GPU?): 1.12.1+cu113 (False)
- Tensorflow version (GPU?): 2.8.2 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


### Who can help?

@patil-suraj

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Run following code without ftfy installed.

```py
from transformers import CLIPTokenizer
tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-large-patch14")
tokenizer("résumé") # {'input_ids': [49406, 15077, 49407], 'attention_mask': [1, 1, 1]}
```

2. Run following code with ftfy installed.

```py
from transformers import CLIPTokenizer
tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-large-patch14")
tokenizer("résumé") # {'input_ids': [49406, 29106, 7054, 4166, 49407], 'attention_mask': [1, 1, 1, 1, 1]}
```

### Expected behavior

They should work consistently.

Comments:
This happens because `BasicTokenizer`, which is [used as](https://github.com/huggingface/transformers/blob/main/src/transformers/models/clip/tokenization_clip.py#L159) fallback text fix function, [strips accents if `do_lower_case=True`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/tokenization_bert.py#L174-L176).

We may fix this by explicitly set `strip_accents` to `False`; [ViT/L-14 tokenizer](https://huggingface.co/openai/clip-vit-large-patch14/raw/main/tokenizer.json) includes vocabs with accents, so I think stripping accents should not be done.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Problem trying to migrate cache 
Author: mpolinsky
Pull request: no
Labels: name: bug

### System Info

Using MacOS Big Sur v11.6.4 and jupyter lab v3.4.7.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Installed transformers in a new venv with pip
2. Imported autocast from torch and diffusers in jupyter lab, it automatically attempted to migrate the cache and I received this error asking me to open an issue.

code:

```
from torch import autocast
from diffusers import StableDiffusionPipeline
```

<img width="918" alt="differror" src="https://user-images.githubusercontent.com/30514239/191115866-5e239ee8-23b5-4f7d-b55f-0646a63d86a7.png">


### Expected behavior

I opened this because the error messaged asked me to.  I imagine the expected behavior is migrating the cache without incident.

Comments:
cc @sgugger 

I'm not too sure what the problem is, maybe it is due to the intermediate subfolder. This error should only happen once in any case, and you might have lost some cached files, but they will jsut be re-downloaded.

Thanks for the clarity.  Didn't cause a problem otherwise.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
How can I control data feeding order to model using Huggingface Trainer?
Author: SangwonPark0211
Pull request: no

### Feature request

I want to train model in the order in which the data are stored.  
For example, if there are 100 data, then I want to feed 1st, 2nd data together(because I set batch_size=2 in code) and then 3rd, 4th data and then 5th, 6th data together and so on....  
But huggingface Trainer train model using datacollator and this feed data to model randomly by the parameter data_seed.  
**How can I train model feeding data in the order in which the data are stored?**  

```
# load tokenizer
model_checkpoint = "facebook/bart-base"
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

# load model
model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint)

# make batch
data_collator = DataCollatorForSeq2Seq(tokenizer, model=model)

batch_size = 2
epochs = 3
args = Seq2SeqTrainingArguments(
    output_dir = "saved_model",
    overwrite_output_dir = True,
    evaluation_strategy = "epoch",
    save_strategy = "epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    gradient_accumulation_steps=2,
    weight_decay=0.01,
    num_train_epochs=epochs,
    predict_with_generate=True,
    fp16=False,
    dataloader_num_workers=8,
)
trainer = Seq2SeqTrainer(
    model,
    args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["validation"],
    data_collator=data_collator,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)
```

### Motivation

I want to control data feeding order to the model.

### Your contribution

I want to control data feeding order to the model.

Comments:
You can subclass the Seq2SeqTrainer and override the [_get_train_sampler](https://github.com/huggingface/transformers/blob/8d59385f124dd1b330cac7eaa7162799870793ec/src/transformers/trainer.py#L759) method. Instead of creating a RandomSampler object, create a [SequentialSampler](https://pytorch.org/docs/stable/data.html#torch.utils.data.SequentialSampler).

```python3
from transformers.trainer_seq2seq import Seq2SeqTrainer
from torch.utils.data import SequentialSampler

class SequentialSeq2SeqTrainer(Seq2SeqTrainer):
    def _get_train_sampler(self) -> SequentialSampler:
        return SequentialSampler(self.train_dataset)
```

Thank you!! I'll try as you mentioned.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Is BertWordPieceTokenizer behaviour deterministic on the same data?
Author: ZurabDz
Pull request: no

So, I accidentally lost BertWordPieceTokenizer checkpoint, what will happen If I retrain it from the same data? Will the result be the same?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.