noamaanMulla-03/transformers-issues huggingface dataset:
Generate: move generation_*.py src files into generation/*.py
Author: gante
Pull request: yes

# What does this PR do?

Moves `generation_*.py` source files into `generation/*.py`.

I tried a few slow tests locally, no problems were raised.

⚠️ the link to the docs seems broken, can't validate their correctness 🤔 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger now with `~/generation/__init__.py` populated with lazy references (as in most `__init__.py` files). 

I've updated all references to objects outside `/generation` from `generation.file_name.ObjectName` to `generation.ObjectName`, including in the docs. That does make tracking external references easier -- if an object is in `~/generation/__init__.py` it means that it is likely used somewhere else, and should be treated with extra care.

Could you please update `optimum` to reflect these changes?
```
from optimum.onnxruntime import ORTModelForSeq2SeqLM
site-packages/transformers/generation_utils.py:27: FutureWarning: Importing `GenerationMixin` from `src/transformers/generation_utils.py` is deprecated and will be removed in Transformers v5. Import as `from transformers import GenerationMixin` instead.
  FutureWarning,
```

> Could you please update `optimum` to reflect these changes?
> 
> ```
> from optimum.onnxruntime import ORTModelForSeq2SeqLM
> site-packages/transformers/generation_utils.py:27: FutureWarning: Importing `GenerationMixin` from `src/transformers/generation_utils.py` is deprecated and will be removed in Transformers v5. Import as `from transformers import GenerationMixin` instead.
>   FutureWarning,
> ```

The PR https://github.com/huggingface/optimum/pull/536 has just been merged and solves this issue.
layerdrop in Wav2Vec2Adapter
Author: bofenghuang
Pull request: no

### System Info

Hi!

As mentioned in https://github.com/huggingface/transformers/issues/20451, the layer dropout in `Wav2Vec2Adapter` may produce outputs with different lengths.

I understand the use of layerdrop in transformer structure, but do we need it in CNNs (`Wav2Vec2Adapter`)?

https://github.com/huggingface/transformers/blob/61d3928bfb3029bceb5be3e68ca3d4bf8456758f/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L1006-L1009

### Who can help?

cc @anton-l @patrickvonplaten 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I copied the reproduction code of @OllieBroadhurst  in https://github.com/huggingface/transformers/issues/20451

```python
from transformers import Wav2Vec2Model

model = Wav2Vec2Model.from_pretrained("anton-l/wav2vec2-base-lang-id", 
                                      add_adapter=True, 
                                      adapter_stride=2,
                                      adapter_kernel_size=3, 
                                      num_adapter_layers=2)

model.train() # NB

dummy_input = torch.randn((1, 16000))

expected_output_sequence_length = 13

for _ in range(200):
    output_shape = model(input_values=dummy_input)[0].shape[1]
    if output_shape != expected_output_sequence_length:
        print(output_shape)
```

### Expected behavior

The above loop shouldn't print anything out.

Comments:
@sanchit-gandhi could you take a look here? 

Hey @bofenghuang! Thanks for opening this issue. 

For context, we use the adapter layer when combining the Wav2Vec2 model in a sequence-to-sequence combination (see https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition#warm-started-speech-encoder-decoder-model). Here, the purpose of the adapter layer is to better match the time scale of the encoder with that of the decoder (see aforementioned doc).

In this respect, it's fine if the CNN downsamples the Wav2Vec2 output sequence at a stochastic rate for all training samples. This should add some 'robustness' to our text decoder which has to infer the correct target transcription from Wav2Vec2 output sequences of slightly varying length.

You can also disable layer drop by setting `layerdrop=0.0` in the config: https://huggingface.co/facebook/wav2vec2-base-960h/blob/main/config.json#L59

Thanks @sanchit-gandhi !
Support token suppression, forced tokens (besides eos and bos), and decoder prompting for flax generation
Author: andyehrenberg
Pull request: no

### Feature request

Add logits processors for token suppression and forced tokens at specific indices.
Enable prompting the decoder of encoder-decoder models with decoder_input_ids.

### Motivation

Currently, the flax generation utilities do not support token suppression, forcing specific tokens to be decoded at specific response indices, nor prompting the decoder (helpful for models like Whisper that support decoder prompts - Flax Whisper is implemented in #20479). Adding these would move the flax utilities closer to feature parity with the pytorch generation utilities. Adding these features would fully unlock a flax implementation of Whisper inference.

### Your contribution

I already have these features implemented in a branch of my fork - happy to open a PR!

Comments:
@sanchit-gandhi @patil-suraj @gante 

Did we decide to implement these features in the Flax Whisper PR in the end? cc @ArthurZucker

@sanchit-gandhi @ArthurZucker I just added these back into the Flax Whisper PR

Cool! Closing this issue in favour of the PR https://github.com/huggingface/transformers/pull/20479
make states contiguous for past_key_values
Author: xyjigsaw
Pull request: yes

Add contiguous for key and value states.

If someone uses past_key_values, it seems to raise the following exception:

`
RuntimeError: view size is not compatible with input tensor‘s size and stride ...
`
since bart executes torch.cat in BartAttention class:

`
key_states = torch.cat([past_key_value[0], key_states], dim=2)
`
`
value_states = torch.cat([past_key_value[1], value_states], dim=2)
`
Thus, we should make key_states and value_states contiguous.


@patrickvonplaten

---
More, bart cannot correctly process the length of attention_mask when the item of past_key_values is added.



Comments:
_The documentation is not available anymore as the PR was closed or merged._
minor fix in jax attention bias
Author: amankhandelia
Pull request: yes

Fixes #19897 

Adding this PR to check if this passes all the test cases, or fix have potential issues

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
cc @sanchit-gandhi 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19899). All of your documentation changes will be reflected on that endpoint.

Hey @amankhandelia! In general, I'm happy with this change. What worries me is that the tests for BART and BART-dervied models currently pass on main, which suggests there shouldn't be a need to change the attention mask value. It suggests that there could be an issue with the FlaxMBartForCausalLM model that you're adding. I've replied more in-depth on the issue as it's more relevant there https://github.com/huggingface/transformers/issues/19897#issuecomment-1294648919. Keeping this PR open until we determine whether it's a generic Flax BART issue or a FlaxBartForCausalLM one!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Unable to Finetune Deberta
Author: ra-MANUJ-an
Pull request: no

I am trying to finetune deberta for irony detection task, colab's notebook link can be found [here](https://colab.research.google.com/drive/1mZI5W2ozc8speZiwzOtCrWABhSaxqhWD?usp=sharing)

When I try to use 'microsoft/deberta-v3-base' checkpoint with AutoModel, I'm getting the following error :

RuntimeError: Expected target size [32, 2], got [32]
but when I use the same model with 'bert-base-uncased' or roberta (with some changes in head) it works fine. The one can find working code for bert based in [this](https://colab.research.google.com/drive/1PXacY2YgAfk6IYC0sAynp88Z6ndWrqYQ?usp=sharing) notebook.

When I printed the shapes of predictions and labels, I got outputs as torch.Size([32, 30, 2]), torch.Size([32]) respectively. In the case of bert, shapes of outputs were torch.Size([32, 2]), torch.Size([32]) for predictions and labels.

Here 32 is the batch size, and 30 is the sequence length.

Can someone let me know what I'm doing wrong?

Comments:
@sgugger 
@patil-suraj 
@patrickvonplaten 

Please use the [forums](https://discuss.huggingface.co/) to get help debug your code. In this instance you are using the base pretrained model (without a classification head) to do classification so it does not work. You should consider using AutoModelForSequenceClassification`.

okay, sure will take care from next time and thanks for the response! Just one question, do bert and roberta provide classification heads in their base models?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Should always set the pad_to_max_length =False when do whole word mask language model fine-tune
Author: bugm
Pull request: no

### Feature request

I am researching some work about whole word mask language model fine-tune and I am doing some customize change on the code from the official example (https://github.com/huggingface/transformers/tree/main/examples/research_projects/mlm_wwm)

I find there is a argument field in class DataTrainingArguments    
pad_to_max_length: bool = field(
        default=False,
        metadata={
            "help": (
                "Whether to pad all samples to `max_seq_length`. "
                "If False, will pad the samples dynamically when batching to the maximum length in the batch."
            )
        },
    )

although its default value is False, but when we set it to True, it will make some problem when we use the  DataCollatorForWholeWordMask collator.

### Motivation

according to the source code about DataCollatorForWholeWordMask, I think it selects the tokens to be masked among all input_tokens, if we do padding before collator processing, then during the word mask process we may get lots of candidate index with PAD token. I believe this kind of PAD token is no meaning for word mask model fine tune.

Although the default value pad_to_max_length is set to False, but I do found lot of people customize the code from the official examples by set tokenizer.padding to "max_length" and then call the map function on dataset.

   def tokenize_function(examples):
        # Remove empty lines
        examples["text"] = [line for line in examples["text"] if len(line) > 0 and not line.isspace()]
        return tokenizer(examples["text"], padding="max_length", truncation=True, max_length=data_args.max_seq_length)

    tokenized_datasets = datasets.map(
        tokenize_function,
        batched=True,
        num_proc=data_args.preprocessing_num_workers,
        remove_columns=[text_column_name],
        load_from_cache_file=not data_args.overwrite_cache,
    )

### Your contribution

I suggest to remove the padding action before the DataCollatorForWholeWordMask collator process and emphasize that padding action before the collator process may influence the training of our model.

Comments:
Note that this is not a maintained example, so we are not planning on making any changes to that script.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Spanish translation of the file multilingual.mdx
Author: SimplyJuanjo
Pull request: yes

# What does this PR do?

Adds the Spanish version of [multilingual.mdx](https://github.com/huggingface/transformers/blob/master/docs/source/en/multilingual.mdx) to [transformers/docs/source_es](https://github.com/huggingface/transformers/tree/main/docs/source/es)

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes https://github.com/huggingface/transformers/issues/15947 (issue)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?

## Who can review?

@omarespejel @osanseviero  

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

any other thing left to do? @omarespejel 

@SimplyJuanjo thank you! 🤗 

@sgugger LGTM 👍
Transformers from GCS (or custom filesystem).
Author: nsthorat
Pull request: no

### Feature request

Hi! I'm wondering if there will be support for a custom filesystem argument to "from_pretrained" for transformers, just like there is for datasets (https://huggingface.co/docs/datasets/filesystems).

### Motivation

This ideally would be great for running models in the cloud in "diskless" mode where there is no access to a real filesystem and model assets could be read into RAM via the same filesystem API that is used for datasets.

This would solve the issue of decoupling a binary from its data dependencies, in the same way it's done for datasets.

Thank you!

### Your contribution

Would love to help here, but presumably a HF expert would be much more suited to solve this problem. Happy to be eyes and a tester!

Comments:
Hi there! We don't plan on adding support for something else than the Hub/local disk for pretrained model in Transformers.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Floating point exception (core dumped) when using `transformers.onnx`
Author: blakechi
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.22.2
- Platform: Linux-5.13.0-41-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.10.0
- PyTorch version (GPU?): 1.12.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
### Who can help?

@LysandreJik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

#### Code
```bash
python -m transformers.onnx --model=google/long-t5-tglobal-base --feature=seq2seq-lm onnx
```

#### Error Message
```bash
Validating ONNX model...
Floating point exception (core dumped)
```

#### Another Trial
Although the model isn't an official one, but it appears in the [document](https://huggingface.co/docs/transformers/model_doc/longt5#transformers.LongT5ForConditionalGeneration.forward.example). Showed the same message as above.
```bash
python -m transformers.onnx --model=Stancld/longt5-tglobal-large-16384-pubmed-3k_steps --feature=seq2seq-lm onnx
```

### Expected behavior

It shouldn't raise `Floating point exception (core dumped)`.

Comments:
cc @lewtun 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Flan-T5 returns incomplete results
Author: dyxohjl666
Pull request: no

### System Info

transformer version: 4.19.2
platform: Linux
python: 3.8.13

### Who can help?

@LysandreJik

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-large")
tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-large")

inputs = tokenizer("Summarize the following text: Peter and Elizabeth took a taxi to attend the night party in the city. While in the party, Elizabeth collapsed and was rushed to the hospital. Since she was diagnosed with a brain injury, the doctor told Peter to stay besides her until she gets well. Therefore, Peter stayed with her at the hospital for 3 days without leaving.", return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

>>> ['Peter and Elizabeth went to a party together. Elizabeth collapsed and was rushed to the']
```

### Expected behavior

The generated text isn't complete. It seems to be truncated. I just use the example codes, so I have no idea about this problem.

Thanks for your help :)

Comments:
You can set min_length and max_length in model.generate() to adjust the length of generation settings. By default, max_length is not very long.

It works! Thanks!

> You can set min_length and max_length in model.generate() to adjust the length of generation settings. By default, max_length is not very long.
[Vision] fix small nit on `BeitDropPath` layers
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes a small nit for `DropPath` layers pointed out in: https://github.com/huggingface/transformers/pull/20550#discussion_r1039395745 & https://github.com/huggingface/transformers/pull/20550#discussion_r1039459045 

Preferred to fix this separately in a PR to avoid modifying too much files in #20550

cc @NielsRogge @patrickvonplaten


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add entries to `FEATURE_EXTRACTOR_MAPPING_NAMES`
Author: ydshieh
Pull request: yes

# What does this PR do?

Add entries to `FEATURE_EXTRACTOR_MAPPING_NAMES`

Not sure if there was any reason not to add these entries in `FEATURE_EXTRACTOR_MAPPING_NAMES`.

Furthermore, without these entries, we get some test failures for the (WIP) improved pipeline tests, because we now can generate tiny models for these config classes with the corresponding tokenizer/processor. (Previously these couldn't be generated). 
The failures are because this line
https://github.com/huggingface/transformers/blob/cc3d0e1b017dbb8dcbba1eb01be77aef7bacee1a/tests/pipelines/test_pipelines_feature_extraction.py#L182
is not able to skip relevant tests for these configs/models.

**Remark: I am going to add them to `TOKENIZER_MAPPING_NAMES` too**



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Cleanup config attrs
Author: ydshieh
Pull request: yes

# What does this PR do?

These are vision models, and they don't form encoder-decoder themselves (unlike some text models like `Bart`).

Furthermore, the current default value (specified in each config  class `__init__`) for these configs are `False`, which is the same as the default value in `PretrainedConfig`. So we can just remove it from the parameters, and rely on `**kwargs` in the call to `super.__init__` .


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix flax GPT-J-6B linking model in tests
Author: JuanFKurucz
Pull request: yes

# What does this PR do?

Fixes the loading of the model `EleutherAI/gpt-j-6B` as the current code links `EleutherAI/gptj-6B` which does not exist and ends up failing the test.

Comments:
_The documentation is not available anymore as the PR was closed or merged._