noamaanMulla-03/transformers-issues huggingface dataset:
Add ASR CTC streaming example 
Author: anton-l
Pull request: yes

# What does this PR do?

This enables the use of streamable datasets to fine-tune CTC models.
Example args:
```
--dataset_name="common_voice"
--model_name_or_path="ntu-spml/distilhubert"
--tokenizer_name_or_path="infinitejoy/wav2vec2-large-xls-r-300m-abkhaz"
--dataset_config_name="ab"
--output_dir="./dummy_ctc"
--overwrite_output_dir
--per_device_train_batch_size="4"
--gradient_accumulation_steps="1"
--learning_rate="5e-5"
--max_steps="3000"
--warmup_steps="500"
--evaluation_strategy="steps"
--text_column_name="sentence"
--save_steps="500"
--eval_steps="5"
--logging_steps="1"
--layerdrop="0.0"
--save_total_limit="1"
--mask_time_prob="0.3"
--mask_time_length="10"
--mask_feature_prob="0.1"
--mask_feature_length="64"
--freeze_feature_encoder
--chars_to_ignore=", ? . ! - \; \: \" % ‘ �"
--fp16
--do_train
--do_eval
--gradient_checkpointing
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

One thing left to add: support for infinite streaming if we reach the end of the dataset, so that `Trainer` doesn't stop until `max_steps` is reached.
Reference: https://github.com/huggingface/transformers/blob/48bf7e47a01f310ca8e76bd90be14e06dbd08329/examples/research_projects/codeparrot/scripts/codeparrot_training.py#L20

Otherwise this example yields:
```
There seems to be not a single sample in your epoch_iterator, stopping training at step 6! This is expected if you're using an IterableDataset and set num_steps (3000) higher than the number of available samples.
```

Maybe run the script for just 5 epochs both with streaming and without so that we have some exact numbers and can compare perf as well

Feel free to merge and announce this @anton-l once there have been some testing :-)

The ASR README now includes a streaming example+benchmark :) 

A couple of last notes:
1. streaming will run in distributed mode only with the Trainer fixes from this PR (to support `IterableDatasetShard`)
2. https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0 is very flaky (connection drops after a couple of epochs, possibly due to a temporary IP ban), so the benchmark only succeeded with `common_voice`

TODO: Merge this after https://github.com/huggingface/transformers/pull/15539
[Trainer] Deeper length checks for IterableDatasetShard
Author: anton-l
Pull request: yes

# What does this PR do?

As discussed in https://github.com/huggingface/transformers/pull/15309#discussion_r795553017 , in cases where `IterableDatasetShard` wraps a non-sized dataset, the check for `isinstance(dataset, Sized)` passes, while `len(dataset)` fails. 
This PR adds a universal `has_length(dataset)` that tries to get the dataset's length explicitly without relying on type checking.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Wav2Vec2 models must either throw or deal with add_apater
Author: FremyCompany
Pull request: yes

Using `Wav2Vec2Model` models with `config.add_adapter=True` as `Wav2Vec2ForCTC` (for instance) doesn't throw on creation, but eventually fails due to mismatch in the output dimension (`config.output_hidden_size`) and the `lm_head` layer put on top of it (`config.output_hidden`).

# What does this PR do?

I propose:

  1. to fix this for two sub-models that only rely on the last output (by using `output_hidden_size` when `add_adapter` is set), and 
  2. to raise an exception for the other types of models (that try to average multiple layers). Raising an explicit error is better than failing during the forward pass.

Another option is:

  3. to always throw if add_adapter is set.

## Before submitting
/!\ This is a draft pull request for review of the general proposal. I did not test this code or write tests yet. I edited the file online directly on github.


## Who can review?

Cc @patrickvonplaten since this is related to the robust-speech-event (community event)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Based on the current CI results:
1. This needs reformatting if we want to merge.
2. All the Wav2Vec2-like models should mirror this change.
3. Ideally, a test would be written to ensure this doesn't regress.

But I still want to have feedback on the preferred approach before considering to go through all the hoops.

Hey @FremyCompany,

Thanks for the PR. I think it's a welcoming change. Could you run:

```make fix-copies``` so that this change is automatically copied to other varients of Wav2Vec2?

There is an issue with the changes propagated by the fix-copies command.
```
AttributeError: 'UniSpeechSatConfig' object has no attribute 'add_adapter'
```

I'm not sure how that should be fixed. Any idea?

I went for an `hasattr` check. 
(Assumption: if the key is absent, it defaults to False).
Remove Longformers from ONNX-supported models
Author: lewtun
Pull request: yes

# What does this PR do?

This PR removes Longformer from the list of ONNX-supported models due to a current limitation in the supported ops in JIT - see issue #15217  

This slipped through our unit tests because we didn't include `longformer` in the list of models to be tested.

Long term, we'll either need to wait for JIT to support the problematic ops or tweak the Longformer implementation.


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Create a custom model guide
Author: stevhliu
Pull request: yes
Labels: name: Documentation

First draft of a guide for how to create a model without using any of the `Auto...` classes to give users a better idea of what's happening behind the automagic. The goal of this guide is to show users alternative methods for creating a model. It also demonstrates how users can instantiate these classes themselves if they want to customize or experiment with the default attributes/parameters loaded from a pretrained model. So in a sense, it is also a guide for creating a custom model.

Some feedback I would appreciate:
- Would adding a graphic showing `configuration -> model <- tokenizer/feature extractor/processor` help show how a model is created?
- Would adding an end-to-end code sample (setting up a custom configuration, model and tokenizer) at the end help tie everything together?
- Are there more details I can add around creating a custom model?
- How is my tone for training a model from scratch? I feel like it might be a bit too harsh (see line 118). 😅 

Thanks and let me know if I'm missing anything!

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix TF T5/LED missing cross attn in retrun values
Author: ydshieh
Pull request: yes

# What does this PR do?

Add cross attention to return values for TF T5/LED (so aligned with PT's version).

(Also needed to add a fix of `undoing padding`. Same undoing padding is required for longformer, but it is done in another PR)

@patrickvonplaten 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Great PR - thanks a lot!

CI failure is unrelated
Make TF Wav2Vec2 outputs the same as PT's version
Author: ydshieh
Pull request: yes

# What does this PR do?

Current TF Wav2Vec2  doesn't return `extract_features`, unlike in PT's version.
This PR adds this to TF Wav2Vec2 outputs.

@patrickvonplaten 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
No module named 'transformers.models.bort'
Author: abhilashreddys
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.16.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU?): 1.10.0+cu111 (True)
- Tensorflow version (GPU?): 2.7.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help 
@sgugger @Narsil

## Error
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/transformers/file_utils.py", line 2704, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'transformers.models.bort'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/content/infoADAPET/src/test.py", line 5, in <module>
    from transformers import *
  File "<frozen importlib._bootstrap>", line 1031, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 1032, in _handle_fromlist
  File "/usr/local/lib/python3.7/dist-packages/transformers/file_utils.py", line 2692, in __getattr__
    value = self._get_module(name)
  File "/usr/local/lib/python3.7/dist-packages/transformers/file_utils.py", line 2708, in _get_module
    ) from e
RuntimeError: Failed to import transformers.models.bort because of the following error (look up to see its traceback):
No module named 'transformers.models.bort'
```


Comments:
v4.16.0  has this issue changing to v4.15.0 fixed it

What command did you run exactly?

from transformers import *

Thanks for flagging, a fix is on its way :-)

Fixing on https://github.com/huggingface/transformers/pull/15378
Fix RuntimeError on generation_utils.py
Author: ipari
Pull request: yes

This PR fixes a below runtime error on generation_utils.py

```
RuntimeError: Integer division of tensors using div or / is no longer supported, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.
```

Comments:
Think https://github.com/huggingface/transformers/pull/15498 superseeds this PR a bit as it corrects all occurances of PyTorch's integer divison. Hope it's ok if we go for https://github.com/huggingface/transformers/pull/15498 instead :sweat_smile:
gptj doc tied weights mistake?
Author: Muennighoff
Pull request: no

https://github.com/huggingface/transformers/blob/5f1918a4a8ed893822aa7dd2b75acf83f255ad79/src/transformers/models/gptj/modeling_gptj.py#L686
```
    The GPT-J Model transformer with a language modeling head on top (linear layer with weights tied to the input
    embeddings).
```

The lm head isn't tied to the input embeddings for gpt-j

Comments:
Good catch!
very large model on multi gpu
Author: ahtamjan
Pull request: no

I converted XLMR-xxl from fairseq to transformers, and the model is too large to load in one GPU (the model is about 40g). I have 4 V100-32G gpu and how can I fix this problem ? How can I split and load the model to 2 gpus?

Comments:
Hi @ahtamjan, I would advise you to read this very complete document that @stas00 created: [Transformers docs: Performance and scalability](https://huggingface.co/docs/transformers/master/en/performance#performance-and-scalability-how-to-fit-a-bigger-model-and-train-it-faster)

Probably head directly to Deepspeed docs: https://huggingface.co/docs/transformers/master/en/main_classes/deepspeed#trainer-deepspeed-integration

I will update the performance doc to make the link as it mentions it but isn't linking to it directly.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Add support for TPU training with CodeParrot!
Author: ncoop57
Pull request: yes

# What does this PR do?

This PR adds support for training CodeParrot-style models on TPUs! It also fixes a small issue in the model initialization script.

You can verify it works using this colab notebook: https://colab.research.google.com/drive/143qMz_0yf1Irb1ZrsYIWu39nqBYh9OTd?usp=sharing

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@lvwerra 


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@ncoop57 let me know when you had a chance to test this and I'll reopen the PR.
Add TFSpeech2Text
Author: gante
Pull request: yes
Labels: name: TensorFlow

# What does this PR do?

This PR adds a TF port of Speech2Text. A summary of the changes:
- This model borrows a lot of code from TFBart, just like Speech2Text borrowed from Bart;
- Tried to follow the changes in other PRs to enable smooth interoperation with other parts of transformers (e.g. auto classes), might be missing a few things 👼 ;
- This seems to be the first TF model with speech as input, so I had to touch common TF code to enable correct data piping and misc operations (e.g. enable loading Conv1D PT weights into TF);
- Likewise, there were a few tests in `test_modelling_tf_common.py` that didn't quite fit this new kind of model.

TODO: 
- [x] create TF version of the weights, so we can load a model without `from_pt=True`

Comments:
Tagging @sgugger as a core dev, @patil-suraj as a core dev + original creator of our `Speech2Text`, and @Rocketknight1 as the TensorFlow boy. Feel free to redirect the reviews if you know of better people to review.

Some pipeline tests are failing almost surely due to the changes in `generation_tf_utils.py`, as some models expect `(encoder_outputs, past)` in the `past` variable and others don't -- having a look, but open to suggestions.

Cloned your branch and did some experimentation and LGTM! I noticed one issue - `model.save()` seems to encounter problems, but I'm not totally sure of why, or whether it's limited to this model or not. `save_pretrained` and `save_weights` worked correctly, and saving in `SavedModel` format has always been a bit shaky for us, so this isn't critical.

Pending the results of automated tests, this should be the last planned commit. We already have `facebook/s2t-small-librispeech-asr` as a TF model, and I will upload the TF version of the others today. 

@patrickvonplaten -- One important final change that I'd like to ask for a double-check is the removal of the positional embeddings weights from the `nn.Parameter()` wrapper, in the PT model. It is a constant that was not being saved nor loaded, and was causing issues in the `pt_tf` tests (the TF model had no such variable, and technically it is not a parameter). 

To check the changes, I've run:
```
RUN_SLOW=1 pytest tests/test_modeling_tf_vision_encoder_decoder.py
RUN_SLOW=1 pytest tests/test_modeling_tf_bart.py
RUN_SLOW=1 pytest tests/test_modeling_tf_t5.py
RUN_SLOW=1 pytest tests/test_modeling_tf_rag.py
RUN_SLOW=1 pytest tests/test_modeling_speech_to_text_2.py
RUN_PT_TF_CROSS_TESTS=1 RUN_SLOW=1 pytest tests/test_modeling_speech_to_text.py
RUN_PT_TF_CROSS_TESTS=1 RUN_SLOW=1 pytest tests/test_modeling_tf_speech_to_text.py
```

EDIT: TF models uploaded.

@patrickvonplaten reverted the previous change and added the embedding weights as named variables in TF
Add in `model.generate` support to encoder outputs  for flax models
Author: borisdayma
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->
Add support to encoder outputs in `model.generate` for flax models (similar to Pytorch models).

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->
With JAX, we need to pass a seed for each generation iteration. So when we want different samples, we may need to run the same function multiple times.

For Encoder/Decoder models such as Bart, we could calculate the encoder outputs once and then directly pass them to the decoder, which could potentially save half of the computations (when encoder/decoder have same size).

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

Not sure the best way to implement it but it seems that they typically come from `model._prepare_encoder_decoder_kwargs_for_generation`

@patil-suraj @patrickvonplaten 

Comments:
Hey @borisdayma, 

Aren't we already doing this here: https://github.com/huggingface/transformers/blob/b8810847d0576e3c142854ad3b8a607ecd3df291/src/transformers/generation_flax_utils.py#L140 ? 

If you do:
```python
model.generate(…)
model.generate(…)  #same params
```

You will get twice a call to `model.encode(…)`.

You will call several times `model.generate(…)` if you want for example a bunch of predictions (dalle-mini case) and then sort them with another model (CLIP scoring).

In that case the encoder part does not depend on the key so we could cache it and allow passing it as an argument to `model.generate(encoder_outputs = manually_cached_value)`.

I think it's just about checking if `encoder_outputs` is part of `model_kwargs` in `_prepare_encoder_decoder_kwargs_for_generation`.

I see! Yeah you're right it should be as easy as this. We can add a `encoder_outputs` to the `generate()` function and then only call `encode()` if it's not present (that's what we're doing in PyTorch as well). Would you like to open a PR about it maybe? :-)
feat(flax): allow encoder_outputs in generate
Author: borisdayma
Pull request: yes

# What does this PR do?
Allow `encoder_outputs` in flax `model.generate`

Fixes #15161 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@patrickvonplaten @patil-suraj 
<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

It seems to work fine.
Not a big speedup though as sampling iteratively is probably what takes most time.
Add Wav2Vec2 Adapter Weights to Flax
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Fixes #15476

- Adds an adapter to the Flax Wav2Vec2 model to reduce the time dimension of the extracted feature vectors beyond that of the standard Wav2Vec2 model. The encoder's output hidden states thus have a time context window that is more similar to that of a subword token instead of just a character. 
- Shape and values of Flax output logits match those of the PyTorch model.
- Flax model uses all PyTorch model weights, including those of the adapter. Running the script in #15476 resolved to yield identical results (within 4e-2 threshold).

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I think the commit history is messed up here - the best is usually to just reopen a new PR and to just extract your changes from this PR.