noamaanMulla-03/transformers-issues huggingface dataset:
T5 mlm Flax streaming example
Author: gsarti
Pull request: yes

# Added T5 mlm Flax streaming example

This PR adds an example script for T5 MLM Pretraining using 🤗 Datasets streaming feature. A new script `run_mlm_t5_flax_stream.py` is added in the `jax-projects/dataset-streaming` folder, and the `README.md` is updated accordingly with a training example for `t5-small` on the `mc4/en` corpus in streaming mode.

As mentioned in the Slack channel, I ran some preliminary tests on mc4/it and I get pretty weird results (train loss converges very early, eval metrics remain very low), possibly due to some problem with the adapted collating/tokenization, so this PR would greatly benefit from reviewing before merging.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->


## Who can review?

@patrickvonplaten @patil-suraj


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Gradient checkpointing in Wav2Vec2
Author: Getmany1
Pull request: no

Hi,
Has anyone managed to fine-tune a **Wav2Vec2** model on long audio recordings which cannot fit into a GPU even with `batch_size=1`?  I tried out to set `gradient_checkpointing=true`, but it didn't help to solve the _CUDA Out of Memory Error_. Could it mean that gradient checkpointing does not work properly with **Wav2Vec2** models or are there other tricks needed to be added to the fine-tuning script in addition to the gradient checkpointing?

Comments:
Hi! You can check out this related issue: https://github.com/huggingface/transformers/issues/10366

Thanks @LysandreJik! Yeah, one solution could be modify the `run_asr.py` script and add segmenting long speech samples by manually splitting each `batch["speech"]` into smaller chunks before passing to the _wav2vec2 processor_ and converting to input values, then passing one chunk at a time to the model and after that merging the outputs into a single transcription and calculating the loss. Just wondering if there could be any other ways / built-in features to split a batch of size 1 into smaller mini-batches.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi @Getmany1  I also meet this error，But my audio file is no longer than 15s. How to solve the error? and how to pass one chunk at a time to the model.
t5 base not found.
Author: s4sarath
Pull request: no

- 't5-base' is a correct model identifier listed on 'https://huggingface.co/models'

- or 't5-base' is the correct path to a directory containing a config.json file

Comments:
Hey @s4sarath,

https://huggingface.co/t5-base looks normal to me - what exactly is your error?

Hi @patrickvonplaten , t5-small is working fine. Transformers version is **4.9.0.dev0**

```
from transformers import TFT5Model
model = TFT5Model.from_pretrained("t5-base")
```

OSError: file t5-base/config.json not found
OSError: Can't load config for 't5-base'. Make sure that:

- 't5-base' is a correct model identifier listed on 'https://huggingface.co/models'

- or 't5-base' is the correct path to a directory containing a config.json file

Do you have a folder called `t5-base` in your working directory? The `TFT5Model` may be trying to load from that directory rather than from the hub.

Hi Lysandre. You were true. My bad.
Closing this ticket.

Thanks for your help.

On Wed, 11 Aug, 2021, 6:58 pm Lysandre Debut, ***@***.***>
wrote:

> Do you have a folder called t5-base in your working directory? The
> TFT5Model may be trying to load from that directory rather than from the
> hub.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/13073#issuecomment-896827708>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACRE6KG7EFEL3Y4QQQCD4Q3T4J3H7ANCNFSM5B4SPOOA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>
> .
>


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Problem saving tf wav2vec in savedmodel format
Author: ahmedlone127
Pull request: no

![image](https://user-images.githubusercontent.com/66001253/128309653-9939e715-2792-419c-b310-0a18fb92194d.png)
this is my code 

Comments:
Hey @ahmed451 - could you please add a code snippet to reproduce the error instead of a screenshot? Thanks!

sure

 ```
from transformers import TFWav2Vec2ForCTC

model= TFWav2Vec2ForCTC.from_pretrained('patrickvonplaten/wav2vec2-base-timit-demo',from_pt = True)

model.save_pretrained("/content/test",saved_model = True) ```

I've run your code snippet in the following environment:

```
- `transformers` version: 4.10.0.dev0
- Platform: Linux-4.15.0-112-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.5
- Tensorflow version (GPU?): 2.3.0 (False)
```


and to me it looks like this is a problem coming from Tensorflow directly. I.E. the error output is:

```
UnimplementedError: The Conv2D op currently does not support grouped convolutions on the CPU. A grouped convolution was attempted to be run because the input depth of 768 does not match the filter input depth of 48 [Op:Conv2D]
```

Also gently pinging @will-rice here in case he has seen something like this before :-)

This is/was a TensorFlow limitation, but according to [this](https://github.com/tensorflow/tensorflow/issues/29005) `2.6` may have solved it. First I would try upgrading to `2.6` or the latest nightly. Another option could be the [workaround](https://github.com/tensorflow/tensorflow/issues/40044) for this problem in TFLite. I will say the workaround is slower though.

@patrickvonplaten  How do I install the transformers 4.10.0 version?

It's current master :-) So `!pip install git+https://github.com/huggingface/transformers.git@master` should do

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
ResourceExhaustedError: Failed to allocate request for 64.00MiB (67108864B) on device ordinal 0
Author: Jassy87
Pull request: no

I'm training an XLM model from huggingface on a small dataset using Colab TPU. However, I get the following error:
<ResourceExhaustedError: Failed to allocate request for 64.00MiB (67108864B) on device ordinal 0>

I tried reducing the batch size and restarting the kernel and freeing memory but nothing is working.
Any help would be appreciated.

Comments:
Can you give some additional info? like the number of layers and the batch size?

I'm following this notebook ->https://www.kaggle.com/dimasmunoz/text-classification-with-roberta-and-tpus
I've tried Roberta, BERT, DistilBert and all working okay on my dataset. Except for XLM !
`MODEL_NAME = 'xlm-mlm-en-2048'    ## 12 layers, 16 heads
MAX_LEN = 256
ARTIFACTS_PATH = '../artifacts/'

BATCH_SIZE = 4 * strategy.num_replicas_in_sync
EPOCHS = 3`

Worked fine for the EnghlishGerman model "xlm-mlm-ende-1024" with 6 layers, 8 heads
[megatron_gpt2] checkpoint v3
Author: stas00
Pull request: yes

The BigScience generates Megatron-Deepspeed checkpoints en masse, so we need a better v3 checkpoint support for megatron_gpt2 (also working on a direct Megatron-Deepspeed to HF-style script).

This PR:
- removes the requirement that the source checkpoint is a zip file. Most of the checkpoints aren't .zip files, so supporting both.
- removes the need to manually feed the config, when all the needed data is already in the checkpoint. (while supporting the old format)
- disables a debug print that repeats for each layer
- switch to default `gelu` from `gelu_new` - which is what the current megatron-lm uses
- fixes a bug in the previous version. The hidden size is dimension 1 as it can be seen from:
https://github.com/NVIDIA/Megatron-LM/blob/3860e995269df61d234ed910d4756e104e1ab844/megatron/model/language_model.py#L140-L141
The previous script happened to work because `max_position_embeddings` happened to be equal to `hidden_size`

@LysandreJik, @sgugger 


Comments:
Add error message to Wav2Vec2 & Hubert if labels > vocab_size
Author: patrickvonplaten
Pull request: no
Labels: name: Good First Issue

# 🚀 Feature request

Add better error message to `HubertForCTC`, `Wav2Vec2ForCTC` if labels are bigger than vocab size.

## Motivation

Following this issue: https://github.com/huggingface/transformers/issues/12264 it is clear that an error message should be thrown if any of the any of the labels are > `self.config.vocab_size` or else silent errors can sneak into the training script.

So we should modify: `Wav2Vec2ForCTC`, `TFWav2Vec2ForCTC`, and `HubertForCTC` to add a nice error message in this case.

## Your contribution

This is a first good issue and should be rather easy to accomplish. I'm happy to give more guidance if needed.


Comments:
I will create a PR to fix this.

@vasudevgupta7 @patrickvonplaten Is this fixed? If not, I will like to work on it. 

Hey, this issue has been fixed in this PR: https://github.com/huggingface/transformers/pull/12288

Thanks for informing. I had seen it, but since the issue is still open, I thought something might be left. 

Closing as fixed :)
[typo] invalid URL
Author: Freed-Wu
Pull request: no

In <https://github.com/huggingface/transformers/blob/master/docs/source/training.rst?plain=1#L36>

```
We will use the `🤗 Datasets <https:/github.com/huggingface/datasets/>`__ library to download and preprocess the IMDB
```

Correct URL should be <https://github.com/huggingface/datasets/>.

Comments:
Closed by #13644
Dead link in docs
Author: dingusagar
Pull request: no

https://huggingface.co/transformers/model_doc/distilbert.html
In this page, under the paragraph,

_This model was contributed by victorsanh. This model jax version was contributed by kamalkraj. The original code can be found [here](https://github.com/huggingface/transformers/blob/master/examples/research-projects/distillation)._

The link to the original code is broken.

Comments:
Fixed by #13643
Key Error: 'pre-processing' during conversion from tatoeba to Marian model
Author: velocityCavalry
Pull request: no
Labels: name: WIP

## Environment info

- `transformers` version: `4.6.0.dev0`
- Platform: `CentOS Linux release 7.7.1908 (Core)`
- Python version: `3.8.5`
- PyTorch version: `1.8.1 + cuda 10.2`
- Tensorflow version: N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

Marian: @patrickvonplaten , @patil-suraj 

## Information

Model I am using (Bert, XLNet ...): Marian

The problem arises when using:
* [x] the official example scripts: tatoeba to marian model script
* [ ] my own modified scripts

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: machine translation
* [ ] my own task or dataset

## To reproduce


Following the script from [scripts/tatoeba/README.md ](https://github.com/huggingface/transformers/tree/master/scripts/tatoeba)
1. 
```git clone git@github.com:huggingface/transformers.git
cd transformers
pip install -e .
pip install pandas GitPython wget
```
2. 
```
curl https://cdn-datasets.huggingface.co/language_codes/language-codes-3b2.csv  > language-codes-3b2.csv
curl https://cdn-datasets.huggingface.co/language_codes/iso-639-3.csv > iso-639-3.csv
```
3. `git clone git@github.com:Helsinki-NLP/Tatoeba-Challenge.git`
4. `python src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py --models kor-eng eng-kor --save_dir converted/`

Error message:
```
Traceback (most recent call last):
  File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 1267, in <module>
    resolver = TatoebaConverter(save_dir=args.save_dir)
  File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 58, in __init__
    reg = self.make_tatoeba_registry()
  File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 258, in make_tatoeba_registry
    return [(k, v["pre-processing"], v["download"], v["download"][:-4] + ".test.txt") for k, v in results.items()]
  File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 258, in <listcomp>
    return [(k, v["pre-processing"], v["download"], v["download"][:-4] + ".test.txt") for k, v in results.items()]
KeyError: 'pre-processing'
```

## Expected behavior

Conversion of the model from Tatoeba to Marian for the chosen language pair with no errors.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

unstale

@patil-suraj - It would be really nice if we could tackle the tatoeba models at some point...

This seems to be related: https://github.com/huggingface/transformers/pull/12192
https://github.com/huggingface/transformers/issues/10943
[SequenceFeatureExtraction] Move padding logic from pure python to numpy
Author: patrickvonplaten
Pull request: no

Currently the padding for sequential feature extraction: https://github.com/huggingface/transformers/blob/9d60eebeb52ed3c266ab8e0cc6871ebeb08a5bc1/src/transformers/feature_extraction_sequence_utils.py#L38 is done in pure Python even though the two feature extractors that rely on sequential both always return either a single numpy (or PT or TF) tensor or a list of numpy tensors - see:

- https://github.com/huggingface/transformers/blob/9d60eebeb52ed3c266ab8e0cc6871ebeb08a5bc1/src/transformers/models/wav2vec2/feature_extraction_wav2vec2.py#L205 for Wav2Vec2
- https://github.com/huggingface/transformers/blob/9d60eebeb52ed3c266ab8e0cc6871ebeb08a5bc1/src/transformers/models/speech_to_text/feature_extraction_speech_to_text.py#L209 

More specifically: 

1) `feature_extractor(raw_audio, return_tensors=None, padding=True)` => returns a list of  numpy arrays
2) `feature_extractor(raw_audio, return_tensors="np", padding=True)` => returns a 2D numpy array (PT and TF return a 2D array in PT and TF correspondingly)

For some background the text tokenizers all have the option to return "pure" Python lists which was then implemented analogous for the speech feature extractors. However it was decided that the each individual sequence will always have be a numpy array. This is because a very common use case is to load audio files via `torchaudio` or `librosa` which return a list of numpy arrays and then run the feature extractor on it. Running `feature_extractor(list_of_numpy_arrays, padding=True)` should then not change the inner numpy arrays to pure python as this would be confusing for the user and also very be very slow. 

Therefore I think the current design is good as is and that we can move the padding logic in https://github.com/huggingface/transformers/blob/9d60eebeb52ed3c266ab8e0cc6871ebeb08a5bc1/src/transformers/feature_extraction_sequence_utils.py#L38 from "pure" Python to a numpified approach.. This would both improve speed and prevent instability issues like: https://github.com/huggingface/transformers/pull/13538 . This won't change anything in the user-facing API since at the moment the the "inner" output is always a numpy vector anyways. 

Do you agree on this @LysandreJik @sgugger @anton-l @patil-suraj  ?

@anton-l - are you maybe interested in giving this PR a try? 

Comments:
I agree with this proposition.

Yeah, I'll take this one, thanks for the pointers @patrickvonplaten :+1: 

Very much agree! +1

This sounds great!
[SequenceFeatureExtractor] Rewrite padding logic from pure python to numpy
Author: anton-l
Pull request: yes

# What does this PR do?

Resolves #13539 

Since speech models universally use Numpy `float32` arrays as input features (standard way of representing waveforms), it was decided to rewrite `SequenceFeatureExtractor` from pure python lists (akin to traditional tokenizers) to numpy arrays. It will also help with solving some inconsistent normalization issues (#13538, #13585) due to `float->np.float32` conversions.

The feature extractor itself is still dtype-agnostic (can pad `np.float64` in the future if needed), while the model-specific feature extractors were updated to only work with `np.float32`

Comments:
Also did you notice a speed-up for larger inputs?

@patrickvonplaten the benchmarking results are pretty promising:

1. Input lengths from 8000 to 16000 (1 sec max), batch size 64, feature_extractor only: 
    - Python: 52.1 ms ± 2.35 ms
    - Numpy: **32.1 ms** ± 1.13 ms

2. Input lengths from 8000 to 160000 (10 sec max), batch size 64, feature_extractor only: 
    - Python: 276 ms ± 950 µs
    - Numpy: **68.2 ms** ± 491 µs
    

Great job @anton-l - feel free to merge!
type object 'AutoModelForSequenceClassification' has no attribute 'from_config'
Author: sujeethshetty
Pull request: no

I'm using Transformer version 4.4.2 and have been getting "type object 'AutoModelForSequenceClassification' has no attribute 'from_config'" error. Here is my code snippet. I went through the document, the syntax seem to be correct. Your help is very much appreciated. 

from transformers import AutoConfig, AutoTokenizer, AutoModel, AutoModelForSequenceClassification, Trainer, TrainingArguments

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
config = AutoConfig.from_pretrained('distilbert-base-uncased', num_labels=2)



Comments:
Hello! I just tried the following code snippet in both `v4.9.2` and `v4.4.2` and both seem to work:

```py
from transformers import AutoConfig, AutoTokenizer, AutoModel, AutoModelForSequenceClassification, Trainer, TrainingArguments

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
config = AutoConfig.from_pretrained('distilbert-base-uncased', num_labels=2)

AutoModelForSequenceClassification.from_config(config)
```


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Can we directly replace gpt2LMHeadModel with BertLMHeadModel to see bert's performance? #7
Author: ZonglinY
Pull request: no

I have a code for gpt2LMHeadModel which runs well, and I want to test my code on BertLMHeadModel. 
But when I directly replace gpt2LMHeadModel with BertLMHeadModel and replace gpt2Tokenizer with BertTokenizer, the ppl remains at 1 (the BertLMHeadModel predicts perfectly the same with labels)

So can anyone help me, is there any difference of the input format between gpt2LMHeadModelwith and BertLMHeadModel? 

Thanks so much!

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[SinusoidalPositionalEmbedding] incorrect dtype when resizing in `forward`
Author: stas00
Pull request: yes

This PR fixes a potential performance issue in general and a failure under Deepspeed when the following models are used under mixed precision with positional embedding resizing at `forward` time:
- speech_to_text
- m2m_100
- fsmt

Currently when `SinusoidalPositionalEmbedding.forward` is called if it resizes the embeddings it ignores the original correct dtype and forces the embeddings into `fp32`, so the inputs are in `fp32` now.

I detected the issue with deepspeed, which doesn't use `amp` but forces the model into `fp16` and then of course if the input is in the wrong dtype we get:
```
deepspeed  examples/pytorch/translation/run_translation.py --train_file tests/fixtures/tests_samples/wmt_en_ro/train.json --source_lang en --target_lang ro --model_name_or_path hf-internal-testing/tiny-random-m2m_100 --do_train --max_train_samples 4 --per_device_train_batch_size 2 --num_train_epochs 1 --fp16 --report_to none --overwrite_output_dir --deepspeed tests/deepspeed/ds_config_zero2.json --output_dir /tmp/tmpi4k4wz8s --save_steps 1
[...]
  File "/mnt/nvme1/code/huggingface/transformers-ds-model-zoo-2/src/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward
    hidden_states = self.final_layer_norm(hidden_states)
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/torch/nn/modules/normalization.py", line 173, in forward
    return F.layer_norm(
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/torch/nn/functional.py", line 2346, in layer_norm
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: expected scalar type Float but found Half
```

So `hidden_states` ends up being `fp32` instead of `fp16` because the pos_emb is `fp32`.

I checked all models matching `SinusoidalPositionalEmbedding` and all the others that aren't modified by this PR don't do dynamic resizing at run time.

I haven't checked non-`SinusoidalPositionalEmbedding` - perhaps those have an issue too.

The test will be in https://github.com/huggingface/transformers/pull/12695 as soon as this PR gets merged.

@patil-suraj, @LysandreJik, @sgugger 


Comments: