noamaanMulla-03/transformers-issues huggingface dataset:
Can't load config for hosted model, works when downloaded
Author: MatejUlcar
Pull request: no

I have recently (19 hours ago) uploaded a new model to huggingface: https://huggingface.co/EMBEDDIA/sloberta

When attempting to load it with `model = AutoModelForMaskedLM.from_pretrained("EMBEDDIA/sloberta")`  I get the following error:

```
Traceback (most recent call last):
  File "/home/mulcar/.conda/envs/transformerslatest/lib/python3.8/site-packages/transformers/configuration_utils.py", line 353, in get_config_dict
    raise EnvironmentError
OSError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mulcar/.conda/envs/transformerslatest/lib/python3.8/site-packages/transformers/modeling_auto.py", line 1105, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
  File "/home/mulcar/.conda/envs/transformerslatest/lib/python3.8/site-packages/transformers/configuration_auto.py", line 272, in from_pretrained
    config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/mulcar/.conda/envs/transformerslatest/lib/python3.8/site-packages/transformers/configuration_utils.py", line 362, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for 'EMBEDDIA/sloberta'. Make sure that:

- 'EMBEDDIA/sloberta' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'EMBEDDIA/sloberta' is the correct path to a directory containing a config.json file
```

If I download the model from the huggingface.co, eg. cloning the model's repo, it loads perfectly fine. Is there a waiting time before a new model is completely added to the system? Or is there an other issue going on?

Comments:
Hello! I can do `model = AutoModelForMaskedLM.from_pretrained("EMBEDDIA/sloberta")` without any issues on my end.

Could it be linked to a connection issue?

```py
>>> from transformers import AutoModelForMaskedLM
>>> model = AutoModelForMaskedLM.from_pretrained("EMBEDDIA/sloberta")
Downloading: 100%|██████████| 520/520 [00:00<00:00, 215kB/s]
Downloading: 100%|██████████| 443M/443M [00:38<00:00, 11.5MB/s]

```

Tested from two computers on two different networks, didn't work. Managed to load it on Google Colab, though.

Anyway, it does work now. I was foiled by conda loading an old version of transformers instead of a newer one. Thanks!
Import error for class Speech2TextProcessor, Speech2TextTransformerForConditionalGeneration
Author: plutonium-guy
Pull request: no

## Environment info
- `transformers` version: 4.4.0.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.8
- PyTorch version (GPU?): 1.7.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help

Models:
- Speech2TextProcessor, Speech2TextTransformerForConditionalGeneration @patil-suraj 


## Information

Model I am using (Speech2TextProcessor, Speech2TextTransformerForConditionalGeneration):

The problem arises when using:
* trying to import the model


## To reproduce

Steps to reproduce the behavior:

1. first install master brach 
2. then try to import the  Speech2TextProcessor, Speech2TextTransformerForConditionalGeneration class then showing import error
```
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-8149e8a8d76d> in <module>
      1 import torch
----> 2 from transformers import Speech2TextProcessor, Speech2TextTransformerForConditionalGeneration
      3 from datasets import load_dataset
      4 import soundfile as sf

ImportError: cannot import name 'Speech2TextProcessor' from 'transformers' (unknown location)
```
## Expected behavior
able to run the code from _https://huggingface.co/facebook/s2t-large-librispeech-asr_ from docs


one suggestion, if any fine-tuning script for s2t-large-librispeech-asr model it would be a great help

thankyou

Comments:
hi @amiyamandal-dev 
Thank you for your interest in `S2T`. It's still a work in progress and not available on master yet. If you want to try it, you could checkout this PR #10175

Hey @amiyamandal-dev ,

The model is now available on [master](https://huggingface.co/transformers/master/model_doc/speech_to_text.html)! You could install transformers from the source if you want to try it.
[Question] How do I prevent a lack of VRAM halfway through training a (Pegasus) model?
Author: lars-at-styx
Pull request: no

I'm taking a pre-trained pegasus model (specifically, google/pegasus-cnn_dailymail, and I'm using Huggingface transformers through Pytorch) and I want to finetune it on my own data. This is however quite a large dataset and I've run into the problem of running out of VRAM halfway through training, which because of the size of the dataset can be a few days after training even started, which makes a trial-and-error approach very inefficient.

I'm wondering how I can make sure ahead of time that it doesn't run out of memory. I would think that the memory usage of the model is in some way proportional to the size of the input, so I've passed truncation=True, padding=True, max_length=1024 to my tokenizer, which if my understanding is correct should make all the outputs of the tokenizer of the same size per line. Considering that the batch size is also a constant, I would think that the amount of VRAM in use should be stable. So I should just be able to cut up the dataset into managable parts, just looking at the ram/vram use of the first run, and infer that it will run smoothly from start to finish.

However, the opposite seems to be true. I've been observing the amount of VRAM used at any time and it can vary wildly, from ~12GB at one time to suddenly requiring more than 24GB and crashing (because I don't have more than 24GB).

So, how do I make sure that the amount of vram in use will stay within reasonable bounds for the full duration of the training process, and avoid it crashing due to a lack of vram when I'm already days into the training process?


Comments:
I think I may have been an idiot; shortly after posting this I found that instead of `padding=True` I should set `padding='max_length'`. Woops.
Unable To Load Pretrained Longformer Models' Tokenizers
Author: UmerTariq1
Pull request: no

## Environment info
- `transformers` version: 4.4.0.dev0
- Platform: Windows
- Python version : 3.7.10
- Using GPU in script?: Issue is with both
- Using distributed or parallel set-up in script?: Single device

@patrickvonplaten   (because the issue is with longformers)
@LysandreJik            (because the issue is with tokenizers)

## Information
Model I am using : Longformer.
The problem arises when loading tokenizer using from_pretrained() function. 
The tasks I am working on is Question Answering but it does not matter since I am facing this issue while loading any kind of Longformer:

## To reproduce
Steps to reproduce the behavior:
1. Install Transformers
2. import Transformers
3. run tokenizer = transformers.AutoTokenizer.from_pretrained(MODEL_NAME)

## Reference Code:
```
!pip3 install git+https://github.com/huggingface/transformers
import transformers
DEEP_LEARNING_MODEL_NAME = "mrm8488/longformer-base-4096-finetuned-squadv2"               # Not working for 4.4.0.dev0
# DEEP_LEARNING_MODEL_NAME = "a-ware/longformer-QA"                                       # Not working for 4.4.0.dev0
# DEEP_LEARNING_MODEL_NAME = "valhalla/longformer-base-4096-finetuned-squadv1"            # Not working for 4.4.0.dev0
# DEEP_LEARNING_MODEL_NAME = "allenai/longformer-base-4096"                               # Not working for 4.4.0.dev0
# DEEP_LEARNING_MODEL_NAME = "deepset/roberta-base-squad2"                                # Working for 4.4.0.dev0
# DEEP_LEARNING_MODEL_NAME = "mrm8488/bert-base-portuguese-cased-finetuned-squad-v1-pt"   # Working for 4.4.0.dev0

tokenizer = transformers.AutoTokenizer.from_pretrained(DEEP_LEARNING_MODEL_NAME)
```

## Reference Colab notebook:
https://colab.research.google.com/drive/1v10E77og3-7B2_aFfYhrHvzBZzRlo7wo#scrollTo=2zHj2lMsFuv3

## Further Information:
- This issue started appearing today. It **was working fine till yesterday.** 
- This **issue is only with 4.4.0** dev version. This issue **does not** occur for pip install transformers (which is currently on version **4.3.3**)
- The issue is only while loading tokenizers, not models
- The issue is only while loading longformers (any longformer model). Other models' tokenizers are loaded correctly (for example 'deepset/roberta-base-squad2' tokenizer can be loaded correctly)

Comments:
Indeed, I can reproduce and traced it back to https://github.com/huggingface/transformers/issues/10624. Investigating!

Found the issue, opening a PR shortly.

It should now be fixed on `master`. Thanks a lot for using the `master` branch and letting us know of the issue!
DistilBertTokenizerFast ignores "do_lower_case=False" parameter
Author: PierceEigirthon
Pull request: no

Hi, hope all is well :)

It looks like DistilBertTokenizerFast doesn't take do_lower_case into account.

```
from transformers import DistilBertTokenizerFast, DistilBertTokenizer

PRE_TRAINED_MODEL_NAME = "distilbert-base-uncased" 
tokenizer_f = DistilBertTokenizerFast.from_pretrained(PRE_TRAINED_MODEL_NAME, do_lower_case=False)
tokenizer_s = DistilBertTokenizer.from_pretrained(PRE_TRAINED_MODEL_NAME, do_lower_case=False)   

sample = "Hello, world. How are you?"

tokens_f = tokenizer_f.tokenize(sample)
tokens_s = tokenizer_s.tokenize(sample)

print(tokens_f)
print(tokens_s)
```

output:
```
['hello', ',', 'world', '.', 'how', 'are', 'you', '?']
['[UNK]', ',', 'world', '.', '[UNK]', 'are', 'you', '?']
```

expected:
```
['[UNK]', ',', 'world', '.', '[UNK]', 'are', 'you', '?']
['[UNK]', ',', 'world', '.', '[UNK]', 'are', 'you', '?']
```


packages:
```
argon2-cffi==20.1.0
async-generator==1.10
attrs==20.3.0
backcall==0.2.0
bleach==3.3.0
certifi==2020.12.5
cffi==1.14.5
chardet==4.0.0
click==7.1.2
decorator==4.4.2
defusedxml==0.7.1
entrypoints==0.3
filelock==3.0.12
idna==2.10
ipykernel==5.5.0
ipython==7.21.0
ipython-genutils==0.2.0
ipywidgets==7.6.3
jedi==0.18.0
Jinja2==2.11.3
joblib==1.0.1
jsonschema==3.2.0
jupyter-client==6.1.11
jupyter-core==4.7.1
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.0
MarkupSafe==1.1.1
mistune==0.8.4
nbclient==0.5.3
nbconvert==6.0.7
nbformat==5.1.2
nest-asyncio==1.5.1
notebook==6.2.0
numpy==1.20.1
packaging==20.9
pandocfilters==1.4.3
parso==0.8.1
pexpect==4.8.0
pickleshare==0.7.5
prometheus-client==0.9.0
prompt-toolkit==3.0.16
ptyprocess==0.7.0
pycparser==2.20
Pygments==2.8.1
pyparsing==2.4.7
pyrsistent==0.17.3
python-dateutil==2.8.1
pyzmq==22.0.3
regex==2020.11.13
requests==2.25.1
sacremoses==0.0.43
Send2Trash==1.5.0
six==1.15.0
terminado==0.9.2
testpath==0.4.4
tokenizers==0.10.1
torch==1.8.0+cu111
tornado==6.1
tqdm==4.59.0
traitlets==5.0.5
transformers==4.3.3
typing-extensions==3.7.4.3
urllib3==1.26.3
wcwidth==0.2.5
webencodings==0.5.1
widgetsnbextension==3.5.1
```

Python version:
`Python 3.8.6`

System:
PopOS 20, happy to provide more info on system specs such as hardware if needed

Comments:
Hi @PierceEigirthon! Thanks for submitting, we're aware of this bug, it's related to #10390 and on my backlog

I'll close this one as duplicate, you'll be able to follow progress on #10390 ;)
Ensure metric results are JSON-serializable
Author: sgugger
Pull request: yes

# What does this PR do?

Metrics returned from numpy (with an `np.mean()` for instance) are not real Python floats but `np.float32` (or other type) objects that are not serializable. This causes problems when the metrics are saved in JSON format in the `Trainer`, for instance when using `load_best_model_at_end`. This PR fixes that by recursively applying a `.item()` on the metrics dictionary.

Fixes #10299

Comments:
[XLSR-Wav2Vec2] Add multi-lingual Wav2Vec2 models
Author: patrickvonplaten
Pull request: yes

The model is identical to Wav2Vec2, but comes with a new paper and new checkpoints. 

So this PR only adds a new doc page and a tiny change in the conversion script.

Check out the new models here: https://huggingface.co/models?search=wav2vec2-large-xlsr

Comments:
Remove special treatment for custom vocab files
Author: sgugger
Pull request: yes

# What does this PR do?

This PR follows up from #10624 and removes the ability to specify a custom vocab file that doesn't lie in the model repo for a tokenizer (which prevents using the versioning system for those tokenizers).

It also cleans up a tiny bit the `from_pretrained` method, mainly:
- use f-strings instead of `.format()`
- remove double try since you can have several except in cascade
- use a FutureWarning for a deprecation warning that was just sent to the logs and set an end date

Comments:
from_pretrained() - some model weights not initialized message
Author: jsrozner
Pull request: no

## Environment info
- `transformers` version: 4.0.1
- Platform: Linux-4.15.0-132-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1+cu110 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: no

Note also: cookiecutter dependency is not included in pip install transformers so transformers-cli env initially fails

### Who can help
(T5) @patrickvonplaten, @patil-suraj, @sshleifer 

When using `T5ForConditionalGeneration.from_pretrained('t5-base')`, I get the following warning at load:
```
Some weights of the model checkpoint at t5-large were not used when initializing T5ForConditionalGeneration: ['decoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight']
- This IS expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining m
odel).                                                                                                                 
- This IS NOT expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification mode
l). 
```

If I load from a checkpoint that I create (i.e. local file), I get the same message.

But I think that all weights are, in fact, identical: 
- evaluation code on the model I finetune before saving AND
- evaluation code on the model I finetune, save, and then reload 
are identical.

This suggests that *all* weights are identical, since performance is identical. This contradicts the warning message.

Questions:
1) Are some weights actually not being loaded? If so, how could I observe identical behavior on metrics. Or is this warning wrong?
2) If this warning is correct, how can I force the model to fully load the model exactly as I saved it.
3) Is there any other difference (randomly initialized head, randomly initialized weights) between the t5 that is pretrained and the T5ForConditionalGeneration?


Comments:
Duplicate of https://github.com/huggingface/transformers/issues/8933 . This wrong waring should have been fixed in newer versions, see: https://github.com/huggingface/transformers/blob/63c295ac05962b03701bdda87a90595b5f864075/src/transformers/models/t5/modeling_t5.py#L1188

Great! So *all* weights are in fact loaded from pretrained, or when loading from a checkpoint, then, right?
How to use deepspeed finetune RAG?
Author: qixintechnology
Pull request: no

Hi Stas!@stas00 Thanks for your great work! I have a question that is it possible to use deepspeed finetune RAG (finetune_rag.py)? Thanks!

Comments:
Thank you for your kind words, @qixintechnology 

I haven't tried it with RAG, but I don't see any reason why it shouldn't work - if you encounter any problems please open a specific issue with details so that we could reproduce it.
MarianMT - tokenizer.supported_language_codes -> 'NoneType' object has no attribute 'supported_language_codes'
Author: gagy3798
Pull request: no

## Environment info
- colab.research.google.com

### Who can help
@patrickvonplaten

Models:
- MarianMT

Examples:
https://huggingface.co/transformers/model_doc/marian.html

## Information
I'm trying to run example code in colab but it fails

`from transformers import MarianMTModel, MarianTokenizer
src_text = [
    '>>fra<< this is a sentence in english that we want to translate to french',
    '>>por<< This should go to portuguese',
    '>>esp<< And this to Spanish'
]

model_name = 'Helsinki-NLP/opus-mt-en-roa'
tokenizer = MarianTokenizer.from_pretrained(model_name)
print(tokenizer.supported_language_codes)
model = MarianMTModel.from_pretrained(model_name)
translated = model.generate(**tokenizer.prepare_seq2seq_batch(src_text, return_tensors="pt"))
tgt_text = [tokenizer.decode(t, skip_special_tokens=True) for t in translated]`

---> 10 print(tokenizer.supported_language_codes)
AttributeError: 'NoneType' object has no attribute 'supported_language_codes'

Could you please provide working translation sample.


Comments:
Hi @gagy3798 

I couldn't reproduce the issue on master and with the latest pypi version as well. What is your transformers version? 
(please always make sure to post the env info when opening an issue)

Hi @patil-suraj 

please try it on this colab https://colab.research.google.com/drive/1z9UtSETxVrDhYnH1eN9lyMv2g-YTFNrz?usp=sharing

transformers 4.3.3



probably `sentencepiece` is not installed. Please install `sentencepiece` and restart the colab. That should resolve the issue.

Ok, thank you.
Onnx fix test
Author: mfuntowicz
Pull request: yes

GPT2 `past_keys_values` format seems to have changed since last time I checked, now exporting for each layer tuple with 2 elements.

PyTorch's ONNX exporter doesn't seem to handle this format, so it was crashing with an error.

The PR assumes we don't currently support exporting `past_keys_values` for GPT2 and then disable the return of such values when constructing the model.

In order to support this behavior, `pipeline()` now ha a `model_kwargs: Dict[str, Any]` parameter which forwards the dict of parameters to model's `from_pretrained(..., **model_kwargs)`.  

Comments:
Merging now to rebase the slow tests and re-run them.
Fix integration slow tests
Author: sgugger
Pull request: yes

# What does this PR do?

This PR fixes the following slow tests which are failing because of the change of beahvior in the `Embeddings` layer in PyTorh 1.8. This is done by adding an attention mask to ignore the padding token and checking a slice that does not contain the padding hidden states.
```
tests/test_modeling_albert.py::AlbertModelIntegrationTest::test_inference_no_head_absolute_embedding
tests/test_modeling_bert.py::BertModelIntegrationTest::test_inference_no_head_absolute_embedding
tests/test_modeling_bert.py::BertModelIntegrationTest::test_inference_no_head_relative_embedding_key
tests/test_modeling_bert.py::BertModelIntegrationTest::test_inference_no_head_relative_embedding_key_query
tests/test_modeling_convbert.py::ConvBertModelIntegrationTest::test_inference_masked_lm
tests/test_modeling_deberta.py::DebertaModelIntegrationTest::test_inference_no_head
tests/test_modeling_deberta_v2.py::DebertaV2ModelIntegrationTest::test_inference_no_head
tests/test_modeling_distilbert.py::DistilBertModelIntergrationTest::test_inference_no_head_absolute_embedding
tests/test_modeling_electra.py::ElectraModelIntegrationTest::test_inference_no_head_absolute_embedding
tests/test_modeling_squeezebert.py::SqueezeBertModelIntegrationTest::test_inference_classification_head
```

It also fixes
```
tests/test_modeling_mbart.py::MBartEnroIntegrationTest::test_enro_generate_batch
```
that was failing since the change in `prepare_seq2seq_batch`. For some reason a word is different but it was consistent in PyTorch 1.7/PyTorch 1.8 so I changed the desired target. @patil-suraj if you want to take a closer look, I'll leave it to you.

Comments:
Specify minimum version for sacrebleu
Author: LysandreJik
Pull request: yes

The `_tests_requirements.txt` require an install of sacrebleu without any version specified. However, some `sacrebleu` versions don't have the same API. I've had problems with version `1.2.10`, and @lhoestq confirmed the issue is not present in `1.4.12`.

The error was the following:

```
    def _compute(
        self,
        predictions,
        references,
        smooth_method="exp",
        smooth_value=None,
        force=False,
        lowercase=False,
        tokenize=scb.DEFAULT_TOKENIZER,
        use_effective_order=False,
    ):
        references_per_prediction = len(references[0])
        if any(len(refs) != references_per_prediction for refs in references):
            raise ValueError("Sacrebleu requires the same number of references for each prediction")
        transformed_references = [[refs[i] for refs in references] for i in range(references_per_prediction)]
>       output = scb.corpus_bleu(
            sys_stream=predictions,
            ref_streams=transformed_references,
            smooth_method=smooth_method,
            smooth_value=smooth_value,
            force=force,
            lowercase=lowercase,
            tokenize=tokenize,
            use_effective_order=use_effective_order,
        )
E       TypeError: corpus_bleu() got an unexpected keyword argument 'smooth_method'
/mnt/cache/modules/datasets_modules/metrics/sacrebleu/b390045b3d1dd4abf6a95c4a2a11ee3bcc2b7620b076204d0ddc353fa649fd86/sacrebleu.py:114: TypeError
```

Full stack trace:

```
E             File "/__w/transformers/transformers/src/transformers/trainer_seq2seq.py", line 74, in evaluate
E               return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
E             File "/__w/transformers/transformers/src/transformers/trainer.py", line 1650, in evaluate
E               output = self.prediction_loop(
E             File "/__w/transformers/transformers/src/transformers/trainer.py", line 1823, in prediction_loop
E               metrics = self.compute_metrics(EvalPrediction(predictions=preds, label_ids=label_ids))
E             File "/__w/transformers/transformers/examples/seq2seq/run_seq2seq.py", line 563, in compute_metrics
E               result = metric.compute(predictions=decoded_preds, references=decoded_labels)
E             File "/opt/conda/lib/python3.8/site-packages/datasets/metric.py", line 403, in compute
E               output = self._compute(predictions=predictions, references=references, **kwargs)
E             File "/mnt/cache/modules/datasets_modules/metrics/sacrebleu/b390045b3d1dd4abf6a95c4a2a11ee3bcc2b7620b076204d0ddc353fa649fd86/sacrebleu.py", line 114, in _compute
E               output = scb.corpus_bleu(
```

I'm unsure about the minimum version required here, I just know that 1.2.10 doesn't work. Please advise if you think a better minimum version would be better.

Comments:
@patil-suraj, you must have meant `1.4.12`