noamaanMulla-03/transformers-issues huggingface dataset:
Support for torch 1.9.0
Author: LysandreJik
Pull request: yes

This PR adds support for torch 1.9.0. It upgrades the CPU CI to use torch 1.9.0, and the GPU CI to use PyTorch's 1.9.0 docker image to run tests.

As discussed with @michaelbenayoun, this puts a hard requirement on having a specific torch version for torch fx to be run. The idea is that:
- The torch fx support in `transformers` is currently experimental, and will be updated *without* backwards compatibility requirements
- To that end, it should always support the latest PyTorch version and not the earlier ones.
- However PyTorch 1.9.0 will not be supported due to https://github.com/pytorch/pytorch/pull/59569
- To that end, we setup a specific version requirement on `torch` in order to offer torch FX support.

Running on torch 1.8.0 and torch 1.8.1, as well as the various torch 1.8.1-cu111 and other 1.8.x versions works correctly.
Running on torch < 1.8 or torch > 1.8 returns:

```
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/opt/pycharm-professional/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/opt/pycharm-professional/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/lysandre/.config/JetBrains/PyCharm2021.1/scratches/torchfx.py", line 6, in <module>
    traced_model = symbolic_trace(
  File "/home/lysandre/transformers/src/transformers/modeling_fx_utils.py", line 374, in symbolic_trace
    tracer = HFTracer(batch_size=batch_size, sequence_length=sequence_length, num_choices=num_choices)
  File "/home/lysandre/transformers/src/transformers/modeling_fx_utils.py", line 152, in __init__
    raise ImportError(
ImportError: Found an incompatible version of torch. Found version 1.9.0, but only version 1.8 is supported.
```

Comments:
Shouldn't the fx tests be skipped correspondingly? I see the CI logs show that they all passed with 1.9.0 - how is that possible?

The `is_torch_fx_available` returns `False` as the versions aren't compatible. The tests for torch.fx require `is_torch_fx_available` to be `True` in order to run!

Yes, switching back to > 1.9.0 once the issue is fixed works for me.

OK, but the tests were reported as passed and not skipped. So another todo for down the road is to add a skip rule, so that we don't get a misleading report and have a skipped test appearing as passed. Don't have to do it now.

They should have a decorator for that, rather than the in-test check. Would be better reported as skipped, indeed!
fix pt-1.9.0 `add_` deprecation
Author: stas00
Pull request: yes

This PR fixes a new pt-1.9.0 `add_` deprecation in several places. 

The deprecation warnings:
```
UserWarning: This overload of add_ is deprecated:
        add_(Number alpha, Tensor other)
Consider using one of the following signatures instead:
        add_(Tensor other, *, Number alpha) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:1025.)
  exp_avg_sq.mul_(beta2t).add_(1.0 - beta2t, update)
```

The new API is at https://pytorch.org/docs/stable/generated/torch.Tensor.add_.html

## Backward compatibility alert

I tracked this API down to need pt-1.5.0 or higher

Requesting an easier way to do this kind of process: https://github.com/pytorch/pytorch/issues/60149

I still have no idea which minimal pytorch version `transformers` is meant to support. Merging this PR will push it at least to `torch>=1.5.0`. Last I [checked](https://github.com/huggingface/transformers/pull/7985) some 8 months ago we barely supported `torch>=1.4.0`.

If you're OK with `torch>=1.5.0` then we should revive and update [this](https://github.com/huggingface/transformers/pull/7985) or make a new PR or fix it here.

## Readability

Unfortunately since we have to use the named arg now, the autoformatter makes the code less readable, by forcing whitespace in the expression.

I wrote these as:
```
exp_avg.mul_(group["beta1"]).add_(update, alpha=1-group["beta1"])
```
to make it clear, it's an expression, but it made it into:
```
exp_avg.mul_(group["beta1"]).add_(update, alpha=1 - group["beta1"])
```
now it looks like alpha is 1. grrrr. perhaps `()` are needed for improved readability. i.e.:
```
exp_avg.mul_(group["beta1"]).add_(update, alpha=(1 - group["beta1"]))
```
@sgugger, @LysandreJik 

Comments:
Can we had an import error in AdaFactor to error if the version is les than 1.5 then? It seems the code is only there.

Sure, I'm just not sure where we are at `transformers`-wise with minimal pt version, so it might be simpler to require pt-1.5+, but the suggestion you made works too for now.

Would it help to add `()` for `alpha` as described in the last section of OP? 

Yes, I missed that part. Adding parenthesis is fine!

@sgugger, it's in AdamW too - it's just whoever coded it hasn't checked back-compat (they didn't know), i.e. search for `add_` - so I think we either need a wrapper or cut off at pt-1.5.0 project-wise.

Found this now, as I was adding `()` for clarity. See the new diff.

as discussed on slack for now adding:
```
require_version("torch>=1.5.0")  # add_ with alpha
```
for AdamW and Adafactor.
AutoTokenizer: infer the class from the tokenizer config if possible
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds the functionality to load a tokenizer with `AutoTokenizer.from_pretrained` after saving it locally (without saving the model config in the same folder).

To do this, the proper tokenizer class is saved in `tokenizer_config.json` and the `AutoTokenizer.from_pretrained` method will first look in this file before defaulting to the model config (like before).


Comments:
update desc for map in all examples
Author: bhavitvyamalik
Pull request: yes

# What does this PR do?

Closes #11797. I've added the remaining the `desc` for `summarization`, `token-classification,` `translation`, `language-modeling ` and updated `requirements.txt` as well. I wasn't sure what to add in `desc` at some places so I've added `# not sure if it's right` comment there. Please let me know if that description looks good or should I replace it with something else?

## Who can review?
@stas00 @sgugger 

Comments:
Depreciate pythonic Mish and support PyTorch 1.9 version of Mish
Author: digantamisra98
Pull request: yes

This PR removes the old pure pythonic version of [Mish](https://arxiv.org/abs/1908.08681) and now enables support for the [PyTorch 1.9 Mish version](https://pytorch.org/docs/stable/generated/torch.nn.Mish.html#torch.nn.Mish). It also removes isolated references of the function where it is not used. 


Comments:
Done
Wrong output used by RobertaForSequenceClassification classification head
Author: slowwavesleep
Pull request: no

Hi,

According to the [documentation](https://huggingface.co/transformers/model_doc/roberta.html#transformers.RobertaForSequenceClassification) classification head should work `on top of the pooled output`, which makes sense, considering the fact that RoBERTa, unlike BERT, wasn't trained on Next Sentence Prediction task, so the `
` token equivalent to `CLS` is not as useful. However, if you look at the at the code [here](https://github.com/huggingface/transformers/blob/master/src/transformers/models/roberta/modeling_roberta.py#L1166) and [here](https://github.com/huggingface/transformers/blob/master/src/transformers/models/roberta/modeling_roberta.py#L1394) you'll see that it doesn't appear to actually use the pooler output for Sequence Classification, as one would expect. Meanwhile, RobertaForMultipleChoice does use it.

It's not clear to me whether this is intended or not, however RoBERTa using a representation of `
` for classification may perform considerably _worse_ than a regular BERT on some tasks.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi,

I'd still like to get a comment on whether this is the intended behavior. If it is, then why it is done this way?

Hello! Sorry for getting back to this so late.

When porting models over to the `transformers` library, we aim to keep them identical to their original implementation. The original RoBERTa implementation in fairseq uses the same classification head, hence why it was ported like this: https://github.com/pytorch/fairseq/blob/c2e8904b6072d8eddab362ac50b324e374b5951d/fairseq/models/roberta/model.py#L382

I recommend opening an issue over at fairseq if you have questions relative to how they designed their architecture. Thank you!

Ah, I see. Didn't realize it's done the same way in the original implementation. Thank you for your response!
RobertaForMaskedLM.from_pretrained throwing some weights not initialized error when loading same model type
Author: dhuruvasaditya
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.7.0
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu101 (False)
- Tensorflow version (GPU?): 2.5.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

## Information

I am using a pretrained RobertaForMaskedLM . When I try to load the model I get the following error:

Some weights of RobertaForMaskedLM were not initialized from the model checkpoint at mldmm/GlassBERTa and are newly initialized: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.weight', 'lm_head.decoder.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

The config of the model is as follows:
```
{
  "architectures": [
    "RobertaForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dim": 96,
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "roberta",
  "n_heads": 3,
  "n_layers": 3,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "position_embedding_type": "absolute",
  "transformers_version": "4.6.1",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 541
}
```
## To reproduce

Steps to reproduce the behavior:
```
from transformers import AutoModelForMaskedLM,AutoTokenizer
tok = AutoTokenizer.from_pretrained('mldmm/GlassBERTa')
mod = AutoModelForMaskedLM.from_pretrained('mldmm/GlassBERTa')
```
or
```
from transformers import RobertaForMaskedLM,AutoTokenizer
tok = AutoTokenizer.from_pretrained('mldmm/GlassBERTa')
mod = RobertaForMaskedLM.from_pretrained('mldmm/GlassBERTa')
```

Same goes for many other pre-trained models hosted ('beomi/kcbert-base','hfl/chinese-bert-wwm-ext')
## Expected behavior
Since I'm loading for the same architecture I am expecting a clean import without any errors, might lead to random output due to those newly initialized layers and can mess the output in fill-mask pipeline. 

Comments:
[doc] t5 incomplete example
Author: stas00
Pull request: no

https://huggingface.co/transformers/model_doc/t5.html#training has:
```
input_ids = tokenizer('translate English to German: The house is wonderful.', return_tensors='pt').input_ids
labels = tokenizer('Das Haus ist wunderbar.', return_tensors='pt').input_ids
# the forward function automatically creates the correct decoder_input_ids
loss = model(input_ids=input_ids, labels=labels).loss
```
which is broken unless the right model type is used. And when none if specified it typically presumes `AutoModel`, which gives: `TypeError: forward() got an unexpected keyword argument 'labels'`

So probably need to use an explicit `T5ForConditionalGeneration` and then it works.

@patrickvonplaten, @patil-suraj


Comments:
A possible approach proposed here: https://github.com/huggingface/transformers/pull/12239
Merge strings that are being concatenated in the same line
Author: orestisfl
Pull request: yes

After fa84ae26d6, some strings are needlessly concatenated even though
they are not wrapped to multiple lines.

This change makes the code slightly less confusing and more grepable.

# What does this PR do?

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
license missing for xlm-roberta-large, and bert-base-spanish-wwm models
Author: beomseok-lee
Pull request: no

Hi HuggingFace team,
I am gratefully using many of huggingface models, but I have found 'xlm-roberta-large' model is missing its license policy.
also 'dccuchile/bert-base-spanish-wwm-uncased' and 'dccuchile/bert-base-spanish-wwm-cased'


[xlm-roberta-large](https://huggingface.co/xlm-roberta-large)
[dccuchile/bert-base-spanish-wwm-uncased](https://huggingface.co/dccuchile/bert-base-spanish-wwm-uncased)
[dccuchile/bert-base-spanish-wwm-cased](https://huggingface.co/dccuchile/bert-base-spanish-wwm-cased)

Could you please add license information using readme or model card or this thread?

Thanks !

Comments:
xlm-roberta should be licensed under MIT like all models available through [fairseq](https://github.com/pytorch/fairseq#license). @aconneau could you please confirm?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
RuntimeError: Error(s) in loading state_dict for Wav2Vec2ForCTC
Author: kamil-bentounes
Pull request: no

## Environment info

- `transformers` version: 4.5.0
- Platform: Ubuntu 18.04 
- Python version: 3.7.4

### Who can help

- @patrickvonplaten 

## Information

Model Im using is Wav2vec 2.0.
The problem arises when loading the pretrained/fintuned model:
```
RuntimeError: Error(s) in loading state_dict for Wav2Vec2ForCTC:
	size mismatch for lm_head.weight: copying a param with shape torch.Size([123, 768]) from checkpoint, the shape in current model is torch.Size([132, 768]).
	size mismatch for lm_head.bias: copying a param with shape torch.Size([123]) from checkpoint, the shape in current model is torch.Size([132]).
```

The tasks I am working on is:
* Finetuning Wav2vec 2.0 on my own data from the finetuned xlsr french model.  

## To reproduce

I followed the steps mentioned [here](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2). I started to finetune the base model on my own dataset. Then, I tried to finetune it from the finetuned french xlsr model. 

With fairseq, to fix this problem, we must add the `--restore` argument to say that this model is a finetuned model from a X-architecture.

Any idea about how can we do it using Transformers ?  

Here is the whole error: 

```
RuntimeError: Error(s) in loading state_dict for Wav2Vec2ForCTC:
	size mismatch for lm_head.weight: copying a param with shape torch.Size([123, 768]) from checkpoint, the shape in current model is torch.Size([132, 768]).
	size mismatch for lm_head.bias: copying a param with shape torch.Size([123]) from checkpoint, the shape in current model is torch.Size([132]).
```

Comments:
Hey @Kamilbentounes,

It looks like your `config.vocab_size` does not match the `config.vocab_size` of the fine-tuned French Wav2Vec2 model. It looks like you want to initialize the model with 132 characters, but the original vocab size is 123. Could you try to align the vocab size to the one of the fine-tuned model? :-) 

If this doesn't fix the problem, please ping me here again

Hey @patrickvonplaten 

Thanks for your quick reply. Yes, before writing this issue I tried it but I had an error during training phase :/ 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Added BERT pretraining example running on Graphcore IPUs to research projects
Author: jimypbr
Pull request: yes

# What does this PR do?

Adds BERT pretraining example running on Graphcore IPUs to research projects


## Before submitting

- This was discussed in the HuggingFace/Graphcore meeting


## Who can review?

@LysandreJik


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Predict single sentence for Glue Tasks
Author: Nickil21
Pull request: no

I have trained a custom binary classifier using `run_glue.py` and have the `pytorch_model.bin` file saved to a directory. Is there a way to predict for a given sentence and extract its label?

I know `trainer.predict(test_dataset)` does it. But I am having some trouble converting the string to the format that it takes.


Comments:
Here is how I managed to do it. I have considered a `pandas` dataframe, but you can easily extend it to predict individual sentence too.

```
import pandas as pd 
import numpy as np
from datasets import Dataset, load_dataset
from scipy.special import softmax
from transformers import Trainer
from transformers import BertTokenizer, BertForSequenceClassification

tokenizer = BertTokenizer.from_pretrained(<model_name_or_path>)
model = BertForSequenceClassification.from_pretrained(<model_name_or_path>)

def preprocess_function(examples):
    # Tokenize the texts
    result = tokenizer(examples['sentence'], padding=False, max_length=None, truncation=True, verbose=False)
    return result

def predict(dataframe):
    eval_dataset = Dataset.from_pandas(dataframe)
    eval_dataset = eval_dataset.map(preprocess_function, batched=False, load_from_cache_file=True)
    # Initialize our Trainer
    trainer = Trainer(model=model, tokenizer=tokenizer)
    predictions = trainer.predict(test_dataset=eval_dataset).predictions
    # Adding a softmax layer to get probabilities. If you want class labels instead -  np.argmax(predictions, axis=1)
    predictions = np.array([softmax(element) for element in predictions])[:, 1]
    return predictions
```

@Nickil21 
I used the torch model in the trainer. It's much faster than using pandas and creating a dataset.

`import torch
def test_2(trainer, sentence1, sentence2):
    id_tolabel = {0:'negative', 1: 'positive'}
    model = trainer.model.eval()
    tokenized = tokenizer(sentence1, sentence2, return_tensors='pt').to(model.device)
    with torch.no_grad():
        label = torch.argmax(trainer.model.forward(**tokenized).logits, dim=1)[0].cpu().item()
    return id_tolabel[label]
print(test_2(trainer, 'it is not possible', 'this is impossible'))`
Add code to pretrain T5 model from scratch
Author: LiweiPeng
Pull request: no

# 🚀 Feature request

The T5 model can significantly improve NLP task accuracies. However, the existing pretrained models are all in English. I'd like to pretrain T5 model on different language datasets from scratch.

Can you add code on pretraining T5 model? Thanks.






Comments:
@patrickvonplaten Can we pre-train T5 from scratch on any task? I want to use it for Question Answering.

Many notebooks for T5 are now added to the community notebooks :-) 

@patrickvonplaten can you share the notebook which show T5 pre-training if it is available ?

@patrickvonplaten as of today, I didn't find any notebook that is related to T5 *pretraining* in the [community notebooks collection ](https://huggingface.co/transformers/master/community.html#community-notebooks). Could you elaborate more on where there is a codebase to do the pretraining? Thanks!

> @patrickvonplaten as of today, I didn't find any notebook that is related to T5 pretraining in the community notebooks collection . Could you elaborate more on where there is a codebase to do the pretraining? Thanks!

Yes I agree there is no guide for pretraining
Request for feature for setting batch size in pipeline when inference
Author: yanan1116
Pull request: no

```
from transformers import pipeline

from transformers import AutoModelWithLMHead, AutoTokenizer
model = AutoModelWithLMHead.from_pretrained("Helsinki-NLP/opus-mt-en-fr")
tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-fr")
nlp = pipeline("translation_en_to_fr", model=model, tokenizer=tokenizer, device=0)

nlp(ds.df_train.sample(32)['content'].tolist(), max_length=300)
```
I am suing pipeline instance for inference on trunks of sentences.
When the size of the trunk data is small, like the 32, it is ok to fit into the GPU memory.

However, when increasing the size of this input,  memory error comes out:

> RuntimeError: CUDA out of memory.

Is there any way to set the batch size inside the `nlp()` to automatically to fit into the GPU to make trunks of inferences?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.