noamaanMulla-03/transformers-issues huggingface dataset:
fix `tokenizer_class_from_name` for models with `-` in the name
Author: stas00
Pull request: yes

https://github.com/huggingface/transformers/pull/13023 breaks for some models with `-` in its name. e.g. `xlm-roberta`,

For example:

```

Traceback (most recent call last):
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/pytorch/language-modeling/run_mlm.py", line 550, in <module>
    main()
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/pytorch/language-modeling/run_mlm.py", line 337, in main
    tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, **tokenizer_kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/models/auto/tokenization_auto.py", line 432, in from_pretrained
    tokenizer_class = tokenizer_class_from_name(tokenizer_class_candidate)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/models/auto/tokenization_auto.py", line 226, in tokenizer_class_from_name
    module = importlib.import_module(f".{module_name}", "transformers.models")
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'transformers.models.xlm-roberta'
```

as you can see it tries to import "`transformers.models.xlm-roberta`", to reproduce:

```
RUN_SLOW=1 pytest tests/deepspeed -k clm_xlm_roberta
```

```
# module_name, tokenizers debug print:
xlm-roberta ('XLMRobertaTokenizer', 'XLMRobertaTokenizerFast')
```

This PR fixes it:
```
 module = importlib.import_module(f".{module_name.replace('-', '_')}", "transformers.models")
```

Oddly enough I don't get this problem if I run `xlm-roberta-base`, so this is an edge case. As the core models seems to not trigger the problem. Not sure why.

In the deepspeed test suite the 2 failing tests were:

```
RUN_SLOW=1 pytest tests/deepspeed -k clm_xlm_roberta
```

Now it has a core test - thanks @LysandreJik 

@LysandreJik also pushed a fix for model names that mismatch model files which is the case with `openai-gpt`

@LysandreJik, @sgugger 

Comments:
Should we turn your code snippet into a test? It's almost complete - just needs an assert.

Yes, that would be great!

OK, done. 

It's not a perfect test as it'll fail on the first invalid entry rather than test them all, but it's probably good enough in this situation.

Thank you for doing the heavy lifting for adding this test, @LysandreJik 

Hi there! Thanks a lot for fixing this while I was away. There is a `model_type_to_module_name` function defined in `configuration_auto` that already does all what you added. Will make a PR to switch to that. The tests should make sure it doesn't break anything.
Movement Pruning does not achieve expected results
Author: iamweiweishi
Pull request: no

Transformer version: master or 4.6.1
tensorflow-gpu == 2.3.1
pytorch == 1.7.1
flax == 0.3.4

I tried to reproduce the results based on the script given in examples/research_projects/movement-pruning. 
The experimental results I got are not as good as the paper shows:

06/01/2021 22:55:40 - INFO - __main__ -   ***** Running evaluation  *****
06/01/2021 22:55:40 - INFO - __main__ -     Num examples = 10833
06/01/2021 22:55:40 - INFO - __main__ -     Batch size = 32
Evaluating: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 339/339 [01:42<00:00,  3.32it/s]
06/01/2021 22:57:22 - INFO - __main__ -     Evaluation done in total 102.215492 secs (0.009436 sec per example)
06/01/2021 22:58:27 - INFO - __main__ -   Results: {'exact': 0.33112582781456956, 'f1': 7.122522334334856, 'total': 10570, 'HasAns_exact': 0.33112582781456956, 'HasAns_f1': 7.122522334334856, 'HasAns_total': 10570, 'best_exact': 0.33112582781456956, 'best_exact_thresh': 0.0, 'best_f1': 7.122522334334856, 'best_f1_thresh': 0.0}

I almost followed each step as the 'README.md' says. The only difference I modified is 'BertLayerNorm = torch.nn.LayerNorm' according to [this](https://github.com/huggingface/transformers/issues/10892).

I also noticed the requirements in the [setup ](https://github.com/huggingface/transformers/tree/master/examples/research_projects/movement-pruning#setup) step. 

I wonder what should I do to reproduce the same results from the master branch.

Thank you. @VictorSanh 

Comments:
that's very low of a score... i imagine it''s on squad.
could you give me more details about the command you are running?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I have the same problem. The accuracy is quite low when performing movement pruning. Have you solved it?
Examples: label mapping for text classication tasks are not written into configuration
Author: stefan-it
Pull request: no

Hi,

I've shortly discussed this with @patrickvonplaten , and we came to the conclusion that the following scenario is a bug:

When using the text classification for a GLUE task, no label mapping will be written into the configuration of the final fine-tuned model.

This leads to an unesthetic "label_0" on the model hub, as it can be seen here:

![Bildschirmfoto_2021-08-27_12-17-09](https://user-images.githubusercontent.com/20651387/131112292-ab2fc8f8-2fad-42cf-b436-0a0a5e0a4475.png)

One has to manually extend the `config.json`:

```json
"id2label": {
    "0": "NEGATIVE",
    "1": "POSITIVE"
  },
  "label2id": {
    "NEGATIVE": 0,
    "POSITIVE": 1
  }
```

to get the following output:

![Bildschirmfoto_2021-08-27_12-17-29](https://user-images.githubusercontent.com/20651387/131112468-bff93d0f-4f3f-428d-ac3b-696ae6e08543.png)

The text classification examples should be extended, so that those label mappings are automatically added to the configuration file.

Comments:
cc @sgugger @Rocketknight1 

Should be fixed by the PR above, I forgot the `label_to_name` dict was left to None for GLUE tasks.
How could I convert output tensor from transformer to text generation?
Author: shp776
Pull request: no

# 🚀 Feature request

https://github.com/onnx/models/blob/master/text/machine_comprehension/gpt-2/dependencies/GPT2-export.py

I succeeded in extracting the output tensor value for the example input text using the above example. The link above imports  your hugging face transformer, so I wonder how I can text generate with the output tensor value I got.

Is there a code or a link I can refer to? (Pytorch or python code..)

The code I tried is as follows. But it didn't work.
![image](https://user-images.githubusercontent.com/42932221/126746354-2c3387aa-1eb1-4135-bd33-87a50c3be28b.png)

'ort_outputs_exmodel' above image is same as 'res' link below
https://github.com/onnx/models/blob/ad5c181f1646225f034fba1862233ecb4c262e04/text/machine_comprehension/gpt-2/dependencies/GPT2-export.py#L110

My final goal of the project is to load the onnx model using onnx runtime's C/C++ API and write the C/C++ code to generate text using output tensor values.

I'll be waiting for your reply. (looking forward to...)
Thank u very much.

## Motivation

I need advice on how to run text generation using output tensor values. 

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
run_mlm_no_trainer.py requires --model_name_or_path
Author: david-waterworth
Pull request: no

The `examples/pytorch/language-modeling/run_mlm_no_trainer.py` script has

    parser.add_argument(
        "--model_name_or_path",
        type=str,
        help="Path to pretrained model or model identifier from huggingface.co/models.",
        default=None,
        required=True,
    )

Despite there being several checks in the code implying it may be None ie

    if args.model_name_or_path:
        model = AutoModelForMaskedLM.from_pretrained(
            args.model_name_or_path,
            from_tf=bool(".ckpt" in args.model_name_or_path),
            config=config,
        )
    else:
        logger.info("Training new model from scratch")
        model = AutoModelForMaskedLM.from_config(config)

As far as I can see it's optional, falling back to training a new model from scratch - just like run_mlm.py (I commented out `required=True` without any obvious issues).


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Generate text from inputs_embeds for seq2seq models like BART or T5
Author: lamthuy
Pull request: no

# 🚀 Feature request

Generate text from inputs_embeds for seq2seq models like BART or T5

## Motivation

Many advanced researches now rely on generation of text from an embedding vector. Currently we can perform forward pass through BART or T5 by just inputing inputs_embeds but this is not possible in generate function.

At the moment I have to go around this issue by calling the encoder to get encoder_outputs, and used it as  an input to the generation function directly:
```python
 encoder_outputs = model.base_model.encoder(inputs_embeds=input_embeds, return_dict=True)
 out = model.generate(encoder_outputs=encoder_outputs)
```
Even this does not show any error message but I am not sure this is the right thing to do, any comment? Can you please support this function as did with the forward function?


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Flaky tests
Author: PhilipMay
Pull request: no

Hi,
in this PR #12794 I somehow have flaky tests.

See the two commits from here
https://github.com/huggingface/transformers/pull/12794/commits/0ea13d6cf8a6a1ca5772f6a129bea9a7c06f5571
to here
https://github.com/huggingface/transformers/pull/12794/commits/e93eb12c01f8de9df6a1130041fe315372f9378b

The 2nd commit is empty and fails at an other location.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Cannot load .pt model using Transformers
Author: 14H034160212
Pull request: no

Hi,

I want to use Transformers to load .pt model. How can I do that? I know how to load .bin using transformers, but I do not know how to load .pt model using transformers. Thanks.

    config = config_class.from_pretrained(
        args.config_name if args.config_name else args.model_name_or_path,
        num_labels=num_labels,
        finetuning_task=args.task_name,
        cache_dir=args.cache_dir if args.cache_dir else None,
    )
    tokenizer = tokenizer_class.from_pretrained(
        args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
        do_lower_case=args.do_lower_case,
        cache_dir=args.cache_dir if args.cache_dir else None,
    )
    model = model_class.from_pretrained(
        args.model_name_or_path,
        from_tf=bool(".ckpt" in args.model_name_or_path),
        config=config,
        cache_dir=args.cache_dir if args.cache_dir else None,
    )


Comments:
What is your .pt model? Where did you obtain it from?

While we don't have the complete details following is a possible solution:
You can initialize your model using transformers and simply load the weights using
`model.load_state_dict(torch.load('model.pt'))`

In case this is not what you're looking for please add further details.

Hi @Ap1075, Thanks for your reply. It is working to load the `model.pt` if I define the `model` class, but do you know if I want to load the tokenizer from the `model.pt`. How can I do that? For example, I can load the tokenizer by this way from huggingface `tokenizer = AutoTokenizer.from_pretrained(pretrained_model, do_lower_case=True)`, but I cannot do that if the `pretrained_model='model.pt'`.

Also, for this command, `model.load_state_dict(torch.load('model.pt'))`, if what is the `model` from `model.load_state_dict()`? How to define the model here? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
bert:What is the tf version corresponding to tensformers?
Author: xmcs111
Pull request: no

 I use python3.7, tf2.4.0, cuda11.1 and cudnn 8.0.4 to run bert-base-un and report an error
- albert, bert, xlm: @LysandreJik
- tensorflow: @Rocketkn


Comments:
@Rocketknight1 

Hello! Do you mind providing the error you're seeing? Thank you!

'''
I:?[35mVENTILATOR?[0m:freeze, optimize and export graph, could take a while...
2021-08-30 20:07:49.360788: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
d:\users\...\pycharmprojects\machine learning i don't study\venv37\lib\site-packages\bert_serving\server\helper.py:176: UserWarning: Tensorflow 2.4.0 is not tested!
 It may or may not work. Feel free to submit an issue at https://github.com/hanxiao/bert-as-service/issues/
  'Feel free to submit an issue at https://github.com/hanxiao/bert-as-service/issues/' % tf.__version__)
E:?[36mGRAPHOPT?[0m:fail to optimize the graph!
Traceback (most recent call last):
  File "D:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "D:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:\Users\...\PycharmProjects\machine learning I don't study\venv37\Scripts\bert-serving-start.exe\__main__.py", line 7, in <module>
  File "d:\users\...\pycharmprojects\machine learning i don't study\venv37\lib\site-packages\bert_serving\server\cli\__init__.py", line 4, in main
    with BertServer(get_run_args()) as server:
  File "d:\users\...\pycharmprojects\machine learning i don't study\venv37\lib\site-packages\bert_serving\server\__init__.py", line 71, in __init__
    self.graph_path, self.bert_config = pool.apply(optimize_graph, (self.args,))
TypeError: cannot unpack non-iterable NoneType object
'''

Hi, it seems like you're using [bert-as-service](https://github.com/hanxiao/bert-as-service) with an unsupported version of Tensorflow. That isn't a Huggingface project, so we can't really support it here, unfortunately! Try filing an issue at that repo instead!

If you're interested in learning to use the Transformers library, you can check out our [documentation](https://huggingface.co/transformers/quicktour.html), our [course](https://huggingface.co/course/chapter1) or our [example code](https://github.com/huggingface/transformers/tree/master/examples), but we can't really answer questions on any of that in GitHub issues - try the [forums](https://discuss.huggingface.co/) instead!

I'm going to close this issue, but if you believe there's an actual problem in the Transformers library, separate from `bert-as-service`, please feel free to add more info and re-open it.
Use DS callable API to allow hf_scheduler + ds_optimizer
Author: tjruwase
Pull request: yes
Labels: name: DeepSpeed

This PR:

- Used the (new) Callable api of deepspeed.initialize() to enable combining hf schedulers with deepspeed optimizers.  
- `create_scheduler` now has an optional `optimizer` arg
- Updates relevant unit test.

Blocking events: All unblocked now.

- [x] depends on deepspeed PR [1316](https://github.com/microsoft/DeepSpeed/pull/1316).
- [x] needs new deepspeed version after PR is merged and need to update the dependencies when that happens.

deepspeed: @stas00. 


Comments:
* [x] https://github.com/microsoft/DeepSpeed/pull/1316 is merged
* [x] v0.5.1 released to PyPI: https://pypi.org/project/deepspeed/0.5.1/
[Docs] Function signatures on website not correctly reflecting current code.
Author: qqaatw
Pull request: no

## Environment info

Tested v4.9.2 and master.

### Who can help

@sgugger 

## Information

In [Trainer docs](https://huggingface.co/transformers/main_classes/trainer.html#transformers.Trainer), the first argument `model` in the function signature should be 

`model: Union[transformers.modeling_utils.PreTrainedModel, torch.nn.modules.module.Module] = None` as defined in the [code](https://github.com/huggingface/transformers/blob/1fec32adc6a4840123d5ec5ff5cf419c02342b5a/src/transformers/trainer.py#L267). 

But it currently shows

`model: torch.nn.modules.module.Module = None`, which seems to be outdated, however.

I've tested building the docs on my system, and the resulting html is correct.
 

Comments:
Would you like to open a PR to correct it @qqaatw ? :-)

@patrickvonplaten I could try, but the problem seems to be related to the CI/CD since function signatures are automatically generated by Sphinx, and the problem didn't occur on the docs I built manually on my machine. (Tested ubuntu 18.04 with python 3.8 / windows 10 with python 3.8) 

Closed as the PR has been merged.
Add `--max_length` argument in seq2seq trainer.
Author: sbmaruf
Pull request: no

# 🚀 Feature request

Currently [seq2seq Trainer](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_seq2seq.py) uses  `--max_length` for prediction step. However in the trainer there is no argument `--max_length` in [here](https://huggingface.co/transformers/main_classes/trainer.html?highlight=trainingargument#transformers.Seq2SeqTrainingArguments) and [here](https://huggingface.co/transformers/_modules/transformers/training_args_seq2seq.html#Seq2SeqTrainingArguments).

During training (with `--predict_with_generate`) when the evaluate function is called, it performs [prediction step ](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_seq2seq.py#L128) with `model.config.max_length` by this [line](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_seq2seq.py#L166). Unless you call the `trainer.evaluate(eval_dataset = eval_dataset, max_length=max_target_length)` manually, in the training time it uses `model.config.max_length`.

Also without reviewing the source code, it is difficult to grasp this. 



So in the training time, for `prediction_loop`, the model performs evaluation based on [this](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_seq2seq.py#L166). It uses `self.model.config.max_length` for doing prediction. It is kind of confusing I would say. Let's look into this, 

```
>>> import transformers
>>> transformers.__version__
'4.10.0.dev0'
>>> model = transformers.AutoModel.from_pretrained("google/mt5-large")
Some weights of the model checkpoint at google/mt5-large were not used when initializing MT5Model: ['lm_head.weight']
- This IS expected if you are initializing MT5Model from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing MT5Model from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
>>> model.config.max_length
20
```

A user who is not careful about this argument would totally miss this. Personally I spent quite a few time on this. My `compute_metrics()` values at the training time on dev set was not good but at the end of training prediction on the test dataset score (using my own call `trainer.evaluate()`) was high. 


## Motivation

Adding `--max_length` in [Seq2SeqTrainer](https://github.com/huggingface/transformers/blob/master/src/transformers/training_args_seq2seq.py#L27) would help the user to be-aware of this parameter. 


@sgugger 



Comments:
This is added by the PR mentioned above.

Thanks a lot for the new feature.  Closing the issue.
[s2s examples] Replace -100 token ids with the tokenizer pad_id for compute_metrics
Author: olinguyen
Pull request: yes

# What does this PR do?

This PR is a small fix that replaces the -100 token ids with the tokenizer pad_id when decoding sequences to compute metrics as was done in [this HF blog post](https://huggingface.co/blog/warm-starting-encoder-decoder)

## When does this problem occur?

When running `examples/seq2seq/finetune_trainer.py` with padding to the `max_seq_len`, an error is thrown at the evaluation step:

```
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 1004, in _maybe_log_save_evaluate
    metrics = self.evaluate()
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_seq2seq.py", line 96, in evaluate
    return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 1442, in evaluate
    output = self.prediction_loop(
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 1601, in prediction_loop
    metrics = self.compute_metrics(EvalPrediction(predictions=preds, label_ids=label_ids))
  File "/eai/transformers/examples/seq2seq/utils.py", line 98, in translation_metrics
    pred_str, label_str = decode_pred(pred)
  File "/eai/transformers/examples/seq2seq/utils.py", line 85, in decode_pred
    label_str = tokenizer.batch_decode(pred.label_ids, skip_special_tokens=True)
  File "/opt/conda/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 3070, in batch_decode
    return [
  File "/opt/conda/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 3071, in <listcomp>
    self.decode(
  File "/opt/conda/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 3109, in decode
    return self._decode(
  File "/opt/conda/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 495, in _decode
    text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
OverflowError: out of range integral type conversion attempted
```

This is because in the prediction loop, the labels will be padded with -100 if the prediction or labels have different sequence length https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L1637.



Comments: