noamaanMulla-03/transformers-issues huggingface dataset:
Add CodeCarbon Integration
Author: JetRunner
Pull request: yes

# What does this PR do?

This PR adds `codecarbon` for carbon footprint tracking. This is also useful for BigScience.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
cc @sashavor
Add output in a dictionary for TF `generate` method
Author: stancld
Pull request: yes

This PR adds two components into the TF `generate` method:
1. It enables the model outputs `attentions`, `hidden_states` and `scores`
2. It enables `return_dict_in_generate`

This PR thus narrows the gap between PyTorch and TF `generate` method implementations.

This PR also adds two tests for the dictionary output.

Besides, this PR fixes handling of 2-tuples of attentions for the XLNet model when `target_mapping is not None`.

**Reviewers:** @Rocketknight1 @patrickvonplaten @sgugger  (anyone else in the community)

<hr>

Edit: The above-mentioned features are not implemented for the `generate` method of `TFRagSequenceForGeneration` model.

Comments:
Great job @stancld !
run_ner.py training data file format
Author: pranav-s
Pull request: no

# 🚀 Feature request

Would it be possible to support text files as input files for the run_ner.py script similar to what was supported in examples/legacy/token-classification/run_ner.py ? This, I believe is the CoNLL-2003 format. The current version of the run_ner.py script in /examples/token-classification appears to support only json and csv format for training input. Could someone also tell me where I can find example files for aforementioned json and csv files ? Looking at some example files would help me format my labeled data in the required format if supporting text files is not possible.

## Motivation

I have some training data formatted as text files with one token and label per line and would like to use that as input to the run_ner.py script if possible.

Comments:
Hi @pranav-s 

We only support the `json` and `csv` files in the examples. To convert your text files to the `json` format, you could use this `datasets` script as references, which converts the `conll` text files to the `datasets` format.
https://github.com/huggingface/datasets/blob/master/datasets/conll2003/conll2003.py 

Hi @pranav-s,
I found this script from `test_examples.py` file,
```
run_ner.py
  --model_name_or_path bert-base-uncased
  --train_file tests/fixtures/tests_samples/conll/sample.json
  --validation_file tests/fixtures/tests_samples/conll/sample.json
  --output_dir {tmp_dir}
  --overwrite_output_dir
  --do_train
  --do_eval
  --warmup_steps=2
  --learning_rate=2e-4
  --per_device_train_batch_size=2
  --per_device_eval_batch_size=2
  --num_train_epochs=2
```
at this path `tests/fixtures/tests_samples/conll/sample.json` you can find an example file which is having required input format

Thank you @bhadreshpsavani and @patil-suraj . This was helpful.

Thank you @bhadreshpsavani . You saved my life!
Clean push to hub API
Author: sgugger
Pull request: yes

# What does this PR do?

This PR reworks the way the push to hub API works by avoiding cloning in a temporary directory, copying the files saved then pushing, by creating and cloning the repo first, then saving the object and lastly pushing it to the hub. The general commands does not bring any major breaking change although the behavior of `object.push_to_hub(repo_name)` and `object.save_pretrained(push_to_hub=True)` changes slightly (see below). That last API is not used much however, since it's new and completely undocumented, so I think it's okay.

`push_to_hub` takes a `repo_name_or_path` and will create a local clone of the repo if it does not exist. That local clone is synced with the distant repo. This is a bit different from before where a temp dir was used for the clone and push, this behavior is still accessible by passing along `temp_dir=True`.

## `push_to_hub` API for models, tokenizers and configs

Works like before with a slight change of behavior::
```
model.push_to_hub(repo_name_or_path="my-awesome-model")
```
will push to the hub the model by creating the repo, cloning it if it exists, saving the model inside and pushing. The change with before is that a local folder named "my_awesome_model" will be created if it does not exist, and if it exists it will either:
- be put in sync with the distant repo if the distant repo exists
- error if it is not a local clone of the distant repo
 
In the same vein
```
model.save_pretrained(my_folder, push_to_hub=True)
```
will use `my_folder` as a working directory and create it if it exists, error if it exists and is not a local clone of the distant repo and do a `git pull` if it exists and is a local clone of the distant repo.

In both cases, the previous behavior can be activated by passing along `temp_dir=True`.

Side note: this PR adds tests for the `FlaxPretrainedModel.push_to_hub` method.

## `push_to_hub` API for the Trainer

Here there are also slightly breaking changes in the sense that the control over the repo to which we push moves from arguments in the `push_to_hub` method to the fields in `TrainingArguments`. This is because the repo is now initialized at init, so we need to know the repo name, organization and potential token there.

The `Trainer` adds an automatic `.gitignore` to ignore all checkpoints folder, which can be changed by the user (we can add a CLI argument to control that in the future) and the `push_to_hub` method now just triggers a save, writes the model card then push the whole output dir to the distant repo.

Another slightly breaking change is that the default for the `logging_dir` (for TensorBoard) changes, so that the logs are inside the output_dir and also pushed to the hub.

Comments:
torch.nn.modules.module.ModuleAttributeError: 'RecursiveScriptModule' object has no attribute 'resize_token_embeddings'
Author: Mounika2405
Pull request: no

## Environment info

- `transformers` version: 2.0.0 (tried with 4.1.1 as well)
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7(False)
- Tensorflow version (GPU?): 1.14.0(False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

@LysandreJik @mfuntowicz

## Information
Model I am using: GPT2

The tasks I am working on is:
* Question generation given a paragraph, clue, style, and answer

The problem arises when using:
* Torchscript version of fine-tuned GPT2. I have an inference script in which I load the pre-trained tokenizer and add special tokens to it. I resize the token embeddings using the model.resize_token_embeddings() function after adding the special tokens. It works fine for the original PyTorch GPT2 model but fails for the traced(Torchscript) model. The code snippet is as follows:

    tokenizer = GPT2Tokenizer.from_pretrained(args.model_name_or_path)
    if args.model_name != "":
        model = GPT2LMHeadModel.from_pretrained(args.model_name)
    else:
        if args.torchscript:
            model = torch.jit.load(args.ts_model_name_or_path)
        else:
            model = GPT2LMHeadModel.from_pretrained(args.model_name_or_path)
    tokenizer.add_tokens(SPECIAL_TOKENS)
    model.resize_token_embeddings(len(tokenizer))

Following is the error stack trace:

Traceback (most recent call last):
  File "QG_gpt2_generate.py", line 5, in <module>
    run()
  File "/content/drive/MyDrive/home/FQG/src/model/FactorizedQG/GPT2_QG/interact.py", line 231, in run
    model.resize_token_embeddings(len(tokenizer))
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/_script.py", line 558, in __getattr__
    return super(RecursiveScriptModule, self).__getattr__(attr)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/_script.py", line 288, in __getattr__
    return super(ScriptModule, self).__getattr__(attr)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 779, in __getattr__
    type(self).__name__, name))
torch.nn.modules.module.ModuleAttributeError: 'RecursiveScriptModule' object has no attribute 'resize_token_embeddings'

Is there any other way in which I can perform the same operation of resizing for torchscript models? 
Thanks

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi @Mounika2405. Were you able to find a solution for this issue? I am facing a similar issue with another torch_script model

Facing the same issue.
Cannot import RobertaPreTrainedModel
Author: dogatekin
Pull request: no

## Environment info
I tried with both transformers 3.5.1 and 4.6.1.

### Who can help
Maybe @julien-c since they contributed RoBERTa.

## Information
I want to derive my own class from RobertaPreTrainedModel, but I cannot import that class like I can import e.g. BertPreTrainedModel or AlbertPreTrainedModel. More specifically,

```from transformers import BertPreTrainedModel```
and
```from transformers import AlbertPreTrainedModel```
works, but
```from transformers import RobertaPreTrainedModel```
returns `ImportError: cannot import name 'RobertaPreTrainedModel'`.

Is this the intended behavior or could it be a bug?

## To reproduce

Try `from transformers import RobertaPreTrainedModel`

## Expected behavior

The RobertaPreTrainedModel class should be imported like it works for other transformers.

Comments:
Add all XxxPreTrainedModel to the main init
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds all XxxPreTrainedModel classes to the main init, making them public, and more generally, adds a CI quality check to make sure every object in the modeling files that is a subclass of `PreTrainedModel`, `TFPreTrainedModel` or `FlaxPreTrainedModel` is public (except the Encoder, Decoder, Wrapper and the ones explicitly listed as privates).

Fixes #12193

Comments:
CPU Memory Leak when using RoBERTa for just word vector representation
Author: ZahraGithub
Pull request: no

Hi,
I do not use model for training or fine tuning. I just want to give strings and take their representations. My dataset is Robust04 (about 2G) and max length of each document is truncated to 1024 tokens. So, I break each documents to pieces of 64 tokens and represent it and then concatenate the representation of each 64 tokens pieces to get word vector representation of document with length of 1024 (1024*768). I use 64GB CPU RAM but it crashed after about 40% of documents represented. The used code is in the following:
`tokenizer = AutoTokenizer.from_pretrained('roberta-base')`
`model = AutoModel.from_pretrained('roberta-base')`
`cl_text = tokenizer.encode("I am using RoBERTa")`
`piece = tokenizer.decode(cl_text)`
`piece = tokenizer(piece, return_tensors="pt", max_length=le, pad_to_max_length=True, truncation=True,
					   return_attention_mask=True, return_token_type_ids=True, add_special_tokens=False)`
`piece = model(**piece)`
`piece = piece.last_hidden_state`
`piece.detach()`

Would you please guide me?

Thanks in advance,
Regards

Comments:
Hi @ZahraGithub, 
1) Please use `model.eval()` to reduce your memory consumption.
2) From what I understood, you're processing 64 tokens in one go and total tokens are 1024. This means you'll get vector representation of (16,768) for each document right? How are you storing these representations? You won't be able to load all of them in your RAM hence that CPU memory leak. Try saving these representations to disk for each (that can vary) document to avoid memory leak.

Hi @bhavitvyamalik

1. I did it but again the memory consumption is high.
2. I can not understand what is the amount of 16768?

For 2, I think what's happening is you're storing all your representations in a list or something. You should load them off your RAM and store in it your disk (you can save it as .npy file and later load them) so as to avoid 100% memory consumption.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Changed modeling_fx_utils.py to utils/fx.py for clarity
Author: michaelbenayoun
Pull request: yes

Moved **modeling_fx_utils.py** to **utils/fx.py** to make it clear that it is not "modeling_flax_utils.py".

Since there are a modeling_utils.py for PyTorch, and a modeling_tf_utils.py for TensorFlow, modeling_fx_utils.py could be believed to be the Flax counterpart, but it is actually related to the torch.fx feature, hence the pull request.

Comments:
[Flax T5] Fix weight initialization and fix docs
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Apply fixes to Flax T5 according to comments on https://github.com/huggingface/transformers/pull/12150

Comments:
The failing hub test is unrelated IMO
UpdateDescription of TrainingArgs param save_strategy
Author: sam-writer
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #12315

TrainingArguments parameter docs: mention in `save_strategy` param description that `load_best_model_at_end` can override.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? N/A


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger


Comments:
Exporting the operator repeat_interleave to ONNX opset version (<=12) is not supported!
Author: JaheimLee
Pull request: no

When I export Roformer model to onnxruntime, this error occurred! Is there any ops can replace 'torch.repeat_interleave'?
It's in [https://github.com/huggingface/transformers/blob/master/src/transformers/models/roformer/modeling_roformer.py](url), at line 330 and 332

Comments:
I have a solution, but maybe it's not the best way.

sin_pos = torch.zeros_like(sin).repeat(1, 1, 1, 2)
sin_pos[..., ::2] = sin
sin_pos[..., 1::2] = sin
cos_pos = torch.zeros_like(cos).repeat(1, 1, 1, 2)
cos_pos[..., ::2] = cos
cos_pos[..., 1::2] = cos

It seems that you haven't change roformer's positional embedding codes yet. Demo codes are like:
```python
import torch
sinusoidal_pos = torch.randn(1,12,16,32)
sin, cos = sinusoidal_pos.chunk(2, dim=-1)

sin_pos = torch.repeat_interleave(sin, 2, dim=-1)
cos_pos = torch.repeat_interleave(cos, 2, dim=-1)

sin_pos_newway = torch.stack([sin,sin],axis=-1).reshape_as(sinusoidal_pos)
cos_pos_newway = torch.stack([cos,cos],axis=-1).reshape_as(sinusoidal_pos)

assert sin_pos.equal(sin_pos_newway)
assert cos_pos.equal(cos_pos_newway)
```
Missing tokenizer_class for `mbart-large-50-many-to-one-mmt` model
Author: Mehrad0711
Pull request: no

Hi @patil-suraj,

I noticed all `mbart-50` models have their tokenizer_class set to "MBart50Tokenizer" in their config file except for `mbart-large-50-many-to-one-mmt`. This causes the wrong tokenizer to be loaded for this model (`tokenization_mbart` instead of `tokenization_mbart50`). Please check. Thanks! 

Comments:
Good catch @Mehrad0711 ! Thank you for reporting this. I just fixed it https://huggingface.co/facebook/mbart-large-50-many-to-one-mmt/blob/main/config.json
[trainer] fix label smoothing for default compute_loss
Author: kolakows
Pull request: yes

# What does this PR do?

Keeps 'labels' in the inputs which are passed to model.

Without this change, the model I'm using (PegasusForConditionalGeneration) can't calculate loss and generate outputs. Change assumes that all other models also need 'labels' in inputs.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->



## Who can review?

trainer: @sgugger


Comments:
No, if you leave the labels in the outputs, the model will then compute the loss without label smoothing which is inefficient (since it will re-compute the proper loss afterwards).

Oh you are right, I guess I was overeager with the PR, sorry.

After looking more into the way arguments are passed to forward, I'll just need to shift tokens in 'labels' and add them to inputs as 'decoder_input_ids' (my confusion was from the fact that, for the model I'm using it is done automatically, but inside forward)

Thanks for the quick answer!

No worries! And let us know if there is a problem with the generation of `decoder_input_ids` for Pegasus as we would need to fix it. :-)