noamaanMulla-03/transformers-issues huggingface dataset:
Multi-GPU memory usage
Author: griff4692
Pull request: no

Hi - I am just using the Seq2SeqTrainer for a fairly vanilla BART fine-tuning use case.  In a multi-GPU setup, the memory, however, is really high for the first GPU (15GB and lower (as expected) for the other 7 (5GB).  What could be causing this?  Here are my training arguments.

>     training_args = Seq2SeqTrainingArguments(
        run_name=args.experiment,
        output_dir='./results',  # output directory
        overwrite_output_dir=True,
        max_steps=20000,  # total number of optimization steps
        per_device_train_batch_size=8,  # batch size per device during training
        per_device_eval_batch_size=8,  # batch size for evaluation
        eval_accumulation_steps=1,
        warmup_steps=0 if args.debug else 200,  # number of warmup steps for learning rate scheduler
        weight_decay=0,  # strength of weight decay
        logging_dir='./logs',  # directory for storing logs
        logging_steps=10,
        logging_first_step=True,
        no_cuda=args.cpu,
        fp16=True,
        do_train=True,
        do_eval=True,
        do_predict=True,
        learning_rate=5e-5,
        dataloader_num_workers=0 if args.debug else 8,
        label_smoothing_factor=0.1,
        save_strategy='steps',
        save_steps=999999 if args.debug else 2500,
        save_total_limit=3,
        seed=1992,
        evaluation_strategy='steps',
        eval_steps=999999 if args.debug else 2500,
        load_best_model_at_end=True,
        metric_for_best_model='mean_f1',
        greater_is_better=True,
    )

Comments:
cc @sgugger 

How are you launching your script? If you are using `python` and not the `torch.distributed.launch` module, then this is completely normal as the gradients and optimizer state are only stored on GPU 0, not the other ones.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Inconsistency in  class naming
Author: sadakmed
Pull request: no

@patrickvonplaten

in `modeling_utils.py` the pretrained model class is written as `PreTrainedModel` While in `configuration_utils` the pretrained config class is named as `PretrainedConfig`
https://github.com/huggingface/transformers/blob/421929b556aedf022a1c4a1f3b2f116b14a7b88a/src/transformers/modeling_utils.py#L413

https://github.com/huggingface/transformers/blob/421929b556aedf022a1c4a1f3b2f116b14a7b88a/src/transformers/modeling_utils.py#L30


and not just that, in `transformers` there's `PretrainedFSMTModel, PretrainedBartModel, PreTrainedModel, PretrainedConfig, PreTrainedTokenizer`
and some part of the documentations
https://github.com/huggingface/transformers/blob/421929b556aedf022a1c4a1f3b2f116b14a7b88a/src/transformers/modeling_utils.py#L425-L433


Edit:
Also in the `transformers.__init__.py` there's this type of inconsistency 

example:

https://github.com/huggingface/transformers/blob/421929b556aedf022a1c4a1f3b2f116b14a7b88a/src/transformers/__init__.py#L728

https://github.com/huggingface/transformers/blob/421929b556aedf022a1c4a1f3b2f116b14a7b88a/src/transformers/__init__.py#L580



Comments:
Hey @sadakmed,

Very good observation! We are aware of these slightly different names :-) However it would create quite some problems with backward compatibility so that we decided to leave as is. Hope this is ok for you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

> However it would create quite some problems

@patrickvonplaten  Totally agree, it's ok for me anyway!
I would only suggest to keep it in mind for the new upcoming models
cannot load character bert
Author: ghost
Pull request: no

## Environment info
- `transformers` version: 4.11.3
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

@LysandreJik I'm looking for help with character bert 

I am trying to use Character bert:

The problem arises when trying to load the model:

```
from transformers import AutoTokenizer, AutoModelForPreTraining
  
tokenizer = AutoTokenizer.from_pretrained("helboukkouri/character-bert")

model = AutoModelForPreTraining.from_pretrained("helboukkouri/character-bert")
```

Steps to reproduce the behavior:

```
from transformers import AutoTokenizer, AutoModelForPreTraining
  
tokenizer = AutoTokenizer.from_pretrained("helboukkouri/character-bert")

model = AutoModelForPreTraining.from_pretrained("helboukkouri/character-bert")

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/pupper/anaconda3/envs/work/lib/python3.7/site-packages/transformers/models/auto/tokenization_auto.py", line 463, in from_pretrained
    config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/Users/pupper/anaconda3/envs/work/lib/python3.7/site-packages/transformers/models/auto/configuration_auto.py", line 529, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
  File "/Users/pupper/anaconda3/envs/work/lib/python3.7/site-packages/transformers/models/auto/configuration_auto.py", line 278, in __getitem__
    raise KeyError(key)
KeyError: 'character_bert'
```

I expected it to load the model and tokenizer for use but it throws the error instead. Can anyone help here? Thanks!


Comments:
@Nitinram23  because it still unavailable #10053
![under construction](https://4efrxppj37l1sgsbr1ye6idr-wpengine.netdna-ssl.com/brushy-fork-institute/wp-content/uploads/sites/38/2018/04/Under-Construction-Sign-1024x483-300x142.png)


Meanwhile, you can install it from the PR branch,

`pip install   git+https://github.com/helboukkouri/transformers.git@add-character-bert`


and use it like this
```
from transformers.models.character_bert import   CharacterBertTokenizer, CharacterBertForPreTraining

tokenizer = CharacterBertTokenizer.from_pretrained("helboukkouri/character-bert")
model = CharacterBertForPreTraining.from_pretrained("helboukkouri/character-bert")
```


Much appreciated!
early_stopping_patience_counter increasing + 2 per epoch in distributed mode
Author: avercau
Pull request: no

## Environment info
- `transformers` version: 4.4.0
- Platform: Linux-5.4.0-1047-azure-x86_64-with-glibc2.27
- Python version: 3.8.0
- PyTorch version (GPU?): 1.7.1+cu101 (True)
- Tensorflow version (GPU?): 2.2.1 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes


### Who can help

- trainer: @sgugger

## Information

Model I am using (Bert, XLNet ...): LayoutLMForTokenClassification

The problem arises when using:
* [ ] my own modified scripts
I am training a LayoutLmForTokenClassification model in distributed mode, and added logging for the EarlyStoppingCallback.early_stopping_patience_counter to gain more insights. I noticed that the count augments with 2 at each epoch, instead of with one (it is called on_epoch_end), I guess from each parallel computation?


## To reproduce

Steps to reproduce the behavior:

1. Train LayoutLMForTokenClassification with trainer class with evaluation_strategy=IntervalStrategy.EPOCH and using EarlyStoppingCallback 


## Expected behavior

increase of patience counter + 1 at each epoch


Comments:
In distributed training, each process will have their own `EarlyStoppingCallback` that are independent and incremented separately, so that's not the issue here.
I tested locally and I see it increases one by one normally even in distributed training.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Enable passing config directly to PretrainedConfig.from_pretrained()
Author: RylanSchaeffer
Pull request: no

The `PretrainedConfig` class has a method for loading configs `from_pretrained()`. As the documentation states, the input can be either a string or a path to a config file:

```
            pretrained_model_name_or_path (:obj:`str` or :obj:`os.PathLike`):
                This can be either:

                - a string, the `model id` of a pretrained model configuration hosted inside a model repo on
                  huggingface.co. Valid model ids can be located at the root-level, like ``bert-base-uncased``, or
                  namespaced under a user or organization name, like ``dbmdz/bert-base-german-cased``.
                - a path to a `directory` containing a configuration file saved using the
                  :func:`~transformers.PretrainedConfig.save_pretrained` method, e.g., ``./my_model_directory/``.
                - a path or url to a saved configuration JSON `file`, e.g.,
                  ``./my_model_directory/configuration.json``.
```

**Request:** I would like to be able to pass the config dictionary directly to this function.

**Motivation:** if I have a config in memory, I shouldn't have to write the config to disk and then pass that file path to the method.

**Proposed Solution:** Add an extra argument `config_dict` and then replace

```
config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
```

with something like

```
if config_dict is not None and pretrained_model_name_or_path is not None:
    assert ValueError('Only one of config_dict and pretrained_model_name_or_path can be provided.`)
if config_dict is None:
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
```

Comments:
My current workaround isn't onerous, but I feel like this my request is pretty simple :)

```
    student_config = some dict
    # Frustratingly, the .from_pretrained method doesn't permit passing a config directly.
    # Consequently, we write the config to disk and then pass that path.
    os.makedirs('tmp', exist_ok=True)
    with open('tmp/student_config.json', 'w') as fp:
        json.dump(obj=student_config, fp=fp)
    stu_architecture_config = student_config_class.from_pretrained('tmp/student_config.json')

```

Hi @RylanSchaeffer, this can be done with the `from_dict` method.

```
config = BertConfig.from_pretrained("bert-base-uncased", hidden_dropout_prob=0.5)
second_config = BertConfig.from_dict(config.to_dict())

print(second_config.hidden_dropout_prob) # output: 0.5
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
convert pytorch checkpoints to TF1.x checkpoints (reverse of transformers-cli convert)
Author: fangli80
Pull request: no

# 🚀 Feature request

Hello, 
Some BERT models were pre-trained using Huggingface/Facebook/NIVIDA's implementations and only have `pytorch.bin`. I am wondering if I can convert the checkpoints to  TF1.x (compatible with google's BERT https://github.com/google-research/bert).




Comments:
Hello! We don't have a conversion script capable of doing that. Is there a feature of google-research/bert we could implement to make your life easier with `transformers` ?

This is simply for comparison reasons. We want to see if there is a performance difference due to the implementation. 

By the way, I also tried to convert the PyTorch checkpoint to TF2 using the `convert_pytorch_checkpoint_to_tf2.py` script. But I got this error:

```
$ python convert_pytorch_checkpoint_to_tf2.py 
Traceback (most recent call last):
  File "convert_pytorch_checkpoint_to_tf2.py", line 21, in <module>
    from . import (
ImportError: attempted relative import with no known parent package
```

I am using transformers v4.10.0 and I have installed this package. 

Thanks! 
Li


By the way, my python version is 3.8.11
I am using ubuntu 18.04

> Is there a feature of google-research/bert we could implement to make your life easier with transformers? 

the `run_classifier.py` script supports input of tsv text. the `datasets` library of huggingface only supports csv format. tsv format is easier to parse than csv format. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
input params in RobertaForQuestionAnswering
Author: TarangRanpara
Pull request: no

can anyone please help me understand what are `start_positions`  and `end_positions` vars that are being passed in the model? 
(example from documentation)
[link](https://huggingface.co/transformers/model_doc/roberta.html#tfrobertaforquestionanswering): 
below is code example:
```
>>> from transformers import RobertaTokenizer, RobertaForQuestionAnswering
>>> import torch

>>> tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
>>> model = RobertaForQuestionAnswering.from_pretrained('roberta-base')

>>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
>>> inputs = tokenizer(question, text, return_tensors='pt')
>>> start_positions = torch.tensor([1])
>>> end_positions = torch.tensor([3])

>>> outputs = model(**inputs, start_positions=start_positions, end_positions=end_positions)
>>> loss = outputs.loss
>>> start_scores = outputs.start_logits
>>> end_scores = outputs.end_logits
```




Comments:
I also have the same question as you.

The `start_positions` and `end_positions` are the labels that you provide to the model. With these, you tell the model which token is at the start of the answer, and which token is at the end of the answer.

In the example you list above, we have a single question + text. The answer to the question is "a nice puppet". So what should be the start and end positions here?

Well, first we need to tokenize the question + text using the tokenizer. We can do this as follows:

```
from transformers import RobertaTokenizer

tokenizer = RobertaTokenizer.from_pretrained("roberta-base")
question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
encoding = tokenizer(question, text, return_tensors="pt")
```
We can now check the tokens, by converting the `input_ids` back to text using the tokenizer's `decode` method:

```
for idx, id in enumerate(encoding.input_ids.squeeze().tolist()):
  print(idx, tokenizer.decode([id]))
```
This prints:
```
0 

1 Who
2  was
3  Jim
4  H
5 enson
6 ?
7 

8 

9 Jim
10  H
11 enson
12  was
13  a
14  nice
15  puppet
16 

```
So in this case, the start position should be 13 and the end position should be 15. 

Note that the `start_positions` and `end_positions` should be of shape (batch_size,), indicating the start token index and end token index respectively for every example in the batch. As we only have a single example in this case, `start_positions` should be `torch.tensor([[13]])`. Similarly, `end_positions` should be `torch.tensor([[15]])`. 

The reason `start_positions` and `end_positions` are set to 1 and 3 in the docs is to just illustrate it, they are actually incorrect.



Note: this is explained in detail in the official [question-answering notebook](https://github.com/huggingface/notebooks/blob/master/examples/question_answering.ipynb).

Thank you so much @NielsRogge. I finally understood the topic.
Best model not loaded in Trainer.hyperparameter_search
Author: Aktsvigun
Pull request: no

Hi,

I am inspecting the `hyperparameter_search` of `Trainer` class. It does not ignore the `patience` argument, however, it does ignore the `load_best_model_at_end`. This is a bug in my view, since `patience` should be ignored when `load_best_model_at_end` is unavailable. Attaching an example for `patience == 1`, `load_best_model_at_end == True` and `metric_for_best_model == "accuracy"`:
<img width="579" alt="Снимок экрана 2021-10-06 в 15 04 04" src="https://user-images.githubusercontent.com/36672861/136198585-a32588b6-ec1c-465f-b47b-5a2249bdbae2.png">
After this the training part is over, however, the reported final value is not the best one:
<img width="313" alt="Снимок экрана 2021-10-06 в 15 05 44" src="https://user-images.githubusercontent.com/36672861/136198804-49b8ba60-1876-48b6-9c2b-0dad5ae58b15.png">

I can make the merge request to solve the issue if necessary.



Comments:
Hi @Aktsvigun  would mind sharing your notebook or py file. I think it can be fixed without PR?

@pratikchhapolika sure, please find the notebook attached. In this notebook have a look at run 17 (penultimate cell, penultimate run). On the first epoch, the accuracy equals 0.45, still the "best run" has the accuracy of 0.44.
[hyp_trainer_example.txt](https://github.com/huggingface/transformers/files/7295195/hyp_trainer_example.txt)

Github does not allow me to attach .ipynb file, hence I changed the extension with .txt. To open it, change its extension from .txt to .ipynb

Maybe of interest to @sgugger 

Good afternoon! @sgugger any updates on this?

Yes the hyperparameter search framework ignore the `load_best_model_at_end` argument, and this is not something that can be changed as far as I can tell since they do not accept such an argument for the reporting. `load_best_model_at_end` should just not be used with HP-search, it's meant for standard trainings only.

See it, thanks! Closing then.
Add BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese
Author: datquocnguyen
Pull request: yes

# What does this PR do?


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Please help have a look: @patrickvonplaten, @patil-suraj Thanks.




Comments:
Thanks, @patil-suraj and @sgugger 
I fixed the pull request based on you guys' comments.

The same comment I get from both of you is regarding the vocab_file. Here is a summary:

I did not train a sentencepiece for Vietnamese.
`bartpho-syllable` employs the existing pre-trained "sentencepiece" model from XLMRoBERTaTokenizer, and this pre-trained "sentencepiece" model is referred to as a `vocab_file` of 250K types.
`reduced_vocab_file` is a vocab containing 40K Vietnamese-specificized types extracted from the XLMRoBERTaTokenizer vocab of 250K types.

Usecase of BartphoTokenizer: Other languages can thus simply reuse BartphoTokenizer with their own `reduced_vocab_file`. The goal here is to reduce model sizes of existing pre-trained XLM-RoBERTa/mBART models when applying to a smaller set of languages instead of the whole 50/100 languages.

@sgugger , @LysandreJik, @patil-suraj , @SaulLu  and @patrickvonplaten
Please could you have a look and provide your feedback for my recent changes? Thanks.

Thanks @LysandreJik 

My pull request suddenly failed the check `run_tests_torch_and_flax`:

`FAILED tests/test_modeling_flax_clip.py::FlaxCLIPModelTest::test_equivalence_flax_to_pt`
`FAILED tests/test_modeling_flax_clip.py::FlaxCLIPModelTest::test_equivalence_pt_to_flax`

They are out of my control, not relating to BartphoTokenizer.


Yes this is a problem unrelated to this PR so you can ignore those failures. They should be fixed tomorrow :-)

@sgugger I made a revision following your last comment. Thanks.
FYI, two failed tests are not related to BartphoTokenizer.

Thanks a gain for your contribution!
Add an API to register objects to Auto classes
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds an API to register custom to the auto classes.

Fixes #13522

Comments:
Thanks a lot for working on this! I think this would greatly benefit from having an entry/tutorial in the `Auto Classes` section:

![image](https://user-images.githubusercontent.com/30755778/137330917-4ccf2167-02db-4378-9234-9cab4f54df52.png)


WDYT?

Great idea!
Fix save when laod_best_model_at_end=True
Author: sgugger
Pull request: yes

# What does this PR do?

As pointed out on the [forums](https://discuss.huggingface.co/t/why-save-steps-should-be-a-round-multiple-of-eval-steps-when-load-best-model-at-end-true/10841/3), the `Trainer` keeps saving every `eval_steps` when `load_best_model_at_end=True` even though we now enforce that `save_steps` can be independent as long as it's a round multiple of `eval_steps`.

This PR fixes that.

Comments:
Error in code
Author: yanan1116
Pull request: no

hello,

https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization_no_trainer.py
in Line 87, the description is wrong. It should be 

> Finetune a transformers model on a summarization task

Please correct it.



Comments:
Hello,

Thanks for spotting this! Could you open a PR to fix this? Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.