noamaanMulla-03/transformers-issues huggingface dataset:
Training loss is not logged correctly when doing evaluation with Trainer
Author: gau-nernst
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.2
- Platform: Linux-5.4.0-62-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@sgugger 

## Information

Model I am using (Bert, XLNet ...): Reformer

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

I'm doing a text generation with Reformer using my own dataset.

## To reproduce

Steps to reproduce the behavior:

1. Set `logging_steps=10` and `evaluation_strategy="steps`, `eval_steps=20` in `TrainingArguments`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

I'm doing training in Jupyter notebook. With the settings above, the logging output shows training loss and validation loss every 20 steps (which is the `eval_steps`). However, I want the training loss to be logged at higher frequency than the validation loss, for example at 10 steps like above. This is because running evaluation will take some time for large validation set, while I still want to monitor mini-batch training loss.

When inspecting logs with Tensor Board, no training loss is logged at all (even the values every 20 steps).

If I disable evaluation (`evaluation_strategy="no"`), the training loss is logged every 10 steps as expected.

## Expected behavior

When enabling evaluation in Trainer, training loss should be logged every `logging_steps`, while validation loss is logged every `eval_steps`


Comments:
I am not able to reproduce, on my side I do see logs every logging_steps for the training loss and learning rate, and every eval_steps for the validation loss and metrics, both in the console and TensorBoard. Could you try again with a source install?

After I restarted TensorBoard, training loss showed up correctly again. Maybe something went wrong with TensorBoard. Thank you for your prompt response! FYI it worked correctly with my current version `transformers=4.4.2`. It was a silly mistake from my side.
Errors in inference API
Author: guplersaxanoid
Pull request: no

I understand that the inference API returns a json with "error" field if an error occurs. Where can I find the list of such possible errors?

Comments:
Maybe of interest to @Narsil 

@DeveloperInProgress .
Sorry, there is no exhaustive list of those as of yet (as a number of them are actual exceptions raised by transformers itself)

What I can say, is that Environment and ValueError are simply displayed as is and treated as user error (usually problem in the model configuration or inputs of the model).

Any other exception is raised as a server error (and looked at regularly).

Any "unknown error" is an error for which we can't find a good message, we try to accompany it (when it's possible) with any warnings that might have been raised earlier by transformers (for instance too long sequences make certain models crash, deep cuda errors are unusable as is, the warning is better).

Does that answer your question ?

@Narsil gotcha
Add nvidia megatron models
Author: jdemouth
Pull request: yes

# What does this PR do?

Add the megatron_gpt2 model. That model reuses the existing GPT2 model. This
commit includes a script to convert a Megatron-GPT2 checkpoint downloaded
from NVIDIA GPU Cloud. See examples/megatron-models/README.md for details.

Add the megatron_bert model. That model is implemented as a modification of
the existing BERT model in Transformers. This commit includes a script to
convert a Megatron-BERT checkpoint downloaded from NVIDIA GPU Cloud. See
examples/megatron-models/README.md for details.

@LysandreJik 



Comments:
@LysandreJik - you'll see that the last test (marked 'slow') in tests/test_modeling_megatron_bert.py points to a checkpoint in examples/megatron-models (downloaded following the instructions described in examples/megatron-models/README.md). I was not sure how to deal with that so suggestions are welcome (for other items too ;)).

We have a few failing tests, let me break them down for you:

### build_doc

The build_doc is failing because of the following errors:
```
/home/circleci/transformers/docs/source/model_doc/megatron_bert.rst:document isn't included in any toctree
```

The `megatron_bert.rst` should be defined in the index of the docs :)

### check_code_quality

The error is:
```
2 files would be reformatted, 783 files would be left unchanged.
```

For this, you should install the quality tools: `pip install -e .[quality]` (from the root of the repo)
and run the following:
```
make fixup
```

This is going to fix some files, and tell you if there are errors it cannot resolve. If there are some, it should tell you how to fix them.

### run_test_flax, run_tests_tf, and run_tests_pipelines_tf

This is due to the following error:

```
____________ ERROR collecting tests/test_modeling_megatron_bert.py _____________
tests/test_modeling_megatron_bert.py:256: in <module>
    class MegatronBertModelTest(ModelTesterMixin, unittest.TestCase):
tests/test_modeling_megatron_bert.py:259: in MegatronBertModelTest
    MegatronBertModel,
E   NameError: name 'MegatronBertModel' is not defined
```

I think this comes from a missing `@require_torch` decorator on one of your tests. This decorator tells the suite that this test requires torch, and to not run this test if torch is not found as a dependency inside the environment.

If that's not it, then it may be that it's missing a dummy object. Running `make fix-copies` should fix this, but you should already have run this if you have done the fix mentioned above relative to the style/code quality.
When I try to import my model I run into an error "TypeError: PyMetaspace.__new__() got an unexpected keyword argument: str_rep"
Author: MLDovakin
Pull request: no

In the Hugging Face repository I have my own model Fidlobabovic / beta-kvantorium-simple-small 

https://huggingface.co/Fidlobabovic/beta-kvantorium-simple-small/tree/main
#7370 
#10148 
When I try to import it, I get an error. What can I do to fix it? Overwrite model files or rename them? What should I write for this?
````
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("Fidlobabovic/beta-kvantorium-simple-small")

model = AutoModelForMaskedLM.from_pretrained("Fidlobabovic/beta-kvantorium-simple-small")

TypeError                                 Traceback (most recent call last)
<ipython-input-5-3f4375d4cdf7> in <module>()
      1 from transformers import AutoTokenizer, AutoModelForMaskedLM
      2 
----> 3 tokenizer = AutoTokenizer.from_pretrained("Fidlobabovic/beta-kvantorium-simple-small")
      4 
      5 model = AutoModelForMaskedLM.from_pretrained("Fidlobabovic/beta-kvantorium-simple-small")

4 frames
/usr/local/lib/python3.7/dist-packages/transformers/models/gpt2/tokenization_gpt2_fast.py in __init__(self, vocab_file, merges_file, tokenizer_file, unk_token, bos_token, eos_token, add_prefix_space, **kwargs)
    150             pre_tok_class = getattr(pre_tokenizers, pre_tok_state.pop("type"))
    151             pre_tok_state["add_prefix_space"] = add_prefix_space
--> 152             self.backend_tokenizer.pre_tokenizer = pre_tok_class(**pre_tok_state)
    153 
    154         self.add_prefix_space = add_prefix_space

TypeError: PyMetaspace.__new__() got an unexpected keyword argument: str_rep
````

Comments:
Maybe @n1t0 knows what might be happening here.

@n1t0 LysandreJik spoked call you  

Dose this problem fixed? I also got the same problem, I trained a SentencePieceBPETokenizer, use save() api to persist it as a tokenizer.json file, but got this error while loading. 

I dig into the rust code, and found that Metaspace is defined as a struct with three attributes but the constructor only accept two. I tried to modify the json file and remove the one not in the constructor method, but got another error: 

`Exception: data did not match any variant of untagged enum PyPreTokenizerTypeWrapper at line 1 column 1449
`
Below is the content around char 1449:
`
lse, "normalized": false}], "normalizer": {"type": "NFKC"}, "pre_tokenizer": {"type": "Metaspace", "replacement": "\\u2581", "add_prefix_space": true}, "post_processor": null, "decoder": {"type": "Meta
`

This has been fixed in the latest version of tokenizers (`0.10.2`)
Don't duplicate logs in TensorBoard and handle --use_env
Author: sgugger
Pull request: yes

# What does this PR do?

This PR fixes a few bugs in the `Trainer` and `TrainingArguments`. First it cleans up the `TensorBoardCallback` to make sure the logs are not duplicated (I think they were not for some convoluted logic with the `tb_writer` never set but now I'm sure).

The second part is more important and handles support for when a user launches a training script using `Trainer` with the `--use_env` option (for instance when using `accelerate launch`). In this case the argument `local_rank` is not passed directly, it's just set in the environment and we did not detect it.

Comments:
considering `pad_to_multiple_of` for run_mlm.py
Author: dorost1234
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.3
- Platform: linux
- Python version: 3.7
- PyTorch version (GPU?): 1.8
- Tensorflow version (GPU?): - 
- Using GPU in script?: - 
- Using distributed or parallel set-up in script?: - 

### Who can help
@sgugger, @patil-suraj

## Information
I have seen in huggingface codes such as run_seq2seq.py that for the case of fp16, they pad to multiple of 8 like below, perhpas for efficiency purpose:

```
 data_collator = DataCollatorForSeq2Seq(
        tokenizer,
        label_pad_token_id=label_pad_token_id,
        pad_to_multiple_of=8 if training_args.fp16 else None,
       )
```

For run_mlm.py codes, the padding condition in case of fp16 is not considered, could it be a bug and possibly performance could get better in run_mlm.py if this was set? kindly appreciate  having a look.

thanks 


Comments:
Indeed, we could use that when the line by line option is set (otherwise there is just no padding). Would you like to make a PR with this?
[tests] relocate core integration tests
Author: stas00
Pull request: yes

This PR 

* moves `deepspeed`/`fairscale`/extended trainer tests from `examples` to `tests`
* updates docs to point to the new sample config files
* adds a new `testing_utils.py`'s context manager `ExtendSysPath` that allows temporary `sys.path` changing to import something locally in the tests and uses it (otherwise sagemaker tests were breaking because they contain `__init__.py`) + doc

Hopefully, it'll be the new home for integration tests for awhile, - specifically for deepspeed tests as the DeepSpeed team would like to run our tests as part of their CIs.

We still need to split off the `fairscale` tests once we start working on this integration again, so for now just moving as is.

@sgugger 

Comments:
@sgugger, @LysandreJik - so after this move we have a problem of dependencies now - the extended integration tests need be able to check score metrics, but the main tests don't have `sacrebleu` and other dependencies installed. How do we resolve this conundrum?

Lysandre replied on slack to add them to `extras["testing"]` - so doing that.

I think we can add
```
sacrebleu >= 1.4.12
rouge-score
nltk
```
to the testing extra. It should be all you need.
Add support for multiple models for one config in auto classes
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds support for having multiple models with the same config be in the same auto class. For instance `FunnelBaseModel` and `FunnelModel` are both valid models for the class `AutoModel` but since they both rely on `FunnelConfig`, only `FunnelModel` was in the model mapping for `AutoModel`.

The mechanism when loading changes slightly: if the mapping finds a tuple for the config at hand, it will look into the `architectures` field and return the model in the tuple corresponding to the architecture found there, or the first model of the tuple as a default.

While diving into this, I realized that TF and Flax pretrained models do not populate the field architectures of their configs, so I added support for this.

The rest of the changes are need to adapt to the fact some model mappings now can have tuple values.

Comments:
[setup] make fairscale and deepspeed setup extras
Author: stas00
Pull request: yes

Based on a request adding support for:
```
pip install transformers[deepspeed]
pip install transformers[fairscale]
```
so moving the version minimums into `setup.py`, and also had to add a helper function `dep_version_check`

@LysandreJik, @sgugger 

Comments:
Bart EncoderLayer masked_fill not working properly with pytorch 1.4
Author: mariamabarham
Pull request: no
Labels: name: wontfix

# 🐛 Bug


## Information

Model I am using (Bert, XLNet ...): Bart 
I'm trying to use the EncoderLayer of Bart but I realized that `attn_weights = attn_weights.masked_fill(reshaped, float("-inf"))` at line [659](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bart.py#L659) does not work when `reshaped` is an `int` or `float` tensor and throws the following error:
```
attn_weights = attn_weights.masked_fill(reshaped, float("-inf"))
RuntimeError: Expected object of scalar type Bool but got scalar type Float for argument #2 'mask' in call to _th_masked_fill_bool_
```
However it does not raise an error when I change `reshaped` type to bool but in that case it returns a tensor of `nan` values.

With @patrickvonplaten helps, we realized that it was related to a pytorch version because upgrading my torch version from 1.4 to 1.5 solved the problem

## To reproduce
```
from transformers.modeling_bart import EncoderLayer 
from transformers import BartConfig 
import torch 
hidden_states = torch.tensor(3 * [ 7 * [ 1024 * [0.4]]]) 
attn_mask = torch.ones(hidden_states.shape[:2]) 
layer = EncoderLayer(BartConfig()) 
layer(hidden_states.transpose(0, 1), attn_mask)  

```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0
- Python version: 3.6
- PyTorch version (GPU?):1.4
- Using GPU in script?: no
- Using distributed or parallel set-up in script?:no


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hello,  I still hava this problem when my pytorch were upgraded  to 1.5 .  I don't know if it's related to the python version . 
Can you give me some suggestions ？ Thank you so much!

Information
`  File "/home/ynos/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/functional.py", line 3937, in multi_head_attention_forward
    float('-inf'),
RuntimeError: Expected object of scalar type Bool but got scalar type Long for argument #2 'mask' in call to _th_masked_fill_bool_
`

Environment info:
- Python version: 3.6.2
- PyTorch version (GPU): 1.5
Make `get_special_tokens_mask` consider all tokens
Author: sgugger
Pull request: yes

# What does this PR do?

As discovered via #11155, some tokenizers do not return the proper special tokens mask in the `get_special_tokens_mask` when `already_has_special_tokens=True` because they only check for the CLS and SEP tokens. This PR fixes that by delegating to the superclass the call when `already_has_special_tokens=True` (the generic method checks for all special tokens).

It also seems from the error message [here](https://github.com/huggingface/transformers/blob/b9b60c1630f63b54b10380ef8bf30ec323985553/src/transformers/tokenization_utils_base.py#L3091) that the `get_special_tokens_mask` method is not supposed to be implemented for fast tokenziers when `already_has_special_tokens=True`, so this PR removes this method from the fast tokenizers where it exists, except for a select few that have a different implementation.

Fixes #11155

Comments:
Thanks a lot for your super-fast feedback! Your projects are big inspiration to me. Thank you.
tokenizer.encode_plus returns torch.tensors loaded on the desired device
Author: sadakmed
Pull request: no

# 🚀 Feature request

add device attribute to tokenizer.encode_plus so when it returns a torch.tensors it loads it on the desired device

## Motivation

- to pass the output of tokenizer to the model, one only can unpack the returned output using ** without bothering about the content of tokenizer, that only true when with cpu, but for gpu u need to unpack the output and load each input to device, and then pass them to the model.
this process will be also frustrating if you don't know the keys from the output or when you want to switch from a model to another -eg from bert to roberta, roberta doesnt need token_type_ids-.



Comments:
Hi! You can cast the `BatchEncoding` output by `encode_plus` to your device:
```py
model_input = tokenizer.encode_plus(xxx, return_tensors="pt")
model_input.to("cuda")
```

niiice, thank you!!
Add a special tokenizer for CPM model
Author: JetRunner
Pull request: yes

Added a special tokenizer (pre-tokenization) for CPM model: https://huggingface.co/TsinghuaAI/CPM-Generate

Comments:
The CI error looks irrelevant

Sure! Thanks for the guidance.

@LysandreJik ping😅

Done @LysandreJik 

> Great, thanks a lot for adding this! Could you also add the model to the main README since it has a doc page? After doing so run `make fix-copies` so the table in the index is also updated.

Hi @sgugger I'm not sure about this! We only have a tokenizer (no modeling file) here so maybe it shouldn't be in the README (from my understanding)?

The README contains BARThez that also only adds a tokenizer so there is a precedent. BORT is also there and only adds a conversion script, re-using other models and tokenizers from the lib.

> The README contains BARThez that also only adds a tokenizer so there is a precedent. BORT is also there and only adds a conversion script, re-using other models and tokenizers from the lib.

Makes sense! I'll do it tmrw!

Done

@LysandreJik Mind taking a look at the CI? Dunno what's wrong there.

Don't worry about it, it's unrelated to your PR. It's an issue with Megatron BERT, we temporarily removed that model from the CI on `master`
Why padding tokens can be masked in albert model? Is it bug or right?
Author: woong97
Pull request: no

I tried to run run_mlm.py for bert model and albert model.
"pad" token is not masked when I run bert-base-uncased model , but "pad" token can be masked when I run albert-base-v2

[bert command]
```
%  python run_mlm.py --model_name_or_path bert-base-uncased --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --do_train --do_eval --output_dir ./tmp/test-mlm --line_by_line

```

[albert command]
```
%  python run_mlm.py --model_name_or_path albert-base-v2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --do_train --do_eval --output_dir ./tmp/test-mlm --line_by_line

```

In examples/language-modeliing/run_mlm.py, I try to call tokenizer.get_special_tokens_mask.
```
tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, **tokenizer_kwargs)
print(tokenizer.get_special_tokens_mask([0, 100, 101, 102, 2, 3, 4], already_has_special_tokens=True))
```

"get_special_tokens_mask" function is called from "class PreTrainedTokenizerBase" when I run bert-base-uncased, but "get_special_tokens_mask" function is called from "class AlbertTokenizerFast"  whenn I run albert-base-v2.


In PretrainedToknizerBase class,
```
def get_special_tokens_mask(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
    ) -> List[int]:
   all_special_ids = self.all_special_ids  # cache the property
   special_tokens_mask = [1 if token in all_special_ids else 0 for token in token_ids_0]

   return special_tokens_mask
```


However in AlbertTokenizerFast class,
```
def get_special_tokens_mask(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
    ) -> List[int]:
    if already_has_special_tokens:
        if token_ids_1 is not None:
            raise ValueError(
                "You should not supply a second sequence if the provided sequence of "
                "ids is already formatted with special tokens for the model."
            )
        return list(map(lambda x: 1 if x in [self.sep_token_id, self.cls_token_id] else 0, token_ids_0))

    if token_ids_1 is not None:
        return [1] + ([0] * len(token_ids_0)) + [1] + ([0] * len(token_ids_1)) + [1]
    return [1] + ([0] * len(token_ids_0)) + [1]
```

=> These two functions are different. Thus when I use bert, all_special_ids( it contains cls, sep, pad id) are ids which cannot be masked. But when i use albert, only cls, sep  ids cannot be masked. Thus pad token can be masked when i use albert.

I don't know why the functions are called from different class when I run bert-base-uncased or albert. 
Do you know why??

And is it correct that pad token will be masked in albert model?? 


 


Comments:
Related to #11163 by @sgugger 

This is solved by #11163