noamaanMulla-03/transformers-issues huggingface dataset:
Can I  print mlm and nsp loss instead of their sum while train the BertForPretraning model?
Author: dancingpipi
Pull request: no

The loss just contain the sum of nsp and mlm loss, I want to know their respective specific values

Is there a way to do that?

Comments:
Hey @dancingpipi! There's no way to get the individual losses out of the box, but you could use the returned logits with your labels to calculate the two losses out of the model.

The loss is computed as such:

```
        if labels is not None and next_sentence_label is not None:
            loss_fct = CrossEntropyLoss()
            masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))
            next_sentence_loss = loss_fct(seq_relationship_score.view(-1, 2), next_sentence_label.view(-1))
            total_loss = masked_lm_loss + next_sentence_los
```

and the output contains the `prediction_logits` (named `prediction_scores` above) and `seq_relationship_logits` (named `seq_relationship_score` above)

@LysandreJik thanks for your reply! 
I refer to the compute_loss, but the problem still exists: how to print them or add the to tensorboard in distributed training?


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Eval rouge = 100.0 on gem/wiki_lingua_english_en
Author: tuvuumass
Pull request: no

Hey everyone!

I just tried the example code `transformers/examples/pytorch/summarization/run_summarization_no_trainer.py` but got 100.0 for eval rouge1, rouge2, and rougeL on a GEM dataset. Seems like there is something wrong here. Am I missing something? (Tagging @patil-suraj as it involves summarization)

My script

```
accelerate launch run_summarization_no_trainer.py \
    --model_name_or_path t5-small \
    --dataset_name "gem" \
    --dataset_config "wiki_lingua_english_en" \
    --source_prefix "summarize: " \
    --output_dir ~/tmp/tst-summarization
```

Training log

```
12/14/2021 18:51:04 - INFO - __main__ - ***** Running training *****
12/14/2021 18:51:04 - INFO - __main__ -   Num examples = 99020
12/14/2021 18:51:04 - INFO - __main__ -   Num Epochs = 3
12/14/2021 18:51:04 - INFO - __main__ -   Instantaneous batch size per device = 32
12/14/2021 18:51:04 - INFO - __main__ -   Total train batch size (w. parallel, distributed & accumulation) = 128
12/14/2021 18:51:04 - INFO - __main__ -   Gradient Accumulation steps = 1
12/14/2021 18:51:04 - INFO - __main__ -   Total optimization steps = 2322
  0%|          | 0/2322 [00:00<?, ?it/s]
  0%|          | 1/2322 [00:00<19:52,  1.95it/s]12/14/2021 18:51:05 - INFO - root - Reducer buckets have been rebuilt in this iteration.
12/14/2021 18:51:05 - INFO - root - Reducer buckets have been rebuilt in this iteration.
12/14/2021 18:51:05 - INFO - root - Reducer buckets have been rebuilt in this iteration.
12/14/2021 18:51:05 - INFO - root - Reducer buckets have been rebuilt in this iteration.
__main__ - {'rouge1': 100.0, 'rouge2': 100.0, 'rougeL': 100.0, 'rougeLsum': 100.0}
Configuration saved in ./output_dir/config.json
Model weights saved in ./output_dir/pytorch_model.bin
tokenizer config file saved in ./output_dir/tokenizer_config.json
Special tokens file saved in ./output_dir/special_tokens_map.json
Copy vocab file to ./output_dir/spiece.model
100%|██████████| 2322/2322 [07:28<00:00,  5.18it/s]
```

Comments:
Hello, thanks for opening an issue! For an additional chance of getting your question answered, could you ask your question on the [forum](https://discuss.huggingface.co) as well, to involve the broader community?

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Add tqdm to pipeline
Author: dunalduck0
Pull request: no

# 🚀 Feature request

Pipeline can process a list of inputs but doesn't print out progress. If the input list is large, it's difficult to tell whether the pipeline is running fine or gets stuck. Add tqdm to the generation loop to show progress.

For example, if I add tqdm to /src/transformers/pipelines/base.py line 1086, then I can see text generation progress. But I don't know how to add such a feature to all paths.

**from tqdm import tqdm**
....
if self.framework == "pt":
    final_iterator = self.get_iterator(
        inputs, num_workers, batch_size, preprocess_params, forward_params, postprocess_params
    )
    outputs = [output for output in **tqdm**(final_iterator)]
    return outputs

Generation progress would show in stdout:

Process 0, Generate starts: Wed Dec 15 12:27:42 2021
  2%|█▊                                                                                             | 2/92 [00:17<13:01,  8.68s/it]

Comments:
Yeah that would be quite nice! cc @Narsil 

Hi @dunalduck0 ,

If you use a `Dataset` instead of a list, it will work.

```python
dataset = MyDataset()

for out in tqdm.tqdm(pipe(dataset)):
    print(out)
```

And converting a raw list to a dataset is relatively easy

```python
class ListDataset(Dataset):
     def __init__(self, original_list)
        self.original_list = original_list

    def __len__(self):
        return len(self.original_list)

    def __getitem__(self, i):
        return self.original_list[i]
```
This helper could live in transformers pipelines utils for sure.

The problem with doing what you suggest is:

- tqdm needs a special command when running in notebooks (not something we should handle IMO)
- `List` existed previously so we need to return a `list` for backward compatibility reasons. That means the pipeline needs to consume and recreate the list, meaning adding tqdm inside is not desirable IMHO (or supporting it by adding special arguments).

My personal take would be to not break backward compatibility for lists, but encourage instead to use iterator enabling user code to decide if they want something like `tqdm` or any other solution (also tweaking all the arguments of `tqdm` like the name of the loop so on and so forth).
The helper to transform the `list` into a `Dataset` is already something (and might cover other frameworks too which is something we have in mind)




This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Occasional Can not load roberta-base tokenizer 
Author: CaoYiqingT
Pull request: no

**Environment info**
transformers 4.12.0.dev

**issue**
when using `tokenizer = AutoTokenizer.from_pretrained('roberta-base')`, got the bug as following:
```
OSError: Can't load config for 'roberta-base'. Make sure that:

- 'roberta-base' is a correct model identifier listed on 'https://huggingface.co/models'
  (make sure 'roberta-base' is not a path to a local directory with something else, in that case)

- or 'roberta-base' is the correct path to a directory containing a config.json file
```
Sometimes this bug occurs while sometimes it works well
@LysandreJik


Comments:
I think this is probably due to an internet connection error, unfortunately

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Nan when training LayoutLM_V2 Model
Author: Asocsar
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.13.0
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU) : 1.10.0+cu111
- Tensorflow version (GPU): 2.7.0
- Flax version: not installed
- Jax version: not installed
- JaxLib version: not installed


### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people. -->

@NielsRogge




## Information

The model used is LayoutLMv2:

The problem arises when using:
* [x] my own modified scripts:


The tasks I am working on is:
* [x] Document Classification

## To reproduce

Steps to reproduce the behavior:

1. Access to the colab notebook created to train LayoutLM_V2 (https://colab.research.google.com/drive/1u4xfrP2tWqMpgnoh8ciuUHeW56ppguse?usp=sharing)
2. Execute every cell in order
3. In the training loop, Accuracy, loss, and output will be printed, and there will be a moment when the output, Accuracy and Loss will become Nan.

## Expected behavior

The model trains, and despite if it accomplishes its task or not, the training loop ends without any Nan.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
why training bigbird-512 model is much slower than bert-512?
Author: jinxinglu
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.15.0
- Platform: 
- Python version: 3.6.8
- PyTorch version (GPU?): 1.10.1+cu102
- Tensorflow version (GPU?):
- Using GPU in script?: one GPU of V100-16GB
- Using distributed or parallel set-up in script?: no

### Who can help
@patrickvonplaten

Model I am using (Bert, XLNet ...): BigBird

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [*] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [*] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

I have compared the training time of bigbird model and bert model, both in the sequence length of 512, with the official recommend pipeline (config -> MLM model -> dataset -> data_collator -> training_args -> trainer ).
But I find the training time of bigbird model is much bigger than bert model, even in the same seq_length, here are the detailed parameters:
num_train_epochs=1
learning_rate = 5e-5
warmup_steps = 0
save_strategy='no'
gradient_checkpointing=False,
seed=222
gradient_accumulation_steps=1  
max_length = 512
the total number of samples is 200_000
1. for bert-512, I can set the batchsize up to 13 (bigger would raise OOM error), the total training time is 2:04:18
2. for bigbird-512, I revise the block_size to 32 to meet the requirements of (5 + 2*num_random_blocks)*block_size < 512, and only can set the batchsize up to 11 (bigger would raise OOM error, this value is smaller than Bert's, I don't know the reaseon, the total parameters of bigger would be bigger than Bert's ?), the total training time is about 4:45:23, that is much slower than Bert-512, 

Anything wrong? looking forward to your response, thanks advance


Comments:
If your maximum sequence length is just 512, it does indeed not make much sense to use BigBird and BERT, AlBERT, or DeBERTa make much more sense. BigBird should be used for models that require sequence lengths > 2048

The main promise of BigBird is not that it is faster than BERT for seq_len < 512, but rather that it can handle sequence lengths of up to 16K tokens

Got it. Thanks very much.
Robust Speech Challenge Evaluation File does not use entire dataset for metric calculation
Author: anuragshas
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.15.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU?): 1.10.0+cu111 (True)
- Using GPU in script?: Yes

### Who can help
@patrickvonplaten, @anton-l

## To reproduce

Steps to reproduce the behavior:

1. ~/transformers/examples/research_projects/robust-speech-event/eval.py --model_id ./ --dataset mozilla-foundation/common_voice_7_0 --config sv-SE --split test --log_outputs

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

1. Evaluation is done only on 10 examples [https://github.com/huggingface/transformers/blob/master/examples/research_projects/robust-speech-event/eval.py#L71 ](https://github.com/huggingface/transformers/blob/master/examples/research_projects/robust-speech-event/eval.py#L71 ) It should be done on entire dataset provided
2. To avoid any ambiguity WER and CER are generally reported in percentages but [https://github.com/huggingface/transformers/blob/master/examples/research_projects/robust-speech-event/eval.py#L26](https://github.com/huggingface/transformers/blob/master/examples/research_projects/robust-speech-event/eval.py#L26) will print the results in range of 0-1

Comments:
Thanks a lot for opening this issue @anuragshas! Indeed one should comment out this line when evaluating the model actually on the full dataset! I've left it here so that one can easily test the script. Will comment it out on master now :-)
Dynamically load model code from the Hub
Author: sgugger
Pull request: yes

# What does this PR do?

This PR allows a user to upload a modeling file along their model weights, and then lets the `Auto` classes load that model. The API is still a bit manual for now, it will be smoothed in follow-up PRs.

In terms of user of the model, the only thing to add is the flag `trust_remote_code=True` when using an auto class:
```
from transformers import AutoModel

model = AutoModel.from_pretrained("sgugger/test_dynamic_model", trust_remote_code=True)
```
This will load a `FakeModel` as defined in [this file](https://huggingface.co/sgugger/test_dynamic_model/blob/main/modeling.py).

For the person uploading the model there is a tiny bit more work. First save the code of the model (with all necessary imports) in a single python file, for instance `modeling.py`. In the rest of this PR description, I'll use MODELING_FILE to represent the name of that file without the suffix (so in my example of `modeling.py`, MODELING_FILE is "modeling"). That file needs to be in the repository you upload.

Second, add a filed `auto_map` to the configuration of the model you want to upload. This needs to be a dictionary with the name of auto classes as keys, and the full name of the corresponding modeling class as values. By full name, I mean MODELING_FILE.CLASS_NAME. For instance `modeling.FakeModel` for a class `FakeModel` defined in the `modeling.py` module. Here is an example:
```
config.auto_map = {"AutoModel": "modeling.FakeModel"}
```

This needs to be done before the model is saved, so that when doing the `model.save_pretrained` call, the config is saved with that field. Once this is done, push everything to the Hub, and you should be good!

Comments:
Forgive me for interjecting and asking, but does this allow custom modules to be uploaded to modelhub and loaded and run with the official Transformers library ? just to be sure.

If that's the case, do you plan to add some code scanning ? I think on some cases where this would be an security vulnerability for remote code execution

The PR requires an extra argument in the call to `from_pretrained`: `allow_custom_model=True` to execute the code from the hub. A user should not pass along that flag without having scanned the code of the repo to make sure they trust it.

Great, thanks for clearing this

hi, quick question about this PR:  

It lets people upload a custom model to the hub great, but it seems that in order to call that model using the API it still requires the model to be using one of the provided Pipelines?  Is there any way of also allowing a custom pipeline when calling the custom models? 

Custom pipelines is something we will add support for, it's on the roadmap, but for now you can't use the custom model in the API indeed.
How does decoder's weight shared with input embeddings?
Author: dancingpipi
Pull request: no

As the title, I haven't found the way to share input embeddings in self.decoder.weight.

Can someone help me point it out.?

```python
class BertLMPredictionHead(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.transform = BertPredictionHeadTransform(config)

        # The output weights are the same as the input embeddings, but there is
        # an output-only bias for each token.
        self.decoder = nn.Linear(config.hidden_size, config.vocab_size, bias=False)

        self.bias = nn.Parameter(torch.zeros(config.vocab_size))

        # Need a link between the two variables so that the bias is correctly resized with `resize_token_embeddings`
        self.decoder.bias = self.bias

    def forward(self, hidden_states):
        hidden_states = self.transform(hidden_states)
        hidden_states = self.decoder(hidden_states)
        return hidden_states
```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
How to implement huggingface BERT + CRF layer?
Author: lkqnaruto
Pull request: no

Hi

I wonder how to add CRF layer to a pretrained BERT model? Do I have to overwrite the loss function in the trainer? I really need some help on this, thank you.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Add support for `push_to_hub` for `AutoFeatureExtractor`
Author: philschmid
Pull request: no

# 🚀 Feature request

Add support for `push_to_hub` for `AutoFeatureExtractor.save_pretrained()`.

Hey, while doing some Speech tests I noticed that
`AutoFeatureExtractor.save_pretrained()` is not supporting `push_to_hub`.
https://github.com/huggingface/transformers/blob/46efc5802458e91a702528332d76d464061d201f/src/transformers/feature_extraction_utils.py#L194
compared to `AutoTokenizer`
https://github.com/huggingface/transformers/blob/0b8c84e110bf9012f30a85c40b9ff8ea868689fd/src/transformers/tokenization_utils_base.py#L1432




Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

unstale

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
pytorch NER example dataset deleted
Author: KentonMurray
Pull request: no

The file examples/pytorch/token-classification/run_ner.py is looking for a hardcoded conll2003 dataset.

In the hardcoded link, the following change removed the dataset:
https://github.com/davidsbatista/NER-datasets/commit/9d8f45cc7331569af8eb3422bbe1c97cbebd5690

To fix, the hardcoded link needs to be changed to a stable location or not be hardcoded.

Comments:
Closing this, as datasets has updated the URL (see https://github.com/huggingface/datasets/issues/3582).
Replace NystromformerTokenizer with AutoTokenizer
Author: novice03
Pull request: yes

# What does this PR do?

This PR replaces `NystromformerTokenizer` with `AutoTokenizer` in `modeling_nystromformer.py` since `NystromformerTokenizer` does not exist.

## Who can review?

@NielsRogge

Comments:
_The documentation is not available anymore as the PR was closed or merged._