noamaanMulla-03/transformers-issues huggingface dataset:
Fix SqueezeBERT for masked language model
Author: forresti
Pull request: yes
Labels: name: model card

# What does this PR do?
This corrects a mistake in the implementation of SqueezeBertForMaskedLM.

Fixes #8277


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Yes: https://github.com/huggingface/transformers/issues/8277
- [x] Did you make sure to update the documentation with your changes? Here are the
- [ ] Did you write any new necessary tests? _No tests added._


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

@sgugger @LysandreJik @ontocord 


Comments:
Fix logging in the examples
Author: jplu
Pull request: yes

# What does this PR do?

This PR updates all the examples to use the Transformers logging util. Before no logs was displayed.

Comments:
The five examples scripts mentioned above all already use the huggingface logger, and already set it to the correct verbosity level. Seeing as the example scripts are *examples*, they really should be as straightforward as possible, and should not repeat unnecessary statements.

Please update the following files: `run_clm.py`, `run_mlm.py`, `run_glue.py`, `run_mlm_wwm.py`, `run_plm.py` so that they do not have useless statements.

@sgugger pointed you towards the correct area, you should be able to find it if you look for "logging".

I don't see the import neither the usage. Can you link it please?

https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_clm.py#L170

Ok got it I was looking at my own updated file ahah.

It is also in your file, below https://github.com/huggingface/transformers/blob/d410b83111238f7b949b7c9c6a4c3f689d29519b/examples/language-modeling/run_clm.py#L176

Ok I have removed the duplicate import and add only the two missing calls.

Already done the changes 👍 

No, the changes are not done:
1. in the scripts like the new `run_glue.py`, the code for logging is duplicated and executed once for every process, then once for the main process.
2. in all the other scripts, the code is executed on all processes.

This is also added in util files or test files where it's just not necessary.

I really don't get what you mean, sorry. What are the exact changes I have to make for each file?

@sgugger Please check the last commit and let me know if it was what you meant. Otherwise, please, can you be more specific in what you want me to change.
Fix check scripts for Windows
Author: jplu
Pull request: yes

# What does this PR do?

The current check-X scripts are reading/writing with `os.linesep` as the newline separator. On Windows it makes the overwritten files in CRLF instead of LF. Same logic is applied on Mac with CR. Now, Python will always use LF to read and write in the files.

Comments:
It doesn't make any change on Linux, and you've tested it on Windows. Could we get someone using MacOS to double-check it doesn't break anything for them before merging?

I think @LysandreJik is on MacOS?

I'm actually between Manjaro and EndeavourOS, but I'll check on a Mac.
Prediction loop: work with batches of variable length (fixed per batch)
Author: JulesGM
Pull request: yes

In the current form, the `prediction_loop` doesn't handle batches with samples of varying lengths (but fixed length per batch). This patch adds this capability.
This is great because it can save a lot of time during training and inference, where using the full length every time is a big sacrifice, knowing self-attention scales in n**2.
Disclaimer: This is a strictly personal contribution, not linked to my professional affiliation in any way.

@sgugger 

https://github.com/huggingface/transformers/issues/8483


Comments:
Please feel free to modify this in any shape or form of course.

This will cause a regression for code expecting a NumPy output, obviously :/

I don't have much time, if any, to dedicate to this. If you're not interested by the idea, it's completely fine, I will close.

I think @jplu is redesigning the TFTrainer. Maybe this should be reopened once that design has been merged in master?

This won't be compliant anymore because the redisign doesn't use custom loops.

Do you support different batch lengths in the new one? @jplu 

It is not on top of the list but, yes for sure, we plan to support it, including for training.
config.attention_head_size for structured pruning out-of-box
Author: dsindex
Pull request: no

# 🚀 Feature request

## Motivation

for structured pruning like `fastformers`, https://github.com/microsoft/fastformers#pruning-models ,

we should modify the source code of transformers for `attention_head_size.

for example,

1. configuration_bert.py
https://github.com/microsoft/fastformers/blob/main/src/transformers/configuration_bert.py#L128

2. modeling_bert.py
https://github.com/microsoft/fastformers/blob/main/src/transformers/modeling_bert.py#L192
https://github.com/microsoft/fastformers/blob/main/src/transformers/modeling_bert.py#L263

is it possible to set `attention_head_size` from outside(config.json) ?

## Your contribution



Comments:
@dsindex  FYI, I am working on creating a PR including this feature for the effort of #8083 .

@ykim362 great!  closing issue here.
module 'torch.nn' has no attribute 'Identity'
Author: ahlawatankit
Pull request: no

Traceback (most recent call last):
  File "trainer.py", line 17, in <module>
    model = XLMForSequenceClassification(config)
  File "/home/ankit/anaconda3/lib/python3.6/site-packages/pytorch_transformers/modeling_xlm.py", line 823, in __init__
    self.sequence_summary = SequenceSummary(config)
  File "/home/ankit/anaconda3/lib/python3.6/site-packages/pytorch_transformers/modeling_utils.py", line 734, in __init__
    self.summary = nn.Identity()
AttributeError: module 'torch.nn' has no attribute 'Identity'



https://github.com/huggingface/pytorch-transformers/blob/2f869dc6651f9cf9253f4c5a43279027a0eccfc5/pytorch_transformers/modeling_utils.py#L734

Comments:
This was added in PyTorch 1.1.0 (see [changelog here](https://github.com/pytorch/pytorch/tree/v1.1.0) :)

So I guess you just have to update your PyTorch version!

Oh yes, I guess we can add a replacement to keep older PyTorch compatibility.
Would be sad to lose backward compatibility just for this.

Is this replacement added in any of the newer versions?
could you please give me a torch example of xlm-roberta-(base/large) for multilingual-text question?
Author: wmathor
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
I read the document of [XLM-RoBERTa](https://huggingface.co/transformers/model_doc/xlmroberta.html#overview), but i also have a lots of question. For example, `should be able to determine the correct language from the input ids.` how to determine?

If you have a example for how to use xlm-roberta about multilingual text question, please show me, Thank you very much!



Comments:
Adding Confusion matrix support in Trainer
Author: prajjwal1
Pull request: no

I want to add confusion matrix support in Trainer. It would be a useful addition. The only dependency would be `sklearn` which this library already uses for metrics. It can allow users to better understand about predictions coming from model. @sgugger Let me know if this is something you want to add.







Comments:
Hi there! We love contributions, but metrics should now be implemented directly in the datasets library, not inside Transformers. So you should check there if it does not already exist, and if not, suggest a Pr on that repository :-)

Okay then, I will close it.
SWA
Author: josh-cooper
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


Comments:
Tagged versions of model in new model hub don't work
Author: brandenchan
Pull request: no

Our model, deepset/roberta-base-squad2 was originally uploaded under the old style model hub.

I have committed a new version of the deepset/roberta-base-squad2 model onto the model hub using the new git based system introduced in transformers 3.5.0. I have 2 tags (v1.0 and v2.0) that I have also pushed to the repo. The tags show up in the model hub drop down but when I click on either of the tags, it says "Not Found: Error: Invalid rev id". 

It seems I cannot load the models when I specify `revision=v1.0` or `revision=v2.0`. If I don't specify a revision, it seems to load a model though I'm not sure which. This is the code I used:
```
tokenizer = AutoTokenizer.from_pretrained(
  "deepset/roberta-base-squad2",
  revision="v2.0" # tag name, or branch name, or commit hash
)
```

What steps can I take so that I can access both versions through the model hub website, and by specifying name and revision?

Thanks,
Branden

 @julien-c

Comments:
Looking into it right now (cc @Pierrci)

looks like we were supporting lightweight tags but not annotated ones. 

Should be fixed: https://huggingface.co/deepset/roberta-base-squad2/tree/v2.0

That fixed the problem! Thanks very much

Thanks, and please keep the feedback coming! <3
[T5] Unused `n_positions` and `max_position_embeddings`.
Author: patrickvonplaten
Pull request: no

The T5Config has the parameter `n_positions` set to 512 and `max_position_embeddigs` referring to `n_positions`. However, neither `max_position_embeddigs` nor `n_positions` is used in the `T5Model` and T5 is not limited to `max_position_embeddings`. *E.g.*:

```python
from transformers import T5Model
model = T5Model.from_pretrained("t5-small")

model.config.max_position_embeddings # shows 512

input_ids = torch.tensor([600 * [0]])  # input of size > 512

model(input_ids, decoder_input_ids=input_ids)  # works fine
```

 I think we should delete the parameter. 

@thomwolf - do you remember why we added `max_position_embeddigs` and `n_positions` to T5? The model does not seem to use these params and also should not be limited to 512 due to its relative position embeddings.

Comments:
Model templates encoder only
Author: LysandreJik
Pull request: yes

Only merge the encoder part, not the encoder-decoder part of #7636. 

Will work on a decoder in the future.

Applied your comments @sgugger @patrickvonplaten, but opened a new PR on a new branch so that we can keep the old one for reference when integrating the encoder-decoder model.

Comments:
Thank you both for your reviews!
Using Pretrained BERT model to add additional words that are not recognized by the model
Author: geo47
Pull request: no

Hello,

I want some help regarding adding additional words in the existing BERT model. I have two quires kindly guide me:

I am working on NER task for a domain:

There are few words (not sure the exact numbers) that BERT recognized as [UNK], but those entities are required for the model to recognize. The pretrained model learns well (up to 80%) accuracy on "bert-base-cased" while providing labeled data and fine-tune the model but intuitively the model will learn better if it recognize all the entities.

1. Do i need to add those unknown entities in vocabs.txt and train the model again?


2. Do i need to train the BERT model on my data from Scratch?

Thanks... 


Comments:
You don't need to do either of those things! You should add tokens to your tokenizer by leveraging the `add_tokens()` method, and then resize your model's embedding matrix.

Then, you should train your model on a dataset that has those entities so that it understands the meaning of these entities. It seems to be what you're doing here, so just make sure to add the tokens to your tokenizer first. You can see the doc about it [here](https://huggingface.co/transformers/internal/tokenization_utils.html?highlight=add_tokens#transformers.tokenization_utils_base.SpecialTokensMixin.add_tokens).

Also, we try to keep the github issues only for bugs and feature requests. Please ask questions/discussions on the [instead](https://discuss.huggingface.co). Thanks!
[WIP] Added token_type_id support to GPT2Model
Author: StuartMesham
Pull request: yes
Labels: name: wontfix

Fixes #6794 #4922 #1339

Added token type embedding support to GPT2Model.

To use `token_type_ids` with `GPT2Model` the user can now supply `type_vocab_size` to the `GPT2Config` constructor and then pass `token_type_ids` as input to the `forward` method as with other models.

Using all the default parameters, the model should continue to work as before, since I have set `type_vocab_size=None` as the default of the `GPT2Config` class, which disables my token type embeddings.

If the user tries to pass `token_type_ids` to the `forward` method when token type embeddings have not been enabled for a model, it will raise an informative error message. The same is true for when the user does not pass `token_type_ids` to the `forward` method when token type embeddings have been enabled for a model.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Incorrect loss calculation for the last batch in TFTrainer if dataloader_drop_last is False
Author: ydshieh
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.1.0
- Platform: Linux-4.15.0-115-generic-x86_64-with-debian-buster-sid
- Python version: 3.6.7
- PyTorch version (GPU?): 1.5.1+cpu (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 -->
 tensorflow: @jplu 


## Description

In [training_step()](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_tf.py#L595) in `trainer_tf.py`, we have

    scaled_loss = per_example_loss / self.total_train_batch_size

However, if `dataloader_drop_last=False`, the last bacth (before being distributed to replicas) won't necessary have `self.total_train_batch_size` examples. If we allow `dataloader_drop_last=False`, we need a way to dynamically calculate the actual number of examples in a global batch, and pass this information in some way to the replicas.


Comments:
Thanks @chiapas! Indeed if the batch size becomes lower for the last step we divide by a wrong number. I will investigate this to better handle this edge case.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Enhance a MarianMT pretrained model from HuggingFace with more training data
Author: stelmath
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**: https://stackoverflow.com/questions/63774619/enhance-a-marianmt-pretrained-model-from-huggingface-with-more-training-data

Comments:
Have you tried the finetune.sh script shown [here](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/finetune.sh)? In addition to the short list of CLI flags listed there, you could try adding:

```
--src_lang "en" \
--tgt_lang "de" \
--num_train_epochs 400 \
--warmup_steps 20 \
--train_batch_size 32 \
--eval_batch_size 32 \
--data_dir "/data/dir" \
--output_dir "/path/to/store/model/etc" \
--cache_dir "/path/for/misc/files" \
--max_source_length 128 \
--max_target_length 128 \
--val_max_target_length 128 \
--test_max_target_length 128 \
--model_name_or_path "</path/to/pretrained>"
```

where the "/path/to/pretrained" could be either a local path on your machine or MarianMT model (Opus-en-de or equivalent). The "data/dir" has a "train.source" and "train.target" for the source & target languages, such that line number x of the target is a translation of line x in the source (and same with "val.source" and "val.target"). I have changed the finetune.py script [here](https://github.com/huggingface/transformers/blob/77cd0e13d2d09f60d2f6d8fb8b08f493d7ca51fe/examples/seq2seq/finetune.py#L415) to 
```
parser = TranslationModule.add_model_specific_args(parser, os.getcwd())

```
and then ran the finetune.sh script.


Note: The gradients blew up when I used the "fp16" flag (with Pytorch 1.6), so I had removed it. Also, you might want to check on the "val_check_interval", "check_val_every_n_epoch", and probably check [this issue](https://github.com/huggingface/transformers/issues/3447) on how to save multiple checkpoints.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Load BERT+GPT2 in EncoderDecoder
Author: AmbiTyga
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
I am working on modelling an EncoderDecoderModel using weights of BERT and GPT2, after going through lots of repo and issues I found that currently its not possible, but I found a model card that has used this model of BERT+GPT2 on dataset cnn-dailymail [here](https://huggingface.co/patrickvonplaten/bert2gpt2-cnn_dailymail-fp16). I would like to know in which version of transformers is that possible and one thing more that there was an attribute passed in `TrainingArguments` module, that was `predict_from_generate`, I can't find that in `transformers`: 3.1.0, 3.0.2, 2.11.0, please clear me in which version does these parameters constitute.

@patrickvonplaten Please answer my query
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->

Comments:
Hi @AmbiTyga ,
Bert2GPT2 is available in the latest release, however `predict_from_generate` from generate is not yet added in `Trainer`. 
You can set `predict_from_generate` to `False`  and `comput_metrics` to `None` if you done't need generative metrics (ROUGE etc) at training time. 

If you want to use `predict_from_generate` from generate then you'll need to install transformers from this fork.
https://github.com/huggingface/transformers/tree/more_general_trainer_metric

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.