noamaanMulla-03/transformers-issues huggingface dataset:
Is `guid` allowed to be None in `InputExample`?
Author: dnaaun
Pull request: no
Labels: name: wontfix

## Environment
-`transformers` version: 3.0.2
Omitted the rest because they most likely don't affect this issue.

## To reproduce

```py
from transformers.data.processors.utils import SingleSentenceClassificationProcessor

processor = SingleSentenceClassificationProcessor(labels=["lbl1", "lbl2"])
processor.add_examples(texts_or_text_and_labels=["example1", "example2"]) # There's a default ids=None
print(processor[0])
```
prints 
```InputExample(guid=None, text_a='example1', text_b=None, label=None)```

If `guid` is allowed to be None, that should be reflected in the type annotation(and documentation) of `InputExample`. If not, then `ids` should not be allowed to be `None`.

### Who can help 
@sgugger because it's a documentation issue, @thomwolf because of `git blame`.



Comments:
The guid is indeed optional, we can add this to the type annotation. We can't add the default `= None` however because it's before `text_a` in the dataclass, which is not optional.

Thanks! I'll probably shape up a bunch of type annotations into a PR sometime soon, so I'll make `guid` Optional(but without a default) in that PR if noone gets to it before me.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
tokenize cache for examples/language-modeling
Author: Jiaxin-Wen
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

I find transformer already has cache for tokenized result in examples/token-classification.
I think language-modeling which with a much larger dataset also need that 


Comments:
as it takes me about 7 minutes to tokenize a train set (size is 400w) every time I start training. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Feed to forward new parameters as computed manually by update rule
Author: meryemmhamdi1
Pull request: no
Labels: name: wontfix

Hi,

I would like to update the parameters of BertModel manually while training without using automatic optimizer to do it. I am trying meta-learning on BertModel and if I use automatic optimization on both inner and outer loops I get in-place operation errors. I also tried deepcopying but I cannot use deepcopy with DataParallel. So, I have opted for configuring my own manual optimization mechanism. how can I feed new parameters I computed from autograd of loss with respect to old parameters and update rule into the forward pass of BertModel manually? Is it a feature that exists? Otherwise is there a workaround?

Thanks,

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Loss becoming nearly zero in first 5K steps when training LM from scratch
Author: 008karan
Pull request: no
Labels: name: wontfix

I am training the ALBERT LM model from scratch.

I have already trained it for Hindi and Bangla and it was working fine but when I am training on Gujarati, the loss is becoming zero in 5K steps.

What could be the reason for the sudden drop in the loss? Can anyone suggest what could be the cause or how to debug such issue? 
Any suggestions?



Comments:
Hi! This is an interesting question, have you tried asking it over on the forums at https://discuss.huggingface.co ? You'll probably get more answers over there.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Update normalising method in oneshot classifier
Author: sachinruk
Pull request: yes

Hi, I know I raised this issue before here: https://github.com/huggingface/transformers/pull/5760#issuecomment-673840015 but I do think that it is worth taking a second look atleast.

So my main concern is, since we are dealing with logits, currently all that happens is to take the softmax over the entailment logits. This does guarantee summing to one, however, it doesn't care for the scale of the logits. For example suppose the logits for two possible classes for a given sentence came out as:
```
[[1000, 10, 10],
 [0.1, 0.1, 0.9]]
```
at the current method would give class1 the higher probability, even though, it can clearly be seen that class1 is a contradiction (since it is 100x contradiction in the log scale), and similarly class2 is clearly an entailment.

Now I do understand that this is an extreme case, but just to cover the bases, what I propose is that:
1. We softmax across every single sentence-class pair to get everything in the same scale.
2. Get a probability measurement over entailment by simply dividing by sum of probabilities. As seen [here](https://github.com/huggingface/transformers/pull/5760#issuecomment-673840015 ) the numbers that we get are different but only slightly for the example shown.

I do apologise about raising this again, just simply trying to help.

Comments:
Pinging @joeddav 

Hey @sachinruk, thanks for taking the time to contribute 🤗

I take your point and the case that you've described can certainly happen with the current method, but I maintain what I said in the other thread that there's not really a single correct way of doing this. We're highjacking the outputs of a model trained on a completely different distribution (NLI data) for own own purposes, so it's just a game of figuring out what makes intuitive sense and what empirically works the best.

I ran a quick benchmark on the AG's News topic classification dataset comparing the current method with the one you've proposed, and they performed similarly. I got a weighted F1 of ~70 with the entailment-only method used in the pipeline and around 68 with the method you've proposed.

If you can show that your proposed method empirically does significantly better, we could look into changing it or adding it as an additional kwarg to specify the method. But as is, I don't think it makes sense to change the behavior of the pipeline under people's feet.
the documents for transformer don't work
Author: dutyhong
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

##   i find the new document for transformer can't work, https://huggingface.co/transformers/v2.5.0/model_doc/bert.html#bertmodel  ; the class link don't work 
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
The link works, but the documentation was not properly done at this version. You should check a more recent version.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Adds license information for default and distilbert models
Author: ankane
Pull request: yes
Labels: name: model card

# What does this PR do?

Adds license information for default and `distilbert*` models. A follow up to #7668.

- Apache 2.0 for `distilbert*` based on https://github.com/huggingface/transformers/issues/3357#issuecomment-614856396
- MIT for `facebook/bart-large-mnli` based on https://github.com/huggingface/transformers/issues/7668#issuecomment-706064737

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), 
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case. #7668 
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? 

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

Model Cards: @julien-c

Comments:
Thanks!

Thanks @julien-c!
KeyError: 'squeezebert' which from model zoo
Author: ORlGlN
Pull request: no

This error occurs when using the pretrained model from https://huggingface.co/squeezebert/squeezebert-mnli

Comments:
Hi! We're still in the process of merging squeezebert. You can see the development [here](https://github.com/huggingface/transformers/pull/7083).

Maybe try again now? SqueezeBERT support was recently merged into the master branch.
Fix docstring in AutoModel class
Author: av-maslov
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes doc string for the class AutoModel

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), 
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? 


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

 @sgugger

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

Comments:
Thanks!
frequent checkpoints have worse performance
Author: wyin-Salesforce
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

Hi All, I often notice an issue when training a model and evaluate on dev set.
Usually we may evaluate on the dev after each epoch, let's call this as setting A;
But we often want to check the system more often on the dev set, so we may evaluate for example 1/5 epoch; let's call this as setting B.

What I noticed is that A and B will get totally different performance in the end.  Since B checks more often, I supposed that B can get the same or at least very close performance with A then evaluate at 5/5 of the training set, 10/5 of the training set, etc. But they are very different. For example, when I train textual entailment model on RTE dataset, A can give me about 86% accuracy on dev, but B can only give about 80%.

What's the issue here? thanks


## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
hi, can you post the link on stackoverflow

Btw, I also face this issue when working with an RTE dataset and have raised an issue here..[https://github.com/huggingface/transformers/issues/5863](url). My dev values after each epoch don't match up when the total number of epoch changes.  Now its making me wonder if its RTE specific. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
examples/seq2seq/test_bash_script.py covers summarization
Author: sshleifer
Pull request: no
Labels: name: Help wanted

name: wontfix

name: Tests


Comments:
Hey @sshleifer, I would like to take this issue, if there no objection

Absolutely!


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
mismatch keys of glue tasks
Author: xujiaze13
Pull request: no
Labels: name: wontfix


```
>>> from transformers import glue_output_modes, glue_processors, glue_tasks_num_labels, glue_compute_metrics``

>>> glue_tasks_num_labels.keys()
dict_keys(['cola', 'mnli', 'mrpc', 'sst-2', 'sts-b', 'qqp', 'qnli', 'rte', 'wnli'])

>>> glue_output_modes.keys()
dict_keys(['cola', 'mnli', 'mnli-mm', 'mrpc', 'sst-2', 'sts-b', 'qqp', 'qnli', 'rte', 'wnli'])

>>> glue_processors.keys()
dict_keys(['cola', 'mnli', 'mnli-mm', 'mrpc', 'sst-2', 'sts-b', 'qqp', 'qnli', 'rte', 'wnli'])
```

and ``glue_compute_metrics`` has ``task_name`` restrictions:

```['cola', 'sst-2', 'mrpc',  'sts-b', 'qqp', 'mnli', 'mnli-mm',  'qnli', 'rte', 'wnli', 'hans']```

Why are these task names inconsistent?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Implement DeLighT: Very Deep and Light-weight Transformers
Author: bratao
Pull request: no
Labels: name: wontfix

name: New model

# 🌟 New model addition

## Model description
DeLight, that delivers similar or better performance than transformer-based models with significantly fewer parameters. DeLighT more efficiently allocates parameters both (1) within each Transformer block using DExTra, a deep and light-weight transformation and (2) across blocks using block-wise scaling, that allows for shallower and narrower DeLighT blocks near the input and wider and deeper DeLighT blocks near the output. Overall, DeLighT networks are 2.5 to 4 times deeper than standard transformer models and yet have fewer parameters and operation
https://arxiv.org/pdf/2008.00623.pdf

<!-- Important information -->

## Open source status

* [X] the model implementation is available: (give details)

It is available at https://github.com/sacmehta/delight

* [ ] the model weights are available: (give details)
* [X] who are the authors: (mention them, if possible by @gh-username)
@sacmehta

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Tips for Tensorflow 2.0 NER task by using fit method.
Author: rainmaker712
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

Any tips for implementing TF2 NER (TF Token Classifier) by using model.fit method?

An official example does not use model.fit for NER task.

Any links or tips will be appreciated.


<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
@aiscientist 
Why would you need to use `model.fit()` method? I think it is a more low level way to train your model. There's an easy way to train HuggingFace models with `TFTrainer`. All you need to do, is to initialize pretrained config, tokenizer and model, create your own Dataset class for NER data, and just pass it to the trainer. It will do all the rest.

> @aiscientist
> Why would you need to use `model.fit()` method? I think it is a more low level way to train your model. There's an easy way to train HuggingFace models with `TFTrainer`. All you need to do, is to initialize pretrained config, tokenizer and model, create your own Dataset class for NER data, and just pass it to the trainer. It will do all the rest.

Yes, I need to finish this job using model.fit() following from the TensorFlow guideline (Like Keras Style), but for NER task it's kinda hard to manage loss because of model.fit().  Using Pytorch is easier but, I need to use model.fit for this task because of several reasons. I just want to know that it is possible to use TF Token Classifier on model.fit(). If not, I need to discuss with my co-workers and find another way to do it.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Decode [UNK] from tokenizer
Author: RodSernaPerez
Pull request: no
Labels: name: wontfix

I am using Bert for NER and I have a problem when the tokenizer uses the token [UNK] to mask a word. How can I easily know which was the original work given its index?

Comments:
@RodSernaPerez I am not sure what you are doing, but the model will do that if it doesn't find your word in the vocab.txt file. You have to use `BertTokenizer` before training, so that it splits all of the words that are not in the vocab file into word tokens. I think it will mark words that it wasn't able to tokenize with `[UNK]`. As to your question, what do you mean by index? I am assuming you know where the `[UNK]` was in white space tokenized word. You can concatenate BERT word tokens first and then you can try to find word marked with `[UNK]` in that list of concatenated tokens.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Cannot preprocess WNUT'17 dataset for token-classification
Author: kushalj001
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):BERT

Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [x] the official example scripts: I am trying to run the run_ner.py script for the WNUT’17 dataset. I followed the preprocessing steps mentioned in the README. I just downloaded the dev dataset. The command  
`python3 scripts/preprocess.py data_wnut_17/dev.txt.tmp $BERT_MODEL $MAX_LENGTH > data_wnut_17/dev.txt`   
does not work and my terminal hangs.
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: Token Classification
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Move into the `token-classification` folder in the `examples` directory.
2. Run the commands for the WNUT’17 dataset as mentioned in the README. 
```
mkdir -p data_wnut_17
curl -L 'https://github.com/leondz/emerging_entities_17/raw/master/emerging.dev.conll' | tr '\t' ' ' > data_wnut_17/dev.txt.tmp
export MAX_LENGTH=128
export BERT_MODEL=bert-large-cased
```
3. The terminal hangs on the next command
`python3 scripts/preprocess.py data_wnut_17/dev.txt.tmp $BERT_MODEL $MAX_LENGTH > data_wnut_17/dev.txt`   


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I would expect the `preprocess.py` script to execute normally and return the control of the terminal back to me. Once hung, I fail to regain control of my command-line even by `ctrl-d/z/c`. 
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: Ubuntu 18.04
- Python version: 3.7.8
- PyTorch version (GPU?): 1.5 (Yes)
- Tensorflow version (GPU?): - 
- Using GPU in script?: The error does not involve GPU usage in the script.
- Using distributed or parallel set-up in script?: No


Comments:
@kushalj001 It may take a while to preprocess the whole dataset. Not sure if you waited long enough. Check if your have enough RAM when you are doing this since if the dataset is too large you might run out of memory.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.