noamaanMulla-03/transformers-issues huggingface dataset:
How can I use a TensorFlow 2.0 model for Named-Entity-Recognition (NER)? (using TFBertForTokenClassification )
Author: romulocosta100
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
How can I use a TensorFlow 2.0 model for Named-Entity-Recognition (NER)? (using TFBertForTokenClassification )

Comments:
Exactly the same question here! Can someone please provide us with a small tutorial or even some general guidelines?

Any response here? I was looking for something similar

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Update setup.py
Author: singhanandVEVO
Pull request: yes

changed update setup file

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1610?src=pr&el=h1) Report
> Merging [#1610](https://codecov.io/gh/huggingface/transformers/pull/1610?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ef1b8b2ae5ad1057154a126879f7eb8de685f862?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1610/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1610?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1610   +/-   ##
=======================================
  Coverage   86.17%   86.17%           
=======================================
  Files          91       91           
  Lines       13595    13595           
=======================================
  Hits        11715    11715           
  Misses       1880     1880
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1610?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1610?src=pr&el=footer). Last update [ef1b8b2...2248c6b](https://codecov.io/gh/huggingface/transformers/pull/1610?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Why should we change this?
solves several bugs in the summarization codes 
Author: rabeehk
Pull request: yes

Hi,

This pull requests solves several small bugs in this summarization script:

- call the evaluation code
- fix the iterating over batch in eval part 
- create the folders before saving encoder/decoder to avoid the crash 
- first creates output_dir, then use it during the saving of the model part.
- tokenizer.add_special_tokens_single_sequence does not exist anymore, changed to tokenizer.build_inputs_with_special_tokens
- fix the empty checkpoints in the evaluation 
- add evaluate=False in load_and_cache_examples functions, to fix the issue for now, but dataset needs to be split in train/validation 
- add missing argument of per_gpu_eval_batch_size 

the code needs more features to be added/bugs to be resolved, this just solves the obvious existing bugs, please see my opened bug request #1674 from yesterday on remaining issues. 





Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1694?src=pr&el=h1) Report
> Merging [#1694](https://codecov.io/gh/huggingface/transformers/pull/1694?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/93d2fff0716d83df168ca0686d16bc4cd7ccb366?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1694/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1694?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1694   +/-   ##
=======================================
  Coverage   85.14%   85.14%           
=======================================
  Files          94       94           
  Lines       13920    13920           
=======================================
  Hits        11852    11852           
  Misses       2068     2068
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1694?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1694?src=pr&el=footer). Last update [93d2fff...5b50eec](https://codecov.io/gh/huggingface/transformers/pull/1694?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thank you for your PR. As I said in the issue, we’re still working on the summarization; you can follow the changes in the ˋexample-summarization` branch.
fix: wrong architecture count in README
Author: gthb
Pull request: yes

Just say “the following” so that this intro doesn't so easily fall out of date :)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2203?src=pr&el=h1) Report
> Merging [#2203](https://codecov.io/gh/huggingface/transformers/pull/2203?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/94c99db34cf9074a212c36554fb925c513d70ab1?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2203/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2203?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2203   +/-   ##
=======================================
  Coverage   81.47%   81.47%           
=======================================
  Files         122      122           
  Lines       18342    18342           
=======================================
  Hits        14945    14945           
  Misses       3397     3397
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2203?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2203?src=pr&el=footer). Last update [94c99db...a297846](https://codecov.io/gh/huggingface/transformers/pull/2203?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This is a good idea :) @thomwolf
Question on AllenNLP vocabulary and huggingface BERT out of sync
Author: pruksmhc
Pull request: no
Labels: name: wontfix

Perhaps this should also be posted in the allennlp repo. 
I'm currently trying to use a pretrained model (clinicalBERT) with a different set of vocabulary with huggingface's BertModel. Even though the code runs, I'm not 100% convinced that the vocabulary index to weight mappings are synced between the allennlp vocabulalry object and the BertModel. The tokenizer used for clinicalBERT is scispacy, not wordpiece. First of all, are there any examples in this repo or other places online that does this (has a stack of allennlp vocab + huggingface BERT, and loading a pretrained BERT model with vocab.txt?).

This is what I do:
` model = pytorch_transformers.BertModel.from_pretrained("my/path/to/clinicalBERT", output_hidden_states=True)`
Then, for the vocabulary, I do: 
    ` vocab = Vocabulary(counter=None, max_vocab_size=max_v_sizes) ` 
    ` vocab.set_from_file(filename="path/to/clinicalBERT/vocab.txt", is_padded=1, namespace="scispacy")` 
I then use resize_tokens on the model, and use a TokenIndexer from allennlp and index my instances with that vocabulary. Even at this point though, ` vocab.get_token_from_index(0, "scispacy")` 
 returns ` @@PADDING@@` , and ` @@UNKNOWN@@`  is at index 101 for the vocab (even though allennlp's vocabulary to my understanding sets ` @@UNKNOWN@@`  to 1).  
What should I do to realign the Vocabulary back to the vocab index -> weight embedding mapping in the model?


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Why the codes of training BERT from scratch are deprecated 
Author: anhnt170489
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
I'm wondering why the team removed the codes of training BERT from scratch, including pregenerate_training_data.py and finetune_on_pregenerated.py. They're very helpful and I still continue developing them to train the BERT as well as Roberta from scratch. 

Comments:
They were community provided and the core team didn't have the bandwidth to maintain them.

Also we want to limit the number of single-model examples now and favor examples that work for a range of models.

If you want to update them to the current version of the repo and add the various models (for instance all the models currently in `run_lm_finetuning`), happy to review a PR.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Error when creating RobertTokenizer for distilroberta-base
Author: fhamborg
Pull request: no

## 🐛 Bug

Model I am using (Bert, XLNet....): DistilRoberta

Language I am using the model on (English, Chinese....): EN

The problem arise when using the official example scripts: https://github.com/huggingface/transformers/tree/master/examples/distillation

## To Reproduce
```
RobertaTokenizer.from_pretrained('distilroberta-base')
```

this will yield an error:
```
OSError: Model name 'distilroberta-base' was not found in tokenizers model name list (roberta-base, roberta-large, roberta-large-mnli). We assumed 'distilroberta-base' was a path or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url.
```

## Expected behavior
should produce a RobertaTokenizer instance, which can also be used for `distilroberta-base`. 

## Environment

* OS: MacOS
* Python version: 3.7
* PyTorch version: 1.3.0
* PyTorch Transformers version (or branch):  2.1.1
* Using GPU ? no


Comments:
It's not in a pip released version yet so you need to pull from master if you want to use it for now.

We'll do a release soon.

Thanks for the info. Do you have an estimation when that pip release would be, @julien-c ?

Reviving this thread. I just cloned 2.2.2 from the master and updated `transformers`. `distilroberta-base` is still not available.  Am I missing something? Thanks, you all!..

> OSError: Model name 'distilroberta-base' was not found in model name list (distilbert-base-uncased, distilbert-base-uncased-distilled-squad, distilbert-base-german-cased, distilbert-base-multilingual-cased)
  

What are the exact commands you typed @oersoy1? 

@julien-c OK, I found out what is happening and will document here just in case someone else falls into the same trap. I wrote my custom script similar to run_glue.py.  I was passing `distilbert` to the `args.model_type` argument and the `model_name` got `distilroberta-base` assigned.  I assumed that it was a subclass of `distilbert` models so the type intuitively looked as if it should have been distilbert. [This list](https://huggingface.co/transformers/pretrained_models.html) certainly gives me that impression.

However, looking at the distillation [examples](https://github.com/huggingface/transformers/tree/master/examples/distillation), I realized the model type needs to be `roberta` not `distilbert`. 

It is a little bit confusing but regardless, I got `distilroberta-base` working and it gave me great results. 

Thanks a lot!

Ah, yeah, you are correct. Moved the model shortcut in the table in ac1b449


No worries. I see that you have corrected the pretrained models list and moved `distilroberta-base` under `roberta` which was the main problem for me. Updating all documentation when you make the changes could be difficult, especially when the gatekeepers for a specific document is different than the ones making the change.
Extract features aligned to tokens from a BertForQuestionAnswering model
Author: Luvata
Pull request: no

## ❓ Questions & Help
I have a finetuned Bert model on a custom Question Answering task (follow original Tensorflow source), I've successful converted this model and loaded it with `BertForQuestionAnswering`
<!-- A clear and concise description of the question. -->
My question is how to get feature embedding aligned to each token from a pretrained `BertForQuestionAnswering`


Comments:
>My question is how to get feature embedding aligned to each token from a pretrained BertForQuestionAnswering

Could you elaborate on what this means? I am currently working with BertForQuestionAnswering but haven't encountered this area before.
The code used to be clean...
Author: Borororo
Pull request: no

Hi, thanks to all contributors!
From my point of view, recent changes in examples codes, especially run_squad script, are kinda confusing.

May I know what is the reason that we delete old utils_squad.py and store all data processing and evaluating scripts under the transformer/data folder? This makes code look quite messy. Particularly, the example and features classes have been changed a lot.

I was reading the old version scripts for a long long time, and I think it was easy for me to change any components in the whole structure, no matter data preprocessing or introducing new LM.  I can easily change the input format of examples, so I could run codes on other datasets with few lines. 

I believe many people have already made task-specific changes on their own codes,  and sometimes some people just want to run codes on new released LMs for testing, so they come here and check. If luckily new LM is available, we would prefer update code by hands instead of the direct downloading whole package again.

But anyway, if changes are compulsory, I am definitely willing to go through it deeply.

Comments:
Updating code by hand seems a very messy way of working. Can't you just make a copy of the (old version of the) examples directory, and make your changes there - regardless of the current installation of  `transformers` itself?
why xlnet requires a long prompt for short inputs while Bert does not ? 
Author: muiPomeranian
Pull request: no
Labels: name: wontfix

 hey guys,

Q1)
can someone give some more insight what @thomwolf explaining about?
'''
#846
The main reason you get bad performance is that XLNet is not good on short inputs (comes from the way it is pretrained, always having a long memory and only guessing a few words in the sequence).

The run_generation example here will show you how to get better performances by adding a random text as initiator.
Aman Rusia also wrote a blog post about that here. We are using his solution in the run_generation example.
'''
I can't understand the difference the way both Bert and XLnetLM works for LMhead task.
Aren't both model having disadvantages if they have short sentence?

It seems he said XLnet has huge disadvantage on short input sentence
while Bert does not(or has less disadvantage). Any detail explanation could be useful !

Q2)
Also, I can't get the point of adding extra padding or adding random padding things to improve XLnetLMHead model. Any snippet or explanation could be appreciated too...(saw the link but could not fully understood). I experimented by just adding extra strings of line:'I believe my sister is because she is a blonde ' + ' ' and it gives much better result than not having at the end....

Q3)
#846 (comment)
Lastly, why do we have better result when we don't use perm_mask ? above link response shows that
not having perm_mask option does give at least better result...But isn't perm_mask supposed to help to get better prediction and what author of paper used for SOTA ?

isn't perm_mask allow model to not seeing the next tokens in the given input while can see the previous tokens? According to the paper and the original code, I could see that if permute order is 3->4->1->2, mask=1,3, then model cannot see masked<1> when it tried to predict masked<3> but the reverse is possible.

Many thanks in advance !

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Bad F1 Score for run_squad.py on SQuAD2.0
Author: WenTingTseng
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
When I run the run_squad.py for SQuAD2.0 like this

python3 run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-cased \
  --do_train \
  --do_eval \
  --train_file /share/nas165/Wendy/transformers/examples/tests_samples/SQUAD/train-v2.0.json \
  --predict_file /share/nas165/Wendy/transformers/examples/tests_samples/SQUAD/dev-v2.0.json \
  --per_gpu_train_batch_size 4 \
  --learning_rate 4e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /share/nas165/Wendy/transformers/examples/SQuAD2.0_debug_bert/
  --version_2_with_negative=True \
  --null_score_diff_threshold=-1.967471694946289

It runs very fast and the f1 score only 7.9% What is wrong with it
The log message like this
<img width="960" alt="擷取" src="https://user-images.githubusercontent.com/32416416/71322341-939daa80-2501-11ea-9313-e179d1760b99.PNG">
Thanks a lot for your help.By the way I clone it today so it is the new version


Comments:
Improve repository structure
Author: aaugustin
Pull request: yes

This PR builds on top of #2255 (which should be merged first).

Since it changes the location of the source code, once it's merged, contributors must update their local development environment with:

        $ pip uninstall transformers
        $ pip install -e .

I'll clarify this when I update the contributor documentation (later).

I checked that:

- `python setup.py sdist` packages the right files (only from `src`)
- I didn't lose any tests — the baseline for `run_tests_py3_torch_and_tf` is `691 passed, 68 skipped, 50 warnings`, see [here](https://app.circleci.com/jobs/github/huggingface/transformers/10684))


Comments:
Last test run failed only because of a flaky test — this is #2240.
Fix doc link in README
Author: upura
Pull request: yes

close https://github.com/huggingface/transformers/issues/2252

- [x] Update `.circleci/deploy.sh`
- [x] Update `deploy_multi_version_doc.sh`

Set commit hash before "Release: v2.3.0".


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2264?src=pr&el=h1) Report
> Merging [#2264](https://codecov.io/gh/huggingface/transformers/pull/2264?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/645713e2cb8307e41febb2b7c9f6036f6645efce?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2264/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2264?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2264   +/-   ##
=======================================
  Coverage   78.35%   78.35%           
=======================================
  Files         133      133           
  Lines       19878    19878           
=======================================
  Hits        15576    15576           
  Misses       4302     4302
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2264?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2264?src=pr&el=footer). Last update [645713e...9d00f78](https://codecov.io/gh/huggingface/transformers/pull/2264?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks @upura!
Loading sciBERT failed
Author: broken-dream
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I am trying to compare the effect of different pre-trained models on RE, the code to load bert is:
`self.bert = BertModel.from_pretrained(pretrain_path)`

When the "pretrain_path" is "pretrain/bert-base-uncased" , everything is fine, but after i changed it to "pretrain/scibert-uncased", i got error:

`-OSError: Model name 'pretrain/scibert-uncased' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased). We assumed 'pretrain/scibert-uncased/config.json' was a path or url to a configuration file named config.json or a directory containing such a file but couldn't find any such file at this path or url.`

The scibert is pytorch model and the two directories are same in structure. 
It seems that if the model name is not in the name list, it won't work.

Thank you very much!

Comments:
Try to use the following commands:

```bash
$ wget "https://s3-us-west-2.amazonaws.com/ai2-s2-research/scibert/huggingface_pytorch/scibert_scivocab_uncased.tar"
$ tar -xf scibert_scivocab_uncased.tar

The sciBERT model is now extracted and located under: `./scibert_scivocab_uncased`.

To load it:

```python
from transformers import BertModel

model = BertModel.from_pretrained("./scibert_scivocab_uncased")
model.eval()
```

This should work 🤗

> Try to use the following commands:
> 
> ```shell
> $ wget "https://s3-us-west-2.amazonaws.com/ai2-s2-research/scibert/huggingface_pytorch/scibert_scivocab_uncased.tar"
> $ tar -xf scibert_scivocab_uncased.tar
> 
> The sciBERT model is now extracted and located under: `./scibert_scivocab_uncased`.
> 
> To load it:
> 
> ```python
> from transformers import BertModel
> 
> model = BertModel.from_pretrained("./scibert_scivocab_uncased")
> model.eval()
> ```
> 
> This should work 🤗

It works! Thank you very much!