noamaanMulla-03/transformers-issues huggingface dataset:
Error reported when fine tuning on my dataset using ''run_language_modeling.py"
Author: jasonachonu
Pull request: no
Labels: name: wontfix

name: Ex: LM (Finetuning)

# 🐛 Bug

## Information

Model I am using (RoBerta):

Language I am using the model on (English):

The problem arises when using:
* [ ] the official example scripts: (give details below) 
      Using the script provided in run_language_modeling.py I tried to fine tune the model on my 
     own dataset. But it shows a "KeyError 1" when It is about to run the first epoch and first iteration
* [ ] my own modified scripts: (give details below)
 i only modified the script to read in my dataset using the LineByLineTextDataset class to read in my sentence which is in an excel file and convert them to a list like in the class


Please advice, if there a proper format my data-set should be in, before fine tuning it.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)
To Fine tuning the language model of Bert on our my own data.


## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?): yes
- Tensorflow version (GPU?):
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
The implementation of GPT2 masked attention mechanism will cause errors when the model was trained after some iterations.
Author: xunzi2020
Pull request: no
Labels: name: wontfix

name: Core: Modeling

# 🐛 Bug

## Information
Model I am using (GPT2):
Language I am using the model on ( Chinese ...): 

File in modeling_gpt2.py：
146        w = w * b - 1e4 * (1 - b)  # here the bias "-1e4" is too big, suggest replace it to "-1e10"

The item value of "query * key" may be close to or lower to "-1e4" ,  then after the softmax operation, the weights of masked attention may be attend to "unseen" context. This will cause inference errors when to evaluate the model.


Comments:
Thanks for the bug report @xunzi2020! Did you by any chance measure the degree results would differ when replacing `-1e4` by `-1e10`? 

Yes, I trained with chinese text (~30GB), it happened.


Patrick von Platen <notifications@github.com> 于2020年3月9日周一 下午4:11写道：

> Thanks for the bug report @xunzi2020 <https://github.com/xunzi2020>! Did
> you by any chance measure the degree results would differ when replacing
> -1e4 by -1e10?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/3181?email_source=notifications&email_token=AOYV777JVZ6HL4BNOKYY7HDRGSQEXA5CNFSM4LEAB3C2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOGCLSI#issuecomment-596387273>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AOYV777AZV3K5GBW4TFVFFDRGSQEXANCNFSM4LEAB3CQ>
> .
>


Do you have a comparison such as: 

| - | GPT2 with -1e4 vs.  GPT2 with -1e10 | 
| ------------- | ------------- | 
| abs mean(softmax(logits) - softmax(logits)) | ???? |
| relative mean(softmax(logits) - softmax(logits)/softmax(logits))  | ???? |

lets say averaged over 100 - 1000 input samples and all GPT2 logits (50256)? 

That would be great to quantify the impact this change would have.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
NER Models not using BIO tagging scheme
Author: guillaume-be
Pull request: no
Labels: name: Ex: Named Entity Recognition

Hello,

I have noticed that most (all?) pre-trained models uploaded seem not to use the BIO tagging scheme, instead returning `I-xxx` and `O` for all categories (`B-xxx` is not used). I believe this is just caused by the way these models have been trained. This makes using these models in actual pipelines difficult as entity parsing becomes impossible (two consecutive entities cannot be differentiated from a multi-tokens entity). 

Are there any plans for updating these models with a BIO labelling scheme?
@stefan-it including you in the discussion as you have uploaded standard models used in the default pipelines

Comments:
Hi @guillaume-be , only the CoNLL-2003 models are using this IOB1 tagging scheme, because it is used in the original dataset. You could convert it (+ needs proper testing) but this requires a modification of the original dataset.

I could train models e.g. on the WikiANN corpus (see [here](https://www.aclweb.org/anthology/P17-1178/) and WikiANN with balanced train/dev/test splits [here](https://www.aclweb.org/anthology/P19-1015/)) for English, but this corpus is then just a "silver standard" :)

Hi @stefan-it ,

Thank you for your response. Assuming the CoNLL-03 model is in IOB1 tagging scheme, it would still use `B-*` labels in case of multi-token entities. For example:
```
Alex I-PER
is O
going O
to O
Los B-LOC
Angeles I-LOC
```
However, the model currently outputs the following for this example:
```
Alex I-PER
is O
going O
to O
Los I-LOC
Angeles I-LOC
```

This means that it is impossible to differentiate multi-token entities (such as Los Angeles) from multiple consecutive entities, e.g. `Alex Bob and Amy are going to Los Angeles`. So far I have never seen the model uploaded generate a `B-*` token. 

`B-*` is only used when an entity follows right after another entity, e.g. different political parties, like this example from the training dataset:

```bash
MAY NNP I-NP O
1996 CD I-NP O
CDU NNP I-NP I-ORG
/ SYM O I-ORG
CSU NNP I-NP I-ORG
SPD NNP I-NP B-ORG
FDP NNP I-NP B-ORG
Greens NNP I-NP B-ORG
PDS NNP I-NP B-ORG
```

Just have a look at the last column. `CDU/CSU` can be considered as one entity, but right after it another political party `SPD` is followed, so it is tagged with a beginning `B-` :)

@stefan-it thank you for the clarification!

@stefan-it are there plans to [merge this pr](https://github.com/huggingface/transformers/pull/3957)? it could help with this issue. i have it myself as i need to pool entity embeddings. https://github.com/huggingface/transformers/issues/3548

@petulla Should be merged soon 👍
Question Answering for TF trainer
Author: jplu
Pull request: yes

This PR add the Question Answering task in the Tensorflow trainer. For the moment the evaluation is not possible due to the complexity of the SQuAD metrics but the training is working.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4320?src=pr&el=h1) Report
> Merging [#4320](https://codecov.io/gh/huggingface/transformers/pull/4320?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/4bf5042240d33286460b83f3dbf9be77500faab3&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `0.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4320/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4320?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4320      +/-   ##
==========================================
- Coverage   78.16%   78.15%   -0.01%     
==========================================
  Files         120      120              
  Lines       20005    20009       +4     
==========================================
+ Hits        15636    15638       +2     
- Misses       4369     4371       +2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4320?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/4320/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyX3RmLnB5) | `17.75% <0.00%> (-0.34%)` | :arrow_down: |
| [src/transformers/training\_args\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/4320/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmluZ19hcmdzX3RmLnB5) | `55.31% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4320/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.77% <0.00%> (ø)` | |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/4320/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `41.09% <0.00%> (+0.28%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4320/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.85% <0.00%> (+0.41%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4320?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4320?src=pr&el=footer). Last update [4bf5042...2a6162b](https://codecov.io/gh/huggingface/transformers/pull/4320?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


LMK when it's ready to merge

Good to merge!
Reason for speedup
Author: mamesmak
Pull request: no
Labels: name: wontfix

I used to develop a code based on the pytorch_transformers package (v1.1) and now transfering to the current version. I see a 2x speedup in Bert run_glue.py. Wondering what is the major reason? Looking at the code, I couldn't find major differences.

Comments:
Depending on which version you are using, it might be that you are using the fast [tokenizers ](https://github.com/huggingface/tokenizers) library which offers a much improved tokenizer interface built on Rust. 

Thanks for the reply. I don't think it is because of the tokenizer. I measured the encoder part and see huge improvement in the encoder speed. Still cannot figure out the difference. comparing pytorch-transformers 1.1 with transformers 2.5.1.

It is unlikely that there have been architecture changes, but it might be caused by a different torch/tensorflow version? Are you testing both transformers versions on the same framework version? It is likely that this is caused by an optimisation of an activation function.  

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
where is the position emdeddings in bert for training a new model from scratch ?
Author: 2hip3ng
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
Right here

https://github.com/huggingface/transformers/blob/31f2437f07cf014a042789f52fa1519a485e8b2b/src/transformers/modeling_bert.py#L150

In the future, please make an effort to write a decent post that explains exactly what you need. This is very low quality.

Sry for low quality. But how about its initialization？ In the paper， it uses sin and cos functions， while the code seems like using random initialization.




| |
Wang
|
|
邮箱：zhipengwang1121@163.com
|

Signature is customized by Netease Mail Master

On 03/10/2020 22:15, Bram Vanroy wrote:

Right here

https://github.com/huggingface/transformers/blob/31f2437f07cf014a042789f52fa1519a485e8b2b/src/transformers/modeling_bert.py#L150

In the future, please make an effort to write a decent post that explains exactly what you need. This is very low quality.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to extract "contiguous tokens" from `NerPipeline` results?
Author: enzoampil
Pull request: no

Using `NerPipeline`, I want to be able to input a string (sequence of tokens), and extract *entity groups*, where an entity group is a contiguous series of tokens, having the same *entity type*.

**Example:**

For the ner code below:

```
from transformers import pipeline, AutoModelForTokenClassification, AutoTokenizer

# Allocate a pipeline for sentiment-analysis
model = AutoModelForTokenClassification.from_pretrained("dbmdz/bert-large-cased-finetuned-conll03-english")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
nlp = pipeline('ner', model=model, tokenizer=tokenizer)
nlp('Enzo works at the Australian National University (AUN)')
```

This returns:

```
[{'entity': 'I-PER', 'score': 0.9983270168304443, 'word': 'En'},
 {'entity': 'I-PER', 'score': 0.9952995777130127, 'word': '##zo'},
 {'entity': 'I-ORG', 'score': 0.9984350204467773, 'word': 'Australian'},
 {'entity': 'I-ORG', 'score': 0.9967807531356812, 'word': 'National'},
 {'entity': 'I-ORG', 'score': 0.9959043264389038, 'word': 'University'},
 {'entity': 'I-ORG', 'score': 0.9900023937225342, 'word': 'AU'},
 {'entity': 'I-ORG', 'score': 0.9763911366462708, 'word': '##N'}]
```

When I want it to return something like:

```
[{'entity_group': 'I-PER', 'score': 0.9983270168304443, 'word': 'Enzo'},
 {'entity_group': 'I-ORG', 'score': 0.9984350204467773, 'word': 'Australian National University'},
 {'entity_group': 'I-ORG', 'score': 0.9900023937225342, 'word': 'AUN'}]
```

I should be able to write a function that performs the above transformation if the indices of the word pieces are also indicated in the dictionary output of `NerPipeline`. Is this currently possible? Please advise me on if there's already an easy way to do this. 

If not, I can fork the repo and introduce an `index` key to the dictionary output. I can send a PR for this if the use case is general enough.

Comments:
What do you think @mfuntowicz?

[Related issue ](https://github.com/huggingface/transformers/issues/2488)

Actually, I realized that instead of `index`, even better would be the word piece `offsets`, similar to that returned by `tokenizer.encode` from the huggingface [tokenizers package](https://github.com/huggingface/tokenizers).

Will get to implementing this within the week if this isn't supported yet!

This recently merged [PR](https://github.com/huggingface/transformers/pull/3957) should solve this issue 🙂
OpusNMT/MarianMT Machine Translation Models
Author: sshleifer
Pull request: no
Labels: name: seq2seq

### Model description
1,026 Language Pair Models, downloadable [here](http://opus.nlpl.eu/Opus-MT/)
Trained with Marian C++ [library](https://github.com/marian-nmt/marian)

### Open source status

* [ x] the model implementation is available: (give details)
* [ x] the model weights are available: (give details)
* [ x] who are the authors: TODO, find gh-usernames of authors!


### Proposed API:
```python
model_name = 'marian/en-fr'
tokenizer = MarianTokenizer.from_pretrained(model_name)
model = MarianModel.from_pretrained(model_name)
src_text = "how are you today?"
tgt_text = "comment allez vous aujourd'hui?"
# Training API
full_inputs: dict = tokenizer.prepare_batch_for_translation(src_text, tgt_text=tgt_text)
loss, logits, *other_outputs = model(
        full_inputs['input_ids'],
        full_inputs['attention_mask'],

        full_inputs['decoder_input_ids'], # this argument is mandatory for the forward pass
)
# Inference/generate API
src_inputs: dict = tokenizer.prepare_batch_for_translation(src_text)
generated_fr_ids = model.generate(src_inputs['input_ids'], src_inputs['attention_mask'])
french_text: List[str] = processor.decode_batch(generated_fr_ids)
```

### Implementation Details
`MarianTokenizer` Signatures

(Not calling it Tokenizer to avoid confusion, but I don't feel strongly.)
- All models require `MosesSentenceSplitter` and `MosesPunctuationNormalizer` preprocessing
    -  There are some additional perl scripts we will not port for pre/post-processing
- 81 of the models require BPE, 960 require SentencePiece.
- We can decide which is which


```python
class MarianTokenizer:
    def __init__(self, vocab_file, source_bpe, target_bpe, source_spm, target_spm):
        # decide whether to use BPE/SPM based on which files are present in S3.
        self.source_lang, self.target_lang #inferred from paths/config
        #self.source_spm =

    @property
    def uses_sentencepiece(self) -> bool:
        return self.source_spm is not None

    def from_pretrained(self, *args, **kwargs):
        # needs to be overwritten or modified to not fail if certain files not present.

    def prepare_batch_for_translation(src_text:str, tgt_text=None, return_tensors='pt', max_length=512, pad_to_max_length=True) -> Dict[str, tensor/list]:
		return {}


    def decode_batch(self, target_lang_ids: List[List[int]]) -> List[str]:


    def decode(self, target_lang_id: List[int]) -> str:
```

#### Edits
- renamed `MarianProcessor` -> `MarianTokenizer` for consistency

Comments:
So no way to get output logits?

The `forward` method returns logits, like other models with language modeling heads. Is that what you meant?
train dev test split with BERT
Author: ksatvat
Pull request: no
Labels: name: wontfix

Does `run_multiple_choice.py` work on train dev test splits?
I need to run BERT on 3 labeled datasets. Train it on my training set, validate it on my validation set (tune hyperparameters and calculate loss), and evaluate it on my test set (report performance). I finally want to do prediction on a forth unlabeled dataset. 
I am wondering which of the codes in your repository includes these 3 modes. Thank you.

Comments:
You can write or add your own version easily..

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
NER tutorial: run_tf_ner.py reports an entity number not matching the one in the .txt files
Author: ChessMateK
Pull request: no
Labels: name: wontfix

name: TensorFlow

name: Ex: Named Entity Recognition

Hi everyone,

I'm following this tutorial https://github.com/huggingface/transformers/tree/master/examples/ner .

Generated results are the same but I noticed that the reported entity numbers ("support" column) are not the same B-entity numbers I can count through Notepad++ in *.txt files used as source for training, validation or test.

Which is my mistake?

Thank you.

UPDATE: run_ner.py works correctly. I suppose there is a bug in run_tf_ner.py.

I also tag @stefan-it who is the only mentioned in tutorial. Thank you Stefan.

Comments:
I'm currently not able to run the TF ner training script - I'm using TF in version 2.0.0b1 and the following error message is thrown:

```bash
...
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/init_ops_v2.py", line 769, in _assert_float_dtype
    raise ValueError("Expected floating point type, got %s." % dtype)
ValueError: Expected floating point type, got <dtype: 'int32'>.
```

🤔

I met the same problem. The script "run_tf_ner.py" can not get the same report result, have you solved it?

I have solved this issue. Because I evaluate the ner model with train.txt, and when the mode is "train", the dataset will be repeated and shuffled, so the support item of metric report is not same with the last report.

When I copy train.txt as dev.txt and change mode from "train" to "dev" during evaluation in the load_and_cache_examples, the dataset will not be repeated and shuffled and the report is reproductive.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[MbartTokenizer] save to sentencepiece.bpe.model
Author: Mehrad0711
Pull request: yes

There is a mismatch between VOCAB_FILES_NAMES for XLMRobertaTokenizer and MBartTokenizer.
When saving tokenizer files with save_pretrained(), XLMRobertaTokenizer's vocab_file_name is used, while during loading using load_pretrained(), MBartTokenizer's vocab_file_name is used.


Comments:
Thanks! 
Can you verify that
```bash
RUN_SLOW=1 pytest --tb=short -p no:warnings tests/test_modeling_bart.py -sv
```
works with this change?

Hi Sam!
Yes, the test passes on my machine.

I think you need to run `make style` for `check_code_quality` to pass.

Thanks, I just did but now black is complaining about other files being reformated. 

sorry about that, there were some changes in the interim.

Try
```bash
pip install flake8 --upgrade
make style
```

you might also need to `git merge master` or `git rebase master`
At the end, this PR should only change `tokenization_bart.py`.



Hi Sam!
The PR is now passing the tests. Is there anything else you would like to update?

Nope, thanks for contributing! Merging!
Request to add MobileBert
Author: msahamed
Pull request: no
Labels: name: New model

# 🌟 New model addition
MobileBERT
## Model description

MobileBERT is a thin version of BERT_LARGE, while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks.

## Open source status

* [ ] the model implementation is available: (give details)
https://github.com/google-research/google-research/tree/master/mobilebert

* [ ] the model weights are available: (give details)
https://storage.googleapis.com/cloud-tpu-checkpoints/mobilebert/uncased_L-24_H-128_B-512_A-4_F-4_OPT.tar.gz

* [ ] who are the authors: (mention them, if possible by @gh-username)
Google LLC
Xiaodan Song
Zhiqing Sun
Hongkun Yu
Denny Zou


Comments:
Duplicate of #4185
[TF generate] Fix issue for batch output generation of different output length.
Author: patrickvonplaten
Pull request: yes

This PR fixes the issue: #4088. 

A wrong variable was used in TF generate to determine the sentence length in the case when multiple outputs have different sentence lengths and the max sentence lengths is < the user defined `max_length`.

Also, both PT and TF generate are refactored a bit so that the `cur_length` variable is incremented directly after the `input_ids` are incremented.

Comments: