noamaanMulla-03/transformers-issues huggingface dataset:
Fix Roberta on TPU
Author: rickysaurav
Pull request: yes
Labels: name: wontfix

Fixes #1569 
- Revert tf.print() to logger , since tf.print() is an unsupported TPU ops.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1570?src=pr&el=h1) Report
> Merging [#1570](https://codecov.io/gh/huggingface/transformers/pull/1570?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/82f6abd98aaa691ca0adfe21e85a17dc6f386497?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1570/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1570?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1570   +/-   ##
=======================================
  Coverage   86.16%   86.16%           
=======================================
  Files          91       91           
  Lines       13593    13593           
=======================================
  Hits        11713    11713           
  Misses       1880     1880
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1570?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_tf\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/1570/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RmX3JvYmVydGEucHk=) | `100% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1570?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1570?src=pr&el=footer). Last update [82f6abd...55c3ae1](https://codecov.io/gh/huggingface/transformers/pull/1570?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hum this situation is a bit annoying because we switched from `logger.error` to `tf.print` to solve #1350

Is there a specific reason why we have such a warning message for Roberta but not for other models? All models based on BERT are require the special tokens.
I was having the same issues as #1350 on my end using the logger (lots of zmq , operationnotallowed errors) . The solution for me was to remove the entire warning message altogether.
Is that viable in this scenario? 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Q / Note: BERT Masked-LM fails to predict last token in sequence if it is not punctuation
Author: IngoMarquart
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I am playing with BERT to see what the distributions of the prediction for a MASK token are. I wrote a quick script that successively masks all words in an input sequence.
This is based on the implementation in the examples (e.g. the lm finetuning script and the examples in the  documentation).

In doing so, I found out that BERT generally fails to predict the last word in the sentence if it is not punctuation. With overwhelmingly high likelihood, BERT expects a normal SVO sentence to end with a full stop. While it can predict the correct word (the correct word usually appears in the top 10 most likely tokens), the likelihood as given by softmax is very low.

So by itself this is perhaps not surprising, because the large majority of examples in pre-training will have punctuation, especially if pre-training is not just the MLM but also the sentence prediction.

But I wonder if it should be best-practice to ensure every sentence is punctuated? If the MLM part of BERT consistently predicts punctuation, then a sentence without it will not be efficiently classified compared to one with punctuation, even on downstream tasks, right?

One thing to confirm, of course, would be that this is not an issue of the pyTorch implementation and how attention masks the <SEP> token or something?

What do you think?




Attached is the script, you should just be able to run it.

[lm_test.py.txt](https://github.com/huggingface/transformers/files/3739434/lm_test.py.txt)


Comments:
If you play with the script a bit, you can see that the loss for BERT with the MLM head is actually quite high, as someone suggested elsewhere, this may be due to pre-training on different tasks than just MLM

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
help: couldn't find such vocabulary files at this path or url
Author: WenxiongLiao
Pull request: no

I want to load the Chinese Roberta model  of pre-trained. When I use RobertaModel.from_pretrained() to load pre-trained model, it can't work.

<img width="1108" alt="屏幕快照 2019-12-25 下午11 56 11" src="https://user-images.githubusercontent.com/25845940/71454755-21f98100-27cd-11ea-8d0d-37beed6cc235.png">
<img width="1058" alt="屏幕快照 2019-12-25 下午11 56 17" src="https://user-images.githubusercontent.com/25845940/71455892-02b12280-27d2-11ea-98d6-ac1b2bd45901.png">


The  Chinese Roberta model is download from https://github.com/brightmart/roberta_zh
I am not sure if it is a problem with the pre-trained model or the transformers framework problem

Comments:
Did you manage to solve your issue? (if you did, how?)
[http] customizable requests user-agent
Author: julien-c
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2231?src=pr&el=h1) Report
> Merging [#2231](https://codecov.io/gh/huggingface/transformers/pull/2231?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a5a06a851e1da79138e53978aa079a093f243dde?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `77.77%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2231/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2231?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2231      +/-   ##
==========================================
- Coverage   81.43%   81.42%   -0.01%     
==========================================
  Files         122      122              
  Lines       18338    18348      +10     
==========================================
+ Hits        14933    14940       +7     
- Misses       3405     3408       +3
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2231?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2231/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2ZpbGVfdXRpbHMucHk=) | `71.36% <77.77%> (-0.07%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2231?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2231?src=pr&el=footer). Last update [a5a06a8...15d897f](https://codecov.io/gh/huggingface/transformers/pull/2231?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Adding new tokens to uncased tokenizers - case insensitivity is lost
Author: mbednarski
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hello!

I'm trying to add new tokens to bert-base-uncased. Let's say my token is '**cool-token**' and it was not present in the original vocab

```
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
print(tokenizer.tokenize('Sentence with cool-token'))
```

Prints as expected:

`['sentence', 'with', 'cool', '-', 'token']`

Now I add this token
`tokenizer.add_tokens(['cool-token'])`

Once again, prints as expected:
`['sentence', 'with', 'cool-token']`

However, when I try to utilize case-insensitivity my new token seems to be not recognized:

`print(tokenizer.tokenize('SenTenCE wIth cOOl-token'))`
prints
`['sentence', 'with', 'cool', '-', 'token']`

I would expect:
`['sentence', 'with', 'cool-token']`

It seems that custom tokens are not lowercased. Is it expected behavior and I have to `.lower()` my text manually or am I doing something wrong?

Anyway, I <3 your library



Comments:
Hmm looks like BertTokenizer's super class handles `.add_tokens()` and the first steps of `.tokenize()`, and doesn't really seem to consider whether the tokens should be made lowercase. I'm not sure whether it's intentional, but I'll make a PR and find out :smile: 

In the meantime, it might be a good idea to manually lowercase your text before tokenization.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Using HuggingFace pre-trained transformer to tokenize and generate iterator for a different text than the one it was trained on
Author: h56cho
Pull request: no
Labels: name: wontfix

Hello,
I am trying to do NLP by using HuggingFace transformers, and I have a question. Is it possible to use the pre-trained HuggingFace Transformer-XL and its pre-trained vocabulary to tokenize and generate BPTTIterator for the WikiText2 dataset instead of the WikiText103 that the transformer was originally trained on? If yes, could someone provide me example codes to illustrate how to 1.tokenize and 2. generate BPTTIterator to analyze the WikiText2, based on the pre-trained HuggingFace transformer XL model and its vocabulary?

NOTE: the WikiText2 can be obtained via

```js
import torchtext
# load WikiText-2 dataset and split it into train and test set
train_Wiki2, val_Wiki2, test_Wiki2 = torchtext.datasets.WikiText2.splits(TEXT)
```
or

```js
import lineflow as lf
import lineflow.datasets as lfds

# load WikiText-103 dataset
train_Wiki103 = lfds.WikiText2('train')
test_Wiki103 = lfds.WikiText2('test')
```

Thank you,

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Is there any efficient way to convert BERT outputs to fit token-level tasks?
Author: jianliu-ml
Pull request: no

Say I have a sentence consisting of two words: S = [“Definitely”, “not”], and what I want is to transfer S into an embedding matrix T with a size of (2, 100), where each row represents a word.

I want to adopt BERT embeddings. But in BERT, each word is represented as a sub-word unit. This means that S will be represented as [“Def”, “##in”, “##ite”, “##ly”, “not”] ( “Definitely” is tokenized as “Def”, “##in”, “##ite”, “##ly”). BERT will output an embedding matrix H with a size of (5, 100) :(.

My goal is to merge some rows of H according to the sub-word units.
For example, for “Definitely”, I should merge the embeddings of [“Def”, “##in”, “##ite”, “##ly”] to get its presentation.

In my current method, I use a head mask vector h = [1, 0, 0, 0, 1] to record the “head” of each word, where 1 indicates the head position:
h = [
1, -> “Def”
0, -> “##in”
0, -> “##ite”
0, -> “##ly”
1 -> “not”
]
So I should merge rows which have a head mask of 0 to that having a head mask of 1. I have to use the `for` computation to enumerate each element in h, which is slow and can not batchfy.

Is there any efficient method to do the above computation?



Comments:
i am also curious about it. ;;
https://github.com/dsindex/etagger/blob/master/feed.py
this is not efficient way. just pooling.

> i am also curious about it. ;;
> https://github.com/dsindex/etagger/blob/master/feed.py
> this is not efficient way. just pooling.

Hi, I think I get the solution. The problem can be solved via simple matrix computation.
For example, let the BERT representation of the above example be B, which has a size of (5, 100).  
We should first construct a matrix according to the sub-words:
m = 
[ 1, 1, 1, 1, 0
  0, 0, 0, 0, 1] 
Then we can simply compute m.dot(B), which is exactly the result.



I have solved this issue.
None in openAi-gpt tokenization
Author: ghost
Pull request: no
Labels: name: wontfix

Hi
I want to concat two sentences, and give it to openAI-gpt, I use cl sentence1 sep sentence2 sep
I got none with openai-gpt in the first position, could you tell me what is the expected format? thanks

Comments:
Are you using the GPT tokenizer? If not try

```
tokenizer = transformers.OpenAIGTPTTokenizer()
input_ids = tokenizer.encode(your_text)
```



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Quickstart BERT Example: Assertion Error
Author: rayed-therap
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arises when:
* I run the official BERT Example in my local Jupyter Lab environment: Copy pasted the code and ran it in one cell. 

## To Reproduce

Steps to reproduce the behavior:

1. Download and install Transformers from source
2. Start Jupyter lab
3. Copy paste and run the Quickstart BERT Example

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

Expected the assertion to pass.

## Environment

* OS: Oracle Linux Server 7.7
* Python version: 3.7.5
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): master
* Using GPU ? No
* Distributed of parallel setup ? No

## Additional context

<!-- Add any other context about the problem here. -->

![transformers_bug](https://user-images.githubusercontent.com/4649183/71553748-ff6e9e80-2a3e-11ea-813e-51d291d78870.png)


Comments:
Not quite sure what happened. Restarted my kernel after installing packages from ```examples/requirements.txt``` and is fixed. Closing the issue.
[debug] Debug Heisenbug, the old school way.
Author: julien-c
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2354?src=pr&el=h1) Report
> Merging [#2354](https://codecov.io/gh/huggingface/transformers/pull/2354?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/bfe870be654a1fc54c5479f9ad0875492d9cd959?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2354/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2354?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2354   +/-   ##
=======================================
  Coverage   73.32%   73.32%           
=======================================
  Files          87       87           
  Lines       14964    14964           
=======================================
  Hits        10972    10972           
  Misses       3992     3992
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2354?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2354?src=pr&el=footer). Last update [bfe870b...c8c4ecd](https://codecov.io/gh/huggingface/transformers/pull/2354?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Support for gpt2-medium, gpt2-large and distilgpt2 in pytorch-pretrained-bert 0.6.2
Author: g-karthik
Pull request: no
Labels: name: wontfix

## 🚀 Feature

Request: Inclusion of the below 3 lines in pytorch-pretrained-bert 0.6.2
https://github.com/huggingface/transformers/blob/ef1b8b2ae5ad1057154a126879f7eb8de685f862/transformers/modeling_gpt2.py#L40-L42

## Motivation

Currently, the above 3 lines exist in the latest version of transformers in PyPI, but not in pytorch-pretrained-bert 0.6.2 (also available in PyPI). Consequently, folks wanting to experiment with the above 3 pre-trained models need to necessarily upgrade to the latest version of transformers immediately. As a relief for such folks who plan to migrate eventually but not immediately, it would be great if the above 3 lines are added in pytorch-pretrained-bert 0.6.2.


Comments:
As far as I know pytorch-pretrained-bert development has been discontinued. That makes sense. If you want the new features, you have to upgrade.

Well technically what I'm asking for isn't a new feature, it's just backwards-compatibility for the above three model artifacts.

I can manually add them to the `modeling_gpt2.py` in my conda environment containing pytorch-pretrained-bert 0.6.2 and verify if these model artifacts work with the old package by invoking the `from_pretrained()` method with each of these three artifact names. I am guessing they would work, but I haven't tried yet.

I feel like this dictionary containing pre-trained artifact names should itself reside in S3, and in `modeling_gpt2.py`, the dictionary should be pulled from S3. Then you could continually add new artifact sizes to that dictionary in S3 and it will work with all versions of this repo, not just some versions. Does that make sense?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Custom language text generation
Author: Radeeswar
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

How to generate text in non english languages. Is xlm-mlm-100-1280 best for this, I tried but results are too worst. 
Also tried things mentioned here:
https://github.com/huggingface/transformers/issues/1414
https://github.com/huggingface/transformers/issues/1068
https://github.com/huggingface/transformers/issues/1407
https://github.com/Morizeyao/GPT2-Chinese

Any better suggestion Please.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
run_generation.py example for a batch
Author: ghost
Pull request: no
Labels: name: wontfix

Hi
I want to use example/run_generation.py to enter a batch of sentences and get a batch of generated outputs. could you please assist me and provide me with the commands how I can do it, is this possible with this code, if not I really appreciate adding this feature. thanks 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
What is the relationship between `run_lm_finetuning.py` and the scripts in `lm_finetuning`?
Author: zphang
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

It looks like there are now two scripts for running LM fine-tuning. While `run_lm_finetuning` seems to be newer, the documentation in `lm_finetuning` seems to indicate that there is more subtlety to generating the right data for performing LM fine-tuning in the BERT format. Does the new script take this into account?

Sorry if I'm missing something obvious!

Comments:
Hi! The folder `lm_finetuning` is especially targeted at BERT. It gives details on two different losses that were used to pre-train BERT: the masked language modeling objective (MLM) and the next sentence prediction objective (NSP). It gives several insights to BERT's fine-tuning.

The file `run_lm_finetuning`, on the other hand, showcases how to fine-tune language modeling on several models: BERT, GPT, GPT-2, and RoBERTa. It only uses a single objective; MLM for BERT and RoBERTa and CLM (causal language modeling) for GPT and GPT-2.

How can we fine-tune on the next sentence prediction task? I did not find the `lm_finetuning` files. Thank you.

Hi @JiajunBao, these scripts were community maintained and have since been removed. We do not have any script working on the next sentence prediction task. I believe the `lm_finetuning` files were last up to date in 1.1.0, so you may look [here](https://github.com/huggingface/transformers/tree/1.1.0/examples/lm_finetuning).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How does arg --vocab_transform help in extract_distilbert.py?
Author: evehsu
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi everyone, I'm new to experiment with bert model distillation. When running extract_distilbert.py on my fine tuned bert model, I came across the argusment vocat_transform.
```
if args.vocab_transform:
        for w in ['weight', 'bias']:
            compressed_sd[f'vocab_transform.{w}'] = state_dict[f'cls.predictions.transform.dense.{w}']
            compressed_sd[f'vocab_layer_norm.{w}'] = state_dict[f'cls.predictions.transform.LayerNorm.{w}']
```
When should we use this argument in running extract_distilbert? Any scenario we could benefit doing so?
Thanks!

Comments:
Hello @evehsu,

BERT uses an additional non-linearity before the vocabulary projection (see [here](https://github.com/huggingface/transformers/blob/master/transformers/modeling_bert.py#L381)).
It's a design choice, as far as I know, XLM doesn't a non-linearity right before the vocab projection (the language modeling head).

I left this option because I experimented with it, but if you want to keep the BERT architecture as unchanged as possible, you should use the `--vocab_transform` to ensure you also extract the pre-trained weights for this non-linearity.

Victor

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
RunTimeError in "run_summarization": expected device cuda:0 and dtype byte but got device cuda: 0 and dtype Bool
Author: junxu-ai
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):
bertabs-finetuned-cnndm-extractive-abstractive-summarization-pytorch_model

Language I am using the model on (, Chinese....):
English

The problem arise when using:
* [ ] the official example scripts: (give details)  run_summarization.py in examples


The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: summarization


## To Reproduce

Steps to reproduce the behavior:

1.  python run_summarization.py --documents_dir .\data --sumaries_output_dir .\output

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Win10
* Python version: 3.6.1
* PyTorch version:1.2.0
* PyTorch Transformers version (or branch): master at 24 Dec
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

errors come from modeling_berabs.py, line 328, in forward


Comments:
Someone suggested to use Pytorch v1.1.0 instead of 1.2.0. But not sure if it is ok. 

it's a version inconsistent issue. 
in ver 1.1.0, torch.gt outputs:
torch.gt(torch.tensor([[1, 2], [3, 4]]), torch.tensor([[1, 1], [4, 4]]))
tensor([[ 0,  1],
        [ 0,  0]], dtype=torch.uint8)
while in ver 1.2.0, it outputs:
>>> torch.ge(torch.tensor([[1, 2], [3, 4]]), torch.tensor([[1, 1], [4, 4]]))
tensor([[True, True], [False, True]])


Please see the pull request #2369