noamaanMulla-03/transformers-issues huggingface dataset:
Create README.md for bio+discharge summary BERT
Author: EmilyAlsentzer
Pull request: yes
Labels: name: model card

Add Bio+ Discharge Summary BERT from Publicly Available Clinical BERT Embeddings

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3230?src=pr&el=h1) Report
> Merging [#3230](https://codecov.io/gh/huggingface/transformers/pull/3230?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e43afb1bb87f01470d0bd16cac2d2aac50a76d7a?src=pr&el=desc) will **increase** coverage by `0.08%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3230/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3230?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3230      +/-   ##
==========================================
+ Coverage   77.94%   78.02%   +0.08%     
==========================================
  Files          98       98              
  Lines       16665    16665              
==========================================
+ Hits        12989    13003      +14     
+ Misses       3676     3662      -14
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3230?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3230/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `93.84% <0%> (+0.27%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3230/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.37% <0%> (+2.14%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3230?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3230?src=pr&el=footer). Last update [e43afb1...2ed661c](https://codecov.io/gh/huggingface/transformers/pull/3230?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[model_cards] 🇹🇷 Add new (cased) DistilBERTurk model
Author: stefan-it
Pull request: yes

Hi,

this PR adds a new distilled BERT model for Turkish: DistilBERTurk 🤗

It was trained with the official Hugging Face [implementation](https://github.com/huggingface/transformers/tree/master/examples/distillation) for model distillation. It uses 7GB of the original training data of BERTurk, and uses the cased BERTurk model as teacher model.

DistilBERTurk was trained for 5 days on 4 RTX 2080 TI.

Performance is really promising: for PoS tagging the model outperforms the 24-layer XLM-RoBERTa and is only 0.69% behind the teacher model. For NER there's a performance diff of 0.44% compared to mBERT and 1.68% compared to the teacher model.

Comments:
Also cc @VictorSanh for the model distillation script.

Thanks @stefan-it this is awesome

I'm supposed to use the same line-breaking options as GitHub for markdown formatting (using marked.js), however this still seems to not render like on GitHub: https://huggingface.co/dbmdz/distilbert-base-turkish-cased

will need to investigate.
Hello, how to upload a .ckpt file in TFBertForSequenceClassification?
Author: hecongqing
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I find the TFBertForSequenceClassification only can upload .h5 file

Comments:
What model is referring to this .ckpt file? If the model is BERT or ALBERT, you can:
- use the `convert_X_original_tf_checkpoint_to_pytorch --tf_checkpoint_path=dir/model.ckpt-xxx`, where _X_ is albert or bert
- load the .pt model into Transformers as usual

> ##  Questions & Help
> I find the TFBertForSequenceClassification only can upload .h5 file

@TheEdoardo93  if load the .pt model into Transformers as usual.  I think it will be pytorch model.
But I want use the TFBertForSequenceClassification to train with the model.fit, and with the mutil gpu with tf Strategy.

What can I do?


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Model card for dkleczek/bert-base-polish-uncased-v1
Author: kldarek
Pull request: yes
Labels: name: model card

Model card for dkleczek/bert-base-polish-uncased-v1

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3221?src=pr&el=h1) Report
> Merging [#3221](https://codecov.io/gh/huggingface/transformers/pull/3221?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d6de6423baf02a971d38ee69824104a1f0f85ad2&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3221/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3221?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3221      +/-   ##
==========================================
- Coverage   78.14%   78.14%   -0.01%     
==========================================
  Files          98       98              
  Lines       16668    16668              
==========================================
- Hits        13026    13025       -1     
- Misses       3642     3643       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3221?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3221/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `93.54% <0.00%> (-0.20%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3221?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3221?src=pr&el=footer). Last update [d6de642...aa7c949](https://codecov.io/gh/huggingface/transformers/pull/3221?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks for sharing => [model page](https://huggingface.co/dkleczek/bert-base-polish-uncased-v1)

will fix images + lowercase the language tag in the next commit
simplify polbert usage example with pipelines
Author: kldarek
Pull request: yes
Labels: name: model card

Indeed this is much simpler now! Pipelines look great :) 

Comments:
Squashed into #3248 

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3241?src=pr&el=h1) Report
> Merging [#3241](https://codecov.io/gh/huggingface/transformers/pull/3241?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a4c75f149269099a98613f51b76cd0b579a109ee?src=pr&el=desc) will **increase** coverage by `0.18%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3241/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3241?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3241      +/-   ##
==========================================
+ Coverage   77.82%   78.01%   +0.18%     
==========================================
  Files          98       98              
  Lines       16665    16665              
==========================================
+ Hits        12970    13001      +31     
+ Misses       3695     3664      -31
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3241?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3241/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `93.56% <0%> (-0.28%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3241/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.37% <0%> (+5.9%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3241?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3241?src=pr&el=footer). Last update [a4c75f1...1fd6564](https://codecov.io/gh/huggingface/transformers/pull/3241?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Error loading pretrained bert-base-multilingual-cased
Author: lucky-bai
Pull request: no

# 🐛 Bug

## Information

Loading `bert-base-multilingual-cased` from pretrained gives an error:

```
Traceback (most recent call last):
  File "error.py", line 4, in <module>
    bert_model = transformers.AutoModel.from_pretrained('bert-base-multilingual-cased', config=config)
  File "/scratch/gobi1/bai/bai-conda/lib/python3.7/site-packages/transformers/modeling_auto.py", line 380, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "/scratch/gobi1/bai/bai-conda/lib/python3.7/site-packages/transformers/modeling_utils.py", line 558, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)
RuntimeError: Error(s) in loading state_dict for BertModel:
        size mismatch for bert.embeddings.word_embeddings.weight: copying a param with shape torch.Size([119547, 768]) from checkpoint, the shape in current model is torch.Size([30522, 768]).
```

## To reproduce

The following snippet produces the error:

```
import transformers

config = transformers.BertConfig(output_hidden_states=True)
bert_model = transformers.AutoModel.from_pretrained('bert-base-multilingual-cased', config=config)
print(bert_model)
```

## Expected behavior

The model should load. Note that `bert-base-uncased` is able to load properly.

## Environment info
     
- `transformers` version: 2.5.1
- Platform: Linux
- Python version: 3.7.5
- PyTorch version (GPU?): 1.4.0 with GPU
- Tensorflow version (GPU?): N/A
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Comments:
You're creating a configuration based on the size of `bert-base-uncased`, so yes, it will work for that checkpoint. For any other checkpoint, however, you would need to change the values which are different (e.g. vocab size, which is what's failing in your case). It is indicated in the [documentation](https://huggingface.co/transformers/model_doc/bert.html#bertconfig):

> This is the configuration class to store the configuration of a BertModel. It is used to instantiate an BERT model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the BERT bert-base-uncased architecture.

In order to load a configuration automatically from a checkpoint, you may use `from_pretrained` on the configuration as well. In your case, that would be:

```py
import transformers

config = transformers.BertConfig.from_pretrained("bert-base-multilingual-cased", output_hidden_states=True)
bert_model = transformers.AutoModel.from_pretrained('bert-base-multilingual-cased', config=config)
print(bert_model)
```
How to encode a batch of sequence?
Author: PosoSAgapo
Pull request: no

Hi,I am trying to learn this transformers package.
I prepared the data as following format:
`(("Mary spends $20 on pizza"),("She likes eating it),("The pizza was great"))`
I saw methods like `tokenizer.encode`,`tokenizer.encode_plus`t and `tokenizer.batch_encode_plus`.However, the `tokenizer.encode` seems to only encode single sentence.
Because when I input the data below,the answer it gives are like this:
```
>>> d[0][0]
'John was writing lyrics for his new album'
>>> d[0][1]
'Franny did not particularly like all of the immigration happening'
>>> input_ids = torch.tensor(tokenizer.encode([d[0][0],d[0][1]]))
>>> input_ids
tensor([101, 100, 100, 102])
```
Obviously,this is not the rights answer for the encoding.
When I was try method tokenizer.encode_plust,it can't even work properly,as the document write 

> "text (str or List[str]) – The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the tokenize method) or a list of integers (tokenized string ids using the convert_tokens_to_ids method)"

It can't even work when I only input a single sentence:
```
>>> input_ids = torch.tensor(tokenizer.encode_plus(d[0][0]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: Could not infer dtype of dict
```
And the method,tokenizer.batch_encode_plust gives the same error message.


Comments:
`batch_encode_plus` is the correct method :-) 

```
from transformers import BertTokenizer
batch_input_str = (("Mary spends $20 on pizza"), ("She likes eating it"), ("The pizza was great"))
tok = BertTokenizer.from_pretrained('bert-base-uncased')
print(tok.batch_encode_plus(batch_input_str, pad_to_max_length=True))
```
batch_encode_plus cannot work properly
Author: PosoSAgapo
Pull request: no

I have data like below:
```
>>> ds[0][0:3]
["John was writing lyrics for his new album.He started experiencing writer 's block.He tried to force himself to write but it would n't do anythingHe tried to force himself to write but it would n't do anything.He took a walk , hung out with some friends , and looked at natureHe took a walk , hung out with some friends , and looked at natureHe took a walk , hung out with some friends , and looked at nature", 'Franny did not particularly like all of the immigration happening.She thought immigrants were coming to cause social problemsShe thought immigrants were coming to cause social problems.Franny was upset when an immigrant moved in next door.The immigrant , Sal , was kind and became friends with FrannyThe immigrant , Sal , was kind and became friends with Franny', 'Ari spends $ 20 a day on pickles.He decides to make his own to save money.He puts the pickles in brine.Ari waits 2 weeks for his pickles to get sour']
>>> ds[1][0:3]
['He felt inspiration and then went back home to write', 'When he finished his paper he went to bedWhen he finished his paper he went to bed', 'Trudey hoped self-publishing would be more profitable']
```
I was trying to do next sentence prediction using BERT model, it is necessary to use text pair to finish this problem.However,when I was tring to encode ds[0] and ds[1] as a batched text pair,I have met following problem.
```
>>> input_wsrq1ids=tokenizer.batch_encode_plus([(ds[0][0:3],ds[1][0:3])],add_special_tokens=True,return_tensors='pt')
>>> input_wsrq1ids
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'inp_wsrq1ids' is not defined
>>> input_wsrq1ids
{'input_ids': tensor([[101, 100, 100, 100, 102, 100, 100, 100, 102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 1, 1, 1, 1]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1]])}
```
The answer is definitely not the right encoding fot this data. I did not quite understand this documentation of batch_encod_plus method,it says for text_pair,look for encode_plus for details.However,the encode_plus only works for non-batched data, and there is no clue and example code to show us how to use batch_enocde_plus properly.

Comments:
This is quite a weird way to encode your data with `ds[0][0:3]`. There are two cases:

1) You have a single string you want to encode:
```
input_str = 'hello, what time is it?'
input_ids_dict = tokenizer.encode_plus(input_str)
```

2) You have a batch of input string data that you want to encode:

```
input_str_batch = ['hello what time is it' , 'hello, how are you?', 'Hey, I'm Peter']
input_ids_dict = tokenizer.batch_encode_plus(input_str_batch, pad_to_max_length=True)
```

Also see #3237
Extract Features for GPT2 and Transformer-XL
Author: danlou
Pull request: no

Hi everyone,

I'm interested in extracting token-level embeddings from the pre-trained GPT2 and Transformer-XL models and noticed that extract_features.py seems to be specific to BERT.

Can you let us know if you have any plans to provide a similar implementation for models other than BERT?

Alternatively, could you possibly provide some hints for us to extract the token-level embeddings the code you already made available with the models?

Many thanks, great work!

Comments:
Hi Dan,
You can extract all the hidden-states of Transformer-XL using the snippet indicated in the readme [here](https://github.com/huggingface/pytorch-pretrained-BERT#12-transfoxlmodel).
For the GPT-2 it's not possible right now.
I can add it in the next release (or you can submit a PR).

@thomwolf  Can we extract final hidden layer representations from GPT-2 1.5 billion models now?
What is the difference between T5Model, T5WithLMHeadModel, T5PreTrainedModel?
Author: g-jing
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I notice that for T5 model, there are more choices(T5Model, T5WithLMHeadModel, T5PreTrainedModel) than BERT or GPT. What is the difference between these three? I think all three are pre-trained model. We do not use T5PreTrainedModel in our downstream task code. Besides, the difference between T5Model and T5WithLMHeadModel is that the latter contains one more linear layer at the end.   Am I right about these?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Implement Electra
Author: flozi00
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

https://github.com/google-research/electra

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

Getting better results for example in QA tasks.

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

I have no idea how to implement, caused by the experience with this library

Comments:
We are currently working on it :-) 
Check out PR: #3257
Update examples/ner/run_ner.py
Author: lifefeel
Pull request: yes

Update the example file by changing the name of AlbertForTokenClassification to AlbertForSequenceClassification.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3242?src=pr&el=h1) Report
> Merging [#3242](https://codecov.io/gh/huggingface/transformers/pull/3242?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a4c75f149269099a98613f51b76cd0b579a109ee?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3242/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3242?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master   #3242      +/-   ##
=========================================
- Coverage   77.82%   77.8%   -0.02%     
=========================================
  Files          98      98              
  Lines       16665   16665              
=========================================
- Hits        12970   12967       -3     
- Misses       3695    3698       +3
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3242?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3242/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `93.42% <0%> (-0.42%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3242?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3242?src=pr&el=footer). Last update [a4c75f1...340f2a7](https://codecov.io/gh/huggingface/transformers/pull/3242?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I found that it is not suitable for guideline. 
Close this request.
Update Mish activation function to use torchscript JIT
Author: iyaja
Pull request: yes
Labels: name: wontfix

This PR modifies the implementation of Mish to match that of the [fastai library](https://github.com/fastai/fastai_dev/blob/0f613ba3205990c83de9dba0c8798a9eec5452ce/dev/local/layers.py#L441). A discussion of the benefits of JIT for the Mish function can be found on the [fastai forums](https://forums.fast.ai/t/meet-mish-new-activation-function-possible-successor-to-relu/53299/587).

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2412?src=pr&el=h1) Report
> Merging [#2412](https://codecov.io/gh/huggingface/transformers/pull/2412?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0ffc8eaf53542092271a208a52e881668e753e72?src=pr&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `35.71%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2412/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2412?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2412      +/-   ##
==========================================
- Coverage   73.24%   73.21%   -0.04%     
==========================================
  Files          87       87              
  Lines       14989    15002      +13     
==========================================
+ Hits        10979    10984       +5     
- Misses       4010     4018       +8
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2412?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2412/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iZXJ0LnB5) | `86.55% <35.71%> (-1.15%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2412?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2412?src=pr&el=footer). Last update [0ffc8ea...286b55b](https://codecov.io/gh/huggingface/transformers/pull/2412?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.