noamaanMulla-03/transformers-issues huggingface dataset:
Error msg when running on the colab
Author: liguangzhe
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

![捕获](https://user-images.githubusercontent.com/43159433/70368968-05c68c00-1880-11ea-8d97-5189348a4eca.PNG)
 
Can anyone tell me where am I wrong or it's not my problem?I cloned whole the files from huggingface. Is it can be fixed? I would appreciate for any suggestion. Thank you.

Comments:
Hi! How did you obtain the train-v2.0 and dev-v2.0 files? Did you put the `--version_2_with_negative` flag to specify you're using SQuAD V2?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
CUDA out of memory for 8x V100 GPU
Author: mittalpatel
Pull request: no
Labels: name: wontfix

```
python -m torch.distributed.launch --nproc_per_node=8 run_squad.py \
    --model_type bert \
    --model_name_or_path bert-base-cased \
    --do_train \
    --do_eval \
    --do_lower_case \
    --train_file $SQUAD_DIR/train-v1.1.json \
    --predict_file $SQUAD_DIR/dev-v1.1.json \
    --learning_rate 3e-5 \
    --num_train_epochs 2 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir ../models/wwm_uncased_finetuned_squad/ \
    --per_gpu_train_batch_size 24 \
    --gradient_accumulation_steps 12
```
We are trying the same command (except  bert-base-cased, we are using bert-large-uncased-whole-word-masking) on 8x V100 GPU but getting CUDA out of memory error (CUDA out of memory. Tried to allocate 216.00 MiB....)

As per the https://github.com/huggingface/transformers/tree/master/examples it should work but it's giving error and stopping in the middle. Any tips would be appreciated. 

Comments:
bert large is bigger than bert base. You're using a batch size of 24 (which is big, especially with 12 gradient accumulation steps). 

Reduce your batch size in order for your model + your tensors to fit on the GPU and you won't experience the same error!

Right @LysandreJik , reducing the batch size did fix the error but it looks like the generated model we receive is not same as provided by huggingface. 

In our demo of closed domain QnA, https://demos.pragnakalp.com/bert-chatbot-demo, the answers are pretty good where we are using the model provided by huggingface (bert-large-uncased-whole-word-masking-finetuned-squad). But when we finetune on our own and even though we get 93.XX f1 score the accuracy of the model is not same as demo.

What other parameters were set by huggingface to generate "bert-large-uncased-whole-word-masking-finetuned-squad" model? 

If the only difference between the command you used and the command available [here](https://huggingface.co/transformers/examples.html#id1) is the batch size, you could try and adjust the gradient accumulation so that the resulting batch size is unchanged. For example if you put batch size equal to 6 (1/4 of the specified batch size, 24), you can multiply by 4 the gradient accumulation steps (-> 48)  so that you keep the same batch size.

What `exact_match` result did you obtain alongside the 93.xx F1 score?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Add T5 model
Author: astariul
Pull request: no
Labels: name: wontfix

# 🌟New model addition

## Model description

Google released paper + code + dataset + pre-trained model about their new **T5**, beating state-of-the-art in 17/24 tasks.

[Paper link](https://arxiv.org/pdf/1910.10683.pdf)

## Open Source status

* [x] the model implementation and weights are available: [Official codebase](https://github.com/google-research/text-to-text-transfer-transformer)


Comments:
+1, it is a very impressive work

https://github.com/google-research/text-to-text-transfer-transformer

However i would prefer seeing Albert implemented before T5.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Please

It's not super-well documented, but it's clearly present: 
https://github.com/huggingface/transformers/blob/dc17f2a1110aed8d1729e77b0619601e3d96b84e/src/transformers/modeling_tf_t5.py
What does the variable 'present' represent?
Author: h56cho
Pull request: no

Hello,

does the variable 'present' shown in [this](https://github.com/huggingface/transformers/blob/4e69104a1fba717026d6909d06288788e684c749/src/transformers/modeling_gpt2.py#L187) line of Hugging Face GPT-2 code represent final output of a single attention-head? (i.e. **not** the final output of the _output head_ , but the final output of the individual _attention head_, which is placed right before the feedforward block of the same layer).

If yes, is there anyway that I can retrieve the value of the variable 'present'?
Would it be possible that Hugging Face will make the value available for everyone?

Thank you,

Comments:
Can I use run_lm_finetuning.py for training models in an uncovered language?
Author: cppntn
Pull request: no

Is it possibile to use run_lm_finetuning.py script to train one of the models from scratch in a language not covered by the available pretrained models? (like spanish, italian, german).

My idea is to replicate something like camemBERT for a language different from french, given that I have the corpora needed for the training.

What are some suggestions that you could give me? What are the changes to make in the script in order to run it correctly for this purpose? How can I deal with a corpus of ~150GB?

Thanks for any help

Comments:
You can now leave `--model_name_or_path` to None in `run_language_modeling.py` to train a model from scratch.

See also https://huggingface.co/blog/how-to-train
Help with finetune BERT pretraining 
Author: yes1234man
Pull request: no

Hi
could you please assist me how I can pretrain the BERT model, so not like SNLI/MNLI when we finetune a pretrained model, but doing pretraining objective.
thanks a lot
and Merry Christmas and happy new year in advance to the team

Comments:
I suggest you follow the [run_lm_finetuning.py](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py) script. Instead of downloading a pretrained model, simply start with a fresh one. Here's an example:

```
from transformers import BertModel, BertConfig, BertTokenizer
model = BertModel(BertConfig())
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
```

I personally don't have a reason to write my own tokenizer, but if you do feel free to do that as well. All you need to do is generate a vocab.txt file from your corpus.

You can now leave `--model_name_or_path` to None in `run_language_modeling.py` to train a model from scratch.

See also https://huggingface.co/blog/how-to-train
Could convert_pytorch_checkpoint_to_tf2.py convert any pytorch model to tf2?
Author: xealml
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->


Comments:
I think **no**. You can use this Python script to convert a PyTorch implementation of one of the models supported by Transformers to TensorFlow 2.0 version.

Have you ever tried to use this Python script to convert **any** PyTorch model to TensorFlow 2.0?

> ##  Questions & Help

@TheEdoardo93 No,have not tried.I read the code.Finding maybe the model costumed support converting. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Why can`t we just use cached pytorch-model without internet
Author: floAlpha
Pull request: no
Labels: name: wontfix

Sorry for my Impolite words. I am very appreciate huggingface team for the excellent repo and 
my original intention is to provide some  suggestion to great repo to help more people like me, forgive me for being rude. 

Comments:
First, yes you can. You should be able to download the model(whether use shell command or python downloader) into a specified directory and modify the loading lines. That works for me. 
Second, please be appreciated to these fantastic works huggingface team pulled off. It's not their fault that you don't have stable access, so help yourself.

Humm wording apart, maybe we could relax the internet connection check indeed, I'll have a look.

Sincerely thank you for your reply, you are the great contributors, I apologize again for my inappropriate expression, mainly because I can`t properly solve the problem till midnight.  I just want to make my own suggestion for people encountered this like me, so I opened this issue. Actually I am very grateful for your work, and appreciate the open source community. Best regards!

Ok, network connection check has been relaxed in the now merged #500.
It will be included in the next PyPI release (probably next week).
In the meantime you can install from `master`.

So nice you are，I cannot continue my research without pytorch-pretrained-BERT，Thank you for your Incredible devotion.With my sincere apology.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
how to get "xlnet-base-cased-pytorch_model.bin" original 'last modified' date?
Author: eyal-orbach
Pull request: no

I'd like to test my finetuning on some wikipedia articles that have not been seen by the model.
For that, I can find the date of creation on the wikipedia article, but I'd also like to verify that this is after the 'last modified' date of the model. 
How can I get the answer to when was the model parameters' last modified?

Comments:
Just click on 'List all files in model' and you will see the upload date [1].

[1] https://huggingface.co/xlnet-base-cased
Update README.md
Author: mrm8488
Pull request: yes
Labels: name: model card

I trained the model for more epochs so I improved the results. This commit will update the results of the model and add a gif using it with **transformers/pipelines**

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2875?src=pr&el=h1) Report
> Merging [#2875](https://codecov.io/gh/huggingface/transformers/pull/2875?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/73028c5df0c28ca179fbe565482a9c2143787f61?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2875/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2875?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2875   +/-   ##
=======================================
  Coverage   75.06%   75.06%           
=======================================
  Files          94       94           
  Lines       15288    15288           
=======================================
  Hits        11476    11476           
  Misses       3812     3812
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2875?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2875?src=pr&el=footer). Last update [73028c5...cb1cba9](https://codecov.io/gh/huggingface/transformers/pull/2875?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks for updating! GIFs on Giphy are a bit over-compressed so you can also host them 
- on one of your GitHub repos
- or even directly in the model card's folder (see severinsimmler/literary-german-bert/README.md as an example)

Also, great results :)

Thank you!!
BertModel.from_pretrained() doesn't accept pathlib.PosixPath anymore
Author: koichikawamura
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): Japanese

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Prepare a directory with `config.json`, `pytorch_model.bin`.
2. Give the directory as a pathlib.PosixPath such like `bert_model = BertModel.from_pretrained(bert_path)`
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
TypeError: argument of type 'PosixPath' is not iterable


## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
This used to load the model with 2.1.1, but started to cause an error since 2.2.0.
`bert_model = BertModel.from_pretrained(str(bert_path))` works.

## Environment

colab
* OS:
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch): 2.2.0 and later
* Using GPU ?
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
It comes from this line https://github.com/huggingface/transformers/blob/master/transformers/modeling_utils.py#L321-L324
If it's a PosixPath, it's not an iterable so `"albert" in path` crashes.
Patch already pushed in previous PR!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Transformers と Simpletransfomrersを使ったAlebertでのNERの対応
Author: dwarfer7634
Pull request: yes

AlbertでNERが出来るようになっていなかったので、その対応をしました。
主に、modeling_albert.pyにAlbertForTokenClassificationを実装したのと、それと整合性をとるために、その他のコードの変更を行いました。

Comments:
BertForSequenceClassification' object has no attribute 'bias
Author: chikubee
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [x] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

<!-- Add any other context about the problem here. -->
I used this script to load the bert model i had finetuned for a classification task following google_research_bert.
I want to convert those TF checkpoints to pytorch.

```python
config = BertConfig.from_pretrained('bert-base-uncased')
config.num_labels=4
# You will need to load a BertForSequenceClassification model
model = BertForSequenceClassification(config)

tf_checkpoint_path = init_checkpoint
# Load weights from tf checkpoint
load_tf_weights_in_bert(model, config, tf_checkpoint_path)

pytorch_dump_path = "./pytorch_bert_output"
# Save pytorch-model
print("Save PyTorch model to {}".format(pytorch_dump_path))
torch.save(model.state_dict(), pytorch_dump_path)
```

When i execute this, i get the following error, BertForSequenceClassification' object has no attribute 'bias. Any leads would be helpful. 
Thanks

Comments:
Can you show us the full error message?

Can it be related to #2109 in some way?

> ##  Bug
> Model I am using (Bert, XLNet....):
> 
> Language I am using the model on (English, Chinese....):
> 
> The problem arise when using:
> 
> * [x]  the official example scripts: (give details)
> * [ ]  my own modified scripts: (give details)
> 
> The tasks I am working on is:
> 
> * [ ]  an official GLUE/SQUaD task: (give the name)
> * [x]  my own task or dataset: (give details)
> 
> I used this script to load the bert model i had finetuned for a classification task following google_research_bert.
> I want to convert those TF checkpoints to pytorch.
> 
> ```python
> config = BertConfig.from_pretrained('bert-base-uncased')
> config.num_labels=4
> # You will need to load a BertForSequenceClassification model
> model = BertForSequenceClassification(config)
> 
> tf_checkpoint_path = init_checkpoint
> # Load weights from tf checkpoint
> load_tf_weights_in_bert(model, config, tf_checkpoint_path)
> 
> pytorch_dump_path = "./pytorch_bert_output"
> # Save pytorch-model
> print("Save PyTorch model to {}".format(pytorch_dump_path))
> torch.save(model.state_dict(), pytorch_dump_path)
> ```
> 
> When i execute this, i get the following error, BertForSequenceClassification' object has no attribute 'bias. Any leads would be helpful.
> Thanks

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[model_cards] 🇹🇷 Add new (cased) BERTurk model
Author: stefan-it
Pull request: yes
Labels: name: model card

Hi,

this PR adds the model card for the (cased) community-driven 🇹🇷 BERTurk model.

Uncased model is coming soon!

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2879?src=pr&el=h1) Report
> Merging [#2879](https://codecov.io/gh/huggingface/transformers/pull/2879?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/6083c1566e261668a5de73cfe484c171ce232812?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2879/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2879?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2879   +/-   ##
=======================================
  Coverage   75.06%   75.06%           
=======================================
  Files          94       94           
  Lines       15288    15288           
=======================================
  Hits        11476    11476           
  Misses       3812     3812
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2879?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2879?src=pr&el=footer). Last update [6083c15...d18f775](https://codecov.io/gh/huggingface/transformers/pull/2879?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Looks good: https://huggingface.co/dbmdz/bert-base-turkish-cased
master branch examples/run_squad.py: missing --predict_file argparse argument
Author: mfeblowitz
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using: albert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [X] the official example scripts: (give details)
examples/run_squad/py: --predict_file not recognized
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: run_squad
* [ ] my own task or dataset: (give details)

## To Reproduce

Try running an eval only run with run_squad.py
Steps to reproduce the behavior:

1. provide all of the requisite arguments to run_squad.py
2. observe error `run_squad.py: error: unrecognized arguments: --predict_file`
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
A correct run
## Environment

* OS:
* Python version: 3.7.5
* PyTorch version: n/a
* PyTorch Transformers version (or branch): master
* Using GPU ? yes
* Distributed of parallel setup ?no
* Any other relevant information:

Quick inspection of examples/run_squad.py reveals missing declaration, present in, e.g., 1.2.0

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
In order to use the **evaluation** mode, you have to pass from script the `do_eval` parameter (in addition to the "classical" input parameters for evaluation).

> ##  Bug
> Model I am using: albert
> 
> Language I am using the model on (English, Chinese....): English
> 
> The problem arise when using:
> 
> * [x]  the official example scripts: (give details)
>   examples/run_squad/py: --predict_file not recognized
> * [ ]  my own modified scripts: (give details)
> 
> The tasks I am working on is:
> 
> * [x]  an official GLUE/SQUaD task: run_squad
> * [ ]  my own task or dataset: (give details)
> 
> ## To Reproduce
> Try running an eval only run with run_squad.py
> Steps to reproduce the behavior:
> 
> 1. provide all of the requisite arguments to run_squad.py
> 2. observe error `run_squad.py: error: unrecognized arguments: --predict_file`
> 
> ## Expected behavior
> A correct run
> 
> ## Environment
> * OS:
> * Python version: 3.7.5
> * PyTorch version: n/a
> * PyTorch Transformers version (or branch): master
> * Using GPU ? yes
> * Distributed of parallel setup ?no
> * Any other relevant information:
> 
> Quick inspection of examples/run_squad.py reveals missing declaration, present in, e.g., 1.2.0
> 
> ## Additional context

Indeed, there was a big refactor of the SQuAD script recently which removed these arguments in favor of `data_dir`, which contains the files. I'll add the possibility to either use `predict_file` and `train_file` instead of `data_dir` later today.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Reduce the CamemBERT dimensions
Author: neuromaancer
Pull request: no
Labels: name: wontfix

name: Usage

I want to reduce the output dimension by adding a linear layer at the end of the camem model.

code :
```python
from transformers import CamembertTokenizer, CamembertModel
import torch
from torch.nn import Sequential, Linear

tokenizer = CamembertTokenizer.from_pretrained('camembert-base')
model = CamembertModel.from_pretrained('camembert-base')
input_ids = torch.tensor(tokenizer.encode("La pose d'un panneau stop.", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
# labels = torch.tensor([1] * input_ids.size(1)).unsqueeze(0)  # Batch size 1
model = Sequential(model, Linear(768, 256))

outputs = model(input_ids)

print(input_ids)
print(outputs[1].size())
print(outputs[0].size())
```
I got this : 

```shell
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1366         - Output: :math:`(N, *, out\_features)`
   1367     """
-> 1368     if input.dim() == 2 and bias is not None:
   1369         # fused op is marginally faster
   1370         ret = torch.addmm(bias, input, weight.t())

AttributeError: 'tuple' object has no attribute 'dim'
```

Additionally, I want to do a word-level embedding, however, the 768 dimensions is too big from my point.

Thanks for your helps.


Comments:
Hi @AlafateABULIMITI sounds like a question that's better suited for Stack Overflow. Thanks!