noamaanMulla-03/transformers-issues huggingface dataset:
CircleCI doesn't run slow tests
Author: sshleifer
Pull request: no

circle_ci.cfg says `RUN_SLOW: yes`, but all my circleci runs have the slow tests skipped.
Is this expected behavior?
@LysandreJik 


Comments:
When deploying on CircleCI, it runs the `build_and_test` job, which runs the [following suites](https://github.com/huggingface/transformers/blob/master/.circleci/config.yml#L126-L133).

The slow tests are ran by the `run_all_tests_torch_and_tf` suite, which only triggers [weekly](https://github.com/huggingface/transformers/blob/master/.circleci/config.yml#L137). 

The slow tests are especially slow, and currently fail on CircleCI because the machines can't run for so long. We're exploring options to run them on a specific machine cc @julien-c 

Got it, thanks. Can I delete this line  https://github.com/huggingface/transformers/blob/81d6841b4be25a164235975e5ebdcf99d7a26633/.circleci/config.yml#L23

it confused me.

If you remove this line the slow tests won't run during the weekly tests though

Oh I get it, was missing
`run_all_tests_torch_and_tf` vs `run_tests_torch_and_tf`

should we rename `run_all_tests_torch_and_tf` to `run_slow_tests_torch_and_tf`?

Well its purpose really is to run all tests, not only the slow tests but the custom tokenizers and soon the doc examples as well, so I feel that the current name is fitting.
tiny issue with distilbertconfig docs
Author: waalge
Pull request: no
Labels: name: Documentation

name: Usage

# 🐛 Bug (barely) 

Discrepancy in variable names between docs and code: 
I presume [``intermediate_size``](https://github.com/huggingface/transformers/blob/520e7f211926e07b2059bc8e21b668db4372e4db/src/transformers/configuration_distilbert.py#L63) refers to [``hidden_dim``](https://github.com/huggingface/transformers/blob/520e7f211926e07b2059bc8e21b668db4372e4db/src/transformers/configuration_distilbert.py#L109)?


Comments:
You're correct! I updated it with 539f601. Thanks.
Add model readme for deepset/roberta-base-squad2
Author: tholor
Pull request: yes
Labels: name: model card

Adding a model readme for https://huggingface.co/deepset/roberta-base-squad2

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2797?src=pr&el=h1) Report
> Merging [#2797](https://codecov.io/gh/huggingface/transformers/pull/2797?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/92e974196fc35eb826f64808ae82d20c4380e3eb?src=pr&el=desc) will **increase** coverage by `1.08%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2797/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2797?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2797      +/-   ##
==========================================
+ Coverage   73.95%   75.03%   +1.08%     
==========================================
  Files          93       93              
  Lines       15272    15272              
==========================================
+ Hits        11295    11460     +165     
+ Misses       3977     3812     -165
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2797?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/2797/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `81.39% <0%> (+1.32%)` | :arrow_up: |
| [src/transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/2797/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jdHJsLnB5) | `94.27% <0%> (+2.2%)` | :arrow_up: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/2797/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `73.21% <0%> (+2.27%)` | :arrow_up: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2797/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `95.77% <0%> (+9.85%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2797/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9weXRvcmNoX3V0aWxzLnB5) | `89.93% <0%> (+81.2%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2797?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2797?src=pr&el=footer). Last update [92e9741...ec005e3](https://codecov.io/gh/huggingface/transformers/pull/2797?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Add albert-base-v3 to pretrained models?
Author: nicolasahar
Pull request: no

# 🚀 Feature request

Albert v3 was recently released on TFHub [here](https://tfhub.dev/google/albert_base/3). Could you please add it to the list of available pretrained models [here](https://huggingface.co/transformers/pretrained_models.html)?

## Motivation

Would provide the community with the most up-to-date albert version. 


Comments:
Hi, as mentioned in their changelog, the only difference between the v2 and v3 is the compatibility with TF 1.15 as they removed the `einsum` operation.

It won't change anything for the huggingface/transformers users as the models available here are only for TF2.

![DeepinScreenshot_select-area_20200210155025](https://user-images.githubusercontent.com/30755778/74188771-25ca5100-4c1d-11ea-9362-22705e1889e3.png)
Text Generation in Hebrew
Author: beneyal
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hi all,

I have 30K tweets in Hebrew and I want to create a sort of chatbot that will answer in the style of those tweets, similar to [this](https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313). The only multilingual models that were trained on Hebrew are BERT and XLM, and they are both MLMs which are not too good at text generation. I thought I could fine-tune XLM and then run `run_generation.py`, but `run_lm_finetuning.py` doesn't support XLM.

Is there a way I can go about my task?

Thanks!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Will you add XLNet text-generation feature ?
Author: efeiefei
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

There is `run_generation.py` in example now. Do you have a plan to add feature of  complete lm_finetune and inference? Just like GPT-2.

Thanks

Comments:
Not in the short term

@thomwolf Thanks a lot

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Find dot product of query and key vectors 
Author: vr25
Pull request: no
Labels: name: wontfix

Hi,

I am following [this popular article](http://jalammar.github.io/illustrated-transformer/) to understand the Transformers. Alongside this, I am using [huggingface transformers](https://huggingface.co/transformers/model_doc/bert.html#bertmodel) to get the attention scores.

On running the following code:
`from transformers import BertTokenizer, BertModel, BertConfig, BertForTokenClassification
import torch

config = BertConfig.from_pretrained('bert-base-uncased', output_hidden_states=True, output_attentions=True)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased', config=config)

input_ids = torch.tensor(tokenizer.encode("Hello my dog is cute", add_special_tokens=False)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)

print(len(outputs))

last_hidden_states, pooler_outputs, hidden_states, attentions = outputs  # The last hidden-state is the first element of the output tuple

print(attentions)`

I get the weighted sum attention matrix of size 5x5. I am actually trying to find the softmax values like 0.88 and 0.12. I was wondering if there is any way I can obtain the dot-product scores.

![image](https://user-images.githubusercontent.com/22553367/70174075-394bbf80-16a2-11ea-972b-4aeb539dbd7e.png)

Thanks!

Comments:
I found [this](https://huggingface.co/transformers/_modules/transformers/modeling_bert.html) code which has transpose_for_scores but I am not sure how this can be used with the above code.

Yes, the `attentions` outputs of the model are the softmax values.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fix circleci cuInit error on Tensorflow >= 2.1.0.
Author: mfuntowicz
Pull request: yes
Labels: name: Tests

Tensorflow 2.1.0 introduce a new dependency model where `pip install tensorflow` would install tf **with GPU support**. Before 2.1.0 it would just install with CPU support only.

CircleCI machines are running without GPU hardware so, at initialisation, TensorFlow tests are looking for NVidia driver version but fails as their is no NVidia Driver running.

This PR introduces an extra (optional) dependency **tf-cpu** which explicitly requires **tensorflow-cpu** and makes sure it `pip install tf-cpu` instead of `pip install tf` while running unit tests.

It should remove the following error on CircleCI: 

```bash
tests/test_modeling_tf_bert.py::TFBertModelTest::test_attention_outputs 2020-02-10 11:14:08.280770: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-02-10 11:14:08.280808: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
2020-02-10 11:14:08.280837: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (40403e139ccb): /proc/driver/nvidia/version does not exist
2020-02-10 11:14:08.281093: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
```

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

Comments:
ALBERT how to obtain the embedding matrix?
Author: alessiocancian
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hi, I'm trying to use ALBERT for word embedding with this library.
ALBERT's doc mentioned an embedding size of 128 independently of the model version (base, large, ...) while the hidden_size changes.
I would like to obtain the 128 word (or subword) vectors but the model gives me only the output of the last hidden state (so for xxlarge a 4096 tensor for each token).

What am I doing wrong?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[model_cards] Add new German Europeana BERT models
Author: stefan-it
Pull request: yes
Labels: name: model card

Hi,

this PR adds the model cards for two new BERT models for Historic German.

The cased and uncased BERT models were trained on a huge corpus: newspapers from [Europeana](http://www.europeana-newspapers.eu/). Time period of these (noisy) OCRed newspapers is 18th - 20th century.

More information can be found [here](https://github.com/dbmdz/berts) and more detailed results on downstream tasks [here](https://github.com/stefan-it/europeana-bert).

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2805?src=pr&el=h1) Report
> Merging [#2805](https://codecov.io/gh/huggingface/transformers/pull/2805?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/68ccc04ee6c762183ff2b34b8b85d139f77cbf14?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2805/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2805?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2805   +/-   ##
=======================================
  Coverage   75.02%   75.02%           
=======================================
  Files          93       93           
  Lines       15275    15275           
=======================================
  Hits        11460    11460           
  Misses       3815     3815
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2805?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2805?src=pr&el=footer). Last update [68ccc04...e1833f7](https://codecov.io/gh/huggingface/transformers/pull/2805?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Use run_lm-finetuning on tpu
Author: abdallah197
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Is it possible to use the script run_lm-finetuning on TPUs, if not, what do you recommend to fine-tune BERT language model on TPUs using the transformers library

Comments:
Hello, the script would need to be adapted to run on TPU to take full advantage of the chips. We're actively working with the Cloud TPU team on scripts for fine-tuning on TPUs, which should be available in the coming weeks.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
The added tokens do not work as expected
Author: wenhuchen
Pull request: no
Labels: name: wontfix

Here is a minimum example, where we add a special token [ENT]
```
from transformers import BertTokenizer
bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased-vocab.txt')

bert_tokenizer.add_tokens(['[ENT]'])
print(len(tokenizer))

x = bert_tokenizer.encode("you are the [ENT] with [ENT] and [ENT]")
print(x)
bert_tokenizer.decode(x)
```
After decoding, we will end up having
```
you are the [ENT]with [ENT]with [ENT]
```
rather than
```
'you are the [ENT] with [ENT] with [ENT]'

```


Comments:
I think you can keep the issue open, this is a bug that should be fixed.

This could be related, I'm on commit  d46147294852694d1dc701c72b9053ff2e726265
![image](https://user-images.githubusercontent.com/1544039/70831841-91c93e00-1dc1-11ea-9bb3-3803312d4456.png)
It's strange that the id for "student" changed after adding the special token


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to use a batch size bigger than zero in Bert Sequence Classification
Author: ayrtondenner
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
[Hugging Face documentation describes](https://huggingface.co/transformers/model_doc/bert.html#bertforsequenceclassification) how to do a sequence classification using a Bert model:

```
from transformers import BertTokenizer, BertForSequenceClassification
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=labels)

loss, logits = outputs[:2]
```

However, there is only example for batch size 1. How to implement it when we have a list of phrases and want to use a bigger batch size?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: https://stackoverflow.com/questions/60170037/how-to-use-a-batch-size-bigger-than-zero-in-bert-sequence-classification

Comments:
I answered your question on stack overflow.
XLNet model params for Question answering 
Author: Swathygsb
Pull request: no
Labels: name: wontfix

Model: XLNet for QA
model(parameters ...)
RuntimeError: CUDA error: device-side assert triggered

Comments:
What parameters should I exactly pass for XLNet QA model?

More details would be nice but please check #1849 #1848 and #1805 . Also set `CUDA_LAUNCH_BLOCKING=1` environment variable to have more details about the error, from the CUDA.

Thanks. It worked after changing the input sequence limit to 128.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
embedding index getting out of range while running gpt2-xl model 
Author: mainulquraishi
Pull request: no
Labels: name: PyTorch

name: Core: Modeling

name: Ex: Generation

name: Usage

I am trying to run [hugginface][1] gpt2-xl model. I ran code from the [quickstart][2] page that load the small gpt2 model and generate text by the following code: 

    ```
    from transformers import GPT2LMHeadModel, GPT2Tokenizer
    import torch
    
    tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
    model = GPT2LMHeadModel.from_pretrained('gpt2')
    
    generated = tokenizer.encode("The Manhattan bridge")
    context = torch.tensor([generated])
    past = None
    
    for i in range(100):
        print(i)
        output, past = model(context, past=past)
        token = torch.argmax(output[0, :])
    
        generated += [token.tolist()]
        context = token.unsqueeze(0)
    
    sequence = tokenizer.decode(generated)
    
    print(sequence)
```
This is running perfectly. Then I try to run `gpt2-xl` model. 
I changed `tokenizer` and `model` loading code like following:

    ```
tokenizer = GPT2Tokenizer.from_pretrained("gpt2-xl")
    model = GPT2LMHeadModel.from_pretrained('gpt2-xl')
```



The `tokenizer` and `model` loaded perfectly. But I a getting error on the following line:

   ` output, past = model(context, past=past)`
The error is:

   ` RuntimeError: index out of range: Tried to access index 204483 out of table with 50256 rows. at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
`
Looking at error it seems that the embedding size is not correct. So I write the following line to specifically fetch the config file of `gpt2-xl`:

`    config = GPT2Config.from_pretrained("gpt2-xl")`
But, here `vocab_size:50257`
So I changed explicitly the value by:

  `  config.vocab_size=204483`
Then after printing the `config`, I can see that the previous line took effect in the configuration. But still, I am getting the same error. 

Comments:
Indeed, there was an error in the code, thank you for letting us know! I've patched it with fd639e5be31f83447c37cf79023fd98bac29f86c.

It is now [updated in the docs](https://huggingface.co/transformers/quickstart.html#using-the-past). Thanks!
Can't get gradients from TF TransformerXL model forward pass
Author: Morizeyao
Pull request: no
Labels: name: wontfix

## 🐛 Bug (Actually I'm not very sure if it's a bug or am I doing something wrong)

<!-- Important information -->

- Model I am using : Transformer-XL

- Language I am using the model on (English, Chinese....): Chinese

- The problem arise when using: my own modified scripts

- The tasks I am working on is: my own task or dataset

## To Reproduce

0. My current testing environment is a CPU machine and a Cloud GPU machine I rented.
1. Clone my repository(https://github.com/Morizeyao/Decoders-Chinese-TF2.0) and install requirements.
2. Copy contents from scripts folder to root folder.
3. Run prepare_data.sh
4. Run train_xl.sh
5. You will see the print result from line 107 of file train_transformer_xl.py shows that the gradients are all zero.
6. And the loss don't change during the training process.

## Expected behavior

I have tested TFGPT2 model from Transformers library, it worked fine (just run train_gpt2.py). Only TFTransfomerXL has this problem.

## Environment

* OS: macOS and Ubuntu
* Python version: 3.7
* PyTorch version: NA
* PyTorch Transformers version (or branch): 2.2.1
* Using GPU ? No
* Distributed of parallel setup ? No

## And...

- Thank you guys for this awesome project!


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Bert cross attention
Author: cristipp
Pull request: no

## ❓ Questions & Help

In the standard Transformer/Bert architecture, what is the intuition behind cross attention doing a weighted average over the encoder_hidden_states? What happens if we set the value layer to decoder hidden_state instead?

See the cross attention value layer being set to the encoder_hidden_states at [modeling_bert.py#L238](https://github.com/huggingface/transformers/blob/8c67b529f615cc24c46864b8323d2d47a15ccd58/src/transformers/modeling_bert.py#L238), and the weighted average being taken at [modeling_bert.py#L266](https://github.com/huggingface/transformers/blob/8c67b529f615cc24c46864b8323d2d47a15ccd58/src/transformers/modeling_bert.py#L266)


Comments:
I would imagine the idea is to incorporate a strong presence of the encoder hidden states - else the conditioning on the encoder might be weak. 

We do an attention without the encoder hidden states anyways before the cross attention.