noamaanMulla-03/transformers-issues huggingface dataset:
Using GPU for gpt2-xl
Author: samer-noureddine
Pull request: no

I want to use gpt2-xl using my PC's GPU (NVIDIA GeForce GTX 1070). The usual way to do this is via conda, but it appears that the latest version of gpt-2 is not available for conda. For example, the following installation commands don't work:
```
conda install transformers
conda install git+https://github.com/huggingface/transformers
```
The latest version of transformers for conda is a[ month old,](https://anaconda.org/conda-forge/transformers) and therefore doesn't include gpt2-xl. How can I use my machine's GPU to run gpt2-xl without using conda? Will there be conda support soon?


Comments:
We have not released a new Pypi/conda version yet, but will do so in the next few days. You can install from source if you want to use GPT2-xl before that!

Via docker:
https://github.com/huggingface/transformers/blob/master/docker/Dockerfile

Via pip:
https://github.com/huggingface/transformers/blob/master/docker/Dockerfile
or 
https://github.com/huggingface/transformers/issues/1837#issuecomment-554594306
Correction for the tuple problem
Author: oleg-sharethis
Pull request: yes

This fixes the problem described and corrected in  https://github.com/huggingface/transformers/issues/831

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1890?src=pr&el=h1) Report
> Merging [#1890](https://codecov.io/gh/huggingface/transformers/pull/1890?src=pr&el=desc) into [xlnet](https://codecov.io/gh/huggingface/transformers/commit/1b35d05d4b3c121a9740544aa6f884f1039780b1?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1890/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1890?src=pr&el=tree)

```diff
@@          Coverage Diff          @@
##           xlnet   #1890   +/-   ##
=====================================
  Coverage   78.9%   78.9%           
=====================================
  Files         34      34           
  Lines       6181    6181           
=====================================
  Hits        4877    4877           
  Misses      1304    1304
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1890?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1890?src=pr&el=footer). Last update [1b35d05...5365fbd](https://codecov.io/gh/huggingface/transformers/pull/1890?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi, we don't have finetune_on_pregenerated.py in the examples anymore but a simpler script, `run_lm_finetuning`. Closing for now.
fixes issue with unrecognized arguments for AdamW
Author: oleg-sharethis
Pull request: yes

as suggested in https://github.com/huggingface/transformers/issues/830

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1891?src=pr&el=h1) Report
> Merging [#1891](https://codecov.io/gh/huggingface/transformers/pull/1891?src=pr&el=desc) into [xlnet](https://codecov.io/gh/huggingface/transformers/commit/1b35d05d4b3c121a9740544aa6f884f1039780b1?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1891/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1891?src=pr&el=tree)

```diff
@@          Coverage Diff          @@
##           xlnet   #1891   +/-   ##
=====================================
  Coverage   78.9%   78.9%           
=====================================
  Files         34      34           
  Lines       6181    6181           
=====================================
  Hits        4877    4877           
  Misses      1304    1304
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1891?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1891?src=pr&el=footer). Last update [1b35d05...dcbea8c](https://codecov.io/gh/huggingface/transformers/pull/1891?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Same as #1890
Where is pytorch-pretrained-BERT?
Author: Foehnc
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

As the title shows, where is pytorch-pretrained-BERT? Please tell me the path, THX.

Comments:
`pytorch-pretrained-BERT` is this library, but four or five months ago. It evolved into `pytorch-transformers` as more models were added to the library, before becoming `transformers` as we now have a front-end for both pytorch and tensorflow.

is this still an issue?

I don't think so. In my opinion, @ShallTearchen would know where it goes pytorch-pretrained-BERT, but I think that @LysandreJik explains very well the transformation of this library to Transformers.
In my opinion, I'll close this "issue"!

> is this still an issue?
tensorflow2.0 does not has mean, but reduce mean
Author: andompesta
Pull request: no

## 🐛 Bug
tensorflow2.0 does not has a mean function but only a reduce_mean().
```
output = tf.mean(hidden_states, axis=1)
```
should be:
```
output = tf.reduce_mean(hidden_states, axis=1)
```
<!-- Important information -->

Model I am using: TFXLMForSequenceClassification with summary_type == 'mean'

Language I am using the model on (English, Chinese....): any

The problem arise when using:
when loading the model, due to the lazy initialisation of tf2.0 you have to run the model with dummy input.
```
TFXLMForSequenceClassification.from_pretrained(os.path.join(args.model_path, "xlm-mlm-17-1280-tf_model.h5"), config=config)
```

The tasks I am working on is:
* [ x] my own task or dataset: I want to finetune the model on a custom english dataset and transfer it to multiple language

## To Reproduce

Steps to reproduce the behavior:
1. load config file and add summary info
```
config = XLMConfig.from_pretrained(os.path.join(args.data_path, "xlm-mlm-17-1280-config.json"))
config.summary_use_proj = True
config.summary_type = 'mean'
config.summary_proj_to_labels = True
config.num_labels = len(LABELS)
config.summary_activation = "tanh"
```
2. load the model
```
model = TFXLMForSequenceClassification.from_pretrained(os.path.join(args.model_path, "xlm-mlm-17-1280-tf_model.h5"), config=config)
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

You should see an error stating that tf2.0 does not has a mean function.
File "transformers/modeling_tf_utils.py", line 442:
    ``output = tf.mean(hidden_states, axis=1)`` should be ``output = tf.reduce_mean(hidden_states, axis=1)``

```AttributeError: module 'tensorflow' has no attribute 'mean'```


## Environment

* OS: Ubuntu16.04
* Python version: 3.7.5
* Tensorflow version: 2.0
* Transformers version (or branch): 2.1.1
* Using GPU: yes
* Distributed of parallel setup : not distributed, only single GPU

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
This bug has been reviewed and corrected in this PR #1876!
Close this issue.
Bart Tokenizer treat symbols in a word as a new word. 
Author: billchenxi
Pull request: no

## 🐛 Bug

<!-- Important information -->

The model I am using is Bart:

The problem arises when using:
* [ ] `tokenizer.encode`
* [ ] `tokenizer.decode`
The tasks I am working on is:
* [ ] Encode a string, then decode it back.

## To Reproduce

```
import torch
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')

test_string = 'text with percentage%'

# encode Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary.
input_ids = tokenizer.encode(test_string)
output = tokenizer.decode(input_ids)
print(output)
```

`>>> text with percentage %`

## Expected behavior

It should be `text with percentage%`, which treats the symbol in the word as one word.


## Environment

* OS: MacOS
* Python version: 3.7
* PyTorch version: 1.3
* PyTorch Transformers version (or branch): Master


Comments:
Hi, this is due to the original BERT's tokenization. You can try it using the [google-research's implementation](https://github.com/google-research/bert):

```py
raw_text = 'text with percentage%'
tokenizer = tokenization.FullTokenizer(vocab_file=vocab_file, do_lower_case=True)
tokens = tokenizer.tokenize(raw_text)

print(tokens)  # ['text', 'with', 'percentage', '%']
```

The goal is to be as close as possible to the original implementation, hence the similar behavior concerning special tokens.
How do I train OpenAIGPTDoubleHeadsModel from scratch?
Author: g-karthik
Pull request: no

## ❓ Questions & Help

It looks like I cannot train `OpenAIGPTDoubleHeadsModel` from scratch because it necessarily needs to be initialized using the `from_pretrained()` method. How should I initialize this model to be able to train it from scratch?

As a hack, perhaps I could initialize it using `from_pretrained()` and then reset the pre-trained initialization so any fine-tuning would essentially be the same as pre-training?

Comments:
You can initialize a model by calling its constructor with a configuration:

```py
from transformers import OpenAIGPTDoubleHeadsModel, OpenAIGPTConfig

config = OpenAIGPTConfig()
model = OpenAIGPTDoubleHeadsModel(config)
```

Cool, so if I instantiate the model with the default config, it will use the same vocab as the pre-trained model but it won't initialize the pre-trained weights, is that correct?

Also, could you please point me to where the model weights are initialized when we don't use pre-trained weights?

If you instantiate a model with a default configuration without specifying any argument, it will instantiate your model according to the default configuration values, [visible here](https://github.com/huggingface/transformers/blob/master/transformers/configuration_openai.py#L59).

The model weights are initialized in the `OpenAIGPTPreTrainedModel`'s [`_init_weights` method](https://github.com/huggingface/transformers/blob/master/transformers/modeling_openai.py#L267).
AlbertPreTrainedModel class is not available in release v2.2.0
Author: bugface
Pull request: no

## ❓ Questions & Help

In the release v2.2.0, the AlbertForSequenceClassification class inherits the AlbertPreTrainedModel class as "class AlbertForSequenceClassification(AlbertPreTrainedModel)". 

However, in the doc, this pre-trained model is not documented. And in the released v2.2.0 transformer, the AlbertPreTrainedModel class is not available to be imported. 

This is not a big issue since we can use BertPreTrainedModel class instead (like RoBERTa). But it should be consistent. Especially in the doc under Albert, there is a class called TFBertForPretraining which makes confusion for users.  



Comments:
Hi, indeed there was a mistake with `TFBertForPretraining` referenced under the ALBERT documentation. This was fixed with 3616209.

The `PreTrainedModel` class is not available for ALBERT, which is the same for CTRL, GPT, GPT-2, DistilBERT, CamemBERT, TransformerXL, XLM and XLNet. 

Why is that class useful for your use-case, seeing as it's a simple wrapper over `PreTrainedModel` with a few overridden attributes?


> Hi, indeed there was a mistake with `TFBertForPretraining` referenced under the ALBERT documentation. This was fixed with [3616209](https://github.com/huggingface/transformers/commit/361620954acf16b27727d763a591257b03f90b5d).
> 
> The `PreTrainedModel` class is not available for ALBERT, which is the same for CTRL, GPT, GPT-2, DistilBERT, CamemBERT, TransformerXL, XLM and XLNet.
> 
> Why is that class useful for your use-case, seeing as it's a simple wrapper over `PreTrainedModel` with a few overridden attributes?

I completely agree we can use PreTrainedModel or BertPreTrainedModel instead.
 
The question is that I do see the implementation of the class AlbertPreTrainedModel(PreTrainedModel)  in the source code (transformers/modeling_albert.py line 313) but I cannot import it. It seems that it is not included in the released version. I just feel it is weird.


Yes, it is not importable as it is an internal used by different models, but I fail to see a use-case where it would be useful for the library users.

Why is that class useful for your use-case, seeing as it's a simple wrapper over `PreTrainedModel` with a few overridden attributes?

I think we could add all the `XXXPretrainedModel` in `__init__` indeed. Would make it easier for people to build custom-made models that can load pretrained checkpoints as well.

Fixed on master
fixed XLNet attention output for both attention streams whenever target_mapping is provided
Author: roskoN
Pull request: yes

XLNet uses two separate attention streams, i.e. there are two separate tensors for representing the model's attention. Both of them need to have their dimensions permuted.

The problem has been described in #1994 .

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2007?src=pr&el=h1) Report
> Merging [#2007](https://codecov.io/gh/huggingface/transformers/pull/2007?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b0ee7c7df3d49a819c4d6cef977214bd91f5c075?src=pr&el=desc) will **increase** coverage by `0.03%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2007/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2007?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2007      +/-   ##
==========================================
+ Coverage   84.05%   84.09%   +0.03%     
==========================================
  Files         105      105              
  Lines       15555    15570      +15     
==========================================
+ Hits        13075    13093      +18     
+ Misses       2480     2477       -3
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2007?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tests/modeling\_xlnet\_test.py](https://codecov.io/gh/huggingface/transformers/pull/2007/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX3hsbmV0X3Rlc3QucHk=) | `96.42% <100%> (+0.29%)` | :arrow_up: |
| [transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/2007/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3hsbmV0LnB5) | `74.22% <100%> (+0.61%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2007?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2007?src=pr&el=footer). Last update [b0ee7c7...76c0bc0](https://codecov.io/gh/huggingface/transformers/pull/2007?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


That's great, thanks for fixing the issue! Looks good to me.

Yes, this is great, thanks a lot @roskoN!
typo fix on the docs as per Pytorch v1.1+
Author: AdityaSoni19031997
Pull request: yes

https://github.com/huggingface/transformers/issues/2010

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2011?src=pr&el=h1) Report
> Merging [#2011](https://codecov.io/gh/huggingface/transformers/pull/2011?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b0ee7c7df3d49a819c4d6cef977214bd91f5c075?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2011/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2011?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2011   +/-   ##
=======================================
  Coverage   84.05%   84.05%           
=======================================
  Files         105      105           
  Lines       15555    15555           
=======================================
  Hits        13075    13075           
  Misses       2480     2480
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2011?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2011?src=pr&el=footer). Last update [b0ee7c7...c356290](https://codecov.io/gh/huggingface/transformers/pull/2011?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks!

Thanks a lot to the creators (and contributors) of this amazing lib for making our lives easier!
[CamemBert] Tokenizer function add_tokens doesn't work
Author: breeky-dev
Pull request: no

## ❓ Questions & Help

Hi, 

I am trying to add new tokens to the CamemBert tokenizer, but when I run the function tokenizer.add_tokens, it doesn't seem to add any token at all :

`from transformers import CamembertTokenizer`
`tokenizer = CamembertTokenizer.from_pretrained('camembert-base')`
`tokenizer.add_tokens(['notfrenchword'])`

`Out[12]: 0`

Whereas with Bert model it works perfectly. Is this a bug or am I doing something wrong ?

Thanks


Comments:
This method is **not** implemented into the CamemBERT tokenizer, at the moment.

> ##  Questions & Help
> Hi,
> 
> I am trying to add new tokens to the CamemBert tokenizer, but when I run the function tokenizer.add_tokens, it doesn't seem to add any token at all :
> 
> `from transformers import CamembertTokenizer`
> `tokenizer = CamembertTokenizer.from_pretrained('camembert-base')`
> `tokenizer.add_tokens(['notfrenchword'])`
> 
> `Out[12]: 0`
> 
> Whereas with Bert model it works perfectly. Is this a bug or am I doing something wrong ?
> 
> Thanks

Hi,
This method is actually implemented (it's a method in the base class of all tokenizer).
The reason is was failing in the present case is that the original fairseq model has a bunch of duplicate tokens in the dictionary:
```
import torch
camembert = torch.hub.load('pytorch/fairseq', 'camembert.v0')
list(camembert.task.source_dictionary[i] for i in range(10))
>>> ['
', '<pad>', '
', '<unk>', '<unk>', '
', '
', ',', '▁de', '.']
```
fixing this in #2065
Typo in modeling_albert.py for mask_token
Author: blmoistawinde
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Albert

Language I am using the model on (English, Chinese....): English

## To Reproduce

```
tokenizer_class, pretrained_weights = AlbertTokenizer, "albert-base-v1"
tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
print(tokenizer.mask_token)   # [MASK]>
print(tokenizer.mask_token_id)  # 1 (same as  <unk>)
```

I think the typo lies here https://github.com/huggingface/transformers/blob/fbaf05bd92249b6dd961f5f8d60eb0892c541ac8/transformers/tokenization_albert.py#L69


<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

mask_token should be "[MASK]" and mask_token_id should be 4

## Environment

* OS: Windows 10
* Python version: 3.6.9
* PyTorch version:  1.2.0
* PyTorch Transformers version (or branch): 2.2.0


Comments:
Indeed, good catch, thanks! Fixed on master.
Fix run_glue.py on QNLI part
Author: adamluo1995
Pull request: yes
Labels: name: wontfix

In QNLI task, the ids should be truncated is the pair cuz that is the huge one. Or we can't load QNLI dataset successfully.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Pr for pplm
Author: mimosavvy
Pull request: yes

Updated paper link and better commands to generate samples.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2060?src=pr&el=h1) Report
> Merging [#2060](https://codecov.io/gh/huggingface/transformers/pull/2060?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/5bfcd0485ece086ebcbed2d008813037968a9e58?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2060/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2060?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2060   +/-   ##
=======================================
  Coverage   83.58%   83.58%           
=======================================
  Files         105      105           
  Lines       15568    15568           
=======================================
  Hits        13012    13012           
  Misses       2556     2556
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2060?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2060?src=pr&el=footer). Last update [5bfcd04...12d18d4](https://codecov.io/gh/huggingface/transformers/pull/2060?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
How to convert the ALBERT tfhub model to pytorch model?
Author: jellying
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I want to apply ALBERT to other QA datasets. But the first question is how to convert the tf_hub model. I downloaded the tfhub model from the repo "google research".  The script ```convert_albert_original_tf_checkpoint_to_pytorch.py``` need the parameter ```--tf_checkpoint_path```. I attempted the directory 'variables', 'assets' and the root directory but all failed.  I haven't solve it by document. Is there some demos for ALBERT?

Comments:
Hi, if you ran the script `run_pretraining.py` in the original ALBERT repo, you should have put as argument an `--output_dir=dir`. In that directory should be several files, among which `model.ckpt-xxx.index`, `model.ckpt-xxx.meta`, `checkpoint` and `model.ckpt-xxx.data-xxxxx-of-xxxxx`.

You can pass this as argument to `convert_albert_original_tf_checkpoint_to_pytorch`: `--tf_checkpoint_path=dir/model.ckpt-xxx`.

A few changes to the script were done today so you might want to install from source to be sure it loads fine.

But I don't have the resources to pretrain my own ALBERT model, I just want to fine-tune the pretrained ALBERT-base model for my task. Are there some other methods to use the google's tfhub model in pytorch? Or if there is other pretrained ckpt type models I can download.

Yeah you can just load them using our API:

```py
from transformers import AlbertModel

model = AlbertModel.from_pretrained("albert-base-v1")
```