noamaanMulla-03/transformers-issues huggingface dataset:
Documentation: Updating docblocks in optimizers.py
Author: DomHudson
Pull request: yes

## Summary
Updating documentation for the optimizer classes. I explicitly state that the scheduler classes are multiplying the optimizer's learning rate by a changing variable.

## Closes
https://github.com/huggingface/transformers/issues/1712

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1737?src=pr&el=h1) Report
> Merging [#1737](https://codecov.io/gh/huggingface/transformers/pull/1737?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e99071f10578adb0191288c1f3301e9a758d6200?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1737/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1737?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1737   +/-   ##
=======================================
  Coverage   83.93%   83.93%           
=======================================
  Files          94       94           
  Lines       13951    13951           
=======================================
  Hits        11710    11710           
  Misses       2241     2241
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1737?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/optimization.py](https://codecov.io/gh/huggingface/transformers/pull/1737/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL29wdGltaXphdGlvbi5weQ==) | `96.62% <ø> (ø)` | :arrow_up: |
| [transformers/tests/modeling\_common\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1737/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX2NvbW1vbl90ZXN0LnB5) | `73.08% <0%> (-0.59%)` | :arrow_down: |
| [transformers/tests/modeling\_tf\_common\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1737/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX3RmX2NvbW1vbl90ZXN0LnB5) | `96.8% <0%> (+1.59%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1737?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1737?src=pr&el=footer). Last update [e99071f...c958962](https://codecov.io/gh/huggingface/transformers/pull/1737?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thanks for this!
Scheduler documentation blocks subtly wrong
Author: DomHudson
Pull request: no

## Summary
Hi, I noticed that in `optimization.py` the way many of the schedulers describe the learning rate is slightly wrong.
For example, `WarmupLinearSchedule` says
```
        Linearly increases learning rate from 0 to 1 over `warmup_steps` training steps.
        Linearly decreases learning rate from 1. to 0. over remaining `t_total - warmup_steps` steps.
```
It actually multiplies the learning rate (set in the optimizer) by this amount. I understand that this is probably implied, but I think the documentation could be a bit clearer on this aspect!

Many thanks,
Dom


Comments:
Indeed, happy to welcome a PR on that if you want to fix this

Closed by https://github.com/huggingface/transformers/pull/1737
Can we fine tune GPT2 using multiple inputs?
Author: tyu0912
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hello. I don't think this is possible looking at the code, but to make sure is it possible to use multiple input texts to fine tune GPT-2? For example, I have 5 news articles. Do I submit them all in one document or should I separate it out? I feel like the latter makes more sense especially if the articles are all written differently. 

Thanks!


Comments:
You can merge them in the same document but add a token that indicates it's the beginning and end of each article. This will allow the generator to understand the concept of a document being a group of text and there are many of those groups in your document.

**Example (assuming one line is one document):**

```
<sod> Today was a good day! <eod>
<sod> Tomorrow is going to be even better <eod>
```

Then at the text generation step (e.g. using ```run_generation.py```), you can specify a ```stop_token``` argument if you want to generate one document at a time.

Fantastic! Thank you for sharing @enzoampil 👍
Released OpenAI GPT-2 1.5B model
Author: TheEdoardo93
Pull request: no

## 🚀 Feature

- [ ] Blog: [https://openai.com/blog/gpt-2-1-5b-release/](url)
- [ ] Code: [https://github.com/openai/gpt-2](url)
- [ ] Dataset: [https://github.com/openai/gpt-2-output-dataset](url)

## Motivation

A bigger model for text generation which is more human than the OpenAI GPT-2 774M parameters

## Additional context

Maybe this is not the most requested feature by AI researchers. Maybe, it's better to implement and standardize **new** NLG models!

Comments:
Hi, GPT-2 XL was added yesterday with commit d7d36181fdefdabadc53adf51bed4a2680f5880a

Yes, sorry for opening a new issue! I didn't see this commit. I'll close the issue! Thank you for your time!
Fixing models inputs_embeds 
Author: julien-c
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1746?src=pr&el=h1) Report
> Merging [#1746](https://codecov.io/gh/huggingface/transformers/pull/1746?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f88c104d8f79e78a98c8ce6c1f4a78db73142eab?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `91.3%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1746/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1746?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1746      +/-   ##
==========================================
+ Coverage   84.03%   84.03%   +<.01%     
==========================================
  Files          94       94              
  Lines       14021    14032      +11     
==========================================
+ Hits        11782    11792      +10     
- Misses       2239     2240       +1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1746?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tests/conftest.py](https://codecov.io/gh/huggingface/transformers/pull/1746/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL2NvbmZ0ZXN0LnB5) | `93.33% <100%> (+3.33%)` | :arrow_up: |
| [transformers/modeling\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/1746/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2Rpc3RpbGJlcnQucHk=) | `95.84% <100%> (+0.01%)` | :arrow_up: |
| [transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/1746/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2JlcnQucHk=) | `87.77% <75%> (ø)` | :arrow_up: |
| [transformers/tests/modeling\_bert\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1746/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX2JlcnRfdGVzdC5weQ==) | `96.5% <91.66%> (-0.43%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1746?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1746?src=pr&el=footer). Last update [f88c104...fd29a30](https://codecov.io/gh/huggingface/transformers/pull/1746?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
How to train from scratch
Author: anandhperumal
Pull request: no

I would like to train the model from scratch. 
How can I drop the trained weight? using the same architecture for Gpt2

Comments:
If you want to randomly initialize a model simply initialize it via its constructor rather than from the `from_pretrained` method:

```py
from transformers import GPT2Config, GPT2Model

config = GPT2Config()  # define your configuration here
model = GPT2Model(config)  # Initialize your model from your config
```

@LysandreJik Thanks for the input.
I did something like this 
```
    config = GPT2Config(vocab_size)
    model = GPT2Model(config)
```
Apart from vocab size, I'm keeping everything else to default value how do I make sure that it doesn't have any pre-trained value?

The values are only loaded if your instantiate the model by calling ˋGPT2Model.from_pretrained`, so you’re fine 🙂

@rlouf Thanks
How to add new pre-trained model pytorch-transformers
Author: ksopyla
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Pytorch-transformers is a great library. I like that it does one thing, give access to the pre-trained SOTA models for NLP. 
I and my team we want to help and start contributing: 
- at first, want to add Polish BERT model like https://github.com/huggingface/pytorch-transformers/pull/688

But we do not know how to do this :( 
Is there any guide or procedure that shows what should be changed in order to add a new model?
We would be grateful if someone guides us.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Subtokens in BPE in GPT2
Author: weiguowilliam
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi, I have a question.

For a word represented by more than 1 subtoken in BPE, I wonder what's the principle to divide it into subtokens?

I use GPT2 small version for example. "bookshelf" is divided into "books","he" and "if".(index for encoder is [3835, 258, 1652]).  But actually "book" and "shelf" are represented by single subtoken.(book is 1492, shelf is 18316). Why is "bookshelf" divided into 3 parts instead of "book" and "shelf"?

Thank you!

Comments:
BPE is a data compression algorithm, the result depends on the statistical properties of the corpus of text on which it has been trained.

You can start with this article https://leimao.github.io/blog/Byte-Pair-Encoding/ to get more of an intuition of how it works.

> BPE is a data compression algorithm, the result depends on the statistical properties of the corpus of text on which it has been trained.
> 
> You can start with this article https://leimao.github.io/blog/Byte-Pair-Encoding/ to get more of an intuition of how it works.

Thank you! That helps!
Error when trying to reuse hidden states in CTRL
Author: bilal2vec
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): CTRL

Language I am using the model on (English, Chinese....): English

The problem arise when using:

My own script, the colab link is available [here](https://colab.research.google.com/drive/143T4sBda4r2nDYzmuNwi-ZFTbhJWfeOW)

The stack trace is:

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-9-ac2d93f8c410> in <module>()
      3   for i in range(3):
      4     print(i)
----> 5     logits, past = model(**inputs, past=past)
      6     logits = logits[0, -1]
      7 

8 frames
/usr/local/lib/python3.6/dist-packages/transformers/modeling_ctrl.py in scaled_dot_product_attention(q, k, v, mask, attention_mask, head_mask)
     64 
     65     if mask is not None:
---> 66         scaled_attention_logits += (mask * -1e4)
     67 
     68     if attention_mask is not None:

RuntimeError: The size of tensor a (13) must match the size of tensor b (7) at non-singleton dimension 3
```

The tasks I am working on is:

Generating text with CTRL

## To Reproduce

Just run the colab from the link I posted above

The main part of the code is:

```
input_ids = torch.tensor(tokenizer.encode("Links Hello, my dog is cute")).unsqueeze(0).to(device)  
inputs = {'input_ids': input_ids}

with torch.no_grad():
  past = None
  for i in range(3):
    print(i)
    logits, past = model(**inputs, past=past)
    logits = logits[0, -1]
    
    next_token = logits.argmax()
    input_ids = torch.cat([input_ids, next_token.view(1, 1)], dim=1)
    
    inputs = {'input_ids': input_ids}
```

## Expected behavior

passing in `past` should not throw an error and should speed up generation

## Environment

* OS: Linux
* Python version: 3.6.8
* PyTorch version: 1.3.0+cu100
* PyTorch Transformers version (or branch): 2.1.1
* Using GPU: yes
* Distributed of parallel setup: No
* Any other relevant information: I'm running on an extended memory colab instance with a K80


Comments:
The same error occurs with the library installed with `git clone` (_master_ version) + torch v1.3.0 + python v3.6.8.
[Here](https://colab.research.google.com/drive/1nawWX6Lrfh9ZVKyRfTLgFSIG355xkPRy#scrollTo=n93UZjq5EIE_) is a more verbose version of the Colab Notebook posted by @bkkaggle with Google Colab.

I've just pushed a fix on the branch `fix-ctrl-past`. It should be in the next release.

Thanks, closing
How to stop wordpiece-tokenizing in BertTokenizer?
Author: RichardHWD
Pull request: no

Default BertTokenizer may split one word into two parts or three parts, which is harmful to token labeling task because it will make sentence length lager than input length. It is difficult to fuse these parts back to one representation.
So, how to stop this operation in BertTokenizer? Or there are alternative Tokenizers without spliting words?


Comments:
No, that's how the model works.

Short of creating your own model (with you own custom tokenizer) there's no way to do what you want to do.

Okkkkkk, it is unreasonable ....
Changing the _read_tsv method in class DataProcessor
Author: Sudeep09
Pull request: no
Labels: name: wontfix

## 🚀 Feature
I would request changing the class method to the following:
@classmethod
def _read_tsv(cls, input_file, quotechar=None)
        """Reads a tab separated value file."""
        lines = []
        df = pd.read_csv(input_file, delimiter='\t')
        for line in (df.values):
            lines.append(line)
        return lines

## Motivation

The reader used to read the files incorrectly.


Comments:
Hi, we need more information about what script you are talking about.

The file path is: /examples/utils_glue.py
The class is DataProcessor(object):

@classmethod
def _read_tsv(cls, input_file, quotechar=None)
"""Reads a tab separated value file."""
lines = []
df = pd.read_csv(input_file, delimiter='\t')
for line in (df.values):
lines.append(line)
return lines

We also need a lot more details and a clear example of why you think the reader used to read the files incorrectly.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Problem with optimizers after migration
Author: ramild
Pull request: no
Labels: name: wontfix

## 📚 Migration

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): Russian

The problem arise when using:
The optimizers. I tried the default parameters from the example with max_grad_norm = 1.0 and lr = 2e-5.

```
warmup_proportion = float(num_warmup_steps) / float(num_total_steps)  # 0.1

### Previously BertAdam optimizer was instantiated like this:
optimizer = BertAdam(model.parameters(), lr=lr, schedule='warmup_linear', warmup=warmup_proportion, t_total=num_total_steps)
### and used like this:
for batch in train_data:
    loss = model(batch)
    loss.backward()
    optimizer.step()

### In PyTorch-Transformers, optimizer and schedules are splitted and instantiated like this:
optimizer = AdamW(model.parameters(), lr=lr, correct_bias=False)  # To reproduce BertAdam specific behavior set correct_bias=False
scheduler = WarmupLinearSchedule(optimizer, warmup_steps=num_warmup_steps, t_total=num_total_steps)  # PyTorch scheduler
### and used like this:
for batch in train_data:
    loss = model(batch)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)  # Gradient clipping is not in AdamW anymore (so you can use amp without issue)
    scheduler.step()
    optimizer.step()
```


The tasks I am working on is:
* [ ] my own task or dataset: private dataset on comments from client support

Details of the issue:

After migration I see that the model converges more slowly and failed to obtain such an accuracy as it was before. In my multi-class classification dataset I get about 0.59 accuracy while the previous version resulted in about 0.63 accuracy after convergence. Do the optimizers in both versions are equivalent? If not, how can I make them absolutely the same?

## Environment

* OS: Ubuntu
* Python version: 3.6
* PyTorch version: 1.0
* PyTorch Transformers version (or branch): 1.0
* Using GPU ? Yes
* Distributed of parallel setup ? No

Comments:
You should get the same behavior than `BertAdam` by setting `correct_bias=False` in `AdamW` and using the `WarmupLinearSchedule` together with it.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Suggestion: exception handling for out-of-vocab in pretrained model
Author: sdeva14
Pull request: no
Labels: name: Need more information

Dear concern,

I appreciate your favor for public implementation.
As you know, all NLP people have an interest in applying your gorgeous model to every NLP problem.

I am writing this to suggest to add exception handling or warning message about out-of-vocabulary when a pretrained model is used.

I had been suffered by an error message for two days with checking every my code, naturally, I had believed it is my programming problem again :)
Then I discovered that it is out-of-vocab problem.
The pretrained BERT model has vocab size 28996, but my dataset with BERT subword tokenizer has more than 30000 vocab.
Then it is an error, when the BERT model meets an index higher than 28996.

If we know the problem, a solution is quite simple, filtering or replacing it to UNK index.
Error messages are caused from somewhat embedding parts in CPU mode, but it was unclear to identify GPU mode, thus other people might be suffered like me.

I know the pretrained model is not designed for application, long documents with unusual vocabulary.
However, I believe many people will appreciate your favor if you add exception handling or warning message.
If you already have recognized the issue, please ignore this message :)


Best
Sungho

Comments:
Hi Sungho,
This should already be taken care of by the BertTokenizer which defaults on the `unk` token when a sub-word is not in the vocabulary (see [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/tokenization.py#L372)).
What BERT model are you using?
Do you have a simple example you can share so I can try to reproduce the behavior?

Hi Tomas,

It is really shame to describe, I think it is just my typo problem again :(
Please ignore this issue.

You are absolutely right, there is an exception handling.
Just I loaded a wrong model in the BertModel after tokenizing.
"bert-base-cased" in the model part, Instead of "bert-base-uncased", which has a smaller vocab size than "uncased".

I am so sorry to waste your time, and appreciate your favor again! :)

Best
Sungho

Ok make sense, good to know.
So there is no problem in the end?

Yep, only my embarrassing here :)

Though there is no performance change, but I believe it is not a problem of pretrained model at all, but 
 just different application.

Thus, no problem for implementation at all, I appreciate it!

Ok, let's close the issue then. Feel free to open a new one if you have other problems.

May I know why cased vocabulary is smaller than uncased?
How to add some parameters in gpt-2 (in attention layer) and initialize the original gpt-2 parameters with pre-trained model and the new introduced parameters randomly?
Author: fabrahman
Pull request: no
Labels: name: wontfix

Hi,

I want to add some weight matrices inside attention layers of gpt-2 model. However, I want to initialize all original parameters with pre-trained gpt-2 and the newly added ones randomly.
Can someone guide me how that's possible or point me to the right direction?


Thanks

Comments:
You should make a class deriving from `GPT2Model` in which:
- the `__init__` method
   * calls its super class `__init__` method (to add the original GPT2 modules),
   * you then add the new modules (with names differents from GPT2 original attributes so you don't overwrite over them).
   * you call `self.init_weights()` at the end to initalize your weights (check the `init_weights` method in `GPT2PreTrainedModel` to be sure it initialize as you want to)
- the  `forward` method has to be written as you want the forward pass to be.

You can then load the pretrained weights and initialize your newly added weights just by doing the usual `model = MyGPT2Model.form_pretrained('gpt2')`.

Thanks @thomwolf . Just to clarify, does that mean if I need to change the attention layer a little bit, then I have to make three classes derived from ``` GPT2Model``` , ```Block``` ,and ```Attention```? And for that, can I use the original Attention modules inside my forward pass of myAttention?

Should it be something like following?
```
class myAttention(Attention):
    def __init__(self, nx, n_ctx, config, scale=False):
        super(myAttention, self).__init__()

    def forward(): ### my customized forward pass


class myBlock(Block):
    def __init__(self, n_ctx, config, scale=False):
        super(myBlock, self).__init__()
    def forward(...):   ### my customized forward pass

class myGPT2Model(GPT2Mode):
    def __init__(self, config):
        super(myGPT2Model, self).__init__(config)
        ....
        self.apply(self.init_weights)
    def forward(...).  ### my customized forward pass
```


Maybe but it depends on what you put in the `....` parts

@thomwolf Is it right that I have to have three separate classes each derived from ```GPT2Model```, ```Block``` and ```Attention``` ?
In general, I want to have one additional input to myGPT2Model forward method and I want to incorporate that in the Attention computation.
What I did is I added that aux input to fw of ```myGPT2Model```, I called the block inside myGPT2Model forward with original and aux input,
Then in the myBlock forward method, I called Attention with the two inputs.

Probably right.

Maybe the easiest in your case would be to copy the `modeling_gpt2` file in whole and modify what you need in the copy.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.