noamaanMulla-03/transformers-issues huggingface dataset:
doc: fix apparent copy-paste error in docstring
Author: gthb
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5626?src=pr&el=h1) Report
> Merging [#5626](https://codecov.io/gh/huggingface/transformers/pull/5626?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/fa5423b1695cd24856bcff47214172e0f540d924&el=desc) will **decrease** coverage by `0.23%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5626/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5626?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5626      +/-   ##
==========================================
- Coverage   77.79%   77.56%   -0.24%     
==========================================
  Files         145      145              
  Lines       25355    25355              
==========================================
- Hits        19726    19667      -59     
- Misses       5629     5688      +59     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5626?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/commands/train.py](https://codecov.io/gh/huggingface/transformers/pull/5626/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb21tYW5kcy90cmFpbi5weQ==) | `0.00% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5626/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `64.90% <0.00%> (-33.90%)` | :arrow_down: |
| [src/transformers/tokenization\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/5626/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxuZXQucHk=) | `66.66% <0.00%> (-23.43%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5626/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `83.20% <0.00%> (-2.76%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/5626/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9yb2JlcnRhLnB5) | `93.36% <0.00%> (+49.37%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5626?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5626?src=pr&el=footer). Last update [fa5423b...959b687](https://codecov.io/gh/huggingface/transformers/pull/5626?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Add beta 1 and beta 2 option in `TrainingArguments` for `AdamW` optimizer.
Author: PhilipMay
Pull request: no

I want to set the Adam Optimizers beta 2 to 0.98 - this is because I want to train a new RoBERTa LM. The paper sais, that it improves stability. The default is 0.999 and it can not be set in `TrainingArguments`.

Could you please add the option to specify beta 1 and 2 for AdamW in the `TrainingArguments`? `adam_epsilon ` can already be specified. If you want me to I can provide a PR.

What do you think?

Comments:
I'm not sure we would like to add this to the `TrainingArguments`. If we add all possible params this could quickly explode. Note that you can instantiate your own optimizer and pass it here: https://github.com/huggingface/transformers/blob/7096e47513127d4f072111a7f58f109842a2b6b0/src/transformers/trainer.py#L158

Also pinging @julien-c here.

Well - my arguments for this change is that adam_epsilon is possible to be set set and so beta 1 and 2 should also be possible to be set. Especialy because the RoBERTa paper suggests an others setting then default. 

2nd argument is that it is not that easy to instantiate your own optimizer because there is a dependency to `model`. See here: https://github.com/huggingface/transformers/blob/7096e47513127d4f072111a7f58f109842a2b6b0/src/transformers/trainer.py#L326-L335


Closing this in favor of #5592
docs(wandb): explain how to use W&B integration
Author: borisdayma
Pull request: yes

Documentation on how to use W&B integration has been added to clear up confusion on how to customize logging.

fix #5262

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5607?src=pr&el=h1) Report
> Merging [#5607](https://codecov.io/gh/huggingface/transformers/pull/5607?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/40d98ebf50c4662bcd6dce6395bbed0b2142ea52&el=desc) will **increase** coverage by `1.23%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5607/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5607?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5607      +/-   ##
==========================================
+ Coverage   76.88%   78.11%   +1.23%     
==========================================
  Files         145      145              
  Lines       25351    25351              
==========================================
+ Hits        19491    19804     +313     
+ Misses       5860     5547     -313     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5607?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/5607/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9yb2JlcnRhLnB5) | `43.98% <0.00%> (-49.38%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5607/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `79.26% <0.00%> (-0.34%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/5607/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9tb2JpbGViZXJ0LnB5) | `96.77% <0.00%> (+73.38%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5607?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5607?src=pr&el=footer). Last update [40d98eb...f38d33b](https://codecov.io/gh/huggingface/transformers/pull/5607?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Cannot control wandb metadata when running examples/seq2seq/finetune.py
Author: sshleifer
Pull request: no
Labels: name: seq2seq

name: wandb

There is no easy way to control the wandb project_name kwarg.
How can we facilitate this without massive number of command line args?


Comments:
You should be able to do it with environment variables. See [these docs](https://docs.wandb.com/library/environment-variables)
Let me know if that solves the issue.

The reason of the current implementation (and auto init of wandb) was to be able to instrument all example scripts without having to add custom wandb code.

There is [documentation on W&B related to transformers](https://docs.wandb.com/library/integrations/huggingface) that probably need to be updated to add this detail. We should probably also find a place to add documentation related to W&B integration in transformers repo. Let me know if I can be of any help.

You could make `examples/wandb.md` with information and link to it from `examples/README.md`? Or just link to [this](https://docs.wandb.com/library/integrations/huggingface) in `examples/README.md`

For the lightning `WandbLogger` integration, does `$WANDB_PROJECT` take precedence over passing `WandbLogger(project='project_name')`?

Thanks, that's a great idea!

For the lightning integration, any parameter you pass explicitly in `WandbLogger` should take precedence.


@sshleifer it seems that the pytorch-lightning integration is commonly used so I also added a reference to it in my PR.
Just curious, is it because you cannot do distributed computing with `Trainer` on multiple machines?

Would it make sense to add an easier integration on `lightning_base` with a similar logic to `Trainer` and `TFTrainer`, ie by using `WandbLogger` on `generic_train` whenever wandb is installed and logged in (and ignore it otherwise)?

As long as we can avoid
(1) generating a lot of logs when I run the unittests on my local machine
(2) enabling gradient watching by default

That sounds like a good change to me. @patil-suraj what do you think? do you use wandb?

Yes, adding wandb  integration on `lightning_base` makes sense to me given wandb is enabled by default in `Trainer` and `TFTraner` , this will enable wandb logging for `run_pl_ner`  and `run_pl_glue`

With `Trainer`  I can disable logging when I'm testing using `WANDB_DISABLED` env variable and gradient watching by setting 
`WANDB_WATCH` . 

env variables should avoid excessive command line args

Thanks for the feedback @sshleifer and @patil-suraj 
I can try to propose something similar for `lightning_base`. I'll wait for PR #5607 to be closed (feel free to comment if you think it's missing details) and I'll update the README accordingly when adding this functionality.

Awesome. I had no idea that PR existed. Tag me next time and sorry for being so slow.
[fix] mbart_en_ro_generate test now identical to fairseq
Author: sshleifer
Pull request: yes

violentele -> violenţa
Slow test was previously failing.

Comments:
I think CI is spurious.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5731?src=pr&el=h1) Report
> Merging [#5731](https://codecov.io/gh/huggingface/transformers/pull/5731?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/c3c61ea01733403210a1d159114e8c3d042dabb7&el=desc) will **increase** coverage by `1.21%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5731/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5731?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5731      +/-   ##
==========================================
+ Coverage   77.22%   78.43%   +1.21%     
==========================================
  Files         146      146              
  Lines       26012    26012              
==========================================
+ Hits        20088    20403     +315     
+ Misses       5924     5609     -315     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5731?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/5731/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9yb2JlcnRhLnB5) | `43.98% <0.00%> (-49.38%)` | :arrow_down: |
| [src/transformers/generation\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5731/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3V0aWxzLnB5) | `97.11% <0.00%> (+0.28%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/5731/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9tb2JpbGViZXJ0LnB5) | `96.77% <0.00%> (+73.38%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5731?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5731?src=pr&el=footer). Last update [c3c61ea...608369d](https://codecov.io/gh/huggingface/transformers/pull/5731?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Latest Version Of HuggingFace is Not Using GPUs
Author: AdityaSoni19031997
Pull request: no

I switched from v2.8 to v2.9 today morning keeping all my experiments, enviornment same etc but the same code which used v2.8 has an epoch time of 5 mins, where as in v2.9 it's more than 2 hours.

Code was custom which was working till yesterday with v2.8.

Please verify the same.

Comments:
If you don't tell us what code you use, we cannot help you. Be detailed and use code block. Don't post screenshots. 

I agree. I apologize, was kinda annoyed that version diff can cause this. Will share a minimal example. (In the mean time,  re-verifying via setting up different env's)

Did you manage to solve the error?

Hey I am sorry for the delay, I am having fever since yesterday, I am trying to replicate it. 

Okay, so i am not able to replicate the same on a dummy test i had run just now; seems all good ; Will ping back if the same thing happens for my code; My apologies to great devs ;

@AdityaSoni19031997  I am also facing the same issue. New transformers version is not using GPU and it's quite slow.

I have solved the issue.
You have to manually set the device and load the tensors to GPU:

```
from transformers import BertTokenizer, BertModel, BertForMaskedLM


def assign_GPU(Tokenizer_output):
    
    tokens_tensor = Tokenizer_output['input_ids'].to('cuda:0')
    token_type_ids = Tokenizer_output['token_type_ids'].to('cuda:0')
    attention_mask = Tokenizer_output['attention_mask'].to('cuda:0')
    
    output = {'input_ids' : tokens_tensor, 
              'token_type_ids' : token_type_ids, 
              'attention_mask' : attention_mask}
    
    return output



sentence  = 'Hello World!'
tokenizer = BertTokenizer.from_pretrained('bert-large-uncased')
model     = BertModel.from_pretrained('bert-large-uncased')

inputs    = assign_GPU(tokenizer(sentence, return_tensors="pt"))
model     = model.to('cuda:0')
outputs   = model(**inputs)
outputs
```

In my case, it was a conflict of driver's IIRC.
How to finetune ELECTRA on glue?
Author: elyesmanai
Pull request: no
Labels: name: Ex: Sequence Classification

name: Core: Modeling

After pretraining my own electra model, I wanted to test it out in Glue using run_glue.py.
However I got this: 
```
ValueError: Unrecognized configuration class <class 'transformers.configuration_electra.ElectraConfig'> for this kind of AutoModel: AutoModelForSequenceClassification.
Model type should be one of DistilBertConfig, AlbertConfig, CamembertConfig, XLMRobertaConfig, BartConfig, RobertaConfig, BertConfig, XLNetConfig, FlaubertConfig, XLMConfig.
```
After taking a look at the source code, It seems like ElectraConfig isn't available for sequence classification, is there a reason for that?  Did anyone finetune electra on glue?

Comments:
I have a pull request here
https://github.com/huggingface/transformers/pull/4257

I just cloned your repo an tried to test with my model and it keeps saying the same: 
![image](https://user-images.githubusercontent.com/21007166/82654565-39745300-9c21-11ea-8da8-ad3bebb8a391.png)

Could you tell me how it's used?

@liuzzi's PR was merged this morning. The `ElectraForSequenceClassification` model is now available, so you can use it directly in `run_glue.py`.

Please make sure to pull the latest changes from the repo, or to wait for `v2.10` which should be released in a few hours.

awesome, it works perfectly, thank you very much!

@elyesmanai Could you please share the code for pretraining Electra from scratch?

I'm using the simpletransformers library for the pretraining since transformers doesn't support it yet.
here's a [link](https://towardsdatascience.com/understanding-electra-and-training-an-electra-language-model-3d33e3a9660d) to how you can do it, it's super easy.
It's built on top of transformers so you can load the model into transformers and use the rest of the lib

The pre-training from scratch for `transformers` is available [here](https://github.com/huggingface/transformers/pull/4656). It is being tested right now.

This problem happened again, when I use ELECTRA on question-answering pipeline. My Transformers version is 2.11.0.

> from transformers import pipeline, AutoTokenizer, AutoModelForQuestionAnswering
> 
> tokenizer = AutoTokenizer.from_pretrained("ahotrod/electra_large_discriminator_squad2_512")
> 
> model = AutoModelForQuestionAnswering.from_pretrained("ahotrod/electra_large_discriminator_squad2_512")
> 
> albert_qa = pipeline('question-answering', model=model, tokenizer=tokenizer)

![image](https://user-images.githubusercontent.com/53075457/87435341-2fb60d00-c61e-11ea-91be-538747c70559.png)
Create README.md for the model card of GPorTuguese-2 model (Portuguese GPT-2 small)
Author: piegu
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5744?src=pr&el=h1) Report
> Merging [#5744](https://codecov.io/gh/huggingface/transformers/pull/5744?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/838950ee44360ca427f345441502d4e7ab2772b8&el=desc) will **increase** coverage by `1.12%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5744/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5744?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5744      +/-   ##
==========================================
+ Coverage   77.33%   78.45%   +1.12%     
==========================================
  Files         146      146              
  Lines       26055    26047       -8     
==========================================
+ Hits        20149    20436     +287     
+ Misses       5906     5611     -295     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5744?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_bart.py](https://codecov.io/gh/huggingface/transformers/pull/5744/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iYXJ0LnB5) | `95.74% <100.00%> (-0.06%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5744/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `64.90% <0.00%> (-33.90%)` | :arrow_down: |
| [src/transformers/generation\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5744/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3V0aWxzLnB5) | `97.11% <0.00%> (+0.28%)` | :arrow_up: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5744/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.71% <0.00%> (+0.50%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/5744/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9tb2JpbGViZXJ0LnB5) | `96.77% <0.00%> (+73.38%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5744?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5744?src=pr&el=footer). Last update [b2505f7...ae466d5](https://codecov.io/gh/huggingface/transformers/pull/5744?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Customize inference widget input
Author: mrm8488
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5743?src=pr&el=h1) Report
> Merging [#5743](https://codecov.io/gh/huggingface/transformers/pull/5743?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/838950ee44360ca427f345441502d4e7ab2772b8&el=desc) will **decrease** coverage by `0.09%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5743/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5743?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5743      +/-   ##
==========================================
- Coverage   77.33%   77.24%   -0.10%     
==========================================
  Files         146      146              
  Lines       26055    26047       -8     
==========================================
- Hits        20149    20119      -30     
- Misses       5906     5928      +22     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5743?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_bart.py](https://codecov.io/gh/huggingface/transformers/pull/5743/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iYXJ0LnB5) | `95.74% <100.00%> (-0.06%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5743/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `64.90% <0.00%> (-33.90%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/5743/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9yb2JlcnRhLnB5) | `93.36% <0.00%> (+49.37%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5743?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5743?src=pr&el=footer). Last update [b2505f7...f8d259d](https://codecov.io/gh/huggingface/transformers/pull/5743?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@JetRunner There's still the possibility to override the default inputs if it makes more sense for the model: https://twitter.com/mrm8488/status/1282778743598194688

(up to @mrm8488 if it does here)

In this case it makes more sense because in last phase the model was fine tuned on Spanish Wikipedia data
GPT-2 language model: multiplying decoder-transformer output with token embedding or another weight matrix
Author: mainulquraishi
Pull request: no
Labels: name: PyTorch

name: Core: Modeling

I was reading the code of GPT2 language model. The transformation of hidden states to the probability distribution over the vocabulary has done in the following line:

`lm_logits = self.lm_head(hidden_states)`
Here,

`self.lm_head = nn.Linear(config.n_embd, config.vocab_size, bias=False)`
However, In the original paper, they suggested multiplying hidden states with the token embedding matrix whereas huggingface implementation used another matrix.

Is there any advantage of this? Am I missing something?

Comments:
Hi, the input embeddings are tied to the output embeddings -> The `lm_head` attribute essentially shares its weights with the embedding layer. Passing the output of the transformer through that layer is the same as multiplying this output (the hidden states) with the token embedding matrix.

@LysandreJik Where is the code that performs this weight tying?

I think I found it:

```
        output_embeddings = self.get_output_embeddings()
        if output_embeddings is not None:
            self._tie_or_clone_weights(output_embeddings, self.get_input_embeddings())

```
in `PreTrainedModel.tie_weights()` in `modeling_utils.py`.