noamaanMulla-03/transformers-issues huggingface dataset:
Where are BERT's pretrained Embeddings loaded?
Author: BramVanroy
Pull request: no

I am trying to better understand the difference between the different types of embeddings that BERT uses (from the BERT paper: token, segment, position). For this purpose, I was hoping to put some print statement in the `pytorch_transformers` source code to see how the IDs change into vector representations for each type of embedding.

First of all I am confused about the embeddings that `pytorch_transformers` uses. Going through the source code for [`BertEmbeddings`](https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_bert.py#L142-L171) I can see

- word embeddings
- position embeddings
- token type embeddings

What are these _token type_ embeddings? Are they the same as segment embeddings?

Secondly, during my quest for better understanding what's going on, I couldn't figure out where the pretrained embedding models are loaded, or even where they are downloaded. I am curious to see the vocab list of all types of embeddings, but I couldn't find them anywhere.

Any pointers?

Comments:
- "token type embeddings" are the BERT paper's segment embeddings 
- embeddings are inside the pretrained weights

Ah that makes sense. So there are no "separate" word2vec-style pretrained embedding models for the different types of embeddings which one could load with `nn.Embedding().from_pretrained`. Rather, they are loaded in a bunch as a set of pretrained weights. Theoretically, though, one could extract the weights for each embedding, and extract the vocab from the tokenizer, and create a simple lookup (`token\tvector`)?

Thanks for the reply and your work.

Sure you could, but I suspect it wouldn’t work too well.

You could say that a large language model’s hidden states are the new way to do word/sentence embeddings (see Sebastian Ruder’s imageNet moment).

Apologies if this is taking too much of your time, but I have a follow-up question. Why wouldn't it work too well? I understand that they are not typical word2vec word representations, since they have been trained together with the whole language model, but why would extracting the embeddings and using them in another task not work well? In other words, what makes the token embeddings of BERT fundamentally different from a typical word2vec model?

I think you'll find this repo (and associated EMNLP 2019 paper) by @nriemers interesting:

https://github.com/UKPLab/sentence-transformers (built on top of `transformers`)

> * "token type embeddings" are the BERT paper's segment embeddings
> * embeddings are inside the pretrained weights

hi, could you tell where the code about BertEmbedding loaded with the pre-trained weights is?
Fix input ids can be none attn mask
Author: patrickvonplaten
Pull request: yes

Make sure `batch_size` is correct for gpt2 and ctrl - these models need a slightly different behavior since the shape of `input_ids` can change depending on whether the past variable is inserted or not. 
See also:
PR: https://github.com/huggingface/transformers/pull/3033
and its issue:
https://github.com/huggingface/transformers/issues/3031

Comments:
Thanks for pointing out @lazarevskiVsg and @julien-c !

@julien-c , will merge right away - small change!

Why don’t you just use input_shape (which is always defined), to be consistent with other models?

> Why don’t you just use input_shape (which is always defined), to be consistent with other models?

The problem is that GPT2 and CTRL have a different behavior (and the `input_ids` shape changes) when the `past` variable is inserted which previously led to problem when the attention_mask is inserted as well: 
#3031 

Therefore this slightly weird implementation.

But in your code in this PR, batch_size is always input_shape[0] anyways, no?

> But in your code in this PR, batch_size is always `input_shape[0]` anyways, no?

I think in the case of CTRL and GPT2, it's actually a bigger inconsistency:

Let's say we have an input_ids tensor of shape `[batch_size, sequence_length] = [5, 4]`.

We call `GPT2Model` and save the last `output embeddings = outputs[0][:, -1, :]` **and** the `past` key/value states to speed up decoding = `outputs[1]`

Now if we want to use `past` GPT expects the `input_ids` to be of shape `[batch_size, 1] .squeezed(-1) = [batch_size]`. Therefore we have to adapt the attention mask here differently than in other models. Which is weird (and a bit suboptimal in my opinion in GPT's and CTLR's API) is that the shape of `input_ids` differs depending on whether `past` is None or not. 

@julien-c
[BART] cleanup: remove redundant kwargs, improve docstrings
Author: sshleifer
Pull request: yes

Small Bart code cleanups before pip release.

### Cleanup
- Deletes unused `value` argument for SelfAttention. (`value` is always the same as `key`.) This might be moderately controversial as most attention modules take query, key, value as arguments, but this change reduces the signature to just query and key (since key always the same as value).
- Deletes redundant `static_kv` argument for SelfAttention. It is always the same as `self.encoder_decoder_attention`. 
  - Context: the `static_kv` variable decides whether we want to extend the keys and values in the cache or, if `True`, use them without modification. 
  - This PR keeps a local `static_kv` variable because that variable name describes the purpose of the variable better than `self.encoder_decoder_attention`. But `static_kv` is no longer a kwarg. This simplifies the API and avoids having the same logic in two places.

#### Two new fast tests
- test coverage for `dummy_inputs` (previously broken)
- test coverage for the default generate kwargs.

Comments:
[FIX] not training when epoch is small
Author: mataney
Pull request: yes

Closes https://github.com/huggingface/transformers/issues/2995

Solving bug where for small epochs and large gradient_accumulation_steps we never train.
Explained further here: https://github.com/huggingface/transformers/issues/2995

Comments:
Cool, thanks @mataney! Do you mind running `make style` at the root of the repository to reformat the file according to `black` and `isort`, as indicated in our [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests)?

This is so the `check_code_quality` passes.

@LysandreJik Thanks for your reply.

Recommited with the black formatting.
It is now failing because something else. Do you think you can rerun the tests? I believe it has something to do with the Indeterminism of the tests.



Can I have your help here @LysandreJik
No optimizer steps when gradient_accumulation_steps smaller than epoch_iterator length
Author: mataney
Pull request: no
Labels: name: Ex: Sequence Classification

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Doesn't matter.

Language I am using the model on (English, Chinese ...): Doesn't matter.

The problem arises when using:
* [x] the official example scripts
* [ ] my own modified scripts

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: Any of the tasks.
* [ ] my own task or dataset

## To reproduce

Steps to reproduce the behavior:

1. Run `run_glue.py` where number of batches for each epoch is smaller than gradient_accumulation_steps.

In https://github.com/huggingface/transformers/blob/8194df8e0cff8e5866ec2bcbda34e3892f10eb39/examples/run_glue.py#L233

`step` is at most `len(train_dataloader)/batch_size`. If the latter is small, then we would never enter the if condition and never call `optimizer.step()` and so on.

I know it's the user responsibility to be aware of this, but this can be easily solved by by altering the condition to be:
```
if (step + 1) % args.gradient_accumulation_steps == 0 or ((step + 1) < args.gradient_accumulation_steps and (step + 1) == len(epoch_iterator)):
```

Should I create a small PR?

Comments:
In this scenario, the user would do a single optimizer step over the whole batch?

Sure, we would welcome a PR! Could you add a comment as well, so that the purpose of this line may be clear for the user?

After the change we would do a single optimizer step over each *epoch*.
Added a comment and created a PR. 
https://github.com/huggingface/transformers/pull/3006
Cheers.
Add LM capabilities to TFTransfoXLLMHead
Author: patrickvonplaten
Pull request: no
Labels: name: Core: Modeling

# 🚀 Feature request

It is currently not possible to generate language with the TFTransfoXLLMHeadModel because 
the `lm_head` is not implemented (see https://github.com/huggingface/transformers/blob/908fa43b543cf52a3238129624f502240725a6a6/src/transformers/modeling_tf_transfo_xl.py#L745)

Doing: 

```
from transformers import TFTransfoXLLMHeadModel
model = TFTransfoXLLMHeadModel.from_pretrained('transfo-xl-wt103') 
model(tf.convert_to_tensor([[1, 5, 8]])) 
```

currently leads to an error. 

## Motivation

Pytorch has TransfoXLLMHead model working - TF should has well.

## Your contribution

@LysandreJik , @thomwolf  I could implement it if needed.


Comments:
Perform MultiLingual Name Matching 
Author: geojolly
Pull request: no
Labels: name: wontfix

I am trying to perform multi-lingual name matching (entity-resolution). To build the pipeline the idea is to use :

- Byte level character embeddings and then use a dense vector similarity. 

Anyone here has some experience in this approach? 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
seems TFBertForSequenceClassification cannot load tf1.x model?
Author: fword
Pull request: no

for TFPreTrainedModel
  model.load_weights(resolved_archive_file, by_name=True)
  
seems by_name is always true.while i see in "tensorflow_core/python/keras/engine/network.py"

 if save_format == 'tf':
      status = self._trackable_saver.restore(filepath)
      if by_name:
        raise NotImplementedError(
            'Weights may only be loaded based on topology into Models when '
            'loading TensorFlow-formatted weights (got by_name=True to '
            'load_weights).')


So, it will always be NotImplementedError

Comments:
All of our TensorFlow models are TF2+ only.
Fix wrong link for the notebook file
Author: Kyeongpil
Pull request: yes

For the tutorial of "How to generate text", the URL link was wrong (it was linked to the tutorial of "How to train a language model").

I fixed the URL.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3344?src=pr&el=h1) Report
> Merging [#3344](https://codecov.io/gh/huggingface/transformers/pull/3344?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f6d813aaaa96cc43fcf55f255b9439ebc22a31a0&el=desc) will **not change** coverage by `%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3344/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3344?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3344   +/-   ##
=======================================
  Coverage   77.63%   77.63%           
=======================================
  Files         100      100           
  Lines       16943    16943           
=======================================
  Hits        13154    13154           
  Misses       3789     3789           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3344?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3344?src=pr&el=footer). Last update [f6d813a...f78b5f0](https://codecov.io/gh/huggingface/transformers/pull/3344?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks a lot for pointing this out @rudvlf0413 and @julien-c
output value of  XLNetModel changes for the same input 
Author: mainulquraishi
Pull request: no

I was trying to use the pre-trained `XLNetModel` model. I found that for the same input, each time I run the model and the output values are different, which is wired to me. Here is a code that I just used:
```
tokenizer =XLNetTokenizer.from_pretrained("xlnet-base-cased")
model=XLNetModel.from_pretrained('xlnet-base-cased') 
sent=tokenizer.encode("I love my dog")
test=torch.tensor(sent)
test=test.view(1,test.shape[0])
```
Now If I run the model, each time the output is different. 
```
out=model(test)
print(out[0]) 

``` 
Why this be behaviour? 

I have another question, isn't this model the pre-trained transformer trained on language modeling task (transformer without the LM head)?  

Comments:
It seems that after fine-tuning if I don't use `model=model.eval()`, this can happen.
TFAlbertMainLayer cannot be imported from the transformers library.
Author: shreyansh05s
Pull request: no

Unlike the TFBertMainLayer class which can be imported from transformers, TFAlbertMainLayer cannot be imported.

Locally I made the changes to init.py:

Import TFAlbertMainLayer from modeling_tf_albert 

Similar to tensorflow version, pytorch version should also be implemented.

Comments:
This seems like a valid point, what do you think @LysandreJik?
Export ALBERT main layer in TensorFlow
Author: LysandreJik
Pull request: yes

closes #3262 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3354?src=pr&el=h1) Report
> Merging [#3354](https://codecov.io/gh/huggingface/transformers/pull/3354?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/3bedfd334763cb5676c2fe92705390ac57d8de5f&el=desc) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3354/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3354?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3354      +/-   ##
==========================================
+ Coverage   77.61%   77.68%   +0.07%     
==========================================
  Files         100      100              
  Lines       16938    16938              
==========================================
+ Hits        13146    13159      +13     
+ Misses       3792     3779      -13     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3354?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/3354/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `98.92% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3354/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.55% <0.00%> (+2.32%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3354?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3354?src=pr&el=footer). Last update [3bedfd3...77a2a4c](https://codecov.io/gh/huggingface/transformers/pull/3354?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Add model cards for huseinzol05/bert-base-bahasa-cased
Author: huseinzol05
Pull request: yes
Labels: name: model card


Comments:
Looks good! [**Model page**](https://huggingface.co/huseinzol05/bert-base-bahasa-cased)

I've also added a filter for the Malay language here:
<img width="792" alt="Screenshot 2020-03-19 at 15 28 42" src="https://user-images.githubusercontent.com/326577/77106980-58beec00-69f6-11ea-8145-4d273d605693.png">
Update 01-training-tokenizers.ipynb (typo issue)
Author: Kyeongpil
Pull request: yes

I found there are two grammar errors or typo issues in the explanation of the encoding properties.

The original sentences:
- **If your was** made of multiple \"parts\" such as (question, context), then this would be a vector with for each token the segment it belongs to
- **If your has** been truncated into multiple subparts because of a length limit (for BERT for example the sequence length is limited to 512), this will contain all the remaining overflowing parts.

I think "**input**" should be inserted after the phrase "If your".

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3343?src=pr&el=h1) Report
> Merging [#3343](https://codecov.io/gh/huggingface/transformers/pull/3343?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f6d813aaaa96cc43fcf55f255b9439ebc22a31a0&el=desc) will **decrease** coverage by `0.08%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3343/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3343?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3343      +/-   ##
==========================================
- Coverage   77.63%   77.54%   -0.09%     
==========================================
  Files         100      100              
  Lines       16943    16943              
==========================================
- Hits        13154    13139      -15     
- Misses       3789     3804      +15     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3343?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3343/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `85.68% <0.00%> (-2.69%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3343?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3343?src=pr&el=footer). Last update [f6d813a...9ecfde1](https://codecov.io/gh/huggingface/transformers/pull/3343?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks!
Simpler Error message when loading config/model with .from_pretrained()
Author: julien-c
Pull request: yes


Comments:
tweaked version of #3247 

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3341?src=pr&el=h1) Report
> Merging [#3341](https://codecov.io/gh/huggingface/transformers/pull/3341?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f6d813aaaa96cc43fcf55f255b9439ebc22a31a0&el=desc) will **decrease** coverage by `0.19%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3341/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3341?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3341      +/-   ##
==========================================
- Coverage   77.63%   77.44%   -0.20%     
==========================================
  Files         100      100              
  Lines       16943    16943              
==========================================
- Hits        13154    13121      -33     
- Misses       3789     3822      +33     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3341?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/configuration\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3341/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3V0aWxzLnB5) | `96.82% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3341/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `82.46% <0.00%> (-5.91%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3341?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3341?src=pr&el=footer). Last update [f6d813a...c9ce50c](https://codecov.io/gh/huggingface/transformers/pull/3341?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Fix #3305: run_ner only possible on ModelForTokenClassification models
Author: julien-c
Pull request: yes

Also, #3305 breaks (if i'm not mistaken) the ability to run the example script from a pip-installed instance of transformers (vs. from an instance installed from source) (This PR does not fix that second issue)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3335?src=pr&el=h1) Report
> Merging [#3335](https://codecov.io/gh/huggingface/transformers/pull/3335?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/20139b7c8d88f380f1e4e0ae2baf0b0ac9351039?src=pr&el=desc) will **decrease** coverage by `0.06%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3335/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3335?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3335      +/-   ##
==========================================
- Coverage   77.63%   77.57%   -0.07%     
==========================================
  Files         100      100              
  Lines       16943    16943              
==========================================
- Hits        13154    13143      -11     
- Misses       3789     3800      +11
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3335?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3335/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `86.4% <0%> (-1.97%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3335?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3335?src=pr&el=footer). Last update [20139b7...7fc00b6](https://codecov.io/gh/huggingface/transformers/pull/3335?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Ok I'll merge this @srush @LysandreJik as it's more correct (feel free to let me know of your feedback anyways)

Thanks. Why does it break the pip version?

Because `MODEL_MAPPING` from `modeling_auto` is not exposed in the package's [`__init__.py`](https://github.com/huggingface/transformers/blob/master/src/transformers/__init__.py)