noamaanMulla-03/transformers-issues huggingface dataset:
Language modeling example script missing the next sentence predicion
Author: salmanmashayekh
Pull request: no
Labels: name: Ex: LM (Pretraining)

The example script in `run_language_modeling.py` does not include the next sentence prediction for pre-training BERT. I was wondering if that is a) an oversight, b) for simplicity, or c) because you have found its impact to be non-significant? 

Comments:
It is both b) and c) :).

Hello @LysandreJik  and @BramVanroy 

Did you have any results in training   run_language_modeling.py    for some language from scratch (i mean with and  without NSP (next sentence prediction) as is in that script) ?

Did you get better or relatively the same losses (and perplexity, accuracy) by first doing MLM then 
NSP ?

what were your warmup steps and  block size (maximum sequence length) for each (MLM , NSP) task , if you have down them separately ? 

as @LysandreJik mentioned here
 
https://github.com/huggingface/transformers/issues/2693#issuecomment-589819382

"the RoBERTa paper has proven that the NSP objective was not particularly helpful"
 
Is that also right for BERT ?  (training for some language from scratch)


If it is right  ,  I think we can set  block size (maximum sequence length)  equal to 64  for MLM
 
Because the  original paper used  128 (except for 10% of the last steps)  , and it had two sentences (A and B for the sake of NSP and model learns that it shouldn't see B for filling masked words in A, because A and B aren't relevant to each other half of the times) which means average block size of 64 for each one 

And it means grate speed up in training BERT, if I say right
because, I haven't got a TPU or even enough GPU to do that 

As mentioned in original paper of BERT "Longer sequences are disproportionately expensive because attention is quadratic to the sequence length"

Perhaps, another idea is that 
words in sentence A do attention on sentence B, anyway
and that attention is very important to get great results in MLM task (here I mean block size of 128 which means 64 for A and another 64 for B)

even regarding the fact that  A and B are related sentences, just half of the times 
(and that attentions are the main inputs for task NSP)

And by using block size of 64 (and not doing task NSP during the training) , I will get very bad results
How to get a spiece.model from customize chinese vocab.txt in Albert xlnet ?
Author: ciel-zhang
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
How to get a spiece.model from customize chinese vocab.txt in Albert xlnet ?

Comments:
Have you taken a look at [sentencepiece](https://github.com/google/sentencepiece)?

请问这个问题您解决了吗

I have the same problem. Have you solved it?

> Have you taken a look at [sentencepiece](https://github.com/google/sentencepiece)?

I have taken a look at sentencepiece documents, but found nothing to build a spiece.model from customized chinese vocab.txt in ALBERT. Do you have any solution to solve this problem?

I think that the chinese version of albert uses wordpiece model instead of sentencepiece model.
[https://github.com/google-research/ALBERT/issues/58](url)

> For Chinese models, we use word piece model provided by Jacob as sentence piece get worse performance on reading comprehension tasks for Chinese.

https://github.com/google-research/ALBERT/blob/master/tokenization.py
```python
class FullTokenizer(object):
  """Runs end-to-end tokenziation."""

  def __init__(self, vocab_file, do_lower_case=True, spm_model_file=None):
    self.vocab = None
    self.sp_model = None
    if spm_model_file:
      self.sp_model = spm.SentencePieceProcessor()
      tf.logging.info("loading sentence piece model")
      self.sp_model.Load(spm_model_file)
      # Note(mingdachen): For the purpose of consisent API, we are
      # generating a vocabulary for the sentence piece tokenizer.
      self.vocab = {self.sp_model.IdToPiece(i): i for i
                    in range(self.sp_model.GetPieceSize())}
    else:
      self.vocab = load_vocab(vocab_file)
      self.basic_tokenizer = BasicTokenizer(do_lower_case=do_lower_case)
      self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab)
    self.inv_vocab = {v: k for k, v in self.vocab.items()}
```
When the sentencepiece model is None, the full tokenizer is initialized with a basic tokenizer and a workpiece tokenizer.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
❓ [BART] Why Decoder Layer Normalization is applied only at the last layer ? 
Author: astariul
Pull request: no

# ❓ Questions & Help

It seems this line :
https://github.com/huggingface/transformers/blob/efbc1c5a9d96048ab11f8d746fe51107cb91646f/src/transformers/modeling_bart.py#L524

was modified when MBART was added. 

---

Before, Layer Normalization was applied after **all** layers of the decoder (_similar to the encoder, if the config was set appropriately_).

But now, Layer Normalization is applied **only at the end**, even for other BART models (_not MBART_).

---

**Is it expected ? What's the reason behind this logic ?**

@sshleifer 

Comments:
That `layer_norm` should be None in `bart-large*`.
See [this comment](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bart.py#L266)

No final `layer_norm` was applied before `mbart` was added, afaict.
Tokenize something with a "." in between Decode these ids, you will find it mismatch
Author: Xunzhuo
Pull request: no

🐛 Bug
Information
Model I am using (Bert, XLNet ...): XLNet

Language I am using the model on (English, Chinese ...): Chinese
The problem arises when using: Tokenizer

 the official example scripts: (give details below) N/A
 my own modified scripts: (give details below) N/A
The tasks I am working on is: Any

 an official GLUE/SQUaD task: (give the name) N/A
 my own task or dataset: (give details below) N/A
To reproduce
Steps to reproduce the behavior:

Load any BERT tokenizer
Tokenize something with a "." in between
Decode these ids, you will find it mismatch
x = tokenizer.encode('AN.C', add_special_tokens=False)
z = tokenizer.decode(x)
It prints:

AN. C
Expected behavior
AN.C
Environment info
transformers version:
Platform: CentOS
Python version: 3.6
PyTorch version (GPU?): GPU
Tensorflow version (GPU?): GPU
Using GPU in script?: NO
Using distributed or parallel set-up in script?: NO


Comments:
Your use case seems specific, so maybe you should try a custom Tokenizer via the `tokenizers` library. I believe the results you're getting are the intended behavior.

For example, any generic sentence where someone forgets to put a space after the period would end up tokenized incorrectly otherwise:

`I love lamp.No I really love lamp.` would leave you with a token `lamp.No`, which is incorrect, eh?

tks~ it helps a lot ~
added functionality for electra classification head
Author: liuzzi
Pull request: yes

Added functionality for an electra classification head following the google repo.  
Tested and trains fine with the run_glue script.  I was testing on my own repurposed classification problems so i didnt evaluate the actual GLUE accuracies, but can confirm that it is getting competitive results on my own problems.
Saw a few requests for this, thought it could benefit the community.

Originally I used a tanh activation in the classification head.  (which in my tests reliably performed better than a gelu activation)  But looking over the google code, it seems that they used gelu, so to keep the integrity of their paper im committing gelu here.

I also opted to make a ClassificationHead class instead of integrating a pooling layer in the base model class because i wasn't sure if that would affect other things, or run time in non sequence-classification tasks.

Comments:
This is great, thanks @liuzzi!

We would need to add the corresponding tests. Can you do it, or would you like me to do it? If so, may I push directly on your fork?

@LysandreJik sure! i just invited you to collaborate.  I haven't been inside the transformers testing code so it'd probably be much faster for you to do.  Let me know if you have any questions/if i can help.

I just pushed the tests on it, thanks for your contribution!

Pushed a style change, rebased and merged into master. Thanks @liuzzi !
FillMaskPipeline crashes when executed on TPU
Author: LeonieWeissweiler
Pull request: no
Labels: name: PyTorch

# 🐛 Bug

## Information

I am following the tutorial in https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb#scrollTo=QDNgPls7_l13 and running on Google Colab using the TPU. The Pipeline object creation works fine, but when I try to run it on the example sentence, the Colab runtime crashes immediately with an unclear cause and no error message. If I remove the TPU and do not install xla, the pipeline works fine.

## To reproduce

Steps to reproduce the behavior:

```python3
!pip uninstall transformers
!git clone https://github.com/huggingface/transformers
!pip install ./transformers

VERSION = "nightly" 
!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
!python pytorch-xla-env-setup.py --version $VERSION

from transformers import pipeline

fill_mask = pipeline(
    "fill-mask",
    model="drive/My Drive/models/EsperBERTo/output/checkpoint-15000",
    tokenizer="drive/My Drive/models/EsperBERTo"
)

fill_mask("La suno <mask>.")

```

Is anyone else experiencing this?

Comments:
Hello! Pipelines are not tested on TPUs yet, unfortunately, and we have not made any effort to support them on that device. We may down the road, once TPU CI is more easily available.
Fix convert_token_type_ids_from_sequences for fast tokenizers
Author: n1t0
Pull request: yes

Before this fix, the generic version of `convert_token_type_ids_from_sequences` from `tokenizer_utils` gets called when called on a `PreTrainedTokenizerFast`. The `type_ids` for the special token are thus not included.
There is no way at the moment to get this information from the rust tokenizers, so we just use the implementation from the original python tokenizers. Tests added as well.

Thanks @dirkgr for reporting this.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4503?src=pr&el=h1) Report
> Merging [#4503](https://codecov.io/gh/huggingface/transformers/pull/4503?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a08652772791fdaeed6f263b1a99926ca64be5dc&el=desc) will **increase** coverage by `0.02%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4503/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4503?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4503      +/-   ##
==========================================
+ Coverage   77.83%   77.86%   +0.02%     
==========================================
  Files         123      123              
  Lines       20514    20526      +12     
==========================================
+ Hits        15968    15982      +14     
+ Misses       4546     4544       -2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4503?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/4503/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmVydC5weQ==) | `95.00% <100.00%> (+0.12%)` | :arrow_up: |
| [src/transformers/tokenization\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/4503/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fcm9iZXJ0YS5weQ==) | `94.52% <100.00%> (+0.49%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4503/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.66% <0.00%> (+0.16%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4503/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.85% <0.00%> (+0.41%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4503?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4503?src=pr&el=footer). Last update [a086527...795f44a](https://codecov.io/gh/huggingface/transformers/pull/4503?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Added huseinzol05/t5-small-bahasa-cased README.md
Author: huseinzol05
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4522?src=pr&el=h1) Report
> Merging [#4522](https://codecov.io/gh/huggingface/transformers/pull/4522?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/bd6e3018322766b3a71ae6675552607923c02636&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4522/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4522?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4522   +/-   ##
=======================================
  Coverage   77.85%   77.85%           
=======================================
  Files         123      123           
  Lines       20551    20551           
=======================================
+ Hits        15999    16001    +2     
+ Misses       4552     4550    -2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4522?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4522/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.53% <0.00%> (+0.11%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4522/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.83% <0.00%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4522?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4522?src=pr&el=footer). Last update [bd6e301...d1e772c](https://codecov.io/gh/huggingface/transformers/pull/4522?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
get output from a particular layer of pre-trained transformer (xlnet) 
Author: mainulquraishi
Pull request: no

As the title, how can I do this in PyTorch version of pre-trained transformer?   

Comments:
Sure, these are called hidden states! Here's the [documentation of the XLNet model](https://huggingface.co/transformers/model_doc/xlnet.html?highlight=output_hidden_states#transformers.XLNetModel).

Please note the third return:

> **hidden_states (tuple(torch.FloatTensor), optional, returned when config.output_hidden_states=True):**
> 
> Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
> 
> Hidden-states of the model at the output of each layer plus the initial embedding outputs.
[Summarization Pipeline]: Fix default tokenizer
Author: sshleifer
Pull request: yes

`pipeline.tokenizer` cannot be a dict!

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4506?src=pr&el=h1) Report
> Merging [#4506](https://codecov.io/gh/huggingface/transformers/pull/4506?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a08652772791fdaeed6f263b1a99926ca64be5dc&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4506/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4506?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4506      +/-   ##
==========================================
- Coverage   77.83%   77.82%   -0.01%     
==========================================
  Files         123      123              
  Lines       20514    20514              
==========================================
- Hits        15968    15966       -2     
- Misses       4546     4548       +2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4506?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/4506/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `76.11% <ø> (ø)` | |
| [src/transformers/hf\_api.py](https://codecov.io/gh/huggingface/transformers/pull/4506/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9oZl9hcGkucHk=) | `93.06% <0.00%> (-4.96%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4506/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.53% <0.00%> (+0.11%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4506/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.83% <0.00%> (+0.32%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4506?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4506?src=pr&el=footer). Last update [a086527...70d3058](https://codecov.io/gh/huggingface/transformers/pull/4506?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Add Type Hints to modeling_utils.py Closes #3911
Author: bglearning
Pull request: yes

Add Type Hints to methods in `modeling_utils.py`

Note: The coverage isn't 100%. Mostly skipped internal methods (and some I wasn't sure of).

Comments:
I think this is good for merge, no? @julien-c 

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3948?src=pr&el=h1) Report
> Merging [#3948](https://codecov.io/gh/huggingface/transformers/pull/3948?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/7b75aa9fa55bee577e2c7403301ed31103125a35&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3948/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3948?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3948      +/-   ##
==========================================
- Coverage   78.39%   78.38%   -0.02%     
==========================================
  Files         120      120              
  Lines       19925    19925              
==========================================
- Hits        15620    15618       -2     
- Misses       4305     4307       +2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3948?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3948/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.96% <100.00%> (-0.13%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3948/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.77% <0.00%> (-0.17%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3948?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3948?src=pr&el=footer). Last update [7b75aa9...99848e8](https://codecov.io/gh/huggingface/transformers/pull/3948?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks @bglearning!
Error in loading Albert model
Author: WenxiongLiao
Pull request: no
Labels: name: wontfix

![image](https://user-images.githubusercontent.com/25845940/76815416-88c09080-6838-11ea-829a-5a5874692009.png)

Help~
The above problems appear in loading the pre-trained Albert by transformers. Albert pre-trained model download address is https://drive.google.com/file/d/1byZQmWDgyhrLpj8oXtxBG6AA52c8IHE-/view

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
❓ How Linear layer difference between TF2 and PT are handled ?
Author: astariul
Pull request: no
Labels: name: PyTorch

name: TensorFlow

# ❓ Questions & Help

There is a difference between TF2 and Pytorch on how to store the weights of a Linear Layer.

As shown in [this Colab notebook](https://colab.research.google.com/drive/1zLWONO3wo09-PImo0kg1bwARsh2jKnpQ?usp=sharing), in order to get the same output for both TF2 and PT when using `torch.nn.Linear` and `tf.keras.layers.Dense`, we need to transpose the weights in PT.

**I couldn't find in this library where this is handled** (when loading a Pytorch checkpoint to TF2 for example).  
Can someone point me out where and how this is handled ?

Comments:
[Here's an example when loading a TF BERT model in PyTorch.](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L122)