noamaanMulla-03/transformers-issues huggingface dataset:
added t5 base and small bahasa summarization readme
Author: huseinzol05
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6269?src=pr&el=h1) Report
> Merging [#6269](https://codecov.io/gh/huggingface/transformers/pull/6269?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/bd0eab351a338175053998ddfc059f1cb6424ab4&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6269/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6269?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #6269   +/-   ##
=======================================
  Coverage   79.29%   79.29%           
=======================================
  Files         146      146           
  Lines       26684    26684           
=======================================
+ Hits        21158    21160    +2     
+ Misses       5526     5524    -2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6269?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6269/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.71% <0.00%> (+0.25%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6269/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `80.30% <0.00%> (+0.25%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6269?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6269?src=pr&el=footer). Last update [bd0eab3...02faea5](https://codecov.io/gh/huggingface/transformers/pull/6269?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Can't reproduce XNLI zero-shot results from MBERT in Chinese
Author: edchengg
Pull request: no

## ❓ Questions & Help

Hi guys,
I am trying to reproduce the XNLI zero-shot transfer results from MBERT.
With the same code and same checkpoint but different language for the test set, I am not able to reproduce the results for Chinese, Arabic, and Urdu. Does anyone encounter the same problem? Thanks!


Model |English | Chinese | Arabic | German | Spanish | Urdu
-- |-- | -- | -- | -- | -- | --
From MBERT github page | 81.4 | 63.8 | 62.1 | 70.5 | 74.3 | 58.3
My results | 82.076 | 36.088 | 35.327 | 68.782 | 70.419 | 35.170





Comments:
Hey @edchengg, I'm running into the same problem. Were you able to figure this out? thanks


Anyone else who stumbles here.
**Fix**: Just use bert-based-multilingual-cased as shown here https://huggingface.co/transformers/v2.3.0/examples.html.
When I used Google's mBERT and made it pyTorch compatible using the convert_tf_original..... script from src/transformers, somehow it doesn't learn properly. Couldn't figure out why, hence opening a new issue here : https://github.com/huggingface/transformers/issues/5019

@bsinghpratap Did you manage to run the script in only evaluation mode? When I try to evaluate an mBERT model trained on MNLI, it just freezes at 99%.

Yeah, I ran it on eval mode as well. Works fine for me.
Generation with EncoderDecoder Model
Author: manzar96
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

Hi, 
I am using the EncoderDecoder Model and I would like to use the generate method for sequence generation. As I have read in docs, the generate method can be used with any Pretrained HF model with  LM head on top.
So wrapping a pretrained LM model (e.g. GPT2LMHeadModel, BertForMaskedLM) (and having a encoder too) with the Encoder-Decoder class gives null bos_token_id and the generate method does not work properly. However using only a pretrained LM model (without wrapping with the Encoder-Decoder class) gives a valid bos_token_id (because the config file contains bos_token_id).
How I should handle the above issue?

Thank you in advance.

Comments:
Do we have any updates on this issue?

I will take a look at this at the end of next week - will get to you! 

> I will take a look at this at the end of next week - will get to you!

Thanks a lot!

Hi @manzar96,
Multiple bugs were fixed in #4680 . Can you please take a look whether this error persists?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Returning the attention heads using Longformer
Author: codeninja
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
Is it possible to get the longformer classification model to return the attentions learned? I don't see a way to do this with in the documentation or code.


Comments:
Add strip_accents to basic BertTokenizer.
Author: PhilipMay
Pull request: yes

The BertTokenizerFast can turn off strip_accents with `strip_accents=False`. This PR also adds this option to the basic BertTokenizer.

Also see #6186

Comments:
Strange CI problem with checksum of Torch:
```
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    torch from https://files.pythonhosted.org/packages/38/53/914885a93a44b96c0dd1c36f36ff10afe341f091230aad68f7228d61db1e/torch-1.6.0-cp36-cp36m-manylinux1_x86_64.whl#sha256=7669f4d923b5758e28b521ea749c795ed67ff24b45ba20296bc8cff706d08df8 (from transformers==3.0.2):
        Expected sha256 7669f4d923b5758e28b521ea749c795ed67ff24b45ba20296bc8cff706d08df8
             Got        8188c461d0b762aa4ae6e72105b3c3a01f7bb2863b46a392ff8537a0854e8967
```

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6280?src=pr&el=h1) Report
> Merging [#6280](https://codecov.io/gh/huggingface/transformers/pull/6280?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/31da35cc8939e19a292cb7f15cad5c9a1ddf7f23&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6280/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6280?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #6280   +/-   ##
=======================================
  Coverage   79.64%   79.64%           
=======================================
  Files         147      147           
  Lines       27120    27125    +5     
=======================================
+ Hits        21600    21605    +5     
  Misses       5520     5520           
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6280?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/6280/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmVydC5weQ==) | `91.51% <100.00%> (+0.19%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6280?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6280?src=pr&el=footer). Last update [31da35c...547df96](https://codecov.io/gh/huggingface/transformers/pull/6280?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


You can ignore the HASH errors, we're working on solving these but they're unrelated to your PR.

This is ready to be merged IMO. 😁

Wow - this was really fast from first commit to merge. Many thanks to the contributors. This makes open source development twice as much fun.
Remove inconsistency between BertTokenizer and BertTokenizerFast 
Author: PhilipMay
Pull request: no

# 🚀 Feature request
`BertTokenizerFast` has the option to specify `strip_accents=False`. The `BertTokenizer` does not have this option. This inconsistency should be removed by adding the `strip_accents` parameter to `BertTokenizer`.

## Motivation
Without adding this, the `BertTokenizer` can not be used for language models which are lowercase but have accents.

In case of a language model with lowercase and with accents you are forced to load the tokenizer by this:

```python
tokenizer = AutoTokenizer.from_pretrained("<model_name_or_path>", use_fast=True, strip_accents=False)
```

This will NOT work: `tokenizer = AutoTokenizer.from_pretrained("<model_name_or_path>")`

And even this would not work: `tokenizer = AutoTokenizer.from_pretrained("<model_name_or_path>", strip_accents=False)`

## Your contribution
With some hints I am willing to contribute.

Comments:
Fixed with #6280
Adds comet_ml to the list of auto-experiment loggers
Author: dsblank
Pull request: yes

This PR does three things:

* abstracts the auto-experiment-loggers' is_available() functions into an "integrations.py" file
* adds comet_ml to the list of auto-loggers available
* updates and reorganizes the docs slightly 

Comments:
@LysandreJik Will do! When I run `make style` it makes corrections to many other files, and also (for example) changes the import sort order on code that I didn't touch in the code that I am editing. Am I doing something wrong? Or should I commit those changes (in the files I am editing)?

Sounds weird, maybe you're missing some packages in your environment. Let me know when you're finished and I'll push the styling on your branch.

@sgugger Should have all of the review comments addressed. Thanks to everyone!

Great, thanks for iterating @dsblank!

You're welcome, and thanks for all of the work on this project! Looking forward to productive ML!
Question about BERT model size (transformer block number) 
Author: ZLKong
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
Hi,

Thank you for your interesting work! I have just started to learn BERT and distillation recently. I have some general questions regarding this topic.

1. I want to compare the performance of BERT with different model size (transformer block number). Is it necessary to do distillation? If I just train a BERT with 6 Layers without distillation, does the performance look bad?

2. Do you have to do pretraining every time you change the layer number of BERT? Is it possible to just remove some layers in an existing pre-trained model and finetune on tasks?

3. Why BERT has 12 blocks? Not 11 or 13 etc. ? I couldn't find any explanation.

Thanks,
ZLK

Comments:
Hi ZLK, we're trying to keep Github issues on our repo strictly related to code. [Our forum](https://discuss.huggingface.co) is a much better place for general questions like yours, and the people over there will better be able to answer it! Cheers
TFRobertaMarkedLM model output issue
Author: yxu02
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
I fine tuned a TFRobertaMarkedLM thanks to this notebook:
https://www.kaggle.com/riblidezso/finetune-xlm-roberta-on-jigsaw-test-data-with-mlm

My model config is like this:
{
  "architectures": [
    "RobertaForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "model_type": "roberta",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "type_vocab_size": 1,
  "vocab_size": 50265
}

I found my model output shape is **[n, sent_max_len, vocb_size]**, whereas I'd like to get **[n, sent_max_len, emb_size]**. 

Is it because I should not use model.predict(test_sent)[0]? Thanks!

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
Hey @yxu02, we are trying to move questions on how to use transformer models to https://discuss.huggingface.co/. Would you mind posting it there again? :-)
Incorrect tokenization for MarianNMT models in example script.
Author: yvespeirsman
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2
- Platform: Linux-4.4.0-1110-aws-x86_64-with-Ubuntu-16.04-xenial
- Python version: 3.7.8
- PyTorch version (GPU?): 1.5.1+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
 Marian: @sshleifer 

## Information

Model I am using (Bert, XLNet ...): Marian (Helsinki-NLP/opus-mt-nl-fr)

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

I'm trying to finetune a MarianMT model (Helsinki-NLP/opus-mt-nl-fr), using the example finetuning script `examples/seq2seq/finetune.py`. Marian models have different sentencepiece models for the encoder and decoder, but it appears the script does not take this into account, as the source line and the target line are tokenized in exactly the same way (in `utils.py`, lines 115-116):

```
        source_inputs = encode_line(self.tokenizer, source_line, self.max_source_length)
        target_inputs = encode_line(self.tokenizer, tgt_line, self.max_target_length)
```

I suggest checking whether the tokenizer is a Marian tokenizer and if so, encoding with the tokenizer's `prepare_translation_batch` method. Doing this improves BLEU score on my validation data by >3 points.

## To reproduce

Steps to reproduce the behavior:

1. Simply run the `examples/seq2seq/finetune.sh` with a MarianModel.


Comments:
Thanks for this catch and detailed report. Feel free to chime in on the Linked pull request #6293 !
[s2s]Use prepare_translation_batch for Marian finetuning
Author: sshleifer
Pull request: yes

Fix #6262 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6293?src=pr&el=h1) Report
> Merging [#6293](https://codecov.io/gh/huggingface/transformers/pull/6293?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d5bc32ce92ace9aaec7752e0b89d51ba18903a1b&el=desc) will **decrease** coverage by `0.55%`.
> The diff coverage is `0.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6293/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6293?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6293      +/-   ##
==========================================
- Coverage   79.56%   79.00%   -0.56%     
==========================================
  Files         147      147              
  Lines       27125    27127       +2     
==========================================
- Hits        21581    21433     -148     
- Misses       5544     5694     +150     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6293?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_marian.py](https://codecov.io/gh/huggingface/transformers/pull/6293/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbWFyaWFuLnB5) | `66.95% <0.00%> (-26.85%)` | :arrow_down: |
| [src/transformers/tokenization\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/6293/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxtLnB5) | `16.26% <0.00%> (-66.67%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6293/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `82.44% <0.00%> (+0.25%)` | :arrow_up: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6293/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.46% <0.00%> (+5.76%)` | :arrow_up: |
| [src/transformers/tokenization\_bart.py](https://codecov.io/gh/huggingface/transformers/pull/6293/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmFydC5weQ==) | `95.77% <0.00%> (+35.21%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6293?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6293?src=pr&el=footer). Last update [d5bc32c...ea610f8](https://codecov.io/gh/huggingface/transformers/pull/6293?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


thanks for the nit!
MarianMT - How to find out the actual names of the languages? - Only language-codes are available
Author: sunnyville01
Pull request: no

# Question

## Information

I want to use the [MarianMT](https://huggingface.co/transformers/model_doc/marian.html) library for creating a translation application. There are lots of languages available with their language-codes mentioned such as "en" and "es", but no where can I find the actual names of the languages for each code. I know "en" is for English and "fr" for French, but what "bg" and "lu" and so many others that are available. [This resource](https://huggingface.co/Helsinki-NLP) doesn't mention the actual names as well.

Is there any place where I can find the corresponding names for these codes. I tried using some online source, but there seems to be a miss-match and I can't be sure if I got the match correct.


Comments:
https://huggingface.co/languages

@patil-suraj Thank you.
Reformer now requires PyTorch 1.6.0
Author: sgugger
Pull request: no

@patrickvonplaten, one of your recent PR (#6244) on Reformer introduces a dep on PyTorch 1.6.0 minimum by using `torch.cuda.default_generators`. We should see if we can find a way to work around this.

Comments:
yes! I think the change #6244 is also only necessary in PyTorch 1.6 -> maybe we can even hard-code `torch.__version__` in the code and use the appropriate functions accordingly. Think this is also very much related to the previous bug: https://github.com/pytorch/pytorch/issues/33546

#6300 should fix it.
[Reformer] fix default generators for pytorch < 1.6
Author: patrickvonplaten
Pull request: yes

As far as I know in PyTorch < 1.6 torch.cuda.default_generators is only defined when running on GPU => so we can simply add a `hasattr(torch.cuda, "default_generators")` to fix Reformer for PyTorch < 1.6.

Comments:
Good to merge for me.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6300?src=pr&el=h1) Report
> Merging [#6300](https://codecov.io/gh/huggingface/transformers/pull/6300?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/2f2aa0c89cab9a77560e6845578f917a61081c67&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6300/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6300?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #6300   +/-   ##
=======================================
  Coverage   79.14%   79.15%           
=======================================
  Files         148      148           
  Lines       27191    27191           
=======================================
+ Hits        21521    21522    +1     
+ Misses       5670     5669    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6300?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_reformer.py](https://codecov.io/gh/huggingface/transformers/pull/6300/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yZWZvcm1lci5weQ==) | `95.68% <100.00%> (ø)` | |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6300/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.71% <0.00%> (+0.25%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6300?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6300?src=pr&el=footer). Last update [2f2aa0c...f029d86](https://codecov.io/gh/huggingface/transformers/pull/6300?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Default value of `n_tpu_cores` in lightning_base.py
Author: xujiaze13
Pull request: no

https://github.com/huggingface/transformers/blob/2804fff8393dbda5098b8c9f5e36235e89c50023/examples/lightning_base.py#L294

The default setting ``0`` raise Error
``pytorch_lightning.utilities.exceptions.MisconfigurationException: `tpu_cores` can only be 1, 8 or [<1-8>]``
It should be replaced by ``None``.

Comments: