noamaanMulla-03/transformers-issues huggingface dataset:
[examples] Cleanup summarization docs
Author: sshleifer
Pull request: yes

Don't think we need a download_cnn_dailymail.py script


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4876?src=pr&el=h1) Report
> Merging [#4876](https://codecov.io/gh/huggingface/transformers/pull/4876?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/02e5f79662d72cccdca81a47e3001a5f6d36e5b1&el=desc) will **decrease** coverage by `0.59%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4876/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4876?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4876      +/-   ##
==========================================
- Coverage   76.46%   75.86%   -0.60%     
==========================================
  Files         128      128              
  Lines       21502    21502              
==========================================
- Hits        16442    16313     -129     
- Misses       5060     5189     +129     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4876?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/4876/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `28.15% <0.00%> (-63.03%)` | :arrow_down: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/4876/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `94.73% <0.00%> (+10.04%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4876?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4876?src=pr&el=footer). Last update [02e5f79...954557d](https://codecov.io/gh/huggingface/transformers/pull/4876?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
tokenizer.encode_plus stopped returning `attention_mask`
Author: wasiahmad
Pull request: no

I have a codebase which was working fine but today when I was trying to run, I observed that `tokenizer.encode_plus` stopped returning `attention_mask`. Is it removed in the latest release? Or, I need to do something?

The following piece of code was working for me.

```
encoded_dict = tokenizer.encode_plus(
                truncated_query,
                span_doc_tokens,
                max_length=max_seq_length,
                return_overflowing_tokens=True,
                pad_to_max_length=True,
                stride=max_seq_length - doc_stride - len(truncated_query) - sequence_pair_added_tokens,
                truncation_strategy="only_second",
                return_token_type_ids=True,
                return_attention_mask=True
            )
```

But now, I get only `dict_keys(['input_ids', 'token_type_ids'])` from `encode_plus`. Also, I realized that the returned `input_ids` are not padded to `max_length`.

Comments:
How did you resolve this? I am facing this issue as well.

@shikharsingla, what is your exact bug? Which tokenizer are you using, with which code?

Just saw your issue https://github.com/huggingface/transformers/issues/4868, looking into it.
how to load checkpoint?
Author: susht3
Pull request: no

i download the model from bert, it only has model.ckpt.data，model.ckpt.meta and model.ckpt.index, i donnot which to load, what is checkpoint file for convert.py?

Comments:
Converting TensorFlow checkpoint from ../dataset/bert/uncased_L-12_H-768_A-12/bert_model
Traceback (most recent call last):
  File "convert_tf_checkpoint_to_pytorch.py", line 111, in <module>
    convert()
  File "convert_tf_checkpoint_to_pytorch.py", line 60, in convert
    init_vars = tf.train.list_variables(path)
  File "/home/susht3/local/anaconda3/envs/susht/lib/python3.6/site-packages/tensorflow/python/training/checkpoint_utils.py", line 95, in list_variables
    reader = load_checkpoint(ckpt_dir_or_file)
  File "/home/susht3/local/anaconda3/envs/susht/lib/python3.6/site-packages/tensorflow/python/training/checkpoint_utils.py", line 64, in load_checkpoint
    return pywrap_tensorflow.NewCheckpointReader(filename)
  File "/home/susht3/local/anaconda3/envs/susht/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 326, in NewCheckpointReader
    return CheckpointReader(compat.as_bytes(filepattern), status)
  File "/home/susht3/local/anaconda3/envs/susht/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for ../dataset/bert/uncased_L-12_H-768_A-12/bert_model

@susht3 what was your fix? 

I encountered a similar issue and didn't find a solution with ALBERT. I tried using the `export_checkpoint.py` file in ALBERT and sent that into the `convert_tf_checkpoint_to_pytorch` command and there was no error. However the resulting `pytorch.bin` output was unusable :\

@dan-hu-spring do you mind opening a new issue with your issue, so that we may take a look?
Problem with https://transformer.huggingface.co/doc/gpt2-xl
Author: MarxEngelsLeninStalin
Pull request: no
Labels: name: wontfix

Hi, when using it, simple switch the button up to xl one. When you try to generate a prediction of words; it will simply never load regardless of what all the other setting are on. Every other gpt2 model size works; expect the xl one. Tried it on Android and Linux same problem and different browsers Firefox and Google Chrome same problem nothing to with addons cause the same problem occurs with them all off.

Comments:
Duplicate of https://github.com/huggingface/transformers/issues/3452.

We should just remove the option cc @julien-c 

You’re right, I’ll get on it

Also best username ever, @MarxEngelsLeninStalin 

Oh, I like that model as someone who has problems with spelling/grammar and fatigue it's useful to use xl one has more knowledge on it (yes I know it's not accurate but more accurate then large model). Is there any prospect of it being turned on again in the future?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@Marxist-Leninist, unfortunately adding it back isn't on our near future roadmap.
Would the weights for the main body of the pertained GPT2Model and pertained GPT2DoubleHeadsModel be identical?
Author: h56cho
Pull request: no
Labels: name: wontfix

Hello,

From my understanding, the difference between the GPT2Model and GPT2DoubleHeadsModel is that GPT2Model does not include any output head, whereas GPT2DoubleHeadsModel includes two types of output heads (lmhead and mchead).

I am wondering, would the weights used in the pertained GPT2Model identical to the weights used in the main body (every parts of the model except the output heads) of the pertained GPT2DoubleHeadsModel? or would the two sets of weights be different since GPT2DoubleHeadsModel was trained after including the output heads, whereas GPT2Model was trained without any output head?

Thank you (I hope my question is understandable),

Comments:
Hi, except if you freeze the transformer's (transformer being the base model) weights during the training, these will be modified during the fine-tuning. I believe none of our scripts in the library freeze the transformer's weight; so the weights would not be identical.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Why the activation function is tanh in BertPooler
Author: xinliweiyuan
Pull request: no

I found the activation function in the BertPooler layer is tanh, but Bert never mentions that it uses the tanh. It says gelu activation function is applied in the paper.

So why there is a tanh here ? Waiting for some explanation. Thanks.

```
class BertPooler(nn.Module):
    def __init__(self, config):
        super(BertPooler, self).__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.activation = nn.Tanh()

    def forward(self, hidden_states):
        # We "pool" the model by simply taking the hidden state corresponding
        # to the first token.
        first_token_tensor = hidden_states[:, 0]
        pooled_output = self.dense(first_token_tensor)
        pooled_output = self.activation(pooled_output)
        return pooled_output
```

Comments:
Because that's what Bert's authors do in the official TF code:
https://github.com/google-research/bert/blob/bee6030e31e42a9394ac567da170a89a98d2062f/modeling.py#L231

Just wanted to point out for future reference the motivation has been answered by the original BERT authors in [[this GitHub issue]](https://github.com/google-research/bert/issues/43).
default of weight_decay  for run_language_modeling.py 
Author: mahdirezaey
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
Hi 

the default value of weight_decay is "0" for run_language_modeling.py  , why is  that ?

shouldn't it be  0.01  according to original paper of BERT ?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Latest version of transformers available via conda?
Author: rmartinshort
Pull request: no
Labels: name: wontfix

name: Installation

name: Should Fix

# 🚀 Feature request

I notice that the version of transformers available via conda-forge is quite outdated (v2.1.1). Could you make later versions available there too? Thanks!


Comments:
It would be nice if it had it's own channel like PyTorch has, as put a distribution of the packages (transformers, tokenizers, etc.) there.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Any update on this request?
The latest version in conda forge is 2.1.1. 
Thanks.
Any Ideas on how to generate in bulk with CTRL?
Author: AdaUchendu
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Is it possible to generate multiple articles with a list of prompts using CTRL?
Any ideas will be greatly appreciated
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->


Comments:
CTRL is a very large model so generating in bulk would require a lot of RAM. 
Also generating with padded batches is not really supported yet, see: https://github.com/huggingface/transformers/issues/3021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
AutoModelForSequenceClassification not working with prunebert model
Author: atowey01
Pull request: no

I am having issues loading the new prunebert model for sequence classification using AutoModelForSequenceClassification.from_pretrained(). 

```
from transformers import AutoModelForSequenceClassification, BertForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained("huggingface/prunebert-base-uncased-6-finepruned-w-distil-mnli")
```
The above code produces the error: KeyError: 'masked_bert'.

The model loads fine using BertForSequenceClassification.from_pretrained, the issue only seems to occur with AutoModelForSequenceClassification. 

```
model = BertForSequenceClassification.from_pretrained('huggingface/prunebert-base-uncased-6-finepruned-w-distil-mnli')
```


Comments:
This is because the `huggingface/prunebert-xxx` configurations have a model type `masked_bert`.

Since these files should be loaded directly in `BertForXXX` classes, it would probably be best to update that field to `bert`, right @julien-c?

Yes. Can you confirm @VictorSanh?

Seems to be working fine now, thanks @LysandreJik and @julien-c
Replace `config.output_attentions` parameter with function argument `output_attentions`
Author: patrickvonplaten
Pull request: no
Labels: name: Good First Issue

# 🚀 Feature request

Currently the user has to decide whether the model should output the attentions when she/he creates the config of a model: config.output_attentions = True/False. It would be nice if the user can decide this when calling the models `forward()` / `call()` with a flag `output_attentions`. This should be done for all TF and PT models that can output attentions.

A very similar recent change was done for the variable `config.output_past` -> see PR:#3734

## Motivation

The user has more flexibility when the hidden states should be output or not.

## Your contribution

If someone feels like contributing to the library, this would be a great first PR. I'm very happy to guide the contributor through the PR!


Comments:
Hi, I would like to work on this issue.

That's great :-) Do you want to open a PR and do the changes analogous to PR: #3734 ? 

Is this still be worked on? If not, I'd be happy to make a first contribution here

First PR first serve ;-) Still an open issue

Any tips on how I should proceed? I was thinking of following the changes made for `config.output_past` (01c37dc), but for `config.output_attentions` instead.

Oh sorry @drjosephliu didn't notice the comment as I was working on it earlier today, my apologies 😞 

Hey @patrickvonplaten, i noticed this issue has been closed. Any updates on what changes were made and any updates to the PR i still need to make?

Hey @drjosephliu, I will take a closer look at your PR tomorrow :-)
[ctrl] fix pruning of MultiHeadAttention
Author: aretius
Pull request: yes

@sshleifer 
Implemented the pruning logic. Fixes - #4798 
After enabling `test_pruning` all the previously failing tests are passing.

Comments:
Hi @aretius, some (if not all) the tests failing are unrelated to your PR, and should have been solved by the recently merged #4903. Do you mind rebasing on `master` and force pushing so that we may see if all the tests pass?

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4904?src=pr&el=h1) Report
> Merging [#4904](https://codecov.io/gh/huggingface/transformers/pull/4904?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ac99217e92c43066af7ec96554054d75532565d7&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `93.33%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4904/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4904?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4904      +/-   ##
==========================================
+ Coverage   76.99%   77.01%   +0.01%     
==========================================
  Files         128      128              
  Lines       21602    21615      +13     
==========================================
+ Hits        16633    16647      +14     
+ Misses       4969     4968       -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4904?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/4904/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jdHJsLnB5) | `99.14% <93.33%> (+0.50%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4904?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4904?src=pr&el=footer). Last update [ac99217...bf94b7e](https://codecov.io/gh/huggingface/transformers/pull/4904?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@sshleifer do you want to take a look?
[ctrl] has broken code for pruning that is not tested
Author: sshleifer
Pull request: no

references an `attn` parameter than should be `multiheaded_attn`. Easy-ish fix.


Comments:
Yeah change 
```python
self.h[layer].attn.prune_heads(heads)
``` 
to 
```python
self.h[layer].multi_head_attention.prune_heads(heads)
``` 
in modeling_ctrl.py

and set `test_pruning=True` in `CTRLModelTest`

@sshleifer 
I did the above-mentioned changes however the testing of pruning still failed. There is no `function` called `prune_heads` implemented in the `MultiHeadAttention` class.
In `modelling_xlm.py` I do observe the implementation of `prune_heads`. 
Should I just raise a PR with the above-mentioned changes or look into implementing `prune_heads` for `ctrl`(Would require some help there 😓)?
![Annotation 2020-06-10 120040](https://user-images.githubusercontent.com/18247856/84234521-0b907900-ab12-11ea-8cec-3f798866feac.png)


You could try to get that test passing or 
work on https://github.com/huggingface/transformers/issues/4902, which is easier in my opinion.
Don't init TPU device twice
Author: LysandreJik
Pull request: yes

closes #4893

The TPU device was initialized twice when using a the `xla_spawn.py` script. Removing this initialization solves the issue.

@patrickvonplaten, is this necessary for the benchmarking script?

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4916?src=pr&el=h1) Report
> Merging [#4916](https://codecov.io/gh/huggingface/transformers/pull/4916?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ef2dcdccaa9a115aca44d81f31c6dc4d32bebb3f&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4916/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4916?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4916   +/-   ##
=======================================
  Coverage   77.12%   77.13%           
=======================================
  Files         128      128           
  Lines       21650    21649    -1     
=======================================
+ Hits        16698    16699    +1     
+ Misses       4952     4950    -2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4916?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4916/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `74.19% <100.00%> (+0.69%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4916?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4916?src=pr&el=footer). Last update [ef2dcdc...acb4cb4](https://codecov.io/gh/huggingface/transformers/pull/4916?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Oh, my bad! Not sure why I left it there

No worries!
Run a single wandb instance per TPU run
Author: LysandreJik
Pull request: yes


Comments:
Why is it specific to tpu only?
Would the same logic apply in all cases?

Yes I'm guessing we only want the global master to log to wandb in DDP, like we do for Tensorboard.

Not sure why we hadn't done it like that before, @borisdayma – thoughts?

I did not consider DP/DDP at the time. Actually Tensorboard logging does not consider world master either (only for logging config parameters but not metrics).

I understand we should wrap the entire `wandb` and Tensorboard logics within a simple `if self.is_world_master`.

May I suggest the following:

* refactor logging through PR #4756 
* add an equivalent `TFTrainer.is_world_master`
* wrap relevant Tensorboard & wandb sections of `log_metrics` by checking `is_world_master`
* call `setup_wandb` only for world master (checked either within `Trainer` & `TFTrainer` or within `setup_wandb`)

Let me know if you want me to add those changes.

I think those changes would be welcome. Do you agree @julien-c ?

Yes I agree. Should we already merge this PR though?

Sure, it won't do any harm.