internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: huggingface/transformers

Issue:
Turning off Verbosity on QA model using Pipeline
Hi,

I am looking for a way to turn off the logs warnings. Please refer to the screenshot.

![image](https://user-images.githubusercontent.com/34964824/79020501-9e2a9100-7b3e-11ea-8f63-afb26b11a5a1.png)

This is currently what I am doing.

![image](https://user-images.githubusercontent.com/34964824/79020549-ba2e3280-7b3e-11ea-893d-a527939181a7.png)

Is there any way?


Discussion:
I was just wondering the same thing.

Resolving patch:
diff --git a/src/transformers/data/processors/squad.py b/src/transformers/data/processors/squad.py
--- a/src/transformers/data/processors/squad.py
+++ b/src/transformers/data/processors/squad.py
@@ -262,7 +262,15 @@ def squad_convert_example_to_features_init(tokenizer_for_convert):
 
 
 def squad_convert_examples_to_features(
-    examples, tokenizer, max_seq_length, doc_stride, max_query_length, is_training, return_dataset=False, threads=1
+    examples,
+    tokenizer,
+    max_seq_length,
+    doc_stride,
+    max_query_length,
+    is_training,
+    return_dataset=False,
+    threads=1,
+    tqdm_enabled=True,
 ):
     """
     Converts a list of examples into a list of features that can be directly given as input to a model.
@@ -315,12 +323,15 @@ def squad_convert_examples_to_features(
                 p.imap(annotate_, examples, chunksize=32),
                 total=len(examples),
                 desc="convert squad examples to features",
+                disable=not tqdm_enabled,
             )
         )
     new_features = []
     unique_id = 1000000000
     example_index = 0
-    for example_features in tqdm(features, total=len(features), desc="add example index and unique id"):
+    for example_features in tqdm(
+        features, total=len(features), desc="add example index and unique id", disable=not tqdm_enabled
+    ):
         if not example_features:
             continue
         for example_feature in example_features:
diff --git a/src/transformers/pipelines.py b/src/transformers/pipelines.py
--- a/src/transformers/pipelines.py
+++ b/src/transformers/pipelines.py
@@ -961,6 +961,7 @@ def __call__(self, *texts, **kwargs):
                 kwargs["doc_stride"],
                 kwargs["max_question_len"],
                 False,
+                tqdm_enabled=False,
             )
             for example in examples
         ]


Test patch:
Repository: huggingface/transformers

Issue:
Documentation and code mismatch in BertForMaskedLM forward method
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):

BertForMaskedLM

Language I am using the model on (English, Chinese ...):

Not important.

The problem arises when using:
* [x] my own modified scripts: (give details below)

The tasks I am working on is:

I am trying to obtain LM probabilities for research purposes.

## To reproduce

Steps to reproduce the behavior:

1. According to the documentation, the code below should work as expected.

However, masked_lm_loss and ltr_lm_loss values are in fact in 2nd and 1st positions. This is apparent if the code is inspected, i.e. lm_labels related code is executed after masked_lm_labels related code.

See https://github.com/huggingface/transformers/blob/908fa43b543cf52a3238129624f502240725a6a6/src/transformers/modeling_bert.py#L1001-L1014

and

https://github.com/huggingface/transformers/blob/908fa43b543cf52a3238129624f502240725a6a6/src/transformers/modeling_bert.py#L1014

The following code is an excerpt from my code.

```python
model = BertForMaskedLM.from_pretrained('bert-base-cased')
with torch.no_grad():
    outputs = model(token_ids_in_sentence_tensor,
                    masked_lm_labels=token_ids_in_sentence_tensor,
                            lm_labels=token_ids_in_sentence_tensor,
                            token_type_ids=segments_tensors)
    masked_lm_loss = outputs[0]
    ltr_lm_loss = outputs[1]
    predictions = outputs[2]
```

## Expected behavior

Explained above.

## Environment info
     
- `transformers` version:
- Platform: Darwin XXX-3.local 19.3.0 Darwin Kernel Version 19.3.0: Thu Jan  9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64 x86_64
- Python version: 3.7.4
- PyTorch version (GPU?): no GPU, 1.3.1
- Tensorflow version (GPU?): 2.0.0
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No



Discussion:

Resolving patch:
diff --git a/src/transformers/modeling_bert.py b/src/transformers/modeling_bert.py
--- a/src/transformers/modeling_bert.py
+++ b/src/transformers/modeling_bert.py
@@ -957,7 +957,7 @@ def forward(
             ltr_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), lm_labels.view(-1))
             outputs = (ltr_lm_loss,) + outputs
 
-        return outputs  # (masked_lm_loss), (ltr_lm_loss), prediction_scores, (hidden_states), (attentions)
+        return outputs  # (ltr_lm_loss), (masked_lm_loss), prediction_scores, (hidden_states), (attentions)
 
 
 @add_start_docstrings(


Test patch:
Repository: huggingface/transformers

Issue:
❓ Summarization example : Why no shuffling ?
# ❓ Questions & Help

Usually, when loading using Dataloader, data is shuffled for the training set. But in the case of the summarization example, data is not shuffled for training set :

https://github.com/huggingface/transformers/blob/1dc9b3c7847269961458c059ad8ad443b26bf60d/examples/summarization/bart/finetune.py#L105-L108

---

**Why data is not shuffled ?**


Discussion:
No good reason.
Do you feel comfortable sending a PR that shuffles for train loader only?


Resolving patch:
diff --git a/examples/summarization/bart/finetune.py b/examples/summarization/bart/finetune.py
--- a/examples/summarization/bart/finetune.py
+++ b/examples/summarization/bart/finetune.py
@@ -102,13 +102,13 @@ def test_epoch_end(self, outputs):
 
         return self.test_end(outputs)
 
-    def get_dataloader(self, type_path: str, batch_size: int) -> DataLoader:
+    def get_dataloader(self, type_path: str, batch_size: int, shuffle: bool = False) -> DataLoader:
         dataset = SummarizationDataset(self.tokenizer, type_path=type_path, **self.dataset_kwargs)
-        dataloader = DataLoader(dataset, batch_size=batch_size, collate_fn=dataset.collate_fn)
+        dataloader = DataLoader(dataset, batch_size=batch_size, collate_fn=dataset.collate_fn, shuffle=shuffle)
         return dataloader
 
     def train_dataloader(self) -> DataLoader:
-        dataloader = self.get_dataloader("train", batch_size=self.hparams.train_batch_size)
+        dataloader = self.get_dataloader("train", batch_size=self.hparams.train_batch_size, shuffle=True)
         t_total = (
             (len(dataloader.dataset) // (self.hparams.train_batch_size * max(1, self.hparams.n_gpu)))
             // self.hparams.gradient_accumulation_steps


Test patch:
Repository: huggingface/transformers

Issue:
GPT2 is not fully torch.jit.trace-able
# 🐛 Bug

## Information

I'm trying to use PyTorch's tracing on a pre-trained GPT2 model and running into the following warning emitted from torch.jit.trace:
```
/opt/miniconda3/envs/py3/lib/python3.6/site-packages/transformers/modeling_gpt2.py:144: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  w = w / math.sqrt(v.size(-1))
```

Additionally, if I inspect the graph generated from the trace, I can see that the denominator in that division expression is a constant value, not one determined by the size of the tensor.

## To reproduce

The following snippet can be used to repro the problem (based on the example in the Quickstart guide)
```python
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2').eval()
example_input = torch.tensor([ tokenizer.encode("The Manhattan bridge")])
traced_model = torch.jit.trace(model, example_input)
```

## Environment info
     
- `transformers` version: 2.8.0
- Python version: 3.6.10
- PyTorch version (GPU?): 1.4.0 (False)



Discussion:

Resolving patch:
diff --git a/src/transformers/modeling_gpt2.py b/src/transformers/modeling_gpt2.py
--- a/src/transformers/modeling_gpt2.py
+++ b/src/transformers/modeling_gpt2.py
@@ -17,7 +17,6 @@
 
 
 import logging
-import math
 import os
 
 import torch
@@ -143,7 +142,7 @@ def prune_heads(self, heads):
     def _attn(self, q, k, v, attention_mask=None, head_mask=None):
         w = torch.matmul(q, k)
         if self.scale:
-            w = w / math.sqrt(v.size(-1))
+            w = w / (v.size(-1) ** 0.5)
         nd, ns = w.size(-2), w.size(-1)
         mask = self.bias[:, :, ns - nd : ns, :ns]
         w = torch.where(mask, w, self.masked_bias)


Test patch:
Repository: huggingface/transformers

Issue:
max_qa_length is needed for funetune on multiple-choice problems
# 🚀 Feature request
In examples/multiple-choice/run_multiple_choice.py, the max_qa_length is not implemented as that in google/Albert, which would cause trouble when funetuning RACE.
This should be easy becauce there are similar codes in codes for run_squad.py.
### Your contribution
I have done the script and get similar RACE performance compared to google/Albert.

# Current Bug
In examples/multiple-choice/utils_multiple_choice.py
```python
# From 537 lines: 
            # It should be => 
            # inputs = tokenizer.encode_plus(
            #    text_a, text_b, add_special_tokens=True, max_length=max_length, pad_to_max_length=True, return_overflowing_tokens=True
            # )
            # Or the log info below would never be activated.
            inputs = tokenizer.encode_plus(
                text_a, text_b, add_special_tokens=True, max_length=max_length, pad_to_max_length=True,
            )
            if "num_truncated_tokens" in inputs and inputs["num_truncated_tokens"] > 0:
                logger.info(
                    "Attention! you are cropping tokens (swag task is ok). "
                    "If you are training ARC and RACE and you are poping question + options,"
                    "you need to try to use a bigger max seq length!"
                )
```



Discussion:

Resolving patch:
diff --git a/examples/multiple-choice/utils_multiple_choice.py b/examples/multiple-choice/utils_multiple_choice.py
--- a/examples/multiple-choice/utils_multiple_choice.py
+++ b/examples/multiple-choice/utils_multiple_choice.py
@@ -535,7 +535,12 @@ def convert_examples_to_features(
                 text_b = example.question + " " + ending
 
             inputs = tokenizer.encode_plus(
-                text_a, text_b, add_special_tokens=True, max_length=max_length, pad_to_max_length=True,
+                text_a,
+                text_b,
+                add_special_tokens=True,
+                max_length=max_length,
+                pad_to_max_length=True,
+                return_overflowing_tokens=True,
             )
             if "num_truncated_tokens" in inputs and inputs["num_truncated_tokens"] > 0:
                 logger.info(


Test patch:
Repository: huggingface/transformers

Issue:
❓ Warning : This overload of addcdiv_ is deprecated
# ❓ Questions & Help

When running the [official Colab example of GLUE](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/trainer/01_text_classification.ipynb), during training I receive a `UserWarning` :

```
/pytorch/torch/csrc/utils/python_arg_parser.cpp:756: UserWarning: This overload of addcdiv_ is deprecated:
	addcdiv_(Number value, Tensor tensor1, Tensor tensor2)
Consider using one of the following signatures instead:
	addcdiv_(Tensor tensor1, Tensor tensor2, *, Number value)
```

---

**Is it expected ?**




Discussion:
Not expected, but shouldn't be an issue. Feel free to open a PR swapping args in https://github.com/huggingface/transformers/blob/31eedff5a0fc47d60609089627af6698c21da88d/src/transformers/optimization.py#L165

Resolving patch:
diff --git a/src/transformers/optimization.py b/src/transformers/optimization.py
--- a/src/transformers/optimization.py
+++ b/src/transformers/optimization.py
@@ -162,7 +162,7 @@ def step(self, closure=None):
                     bias_correction2 = 1.0 - beta2 ** state["step"]
                     step_size = step_size * math.sqrt(bias_correction2) / bias_correction1
 
-                p.data.addcdiv_(-step_size, exp_avg, denom)
+                p.data.addcdiv_(exp_avg, denom, value=-step_size)
 
                 # Just adding the square of the weights to the loss function is *not*
                 # the correct way of using L2 regularization/weight decay with Adam,


Test patch:
Repository: huggingface/transformers

Issue:
MMBT doesn't inherit from nn.Module
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): MMBT

Language I am using the model on (English, Chinese ...): not related

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Minimal reproduction:
```python
from transformers import MMBTConfig, MMBTModel, AutoConfig, AutoModel

electra_config = AutoConfig.from_pretrained("google/electra-small-discriminator")
mmbt_config = MMBTConfig(electra_config)

model = AutoModel.from_config(electra_config)

mmbt = MMBTModel(mmbt_config, model, None)

mmbt()
```

output:
```
Traceback (most recent call last):
  File "mmbt_debug.py", line 11, in <module>
    mmbt()
TypeError: 'MMBTModel' object is not callable
```

You can see in the [source code](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_mmbt.py#L152) that it's currently only inheriting from `ModuleUtilsMixin`, but not `torch.nn.Module`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

We should be seeing a downstream error since I didn't pass in a real modal encoder or any input.  It should at least call `forward()`

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.9.1 (also tried 2.10.0)
- Platform: Darwin-19.4.0-x86_64-i386-64bit
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no (doesn't matter)
- Using distributed or parallel set-up in script?: no (doesn't matter)



Discussion:

Resolving patch:
diff --git a/src/transformers/modeling_mmbt.py b/src/transformers/modeling_mmbt.py
--- a/src/transformers/modeling_mmbt.py
+++ b/src/transformers/modeling_mmbt.py
@@ -149,7 +149,7 @@ def forward(self, input_modal, start_token=None, end_token=None, position_ids=No
     MMBT_START_DOCSTRING,
     MMBT_INPUTS_DOCSTRING,
 )
-class MMBTModel(ModuleUtilsMixin):
+class MMBTModel(nn.Module, ModuleUtilsMixin):
     r"""
         Outputs: `Tuple` comprising various elements depending on the configuration (config) and inputs:
             **last_hidden_state**: ``torch.FloatTensor`` of shape ``(batch_size, sequence_length, hidden_size)``


Test patch:
Repository: huggingface/transformers

Issue:
Loading config file bug
Hi guys.

I have released a new (XLNet based) transformer model for low-resource language Tigrinya 
 [(TigXLNet)](https://github.com/abryeemessi/Transferring-Monolingual-Model-to-Low-Resource-Language)  and found a bug when loading a pre-trained config file: 

My config file looks like:
https://s3.amazonaws.com/models.huggingface.co/bert/abryee/TigXLNet/config.json


       config = AutoConfig.from_pretrained("abryee/TigXLNet")
       print(config.d_head) #prints 48 even though d_head in the given config file is 64.




Discussion:
Hello! The `d_head` is actually computed in the configuration: `self.d_head = d_model // n_head`. 

It would probably be better to handle `d_head` directly, but currently the `d_model` and `d_head` are linked to each other. 

This would crash in your case as  `d_model // n_head != d_head`
@LysandreJik Thanks a lot. Yes, I am setting d_head directly before loading my model. But it would be nice to see the model load its configuration from the given config file. Just my opinion though :)
You're right! Raising an error if the `d_head` is wrong in #4747 

Resolving patch:
diff --git a/src/transformers/configuration_xlnet.py b/src/transformers/configuration_xlnet.py
--- a/src/transformers/configuration_xlnet.py
+++ b/src/transformers/configuration_xlnet.py
@@ -163,6 +163,10 @@ def __init__(
         self.n_layer = n_layer
         self.n_head = n_head
         assert d_model % n_head == 0
+        if "d_head" in kwargs:
+            assert kwargs["d_head"] == d_model // n_head, (
+                f"`d_head` ({kwargs['d_head']}) should be equal to `d_model // n_head` ({d_model // n_head})"
+            )
         self.d_head = d_model // n_head
         self.ff_activation = ff_activation
         self.d_inner = d_inner


Test patch:
Repository: huggingface/transformers

Issue:
Tokenizer.encode documentation not correct
# 🐛 Bug

## Information

Model I am using Bert (bert-base-german-cased):

Language I am using the model on German:

The problem arises when using:
* tokenizer of this model

The tasks I am working on is:
* encoding

## To reproduce

Steps to reproduce the behavior:

1. lang_model = 'bert-base-german-cased'
2. tokenizer = BertTokenizer.from_pretrained(lang_model)
3. test_sentence = 'Das war gut'
4. tokenizer.encode(test_sentence)
output: [3, 295, 185, 1522, 4]
5. tokenizer.convert_tokens_to_ids(tokenizer.tokenize(test_sentence))
output: [295, 185, 1522]

## Expected behavior

According to the documentation of the encoding method in

https://huggingface.co/transformers/main_classes/tokenizer.html 

these two outputs should be the same. The problem is, that the _encode_ method adds special tokens [CLS] and [SEP] at the beginning and the end. However, the transformation in line 5. does not. This is not a problem at all but one should consider correcting the online-documentation. That is, the hint 

**Same as doing self.convert_tokens_to_ids(self.tokenize(text))**

in the encode method is misleading. Instead, one could also add, that these two commands are the same if _add_special_tokens_ is set _False_ in the _encode_ method.

## Environment info
     
- `transformers` version: 2.8.0
- Platform: Windows 10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.5.0+cpu
- Tensorflow version (GPU?): None
- Using GPU in script?:No
- Using distributed or parallel set-up in script?:No



Discussion:
You're right! The documentation should be updated.

Resolving patch:
diff --git a/src/transformers/tokenization_utils.py b/src/transformers/tokenization_utils.py
--- a/src/transformers/tokenization_utils.py
+++ b/src/transformers/tokenization_utils.py
@@ -1377,9 +1377,10 @@ def encode(
         **kwargs
     ):
         """
-        Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary.
+        Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary. Adds the model-specific
+        special tokens (such as beginning of sequence, end of sequence, sequence separator).
 
-        Same as doing ``self.convert_tokens_to_ids(self.tokenize(text))``.
+        If specifying ``add_special_tokens=False``, same as doing ``self.convert_tokens_to_ids(self.tokenize(text))``.
 
         Args:
             text (:obj:`str`, :obj:`List[str]` or :obj:`List[int]`):


Test patch:
Repository: huggingface/transformers

Issue:
GPT2TokenizerFast raises pad_token error even if not used
# 🐛 Bug

## Information

Model: GPT2

Language: English

Encoding with the `GPT2TokenizerFast` causes a `pad_token` error to be sent to stderr, despite not attempting to access that property.

## To reproduce

```import transformers
from transformers import GPT2TokenizerFast

tokenizer = GPT2TokenizerFast.from_pretrained('gpt2')

res = tokenizer.encode("This is a sentence")

print(transformers.__version__)
```

Output:
```
Using pad_token, but it is not set yet.
2.11.0
```

## Expected behavior

I'm aware that GPT-2 doesn't include a pad token (#2630) - I haven't tried to use it. I would expect no error to be displayed until I try to access that property.

## Environment info

- `transformers` version: 2.11.0
- Platform: Ubuntu
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0 with CUDA
- Tensorflow version (GPU?): n/a
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no



Discussion:
I can reproduce and think I've found the cause, working on this.

Resolving patch:
diff --git a/src/transformers/tokenization_utils.py b/src/transformers/tokenization_utils.py
--- a/src/transformers/tokenization_utils.py
+++ b/src/transformers/tokenization_utils.py
@@ -2467,7 +2467,7 @@ def batch_encode_plus(
             strategy=truncation_strategy,
             pad_to_max_length=pad_to_max_length,
             padding_side=self.padding_side,
-            pad_token_id=self.pad_token_id,
+            pad_token_id=self.pad_token_id if self._pad_token is not None else None,
             pad_token_type_id=self.pad_token_type_id,
             pad_token=self._pad_token,
         ):


Test patch: