noamaanMulla-03/transformers-issues huggingface dataset:
Loading DistilBertModel with AutoModel gives 12 layers  
Author: sasaadi
Pull request: no
Labels: name: wontfix

I am using ``AutoModel`` to load  ``distilbert-base-uncased`` and save the fine-tuned model after training using ``model.save_pretrained('path_to_save')``. However, when I load the fine-tuned model using ``AutoModel.from_pretrained('path_to_the_saved_model')``, it extracts 12 layers instead of 6 layers. I also checked the ``config.json`` file that was saved automatically and the number of layers is still 6. When I load the model with ``DistilBertModel.from_pretrained()`` it extracts 6 layers. In the following, I copied the ``config.json`` file. Does anyone know why this happens? am I lacking some packages or files when loading/saving the model?

> {
  "activation": "gelu",
  "architectures": [
    "DistilBertModel"
  ],
  "attention_dropout": 0.1,
  "bos_token_id": 0,
  "dim": 768,
  "do_sample": false,
  "dropout": 0.1,
  "eos_token_ids": 0,
  "finetuning_task": null,
  "hidden_dim": 3072,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "length_penalty": 1.0,
  "max_length": 20,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "num_beams": 1,
  "num_labels": 2,
  "num_return_sequences": 1,
  "output_attentions": true,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": 0,
  "pruned_heads": {},
  "qa_dropout": 0.1,
  "repetition_penalty": 1.0,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "temperature": 1.0,
  "tie_weights_": true,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 40000
}



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
DOC: Fix typos in modeling_auto
Author: bharatr21
Pull request: yes

Fix typo of word `dictionnary` => `dictionary`

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4534?src=pr&el=h1) Report
> Merging [#4534](https://codecov.io/gh/huggingface/transformers/pull/4534?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e19b978151419fe0756ba852b145fccfc96dbeb4&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4534/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4534?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4534      +/-   ##
==========================================
- Coverage   77.86%   77.86%   -0.01%     
==========================================
  Files         123      123              
  Lines       20566    20566              
==========================================
- Hits        16014    16013       -1     
- Misses       4552     4553       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4534?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/4534/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `78.57% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4534/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.50% <0.00%> (-0.17%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4534?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4534?src=pr&el=footer). Last update [e19b978...847fb92](https://codecov.io/gh/huggingface/transformers/pull/4534?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Architectures for Dialogue
Author: lukasfrank
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
Hi 👋
I'm trying to build a dialogue system which should reply based on a history, memory (which is represented as a string) and a confidence if the memory content is correct and should be used. 

Here two examples: 

- history: _Hi_ 
memory: _name: Max_ 
confidence: _0.2_
=> expected output: _Hi, what's your name?_

- history: _Hi_ 
memory: _name: Max_ 
confidence: _0.9_
=> expected output: _Hi Max_


First of all, are there some best practices how to encode non textual input or inject such information into a model? 

I already trained a Bert2Bert model which is not really performing very well. The generated response seems to be not conditioned on the encoder output. 

Are there any recommendations what to try next? 

Many thanks in advance for your hints! 






Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Did you check out [DialoGPT](https://huggingface.co/microsoft/DialoGPT-large) by @dreasysnail?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Error in loading  albert-base-v2
Author: WenxiongLiao
Pull request: no
Labels: name: wontfix

![image](https://user-images.githubusercontent.com/25845940/76815952-f3be9700-6839-11ea-9ef9-53bc3655cefa.png)
![image](https://user-images.githubusercontent.com/25845940/76815974-020cb300-683a-11ea-830a-8a26b1eb6822.png)

Help~~
The above problems appear in loading the pre-trained Albert by transformers.  

Comments:
@anjubaoGDUT If you can provide the code in text (instead of image) that can copy and paste, it is easy to test. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
RuntimeError: The size of tensor a (1025) must match the size of tensor b (1024) at non-singleton dimension 3
Author: rautnikita77
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): gpt2

Language I am using the model on (English, Chinese ...): english

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)





The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
I want to generate samples with length = 2000
## To reproduce

Steps to reproduce the behavior:
1. run the command
python run_generation.py     --model_type=gpt2     --model_name_or_path=<output_dir_of_finetuned_model>     --length=2000     --num_return_sequences=10     --stop_token='

'


## Expected behavior
The error that I am getting is
w = torch.where(mask.bool(), w, self.masked_bias.to(w.dtype))
RuntimeError: The size of tensor a (1024) must match the size of tensor b (1025) at non-singleton dimension 3

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: Python 3.8
- Python version:
- PyTorch version (GPU?): 10.1
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Comments:
Length of 2000 is too long for GPT2, this won't be possible. 
You can do two things here:

1) You chunk your generation which means that you first produce a length of up to 1000 and then use a bit of that (100 or so tokens) as context to generate the next 900 tokens and the same again until you hit 2000.

2) You can use this Reformer model: https://huggingface.co/google/reformer-enwik8 which can handle sequences up to 65000. Currently generation with Reformer is painfully slow though :-/ This should be improved in the coming weeks :-)
Reproducing SQuAD v1.1 with xlnet-base cased?
Author: JJumSSu
Pull request: no
Labels: name: wontfix

Hi, first of all, thanks for the great library you guys are providing. I'm currently using the latest version of huggingface/transformers, and I'm trying to get a score for the SQuAD V1.1 with XLNET base-cased. However, it seems that the performance I get is only 0.10 for EM and 0.64 for F1. 

When getting the score with BERT base-cased, the score comes out appropriately. (F1 about 88.5)

Is there any bugs or sth else I should be aware of?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
implementation of transformers for abstractive summarization task
Author: fatihbeyhan
Pull request: no

Hello, I am new to whole NLP world and PyTorch. I am trying to learn the concepts and that is taking some time for a rookie. I have a project to finish and I want to implement transformers & BERT on my abstractive summarization project. I tried to find some implementation tutorial on this topic but I could not find any. Do you guys have any suggestions about clear implementation of any pre-trained model that I can fine-tune on my dataset to get some solid results. I am not looking for this just for finishing up the project but also learn how to implement. Therefore, I need a clear tutorial.


Data:
I am using 0.25 of XSum dataset so I have 45k news and their one-sentence summary. 


Thank you in advance.

Comments:
@sshleifer might be able to help you here. @sshleifer - hope it's fine that I link you here :-) 

This might also help: https://github.com/huggingface/transformers/pull/4539/files?short_path=3a2ba7b#diff-3a2ba7b492f00029d14cec3994b73ac7

> This might also help: https://github.com/huggingface/transformers/pull/4539/files?short_path=3a2ba7b#diff-3a2ba7b492f00029d14cec3994b73ac7

Thank you very much.  I will look and try to implement and let you know about the result!

> This might also help: https://github.com/huggingface/transformers/pull/4539/files?short_path=3a2ba7b#diff-3a2ba7b492f00029d14cec3994b73ac7

it seems working! thank you!
Add DistilBERT to supported run_language_modeling models
Author: antmarakis
Pull request: yes

As per the code, distilbert is indeed supported by the `run_language_modeling.py` script, even though the README states otherwise.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4558?src=pr&el=h1) Report
> Merging [#4558](https://codecov.io/gh/huggingface/transformers/pull/4558?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a34a9896ac2a4a33ff9cd805c76eed914c8d8965&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4558/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4558?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4558      +/-   ##
==========================================
- Coverage   77.87%   77.86%   -0.01%     
==========================================
  Files         123      123              
  Lines       20566    20566              
==========================================
- Hits        16016    16014       -2     
- Misses       4550     4552       +2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4558?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4558/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.44% <0.00%> (-0.42%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4558/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.41% <0.00%> (-0.12%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4558/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.66% <0.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4558?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4558?src=pr&el=footer). Last update [a34a989...ce96482](https://codecov.io/gh/huggingface/transformers/pull/4558?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks! That's correct
Added reference to use for citing this model
Author: alisafaya
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4556?src=pr&el=h1) Report
> Merging [#4556](https://codecov.io/gh/huggingface/transformers/pull/4556?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a34a9896ac2a4a33ff9cd805c76eed914c8d8965&el=desc) will **increase** coverage by `0.05%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4556/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4556?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4556      +/-   ##
==========================================
+ Coverage   77.87%   77.93%   +0.05%     
==========================================
  Files         123      123              
  Lines       20566    20566              
==========================================
+ Hits        16016    16028      +12     
+ Misses       4550     4538      -12     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4556?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4556/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.84% <0.00%> (-0.83%)` | :arrow_down: |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/4556/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `34.07% <0.00%> (+5.41%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4556?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4556?src=pr&el=footer). Last update [a34a989...433d479](https://codecov.io/gh/huggingface/transformers/pull/4556?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Create model card
Author: mrm8488
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4578?src=pr&el=h1) Report
> Merging [#4578](https://codecov.io/gh/huggingface/transformers/pull/4578?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a34a9896ac2a4a33ff9cd805c76eed914c8d8965&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4578/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4578?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4578      +/-   ##
==========================================
- Coverage   77.87%   77.86%   -0.01%     
==========================================
  Files         123      123              
  Lines       20566    20566              
==========================================
- Hits        16016    16014       -2     
- Misses       4550     4552       +2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4578?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4578/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.44% <0.00%> (-0.42%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4578/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.41% <0.00%> (-0.12%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4578/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.66% <0.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4578?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4578?src=pr&el=footer). Last update [a34a989...03b9ffc](https://codecov.io/gh/huggingface/transformers/pull/4578?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
MMBT doesn't inherit from nn.Module
Author: shoarora
Pull request: no
Labels: name: PyTorch

name: Core: Modeling

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): MMBT

Language I am using the model on (English, Chinese ...): not related

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Minimal reproduction:
```python
from transformers import MMBTConfig, MMBTModel, AutoConfig, AutoModel

electra_config = AutoConfig.from_pretrained("google/electra-small-discriminator")
mmbt_config = MMBTConfig(electra_config)

model = AutoModel.from_config(electra_config)

mmbt = MMBTModel(mmbt_config, model, None)

mmbt()
```

output:
```
Traceback (most recent call last):
  File "mmbt_debug.py", line 11, in <module>
    mmbt()
TypeError: 'MMBTModel' object is not callable
```

You can see in the [source code](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_mmbt.py#L152) that it's currently only inheriting from `ModuleUtilsMixin`, but not `torch.nn.Module`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

We should be seeing a downstream error since I didn't pass in a real modal encoder or any input.  It should at least call `forward()`

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.9.1 (also tried 2.10.0)
- Platform: Darwin-19.4.0-x86_64-i386-64bit
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no (doesn't matter)
- Using distributed or parallel set-up in script?: no (doesn't matter)


Comments:
Model card: Updated the link to the paper
Author: oliverguhr
Pull request: yes
Labels: name: model card

The conference has changed the link to the paper, so I updated it. 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4570?src=pr&el=h1) Report
> Merging [#4570](https://codecov.io/gh/huggingface/transformers/pull/4570?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a34a9896ac2a4a33ff9cd805c76eed914c8d8965&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4570/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4570?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4570      +/-   ##
==========================================
- Coverage   77.87%   77.87%   -0.01%     
==========================================
  Files         123      123              
  Lines       20566    20566              
==========================================
- Hits        16016    16015       -1     
- Misses       4550     4551       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4570?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4570/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.44% <0.00%> (-0.42%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4570/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.41% <0.00%> (-0.12%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4570/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.83% <0.00%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4570?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4570?src=pr&el=footer). Last update [a34a989...78cf772](https://codecov.io/gh/huggingface/transformers/pull/4570?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).