noamaanMulla-03/transformers-issues huggingface dataset:
Transfer Learning on Text Summarization Model
Author: imayachita
Pull request: no
Labels: name: seq2seq

Hi all,
Is there any way to do transfer learning on the Text Summarization model (bertabs-finetuned-cnndm)? I would like to continue training it on my dataset.
The code run_summarization.py only does prediction. Thanks!

Comments:
@imayachita If your succeeded to use transfer learning on your own data, please update here.

`examples/summarization/bart/run_bart_sum.py` now exists :)
[BART] Update encoder and decoder on set_input_embedding
Author: dougian
Pull request: yes

Since  _resize_token_embeddings will create a new embedding layer,
resizing the input embeddings for BART will currently break, as the
model.shared will refer to the new embedding created but the
model.encoder.embed_tokens and the model.decoder.embed_tokens will
still refer to the old embedding created.

We need to re-assign the encoder/decoder or just their weights, I opted for the second option.
Unfortunately can't see how to write a test in the test_resize_tokens_embeddings to capture this without putting a BART-specific if statement there, but this is also related to https://github.com/huggingface/transformers/issues/3378

Run tests:
 733 passed, 319 skipped, 80 warnings in 269.62s (0:04:29) 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3501?src=pr&el=h1) Report
> Merging [#3501](https://codecov.io/gh/huggingface/transformers/pull/3501?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/601ac5b1dc1438f00d09696588f2deb0f045ae3b&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3501/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3501?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3501   +/-   ##
=======================================
  Coverage   77.79%   77.80%           
=======================================
  Files         100      100           
  Lines       17051    17053    +2     
=======================================
+ Hits        13265    13268    +3     
+ Misses       3786     3785    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3501?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_bart.py](https://codecov.io/gh/huggingface/transformers/pull/3501/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iYXJ0LnB5) | `97.59% <100.00%> (+0.01%)` | :arrow_up: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3501/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.94% <0.00%> (+0.13%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3501?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3501?src=pr&el=footer). Last update [601ac5b...5e11181](https://codecov.io/gh/huggingface/transformers/pull/3501?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[bart-tiny-random] Put a 5MB model on S3 to allow faster examples test
Author: sshleifer
Pull request: yes

- Vocab size is the same to avoid requiring a new tokenizer.
- this allows examples that parametrize `model_name`, like `evaluate_cnn.py`, to run much quicker: `summarization/bart/test_bart_examples.py` runs in 6 seconds vs 22 + download time previously.
- Would be happy to do this for more models.
- Will update `run_sum.py` if this idea is OK with people.
- this also makes local debugging easier. You can easily pull down a tiny model without making a config.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3488?src=pr&el=h1) Report
> Merging [#3488](https://codecov.io/gh/huggingface/transformers/pull/3488?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/17dceae7a1de5577cd0c07a97dcd5821a08af07c&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3488/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3488?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3488      +/-   ##
==========================================
- Coverage   77.80%   77.79%   -0.01%     
==========================================
  Files         100      100              
  Lines       17051    17051              
==========================================
- Hits        13266    13265       -1     
- Misses       3785     3786       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3488?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3488/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.81% <0.00%> (-0.14%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3488?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3488?src=pr&el=footer). Last update [17dceae...3e0a394](https://codecov.io/gh/huggingface/transformers/pull/3488?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


LGTM

Merging to unblock other testing efforts!
Documentation markup for model descriptions
Author: BramVanroy
Pull request: no
Labels: name: wontfix

## 🐛 Bug

Looking at [the documentation](https://huggingface.co/transformers/model_doc/bert.html#bertforsequenceclassification), it seems something went wrong in markup land. In some models (but not all, e.g. BertModel, BertForMaskedLM, BertForNextSentencePrediction), the model description (i.e. the first paragraph) is split up in one highlighted line (grey background) and the rest of the text is regular. Seems like a formatting issue in the source code.

Comments:
Hi! Indeed there were quite a few issues with the documentation. #2532 was merged this morning, and hopefully fixes all these issues!

Would love your feedback on the new documentation (be sure to refresh your cache to see the new doc on https://huggingface.co/transformers). 

Ah, sorry, didn't check the recent commits. Just checked a couple of items. Everything seems in order except for a small inconsistency in the tokenizers:

Some (e.g. [openai](https://huggingface.co/transformers/model_doc/gpt.html#openaigpttokenizer)) put the first line (the one with 'peculiarities') in a highlighted block, while others (e.g. [XLNet](https://huggingface.co/transformers/model_doc/xlnet.html#xlnettokenizer)) don't.

Good catch! Up to now I've reworked the configuration + models + glossary. I've yet to do the tokenizers as well as the abstract classes, will work on them in the coming days.

If you've finalized that, feel free to close this issue through a commit. I'll have another look, then!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
TypeError when using Feature Extraction Pipeline with XLM roberta
Author: aeshapar
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): XLM-R

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. after importing the required libraries, run the following snipped of code:
```
xlmr_model = XLMRobertaForSequenceClassification.from_pretrained("xlm-roberta-base")
xlmr_tokenizer = XLMRobertaTokenizer.from_pretrained("xlm-roberta-base")
nlp = pipeline(task ="feature-extraction", model = xlmr_model, tokenizer=xlmr_tokenizer, framework="tf")
features = nlp('We are very happy to include pipeline into the transformers repository.')
print(features)
```
2. You should get the following error: result = self.forward(*input, **kwargs)
**TypeError: forward() got an unexpected keyword argument 'training'**

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I would expect this code to run without this error.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.6.1
- Platform: Ubuntu 16.04.6 LTS
- Python version: 3.6.1
- PyTorch version (GPU?): No GPU
- Tensorflow version (GPU?): No GPU
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
Bad Results with Albert
Author: chikubee
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
Trying to understand why is the cosine similarity between tokens with Albert way bad in comparison to DistilBert. 
Any inferences on the same would be helpful.
Thanks in advance. 

Embeddings constructed for a token by summing the last 4 encoded layers.
Distance metric: cosine

Results with DistilBert
<img width="754" alt="Screenshot 2020-01-22 at 4 05 06 PM" src="https://user-images.githubusercontent.com/25073753/72887172-2e6feb80-3d31-11ea-876b-0ba8eac22234.png">
<img width="613" alt="Screenshot 2020-01-22 at 4 12 27 PM" src="https://user-images.githubusercontent.com/25073753/72887650-25334e80-3d32-11ea-8255-26109f604c84.png">

Results with Albert
<img width="735" alt="Screenshot 2020-01-22 at 4 04 56 PM" src="https://user-images.githubusercontent.com/25073753/72887227-447dac00-3d31-11ea-8797-9873c8439879.png">
<img width="499" alt="Screenshot 2020-01-22 at 4 10 16 PM" src="https://user-images.githubusercontent.com/25073753/72887665-2bc1c600-3d32-11ea-95aa-83b3192e9d49.png">




Comments:
Hi. Many people are reporting unstable results or just unexpected results. You can search for issues in this library, and even in other ones (e.g. https://github.com/deepset-ai/FARM/issues/202#issuecomment-577077201). It seems that ALBERT is very sensitive to hyperparameters and even then... For now there seems to be no solution. It is probably best to stick to another model. I'd recommend RoBERTa but it depends on your use-case.

@BramVanroy I tried with roberta-base as well, the token level similarity is coming out very bad. 
Smoking is getting matched with software.

Can you share a repo to your full code?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Training GPT or GPT-2 from scratch
Author: LeonCrashCode
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I am trying to retrain GPT or GPT-2 from scratch, is there any implementation for this?

Comments:
I think, just create an instance of the model (without loading from pretrained one), switch it to train mode and run. That's all.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I am currently trying to implement this as well. Once I figured it out i'll let you know !
Isort installed from github branch does not correspond to circle ci isort
Author: patrickvonplaten
Pull request: no

# 🐛 Bug

## Information

Installing isort via: 

`$ pip install -U git+git://github.com/timothycrosley/isort.git@e63ae06ec7d70b06df9e528357650281a3d3ec22#egg=isort`

does not correspond to the circle ci version anymore, so that `make style` formatting leads to circle-ci make the code quality test fail. 
     
- `transformers` version: 2.6.0                                                                          
- Platform: Linux-5.3.0-42-generic-x86_64-with-Ubuntu-18.04-bionic                                       
- Python version: 3.6.9                                                                                  
- PyTorch version (GPU?): 1.4.0+cpu (False)                                                              
- Tensorflow version (GPU?): 2.1.0 (False)                        


Comments:
Looking at the CI config, it should install the correct version, though. Not sure what I am missing here.

https://github.com/huggingface/transformers/blob/e5c393dcebf42eaec9c1e1d619b5a7788a2d7c65/.circleci/config.yml#L89

Hi @BramVanroy, 

Thanks for your answer :-) 
I get the feeling that it's somehow related to my computer.

Running `pip install git+git://github.com/timothycrosley/isort.git@e63ae06ec7d70b06df9e528357650281a3d3ec22#egg=isort` in my terminal installs the same isort version / the exact same code (isort-4.3.21)

as if running 

`pip install isort`

When I `pip uninstalled isort` and reinstalled it with `pip install git+git://github.com/timothycrosley/isort.git@e63ae06ec7d70b06df9e528357650281a3d3ec22#egg=isort` the library was not the same anymore. Any ideas what could be happening here on my computer? I tried deleting the pip cache, but no success yet :-/. 

Ok for some reason, upgrading python3.6 to python3.7 solved the problem for me
TFAlbertForMaskedLM Decoding Error
Author: zzj0402
Pull request: no

# 🐛 Bug

## Information

### Model
TFAlbertForMaskedLM <https://huggingface.co/transformers/model_doc/albert.html#tfalbertformaskedlm>

### Language
English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)
```python
import tensorflow as tf
from transformers import AlbertTokenizer, TFAlbertForMaskedLM

tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
model = TFAlbertForMaskedLM.from_pretrained('albert-base-v2')
input_ids = tf.constant(tokenizer.encode("This is a test!"))[
    None, :]  # Batch size 1
outputs = model(input_ids)
prediction_scores = outputs[0]
outputTokens = tf.math.argmax(prediction_scores, axis=2)
outputTokens = tf.keras.backend.eval(outputTokens[0])
outputTokens = tokenizer.decode(outputTokens)
print(outputTokens)

```
The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Copy the above code
2. See output in terminal
```
time this is a test! your
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Output
`This is a test!`
## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: How to install transformers-cli?
- Platform:  MacOS 10.15.2
- Python version: 3.6.6
- PyTorch version (GPU?): 1.4.0 CPU
- Tensorflow version (GPU?): 2.1.0 CPU
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
When you encode with `tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')` a [CLS] token is added in the beginning and a [SEP] token is added at the end. You can verify this by: 

```python 
from transformers import AlbertTokenizer, TFAlbertForMaskedLM

tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
tokenizer.decode(tokenizer.encode("This is a test!"))  # gives '[CLS] this is a test![SEP]'
```

Now the encoded string has two added tokens, one in the begging, one in the end. This means that the two new tokens also produce two logits whose argmax token that in your case happened to be `time` and `your`. If you don`t want to add [CLS] and [SEP] when encoding, use:

```python 
from transformers import AlbertTokenizer, TFAlbertForMaskedLM

tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
tokenizer.decode(tokenizer.encode("This is a test!", add_special_tokens=False))  # gives 'this is a test!'
```



So they can't output special tokens properly.

I mean they can, but the model does not have to output "[CLS]" when you feed "[CLS]" in the model.
Albert on QQP inference
Author: search4mahesh
Pull request: no
Labels: name: wontfix

While using Albert model trained on QQP data, i am using following code for inference.
How to manage two sentences and two labels (0,1) like QQP?


ffrom transformers import AlbertTokenizer, AlbertForSequenceClassification
import torch

tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
model = AlbertForSequenceClassification.from_pretrained('albert-base-v2')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=labels)
loss, logits = outputs[:2]

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[Docs] Update usage doc regarding generate fn
Author: patrickvonplaten
Pull request: yes

Update `model.generate()` docs

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3504?src=pr&el=h1) Report
> Merging [#3504](https://codecov.io/gh/huggingface/transformers/pull/3504?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/601ac5b1dc1438f00d09696588f2deb0f045ae3b&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3504/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3504?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3504      +/-   ##
==========================================
- Coverage   77.79%   77.76%   -0.03%     
==========================================
  Files         100      100              
  Lines       17051    17051              
==========================================
- Hits        13265    13260       -5     
- Misses       3786     3791       +5     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3504?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3504/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `90.65% <0.00%> (-0.84%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3504/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.32% <0.00%> (+0.17%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3504?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3504?src=pr&el=footer). Last update [601ac5b...5779256](https://codecov.io/gh/huggingface/transformers/pull/3504?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Update README.md
Author: mrm8488
Pull request: yes
Labels: name: model card

- Show that the last uploaded version was trained on more data (custom_license files)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3552?src=pr&el=h1) Report
> Merging [#3552](https://codecov.io/gh/huggingface/transformers/pull/3552?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/83d1fbcff608f84a27234e20d6531b4404dc059e&el=desc) will **decrease** coverage by `0.49%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3552/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3552?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3552      +/-   ##
==========================================
- Coverage   78.31%   77.81%   -0.50%     
==========================================
  Files         100      100              
  Lines       17064    17064              
==========================================
- Hits        13363    13278      -85     
- Misses       3701     3786      +85     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3552?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/3552/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `28.89% <0.00%> (-27.60%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3552?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3552?src=pr&el=footer). Last update [83d1fbc...cd4f658](https://codecov.io/gh/huggingface/transformers/pull/3552?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).