noamaanMulla-03/transformers-issues huggingface dataset:
Gradient accumulation
Author: okanlv
Pull request: no
Labels: name: wontfix

Shouldn't we include `len(train_dataloader)` along with `step` here considering `len(train_dataloader)` might be a odd number? In that case, we could accumulate the gradients more times than `gradient_accumulation_steps`.
 
https://github.com/huggingface/transformers/blob/0412f3d9298cdb8ba7f69570753ec6a07d240c87/examples/run_squad.py#L231

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Support longer sequences with BertForSequenceClassification
Author: eladbitton
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I am using `BertForSequenceClassification` for solving a regression task. I have a long sequence as an input and the model outputs a float in range [0,1].

Most of my sequences are longer than 512, which is the max sequence length in the current bert pretrained models.

To handle longer sequence you need to split the input with some stride value, As suggested here:
https://github.com/google-research/bert/issues/27#issuecomment-435265194

It seems that it was implemented in the `BertForQuestionAnswering` and the `Squad` example but not in the `BertForSequenceClassification` which i use.
But still, I do not understand how that would really work. I do not understand the use of `start_positions` and `end_positions` enough to implement it on my own.

Given my regression task how do i handle the output of the model for each chunk of my input and get unified output of the model?




Comments:
A little question because we are trying to organize the issues better:
- what made you not use the issue templates we have added?

> A little question because we are trying to organize the issues better:
> 
> * what made you not use the issue templates we have added?

Didn't know about it...

@eladbitton Can you link lines of code to where this was done for `BertForQuestionAnswering` and `SQuAD`? I'd be willing to take a stab at implementation.

Hey @maxzzze. I was looking at:
https://github.com/huggingface/pytorch-transformers/blob/0d1dad6d5323cf627cb8d7ddd428856ab8475f6b/pytorch_transformers/modeling_bert.py#L1112

Now that i look at it, i am not sure if they implemented it there.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Same issue here.
Since this package is presented as a "plug n play" solution for this kind of task, that is really unfortunate.

@eladbitton, I believe the start and end positions in BertForQuestionAnswering are for filtering tokens when computing the loss (since the loss is given by the cross-entropy between the predicted and true distributions of the start token, the latter of which is a one-hot vector; similarly for the end token), not for converting a large sequence into a batch of shorter sequences.

@thomwolf, are there plans to add the functionality mentioned [here](https://github.com/google-research/bert/issues/27#issuecomment-435265194) by Devlin (or would you be able to suggest any alternatives that might work)?
How pipeline can use a  ner finetuned model from a local directory ?
Author: lecidhugo
Pull request: no
Labels: name: wontfix

name: Core: Pipeline

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, 
I fine tuned XLM-Roberta on my own NER dataset so I got a folder containing pytorch_model.bin and all the other stuff.
But the problem is that I do not figure out how to use this model with the pipeline.
Below is an example of how I used it and the generated error:
Usage:
nlp = pipeline('ner',model= XLMRobertaForTokenClassification.from_pretrained('./1-out/checkpoint-24924/'))
Error:
OSError: Model name './1-out/checkpoint-24924/' was not found in model name list (xlm-roberta-base, ...)

Instead of XLMRobertaForTokenClassification I tried AutoModel and PreTrainedModel but I still get the same error. I also added tokenizer=AutoTokenizer.from_pretrained, etc but with no luck.

Any help is appreciated!
Thank you!

Comments:
Hi,
Just an update on this issue. I managed to get it work like this:
`model = XLMRobertaForTokenClassification.from_pretrained('./2-out/')`
`tokenizer = XLMRobertaTokenizer.from_pretrained('./2-out/')`
`nlp = pipeline('ner',model= model,tokenizer=tokenizer)`
`nlp('blabla').`

The problem is that the output gives labels for individual tokens and not for complete words. This issue was mentioned also [here](https://github.com/huggingface/transformers/issues/2488).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Added scibert-nli model card
Author: gsarti
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3376?src=pr&el=h1) Report
> Merging [#3376](https://codecov.io/gh/huggingface/transformers/pull/3376?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/cf72479bf11bf7fbc499a518896dfd3cafdd0b21&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3376/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3376?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3376   +/-   ##
=======================================
  Coverage   77.55%   77.56%           
=======================================
  Files         100      100           
  Lines       16970    16970           
=======================================
+ Hits        13161    13162    +1     
+ Misses       3809     3808    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3376?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3376/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `84.09% <0.00%> (+0.17%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3376?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3376?src=pr&el=footer). Last update [cf72479...0c19c77](https://codecov.io/gh/huggingface/transformers/pull/3376?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
AlbertDoublehHeadsModel
Author: hasnain2808
Pull request: no
Labels: name: wontfix

# 🌟New model addition

## Model description
Like we have OpenAIGPTDoubleHeadsModel. I actually want to know if someone is already working on similar model for Albert
If not that with some help I would want to contribute towards it
<!-- Important information -->

## Open Source status

* [ ] the model implementation is available: (give details)
* [ ] the model weights are available: (give details)
* [ ] who are the authors: (mention them)

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to load tf1 BERT checkpoints and sentencepiece model from local folder?
Author: sharavsambuu
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
We have BERT checkpoints trained for [Mongolian](https://github.com/tugstugi/mongolian-bert) language and planning to upload it to transformers library.

In order to do that, we have to check compatibility. I have following questions.

- How to load sentencepiece model from local folder?
- How to load TF1 checkpoints from local folder?

Thanks.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Create card for the model
Author: mrm8488
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3391?src=pr&el=h1) Report
> Merging [#3391](https://codecov.io/gh/huggingface/transformers/pull/3391?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/cf72479bf11bf7fbc499a518896dfd3cafdd0b21&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3391/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3391?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3391      +/-   ##
==========================================
+ Coverage   77.55%   77.56%   +0.01%     
==========================================
  Files         100      100              
  Lines       16970    16970              
==========================================
+ Hits        13161    13163       +2     
+ Misses       3809     3807       -2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3391?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3391/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.72% <0.00%> (+0.13%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3391/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `84.09% <0.00%> (+0.17%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3391?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3391?src=pr&el=footer). Last update [cf72479...c941e45](https://codecov.io/gh/huggingface/transformers/pull/3391?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
More details about DistilBERT experiment setting.
Author: silencio94
Pull request: no

NIPS workshop paper (http://arxiv.org/abs/1910.01108) does not provide loss weights and other distillation hyper-paramerters (temperature, learning rate, epoch, step... ). and also (https://medium.com/huggingface/distilbert-8cf3380435b5). So when I try to experiment with this scripts(/transformers/examples/distillation/), it's hard to set hyper parameters. 

Actually, I tried to train my model with distilation/ scrpts (loss weights are set equally 0.33), and got undesirable result. :sob: I think it would be helpful for others to explicitly write experimental settings of DistilBERT in the readme file. Have a good day.

Comments:
Did you take a look at the [distillation README](https://github.com/huggingface/transformers/tree/master/examples/distillation)? It shows the command that was used to train the distilled model.

Hey @silencio94, if you need help with distillation, drop me an email at clement [at] huggingface [dot] co.

Thanks for replying. I'm planing to do some experiments on my DistilBERT after distillation. after Then I'll send an email or leave another issue. thanks!
Fix output name
Author: glicerico
Pull request: yes
Labels: name: wontfix

Output variable name `all_hidden_states` found in README is inconsistent with documentation's `hidden_states`: https://huggingface.co/transformers/model_doc/bert.html#bertmodel

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2555?src=pr&el=h1) Report
> Merging [#2555](https://codecov.io/gh/huggingface/transformers/pull/2555?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/6e2c28a14a3d171e8c4d3838429abb1d69456df5?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2555/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2555?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2555   +/-   ##
=======================================
  Coverage   74.66%   74.66%           
=======================================
  Files          87       87           
  Lines       14802    14802           
=======================================
  Hits        11052    11052           
  Misses       3750     3750
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2555?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2555?src=pr&el=footer). Last update [6e2c28a...e268f1c](https://codecov.io/gh/huggingface/transformers/pull/2555?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
squad convert example to features potential bug
Author: phuongpm241
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I used an older version of run_squad.py (and everything else in the example). My dataset contains very long documents (1000-2000 tokens). In the past, convert_example_to_features returns about 913 features per 12 examples. However, after a pull I did last night, the number of features is now 14. Both f1 and exact match drops tremendously because of that. I wonder if there were any changes in the pull I did.

Comments:
The new and old versions of SQuAD should behave exactly the same when building features. Do you think you could provide an example script that replicates this issue, so that I may take a look at it?

Possibly related to #2548 

I was trying to run `run_squad.py` using this script
```
CUDA_VISIBLE_DEVICES=1 python run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-cased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file ../clicr_train_squadstyle.1.0.json \
  --predict_file ../clicr_dev_squadstyle.1.0.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir ../squad_results/
```

`max_seq_length` was set to be larger than `doc_stride`.

It might be my misunderstanding, but the old version of `examples` was caching a lot of checkpoints and log of the number of iterations over 913 while the new version has 14 iterations per epoch.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Using T5
Author: ahmedbahaaeldin
Pull request: no
Labels: name: wontfix

How can i use T5 model like in the paper , input to the model "Machine Translation #Some Text#" and it outputs its translation ?


Comments:
I am also looking for inference example. I tried using GPT-2 style inference but it does not work at all

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Downloading mlm-17-1280 community model
Author: patrickvonplaten
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): mlm-17-1280

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

```
from transformers import AutoConfig
conf = AutoConfig.from_pretrained('mlm-17-1280')
```

## Expected behavior

The config should be loaded correctly.
All files exist are seem to be correct. There seems to be a problem with the etag.
When debugging, the call jumps into this statement:
https://github.com/huggingface/transformers/blob/8becb732931bbab5dd75cca5f5e7c75b2516d10b/src/transformers/file_utils.py#L449

and never manages to store the `config.json` file. Not sure what's going on here.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.5.1
- Platform: Linux-5.3.0-40-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0+cpu (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No



Comments:
@julien-c 

This was a bogus model file, rm'ed it.
fix sklearn release circle ci [temporary]
Author: patrickvonplaten
Pull request: yes

new sklearn release (https://github.com/scikit-learn/scikit-learn/releases) seems to be broken or leads to errors for PR since this morning - go back to previous version for now to avoid circle ci errors

Comments:
@julien-c this should fix the sklearn problem for the moment

👍 


reverted on master as they pushed a fixed release right after that one.
update the config.is_decoder=True before initialize the decoder
Author: zlinao
Pull request: yes
Labels: name: wontfix

Currently the PreTrainedEncoderDecoder class fails to initialize the "cross-attention layer" since it updates decoder.config.is_decoder = True after decoder initialization.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2435?src=pr&el=h1) Report
> Merging [#2435](https://codecov.io/gh/huggingface/transformers/pull/2435?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9261c7f771fccfa2a2cb78ae544adef2f6eb402b?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `25%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2435/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2435?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2435      +/-   ##
==========================================
- Coverage   73.24%   73.24%   -0.01%     
==========================================
  Files          87       87              
  Lines       15001    15004       +3     
==========================================
+ Hits        10988    10989       +1     
- Misses       4013     4015       +2
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2435?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_encoder\_decoder.py](https://codecov.io/gh/huggingface/transformers/pull/2435/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19lbmNvZGVyX2RlY29kZXIucHk=) | `25.58% <25%> (+0.28%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2435?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2435?src=pr&el=footer). Last update [9261c7f...b4418d3](https://codecov.io/gh/huggingface/transformers/pull/2435?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Indeed, the cross attention is initialized in `BertLayer` and needs knowledge of the `is_decoder` boolean to ensure it is correctly initialized.

Looks good to me, thanks @zlinao 

> Indeed, the cross attention is initialized in `BertLayer` and needs knowledge of the `is_decoder` boolean to ensure it is correctly initialized.
> 
> Looks good to me, thanks @zlinao

Yes, exactly.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[Bart]example---BartForConditionalGeneration
Author: qiunlp
Pull request: no

**when I run your example:**
from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig
model = BartForConditionalGeneration.from_pretrained('bart-large-cnn')
tokenizer = BartTokenizer.from_pretrained('bart-large-cnn')
ARTICLE_TO_SUMMARIZE = "My friends are cool but they eat too many carbs."
inputs = tokenizer.batch_encode_plus([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors='pt')
print(inputs)
summary_ids = model.generate(inputs['input_ids'], attention_mask=inputs['attention_mask'], num_beams=4, max_length=5)
print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids])

**model :**
https://s3.amazonaws.com/models.huggingface.co/bert/facebook/bart-large-cnn/pytorch_model.bin

**the results:**
{'input_ids': tensor([[    0,  1308,   964,    32,  3035,    53,    51,  3529,   350,   171,
         33237,     4,     2]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}
Traceback (most recent call last):
  File "/home/qwh/桌面/OpenNMT/bart.py", line 17, in <module>
    summary_ids = model.generate(inputs['input_ids'], attention_mask=inputs['attention_mask'], num_beams=4, max_length=5)
  File "/home/qwh/.local/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
    return func(*args, **kwargs)
**TypeError: generate() got an unexpected keyword argument 'attention_mask'**

thank you!

Comments: