noamaanMulla-03/transformers-issues huggingface dataset:
TF Flaubert w/ pre-norm
Author: LysandreJik
Pull request: yes

#5614 fixed TF Flaubert without pre-norm, but didn't fix it with the pre-norm.

Fixes #6084


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6841?src=pr&el=h1) Report
> Merging [#6841](https://codecov.io/gh/huggingface/transformers/pull/6841?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/05c3214153d30245928279724ce2a9b701ec8aab?el=desc) will **decrease** coverage by `0.10%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6841/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6841?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6841      +/-   ##
==========================================
- Coverage   80.27%   80.16%   -0.11%     
==========================================
  Files         157      157              
  Lines       28586    28586              
==========================================
- Hits        22946    22916      -30     
- Misses       5640     5670      +30     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6841?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_flaubert.py](https://codecov.io/gh/huggingface/transformers/pull/6841/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9mbGF1YmVydC5weQ==) | `88.34% <ø> (ø)` | |
| [src/transformers/tokenization\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/6841/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fY3RybC5weQ==) | `78.64% <0.00%> (-17.48%)` | :arrow_down: |
| [src/transformers/tokenization\_xlm\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/6841/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxtX3JvYmVydGEucHk=) | `84.52% <0.00%> (-10.72%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6841/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `85.21% <0.00%> (-1.01%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6841/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `82.66% <0.00%> (+0.25%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6841?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6841?src=pr&el=footer). Last update [05c3214...f863f8e](https://codecov.io/gh/huggingface/transformers/pull/6841?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
pegasus-large: Can we have input text descriptions more than the maximum input length of 512?
Author: shamanez
Pull request: no

As mentioned pegasus paper, authors used sinusoidal positional encoding to make sure the model can work with inputs longer than default 512? 

So with huggingface implementation can we use inputs with longer lengths? 

Comments:
Yes ! https://discuss.huggingface.co/t/pegasus-questions/838/8?u=valhalla

Thanks
Finetune T5 on other Dataset, AssertionError: assert tokenized.input_ids.shape[1] == max_length
Author: ShoubhikBanerjee
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

# ❓ Questions & Help
## Details
I was trying to finetune Amazon Food Review Dataset on T5 using **Latest code from master**.

I formatted the data, as .source and .target. 

When running [finetune_t5.sh](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/finetune.py),

First it gives error :  **Keyword arguments {'add_prefix_space': True} not recognized.**

And then  : `File "/content/transformers/examples/summarization/utils.py", line 51, in encode_file
    assert tokenized.input_ids.shape[1] == max_length
AssertionError`

Could you please say, whether it is a bug or I am doing something wrong?


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Resuming training from a checkpoint on Windows does not resume at the correct global_step
Author: jncasey
Pull request: no

### Who can help
Trainer: @sgugger


## Information

I've noticed that resuming training from a checkpoint on Windows fails to start at the right global step (though learning rate, loss, etc continue from where the training left off). The same doesn't happen under macOS.

## Solution?

I think the issue is in Trainer.train, specifically this line:
`self.global_step = int(model_path.split("-")[-1].split("/")[0])`

And it seems to work as expected by swapping `"/"` with `os.sep`



Comments:
Seems wrong indeed. Would you mind fixing with a PR?

I'm embarrassed to say I'm not super familiar with git or open source collaboration in general, so even though it seems super trivial, I'm worried I'll mess something up. 

I can fix this but it'll be next week as I'm off tonight and tying a few other loose ends.

Thank you! I'm in no personal rush – I got my aborted training to resume. I just wanted someone on the project to know about the problem/solution I found.
How to get multiple answers from the context using BertForQuestionAnswering
Author: MaheshChandrra
Pull request: no
Labels: name: wontfix

How do I get multiple answers from the text using **BertForQuestionAnswering**, just like for the below question there are two possible answers:

1. a nice puppet
2. a software engineer

**Below is the code snippet for the same:**

```
from transformers import BertTokenizer, BertForQuestionAnswering
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')

question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet.Jim Henson was a software engineer."
input_ids = tokenizer.encode(question, text)
token_type_ids = [0 if i <= input_ids.index(102) else 1 for i in range(len(input_ids))]
start_scores, end_scores = model(torch.tensor([input_ids]), token_type_ids=torch.tensor([token_type_ids]))

all_tokens = tokenizer.convert_ids_to_tokens(input_ids)
answer = ' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1])
print(answer)

'a software engineer' 
```

Thanks in advance!!

Comments:
this might help https://github.com/google-research/bert/issues/657

Thanks @chutaklee  for the quick response, but may I please know whether we can do the same with the existing pretrained BERT model,by changing any parameters anywhere,as I currently do have question-answer pairs and I'm not not training the model, so just wanted to use pre trained to get the answers.

Thanks in advance!!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@MaheshChandrra Going through the same problem. Did you find any solution ?

@subigyaup ,No luck yet!! Will do drop the fix if I find anything.
2 Slow Test Failures That Sam Can Fix
Author: sshleifer
Pull request: no
Labels: name: Tests

Both related to fp16

1)
```
        if torch.cuda.is_available():
            testargs += ["--fp16", "--gpus=1"]
    
        with patch.object(sys, "argv", testargs):
>           result = run_pl_glue.main()
```


2)
```
examples/seq2seq/test_bash_script.py::test_train_mbart_cc25_enro_script
```
needs rerun/ cannot use fp16

Comments:
🐛 [BART] Pipeline OOM 
Author: astariul
Pull request: no
Labels: name: wontfix

# 🐛 Bug

I try to run BART model myself versus running the model through `pipeline`.

Running the BART model myself is fine, but I have OOM on my GPU if I try to run the same model through pipeline.

Please see the following code : https://gist.github.com/Colanim/4fae6ab52c05716062a0f20c4a6b9737

_(It assume you have a file `cnndm/test.source` with an article on each line)_

Run with : 
`python pipeline_oom.py --model HuggingFace --batch-size 32`  
(Should **not** produce OOM on 11G-GPU)

and `python pipeline_oom.py --model Pipeline --batch-size 32`  
(Should produce OOM on 11G-GPU)

---

**Why the pipeline use more memory ?**

@sshleifer 

Comments:
@sshleifer Can you reproduce on your side or is it just me ?

Yes I can replicate, sorry for the slow response. I am still trying to figure out why this is happening.

OK I figured out the problem
Long articles are not getting truncated anymore by pipeline.
Will have a look.
If you look at the second val.source example it's 1583 tokens, and pipeline does not truncated it, whereas `Huggingface` does.

Related: #4236 

May be related #5398

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Bart can make decoder_input_ids from labels
Author: sshleifer
Pull request: yes

mimics a nice t5 feature

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6758?src=pr&el=h1) Report
> Merging [#6758](https://codecov.io/gh/huggingface/transformers/pull/6758?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a75c64d80c76c3dc71f735d9197a4a601847e0cd?el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6758/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6758?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6758      +/-   ##
==========================================
- Coverage   78.96%   78.96%   -0.01%     
==========================================
  Files         157      157              
  Lines       28486    28488       +2     
==========================================
+ Hits        22495    22496       +1     
- Misses       5991     5992       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6758?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_bart.py](https://codecov.io/gh/huggingface/transformers/pull/6758/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iYXJ0LnB5) | `95.57% <100.00%> (+0.01%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6758/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `82.41% <0.00%> (-0.26%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6758?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6758?src=pr&el=footer). Last update [a75c64d...06e7cc5](https://codecov.io/gh/huggingface/transformers/pull/6758?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
MarianMTModel.generate error: Segmentation fault (core dumped)
Author: sdhzlxm
Pull request: no

When I use MarianMTModel to generate sequences, that is running "translated = model.generate(**tokenizer.prepare_seq2seq_batch(src_text))", the python development environment will quit automatically. There is a message, i. e. Segmentation fault (core dumped). 
When I run the program in Jupyter, it will also quit. Leave the message like this, "Kernel restarting The kernel appears to have died. It will restart automatically"

Comments:
I run the same demo program on another server. The program can work properly.
num_beams error in GPT2DoubleHead model
Author: vibhavagarwal5
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.9.1
- Platform: Linux
- Python version: 3.6
- PyTorch version (GPU?): 1.5
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help
@LysandreJik @patil-suraj 
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 tensorflow: @jplu 
documentation: @sgugger
 -->

## Information

I am trying to use `model.generate()` for the GPT2DoubleHeadModel but the beam search is giving an error.
Setting the `num_beams > 1` results in the following error:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/transformers/modeling_utils.py", line 1125, in generate
    model_specific_kwargs=model_specific_kwargs,
  File "/home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/transformers/modeling_utils.py", line 1481, in _generate_beam_search
    past = self._reorder_cache(past, beam_idx)
  File "/home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/transformers/modeling_utils.py", line 1551, in _reorder_cache
    return tuple(layer_past.index_select(1, beam_idx) for layer_past in past)
  File "/home/hdd1/vibhav/anaconda3/envs/vesnli/lib/python3.7/site-packages/transformers/modeling_utils.py", line 1551, in <genexpr>
    return tuple(layer_past.index_select(1, beam_idx) for layer_past in past)
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
```

However, things are working fine for `num_beams=1` and for GPT2LMHeadModel(both beam search and non beam search)


Comments:
encountered the same issue

I think @patrickvonplaten  might have some ideas.
Model cards for loodos
Author: hknor
Pull request: yes
Labels: name: model card

Adds cards for 6 models in the ./models/loodos directory.

albert-base-turkish-uncased
bert-base-turkish-uncased
electra-base-turkish-uncased
electra-base-turkish-64k-uncased-discriminator
electra-small-turkish-cased-discriminator
electra-small-turkish-uncased-discriminator

Comments:
[t5] Missing requirements in examples/seq2seq
Author: jeff-da
Pull request: no

Hi! This is a very small fix. But it seems like some requirements for examples/seq2seq are missing? Namely, rouge-score, gitpython, sacrebleu. Is this intentional (conflicts with other example requirements)? Of course, would be happy to open a PR

## Environment info

- `transformers` version: 8b884dadc6bd70600c98bb35b522beb0005e3f28
- Platform: Linux
- Python version: 3.8
- PyTorch version (GPU?): GPU, '1.6.0+cu101'
- Tensorflow version (GPU?): n/a
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

 Summarization: @sshleifer

## Information

Model I am using (Bert, XLNet ...): T5

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name) CNN Summary
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Create new `conda` env
2. Install requirements.txt in /examples
3. Try running finetune_t5.sh

## Expected behavior

Finetune should run correctly.

Comments:
Those requirements are all in [here](https://github.com/huggingface/transformers/blob/master/examples/requirements.txt). Are you sure you ran `pip install -r ./examples/requirements.txt` as mentioned in the [README of all examples](https://github.com/huggingface/transformers/tree/master/examples#important-note)?

They are not, and won't be requirements of the main library since they are only used for some specific tasks.

Huh, probably just a local issue then. Thanks!
How to use the reformer for question answering?
Author: Krak91
Pull request: no

In the code example in the documentation the tokenizer gets passed just a single string. Doesn't it need a context and a question? Also, how exactly to get the answer?

Comments:
Hey @Krak91,

Thanks for this issue. The PR attached to this issue should fix it. 
Note that even though the new example code will work for Reformer, it won't yield any good results because there is no pretrained ReformerModel yet.
Printing probabilities
Author: Mahmedturk
Pull request: no

Hi,

I apologize if that's a stupid question. How can I print probabilities during inference produced by the softmax layer with BertForSequenceClassification?

Comments:
Hi, you have an example of how to do exactly this in the [documentation](https://huggingface.co/transformers/task_summary.html#sequence-classification):

```py
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased-finetuned-mrpc")
model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased-finetuned-mrpc")

classes = ["not paraphrase", "is paraphrase"]
sequence_0 = "The company HuggingFace is based in New York City"
sequence_1 = "Apples are especially bad for your health"
sequence_2 = "HuggingFace's headquarters are situated in Manhattan"

paraphrase = tokenizer(sequence_0, sequence_2, return_tensors="pt")
not_paraphrase = tokenizer(sequence_0, sequence_1, return_tensors="pt")

paraphrase_classification_logits = model(**paraphrase).logits
not_paraphrase_classification_logits = model(**not_paraphrase).logits

paraphrase_results = torch.softmax(paraphrase_classification_logits, dim=1).tolist()[0]
not_paraphrase_results = torch.softmax(not_paraphrase_classification_logits, dim=1).tolist()[0]

# Should be paraphrase
for i in range(len(classes)):
    print(f"{classes[i]}: {int(round(paraphrase_results[i] * 100))}%")

# Should not be paraphrase
for i in range(len(classes)):
    print(f"{classes[i]}: {int(round(not_paraphrase_results[i] * 100))}%")
```
Example config code uses invalid 'output_attention' rather than 'output_attentions'
Author: lannelin
Pull request: no

Looks like documentation only bug. `output_attention` used rather than `output_attentions`.

Occurs in multiple places in repo.

Maybe linked to #2985 


## Environment info

- `transformers` version: 3.1.0
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.4
- PyTorch version (GPU?): 1.5.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help
 documentation: @sgugger 

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

From docs L4-L5 of AutoConfig example: 
[docs](https://huggingface.co/transformers/model_doc/auto.html#transformers.AutoConfig)
or 
[code](https://github.com/huggingface/transformers/blob/master/src/transformers/configuration_auto.py#L264)

```python
config = AutoConfig.from_pretrained('bert-base-uncased', output_attention=True, foo=False)
assert config.output_attention == True
```

causes:

```
AttributeError: 'BertConfig' object has no attribute 'output_attention'
```


## Expected behavior

The assertion given in documentation passes.



Comments:
Thanks for reporting! The PR mentioned above should fix all of those.
Greedy decoding for non-beam-search appears to ignore postprocessing
Author: alexeyr
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2
- Platform: Linux-4.4.0-18362-Microsoft-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

TextGeneration: @TevenLeScao 

## Information

Model I am using (Bert, XLNet ...): Bart

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

I was experimenting with `generate` method to understand behavior on small cases.

## To reproduce

See https://github.com/huggingface/transformers/blob/8f2723caf0f1bf7e1f639d28d004f81c96d19bbc/src/transformers/generation_utils.py#L535-L566

The last line should probably take `argmax` of post-processed `scores` instead of `next_token_logits`. This should manifest in not respecting minimum length, generating bad words and repeats.

On a more minor note, are `next_token_logscores` really _log_ scores?

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Didn't realize that `postprocess_next_token_scores` mutates its argument.