noamaanMulla-03/transformers-issues huggingface dataset:
Documentation on how to get results out of trainer is missing.
Author: PhilipMay
Pull request: no
Labels: name: wontfix

Hi,
some time ago it was possible to get the results out of the trainer by `trainer.log_history`.
This now changed to `trainer.state.log_history`. But everything is not documented. I suggest to add documentation on how to get results out of the trainer.

Comments:
Hello! Indeed, do you want to open a PR to fix this? Thanks!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Description of how to preprocess text corpus for roBERTa LM training
Author: PhilipMay
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

There are different locations where you desribe how to train roBERTa models. For example here:
- https://huggingface.co/blog/how-to-train
- https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb
- https://gist.github.com/aditya-malte/2d4f896f471be9c38eb4d723a710768b

But nobody sais how to preprocess the text corpus. I think it must be one sentence per row. But does it need empty lines between documents? Is it ok to shuffle the text line by line?

Could you please claify this?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Well - this issue is still open IMO...

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Maybe better to ask to the original RoBERTa authors or on the forum at https://discuss.huggingface.co?

We are trying to keep the issues for bug/features reports now.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
sentencepiece 0.1.94 causing segmentation fault
Author: mejran
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.4.0 and 3.3.1
- Platform: Linux/Sagemaker
- Python version: 3.7
- PyTorch version (GPU?): 1.4
- Tensorflow version (GPU?): 
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

## Information

## To reproduce

Steps to reproduce the behavior:

1. `pip install transformers[torch]`
2. `from transformers.trainer import TrainingArguments, Trainer`
`import torch`
3. `torch.tensor([1,2,3])`

`transformers` 3.3.1 seg faults at step 3, `transformers` 3.4 seg faults at step 2.

## Expected behavior

No segmentation fault

Comments:
This can be worked around, for anyone hitting this issue, by setting `sentencepiece==0.1.91` explicitly.

Maybe we could set `sentencepiece==0.1.91` in the setup.py to prevent this from happening, as we already had the issue with the 0.1.92.

Do you want to open a PR for that?

This should also be fixed by #8073 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Update Conda Release
Author: lukasfolle
Pull request: no
Labels: name: wontfix

When trying to get started with transformers using some examples from the model cards, the installation with conda results in an outdated version (2.1.1).

As a result, the example cannot run. However, when using pip to install transformers the newest version (2.11) is correctly used.

I would suggest updating the conda release in order to avoid installing via pip.

Comments:
This might provide more information: https://github.com/conda-forge/transformers-feedstock/issues/3

In short, the conda-forge update is blocked by `sentencepiece` not being available through conda.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


The feedstock has now been updated for Linux and is available. Other OSes are still waiting on sentencepiece.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This should be fixed by #8073 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Poor f1 score when validating existing models
Author: omrishsu
Pull request: no
Labels: name: wontfix

I have been attempting to validate the results on several models, but have been unable to match them with the results posted on the model cards.

For example, I took an Albert model: https://huggingface.co/ahotrod/albert_xxlargev1_squad2_512
and I run:
```
python run_squad.py 
--model_type albert 
--model_name_or_path ahotrod/albert_xxlargev1_squad2_512 
--do_eval 
--predict_file ../squd/dev-v2.0.json 
--per_gpu_eval_batch_size 8  
--max_seq_length 512 
--doc_stride 128 
--output_dir ../squd/output/albert 
--overwrite_output_dir 
--threads 16 
--verbose 
--version_2_with_negative
```

I got: 
```
exact: 77.39408742525058
f1: 81.6576936707378
total: 11873
HasAns_exact: 71.60931174089069
HasAns_f1': 80.14875117285251
HasAns_total': 5928
NoAns_exact': 83.16232127838519
NoAns_f1': 83.16232127838519
NoAns_total': 5945
best_exact': 77.38566495409754
best_exact_thresh': 0.0
best_f1': 81.64927119958456
best_f1_thresh': 0.0
```

While on the model card stated:
```
exact: 86.11134506864315
f1: 89.35371214945009
total': 11873
HasAns_exact': 83.56950067476383
HasAns_f1': 90.06353312254078
HasAns_total': 5928
NoAns_exact': 88.64592094196804
NoAns_f1': 88.64592094196804
NoAns_total': 5945
best_exact': 86.11134506864315
best_exact_thresh': 0.0
best_f1': 89.35371214944985
best_f1_thresh': 0.0
```

I did not alter any code, I was simply trying to validate the results. What am I overlooking here?

(the same diff also found using roberta models, bert, and others)

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Setting os.environ['CUDA_VISIBLE_DEVICES'] = ‘1’, but always training on GPU0, how to set it(GPT2)?
Author: TheoRenLi
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
Hi @TheoRenLi, this question should go on https://discuss.huggingface.co

We keep the issues of the repo for bug and features request (with clear descriptions).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Example for running T5 for translation 
Author: rabeehk
Pull request: no
Labels: name: wontfix

Hi,
I am having hard time finetuning T5-small on WMT-14 de/en and bleu score does go high. I followed the notebooks of question answering with T5, is there any specific point to consider for translation, so any specific parameter which needs to be given to model.generate ? 
could you assist me with showing me some example codes you have made it work for translation task?
thanks

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Sentence transformer Segmentation Fault - Pytorch 1.4.0, 2.80
Author: elangovana
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: Linux
- Python version:  3.7
- PyTorch version (GPU?):  1.4 (Both)
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Doesn't Matter

### Who can help
@LysandreJik 
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert):

The problem arises when using:
* [ ] the official example scripts: (give details below)


## To reproduce

Steps to reproduce the behavior:

1. Open python console, with torch 1.4 installed

```python
import transformers
```

This crashes with error
```text
segementation fault
```

## Expected behavior
Works as normal without error

<!-- A clear and concise description of what you would expect to happen. -->


The only way to fix this is to force install sentencepiece==0.1.91 . The root cause is that https://github.com/huggingface/transformers/blob/v2.8.0/setup.py doesn't fix the version of sentencepiece.  


Comments:
This will also be fixed by #8073 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Remove tolerance + drop_rows_to_fit by default
Author: LysandreJik
Pull request: yes

Please take a look @NielsRogge.

I'm setting `drop_rows_to_fit=True` when the user wants that truncation. I don't think that attribute really means anything anymore given the way we handle truncation in the encoding methods, so I think it can be removed altogether.

Regarding the integration tests, I finally chose to go with a per-test tolerance instead of a relative tolerance, as the TAPAS model can output very large negative numbers; for example for the `test_inference_question_answering_head_conversational` test:

```py
        expected_tensor = torch.tensor(
            [
                [
                    -9997.22461,
                    -9997.22461,
                    -9997.22461,
                    -9997.22461,
                    -9997.22461,
                    -9997.22461,
                    -9997.22461,
                    -9997.22461,
                    -9997.22461,
                    -16.2628059,
                    -10004.082,
                    15.4330549,
                    15.4330549,
                    15.4330549,
                    -9990.42,
                    -16.3270779,
                    -16.3270779,
                    -16.3270779,
                    -16.3270779,
                    -16.3270779,
                    -10004.8506,
                ]
            ],
            device=torch_device,
        )
```

I think in these cases it's helpful to see what difference we're looking at directly in the test, and I'm not sure a relative difference would handle such ranges, but I may be mistaken here.


Comments:
The tests look ok to me!

@NielsRogge removed the `drop_rows_to_fit` attribute in the last commit.
Make doc styler detect lists on rst and better support for Windows
Author: sgugger
Pull request: yes

# What does this PR do?

The new lines for the lists in rst files were not actually added because I made a mistake, this PR fixes that. @patrickvonplaten it adds some new lines in the benchmarking files which I think are okay, but let me know if I should write some special code to get the scripts to ignore them.

Also, changed the line that added the new lines before doc special words since it seems to be not working properly on Windows. Let's see if this version is better!

(Failures are because master is red at the time of this PR.)

Fixes #9438

Comments:
[TF Led] Fix flaky TF Led test
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

The reason why the TF LED test is flaky was not fully fixed in: https://github.com/huggingface/transformers/pull/9459
and is actually the following: 

Currently the `decoder_attention_mask` can have a `0` at its first input:

```python
decoder_attention_mask[:, 0] == 0
```

Since the decoder uses a causal mask, this however leads to problems as a softmax over only very large negative numbers in computed. Now since TF and PT use slightly different large numbers, we can see significant differences between the models. The solution is to make sure that the `decoder_attention_mask` used for the `tf_pt_equivalence` test cannot be zero at the first position (I've done the same changes for all TFBart models in: https://github.com/huggingface/transformers/pull/9497 and also made sure in https://github.com/huggingface/transformers/pull/9497 that the TF templates are correctly updated )



Comments:
cc @LysandreJik @sgugger @jplu 

Thanks for fixing!
[trainer] round numbers in trainer state
Author: stas00
Pull request: yes

This PR rounds very long fractions in trainer state e.g., 

```
{'loss': 14.846837043762207, 'learning_rate': 6e-06, 'epoch': 0.3333333333333333} 
```
to:
* epoch 2 decimals
* loss 4 decimals

resulting in:

```
{'loss': 14.8468, 'learning_rate': 6e-06, 'epoch': 0.33}
```` 

If you want any other small tweaks for me to add please let me know.

Fixes: #9475 

@sgugger 

Comments:
Model previews not working for models that require MecabTokenizer
Author: BramVanroy
Pull request: no
Labels: name: dependencies

As brought up [on Twitter](https://twitter.com/polm23/status/1348520920948695043) by user pol23 @polm, Japanese (and other?) models do not work at all on the model page. They will throw an error `You need to install fugashi to use MecabTokenizer.See https://pypi.org/project/fugashi/ for installation.` Perhaps the environment that these models run in should include all optional dependencies, too?

You can try it yourself by picking [a model](https://huggingface.co/daigo/bert-base-japanese-sentiment?text=I+likw+po) and trying the inference widget.

Comments:
Can we add those `pip install -e .[ja]` dependencies to the hosted Inference API @Narsil?

@julien-c I'm not sure if that is possible, but perhaps this can even be derived from the model card? If a model card specifies Japanese, then the env could include the `[ja]` option.

If I'm not mistaken all models run in the same env so it's probably not an issue to add a dependency, but I'll let @Narsil answer!

Yes I created a patch for this, should be up soon.

and up !

Confirmed it works, thanks for the quick fix!
[make docs] parallel build
Author: stas00
Pull request: yes

This PR enables multi-worker doc building.

After experimenting with different number of workers https://github.com/huggingface/transformers/issues/9496#issuecomment-758145868 4-5 workers seems to be the most optimal - let's go with 4 as surely we wouldn't find a cpu with less cores these days.

Fixes part of https://github.com/huggingface/transformers/issues/9496

@sgugger


Comments:
TF Models have no attribute .train() or .eval()
Author: jamescolless
Pull request: no
Labels: name: wontfix

## 🐛 Bug

Using any of the TF models I am unable to set the **.eval()** or **.train()** properties. In addition, when loading from a pre-trained path (which the documentation seems to imply would mean that the models will be set to eval mode) I see non deterministic outputs given the same input indicating the models do not have dropout turned off.

Basic example:

```
model = TFBert.from_pretrained('path_to_model_directory', from_pt=True)
model.eval() #### This errors with "TFBertModel object has no attribute 'eval'"
tokenizer = BertTokenizer.from_pretrained('path_to_model_directory')
inputs = tokenizer.encode_plus('Dummy text here.', return_tensors='tf')['input_ids']

print(model(inputs)) ## these outputs
print(model(inputs)) ## **will not** be the same

print(model(inputs, training=False)) ## these outputs
print(model(inputs, training=False)) ## **will** be the same
```

Any help would be greatly appreciated!

## Environment

* OS: Windows
* Python version: 3.6
* PyTorch version: 1.4
* PyTorch Transformers version (or branch): 2.3
* Using GPU ? Yes


Comments:
`model.eval()` is a PyTorch directive. It will disable dropout/norm, as you point out. On top of that, though, you'd also set the `no_grad` parameter so that weights are not updated.

Typically, your code'd look like this for inference/evaluation/testing.

```python
model.eval()
with torch.no_grad():
    # do stuff
```

I am not sure how this should be done in Tensorflow.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Is there a fix available for this because I am currently running a TFBertForSequenceClassification for classification and it gives non determinististic outcomes. I have written my code in tensorflow framework. Looks like the fix is placing model.eval() during evaluation operation as per https://github.com/google-research/bert/issues/583.
[TFBart] Split TF-Bart
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

TF mirror of: #9343

- Exact same changes as in #9343
- Docs are improved
- TFBlenderbot gets a better integration tests
- tf_saved_model & tf_serving tests are disabled for now and should ideally be fixed in https://github.com/huggingface/transformers/pull/9478 after merging this one

## After PR is merged TODO:

- [x] Open issue about `facebook/blenderbot_small-90M` tokenizer - cannot download files from hub. Weird issue


Comments:
> Awesome work!! Just left few smalll comments. I think we should first find a proper fix #9478 and then merging this one. Switching on/off some tests everytime we touch a model is really not a long term solution, I think a proper template as to be stated first and then afterwards we do the models.

IMO this PR should be merged and the s2s fix should be applied afterward as said offline. This PR is blocking a new release currently

> IMO this PR should be merged and the s2s fix should be applied afterward as said offline. This PR is blocking a new release currently

Ok, nevermind, I didn't know you wanted to have it in the next release.
Adding Stochastic Weight Averaging to transformer optimizers
Author: hasansalimkanmaz
Pull request: no

# 🚀 Feature request

I would like to train my models with SWA optimizer. According to this [paper](https://arxiv.org/pdf/1803.05407.pdf), SWA leads to better models and wider optima. 

## Motivation

As humans, we are all willing to get better results :) . I think adding this feature will lead to better models without costing more and it may be easy to implement.


Comments:
In the meantime, I have started to work on adding SWA to huggingface. After doing some experiments, If I get better results, I will create a PR. You can check my work  from [here](https://github.com/hasansalimkanmaz/transformers/tree/add-SWA-optimizer)

Any feedback will be appreciated.

Based on my custom experiments, I couldn't produce better results with SWA. So I am closing this issue. My implementation for SWA is so custom and I didn't go through all tests. So, I will not create a PR due to the lack of benefits.
ProphetNetNgramAttention: Number of attention heads
Author: guillaume-be
Pull request: no

## Information

Model I am using (Bert, XLNet ...): ProphetNet

The ProphetNet Ngram attention layer seem to refer to a wrong number of heads. The `ProphetNetNgramProphetNetSelfAttention` (seems to be a typo by the way, maybe `ProphetNetNgramSelfAttention` would be more appropriate?) is part of the decoder, and therefore I would expect it should contain a number of attention head equal to the configuration parameter `num_decoder_attention_heads `.

However, when instantiated at https://github.com/huggingface/transformers/blob/02e05fb0a532e572b56ba75dad6ba3db625bbdeb/src/transformers/models/prophetnet/modeling_prophetnet.py#L759, it uses the property `num_attention_heads` that equals to the number of **encoder** attention heads.

I assume that the correct value should be `config.num_decoder_attention_heads`. (Luckily?) no issue showed up in most models because pretrained models have the same number of encoder and decoder heads. Looking at the reference implementation, it does seem that the **decoder** number of attention heads is used for the Ngram attention (see https://github.com/microsoft/ProphetNet/blob/1d36bc5c4f334b0ed9b90fdf3a64785c174f5c45/GLGE_baselines/script/prophetnet/ngram_s2s_model.py#L586)

### Who can help

 @patrickvonplaten ? Again not sure since I do not see an owner for ProphetNet

I would be happy to submit a small PR referencing to `config.num_decoder_attention_heads` rather than this property if you agree with this change

Thanks!



Comments:
Hey @guillaume-be,

you're 100% correct about both the naming: We should remove one `ProphetNet` and also about the config. Thanks a lot for reporting this. I'll open a PR
Update 'Develop on Windows' guidelines
Author: SBrandeis
Pull request: yes

# What does this PR do?

Update the `Develop on Windows` guidelines in `CONTRIBUTING.md` to add:

- Instructions to setup git to handle CRLF line endings
- Instructions to add MSYS executables in your PATH to run `make` from another terminal

Fixes #9438

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

@sgugger
@jplu


Comments:
Do you have any idea why GitHub is not showing the diff properly?

Looks like I'm having an issue with CRLFs :( I think I replaced all CRLFs by LFs
I'm currently investigating this

@sgugger problem solved 👌