noamaanMulla-03/transformers-issues huggingface dataset:
Is Reformer supported under Encoder-Decoder framework?
Author: spookypineapple
Pull request: no
Labels: name: wontfix

Hi,
Is it possible to use Reformer with the encoder-decoder framework (i.e Reformer2Reformer)?

Comments:
It's not yet supported sadly

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
WNLI benchmark results clarification
Author: sumanthd17
Pull request: no
Labels: name: wontfix

The BERT paper mentions that the accuracy for WNLI is 62.3% [BERT Repository](https://github.com/google-research/bert) but the model card on HuggingFace reports the WNLI accuracy as  45.07%. Is there any particular for the big gap between the 2 models?

Comments:
Wondering the same! Is it a typo? Since they report the exact same number for tiny, mini, small & medium. However, one would not expect that to be higher than [bert-base-cased that recieved 45.07%](https://github.com/huggingface/transformers/tree/master/examples/text-classification#run-pytorch-version).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Bi-Directional Reformer text multi class classification
Author: Zozos972
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

I am fairly new to transformers, but good experience in ML. I have tried to start by building a RoBERTa multi class classification model but the doc and examples are not clear.

Can you point me to the set of literature that would get me going with RoBERTa?

Afterwards, I would love to support the effort building the Bi-Directional Reformer text multi class classification model. Of if you send me the docs, I can jump directly to the Reformer model.
Many thanks,
Z

Comments:
Have you taken a look at the [RobertaForSequenceClassification](https://huggingface.co/transformers/model_doc/roberta.html#tfrobertaforsequenceclassification) documentation page?

You might also be interested in the [sequence classification task](https://huggingface.co/transformers/task_summary.html#sequence-classification) documentation page.
We aim to have the exact same API for all models, so while this example showcases BERT with the auto models, you can use a RoBERTa architecture instead.


You can see the [ReformerForSequenceClassification](https://huggingface.co/transformers/model_doc/reformer.html#reformerforsequenceclassification) documentation page if you're particularly interested in Reformer.


Excellent! Thanks,
Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Can't find model to down
Author: havetry
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution
I can't find any model to down in "huggingface.cn", so what happened?
![down](https://user-images.githubusercontent.com/49902228/98759351-e202b300-240b-11eb-87ea-e2f495d3afbe.png)
but I find many people downed the model in Noverber 9, so why I can't find model today?
<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
I encountered the same problem

Will release a new version of that UX tomorrow

We just added file sizes, and download links, to the lists of model files, see for instance:

<img width="1592" alt="Screenshot 2020-11-13 at 22 55 23" src="https://user-images.githubusercontent.com/326577/99125288-be896500-25d1-11eb-84f5-03eb9b44f29d.png">

https://huggingface.co/dbmdz/bert-base-turkish-cased/tree/main

Let us know if this solves your use case @havetry @xlxwalex.

> We just added file sizes, and download links, to the lists of model files, see for instance:
> 
> <img alt="Screenshot 2020-11-13 at 22 55 23" width="1592" src="https://user-images.githubusercontent.com/326577/99125288-be896500-25d1-11eb-84f5-03eb9b44f29d.png">
> 
> https://huggingface.co/dbmdz/bert-base-turkish-cased/tree/main
> 
> Let us know if this solves your use case @havetry @xlxwalex.

Good job, thanks!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fix run_mlm_wwm example
Author: changjonathanc
Pull request: yes

# What does this PR do?

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).


Comments:
Wait, the example for `run_mlm_wwm.py` doesn't seem to run:
```
Traceback (most recent call last):
  File "run_mlm_wwm.py", line 334, in <module>
    main()
  File "run_mlm_wwm.py", line 161, in main
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
  File ".../transformers/src/transformers/hf_argparser.py", line 144, in parse_args_into_dataclasses
    raise ValueError(f"Some specified arguments are not used by the HfArgumentParser: {remaining_args}")
ValueError: Some specified arguments are not used by the HfArgumentParser: ['--dataset_name', 'wikitext', '--dataset_config_name', 'wikitext-2-raw-v1']
```

I will try to fix it too.

Thanks for the fixes. I'm unsure the script can run on datasets hosted on the hub because of the need for reference files, that's why I hadn't added the `dataset_name` and `dataset_config_name` arguments in this one.

You can still ignore the `dataset_name` arg and use the old behavior, so adding `dataset_name` should be fine.
I want to mention that I did not run the example with ref file.

Ok, then we should add a check in the post_init to make sure no reference file is passed if we're using `dataset_name`, otherwise it won't work,

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
assert tgt_line, f"empty tgt line for index {index}" with t5 
Author: rabeehkarimimahabadi
Pull request: no

Hi
I am trying to run fineutne_trainer with this command and I got assert error  thanks for your help

export TPU_IB_ADDRESS=10.160.244.2

start=`date +%s`
python xla_spawn.py --num_cores 8 \
    finetune_trainer.py \
    --tokenizer_name t5-small --model_name_or_path t5-small \
    --data_dir wmt_en_de \
    --output_dir /home/rabeeh/outputs/verify --overwrite_output_dir \
    --learning_rate=3e-4 \
    --warmup_steps 500 \
    --per_device_train_batch_size=128 --per_device_eval_batch_size=128\
    --num_train_epochs=1 \
    --save_steps 500 --eval_steps 500 \
    --logging_steps 200 \
    --max_source_length 128 --max_target_length 128 --val_max_target_length 128 --test_max_target_length 128 \
    --task translation --label_smoothing 0.1 \
     --freeze_encoder --freeze_embeds \
     --num_train_epochs=1 \
                 --logging_first_step --logging_steps 200 \
                         --do_train --do_eval --evaluate_during_training \
                             --prediction_loss_only \

    "$@"

end=`date +%s`
runtime=$((end-start))
echo running time $runtime

logs 


Exception in thread Thread-4:
Traceback (most recent call last):
  File "/root/anaconda3/envs/pytorch/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/pytorch/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/parallel_loader.py", line 141, in _loader_worker
    _, data = next(data_iter)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/root/home/rabeeh/examples/seq2seq/utils.py", line 246, in __getitem__
    assert tgt_line, f"empty tgt line for index {index}"
AssertionError: empty tgt line for index 4549523

Comments:
@patil-suraj 

I am guessing maybe this issue is making it slower on tpus? on en-ro which it does not happen it is faster

@rabeehkarimimahabadi did you solve the issue? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
FRCNN in the LXMERT demo outputs different features when using a local image vs. an image from a URL 
Author: ecekt
Pull request: no

## Environment info
- `transformers` version: 3.3.1
- Platform: Linux-5.4.0-52-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.4
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@eltoto1219

## Information
Model I am using: LXMERT, more specifically FRCNN. The problem arises when using the demo.ipynb file given here: https://github.com/huggingface/transformers/blob/master/examples/lxmert/demo.ipynb

## To reproduce

1. Change the URL to the filename of a local image. Both the visualization and the extracted features are affected by this.

## Expected behavior
I noticed that the order of Red, Green, Blue is changed in this line in the utils file: https://github.com/huggingface/transformers/blob/7abc1d96d114873d9c3c2f1bc81343fb1407cec4/examples/lxmert/utils.py#L552

When I comment out this line, the visualization and the features output by the model for a local image are the same as when we use its URL without commenting this line out. 

I would appreciate if you can confirm that commenting out that line does not cause other issues in LXMERT so that I can use local files.

Best,
Ece  


Comments:
Pinging @eltoto1219 

Hi @eltoto1219 I see that my issue and proposed solution have been mentioned in a more recent issue (#8769), should I follow there for the confirmation?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[QA examples] fix inconsistent tokenization in _improve_answer_span 
Author: xiye17
Pull request: yes

Basically, I am trying to fix the QA preprocessing steps when using BPE-based tokenizers (Roberta, Bart, Longformer)

The prior _improve_answer_span has a bug because the way it tokenizes the answer_text is different from the way context is handled. So when the original context is '1987', and the old input ans_span is ['Ġ1987', ','], then the ',' won't be removed.

To fix these cases I use two cleaned forms built by either including prefix_space or not. If a sub_answer_span matches either one of them we'll improve the answer_span. The first form [line 42] is used to handle things with leading punctuations, e.g., orig_answer_context=="73 million" and ans_span is ["Ġ$", "73", "Ġmillion"] (should be cleaned to  ["73", "Ġmillion"] ). The second form is used to handle things with ending punctuations, e.g., the '1987' example mentioned above.


Comments:
@LysandreJik Not sure who I should ping to review this :) btw I am not sure what's the issue with the code quality here. 

Hello, we're in the midst of deprecating that method and shifting the methods to `datasets` instead. Could you hold on until we have the updated approach, which should land sometimes within the next ~2 weeks?

This current method is not tested, so ensuring that a bug fix isn't creating all sorts of other bugs takes us a lot of time, so we're not going to do that here as this method will be deprecated very soon.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[wip/s2s/pl] attempt to sync metrics in DDP
Author: sshleifer
Pull request: yes

This is broken.
Attempted to add 
`AverageMetric` where you just dump python floats and they get averaged and the end, but not working on DDP. 

### Failing command
(fails quickly at val sanity check)

```bash
cd examples/seq2seq
wget https://cdn-datasets.huggingface.co/translation/wmt_en_ro.tar.gz
tar -xzvf wmt_en_ro.tar.gz
export WANDB_PROJECT=dmar
export BS=64
export m=sshleifer/mar_enro_6_3_student
export MAX_LEN=128
python finetune.py \
  --learning_rate=3e-4 \
  --do_train \
  --do_predict \
  --fp16 \
  --val_check_interval 0.25 \
  --data_dir wmt_en_ro \
  --max_source_length $MAX_LEN --max_target_length $MAX_LEN --val_max_target_length $MAX_LEN --test_max_target_length $MAX_LEN \
  --freeze_encoder --freeze_embeds \
  --train_batch_size=$BS --eval_batch_size=$BS \
  --tokenizer_name Helsinki-NLP/opus-mt-en-ro --model_name_or_path $m \
  --warmup_steps 500 --sortish_sampler --logger_name wandb \
  --gpus 2 --fp16_opt_level=O1 --task translation --num_sanity_val_steps=1 --output_dir dmar_met_test_2gpu \
  --num_train_epochs=2 --overwrite_output_dir
```


### Traceback

```bash
  File "/home/shleifer/transformers_fork/examples/seq2seq/finetune.py", line 206, in <dictcomp>
    pl_metrics = {f"pl_{prefix}_avg_{k}": v.compute().item() for k, v in self.metric_stores.items()}
  File "/home/shleifer/miniconda/lib/python3.8/site-packages/pytorch_lightning/metrics/metric.py", line 214, in wrapped_func
    self._sync_dist()
  File "/home/shleifer/miniconda/lib/python3.8/site-packages/pytorch_lightning/metrics/metric.py", line 177, in _sync_dist
    output_dict = apply_to_collection(
  File "/home/shleifer/miniconda/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py", line 53, in apply_to_collection
    return elem_type({k: apply_to_collection(v, dtype, function, *args, **kwargs)
  File "/home/shleifer/miniconda/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py", line 53, in <dictcomp>
    return elem_type({k: apply_to_collection(v, dtype, function, *args, **kwargs)
  File "/home/shleifer/miniconda/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py", line 49, in apply_to_collection
    return function(data, *args, **kwargs)
  File "/home/shleifer/miniconda/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py", line 100, in gather_all_tensors_if_available
    torch.distributed.all_gather(gathered_result, result, group)
  File "/home/shleifer/miniconda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1185, in all_gather
    work = _default_pg.allgather([tensor_list], [tensor])
RuntimeError: Tensors must be CUDA and dense

```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fixed emmental example.
Author: madlag
Pull request: yes

Added information about loading of squad data in README.
Fixed BertLayerNorm which disappeared some time ago, replace with torch.nn.LayerNorm (which was buggy long time ago it seems).


Comments:
looks good to me!

looks good to me too!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Multi-Workers distributed training
Author: Gforky
Pull request: no

Hi, does transformers support multi-workers distributed training for bert fine-tuning?

Comments:
Hi, here are a few resources to get you started:
- [Examples](https://github.com/huggingface/transformers/tree/master/examples)
- [Docs on distributed training](https://huggingface.co/transformers/examples.html#distributed-training-and-mixed-precision)

> Hi, here are a few resources to get you started:
> 
> * [Examples](https://github.com/huggingface/transformers/tree/master/examples)
> * [Docs on distributed training](https://huggingface.co/transformers/examples.html#distributed-training-and-mixed-precision)

Got it, after I get familiar with pytorch, this problem solved😀
Getting a model to work on a system with no internet access
Author: XapaJIaMnu
Pull request: no

## Environment info
We are trying to use the model from https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment on a system that has no connection to the internet. Normally one can do:
`pipeline = Pipeline(model='model_name")`
and huggingface will fetch everything it needs from the internet. Unfortunately, some datasets reside within a highly protected environment that does not allow for any internet connection and we can't fetch our models. (Hell, we can't even copy/paste errors).

Uploading files to that environment is really cumbersome and every file needs to go through a review process. Through trial and error, i have gotten the model and tokenizer to load, but it is now missing a "vocabulary". Before I got a submit a request for extra files to be uploaded, could someone just confirm for me what files do I need so that the model could be loaded offline. As far as I understand i have to have every single file from here: https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment/tree/main
and my script should look like:
`pipeline = Pipeline(model='/path/to/dir")`

### Who can help

Models:
- pipelines: @LysandreJik


## Information
https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment

The problem arises when using:
Running in a restricted environment


## To reproduce
Use the example script from https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment and try to manually input the files

## Expected behavior
The model should work just as if it were in a system connected to the internet.


Comments:
Yes, if you place all files which you find on the model page on the hub in a directory, then it will work.

You can even just `git clone` the model repo

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
PreTrainedTokenizerFast.save_pretrained() ERROR
Author: liyucheng09
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.1
- Platform: macOS-11.1-arm64-arm-64bit
- Python version: 3.9.1
- PyTorch version (GPU?): 1.8.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


## Description

It a bizarre error. When I run
```
from transformers import AutoTokenizer
t=AutoTokenizer.from_pretrained('distilroberta-base')
t.save_pretrained('vocab/')
```

It gives me the following errors:
```
>>> t.save_pretrained('vocab/')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/liyucheng/miniforge3/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2005, in save_pretrained
    return self._save_pretrained(
  File "/Users/liyucheng/miniforge3/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 528, in _save_pretrained
    vocab_files = self.save_vocabulary(save_directory, filename_prefix=filename_prefix)
  File "/Users/liyucheng/miniforge3/lib/python3.9/site-packages/transformers/models/gpt2/tokenization_gpt2_fast.py", line 172, in save_vocabulary
    files = self._tokenizer.model.save(save_directory, name=filename_prefix)
TypeError: PyModel.save() got an unexpected keyword argument: name
```

It looks that the saving process called the `save_vocabulary` function in `gpt2/tokenization_gpt2_fast.py`. But the tokenizer I wanted to save is `distilroberta-base`.

Does anyone get some ideas?

Comments:
It is because the version of `tokenizers` lib and the version of `transformers` lib do not match.