noamaanMulla-03/transformers-issues huggingface dataset:
Reopen: Unable to use run_squad with xla_spawn.py on TPU
Author: christian-janiake-movile
Pull request: no
Labels: name: wontfix

## Environment info
- `transformers` version: 3.1.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0a0+ab76067 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script? TPU
- Using distributed or parallel set-up in script?: YES

### Who can help
@LysandreJik 

 
## Information
I see there is an issue (#5470) closed on July, because the SQuAD example didn't have trainer support yet, but it seems that now it does have according to the table (https://github.com/huggingface/transformers/tree/master/examples#the-big-table-of-tasks)

Model I am using (Bert, XLNet ...):
BERT

The problem arises when using:
* [X] the official example scripts: (give details below)
the official example scripts: RUN_squad.py + xla_spawn.py
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (give the name)
SQuAD v2.0
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Install pytorch-xla on colab using:
VERSION = "20200325"  #@param ["1.5" , "20200325", "nightly"]
!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
!python pytorch-xla-env-setup.py --version $VERSION
Trying to run_squad on colab TPUs using xla_spawn.py
python examples/xla_spawn.py --num_cores 8 \
   examples/question-answering/run_squad.py \
  --model_type electra \
  --model_name_or_path google/electra-base-discriminator \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file "/content/drive/My Drive/bert/train.json" \
  --predict_file "/content/drive/My Drive/bert/val.json" \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir "/content/drive/My Drive/bert/newdir6"

Error is thrown up
Traceback (most recent call last):
  File "examples/xla_spawn.py", line 72, in <module>
    main()
  File "examples/xla_spawn.py", line 68, in main
    xmp.spawn(mod._mp_fn, args=(), nprocs=args.num_cores)
AttributeError: module 'run_squad' has no attribute '_mp_fn'

Expected behavior
Training should run properly using xla_spawn.py


Comments:
hi @christian-janiake-movile ,
`run_squad` won't work with `xla_spawn` since it doesn't use `Trainer`. You can use `run_squad_trainer.py` with `xla_spawn.py` if you want to fine-tune on TPU

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fine-tuning T5 in Tensorflow
Author: tqdo
Pull request: no
Labels: name: wontfix

I wonder whether there is an example/tutorial about how to fine-tune T5 for a particular task (say translation) in Tensorflow. On the doc website it says that `This model is a tf.keras.Model tf.keras.Model sub-class`. However I am not really sure what the output is for the Keras model. It seems like the output (the target sequence, in the case of translation) are considered input for the T5 model as well.

Thank you in advance.



Comments:
Hi @tqdo, 

Good question. We will add better explanation for TF T5 soon :-) 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@tqdo. I open sourced fine-tuning T5 by customizing training loop in Tensorflow2.0+: https://github.com/wangcongcong123/ttt
Hope this helps.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
RuntimeError: Internal: /sentencepiece/src/sentencepiece_processor.cc(818) [model_proto->ParseFromArray(serialized.data(), serialized.size())] 
Author: magic20191
Pull request: no
Labels: name: wontfix

# ❓ When I used transformers based on jupyter, it can't get vocab,but  the function AlbertModel.from_pretrained  is available

**CODE:**
from transformers import AlbertTokenizer, AlbertModel
import torch
tokenizer = AlbertTokenizer.from_pretrained("./albert-v1/vocab.txt")

**The following error occurs:**

Calling AlbertTokenizer.from_pretrained() with the path to a single file or url is deprecated
------------------------------------------------------------------
RuntimeError                     Traceback (most recent call last)
<ipython-input-10-bf78623a6e4a> in <module>
----> 1 tokenizer = AlbertTokenizer.from_pretrained("./albert-v1/vocab.txt")

~/anaconda3/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in from_pretrained(cls, *inputs, **kwargs)
   1138 
   1139         """
-> 1140         return cls._from_pretrained(*inputs, **kwargs)
   1141 
   1142     @classmethod

~/anaconda3/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in _from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
   1285         # Instantiate tokenizer.
   1286         try:
-> 1287             tokenizer = cls(*init_inputs, **init_kwargs)
   1288         except OSError:
   1289             raise OSError(

~/anaconda3/lib/python3.7/site-packages/transformers/tokenization_albert.py in __init__(self, vocab_file, do_lower_case, remove_space, keep_accents, bos_token, eos_token, unk_token, sep_token, pad_token, cls_token, mask_token, **kwargs)
    153 
    154         self.sp_model = spm.SentencePieceProcessor()
--> 155         self.sp_model.Load(vocab_file)
    156 
    157     @property

~/anaconda3/lib/python3.7/site-packages/sentencepiece.py in Load(self, model_file, model_proto)
    365       if model_proto:
    366         return self.LoadFromSerializedProto(model_proto)
--> 367       return self.LoadFromFile(model_file)
    368 
    369 

~/anaconda3/lib/python3.7/site-packages/sentencepiece.py in LoadFromFile(self, arg)
    175 
    176     def LoadFromFile(self, arg):
--> 177         return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
    178 
    179     def Init(self,

RuntimeError: Internal: /sentencepiece/src/sentencepiece_processor.cc(818) [model_proto->ParseFromArray(serialized.data(), serialized.size())] 


![image](https://user-images.githubusercontent.com/50441790/92065758-cf984780-edd2-11ea-8774-6a9dd5f27b16.png)




Comments:
The `AlbertTokenizer` in `transformers` is a SentencePiece based tokenizer, so it cannot load `vocab.txt`. You could try loading it in `BertTokenizer`, as it seems to be a wordpiece tokenizer vocabulary.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Using 'ner' task in pipeline with a non default model gives me entities as "LABEL-6" , "LABEL-8"  instead of "I-ORG" and "I-LOC"
Author: goutham794
Pull request: no
Labels: name: wontfix

name: Core: Pipeline

name: Ex: Named Entity Recognition

model - dbmdz/bert-base-cased-finetuned-conll03-english

Language - english

The problem arises when using:
* [x] my own modified scripts

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: ner

## To reproduce
```
tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-cased-finetuned-conll03-english")
model = AutoModelForTokenClassification.from_pretrained("dbmdz/bert-base-cased-finetuned-conll03-english")
ner_task = pipeline(task='ner', model=model, tokenizer=tokenizer)
ner_task('Hugging Face is a French company based in New-York.')
```


Comments:
Oh no, there's something wrong with the label alignment (can also be seen in the `config.json`):

```json
"id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1",
    "2": "LABEL_2",
    "3": "LABEL_3",
    "4": "LABEL_4",
    "5": "LABEL_5",
    "6": "LABEL_6",
    "7": "LABEL_7",
    "8": "LABEL_8"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1,
    "LABEL_2": 2,
    "LABEL_3": 3,
    "LABEL_4": 4,
    "LABEL_5": 5,
    "LABEL_6": 6,
    "LABEL_7": 7,
    "LABEL_8": 8
  },
```

Thanks for reporting, I'll try to fix it :)

In the meantime, you could use our large model `dbmdz/bert-large-cased-finetuned-conll03-english` 😅

Also I forgot to add, I see the [CLS], [SEP] tokens and also words like "is" , "a" in the NER results which are not present when I use the default dbmdz/bert-large-cased-finetuned-conll03-english.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


https://s3.amazonaws.com/models.huggingface.co/bert/dbmdz/bert-base-cased-finetuned-conll03-english/config.json
This file needs fixing.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I'm having the same problem with the model: dbmdz/bert-base-multilingual-cased-finetuned-conll03-spanish

> I'm having the same problem with the model: dbmdz/bert-base-multilingual-cased-finetuned-conll03-spanish

One fix is to download the model and config.json to your local and then just manually edit the "config.json" file.
This is what it should actually look like :- 
https://s3.amazonaws.com/models.huggingface.co/bert/dbmdz/bert-large-cased-finetuned-conll03-english/config.json

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Feature request of Sparselty Gated Mixture-of-Experts and PowerNorm
Author: AranKomat
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

- The newer variant of MoE for Transformer as in [GShard: Scaling Giant Models with Conditional
Computation and Automatic Sharding](https://arxiv.org/abs/2006.16668) ([Relevant codes](https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/research/moe.py))
- PowerNorm of [PowerNorm: Rethinking Batch Normalization in Transformers](https://arxiv.org/abs/2003.07845) ([Relevant codes](https://github.com/sIncerass/powernorm/blob/master/fairseq/modules/norms/mask_powernorm.py))

## Motivation

MoE as in GShard should be a crucial addition for this library, since the performance gain from it is rather significant as demonstrated in the paper. For example, Transformer+MoE achieved 44.3 avg BLEU on various NMT tasks with 10x less computes than with the baseline Transformer to get 36.9 avg BLEU. The codes for GShard has not been available yet, and if it will be published, it is most likely to be published in Tensorflow (or JAX) rather than PyTorch due to their use of TPUs. However, the codes for MoE does not seem to be complicated, so it must be easy to implement it.  

PowerNorm improved Wikitext-103 perplexity from 20.9 to 17.9 without additional computes simply by replacing LayerNorm with PowerNorm. Given that there is already PyTorch codes available, which is easy to transplant into this library, I think it's reasonable to suggest its feature request. 

## Your contribution
As for MoE, I'm currently writing my implementation of non-hierarchical MoE in PyTorch (no model parallelism as in GShard for numerous GPUs), and I'm going to compare its performance against the baseline. It should serve as a reference. My collaborator (@lucidrains) may write the same one with more flexibilities later, but no guarantee. You may want hierarchical MoE and GShard components as well. If you want them, my codes may not be as useful. 

As for PowerNorm, I don't think you need much help from me, as all you need is to replace LayerNorm with the above implemenation. I'll try to verify its performance gain, since replication of the results hasn't been done yet.  

Comments:
Is there any update on this? Thanks.

@lucidrains made this [repo](https://github.com/lucidrains/mixture-of-experts) with me.

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
The downloading url of GermEval 2014 dataset is out dated.
Author: YuanEric88
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information
The downloading url in GermanEval 2014 dataset is out dated in Readme file https://github.com/huggingface/transformers/tree/master/examples/token-classification The urls should be substituted by those in run.sh
Model I am using (Bert, XLNet ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
distilbart-cnn reproduction
Author: dhruvrnaik
Pull request: no
Labels: name: wontfix

@sshleifer 
Unable to read `config.json` for `sshleifer/student_cnn_12_6`

[config.json](https://s3.amazonaws.com/models.huggingface.co/bert/sshleifer/student_cnn_12_6/config.json) Line 64 has a format error :
```
  "force_bos_token_to_be_generated", true
}
```

This should be:
 ```
 "force_bos_token_to_be_generated" : true
}
```
This is causing an issue in loading `sshleifer/student_cnn_12_6` using `.from_pretrained()`.


Comments:
Fixed, will add a check. 
Are you running distillation experiments!? FYI that model is not trained.

> Are you running distillation experiments!? FYI that model is not trained.

Yes, I know. Reproducing the results, then planning to run a few experiments with it.

Wasn't able to use `--fp16 `, kept getting OOM errors (using 4 2080TIs).

Cool! 
re: fp16:
Are you in torch 1.6?
Try torch 1.5.1 with apex installed.

I haven't run anything successfully in torch 1.6 and am very suspicious of native amp.

> Try torch 1.5.1 with apex installed.
> 
> I haven't run anything successfully in torch 1.6 and am very suspicious of native amp.

Thanks, I will try that.
Also, did you use `run_eval.py` for the results [here](https://docs.google.com/spreadsheets/d/1EkhDMwVO02m8jCD1cG3RoFPLicpcL1GQHTQjfvDYgIM/edit#gid=0)? 
I tried using `sshleifer/distilbart-cnn-12-6` as well as one I finetuned from `sshleifer/student_cnn_12_6`, but got comparatively lower results.


Yes I did, what were your results?

Validation -     `{'rouge1': 36.902390083382635, 'rouge2': 15.98520126771937, 'rougeL': 25.75566724592724} `
Test -` {'rouge1': 33.980893339399074, 'rouge2': 13.925809496977044, 'rougeL': 23.731267594610095} `

That's awful! Can I see your command?


```
python run_eval.py distilbart-cnn-12-6/best_tfmr $DATA_DIR/val.source dbart_val_generations.txt \
    --reference_path $DATA_DIR/val.target \
    --score_path distilbart-cnn-12-6/cnn_rouge.json \
    --task summarization \
    --n_obs 100 \
    --device cuda \
    --bs 32 \
```

On 100 observations that might not be so bad. 
The 21.26 Rouge 2 is from the following command (a few months ago):

```bash
python run_eval.py sshleifer/distilbart-cnn-12-6 \
cnn_dm/test.source \
dbart_cnn_12_6_test_gens.txt \
--reference_path cnn_dm/test.target \
--score_path dbart_cnn_12_6_test_rouge.json \
--task summarization --bs 32 --fp16
```
in torch 1.5.1.

Reran Today (it took an hour)
```
{'rouge1': 44.2503, 'rouge2': 21.2586, 'rougeL': 30.3729, 'n_obs': 11490, 'runtime': 3569, 'seconds_per_sample': 0.3106}
```




I had tried with 1000 (based on the [comment](https://docs.google.com/spreadsheets/d/1EkhDMwVO02m8jCD1cG3RoFPLicpcL1GQHTQjfvDYgIM/edit#gid=0)),  had similar results. I wouldn't have expected the result to change that much, my bad. Thanks for your help!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Add model_max_length property on fast tokenizers
Author: mfuntowicz
Pull request: yes

Fast tokenizers doesn't have model_max_length property as their "slow" counterpart.

```python
>>> t.model_max_len
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'BertTokenizerFast' object has no attribute 'model_max_len'
```



Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

Comments:
I think this is handled in the base tokenizer class, Morgan.

I think I've got an issue in my dev env, something fucked up...
install sentence-transformers on Linux by python error
Author: kension0929
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

When I install sentence-transformers on Linux by python
![error](https://user-images.githubusercontent.com/15052580/88359057-da0efc80-cda3-11ea-98fb-01deddb09acd.png)
, I got an error message:

ERROR: Could not find a version that satisfies the requirement transformers>=3.0.2 (from sentence-transformers) (fr
om versions: 0.1, 2.0.0, 2.1.0, 2.1.1, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.4.0, 2.4.1, 2.5.0, 2.5.1)
ERROR: No matching distribution found for transformers>=3.0.2 (from sentence-transformers)

system: Linux 4.9.0-12-amd64 #1 SMP Debian 4.9.210-1+deb9u1 (2020-06-07) x86_64 on GCP VM instance.
Is there any suggestion?


Comments:
I am also facing same issue while trying to install transformers version 3.0.2...Did you find any solution?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


![Capture](https://user-images.githubusercontent.com/31931277/99186693-1e217500-2778-11eb-9bb1-086203a65031.PNG)
how to fix this....pls help
Deleting models
Author: RuiPChaves
Pull request: no

I would like to delete the 'bert-base-uncased' and 'bert-large-uncased' models and the tokenizer from my hardrive (working under Ubuntu 18.04). I assumed that uninstalling  pytorch-pretrained-bert would do it, but it did not. Where are these models located at?
Thanks!


Comments:
Models are usually located under `~/.cache/torch/pytorch_pretrained_bert` (older version of this library) or `~/.cache/torch/pytorch_transformers` (now) :)

Thank you! I found it elsewhere, actually. What worked for me (quaintly) was:
 find .  -type f -size +1G -print 2>/dev/null


hey, do you guys know where it is stored on windows? thanks

In my case, it gets stored  in " /tmp/torch"

In my case I found a variable containing the default cache path.
Run the following in python:
```
from transformers import file_utils
print(file_utils.default_cache_path)
```

If it is not there, check your environmental variables.

In my current transformers version `PYTORCH_PRETRAINED_BERT_CACHE`, `PYTORCH_TRANSFORMERS_CACHE` and `TRANSFORMERS_CACHE` can overwrite the default cache path.
Can the GPT2 of Transformers receive output hidden_states from external Encoder?
Author: wulaoshi
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help
<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
I want to use the GPT2 receive the output hidden_states from bert to calculate self_attention ,how can I do?
Thanks.
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
Hey @wulaoshi - I don't fully understand your question. Could you maybe post such a higher level question on the forum at `discuss.huggingface.co` ? :-) 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Specify label name
Author: exelents
Pull request: no

I'm trying to train a classification network on T5 encoder from this training code:
https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb
As I have a special output for label, I need to make trainer associate field from batch with field from my net output dictionary, as they represent output class label. How to do that?

Comments:
converting tensorflow checkpoint to pytorch
Author: mchari
Pull request: no

@LysandreJik , I am trying to convert REALM checkpoints in google-research/language/ to a pytorch checkpoint.

One of the arguments to convert_bert_original_tf_checkpoint_to_pytorch.py is bert config.json file. I don't see this file in the model directory. Just wanted to confirm that I can use any bert_config.json ? 
From the code, I see :
" help="The config json file corresponding to the pre-trained BERT model. \n"
        "This specifies the model architecture.",



Comments: