noamaanMulla-03/transformers-issues huggingface dataset:
Spanbert TACRED model not found, despite model card
Author: michaelroyzen
Pull request: no
Labels: name: wontfix

I am trying to load this model: https://huggingface.co/mrm8488/spanbert-large-finetuned-tacred in Transformers 2.11.0. My exact code is:

` tokenizer = AutoTokenizer.from_pretrained("mrm8488/spanbert-large-finetuned-tacred")  model = AutoModel.from_pretrained("mrm8488/spanbert-large-finetuned-tacred")`.

However, I get a 'not found' error: `OSError: Model name 'mrm8488/spanbert-base-finetuned-tacred' was not found in tokenizers model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, TurkuNLP/bert-base-finnish-cased-v1, TurkuNLP/bert-base-finnish-uncased-v1, wietsedv/bert-base-dutch-cased). We assumed 'mrm8488/spanbert-base-finetuned-tacred' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.
`.

Guidance would be appreciated. Thanks!

Comments:
Vocab file for that model is indeed missig, /cc @mrm8488 

But in the meantime I think you can use:

```python
tokenizer = AutoTokenizer.from_pretrained("SpanBERT/spanbert-base-cased")
```

I will upload it ASAP. Thank you for letting me know!

Tokenizer files have been uploaded!!! @michaelroyzen 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Why run_language_modelling.py does not use segment embeddings or language embeddings?
Author: HuihuiChyan
Pull request: no
Labels: name: wontfix

I am trying to train Bert and XLM on my own data. But I found that it seems run_language_modelling.py only tokenizes input text into ids, but it doesn't create token_type_ids or langs for Bert or XLM.

`batch_encoding = tokenizer.batch_encode_plus(lines, add_special_tokens=True, max_length=block_size)`
`self.examples = batch_encoding["input_ids"]`
As you can see from the codes, the examples only contain input_ids. 

I set breakpoints in the modeling_bert.py and modeling_xlm.py, and there is no token_type_ids or langs as a part of input to Bert or XLM. For the downstream tasks, Bert and XLM always use segment embeddings or language embeddings as a part of their input. Why don't we use them in the pretraining step?
If we do not use segment embeddings or language embeddings during pretraining, but use them in fine-tuning, shouldn't that cause bias?
Sorry if misunderstood something, but I am quite confused now.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Getting very bad F1 Scores when training SQUAD v2.0 with robertadistil-base
Author: manishiitg
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
https://stackoverflow.com/questions/62370365/getting-very-bad-f1-scores-when-training-squad-v2-0-with-robertadistil-base


Here is my notebook

https://github.com/manishiitg/ML_Experiments/blob/master/squad_huggingface_experiment_with_Trainer_TPU.ipynb

I am getting very bad f1 scores.

Any help on what i am doing wrong?



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Add diagnostic dataset of glue tasks for prediction
Author: stdcoutzyx
Pull request: yes
Labels: name: wontfix

As diagnostic dataset didn't have train set, it's very common in current research work to use model fine tuned on MNLI task to conduct prediction of diagnostic dataset.

As experimented, by adding logic in this request, we can achieve 47.1% on diagnostic dataset after fine-tuning 10k steps on MNLI dataset.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
update for roberta and xlm
Author: bsinghpratap
Pull request: yes
Labels: name: wontfix

Two changes are done:

Update the inputs with langs during training and evaluation.
Update the token_type_ids for Roberta otherwise it throws an error while creating features.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Not able to reproduce XNLI results from Google's mBERT weights.
Author: bsinghpratap
Pull request: no
Labels: name: wontfix

# 🐛 Bug

Hello, 
I am using mBERT from https://github.com/google-research/bert/blob/master/multilingual.md and using it with the run_xnli.py file.

Language: Learning the model on English (en) and testing it on French (fr).

The problem arises when I download the mBERT's weights from: https://github.com/google-research/bert/blob/master/multilingual.md and then use https://github.com/huggingface/transformers/blob/master/src/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py to get the PyTorch compatible weights.

I then used the PyTorch compatible weights with run_xnli provided here: https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_xnli.py. Using the language as fr and train_language as en. The model doesn't learn properly and doesn't provide performance above 0.40 on fr. 
Though if I use the weights directly provided by huggingface for bert-base-multilingual-cased (as mentioned here: https://huggingface.co/transformers/v2.3.0/examples.html). It works fine and the mBERT model is able to achieve SOTA performance. Would it be possible to understand as why this might be happening?

The tasks I am working on is XNLI. Learning on English (en) and zero-shot testing on French (fr).

Steps to reproduce the behavior:
1. Get the mBERT model from the link provided here: https://github.com/google-research/bert/blob/master/multilingual.md
2.Get the PyTorch compatible weights using: https://github.com/huggingface/transformers/blob/master/src/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py
3. Copy the vocab.txt and config.json from TF weights. Add "model_type"="bert" to config.
4. run the example run_xnli.py provided here: https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_xnli.py
5. Default parameters can be used with model_path set to the PyTorch weights of mBERT.
     
- `transformers` version: 2.11
- Python version: 3.7
- PyTorch version (GPU): 1.5.0
- Tensorflow version (GPU): 2.2.0
- Using GPU in script: Yes
- Using parallel setup across 3 1080ti GPUs


Comments:
Edit: For some reason the model: xlm-mlm-tlm-xnli15-1024 (https://huggingface.co/transformers/pretrained_models.html), also doesn't learn properly. The train accuracy never goes beyond 0.35.

Fix for XLM is provided here: https://github.com/huggingface/transformers/pull/5035

The issue of mBERT (weights from Google) still not resolved.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Illegal memory access (cudaErrorIllegalAddress)
Author: prajjwal1
Pull request: no
Labels: name: wontfix

name: dependencies

# 🐛 Bug

## Information
This bug/problem has been discussed on [Pytorch](https://github.com/pytorch/pytorch/issues/21819)/[Apex](https://github.com/NVIDIA/apex/issues/319) and [here](https://github.com/huggingface/transformers/issues/460) (bot marked it as stale) also.

I'm using Albert on GLUE (although this issue is model/dataset agnostic). 
I've made a slight modifications in my train loop (as compared to `train()` in `Trainer()`.
The main one which throws this error is when I compute the gradients:
```
grad = torch.autograd.grad(loss, model.parameters(), allow_unused=True)
```
where loss is simply `model(**inputs)[0]`

I'm using Pytorch 1.5.0+cu101, transformers 2.11 on one GPU, no multiGPU, although the instance has 2 by (CUDA_VISIBLE_DEVICES=0). I tried with `torch.cuda.set_device()` also.

Can you suggest a workaround ?


Comments:
Reducing the batch size further doesn't raise this error. But a lot of RAM is left empty. If this is an issue, then RAM demand exceeded error should be raised.

I've seen this error, and think it happens right before `OutOfMemory`.
I agree the traceback should be different.
Marking wontfix for now since this is a torch/apex issue, as you suggest, not a transformers issue. 

I don't think it's an Apex issue also because I ran my code without fp16 integration earlier. Mostly a pytorch issue. I am not sure how RAM usage exceeds in such a short time. Initially, 10 Gigs of RAM is left and suddenly this error pops up. Halving the batch size helped but there are no signs of memory leakage. Not really sure what's happening. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to implement differential learning rates and still ensure "weight_decay" = 0 for the the parameters it should?
Author: ohmeow
Pull request: no
Labels: name: wontfix

I notice that most models are trained with the following parameter groups:
```
no_decay = ["bias", "LayerNorm.weight"]

optimizer_grouped_parameters = [
    {
        "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
        "weight_decay": 1e-5,
    },
    {
        "params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],
        "weight_decay": 0.0,
    },
]
optimizer = AdamW(optimizer_grouped_parameters, lr=5e-3, eps=1e-9)
```

**Questions:**
1. Why do we set `weight_decay` = 0.0 for the `no_decay` named parameters?

2. Assuming we need to set `weight_decay` = 0.0 as such, how would we extend the above to allow for differential learning rates (which may be especially helpful for models that include both an encoder and decoder stack)?

3. Have there been any tests to demonstrate the relative effectiveness of differential learning rates for the various transformer models?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
sentencepiece dependency must be a specific version. 
Author: soni-vikas
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information
Now the `sentencepiece` library has upgraded to `0.1.92` version which is incompatible with `transformers==2.8.0`.

## Error
        `segmentation fault (core dumped)` 

## location.
https://github.com/huggingface/transformers/blob/v2.8.0/setup.py#L113



## configurations
- Platform:
- Torch version: torch==1.3.1
- Transformer version; transformers==2.8.0
- Python version: 3.7.3
- Using GPU in the script?: No.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Training ELECTRA model on TPU with the help of Trainer or TFTrainer classes
Author: DevKretov
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

Hi there,

I am trying to train Electra model from scratch using HF's Trainer interface. My primary source is this colab:

https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb#scrollTo=ri2BIQKqjfHm

There are several questions about Electra training specifics:

1. Which model to use: ElectraForPreTraining or ElectraForMaskedLM? From my perspective, both seem like appropriate choice, however <model>ForMaskedLM is been used in the Colab notebook above. 

2. If I opt for PyTorch model, I am stuck with LineByLineTextDataset. The way LineByLineTextDataset is implemented makes me use small datasets (the whole dataset is loaded into RAM and is preprocessed, so I cannot make use of really huge amount of text (hundreds of millions of sentences, for example). I tried to inherit my own Dataset type from torch's IterableDataset class, but it doesn't support sampling, which is a further step of making the whole pipeline functional. 

Does transformers framework offer another variant of Dataset class? I would much prefer having something similar to TFRecords dataset from TensorFlow 2.

3. If I still overcome the second issue, I see some crucial differences between trainer classes written for torch and TF. The most important one is Data Collator. TF Trainer doesn't offer such a parameter in class constructor, whereas the torch's Trainer does. If I'm not mistaken, data collator is responsible for masking input tokens, so running Electra model on TF Trainer is not correct, since I haven't found any implicit masking inside TF model. 

That's why I'm asking how to train Electra model with the help of TF Trainer class? Since choosing Torch's Trainer seems impossible due to the Datasets being loaded into RAM, the only variant that seems logical to me, is to train Electra model on TPU with TF Records dataset and thus making use of TF Trainer, which doesn't have a vital data preparation step of masking out 15% of tokens.

If I'm mistaken, please give me a hint. Thanks :)


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
text generation like lorem ipsum but human readable
Author: ghost
Pull request: no
Labels: name: wontfix

Hi guys,

Hope you are all well !

We would like to add a fake text generator based on transformers/gpt-2 for the wordpress module called https://github.com/bordoni/fakerpress. For now, It uses only an old and not comprehensible lorem ipsum generator.

Is it possible to generate fake text (paragraphs, headings, taxonomies) with transformers/gpt-2 based on type of topics like reddit or shakespare datasets.

Why it would be useful ? for creating fake wordpress with human readable content and also indexable by a full-text search engine module (eg manticore or elasticsearch).

Thanks in advance for any insights or inputs on that topic

Cheers,
X

Comments:
For conditional output generation, you might want to take a look at `CTRL` or you would have to fine-tuned gpt2. To just produce "any" fake text, you could  just use `GPT2` out of the box as in this demo: 
https://transformer.huggingface.co/doc/gpt2-large

Thanks for your reply :-)

What is cTRL ? do you have any references ?

Sorry I should have linked that: 
https://huggingface.co/transformers/model_doc/ctrl.html

CTRL is a very big model so quite difficult to run on a local machine - it might be easier to fine-tuned gpt2. I think @mariamabarham knows well how to fine-tune gpt2 - maybe you can add a couple lines? :-) 

I think You can use gt2 with [run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py). You can consider the [TextDataset](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py#L66) class or [LineByLineDataset](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py#L107) or define your own dataset class that suits better with your data structure.

Do you have a code example ? I am a little bit a newbie in NLP ^^ 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Found this https://colab.research.google.com/drive/1VI3oBIOQYsym2x5oOux7DTNhpdR0r4uw

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
T5ForConditionalGeneration fp16 issues
Author: sshleifer
Pull request: no
Labels: name: wontfix

This is a continuation of  and very related to https://github.com/huggingface/transformers/issues/4586
but the issue here is `NaN` loss during finetuning, rather than `nan` in model outputs.

### Instructions to reproduce
From `examples/summarization`
Setup cnn tiny:

```bash
wget https://s3.amazonaws.com/datasets.huggingface.co/summarization/cnn_tiny.tgz
tar -xzvf cnn_tiny.tgz
rm cnn_tiny.tgz

export OUTPUT_DIR_NAME=bart_utest_output
export CURRENT_DIR=${PWD}
export OUTPUT_DIR=${CURRENT_DIR}/${OUTPUT_DIR_NAME}

# Make output directory if it doesn't exist
mkdir -p $OUTPUT_DIR

# Add parent directory to python path to access lightning_base.py and utils.py
export PYTHONPATH="../":"${PYTHONPATH}"
```

### Call finetune.py with 'O1'
```bash
python finetune.py \
--data_dir=cnn_tiny/ \
--model_name_or_path=t5-large \
--learning_rate=3e-5 \
--train_batch_size=1 \
--eval_batch_size=2 \
--output_dir=$OUTPUT_DIR \
--num_train_epochs=10  \
--n_gpu=1 \
--fp16 \
--fp16_opt_level=O1 \
--do_train $@

```
Uses 16GB, but loss is always NAN.



### Call finetune.py without fp16

```bash
mkdir t5_large_fp32
python finetune.py \
--data_dir=cnn_tiny/ \
--model_name_or_path=t5-large \
--learning_rate=3e-5 \
--train_batch_size=1 \
--eval_batch_size=1 \
--output_dir=t5_large_fp32 \
--num_train_epochs=10  \
--n_gpu=1 \
--do_train $@

```
Result: OOM on 16GB card, uses 19GB on RTX (24GB Card)


If you try `t5_base`, it seems to use 8GB of GPU ram in fp16, and no issues.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
RobertaTokenizer corner case with empty string
Author: boy2000-007man
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

https://github.com/huggingface/transformers/blob/81484b447b7d8504ff5e1cfff38ec35918383963/src/transformers/tokenization_roberta.py#L239
this will introduce issue if `text == ""`, which will occur if anyone follows `run_glue.py` with QQP task as the `train.tsv` has two lines contains the empty column.
can be corrected to `if add_prefix_space and (not text or not text[0].isspace()):`

Comments:
A PR is welcome!

created PR #3621 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


cc @mfuntowicz @n1t0 

The problem is that the tokenizer does not allow empty strings to be passed (will lead to index out of bounds error when `text[0].isspace()`). An empty string is possible, according to OP, when using the QQP task which has such a format. OP added a PR here that you can have a look at https://github.com/huggingface/transformers/pull/3621

This issue speaks about fixing https://github.com/huggingface/transformers/blob/81484b447b7d8504ff5e1cfff38ec35918383963/src/transformers/tokenization_roberta.py#L239 which seems totally reasonable to me, but #3621 does a lot more than that, half of which I don't even understand.

@boy2000-007man could you update your PR to only fix the relevant line, and maybe add a test?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
New model addition: Blender (Facebook chatbot)
Author: trianxy
Pull request: no
Labels: name: wontfix

name: New model

# 🌟 New model addition

## Model description
- Dialogue response generation model for multiturn conversations, open-sourced by Facebook (28 April 2020)
- 3 models: 90m, 2.7b and 9.4b parameters
- New state-of-the-art "in terms of engagingness and humanness measurements"
- Blog post: https://parl.ai/projects/blender/
- Paper: https://arxiv.org/abs/2004.13637
- Code: Included in Facebook dialogue framework ParlAI https://github.com/facebookresearch/ParlAI

## Open source status

* [ ] the model implementation is available:
- possibly yes, together with the weights
* [x] the model weights are available:
- Need to install ParlAI (https://github.com/facebookresearch/ParlAI). Model weights are then downloaded when running `python parlai/scripts/safe_interactive.py -t blended_skill_talk -mf zoo:blender/blender_3B/model`. Compare https://github.com/facebookresearch/ParlAI/tree/master/parlai/zoo/blender
* [x] who are the authors: (mention them, if possible by @gh-username)

- Facebook AI Research: 
    - Stephen Roller, Emily Dinan, Jason Weston
    - and Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott,Kurt Shuster, Eric M. Smith, Y-Lan Boureau, 

Comments:
@tx1985 awesome idea

Very Cool !

+1

Hey, 
I want to ask a question regarding Facebook Blender Chatbot. I want to train my custom data into blender. how can i do that? Please help

Sorry, I know how to use blender as is but not how to fine tune it.   I
think that you would require a significant amount of compute to do that !

On Fri, 5 Jun 2020 at 12:09, Nitesh Kumar <notifications@github.com> wrote:

> Hey,
> I want to ask a question regarding Facebook Blender Chatbot. I want to
> train my custom data into blender. how can i do that? Please help
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/4142#issuecomment-639415197>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/APESH7GRICJLNEEJIGRPN5DRVDG73ANCNFSM4MYRA2EQ>
> .
>


Very cool!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.