noamaanMulla-03/transformers-issues huggingface dataset:
__init__() got an unexpected keyword argument 'cache_dir' 
Author: jasonyliang
Pull request: no

After reviewing issue #7006 and following the steps (updating Transformers to the latest version or even the 3.1.0 branch), I still get the error:
TypeError: init() got an unexpected keyword argument 'cache_dir'
when running the latest version for transformers (3.1.0).
I'm also running on Colab environment:
Command:
!pip3 install transformers
(also tried #! pip3 install git+git://github.com/huggingface/transformers/)

!wget https://raw.githubusercontent.com/huggingface/transformers/master/examples/language-modeling/run_language_modeling.py

%%bash
export TRAIN_FILE=train_path
export TEST_FILE=valid_path
export MODEL_NAME=gpt2
export OUTPUT_DIR=output

python run_language_modeling.py
--output_dir=output
--model_type=gpt2
--model_name_or_path=gpt2
--do_train
--train_data_file=$TRAIN_FILE
--do_eval
--eval_data_file=$TEST_FILE
--cache_dir=None

Output:
Traceback (most recent call last):
File "run_language_modeling.py", line 313, in
main()
File "run_language_modeling.py", line 242, in main
get_dataset(data_args, tokenizer=tokenizer, cache_dir=model_args.cache_dir) if training_args.do_train else None
File "run_language_modeling.py", line 143, in get_dataset
cache_dir=cache_dir,
TypeError: init() got an unexpected keyword argument 'cache_dir'

Comments:
Do you have a link to the colab?

I was following the tutorial by Joey S and tried to include my own dataset instead. This is the notebook that he has provided:
https://colab.research.google.com/drive/1odn0VHb4SmQnUqtOwK6XQTLpoTSbneCf?usp=sharing

The notebook he provided already has the error.
Thank you so much for taking a look!

The error seems to not be here anymore when doing `!pip3 install git+https://github.com/huggingface/transformers` instead of `pip3 install transformers` and resetting to factory settings before doing the run.

Thank you so much! It's working now!

Glad I could help!
fix the warning message of overflowed sequence
Author: xiye17
Pull request: yes

This PR fixes the warning message (of _**PreTrainedTokenizerBase.prepare_for_model**_) when trying to tokenize a model with a sequence that is longer than the specified maximum sequence length.

The current warning message shows the length of the first input_ids, which can be incorrect when a pair is being encoded and the first input length doesn't exceed the limit. The desired warning message should show the overall length of the encoded ids. 

Comments:
unexpected keyword argument 'force_fusions' when running the onnx notebook
Author: pierre-si
Pull request: no

## Environment info
   
- `transformers` version: 3.1.0
- Platform: Linux-5.8.0-1-amd64-x86_64-with-debian-bullseye-sid
- Python version: 3.7.8
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

## Information

The problem arises when using:
* [x] the official example scripts: the notebook "04-onnx-export"
* [ ] my own modified scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run the notebook's cells. 
First cell installs onnxruntime in version 1.4.0
The "Benchmarking ONNX quantized model" cell calls transformers/convert_graph_to_onnx.py,
which calls the "quantize" function with parameter force_fusions.

`Error: quantize() got an unexpected keyword argument 'force_fusions'`
The parameter was discarded in onnxruntime 1.4.0
Removing the parameter in convert_graph_to_onnx.py solves the issue.
## Expected behavior

No error.


Comments:
The error might be something else, onnxruntime version 1.4.0 does include the `force_fusions` parameter: 

https://github.com/microsoft/onnxruntime/blob/v1.4.0/onnxruntime/python/tools/quantization/quantize.py#L1457

Indeed,
my bad, somehow the notebook used the python 3.8 related onnxruntime package and not the one associated with the conda environment from which I was launching jupyter notebook.
It works perfectly, thanks :+1:
RuntimeError: CUDA out of memory. 
Author: jasonyliang
Pull request: no

I'm trying to run run_language_modeling.py and fine-tune the GPT-2 model with my own text dataset with the following command:
%%bash
export TRAIN_FILE=Models/Data/train.txt
export TEST_FILE=Models/Data/valid.txt
export MODEL_NAME=gpt2
export OUTPUT_DIR=output

python run_language_modeling.py \
    --output_dir=output \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --cache_dir=None


I am currently using the Colab environment to run the script on GPU but encountered the following error:
RuntimeError: CUDA out of memory. Tried to allocate 384.00 MiB (GPU 0; 15.90 GiB total capacity; 14.75 GiB already allocated; 225.81 MiB free; 14.81 GiB reserved in total by PyTorch)

I noticed that it might be possible to resolve this issue by changing the batch but am unsure how I can do this?
Would love any pointers available (  --train_batch_size 16  wasn't an option with this file)!
Thank you so much!

Comments:
Found solution:
    --per_gpu_train_batch_size=1 \
I will be closing this issue now, thank you so much!
Sorry for the trouble!
[s2s] distributed eval cleanup
Author: sshleifer
Pull request: yes

Fixes #7176 

- [x] Fix determinism issue caused by hardcoding num_beams
- [x] local_rank 0 logging
- [x] local_rank 0 tqdm
- [x] same results as `run_eval.py`
- [x] deeper investigation of non-determinism. Gens the same. Labels different?
- [x] save json to one line.


Comments:
cc @stas00
[seq2seq Examples] Use _step instead of generate for val, test
Author: setu4993
Pull request: yes

Addresses a workaround I proposed in #6589 for limiting the memory consumption in `.generate` steps.

@sshleifer 

Comments:
@sshleifer : Does it make sense to still add this? If yes, I can rebase and update based on Suraj's comments earlier. If not, will close.

Let's hold off for now. I think passing --eval_num_beams=1 is close enough to equivalent.
Thanks for trying!
Segmentation fault (core dumped)
Author: SummmerSnow
Pull request: no

Hi, 

   I downloaded pretrained model and vocabulary file， and wanted to test BertModel to get hidden states.
   when this
  ```encoded_layers, _ = model(tokens_tensor, segments_tensors)```  lines run,  I got this error:  Segmentation fault (core dumped).

   I wonder what caused this error

Comments:
Hi, you need to give me more information (a screen copy of a full log of the error).

Actually, this is all I got:

>>  python bert.py
12/15/2018 19:43:06 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file /home/snow/bert_models_path/vocab.txt
12/15/2018 19:43:06 - INFO - pytorch_pretrained_bert.modeling -   loading archive file /home/snow/bert_models_path
12/15/2018 19:43:06 - INFO - pytorch_pretrained_bert.modeling -   Model config {
  "attention_probs_dropout_prob": 0.1,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "type_vocab_size": 2,
  "vocab_size": 30522
}

bert_models_path load!!!
Segmentation fault (core dumped)

There is no special c function in our package, it's all python code.
Maybe you just don't have enough memory to load BERT?
Or some dependency is not well installed like pytorch (or apex if you are using it).

Thanks for your advice.
Maybe because of my pytorch version(0.4.0) I not not sure.
I download the source code instead of pip install and using 0.4.1 version and run successfully.

Thanks for your code and advice again~

![图片](https://user-images.githubusercontent.com/26063832/57267682-8d160c00-70b3-11e9-9d59-5866c1321272.png)
I also have this problem, and my torch version is 1.0.1 . I have tried to download the source code instead of pip install but also failed.

I have the same question  @wyx518 

Did you solve this ? @zhaohongjie 

@zhaohongjie did you solve this? I have the same question too.

Has someone solved this issue by any chance?


> Has someone solved this issue by any chance?

Do you have the same problem? You could try to debug it by import transformers and torch only, then call torch.nn.CrossEntropyLoss() to see if it results in Segmentation fault. I accidentally fixed this error by install more packages

Hello,

Had this error with CamembertForSequenceClassification.from_pretrained(), needed to update torch==1.5.1 and torchvision==0.6.1  

I had the same issue while loading pretrained models.
Updated to the last version of Pytorch (1.5.1) and worked fine.

Yup, that worked guys! Thank you @Daugit and @gabrer 

Update by pip install torch==1.5.1 and the problem solved
Multiple Mask Tokens
Author: zbush548
Pull request: no
Labels: name: wontfix

For those wishing to [MASK] several tokens, here this is.

My question, however, relates to the output. I added "top_k" assuming I'd be able to return multiple sentences, but that was not the case. I am not sure how exactly I can achieve this. 

```
import torch
from transformers import BertTokenizer, BertModel,BertForMaskedLM
 
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
 
input_tx = "[CLS] [MASK] [MASK] [MASK] of the United States mismangement of the Coronavirus is its distrust of science. [SEP]"
tokenized_text = tokenizer.tokenize(input_tx)
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
top_k = 10
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([[0]*25])
 
model = BertForMaskedLM.from_pretrained('bert-base-cased')
 
outputs = model(tokens_tensor, token_type_ids=segments_tensors)
predictions = outputs[0]
predicted_index = [torch.argmax(predictions[0, i]).item() for i in range(0,24)]
predicted_token = [tokenizer.convert_ids_to_tokens([predicted_index[x]])[0] for x in range(1,24)]
print(predicted_token)

```

`Output: 'The', 'main', 'cause', 'of', 'the', 'United', 'States', 'mi', '##sman', '##gement', 'of', 'the', 'Co', '##rona', '##virus', 'is', 'its', 'di', '##st', '##rust', 'of', 'science', '`

Comments:
Hi! This is a very good question. We just opened a forum on [discuss.huggingface.co](https://discuss.huggingface.co/) to discuss those kind of questions exactly. Do you think you could go over there and ask it? Thanks a lot!

Sure, I'll do that now! @LysandreJik 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to download original weights of gpt2
Author: nuradilK
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help
Hi, I have been trying to reproduce losses from [this paper](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf), but I realized that my losses are lower than the ones from the paper. Also, I noticed a warning - Some weights of GPT2LMHeadModel were not initialized from the model checkpoint at gpt2-medium and are newly initialized: ... Does it mean that the original weights of GPT-2 was changed? If so, how can I download the original weights?
For instance my PPL for PTB is 35.20 while the PPL in the paper is 65.85.

Comments:
Hey @nuradilK,

1) The GPT2 weight initialization is a known warning bug we should remove, but which does not affect model performance, 
see https://github.com/huggingface/transformers/issues/5800 e.g.

2) It's very hard for us to track down low performance of specific user code (which data to use, data processing, ...). Do you mind posting a question like "How to reproduce GPT2 PPL results" on https://discuss.huggingface.co/ since it is not really a bug but more a high level question. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Create larger summaries by using Summarization models like T5 or Pegasus
Author: MichaelJanz
Pull request: no

Hi,
I want to create summaries of book-reviews, which should be about half the size of the original text corpus.
My goal is to compare how well the summaries are created and if the important content of the original texts are taken over.
For this I just had a look at different models and saw, that the models have a very fixed max length (often about 256 tokens), which corresponds in creating only 1-2 sentences, while my goal is for example to have 8 sentences, when the original text corpus has 16.

Of course I could split my text corpus in smaller ones, but then I am afraid that the context of the previous sentences gets lost. (Is there maybe a way to pass the context information on into the next predictions, as it could be done in stateful RNNs? Atleast I dont think so)

For sure many others already had the issue, since there are tasks as summarizing whole documents or even books, which cannot be done in one sentence.

I already [asked](https://stackoverflow.com/questions/63904821/using-transformer-for-text-summarization) on SO, but unfortunately I didnt get an answer there.
So I would appreciate any hint on how I could continue on my task.

Thank you!

Comments:
Pinging the summarization expert here @sshleifer 

Can we move [here](https://discuss.huggingface.co/t/summarization-on-long-documents/920/5)

I am super happy to help, but want to help everybody at the same time. @ibeltagy is working hard on summarizing longer docs, but besides his efforts we don't have what you are looking for.

Thank you for posting, I will have a look at it.
Transformer-XL: Remove unused/unnecessary Parameters
Author: RafaelWO
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->
The used configuration parameters `tgt_len` and `ext_len` in the Transformer-XL implementation are not utilized in the source code. I would recommend removing those since the only confuse the users of the model(s).

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

As already mentioned, the unnecessary parameters are confusing and can be likely removed. The parameter `tgt_len` is determined for the input tensor anyway and `ext_len` was only an experimental feature (see [issue from the original repo](https://github.com/kimiyoung/transformer-xl/issues/9)).

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

I am happy to contribute and open a PR with the requested changes if that's also in your interest?

Comments:
ping @TevenLeScao
FSMT Cuda CI Failures
Author: sshleifer
Pull request: no

https://github.com/huggingface/transformers/runs/1130127719?check_suite_focus=true

I'm not sure whether these are spurious, but I wanted to bring them to your attention @stas00 .

Comments:
Thank you for the heads up, @sshleifer 

I'm not able to reproduce those, but let's first thing first let's skip them so that CI is not affected. 
https://github.com/huggingface/transformers/pull/7220
I will investigate and we can try to enable them again later.

I've found some other tests failing with USE_CUDA=1, but which were fine on CI - so I will attend to those first.

Hope we could finalize this issue https://github.com/huggingface/transformers/issues/6349 and get USE_CUDA on by default.
Suggestion: Better to change the task name of  "sentiment-analysis" to "text-classification"
Author: shirayu
Pull request: no

There is a task named ``sentiment-analysis`` in``SUPPORTED_TASKS`` used in piepelines.
https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines.py#L2509

However, it can be used not only for sentiment analysis but also fo more generic tasks.
Therefore, I think it is better to rename it to ``text-classification`` (or add a task named ``text-classification`` for backward compatibility).

Comments:
These are tasks that can be done with a specific pipeline, with a model and tokenizer as default. Having the `sentiment-analysis` task supported with a `TextClassificationPipeline` is the same as having a `ner` task supported with a `TokenClassificationPipeline`. These tasks are not generic text-classification or token-classification, but specific tasks that users can use.

If you want a general pipeline for text classification, then you can simply use the text-classification pipeline directly:
```py
from transformers import TextClassificationPipeline
```
Links seem to be expired for German NER example 
Author: Santosh-Gupta
Pull request: no
Labels: name: Ex: Named Entity Recognition

There is a German Ner example here

https://github.com/huggingface/transformers/tree/master/examples/token-classification

It seems that the links for the training data are no longer working

```
curl -L 'https://sites.google.com/site/germeval2014ner/data/NER-de-train.tsv?attredirects=0&d=1' \
| grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > train.txt.tmp
curl -L 'https://sites.google.com/site/germeval2014ner/data/NER-de-dev.tsv?attredirects=0&d=1' \
| grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > dev.txt.tmp
curl -L 'https://sites.google.com/site/germeval2014ner/data/NER-de-test.tsv?attredirects=0&d=1' \
| grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > test.txt.tmp
```

I tried going to the links directly and got 404.

For convenience, here's a colab notebook that runs the whole example. The contents of the created downloaded and processed data files do not seem correct to me.  

https://colab.research.google.com/drive/1ox23ZuWNwh3pim8OQAW99TF2sTVHzKqH?usp=sharing

Comments:
@stefan-it might know what's up!

Oh, I think this PR should fix it:

https://github.com/huggingface/transformers/pull/6571

I will resolve some merge conflicts, then it should be ready to merge!

Updated, it should work now :+1: 

@Santosh-Gupta Please try to use the new urls for the GermEval dataset in your Colab :)

The links have been updated, thanks @stefan-it
Adding Neutral Score 
Author: MohammadReza-Babaee
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

After performing some experimentation and comparison to VADER, we come to consensus that "pretrained BERT-based Hugging Face transfomer" is performing way beyond the other lexicons, but VADER is also good at social media context + it provides "neutral" label which turns out to be useful in some context. 

I was wondering whether it is possible to manipulate the Transformer Sentiment Analysis in a way that it can calculate the **"neutral" score**?



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hey all, any chance anyone else is working around this? I think a neutral label or a standard sentiment score would be great for such an extensive model. Neutral statements are not caught with this adjustment:

classifier('I do not know the answer.')
Out[16]: [{'label': 'NEGATIVE', 'score': 0.9995205402374268}]

classifier('This is meant to be a very neutral statement.')
Out[17]: [{'label': 'NEGATIVE', 'score': 0.987031102180481}]

classifier('The last president of US is Donald Trump.')
Out[18]: [{'label': 'POSITIVE', 'score': 0.9963828325271606}]

classifier('There is going to be an election in two months.')
Out[19]: [{'label': 'NEGATIVE', 'score': 0.9604763984680176}]

Just raising this thread again to see if there is a common interest...
Cheers!
[WIP] add DeFormer (ACL 2020) example
Author: csarron
Pull request: yes
Labels: name: wontfix

Hi there, I'm one of the authors of the [DeFormer paper](https://www.aclweb.org/anthology/2020.acl-main.411/), and I'd like to adapt the [DeFormer codebase](https://github.com/StonyBrookNLP/deformer) to this awesome transformers library. To get the adaptation done, I put a few high-level todos in the README (also here). I plan to get a working example by/before August but don't have a precise timeline yet. Let me hear what you think. Thanks.

- [ ] use HF preprocessing (use HF nlp library)
- [ ] convert original TF DeFormer to HF version
- [ ] convert pre-trained checkpoints 
- [ ] compare and test accuracy for SQuAD, RACE, and BoolQ
- [ ] prepare demo and upload to model cards


Comments:
Hi Qingqing, good to see you here! Yeah, I checked your official code repo and found it isn't based on 🤗's Transformers. It'll be nice if you can reimplement it and add Deformer in our library! 

~~@thomwolf  Re. `🤗nlp`, should we use `🤗nlp` in `transformers` right away? Since it would make `🤗nlp` a dependency of `transformers/examples`.~~

Never mind, it already is.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Span Mask Fill
Author: BigSalmon2
Pull request: no
Labels: name: wontfix

I see that Transformers does not support Ernie, but am in search of a way to [MASK] phrases. Can somebody guide me to an alternative to Ernie, code, or a way I could do this myself?

Comments:
Maybe you can get inspiration from here: https://github.com/facebookresearch/SpanBERT

> Maybe you can get inspiration from here: https://github.com/facebookresearch/SpanBERT

@RudrakshTuwani 

Thank you for your response. I've actually tried to implement SpanBERT previously, but my status as a beginner must have barred me from doing it properly. 

While SpanBERT has "the same format as the HuggingFace BERT models", the outputs were only special tokens or letters.   

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to generate sentences from Transformer's sentence embeddings?
Author: willanxywc
Pull request: no
Labels: name: wontfix

Is it possible to use the pre-trained transformer models to generate sentences from sentence embeddings?
I konw I can get the continuous representations of a sentence with for example BertModel or GPT2Model.
But can I reconstruct the sentence directly from the sentence representations generated by BertModel, with hugging-face transformer, especially the pre-trained ones?
I mean sentence embeddings as input, the readable sentence as output.

Comments:
Sorry I'm not 100% sure, I get the question. Could you specify what you mean by `sentence embeddings` exactly and the output you would like with some code? :-) 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.