noamaanMulla-03/transformers-issues huggingface dataset:
Unwanted left shift of target tokens in `get_nll`
Author: JamesDeAntonis
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.4
- Platform: ubuntu
- Python version: 3.8
- PyTorch version (GPU?): 1.6
- Tensorflow version (GPU?):
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?:no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 RAG: @patrickvonplaten, @lhoestq
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x ] my own modified scripts: (give details below)
I am training RAG on the FEVER dataset, trying to generate one token from `['[SUPPORTS]', '[REFUTES]', '[INCONCLUSIVE]'.  My loss function is always zero, I think because of the shift left that occurs in `RagTokenForGeneration.get_nll()`, which I think should only happen if special tokens are included.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
Training RAG on FEVER dataset

## To reproduce

Steps to reproduce the behavior:

1. Simply train `RAGTokenForGeneration` on anything, using only one token with no special tokens.  Loss is zero

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
@patrickvonplaten or @lhoestq might have an idea.

Hey @JamesDeAntonis yes this is expected. The same behavior would occur for GPT2 if only one token is provided as the labels. You should at least add an EOS token at the end to `labels` (so that you have two labels tokens) to make sure the loss is not zero.

If you cannot do this you will have to fork the repo and manually change the `get_nll()` function.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Allow none-tensor fields in BatchEncoding
Author: changjonathanc
Pull request: yes
Labels: name: wontfix

# What does this PR do?

This PR allows BatchEncoding to have non-tensor fields. This is useful for example when doing multi-task learning, I can add a task name (str) in the batch, and use it to decide the computation later on. 
Without this PR, I cannot use `.to('cuda')` if there is str in the batch.

I don't know who to tag. @LysandreJik



Comments:
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Longform QA demo breaks after clearing cache
Author: huu4ontocord
Pull request: no
Labels: name: wontfix

## Environment info

Browser: Chrome, running on windows.

Running demo at: http://35.226.96.115:8080/

Linked from https://github.com/huggingface/transformers/tree/master/examples/longform-qa

### Who can help
@sgugger
 
## Information
I had clicked "Clear Cache" in the app and when I did another search errors came up in the browers. subsequent runs also produces errors in the browser.

RuntimeError: Error in void faiss::gpu::allocMemorySpaceV(faiss::gpu::MemorySpace, void**, size_t) at gpu/utils/MemorySpace.cpp:26: Error: 'err == cudaSuccess' failed: failed to cudaMalloc 8987501056 bytes (error 2 out of memory)
Traceback:
File "/home/yacine/anaconda3/lib/python3.7/site-packages/streamlit/ScriptRunner.py", line 322, in _run_script
    exec(code, module.__dict__)
File "/home/yacine/Code/transformers/examples/longform-qa/eli5_app.py", line 78, in <module>
    passages, gpu_dense_index, es_client = load_indexes()
File "/home/yacine/anaconda3/lib/python3.7/site-packages/streamlit/caching.py", line 591, in wrapped_func
    return get_or_create_cached_value()
File "/home/yacine/anaconda3/lib/python3.7/site-packages/streamlit/caching.py", line 575, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/yacine/Code/transformers/examples/longform-qa/eli5_app.py", line 58, in load_indexes
    wiki40b_gpu_index_flat.add(wiki40b_passage_reps)  # TODO fix for larger GPU
File "/home/yacine/anaconda3/lib/python3.7/site-packages/faiss/__init__.py", line 138, in replacement_add
    self.add_c(n, swig_ptr(x))
File "/home/yacine/anaconda3/lib/python3.7/site-packages/faiss/swigfaiss.py", line 4245, in add
    return _swigfaiss.GpuIndexFlat_add(self, arg2, x)
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

Expected to have the results of the ELI5 search

Comments:
This seems to be an out-of-memory error! @yjernite might know what's up.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
error: sentencepiece 0.1.94 is installed but sentencepiece==0.1.91 is required by {'transformers'}
Author: rabeehkarimimahabadi
Pull request: no
Labels: name: wontfix

## Environment info
     
- `transformers` version: 3.5.1
- Platform: google cloud 
- Python version: 3.7 
- PyTorch version (GPU?): TPU, 1.6
- Tensorflow version (GPU?): - 
- Using GPU in script?: - 
- Using distributed or parallel set-up in script?: - 

### Who can help
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 examples/distillation: @VictorSanh
 examples/seq2seq: @patil-suraj

## Information
I am using requirements.txt file inside examples, and installing it, it fails with this error: 

error: sentencepiece 0.1.94 is installed but sentencepiece==0.1.91 is required by {'transformers'}

here is my setup script, as mentioned in requirements of transformers 3.5.1 for running the examples. thank you
   ```
   install_requires=[
        'sentencepiece != 0.1.92',
        'transformers==3.5.1',
        'tensorboard',
        'scikit-learn',
        'seqeval',
        'psutil',
        'sacrebleu',
        'rouge-score',
        'tensorflow_datasets',
        'pytorch-lightning==1.0.4',
        'matplotlib',
        'git-python==1.0.3',
        'faiss-cpu',
        'streamlit',
        'elasticsearch',
        'nltk',
        'pandas',
        'datasets',
        'fire',
        'pytest',
        'conllu',
        'tf-nightly',
        'google-cloud-storage',
      ],

```




Comments:
The error tells you it wants SentencePiece 0.1.91, can you install that version instead?
```
pip install -U sentencepiece==0.1.91
```
We should update the requirements.txt file to reflect this. Do you want to open a PR?

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Get locally cached models programatically 
Author: cdpierse
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

A small utility function to allow users to get a list of model binaries that are cached locally. Each list entry would be a tuple in the form `(model_url, etag, size_in_MB)`. 

## Motivation

I have quite a few environments on my local machine containing the package and have downloaded a number of models. Over time these begin to stack up in terms of storage usage so I thought it would be useful at the very least to be able to retrieve a list of the models that are stored locally as well as some info regarding their size. I had also thought about building on this further and providing a function to remove a model from the local cache programmatically. However, for now I think getting a list is a good start. 

## Your contribution

I have a PR ready to go if you think this would be a suitable feature to add. I've added it inside `file_utils.py` as this seemed like the most appropriate place. The function only adds files to the list that endwith `.bin` so right now only model binaries are included. 

An example usage of the function is below:
```python
from transformers import file_utils
models = file_utils.get_cached_models()
for model in models:
    print(model)

>>> ('https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-pytorch_model.bin', '"2d19e321961949b7f761cdffefff32c0-66"', 548.118077)
>>> ('https://cdn.huggingface.co/distilbert-base-uncased-finetuned-sst-2-english-pytorch_model.bin', '"1d085de7c065928ccec2efa407bd9f1e-16"', 267.844284)
>>> ('https://cdn.huggingface.co/twmkn9/bert-base-uncased-squad2/pytorch_model.bin', '"e5f04c87871ae3a98e6eea90f1dec146"', 437.985356)
```

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
I think that would be a cool addition! What do you think @julien-c?

Yes, I almost wrote something like that a while ago, so go for it 👍

To remove old weights you don't use anymore @cdpierse, we could also document a unix command to `find` files sorted by last access time and `rm` them

(I think @sshleifer or @patrickvonplaten had a bash alias for this at some point?)

@julien-c I have the PR for handling cached models pushed but I've been trying to think of some way to add a function that allows model deletion, we could use the model names and etags returned by `get_cached_models()` to select specfic model `.bin` files to delete but the problem is that it will still leave behind stray config and tokenizer files which probably isn't great. The filenames for tokenizers, configs, and models don't seem to be related so I'm not sure if they can be deleted that way. The two approaches seem to be to delete from last accessed date or just delete model `.bin` files. 

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Can I get logits for each sequence I acqired from model.generate()?
Author: RandolphShi
Pull request: no
Labels: name: wontfix

Hi, I’m currently stucked in getting logits from model.generate. I’m wondering if it is possible to get logits of each seqeucne returned by model.generate. (like logits for each token returned by model.logits)

Comments:
Sadly not at the moment... -> we are currently thinking about how to improve the `generate()` outputs though! 

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
KeyError: 'eval_loss' when fine-tuning gpt-2 with run_clm.py
Author: Potomac
Pull request: no
Labels: name: wontfix

## Environment info
- `transformers` version: 4.0.0-rc-1
- Platform: Linux-4.19.0-12-amd64-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.2.0 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: default option

### Who can help
 albert, bert, GPT2, XLM: @LysandreJik 
 Trainer: @sgugger

## Information

Model I am using (Bert, XLNet ...): GPT2

The problem arises when using:
* [x] the official example scripts: (give details below)
Bug occurs when running run_clm.py file from  transformers/examples/language-modeling/ , the evaluation step (--do_eval) will crash with a python error related to missing KeyError 'eval_loss',

## To reproduce

Steps to reproduce the behavior:

1. Use run_clm.py file from  transformers/examples/language-modeling/
2. Try to fine-tune gpt-2 model, with your own train file and your own validation file
3. When you add "--do_eval" option in run_clm.py then an error will occur when the step "evaluation" is reached :  

```
  File "run_clm.py", line 353, in <module>
    main()
  File "run_clm.py", line 333, in main
    perplexity = math.exp(eval_output["eval_loss"])
KeyError: 'eval_loss'
```
when I try to print the content of eval_output then there is just one key :  "epoch"

the way I execute run_clm.py  : 

```
python run_clm.py \
    --model_name_or_path gpt2 \
    --train_file train.txt \
    --validation_file dev.txt \
    --do_train \
    --do_eval \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --output_dir results/test-clm
```

## Expected behavior
The evaluation step should run without problems.

Comments:
This is weird, as the script is tested for evaluation. What does your `dev.txt` file look like?

Dev.txt contains text in english, one sentence by line.
The PC I use has 2 graphic cards, so run_clm.py uses the 2 cards for the training, perhaps the bug occurs only when 2 or more graphic card are used for the training ?

The script is tested on 2 GPUs as well as one. Are you sure this file contains enough text to have a least one batch during evaluation? This is the only thing I can think of for not having an eval_loss returned.

The dev.txt file contains 46 lines, the train file contains 268263 lines.

the specifications of the PC I use :

- Intel Xeon E5-2650 v4 (Broadwell, 2.20GHz)
- 128 Gb ram
- 2 x Nvidia GeForce GTX 1080 Ti



Like I said, the dev file is maybe too short to provide at least one batch and return a loss. You should try with a longer dev file.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Using the XLNet or Tranformer-XL as an EncoderDecoder 
Author: gulnazaki
Pull request: no
Labels: name: wontfix

I want to train a long sequence dataset (MIDI text event representation like the one in [MuseNet](https://openai.com/blog/musenet/#dataset)) from scratch. Since, I can't split the sequence to "sentences" I am using XLNet (or Transformer-XL). I am modelling the task as a sequence2sequence task (with max input seq length of around 40k tokens and output length of 4k tokens) so I want to use an Encoder Decoder Framework. 

Is it possible to use XLNet as the encoder and decoder, or just the encoder and use GPT-2 for example to do the decoding (because of the smaller output sequence length).

Thank you 🤗

Comments:
@patrickvonplaten any thoughts on this? Since, I found your work on Bert2Bert very informative :)

Hey @gulnazaki - you can use XLNet as an encoder, but not as a decoder because it'll be very difficult to add cross-attention functionality to XLNet for the decoder...

Thanks @patrickvonplaten , I thought so. 
Also, the concept of XLNet is kinda the opposite of uni-directional. 

I will try to increase the sequence length of GPT2 for the output sequence.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
saving checkpoints on gs bucket 
Author: rabeehk
Pull request: no
Labels: name: wontfix

Hi
In case of running on cloud, the model does not work to save the checkpoints on gs bucket, could you help please?
thanks

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Allow to set truncation strategy for pipeline
Author: Backfighter
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

The highlevel pipeline function should allow to set the truncation strategy of the tokenizer in the pipeline.

## Motivation

Some models will crash if the input sequence has too many tokens and require truncation. Additionally available memory is limited and it is often useful to shorten the amount of tokens. Sadly this is currently not possible using the pipeline-API. One has to call the tokenizer manually to set the truncation strategy or hope that the task specific pipeline has truncation turned on by default (the summarization pipeline for example has not).

## Your contribution

I could potentially create a PR for this, but want to confirm first that the change is welcome.

Comments:
Indeed, not being able to pass tokenizer arguments is a limiting factor of the pipelines. We're working on pipelines v2 (cc @mfuntowicz) which will allow such arguments to be passed.

In the meantime, we would definitely welcome a PR offering this functionality - but it would have to be agnostic to the argument, not specific to the truncation strategy.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Continued training of the original BERT models (not to PyTorch)
Author: singhsidhukuldeep
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

I am looking for continued training of original tiny-bert with my own raw data using masked language modelling. 
But I want the final model in TF.

## Motivation

I tried this [LM](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm_wwm.py), but this only works from pytorch to pytorch.

I tried with using original ckpt and from_tf, it always results in a .h5 error 

Please let me know I can explain more OR help in any way.

Basically, we should be able to use tf weights >> masked language modelling >> and have a domain specific pre-trained Tensorflow language model.




Comments:
You should first convert your checkpoint to a huggingface checkpoint. You can check the docs on how to do that [here](https://huggingface.co/transformers/converting_tensorflow_models.html#bert).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Blank line indicates the end of a document for NER training ?
Author: polodealvarado
Pull request: no
Labels: name: wontfix

Hi everyone!

In the <https://huggingface.co/transformers/custom_datasets.html#token-classification-with-w-nut-emerging-entities> , it says that each line of the  dataset file contains either (1) a word and tag separated by a tab, or (2) a blank line indicating the end of a document. 

The blank line should not represent the end of a sentence?





Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
TFTrainer & Eager mode
Author: JulesGM
Pull request: no
Labels: name: wontfix

Is eager mode really not supported with TFTrainer? It's telling me that it is using `tf.gradients`, which is not supported with eager mode.
If that's true, then I maybe it could be displayed a lot more prominently in your documentation ... I wasted so much time implementing custom functions, 🤦‍♂️
@sgugger


Comments:
cc @jplu for TFTrainer.

Hello @JulesGM!

Yes, the TF Trainer don't support eager execution, partially because of the usage of `tf.gradients` that we use because sometime when loading a model for a specific task, not all the layers are used and bring some None values when computing the gradients. `tf.gradients` allows to ignore these None values.

I'm really sorry that it is not enough detailed in the documentation, we are currently working on an improved version of the TF Trainer that you can find [here](https://github.com/huggingface/transformers/pull/8264). And it should be much easier and convenient to use. Sorry once again for the inconvenience you have encountered with the TFTrainer until now.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
The Albert tokenizer file cannot download automatically and the official Albert tokenizer file is wrong, I cannot use it.
Author: probe2
Pull request: no
Labels: name: wontfix

Have anybody meet the same problem?

Comments:
Do you mind following and filing in the issue template?
Thanks

I cannot load the offiline vocab as well:
RuntimeError: Internal: C:\projects\sentencepiece\src\sentencepiece_processor.cc(824) [model_proto->ParseFromArray(serialized.data(), serialized.size())]
but it can run for me to download the vocab automatically.


This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.