noamaanMulla-03/transformers-issues huggingface dataset:
grad checkpoint for T5 model -- and lots of debug not yet removed
Author: moscow25
Pull request: yes
Labels: name: wontfix

@sshleifer here is my grad checkpointing code -- albeit with too much debug info and some FP16 stuff also not deleted.

- all FP16 needs to be off -- else the model does not converge
- with grad checkpoint as hacked now (for every "T5 Stack") -- T5-Large runs with training batch 32
- still unable to train T5-3B even with batch == 1
- still working on grad checkpointing for DDP (which we need for multi-GPU)
- need to add grad checkpointing as config option... and many other cleanup but it does work


<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


Comments:
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
run_squad.py eval metrics meaning
Author: batyas
Pull request: no
Labels: name: wontfix

I am having difficulty understanding what exactly the best_f1 and best_exact scores that are outputted in the run_squad.py evaluation mean. (The scores are computed in the squad_metrics script, found [here](https://github.com/huggingface/transformers/blob/master/src/transformers/data/metrics/squad_metrics.py)). 
What are the "scores" the best calculations are working with, what do the metrics represent, and when/is the best_threshold value employed during training?

Thank you!

Comments:
The model assign correctness probability to every answer it produce. if this probability crossing the threshold it means that the model predict their is no answer to the question. The threshold is picked as the threshold that achieve the best f1/exact score on the dev set. The best f1/exact is the result achieved with the best threshold found.

Unfortunately, their is a [bug](https://github.com/huggingface/transformers/pull/7319) in run_squad.py code which was used for the training and evaluation of most of the models in the library and the results you see in their model cards are incorrect. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
MBartTokenizerTrimmed to support truncated embeddings
Author: sshleifer
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

name: translation

Motivation:
The embeddings table for MBART is huge, but only ~40K of the entries are used/finetuned for most wmt tasks. See https://github.com/pytorch/fairseq/issues/2120 

- needs vocab.json (fairseq Dictionary)
- needs to call `encode_as_pieces` with restricted vocabulary.

I will take this.

Comments:
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Finetuning German BERT for QA on biomedical domain 
Author: sbhttchryy
Pull request: no
Labels: name: wontfix

Hello there and thank you very much for this wonderful work. I am relatively new to this field, so please bear with my amateur question. I want to perform question-answering on a German Biomedical text. From what I understand up to now, I need to fine-tune German BERT on biomedical QA datasets. Is there any script/pipeline that I should be using for this?
I have also posted in StackOverflow and the huggingface forum, but to no avail as of now. 
Thank you very much in advance.

Comments:
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
When i train a GPT-2 which uses BPE, the computed perplexity is per sub-word right?
Author: Nkonstan
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:

Comments:
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Use pretrained bert withou embedding layers.
Author: demdecuong
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->

https://arxiv.org/pdf/2005.07421.pdf

I was inspirited by the above paper. But the author set the input of bert is there soft-masked embeddings which is not the original format of bert.

So are there any way for me to get Bert without its embedding layers.


Comments:
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
offline mode for firewalled envs
Author: stas00
Pull request: yes

This PR implements the proposal from https://github.com/huggingface/transformers/issues/10379 to enable transformers to cache everything it needs and then run in the offline mode - e.g. in a firewalled environment.

This PR:

* [x] adds `is_offline_mode()` helper function that returns `True` when env var `TRANSFORMERS_OFFLINE` is set to `1/YES/ON`
* [x] automatically sets `local_files_only=True` in all 3 `from_pretrained()` methods
* [x] handles `ntlk` download dynamically in `run_seq2seq.py`
* [x] adds offline test (thanks to @lhoestq for the idea for mocking no network in the test) 
* [x] adds doc

This is to match the recently added `HF_DATASETS_OFFLINE=1` in `datasets` (https://github.com/huggingface/datasets/pull/1976). Tested that both work well together.

So now we can run with the network:
```
python examples/seq2seq/run_seq2seq.py --model_name_or_path t5-small --dataset_name wmt16 --dataset_config ro-en ...
```
and then with the same filesystem w/o the network or w/ a firewalled network:
```
HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \
python examples/seq2seq/run_seq2seq.py --model_name_or_path t5-small --dataset_name wmt16 --dataset_config ro-en ...
```
and the latter succeeds since step 1 had all the data pre-fetched and cached.

@sgugger, @LysandreJik 

Comments:
> The constant should be documented in the install page, in the section about caching models I think (https://huggingface.co/transformers/installation.html#caching-models).

Great idea, @sgugger. Please kindly check the doc I added is good when you get a chance.

And also the kind of test I had to add is unorthodox too, so please see if it works for you. The original version couldn't have worked.

Thank you!
Refactoring checkpoint names for multiple models
Author: danielpatrickhug
Pull request: yes

Hi, @sgugger reupload without datasets dir and added tf_modeling files, removed extra decorator in distilbert.
Linked to #10193, this PR refactors the checkpoint names in one private constant. 

one note: longformer_tf has two checkpoints "allenai/longformer-base-4096" & "allenai/longformer-large-4096-finetuned-triviaqa". I set the checkpoint constant to "allenai/longformer-base-4096" and left the one decorator with "allenai/longformer-large-4096-finetuned-triviaqa".

Fixes #10193


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
...The test pasts on my local machine, I ran make test, style, quality, fixup. I dont know why this failed..

I just rebased the PR to ensure that the tests pass. We'll merge if all is green!

Thanks guys
Implementing ELECTRIC training for ELECTRA
Author: stephantul
Pull request: no
Labels: name: Feature request

# 🚀 Feature request

Google released Electric this summer at EMNLP (see: [here](https://www.aclweb.org/anthology/2020.emnlp-main.20.pdf)). Electric is like ELECTRA, but trained using a Noise Contrastive Estimation loss instead of a negative sampling loss. 

## Motivation

Electric is well-suited for modeling perplexity scores, and can model these very efficiently. Modeling these perplexity scores using BERT requires N passes over the input sentence, where N is the number of tokes in the sentence (see [here](https://arxiv.org/abs/1910.14659)).

## Your contribution

Electric has been implemented in the Google Electra repository. From I can see, moving from an Electra to Electric-style training is not a huge code change, but I'm not that familiar with the inner workings of transformers to be able to make a judgment call on this.

Comments:
I have trained Bert on my own data which has been converted to IDs by using BertForMaskedLM, but when I use the model for the further fine-tuned, I found this error 
Author: lisiyuan1209
Pull request: no

@LysandreJik
## code info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
here's my model:


![image](https://user-images.githubusercontent.com/56223656/110211963-61276580-7e99-11eb-8508-42591b5138e6.png)

![image](https://user-images.githubusercontent.com/56223656/110212033-b2cff000-7e99-11eb-8378-116dafe78f8e.png)




## Information

Model I am using my own pre-tained Bert model which is stored in the path "../bert":

The problem arises when using:

![image](https://user-images.githubusercontent.com/56223656/110212083-fb87a900-7e99-11eb-9457-f4b94da1234b.png)
   
here are the files in the path "../bert":

![image](https://user-images.githubusercontent.com/56223656/110212157-55886e80-7e9a-11eb-8b7f-dce93af1b72e.png)

The tasks I am working on is: text matching:
* [ ] my own dataset is like: 

![屏幕快照 2021-03-06 16 29 16](https://user-images.githubusercontent.com/56223656/110211923-2a514f80-7e99-11eb-86ae-c403a7e7001e.png)






## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
what exactly is the problem?

Comments:
[firewalled env] OFFLINE mode
Author: stas00
Pull request: no

This is done - we now have:

* `HF_DATASETS_OFFLINE=1`
* `TRANSFORMERS_OFFLINE=1`

Documented:  [here](https://huggingface.co/transformers/master/installation.html#offline-mode)

The transformers-specific issue is here:
-------------------


Similar to `datasets` https://github.com/huggingface/datasets/issues/1939 `transformers` needs to have an OFFLINE mode where it can work w/o ever making a network call to the outside world.

This issue comes from a need to be able to run `transformers` in a firewalled env, which currently makes the software hang until it times out, as it's unable to complete the network calls.

We assume `DATASETS_OFFLINE=1` will already deal with datasets and metrics as I proposed at https://github.com/huggingface/datasets/issues/1939, so this issue is specific to `transformers` only.

I propose the following approach to solving this problem, using the example of `run_seq2seq.py` as a sample program. There are 2 possible ways to going about it.

## 1. Manual

manually download model files, that is transfer to the firewalled instance and run:

```
TRANSFORMERS_OFFLINE=1 run_seq2seq.py   --model_name_or_path ./t5-small-local ... 
```

`transformers` must not make any network calls and if there is a logic to do that and something is missing it should assert that this or that action requires network and therefore it can't proceed.

## 2. Automatic

In some clouds one can prepare a data storage ahead of time with a normal networked environment but which doesn't have gpus and then one switches to the gpu instance which is firewalled, but it can access all the cached data. This is the ideal situation, since in this scenario we don't have to do anything manually, but simply run the same application twice:

1. on the non-firewalled instance:
```
run_seq2seq.py  --model_name_or_path t5-small ...
```

which should download and cached everything.

2. and then immediately after on the firewalled instance, which shares the same filesystem:
```
TRANSFORMERS_OFFLINE=1 run_seq2seq.py  --model_name_or_path t5-small ...
```

and the model should be cached by the invocation number 1 and any network calls be skipped and if the logic is missing data it should assert and not try to fetch any data from online.

## Specifics

1. We already have `local_files_only=True` for all 3 `.from_pretrained()` calls which make this already possible, but this requires editing software between invocation 1 and 2 in the Automatic scenario which is very error-prone. Thus I propose that `TRANSFORMERS_OFFLINE=1` will turn these flags True from the ouside of the system.

2. There are other issues to check, for example in some `examples` scripts we have:

```
with FileLock(".lock") as lock:
    nltk.download("punkt", quiet=True)
```

which also issues a network call and under `TRANSFORMERS_OFFLINE=1` it should be skipped and replaced with a check that the corresponding nltk data is already available.

Thanks.

@julien-c, @sgugger, @LysandreJik 

Comments:
This is done.
Gpt2 generation of text larger than 1024
Author: rautnikita77
Pull request: no
Labels: name: wontfix

name: Ex: Generation

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

I know the context supported by GPT2 is 1024, but I assume there's some technique they utilized to train and generate text longer than that in their results. Also, I saw many gpt2-based repos training text with length longer than 1024. But when I tried generating text longer than 1024 it throws a runtime error :The size of tensor a (1025) must match the size of tensor b (1024) at non-singleton dimension 3. I have the following questions:
1) Shouldn't it be possible to generate longer text since a sliding window is used?
2) Can you please explain what's necessary to generate longer text? What changes will I have to make to the run_generation.py code?




<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->



Comments:
After looking over `modeling_utils.generate` (the function used for generation by `run_generation.py`), I believe that the sliding window approach is not yet implemented.

This method was implemented for CTRL generation in their repo, so you may be able to adapt some of their code for your use case: https://github.com/salesforce/ctrl/blob/master/generation.py#L186-L189

The iterative approach to generation (using `past`) may work better because you can control the sliding window manually.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@patrickvonplaten @rautnikita77 @minimaxir Has anyone attempted to implement this (using cached keys and values)?
"Write With Transformer" inserts a space whenever accepting a suggestion, even if a space doesn't belong there
Author: flarn2006
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT-2

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [X] the official example scripts: Write With Transformer
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: Write With Transformer
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Type something ending in the middle of a word. (e.g. `See how a modern neural netw`) Alternatively, type something ending with an open parenthesis or quotation mark, such as `Donald Trump tweeted "`.
2. Press Tab and accept a suggestion.
3. Observe how a space is added prior to the accepted text, despite a space not belonging there.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

I'm not sure about the other models, but I know GPT-2 does start continuations with a space when appropriate. So it should be possible to have it only add a space when a space is desirable. Otherwise, I think it would be better to have it not add a space at all, as it's easier to manually add a space before pushing Tab (if one is desired) than it is to go back and delete undesired spaces every time they're generated.

## Environment info
N/A

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue still occurs. How do I reopen?
[run_seq2seq] fix nltk lookup
Author: stas00
Pull request: yes

Hmm, CI crashes every so often on 
```
try:
    nltk.data.find("tokenizers/punkt")
except LookupError:
```

introduced in this PR: https://github.com/huggingface/transformers/pull/10407

https://app.circleci.com/pipelines/github/huggingface/transformers/20635/workflows/989fde0b-e543-4620-9d9a-f213ad53dd9b/jobs/176742
```
__________________ ERROR collecting examples/test_examples.py __________________
examples/test_examples.py:51: in <module>
    import run_seq2seq
examples/seq2seq/run_seq2seq.py:54: in <module>
    nltk.data.find("tokenizers/punkt")
../.local/lib/python3.6/site-packages/nltk/data.py:539: in find
    return FileSystemPathPointer(p)
../.local/lib/python3.6/site-packages/nltk/compat.py:41: in _decorator
    return init_func(*args, **kwargs)
../.local/lib/python3.6/site-packages/nltk/data.py:315: in __init__
    raise IOError("No such file or directory: %r" % _path)
E   OSError: No such file or directory: '/home/circleci/nltk_data/tokenizers/punkt/PY3'
```
which is odd, re-running the job fixed the problem.

So trying to mend it with:

```
try:
    nltk.data.find("tokenizers/punkt")
except (LookupError, OSError):
```
not sure why the Exception was different here. 

@sgugger, @LysandreJik 


Comments:
I'm going to merge this since the issue could be interfering with other PRs.
ImportError: cannot import name 'get_last_checkpoint'
Author: yuxuan2015
Pull request: no

from transformers.trainer_utils import get_last_checkpoint, is_main_process
ImportError: cannot import name 'get_last_checkpoint'

Comments:
HI there,

what's your transformers version ? `get_last_checkpoint` is available on master, so you should install from source to use it

hi @yuxuan2015 , the latest stable release of transformers (4.2.2) has no 'get_last_checkpoint' function, so if you installed via package manager you won't be able to use that function. like patil said, you need to install from source

I solved this error after reinstalling transformers from pip. The version of transformers I installed is 4.3.3

As mentioned in the `examples/readme.md` [here](https://github.com/huggingface/transformers/tree/master/examples#important-note), to run the examples, always install from source.

Closing this issue.