noamaanMulla-03/transformers-issues huggingface dataset:
Running gpt-neo 2.7B with less than 13GB of system memory like Colab
Author: finetunej
Pull request: no
Labels: name: Feature request

# 🚀 Feature request

A way to conserve regular system memory while loading large models. On systems without much system memory, the process crashes because it tries to load both the weight checkpoint and the model into system memory. In the case of gpt-neo 2.7B, this is even worse because the checkpoint is offered only in float32 format, taking up twice as much space.

## Motivation

Free Colab often offers GPUs with 16GB of VRAM but only about 13GB of RAM. This increases the barrier of entry for people to play with these kinds of models. I have found a way to load models in this kind of situation, but it is not general or well integrated. By posting it here, I hope that a more general implementation can be built at some point.

This would also help #11271.

## Your contribution

It is possible to work around this by loading the checkpoint directly into VRAM, casting it to float16, instantiating the model in VRAM and only then applying the weights from the checkpoint.

To do this, first a patch has to be applied to src/transformers/models/gpt_neo/modeling_gpt_neo.py. This is based on the 4.5.1 release.

    703c703
    <         self.h = nn.ModuleList([GPTNeoBlock(config, layer_id=i) for i in range(config.num_layers)])
    ---
    >         self.h = nn.ModuleList([GPTNeoBlock(config, layer_id=i).half().cuda() for i in range(config.num_layers)])
    890,891c890,891
    <         self.transformer = GPTNeoModel(config)
    <         self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
    ---
    >         self.transformer = GPTNeoModel(config).half().cuda()
    >         self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False).half().cuda()

This causes space for the biggest part of the model to be allocated directly on the GPU, which has more space in the free Colab scenario. It also moves the other parts of the model to GPU. Now the model can be instantiated like this:

    from transformers.file_utils import cached_path, WEIGHTS_NAME, hf_bucket_url
    model_name = "EleutherAI/gpt-neo-2.7B"
    archive_file = hf_bucket_url(model_name, filename=WEIGHTS_NAME)
    resolved_archive_file = cached_path(archive_file)
    checkpoint = torch.load(resolved_archive_file, map_location="cuda:0")
    for k in checkpoint.keys():
      checkpoint[k] = checkpoint[k].half()
    model = GPTNeoForCausalLM.from_pretrained(model_name, state_dict=checkpoint).half().to("cuda")
    for k in list(checkpoint.keys()):
      del checkpoint[k]

Comments:
How to load weights from a private server?
Author: hwijeen
Pull request: no

Hi, thank you for the great library!

I am trying instantiate a model with weights uploaded on my private server.  By looking at [`is_remote_url`](https://github.com/huggingface/transformers/blob/8780caa388c7b2aa937454ed96bcdd3f097f851d/src/transformers/modeling_utils.py#L1011) function, it seems that transformers supports loading from a private server, but it seems a bit tricky.
```python
BertModel.from_pretrained('http://my-server/my-bert-cased/pytorch_model.bin') # cannot not find config
BertModel.from_pretrained('http://my-server/my-bert-cased/config.json') # finds config, but cannot find model weights
BertModel.from_pretrained('http://my-server/my-bert-cased', config='http://my-server/my-bert-cased/config.json') # works!
```

Although the third one works, it is cumbersome as I need to download config from private server to my local machine beforehand.

I would appreciate it if someone could share or point to a better way!



Comments:
My workaround was to manually download checkpoints using `cached_file` function to local, and instantiate a model from the download file using `from_pretrained`.

Yes that's indeed the preferred workaround. Thanks!
modify double considering special tokens in `language_modeling.py`
Author: taepd
Pull request: yes

# What does this PR do?

in `class TextDatasetForNextSentencePrediction`, double considering `self.tokenizer.num_special_tokens_to_add(pair=True)`

so, i remove `self.block_size`, and add parameter for `def create_examples_from_document` like `class LineByLineWithSOPTextDataset` do



Fixes # (issue) double considering special tokens in `language_modeling.py`




## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Add early stopping to trainer_tf.py
Author: KMFODA
Pull request: yes

## Summary
This PR adds the early stopping feature to trainer_tf.py.

## Related Issues
Alongside #4186 should close #4894.

## Who can review?

@sgugger & @BramVanroy.


Comments:
Thanks @KMFODA!

I'm not really in favor to do this manually as there are already a Keras callbacks taking care of this https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping that can monitor multiple values on which to base his early stop.

@KMFODA if you want to add the feature to add Keras callbacks handling, it will be more than welcome :) As it is part of the features we plan to add in a near future.

Seeing that I have close to no experience with TF, I won't be able to review this.

Not a problem. @jplu I agree I prefer using Keras callbacks I just only have experience using it with Keras’s model.fit function. I’ll think and experiment with how to use it in a custom built TF model. Hopefully if successful, it should be fairly simple then to add early stopping based on custom metrics rather than just validation loss.

I've just pushed the latest changes to trainer_tf.py that will use Keras's callbacks for early stopping rather than the manual solution I had initially submitted. Setting the callback in the `TFTrainer` function using a command such as this:

`callbacks = [EarlyStopping(monitor='loss', patience=1, verbose=1)]
`

will monitor the training loss and stop the model at the first epoch which fails to improve on the best training loss metric. Using the monitor variable we can also select metrics, other than the training loss, to carry out the early stopping on.

Hi all, anything more I can do to help get this merged?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Bump @LysandreJik @jplu 

We're moving away from the TFTrainer to fully integrate with Keras, so we won't add new functionality to the TFTrainer.

> We're moving away from the TFTrainer to fully integrate with Keras, so we won't add new functionality to the TFTrainer.

Alright, sounds good, This can be closed then?

I'll let the original author close it :-)
[Trainer] fix the placement on device with fp16_full_eval
Author: stas00
Pull request: yes

* `do_train` isn't a reliable arg - it is not required to run `train), so for now add a workaround for the `fp16_full_eval` case to place the model on device if `train()` was called w/o `do_train=True` being passed to the Trainer args.
* while at it fix the `deepspeed` case, now that it's used in eval too, it should never be put on device
* Also `args = self.args` to make the code easier to read/shorter - there is a lot of it in `train`.

Fixes: https://github.com/huggingface/transformers/issues/11200#issuecomment-822631511

@sgugger 

Comments:
We probably need to re-think the "placement on device" logic. And to do it explicitly in each stage and in `__init__` only in those special cases where it's absolutely required.
Load checkpoint without re-creating the model
Author: sgugger
Pull request: yes

# What does this PR do?

This PR avoids recreating the model when loading a checkpoint in the Trainer. As mentioned in #11294, the current loading messes up the model a user passed when some weights are frozen, which will yield to unexpected OOM errors.
A test is also added checking the frozen parameters are kept frozen.

Fixes #11294 and probably #11317

Comments:
run_ner.py example MobileBERT FP16 returns nan loss
Author: tblattner
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.5.1
- Platform: Linux-5.8.0-44-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes (RTX 2080 Ti)
- Using distributed or parallel set-up in script?: No


### Who can help
@sgugger   @stas00  @patil-suraj

## Information

Model I am using MobileBERT:

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (give the name) conll2003
* [ ] my own task or dataset: (give details below)

## To reproduce
Using the example: https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_ner.py

Steps to reproduce the behavior:

1. Add training_args.fp16 = True to main() after initializing training_args
2. parameters to run_ner: 
```
--model_name_or_path
google/mobilebert-uncased
--dataset_name
conll2003
--output_dir
/path/to/output
--do_eval
--do_train
--do_predict
```
3.loss will return nan

First observed nans popping up from the encoder within the forward call in the MobileBertModel class: 
https://huggingface.co/transformers/_modules/transformers/modeling_mobilebert.html



## Expected behavior

When running without FP16, the model trains as expected. Other models that I have tested did not have this issue and converge well with fp16 enabled: RoBERTa, BERT, and DistilBERT.

Comments:
It looks like mobileBERT was pretrained on TPUs using bfloat16, which then often result in NaNs when using FP16 for further fine-tuning (see #11076 or #10956). You'll be best off training in FP32 or use another model compatible with FP16.

Makes sense! That's interesting that affects the training on GPUs! I will pass this info on to my colleague who deals with reproducibility! And for now I shall stick with FP32 when fine-tuning the MobileBERT model!

Many thanks for the reply!

> You'll be best off training in FP32 or use another model compatible with FP16.

And at some point we should also add `--bf16` mode to Trainer, for those who want to do finetuning and inference on hardware that supports it . e.g. high-end Ampere RTX-3090 and A100 should already support it, and of course TPU v2+. 

Does it make sense?

FYI, `bf16` AMP is being discussed here: https://github.com/pytorch/pytorch/issues/55374
RAG with RAY implementation: Ray workers memory slowly increase over time.
Author: shamanez
Pull request: no

I fine-tune the RAG with RAY workers from 50 000 steps. When I checked the MEM% with the top command, I can see the memory consumption keep growing slowly. Usually, it should use around 20GB. After 50000 steps it raises up to 24 GB. 

This could eventually lead to crash the system with OOM error. I did a background check and found the Redis server usually keeps increasing its memory consumption. 

So is it ok to set a value  **object_store_memory**?


I get something similar to this... (found this from the [issue](https://github.com/ray-project/ray/issues/10431) )

![image](https://user-images.githubusercontent.com/16892570/115200801-b5845d00-a148-11eb-8447-b0c71b33b937.png)


@lhoestq  @amogkam 

Comments:
I think I solved the memory leakage issue.  In my case, I simply wanted to update the index parameters in **self.retriever  object**. So I used **def set_index function**.  But I observe for some reason ray works completely cannot flush out the old index and its related objects.

So now when I want to update it, I delete the self.retriever object and re-initialize it. 

```
class RayRetriever:
    def __init__(self):
        self.initialized = False

    def create_rag_retriever(self, config, question_encoder_tokenizer, ctx_encoder_tokenizer,generator_tokenizer, index):
        if not self.initialized:
            self.retriever = RagRetriever(
                config,
                question_encoder_tokenizer=question_encoder_tokenizer,
                ctx_encoder_tokenizer=ctx_encoder_tokenizer,
                generator_tokenizer=generator_tokenizer,
                index=index,
                init_retrieval=False,
            )
            self.initialized = True

    def init_retrieval(self):
        self.retriever.index.init_index()

    def set_index(self,index):
        self.retriever.index=index #with this new index class all the paramters in HFindex class get updated
   

    def clear_object(self):  #we can call this first delete all thing things and again call create_rag_retriever
        del self.retriever
        self.initialized = False
     

    def retrieve(self, question_hidden_states, n_docs):

        doc_ids, retrieved_doc_embeds = self.retriever._main_retrieve(question_hidden_states, n_docs)
        doc_dicts= self.retriever.index.get_doc_dicts(doc_ids)

        return doc_ids, retrieved_doc_embeds,doc_dicts
``````
Update to use datasets remove_cloumns method
Author: sgugger
Pull request: yes

# What does this PR do?

This PR updates the command used in the `Trainer` to drop the columns not used by the model to take advantage of the latest (well not so latest since it landed in datasets 1.4.0) `remove_columns` method. This adds the advantage of not modifying in place the dataset (as was done before) so the user does not have unexpected changes in their original datasets.

In consequence, the little hack needed in the question answering examples is now unnecessary.

Comments:
Add an error message for Reformer w/ .backward()
Author: forest1988
Pull request: yes

# What does this PR do?

Add an error message that fires when Reformer is not in training mode, but one runs .backward().

Fixes #10370


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

- longformer, reformer, transfoxl, xlnet: @patrickvonplaten


Comments:
@forest1988 thank you for your PR! Patrick is off for a couple of weeks, but will review once he's back. Thank you for your patience!

@LysandreJik 
Thank you for letting me know about that! That's totally fine!
How to Paraphrase with GPT2?
Author: BigSalmon2
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->



Comments:
You should rather use a seq2seq model for paraphrasing like T5 or BART. But if you want to do it using GPT-2 then maybe you can use this format

`input: input_text paraphrase: parahrase_text`

 while training,  set attention mask to 0 on the paraphrased text

and when generating just pass `input: input_text paraphrase: `  and sample till the `eos` token

Thank you. I'll be sure to try that! I know there are datasets online that I can use, but are there edits I should make to ensure that I get a suitable output? I've read something about cheating, and I want to avoid that. (I'm sorry if I am using the wrong terminology)

Can we use `T5` or `BART` like we would `GPT-2`?

hi @shamoons , yes you can, have a look at this https://madewithml.com/projects/1094/paraphrase-any-question-with-t5-text-to-text-transformer/

This looks like it’s only for questions. What about arbitrary sentences?

> This looks like it’s only for questions. What about arbitrary sentences?

You can do it with arbitrary sentences as well, but you'll need to fine-tune it yourself.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> hi @shamoons , yes you can, have a look at this https://madewithml.com/projects/1094/paraphrase-any-question-with-t5-text-to-text-transformer/
This url is gone.


@kingglory 

This is the [url](https://github.com/ramsrigouthamg/Paraphrase-any-question-with-T5-Text-To-Text-Transfer-Transformer-) for the repo of that project.

Also there are quite a few paraphrase models on the [hub](https://huggingface.co/models?search=paraphrase) that you can try
Honor contributors to models
Author: sgugger
Pull request: yes

# What does this PR do?

This PR mentions by HF username the person who added a given model in each doc file, and updates the template so this keeps being consistently done.

A few persons are missing, tagging there below. If you are one of the persons tagged, it would be great if you could create a HF account and report your HF username here so we can properly attribute the model addition to you :-)

- Bert japanese: @singletongue
- CamemBert: @louismartin
- DeBerta (v1 and v2): @BigBird01
- ProphetNet and XLM-ProphetNet: @qiweizhen


Comments:
Hi, thanks for this update!
I guess my huggingface username is [`camembert`](https://huggingface.co/camembert) ...?

> # What does this PR do?
> This PR mentions by HF username the person who added a given model in each doc file, and updates the template so this keeps being consistently done.
> 
> A few persons are missing, tagging there below. If you are one of the persons tagged, it would be great if you could create a HF account and report your HF username here so we can properly attribute the model addition to you :-)
> 
> * Bert japanese: @singletongue
> * CamemBert: @louismartin
> * DeBerta (v1 and v2): @BigBird01
    => Pengcheng He: @BigBird01
> * ProphetNet and XLM-ProphetNet: @qiweizhen

Thanks I just replied my name and account inline.

Hi @BigBird01, there is no huggingface account with the user name BigBird01 (I'm not talking about the GitHub account but the account on [huggingface.co](https://huggingface.co/)).

Thanks! It’s DeBERTa😊, DeBERTa (Pengcheng He) (huggingface.co)<https://huggingface.co/DeBERTa>

Thanks!
Pengcheng



Thank you for the mentioning!
The HF account for `bert-japanese` models is [`cl-tohoku`](https://huggingface.co/cl-tohoku).

Failure is spurious, so merging. @qiweizhen if you create (or already have) a HF account, I can add you in a followup PR.
Fix token_type_ids error for big_bird model.
Author: wlhgtc
Pull request: yes

I run the follow code, but get strange outputs, the `token_type_ids` is shorter than `input_ids`:
```python
def demo():
    model_name = "./resources/bigbird-roberta-base"
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    text = 'With power of science ,'
    max_length = 10
    encoded_tokens = tokenizer.encode_plus(
        text=text,
        add_special_tokens=True,
        max_length=max_length,
        truncation=True if max_length is not None else False,
        return_tensors=None,
        return_offsets_mapping=tokenizer.is_fast,
        return_attention_mask=False,
        return_token_type_ids=True,
        return_special_tokens_mask=True,
    )
    print(encoded_tokens)
```
![CleanShot 2021-04-21 at 19 03 13@2x](https://user-images.githubusercontent.com/16603773/115543730-4b98be80-a2d4-11eb-8830-c1a2a20f4d84.png)
Seem it miss the `create_token_type_ids_from_sequences` method, so it run code in [here](https://github.com/huggingface/transformers/blob/95dab34d5588fb155dfed8293ac2fbb1217a95a7/src/transformers/tokenization_utils_base.py#L2665) as default. So I copy the method in BERT to fix it.

I know maybe the big_bird model doesn't need `token_type_ids`, but we also have to make sure it return the right result if we set `return_token_type_ids = True`. 

Comments:
@sgugger 

I think the BigBird QA model actually uses 16 token type ids (https://huggingface.co/google/bigbird-base-trivia-itc/blob/main/config.json), but this is an exception I think and the default case is 2 token type ids => so this looks good to me.

 @vasudevgupta7 what do you think?

Yes, both base & large pre-trained checkpoints accept 2 token type ids while only trivia-qa checkpoint accepts 16 token type ids. So, this should be good.