noamaanMulla-03/transformers-issues huggingface dataset:
Support for running Gpt-Neo 2.7B with 6 GB vram for inference
Author: arrmansa
Pull request: yes

# What does this PR do?
It adds functionality to allow gpt-Neo 2.7B to run in 6gb vram.

If it detects that some modules are on gpu and there is not enough vram, a dict called extrastorage is created which holds the data for model.transformer.h
These weights are loaded from ram to vram one at a time, reducing vram usage.

Expected speed is around 1 token/2s. (slower on the first run)

## Usage
1. Have between 5 and 9.5 Gb Vram

2. run -
```
model.eval().half().to("cpu")
model.transformer.wte.to("cuda")
model.transformer.wpe.to("cuda")
model.transformer.ln_f.to("cuda")
model.lm_head.to("cuda")
torch.cuda.empty_cache()
```
3. Use
`model.generate()
`or 
`model(**inputs)
`

## Motivation
Will become faster as ram->vram (pcie) bandwidth increases
Running larger models on consumer hardware is important

## Incomplete
I need some help with the documentation, also I'm not sure if `import copy` should be inside an if statement or not (line 769).

## Before submitting
 
* [x]  Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
       Pull Request section?
* [ ]  Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
       to it if that's the case.
* [ ]  Did you make sure to update the documentation with your changes? Here are the
       [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
       [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
* [ ]  Did you write any new necessary tests?
 
 
 ## Who can review?
 
 Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
 members/contributors who may be interested in your PR.


Models:
gpt-neo: @patil-suraj

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Problem with mT5 and the official Summarization notebook
Author: demegire
Pull request: no

## Environment info

- `transformers` version: 4.6.0
- Platform: Linux-5.4.109+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@patrickvonplaten @patil-suraj @sgugger 

## Information

I am using mT5-small on the [official summarization notebook](https://github.com/huggingface/transformers/tree/master/examples/pytorch). However when trained, the model gets nan loss values and outputs non-sense.

I made some changes to speed up the training such as loading 5% of the data, changing max input length to 256 from 1024 and the batch size to 8 from 16 however my settings work perfectly fine with t5-small and I get a high rouge score with sensible outputs and loss values. The problem seems to be mT5.

## To reproduce

Here is my [Colab notebook ](https://colab.research.google.com/drive/16-6yIHFQQ1Q8meVYqFn21Tw2eoG9wliU?usp=sharing)which you can see the output at the end.

TrainOutput(global_step=1276, training_loss=nan, metrics={'train_runtime': 340.4428, 'train_samples_per_second': 3.748, 'total_flos': 1196714720985600.0, 'epoch': 1.0, 'init_mem_cpu_alloc_delta': 1543725056, 'init_mem_gpu_alloc_delta': 1200707584, 'init_mem_cpu_peaked_delta': 0, 'init_mem_gpu_peaked_delta': 0, 'train_mem_cpu_alloc_delta': 10055680, 'train_mem_gpu_alloc_delta': 1203914240, 'train_mem_cpu_peaked_delta': 65540096, 'train_mem_gpu_peaked_delta': 4225469440})

## Expected behavior

Training loss should not be nan.


Comments:
disabling fp16 seems to solve the issue of nan loss, but I wouldn't call this issue closed because this doubles the training time :(

Hey @demegire,

Sadly MT5 doesn't really work with fp16. There are a bunch of issues regarding this problem...see:
- https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/5
- https://github.com/huggingface/transformers/issues/10830

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Reconstructing Tokens from Bert Embedding?
Author: patdflynn
Pull request: no

Sorry if this was posted before but I couldn't it after a few searches.

My goal is to take a sentence, run it through BERT, perturb the contextualized embeddings from the output of BERT, and reconstruct the sentence text.

I'm currently using 'bert-base-uncased' as my tokenizer and model and a perturbed output torch tensor with each token embedding size 768. How do I reconstruct the sentence text?

Thanks!

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!
Fixed a typo in readme
Author: MichalPitr
Pull request: yes

# What does this PR do?

Fixes a simple typo in the readme file from "pr" to "or".  



## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Prediction fails for certain batch sizes
Author: jglaser
Pull request: no

## Environment info

- `transformers` version: 4.6.1
- Platform: IBM open-ce / ppcle64
- Python version: 3.8.8
- PyTorch version (GPU?): 1.7.1 NVIDIA V100
- Tensorflow version (GPU?): N/A
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: distributed (but only 1 rank per process group)

### Who can help

@sgugger


## Information

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

Custom-implementation combining two BERT models with a multilayer perceptron that implements a regression on the last hidden layer outputs

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

a regression on pairs of protein sequence and molecule SMILES strings, plus binding affinity

## To reproduce

Steps to reproduce the behavior:

1. Call `trainer.predict()` on the encoded inputs
2. If the number of inputs is 1+`per_device_eval_batch_size`, the last batch has just one member and is treated as a scalar
3. the prediction loop fails with
``` 
File "../affinity_pred/infer_mpi.py", line 189, in main
    df_pred = predict(df)
  File "../affinity_pred/infer_mpi.py", line 180, in predict
    out = trainer.predict(x)
  File "/gpfs/alpine/world-shared/bip214/opence-env/lib/python3.8/site-packages/transformers/trainer.py", line 2065, in predict
    output = eval_loop(
  File "/gpfs/alpine/world-shared/bip214/opence-env/lib/python3.8/site-packages/transformers/trainer.py", line 2162, in evaluation_loop
    logits = self._nested_gather(logits)
  File "/gpfs/alpine/world-shared/bip214/opence-env/lib/python3.8/site-packages/transformers/trainer.py", line 2252, in _nested_gather
    tensors = distributed_concat(tensors)
  File "/gpfs/alpine/world-shared/bip214/opence-env/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 158, in distributed_concat
    concat = torch.cat(output_tensors, dim=0)
RuntimeError: zero-dimensional tensor (at position 0) cannot be concatenated
```

**Example**:

last two batches (`per_device_eval_batch_size` is 32)
```
[tensor([ 0.5737, -0.1372, -0.5283, -0.0641, -0.0641,  0.6353,  0.1035, -0.4148,
         0.2314, -0.3879, -0.4431, -0.3931, -0.2642,  0.5039, -0.4187,  0.0679,
         0.0679, -0.3167, -0.4783, -0.6724, -0.6724, -0.3257,  0.4922,  0.4922,
        -0.4189, -0.3652, -0.4468, -0.2358, -0.3696,  0.1646, -0.2004, -1.0234],
       device='cuda:0', dtype=torch.float16)]
[tensor(0.7144, device='cuda:0', dtype=torch.float16)]
RuntimeError('zero-dimensional tensor (at position 0) cannot be concatenated')
```

## Expected behavior

No exception

Comments:
Fix exception in prediction loop occurring for certain batch sizes
Author: jglaser
Pull request: yes

# What does this PR do?

Fixes #12349 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sgugger 


Comments:
Thanks, I can't push to your branch, so could you push an empty commit since circleCI decided to not run on your PR?
```
git commit --allow-empty -m "Trigger CI"
```
and then a push.

Ok, there are lots of failures that seem related to the PyTorch release. Could you rebase on master (I really want to make sure this does not break any example before merging)? Thank you.

> Ok, there are lots of failures that seem related to the PyTorch release. Could you rebase on master (I really want to make sure this does not break any example before merging)? Thank you.

Done, I must have based this off from a fork that was a few weeks old....

I'll be off now for the weekend, but the box "Allow edits by maintainers" is checked, so feel free to adapt as necessary.

We're all good, thanks for walking through this with me!
Add FlaxBigBird QuestionAnswering script
Author: thevasudevgupta
Pull request: yes

# What does this PR do?

This PR will add flax-bigbird QA script on `natural-questions` dataset.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed.

@patrickvonplaten 

Comments:
@patrickvonplaten, this PR is ready for review & merge (tested all the code after porting here).

Failing test is unrelated to this PR.

Awesome merging for the sprint - we'll fix bugs on the go as it's under `research_projects` :-)
Flax T5
Author: thevasudevgupta
Pull request: yes

# What does this PR do?

This PR will add T5 in Flax/Jax.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed.

@patrickvonplaten

Comments:
Exceptionally merging already given the time constraint of the Flax/JAX community week announcement. 
@patil-suraj @sgugger, I would be very happy if you could nevertheless take a look after merge so that I can correct suggestions in a follow-up PR before the next transformers release.

Is a jax/flax byT5 planned?

Interested in both byT5 and jax... torn.

You can already use ByT5 in jax/flax! Check the models [here](https://huggingface.co/models?filter=jax&search=byt5)

oh, my 🤗

I'm having trouble finding the Jax model training and architecture definition.

Is this just loading a byT5 model into a regular T5 inference scaffolding?

My aim is to experiment with the training / masking code.

Here is a test for FlaxByT5 that could help a bit: https://github.com/huggingface/transformers/blob/332a2458611751e7d9c4d7a21bc454299d50e160/tests/test_modeling_flax_t5.py#L432

Also we have `run_summarization` script in Flax that can be easily tweaked for any seq2seq task. And soon (Monday) we'll have FlaxT5 pretraining as well :-)
[examples] [distributed] process datasets.map only on main process in 
Author: stas00
Pull request: no

# 🚀 Feature request

Switch examples to process `datasets.map` only once on a main process and have the other processes wait, which would make pre-processing much faster. This is based on: https://huggingface.co/docs/datasets/processing.html?highlight=map#mapping-in-a-distributed-setting

@sgugger suggested to add a new context manager `training_args.main_process_first()` to make it simple.

context: https://discuss.huggingface.co/t/slow-processing-with-map-when-using-deepspeed-or-fairscale/7229/ (albeit the title is misleading, the issue the user experienced is with any distributed framework).


Comments:
[trainer] add main_process_first context manager
Author: stas00
Pull request: yes

This PR 
- [x] implements `main_process_first` context manager as discussed in https://github.com/huggingface/transformers/issues/12345
This make the `datasets` pre-processing much faster in the distributed environment as only one process is doing it and not all replicas too at once.
- [x] modifies `run_translation.py` example to use it as a model.
- [x] starts using `log.debug` - since now we have the new shiny `--log_level` trainer arg, so many too-much-information `log.info` can be switched to `log.debug`, and the user can run `--log_level debug` to generate a lot more info when debugging or when filing a bug report.

**Question: not sure what should be done on multi-node setups, since one may or may not use a shared filesystem.**

TODO:
- once merged replicate to other examples

**Kudos for the cool function name goes to @sgugger** 

Fixes: https://github.com/huggingface/transformers/issues/12345

@sgugger 


Comments:
So I won't lose it a magic perl to rewrite all other examples:

```
find examples -type f -exec perl -0777 -pi -e 's|^(\s+)(train_dataset = train_dataset.map\(.*?\))|x($1, $2)|msge; BEGIN {sub x {($p, $t) = @_ ; $t =~ s/^/    /msg; return qq[${p}with training_args.main_process_first(desc="train dataset map pre-processing"):\n$p$t] } }' {} \;
```
[examples] replicate the new `--log_level` feature to all trainer-based pytorch examples
Author: stas00
Pull request: no
Labels: name: Good First Issue

https://github.com/huggingface/transformers/pull/12276 introduced a new `--log_level` feature, which now allows users to set their desired log level via CLI or TrainingArguments. 

`run_translation.py` was used as a "model" for other examples.

Now we need to replicate this to all other Trainer-based examples under examples/pytorch/, the 3 changes are
1. importing datasets
2. using `training_args.get_node_log_level()`  and setting log_level in 3 modules
3. replacing `datasets` object name with `raw_datasets`, since otherwise we have a conflict with `datasets` the module.

and the relevant diff is [here](https://github.com/huggingface/transformers/pull/12276/files?file-filters%5B%5D=.py#diff-09777f56cee1060a535a72ce99a6c96cdb7f330c8cc3f9dcca442b3f7768237a)

and of course since we don't quite have extensive tests for examples, you can just test with a staple cmd from the corresponding README.md with `--log_level=error` and check that almost all logs are gone.

This is open to all.

And thank you.

Comments:
Can I take this?

Yes, of course! Thank you, @bhadreshpsavani 

@bhadreshpsavani, once https://github.com/huggingface/transformers/pull/12309 gets merged, please rebase your branch and rename
```
-    get_node_log_level()
+    get_process_log_level()
```

Thank you!

ok, it's merged now.

Thanks!

How can I contribute?

Please refer to https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md
[Examples] Replicates the new --log_level feature to all trainer-based pytorch
Author: bhadreshpsavani
Pull request: yes

# What does this PR do?
Fixes #12295 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Notes:
- The required changes are added for all the trainer-based examples except `run_generation.py` since it seems very different.
- Please let me know if any modification needed.

## Who can review?
@stas00  @sgugger 


Comments:
Also as mentioned earlier, let's add:
```
    transformers.utils.logging.enable_default_handler()
    transformers.utils.logging.enable_explicit_format()
```
everywhere they are missing - I see there are quite a few places.

Thank you!

@bhadreshpsavani, we forgot to take care of the template `templates/adding_a_new_example_script/\{\{cookiecutter.directory_name\}\}/run_\{\{cookiecutter.example_shortcut\}\}.py`
if you don't mind adding this change there too in another PR.
 
Thank you!


Sure I will add it,
Ya, i totally forgot!
[Examples] Update Example Template for `--log_level` feature
Author: bhadreshpsavani
Pull request: yes

# What does this PR do?
Fixes #12295


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. Discussed on this [PR](https://github.com/huggingface/transformers/pull/12359)
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@stas00 @sgugger

Comments: