noamaanMulla-03/transformers-issues huggingface dataset:
replacing apex.normalization.FusedLayerNorm with torch.nn.LayerNorm
Author: stas00
Pull request: no

It seems that time has arrived to drop `apex.normalization.FusedLayerNorm` in favor of `torch.nn.LayerNorm` 

1. the latter was ported more than a year ago from apex  https://github.com/pytorch/pytorch/pull/27634 (around pt-1.4)
2. it's faster than the apex according to my benchmarks https://github.com/pytorch/pytorch/issues/37713#issuecomment-753434842 (**33% faster on rtx-3090!**, 10% faster on gtx-1070)

**but note:** this same benchmark run here https://github.com/pytorch/fairseq/issues/2012#issuecomment-622607286 on V100 reports the opposite  - that the native is slower (pt-1.5). So it might help to run this very quick benchmark on other cards and compare. In particular if you have access to V100 please report back the findings at this thread: https://github.com/pytorch/pytorch/issues/37713

The main reason for this need is that `apex.normalization.FusedLayerNorm` is buggy (corrupts memory) when it comes to switching devices, which is done a lot under Model Parallel. https://github.com/NVIDIA/apex/issues/1022
With  `apex.normalization.FusedLayerNorm` things fail a lot under MP and requires sticking `torch.cuda.set_device(id)` in many many places as a workaround :( Since this overload is used at model's init time it's not possible to not use it under MP as the latter gets activate after model's init.

I will use that workaround if you find out that apex is faster still on some important-to-consider hardware. And, of course, in that case please report back to the pytorch team so that they could fix it. Otherwise apex support is pretty much no more and it's just a matter of time before apex will be unusable.

The models that need that change are bart/fsmt/prophetnet.

@patrickvonplaten, @LysandreJik


Comments:
I'm good with changing to `torch.nn.LayerNorm`. At @stas00 - do you know what the advantage of `apex.normalization.FusedLayerNorm` is supposed to be? Why did we add `apex.normalization.FusedLayerNorm` in the first place?

Prior to about a year ago, `apex.normalization.FusedLayerNorm` was faster than `torch.nn.LayerNorm`, but then the former got ported to native `torch.nn.LayerNorm`, and now the native appears to be faster - at least the 2 cards I have experimented with. I checked with pt-1.4 .. pt-1.8.dev

If you have other than gtx-1070/rtx-3090 cards which I benchmarked with please run that benchmark and see if it stands true for other cards: https://github.com/pytorch/pytorch/issues/37713#issuecomment-753434842
it only takes a few seconds if you have apex installed already. To install apex:

```
git clone https://github.com/NVIDIA/apex
cd apex
rm -rf build
pip install  --global-option="--cpp_ext" --global-option="--cuda_ext"  .
```

The benchmark measures and reports a total run time, so the smaller the numbers the faster it is.

If you do run the benchmarks please post your results at https://github.com/pytorch/pytorch/issues/37713 so that it can be seen whether it's safe to drop `apex.normalization.FusedLayerNorm` based on hard data and not anecdotal info.

Thank you.
Improve documentation coverage for Bertweet
Author: Qbiwan
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #9035
@sgugger  added docs for Bertweet



## Before submitting

- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Comments:
Thanks @Qbiwan!
replace apex.normalization.FusedLayerNorm with torch.nn.LayerNorm
Author: stas00
Pull request: yes

This PR proposes to drop `apex.normalization.FusedLayerNorm` in favor of faster `torch.nn.LayerNorm`. 

1. For performance and background details please see the discussions in https://github.com/huggingface/transformers/issues/9377
2. It's also needed for https://github.com/huggingface/transformers/pull/9384 since `apex.normalization.FusedLayerNorm` corrupts data under model parallel https://github.com/NVIDIA/apex/issues/1022

Fixes: #9377

@LysandreJik, @sgugger, @patrickvonplaten 


Comments:
Merging since it's blocking #9343 .
added head_mask/decoder_head_mask for BART
Author: stancld
Pull request: yes

Description:
This PR adds `head_mask` and `decoder_head_mask` for BART PyTorch implementation according to BERT implementation.

Motivation:
According to HuggingFace's websites "There is a growing field of study concerned with investigating the inner working of large-scale transformers like BERT (that some call “BERTology”)." This PR enables to mask attention heads in encoder and decoder models exactly like for BERT. This PR thus creates an opportunity to study the importance of attention heads in encoder-decoder BERT-like model.

Reviewer: @patrickvonplaten 


Comments:
[examples/language-modeling] Add dataset download instructions
Author: stas00
Pull request: yes

I had to hunt for instructions to get the dataset used in this set of examples, so this PR proposes to add them to README.md.

@patrickvonplaten


Comments:
Isn’t this one simply on the [HuggingFace datasets hub](https://huggingface.co/datasets) by the way?

Op wo 30 dec. 2020 om 07:50 schreef Stas Bekman <notifications@github.com>

> I had to hunt for instructions to get the dataset used in this set of
> examples, so this PR proposes to add them to README.md.
>
> @patrickvonplaten <https://github.com/patrickvonplaten>
> ------------------------------
> You can view, comment on, or merge this pull request online at:
>
>   https://github.com/huggingface/transformers/pull/9356
> Commit Summary
>
>    - [examples/language-modeling] Add dataset download instructions
>
> File Changes
>
>    - *M* examples/language-modeling/README.md
>    <https://github.com/huggingface/transformers/pull/9356/files#diff-28c51ae2110e09a5e495a1748a8ecc3c2e3cb2f7a244c000c67a9d8c4c37adf6>
>    (9)
>
> Patch Links:
>
>    - https://github.com/huggingface/transformers/pull/9356.patch
>    - https://github.com/huggingface/transformers/pull/9356.diff
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/9356>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABYDIHJOJFTITANOLIIFNY3SXLESTANCNFSM4VOADSUA>
> .
>


Sure, let's have the equivalent instructions to retrieve that from HF `datasets`. It doesn't really matter where it comes from as long as it doesn't require the user to go and search for it.  

FWIW, I went to https://huggingface.co/datasets and:
1. couldn't find it. That is I did find `wikitext`, but how do I know that it's the same as `wikitext-2-raw-v1` that the script expects - it seems to be very specific.
2. it gives me no instructions on how to download it in the format the script expects it in.

p.s. it looks like  Email replies do not support Markdown.


There is no need to download the data manually with the new scripts, it is done automatically by the datasets library. So this should not be added in my opinion.

oh, ok, I guess I didn't pay attention to the command line being changed and assumed that I needed to get the dataset first.

I stand corrected. Thank you for your feedback.
[logging] autoflush
Author: stas00
Pull request: yes

This PR proposes to:

* auto-flush `transformers` logging 

When using logging for tracing signals from different parts of the code and which could be mixed with print debug this aids to get all the logging events synchronized. 

I don't think this change will introduce any performance impacts.

If it helps someone here is the code I used to sync `transformers` logging with various other debug prints.

I was porting bart to MP and I needed to trace that the device switching happens correctly and I added a bunch of `logger.info` calls inside `modeling_bart.py` and also had some other helpers `print` debug messages which weren't logger based:

```

# auto flush std streams
from sys import stdout, stderr
def stdout_write_flush(args, w=stderr.write): w(args); stderr.flush()
def stderr_write_flush(args, w=stderr.write): w(args); stderr.flush()
stdout.write = stdout_write_flush
stderr.write = stderr_write_flush

from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig

import logging
import transformers.utils.logging
import transformers.models.bart.modeling_bart

# I wanted a shorter simpler format
handlers = transformers.utils.logging._get_library_root_logger().handlers
for handler in handlers:
    formatter = logging.Formatter("[%(funcName)s] %(message)s")
    handler.setFormatter(formatter)

transformers.models.bart.modeling_bart.logger.setLevel(transformers.logging.INFO)

# then all the model creation and generate() goes next
```

@LysandreJik, @sgugger, @patrickvonplaten


Comments:
Fix TF Longformer
Author: jplu
Pull request: yes

# What does this PR do?

This PR aims to fix the TF Longformer version in order to make it graph compliant. As seen offline with @patrickvonplaten `all_global_attentions` now is added in the output when `output_attentions=True`. The global attentions are filled with zeros in case `is_global_attn` is False (see line 897 in `TFLongformerSelfAttention`.

# Fix issue
#9333 


Comments:
I have already ran the slow tests as well and they all pass!
Fix  TF Funnel
Author: jplu
Pull request: yes

# What does this PR do?

This PR fixes Funnel to make it full graph compliant. Even though all the slow/quick tests are passing and got similar results with few experiements, @sgugger I would appreciate that you thoroughly look at the changes in order to be sure no bugs have been introduced.


Comments:
@LysandreJik feel free to merge if it looks ok for you and if @sgugger approves the last fix on `pooled_hidden`.
[Good first issue] Documentation links in older docs versions
Author: LysandreJik
Pull request: no
Labels: name: Documentation

name: Good First Issue

# 🚀 Feature request

This is a documentation request in order to make it easier to find corresponding examples in the documentation. 

Good first issue if you want to get acquainted with the docs and how to build docs using Sphinx!

## Current issue

Here's the issue: currently, if one goes to an older documentation version to check the "examples" page, for example, [v2.6.0](https://huggingface.co/transformers/v2.6.0/examples.html), all links point towards the `master` branch. 

For example, the link towards `run_tf_glue.py` is the following: https://github.com/huggingface/transformers/blob/master/examples/run_tf_glue.py

As this points towards the `master` branch, it is prone to breaking as files can (and probably will) be moved around as versions come out. It is the case for this example, as the `run_tf_glue.py` script is not in `examples/` anymore, but in `examples/text-classification/`. 

I think we need a way to ensure that all links point toward their appropriate version, and the easiest would be to point to a given tag. Since we're looking at the version `v2.6.0`, it makes sense to point towards the tag v2.6.0: https://github.com/huggingface/transformers/blob/v2.6.0/examples/run_tf_glue.py

This way links get frozen in time and redirect to actual files corresponding to their description and behaviour as stated in the docs.

## Resolution

I believe the easiest change would be to use sphinx variables in order to do this. Probably either [rst_epilog](https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-rst_epilog) or [rst_prolog](https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-rst_prolog) could be useful here.

Some useful links: [rst_epilog](https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-rst_epilog), [rst_prolog](https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-rst_prolog)

Comments:
Hi, has anyone picked this up? I can give it a go in the coming week or two when I have some let off from releases at work if no-one is doing this. 

Nobody has picked this up yet, would love to see such a contribution!

Awesome, have our last release till year end freeze this week at work.  I'll get in there afterwards. Would like to learn about some of the libraries involved in this project this seems like a good intro.
Fix documentation links always pointing to master.
Author: sugeeth14
Pull request: yes

# What does this PR do?

This PR fixes documentation issue. The issue is that hyperlinks pointing to code file in the docs always point to the master.
Reference issue : https://github.com/huggingface/transformers/issues/7988

I found [extlinks](https://www.sphinx-doc.org/en/master/usage/extensions/extlinks.html) more appropriate than [rst_epilog
](https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-rst_epilog) as mentioned in the original issue because hyperlink replacement is not possible with `rst_epilog` 
Stackoverflow issue reference : https://stackoverflow.com/questions/1227037/substitutions-inside-links-in-rest-sphinx

Although this fixes the issues as discussed I am unable to fix the  issue  in markup files which are https://huggingface.co/transformers/master/contributing.html and https://huggingface.co/transformers/master/notebooks.html
Please suggest if it is possible to do that as well. 


Fixes https://github.com/huggingface/transformers/issues/7988

@sgugger @LysandreJik 


Comments:
Thanks for merging @LysandreJik but I want to know why the errors occurred when I ran `make style ` as I see in https://github.com/huggingface/transformers/pull/9217#discussion_r548898170
Reorganize examples
Author: sgugger
Pull request: yes

# What does this PR do?

This PR reorganizes the examples folder by splitting it in two:
- `examples` that stay in this folder are the base example scripts maintained with the state of the library, expected to work on master. We accept PRs on them and will try our best to fix issues.
- `research-projects` are (often) more complex examples that we don't really maintain. They work on a specific version of the library  (sometimes even a specific commit). We don't accept PRs on them except minor typo fixes *or* PRs from the original authors that want to bring an update to those scripts. Issues opened for those are probably less efficient than directly contacting the authors.

Each example/research project lives in a folder of its own, with its particular requirements in a `requirements.txt` file (instead of a global requirements file as before).

The seq2seq subfolder is less organized than the others, so I did my best to split its research project part from its example part. I made sure all tests are passing and duplicated the needed files, but @stas00 and @patil-suraj please tell me if you something obvious that I missed. We will leave the research-project part as is and clean a bit more the part in the examples in other PRs.



Comments:
another bit - `examples/seq2seq/test_data` is also used by `research_projects/seq2seq` - perhaps symlink? 

Maybe a hard copy in that case, just in case the data changes/moves on the examples side.

> Maybe a hard copy in that case, just in case the data changes/moves on the examples side.

Then need to check which specific sub-dirs are needed - if I'm not mistaken it's only `test_data/wmt_en_ro/`

I'd still use a symlink to avoid git repo bloat and this can always be easily fixed if there is a divergence down the road.

Does this mean we no longer explicitly support pytorch_lightning? 

> Does this mean we no longer explicitly support pytorch_lightning?

I would rather like it if we ditch the custom transformer trainer and just use lightning.
[examples/text-classification] Fix a bug for using own regression dataset
Author: forest1988
Pull request: yes

# What does this PR do?

This PR is to fix https://github.com/huggingface/transformers/issues/9393

Fix a bug in `run_glue.py` so that it can be used for our own dataset of regression tasks.

close #9393 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sgugger
Thank you for checking the issue and giving the comment.



Comments:
@sgugger 
Thank you for checking and merging this PR!
Put back LXMert example
Author: sgugger
Pull request: yes

# What does this PR do?

During the example reorganization, LXMert seems to have slipped into the cracks and got accidentally deleted. This PR puts it back.

Fixes #9309

Comments:
Anyway, I doubt Lxmert is currently not supported for customized/ personal datasets (nlp+images, images are harder to prepare). See issues relevant to feature extraction in [Lxmert](https://github.com/airsplay/lxmert#faster-r-cnn-feature-extraction), for example, [issue#79](https://github.com/airsplay/lxmert/issues/79), [issue#86](https://github.com/airsplay/lxmert/issues/86).
[docs] outline sharded ddp doc
Author: stas00
Pull request: yes

This PR provides an initial outline of HF Trainer integration starting with ZeRO. We have fairscale's Sharded optimizer/gradients supported already and deepspeed is coming 

We won't merge this until fairscale merged all the required fixes and released a new version, but I thought it'd be good to start the doc going so it's ready when fairscale is ready.

I hope to submit a deepspeed integration shortly as well, so we will extend it then with deepspeed info. edit (https://github.com/huggingface/transformers/pull/9211)

@sgugger 

Comments:
Why was DataCollatorForNextSentencePrediction removed ?
Author: dwarfer7634
Pull request: no

# 🚀 Feature request

I want to ask a question about why DataCollatorForNextSentencePrediction was removed.
That class was implement in the pull request down below.
https://github.com/huggingface/transformers/pull/6572
It was so useful for me.
But, this feature is not included in the latest version.
Do anyone know why it was removed?

Or, is there any alternative features?

## Motivation

I need NSP feature to cunduct complete pre-training. 



Comments:
This class is not necessary anymore: it was the same as `DataCollatorForLanguageModeling` while keeping the `nsp_labels` but  `DataCollatorForLanguageModeling` will keep any extra things (like `nsp_labels`) you pass to it. So you can just replace it with `DataCollatorForLanguageModeling`.

Thank you for your quick reply. 
Do you mean you just have to use TextDatasetForNextSentencePrediction before DataCollatorForLanguageModeling to conduct NSP?

Yes.

That makes sense.
Thank you very much for your help!