Vectorized Numpy to Torch based Functions for SpecAugment
Author: 01-vyom
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #10459 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@patrickvonplaten

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Following are the average test results for the `_compute_mask_indices` function when ran 100 times. Each X.1 subtest case is calculated with `attention_mask = None` and each X.2 subtest case is calculated with `attention_mask` calculated with the following code:
```
attention_mask = torch.ones((batch_size, sequence_length), device=torch_device, dtype=torch.long)
attention_mask[:, -sequence_length // 2 :] = 0
```
1) Test - 100 times
batch_size = 4
sequence_length = 60
mask_prob = 0.5
mask_length = 1
Test **1.1** - 
Result - seconds 
New Code GPU: 0.002872414588928223
New Code CPU: 0.0006633639335632324
Old Code: 0.0003826594352722168
Test **1.2** - 
Result - seconds
New Code GPU: 0.002973439693450928
New Code CPU: 0.0006422805786132813
Old Code: 0.0004153728485107422

2) Test - 100 times
batch_size = 100
sequence_length = 60
mask_prob = 0.5
mask_length = 1
Test **2.1** - 
Result - seconds 
New Code GPU: 0.0663988971710205
New Code CPU: 0.014422652721405029
Old Code: 0.008053600788116455
Test **2.2** - 
Result - seconds
New Code GPU: 0.06568058252334595
New Code CPU: 0.01404146671295166
Old Code: 0.008796172142028809

3) Test - 100 times
batch_size = 1000
sequence_length = 60
mask_prob = 0.5
mask_length = 1
Test **3.1** - 
Result - seconds 
New Code GPU: 0.6623778533935547
New Code CPU: 0.14311392545700075
Old Code: 0.08917582988739013
Test **3.2** - 
Result - seconds
New Code GPU: 0.6566315603256225
New Code CPU: 0.13569485664367675
Old Code: 0.08646429538726806

4) Test - 100 times
batch_size = 4
sequence_length = 1000
mask_prob = 0.5
mask_length = 1
Test **4.1** - 
Result - seconds 
New Code GPU: 0.0031879472732543944
New Code CPU: 0.0013749027252197266
Old Code: 0.00248842716217041
Test **4.2** - 
Result - seconds
New Code GPU: 0.0031322765350341795
New Code CPU: 0.0010571050643920898
Old Code: 0.0015622496604919434

5) Test - 100 times
batch_size = 4
sequence_length = 60
mask_prob = 0.5
mask_length = 4
Test **5.1** - 
Result - seconds 
New Code GPU: 0.003424525260925293
New Code CPU: 0.0008220672607421875
Old Code: 0.0003489851951599121
Test **5.2** - 
Result - seconds
New Code GPU: 0.0034962940216064454
New Code CPU: 0.0007469034194946289
Old Code: 0.0003824186325073242

6) Test - 100 times
batch_size = 4
sequence_length = 1000
mask_prob = 0.5
mask_length = 4
Test **6.1** - 
Result - seconds 
New Code GPU: 0.003502027988433838
New Code CPU: 0.0014672994613647461
Old Code: 0.0017711663246154786
Test **6.2** - 
Result - seconds
New Code GPU: 0.0034971165657043455
New Code CPU: 0.0011277437210083009
Old Code: 0.0011361241340637207

7) Test - 100 times
batch_size = 128
sequence_length = 1000
mask_prob = 0.5
mask_length = 4
Test **7.1** - 
Result - seconds 
New Code GPU: 0.10527128219604492
New Code CPU: 0.04762232780456543
Old Code: 0.052808206081390384
Test **7.2** - 
Result - seconds
New Code GPU: 0.1032623028755188
New Code CPU: 0.03513101100921631
Old Code: 0.03523270606994629

Hey @01-vyom,

Great job - thanks a lot for tackling this issue! 

One question regarding the benchmarking it looks like CPU is faster than GPU -> is this actually the case? This would not be very good as it means that performing this computation on GPU would slow training down. Also the statistics about the "old code" does it refer to the old code being run on CPU or GPU? 

> Hey @01-vyom,
> 
> Great job - thanks a lot for tackling this issue!
> 
> One question regarding the benchmarking it looks like CPU is faster than GPU -> is this actually the case? This would not be very good as it means that performing this computation on GPU would slow training down. Also the statistics about the "old code" does it refer to the old code being run on CPU or GPU?

While running with smaller size inputs i.e. inputs with lesser batch size and mask length, this is the case. But, as we increase our mask length and batch size, this implementation is much efficient. Also, while using GPU, I think it is ineffcient because the tensors are being transfered back and forth from CPU and GPU. Old-code being run on CPU as it was only using numpy.

> While running with smaller size inputs i.e. inputs with lesser batch size and mask length, this is the case. But, as we increase our mash length and batch size, this implementation is much efficient. Also, while using GPU, I think it is ineffcient because the tensors are being transfered back and forth from CPU and GPU. Old-code being run on CPU as it was only using numpy.

Let me run some benchmarks on my side to see :-) Ideally we don't have to run anything on CPU 

I have made all the changes. 
@patrickvonplaten any updates on the benchmarks?

Hey @01-vyom,

I couldn't see a big speed-up really when running on GPU -> I think the goal should really be to completely get rid of the for loop. I implemented a simpler version here: https://github.com/huggingface/transformers/pull/11764

which I'm benchmarking now. Will let you know!

Hey @01-vyom,

Sorry I merged the PR now since it was more or less ready. I'm sorry to not have used your PR here! I'm sure we'll find other cool possible contributions for Wav2Vec2 though soon - will ping you if I find something interesting :-) 

No problem. Learned a lot from this issue.