Adding TFWav2Vec2Model
Author: will-rice
Pull request: yes

Adds a TensorFlow version of Wav2Vec2
https://github.com/huggingface/transformers/issues/11603


# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@patrickvonplaten 
@LysandreJik

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Hi @will-rice, just an FYI, there's a CookieCutter template that allows you to automatically create a lot of files (modeling files, test files, documentation pages) for you. You can then edit those files. 

See [here](https://github.com/huggingface/transformers/tree/master/templates/adding_a_new_model) for more info. 

@patrickvonplaten Some of the layers used in the Pytorch version only exist in TensorFlow [addons](https://www.tensorflow.org/addons) and I imagine that we do not want to introduce an additional dependency. Is this normally handled with custom versions of layers I need to be included with the model code? 

> @patrickvonplaten Some of the layers used in the Pytorch version only exist in TensorFlow [addons](https://www.tensorflow.org/addons) and I imagine that we do not want to introduce an additional dependency. Is this normally handled with custom versions of layers I need to be included with the model code?

Hey @will-rice,

Is it the group norm layer? Maybe we can only implement the code for `config.feat_extract_norm == "layer"` in a first step and once this works we can see what to do next

@patrickvonplaten It's the group norm layer. I'm out until Saturday, but I will try your suggestion then. Thank you!

~@patrickvonplaten I'm almost finished with this. However, I'm having trouble getting the `Wav2Vec2PositionalConvEmbedding` to match. I may have it fixed by the time you see this, but if you have any tips on porting the PyTorch version to TensorFlow, I'd appreciate it.~

Edit: Figured it out. Just cleaning up and finishing tests.

@LysandreJik @patrickvonplaten I believe this is ready for a first look. 

The `run_tests_tf` workflow that is failing gives the error message `FAILED tests/test_hf_api.py::HfApiEndpointsTest::test_create_and_delete_repo`. 

The other one, `run_tests_torch` workflow seems to time out with the message:  `Too long with no output (exceeded 10m0s): context deadline exceeded`

I'm not completely sure how my code affects these tests. If there is something I missed please let me know.

This looks very nice! Rerunning the tests as the errors seem to be unrelated. 

Hey @will-rice, 

Amazing work! This is not an easy model to implement, and it seems that we are already very close to merging this PR :-) 

The main thing to change is to make the tests independent of PyTorch so that they are actually run at `run_tests_tf` and not skipped because PT is not installed. I think the easiest would be to just "translate" the PT tests into a TF format. 

Also, we should try to test both encoder architecture the "normal" Wav2Vec2Encoder as well as the "robust" Wav2Vec2RobustEncoder` model - ideally we make to testing classes here just as in PyTorch.


I did a small change to make your code work correctly for the model conversion for and uploaded TF weights to both:
- https://huggingface.co/facebook/wav2vec2-base-960h
- and https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self (weights will be uploaded in 5min - sorry my internet connection is slow today) 

Could you maybe also add a hard-coded test for https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self similar to the one in PyTorch and switch the "inofficial" weights in the tests to `facebook/wav2vec2-base-960h` and `facebook/wav2vec2-large-960h-lv60-self` ? 



@patrickvonplaten When I use `"facebook/wav2vec2-large-960h-lv60-self"` most of the layer weights don't get loaded for the `feature_extractor`. I'm using the exact same model class though, so I would think the weight names wouldn't have changed. Did you have any issues converting the weights other than my `layer_norm` mistake that you fixed? This seems like it would be related to why `TFWav2Vec2PositionalConvEmbedding` only matches when loaded directly from the pytorch layer weights.

```
tf_layer = tf_model.wav2vec2.encoder.pos_conv_embed
pt_layer = pt_model.wav2vec2.encoder.pos_conv_embed

tf_weight_v, tf_weight_g, tf_bias = tf_layer.conv.weights

for layer, conv in zip(tf_layer.weights, tf_layer.conv.weights):
    tf.assert_equal(layer, conv)

pt_weight_v = pt_layer.conv.weight_v.detach().numpy().transpose(2, 1, 0)
pt_weight_g = pt_layer.conv.weight_g.detach().numpy().transpose(2, 1, 0)
pt_bias = pt_layer.conv.bias.detach().numpy()

tf_layer.set_weights([pt_weight_v, pt_weight_g, pt_bias])

assert np.allclose(tf_weight_v, pt_weight_v)
assert np.allclose(tf_weight_g, pt_weight_g)
assert np.allclose(tf_bias, pt_bias)
```

This is what I have to do to get the final outputs to pass `assert_near`. I was able to fix this in my uploaded weights by doing the above. I don't think this should be necessary, but I'm having trouble tracking down exactly where the problem occurs. It looks like the weights for this layer aren't getting translated to TensorFlow well. Or the weights in the Pytorch model get modified by weight norm after being loaded.

Ah yeah the conversion for conv_norm in the pos_embeddings layer might be a bit problematic there indeed. It should show up with a failing:

```
tests/test_modeling_tf_wav2vec2.py::TFWav2Vec2ModelTest::test_pt_tf_model_equivalence
```
test. 

This might be a bit difficult to debug, actually. I think it'd be best to first refactor the TF tests to not include any PyTorch code and add test for the "robust" TFWav2Vec2 model as well. Once the "normal" TF tests all pass, we can take a look at the conversion afterwards together :-) 

@patrickvonplaten I pushed what I have. I had a lot of issues with the behavior of the scatter operation between the two frameworks. The Pytorch version seems to perform intuitively, but the TensorFlow ops have some differences that I don't completely understand. It may be a conceptual deficiency on my part so if you have any suggestions, I would greatly appreciate them.

@patrickvonplaten Thank you so much for all of the debugging. Looking at the changes, I've learned a lot and will be sure apply this in the future.

Thanks for all the work @will-rice! Happy that the commit diffs help a bit! The final changes were super tricky and very specific to how we convert weights between PyTorch and Tensorflow ;-)

Hey! I just reviewed and it looks good. Also, don't worry - the tensorflow `scatter` ops are quite counterintuitive; in one case I avoided them by building a tf.sparse() matrix and then densifying it, which was probably not the most efficient but it did work!

What problems were you encountering with the scatter updates? If you can identify specific problems I can help you write a function that gets the desired behaviour, but only if you think it's necessary.

Test failure is unrelated

Merging - awesome work @will-rice. Feel free to tweet about the addition of TFWav2Vec2 - I'm sure the community would be interested in hearing about it :-)

Hi @will-rice 👋 Thanks for adding this model all these months ago! I realise it's been quite a while, but I was wondering if you were able to answer two questions I have about the design decisions in this port: 

1. Additional arguments

For `TFWav2Vec2ForCTC`, the signature for `call` has some additional arguments which weren't in the equivalent PyTorch `forward` method. For example, `input_embeds` [is included here](https://github.com/huggingface/transformers/blob/d438eee030398e084ecf42b24a7453bcd7764d36/src/transformers/models/wav2vec2/modeling_tf_wav2vec2.py#L1547) but [not in the PT model](https://github.com/huggingface/transformers/blob/d438eee030398e084ecf42b24a7453bcd7764d36/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L1413). This argument also doesn't appear to be used once passed to the main layer. Is there a reason for including these? 

2. Missing arguments

There are also some arguments which are in the function signature for `forward` in the PyTorch model e.g.  `mask_time_indices` [here](https://github.com/huggingface/transformers/blob/d438eee030398e084ecf42b24a7453bcd7764d36/src/transformers/models/wav2vec2/modeling_wav2vec2.py#LL1009C2-L1009C2), which are taken from `kwargs` for the [TF model here](https://github.com/huggingface/transformers/blob/d438eee030398e084ecf42b24a7453bcd7764d36/src/transformers/models/wav2vec2/modeling_tf_wav2vec2.py#L1221). Is there a reason for only passing these in through kwargs? 

@amyeroberts I honestly do not remember why I did it like that. It looks like `input_embeds` should be removed and then `mask_time_indices` added as an argument to `call`.

OK, no worries. Thanks for the quick reply! :)