Microsoft's SpeechT5 for Spoken Language Processing (ASR, TTS, ST...)
Author: sanchit-gandhi
Pull request: no
Labels: name: New model

name: Good Second Issue

### Model description

Motivated by the success of [T5](https://arxiv.org/abs/1910.10683) for pre-training NLP models, [SpeechT5](https://arxiv.org/abs/2110.07205) explores a cross-modal framework for learning joint contextual representations for speech and text data via a shared encoder-decoder structure.

The model architecture consists of an encoder-decoder transformer module and six modal-specific pre/post nets. The pre-nets convert the input speech $\mathbf{X}^{s} \in \mathcal{D}^{s}$ or text $\mathbf{X}^{t} \in \mathcal{D}^{t}$ to a unified space of hidden representations. The hidden representations are then fed into the shared encoder-decoder to perform the sequence-to-sequence conversion. Finally, the post-nets generate the output in the speech or text modality, based on the decoder output.

Extensive evaluations show the superiority of the proposed SpeechT5 framework on a wide variety of spoken language processing tasks, including automatic speech recognition, speech synthesis, speech translation, voice conversion, speech enhancement, and speaker identification.

The paper was accepted at the ACL 2022 main conference: https://arxiv.org/abs/2110.07205

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Code and checkpoints: https://github.com/microsoft/SpeechT5/tree/main/SpeechT5

SpeechT5 combines a transformer encoder-decoder backbone with speech/text specific pre/post-nets. Thus, many of the modules required for the SpeechT5 model are already partially or fully implemented in Transformers.

Model architecture:

1. Transformer encoder block: Wav2Vec2/Hubert-encoder transformer block (Wav2Vec2EncoderLayer) https://github.com/huggingface/transformers/blob/26e5e129b43760138aed2dfc1cc3c75b481a95e6/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L725
2. Transformer decoder block: BERT-decoder transformer block (BertLMHeadModel) https://github.com/huggingface/transformers/blob/26e5e129b43760138aed2dfc1cc3c75b481a95e6/src/transformers/models/bert/modeling_bert.py#L1157
3. Speech-encoder pre-net: convolutional feature extractor of Wav2Vec2 (Wav2Vec2FeatureEncoder) https://github.com/huggingface/transformers/blob/26e5e129b43760138aed2dfc1cc3c75b481a95e6/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L408
4. Speech-decoder pre-net: three fully connected layers with the ReLU activation, fed with the log Mel-filterbank of the speech signal (new, original code: https://github.com/microsoft/SpeechT5/blob/main/SpeechT5/speecht5/models/modules/speech_decoder_prenet.py)
5. Speech-decoder post-net:  linear layer fed with the decoder output to predict the log Mel-filterbank $\mathbf{Y}_f = (\mathbf{y}_f , \dots, \mathbf{y}_f )$, followed by five 1-dimensional convolutional layers to refine the predicted $\mathbf{Y}_f$ (new, original code: https://github.com/microsoft/SpeechT5/blob/main/SpeechT5/speecht5/models/modules/speech_decoder_postnet.py)
6. Text pre/post-net: shared embeddings. See https://github.com/huggingface/transformers/blob/26e5e129b43760138aed2dfc1cc3c75b481a95e6/src/transformers/models/bart/modeling_bart.py#L1151

Comments:
Piecing these modules together should be a fun challenge! Happy to help with integration here :-)

Hi @sanchit-gandhi, if compute resources would not be a problem here ( just have a 4GB GPU on my laptop or Google Colab with me here  :upside_down_face:  ) then I want to help in adding this model as well

Hey @ayushtues! Lovely to meet you :-) Compute shouldn't be a problem! We can begin with 'dummy' versions of the model in order to verify that our implementations work, then scale up to the full size ones and share resources. 

As a starting point, we can start the PR with the 'add-new-model-like' command:
https://github.com/huggingface/transformers/tree/main/templates/adding_a_new_model#add-new-model-like-command from the Wav2Vec2 model:
https://github.com/huggingface/transformers/tree/main/src/transformers/models/wav2vec2
This will take care of the Wav2Vec2 Feature Encoder (speech pre-net) and Wav2Vec2 Encoder (speech encoder block) for us! Having automatically created all the files, the next step will be to verify that the feature extractor and speech encoder block match Microsoft's implementation.

Feel free to ping me on Slack (sanchit[at]huggingface.co) if you have any questions!

Hi @sanchit-gandhi, thanks for the reply, great I'll start reading the paper in detail, checking out its repo, and also Huggingface's contribution documentation, and then start working on the PR.

I am not added on slack, and can't find a public invite link, can you send me an invite if possible?

Awesome! Let me know how you get on :-)

If you drop me an email at sanchit[at]huggingface.co I can send you over an invite!

Hey there @sanchit-gandhi @ayushtues, I’m also interested in contributing, if any assistance is still required. Please do let me know!

Hey @mingboiz! Great to have you on-board! I'll invite you over to the Slack channel too!

Hey there @sanchit-gandhi @ayushtues, I’m also interested in contributing. @sanchit-gandhi I have sent you a DM on slack. 

Hey @mingboiz and @anuragshas, if you guys could drop me an email at sanchit[at]huggingface.co I can send you over email invites to the Slack channel!

Great to see so much interest in adding this model! 🔥 Should be a fun collaborative project!

Hi @sanchit-gandhi, I have send you the rquest to send me the invite link. I would also love to contribute here.


I'm also open to helping out with this.

I created a branch with a model made from wav2vec2.
https://github.com/huggingface/transformers/pull/17982

Is this still being developed on? I'd be happy to contribute here as well. @sanchit-gandhi 

Hey @kasmith11, sorry for the late response! If you drop me an email at sanchit [at] huggingface.co I can add you to the Slack channel for the model addition. There's plenty of opportunity for contribution!

Closed via https://github.com/huggingface/transformers/pull/18922