Speech2Text2 training support
Author: ivangtorre
Pull request: no

# 🚀 Feature request

Training support for Speech2Text2
https://github.com/huggingface/transformers/pull/13186

## Motivation

The Transformer release 4.11 has included theSpeechEncoderDecoder and Speech2Text2 decoder transformer @patrickvonplaten . However Speech2Text2 does not currently support training. There are some available models at [HuggingFace](https://huggingface.co/models?other=speech2text2) but it would be very interesting for the community to be able to train new languages. I am not not sure if it is possible to train other models in fairseq and to export them to huggingface. 

## Your contribution
I can contribute testing the code and providing training examples to the documentation


Comments:
Maybe of interest to @patil-suraj too

Hey @ivangtorre, 

I've now ported the tokenizer files so that the model can be trained. I'll try to publish a notebook this week that showcases how to fine-tune the `SpeechEncoderDecoder` model class.

Hi @patrickvonplaten any update on this ? 

My deadline predictions are again very much off - sorry! It is though definitely on my short-term ToDo List! A conservative estimate is by the end of next week now ;-) 

Thanks for the response @patrickvonplaten , I tried writing one script if possible  can you please let me know if the overall flow is right? I can compile it into a notebook and share here if that helps

https://gist.github.com/arijitx/b853c8bc29c3935e26f69343b32ddb4e#file-s2t_finetune-py

Hi @patrickvonplaten any update on this ? Sorry for bugging :( 

Script works on a dummy model: https://github.com/huggingface/transformers/pull/14792 - now testing on a real example

Hope to have it ready by tomorrow

Hi @patrickvonplaten I tried with your latest commit I'm getting this error 

```
Traceback (most recent call last):
  File "run_seq2seq.py", line 561, in <module>
    main()
  File "run_seq2seq.py", line 488, in main
    processor = AutoProcessor.from_pretrained(training_args.output_dir)
  File "C:\Users\admin\anaconda3\lib\site-packages\transformers\models\auto\processing_auto.py", line 171, in from_pretrained
    return PROCESSOR_MAPPING[type(config)].from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "C:\Users\admin\anaconda3\lib\site-packages\transformers\models\auto\auto_factory.py", line 559, in __getitem__
    raise KeyError(key)
KeyError: <class 'transformers.models.speech_encoder_decoder.configuration_speech_encoder_decoder.SpeechEncoderDecoderConfig'>
```

Hey @arijitx, 

Good observation! With this PR: https://github.com/huggingface/transformers/pull/14881 being merged it should work now. Could you try again? :-)

You will have to re-run this part: 

```python
from transformers import SpeechEncoderDecoderModel, AutoFeatureExtractor, AutoTokenizer, Wav2Vec2Processor

# checkpoints to leverage
encoder_id = "facebook/wav2vec2-base"
decoder_id = "facebook/bart-base"

# load and save speech-encoder-decoder model
# set some hyper-parameters for training and evaluation
model = SpeechEncoderDecoderModel.from_encoder_decoder_pretrained(encoder_id, decoder_id, encoder_add_adapter=True, encoder_feat_proj_dropout=0.0, encoder_layerdrop=0.0, max_length=200, num_beams=5)
model.config.decoder_start_token_id = model.decoder.config.bos_token_id
model.config.pad_token_id = model.decoder.config.pad_token_id
model.config.eos_token_id = model.decoder.config.eos_token_id
model.save_pretrained("./")

# load and save processor
feature_extractor = AutoFeatureExtractor.from_pretrained(encoder_id)
tokenizer = AutoTokenizer.from_pretrained(decoder_id)
processor = Wav2Vec2Processor(feature_extractor, tokenizer)
processor.save_pretrained("./")
```

Hi @patrickvonplaten ,

I'm running again from a build freshly installed from master getting this error,

```
Traceback (most recent call last):
  File "run_seq2seq.py", line 594, in <module>
    main()
  File "run_seq2seq.py", line 286, in main
    model = SpeechEncoderDecoderModel.from_encoder_decoder_pretrained(encoder_id, decoder_id, encoder_add_adapter=True, encoder_feat_proj_dropout=0.0, encoder_layerdrop=0.0, max_length=200, num_beams=5)
  File "/XX/lib/python3.6/site-packages/transformers/models/speech_encoder_decoder/modeling_speech_encoder_decoder.py", line 394, in from_encoder_decoder_pretrained
    encoder = AutoModel.from_pretrained(encoder_pretrained_model_name_or_path, *model_args)
  File "/XX/lib/python3.6/site-packages/transformers/models/auto/auto_factory.py", line 449, in from_pretrained
    f"Unrecognized configuration class {config.__class__} for this kind of AutoModel: {cls.__name__}.\n"
ValueError: Unrecognized configuration class <class 'transformers.models.speech_encoder_decoder.configuration_speech_encoder_decoder.SpeechEncoderDecoderConfig'> for this kind of AutoModel: AutoModel.
Model type should be one of ImageGPTConfig, QDQBertConfig, FNetConfig, SegformerConfig, VisionTextDualEncoderConfig, PerceiverConfig, GPTJConfig, LayoutLMv2Config, BeitConfig, RemBertConfig, VisualBertConfig, CanineConfig, RoFormerConfig, CLIPConfig, BigBirdPegasusConfig, DeiTConfig, LukeConfig, DetrConfig, GPTNeoConfig, BigBirdConfig, Speech2TextConfig, ViTConfig, Wav2Vec2Config, M2M100Config, ConvBertConfig, LEDConfig, BlenderbotSmallConfig, RetriBertConfig, IBertConfig, MT5Config, T5Config, MobileBertConfig, DistilBertConfig, AlbertConfig, BertGenerationConfig, CamembertConfig, XLMRobertaConfig, PegasusConfig, MarianConfig, MBartConfig, MegatronBertConfig, MPNetConfig, BartConfig, BlenderbotConfig, ReformerConfig, LongformerConfig, RobertaConfig, DebertaV2Config, DebertaConfig, FlaubertConfig, FSMTConfig, SqueezeBertConfig, HubertConfig, BertConfig, OpenAIGPTConfig, GPT2Config, TransfoXLConfig, XLNetConfig, XLMProphetNetConfig, ProphetNetConfig, XLMConfig, CTRLConfig, ElectraConfig, FunnelConfig, LxmertConfig, DPRConfig, LayoutLMConfig, TapasConfig, SplinterConfig, SEWDConfig, SEWConfig, UniSpeechSatConfig, UniSpeechConfig, WavLMConfig.


```

Hey @arijitx,

Could you give the full command that you ran (including the code to create the model?). On my side, the example seems to work. Happy to dive a bit deeper into your code if you could make a reproducible error. Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.