[WIP]add transformer transducer
Author: jp1924
Pull request: yes

# What does this PR do?
#20961
This PR adds [Transformer-Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://arxiv.org/abs/2002.02562)

This model is streaming model that model recognize text from audio in real-time and There is no site where model weight has been uploaded.

Transformer-Transducer implement: https://github.com/jp1924/transformer-transducer

RNN-Transducer reference: [https://lorenlugosch.github.io/posts/2020/11/transducer/](https://lorenlugosch.github.io/posts/2020/11/transducer/)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Models: speech models
@sanchit-gandhi
Library: generate
@gante
Maintained examples: pytorch
@sgugger


Comments:
Hey @jp1924! As discussed on the issue https://github.com/huggingface/transformers/issues/20961#issuecomment-1382245091, it's not possible to add a model to `transformers` when official weights are not available (please refer to the message thread for more details).

I would advise that you focus the implementation on a Transformer-Transducer codebase where strong pre-trained weights are available and open-sourced. I'm more than happy to help find a suitable codebase + weights to port! This would be a valuable addition to the `transformers` library.

hey @sanchit-gandhi

I saw several papers(like [MS T-T](https://arxiv.org/pdf/1910.12977.pdf), [Google T-T](https://arxiv.org/pdf/2002.02562.pdf), [Facebook T-T](https://arxiv.org/pdf/2010.11395.pdf)) related to the T-T model, but there is no content related to the official github code in the paper like BERT. So, while looking for an alternative, i found Transformer-Transducer implemented in a library called [openspeech](https://github.com/openspeech-team/openspeech).  

The model weight is not disclosed, but there is a code that can train [T-T](https://github.com/openspeech-team/openspeech/tree/main/openspeech/models/transformer_transducer). So I'm thinking of using openspeech to get the weight of the T-T first and then transferring the model and code to the hugingface, is it possible?

Hey @jp1924!

Cool that you've been able to dig so deeply into the ASR literature! Indeed, these are all fantastic research papers that highlight the strong performance of the T-T architecture. It's a real pity that neither MS, Google nor Meta released open-sourced weights for these papers, as I think a performant T-T model would be of great use to the open-source community, especially for low-latency/on-device applications.

Unfortunately, again with OpenSpeech it's an unofficial implementation without weights, so we probably can't add this to `transformers` either.

I'm really sorry you've invested time here without luck finding a performant set of open-source T-T weights. I had a look through your wandb logs, it does look as though the model is working. We can leave this PR open if you want to continue iterating and provide updates, but we won't be able to merge it to `transformers` without weights from a well established research paper (e.g. from MS, Google, Meta, etc)

Hey @sanchit-gandhi @flozi00 @fxtentacle @YooSungHyun!

Thank you so much for your interest in the T-T model! Unfortunately, it looks like we'll have to close the PR.....

I understand that this sophistication and prudence makes Transformers even better! Most of all, it was really nice to have access to Transformers' philosophy and new features!

---
Hey @sanchit-gandhi!

I have a question about PR. What is the range of the official code and weight? 

I think the emformer of #17302 is an example. The emformer paper does not have an official github code & weight, such as BERT. 

However, the emformer code and weight have been uploaded to torchaudio. So my question is, can I contribute to Transformer by using the code and weight in the certified library even if it's not the code listed in the official paper?

Hey @jp1924! Great question! The short answer is: code and weights in a certified library = yes! Just code = no