Bring back Transformer-encoder, LSTM-decoder models
Author: Ubadub
Pull request: no

### Feature request

HuggingFace used to support (albeit, with bugs), a "Model2LSTM" class that was closed with [this ticket](https://github.com/huggingface/transformers/issues/2849) / [this PR](https://github.com/huggingface/transformers/pull/2968). While [the code](https://github.com/huggingface/transformers/blob/90ab15cb7a8fcf8bf58c05453ddf1aa6a4fa00c1/src/transformers/modeling_encoder_decoder.py#L335) was buggy, I don't think deleting it was the right decision.

### Motivation

Ultimately, I guess, the motivation is... this is a real thing that exists and it seems to be within the scope of the `transformers` project. Concretely, this came up because I wanted to try and implement such a model, and discovered that `transformers` used to support it, but doesn't anymore.

### Your contribution

I'm willing to make code contributions towards this, and already began an exploratory analysis of the code base to see if this is possible, but as a team member I'm not fully aware of the reasons this was dropped in the first place, and what limits there might be on its feasibility.

Comments:
Hi @Ubadub, 

There's many reasons a model might be removed or code deleted. Every piece of code in the library requires maintenance from us, and so it's not possible to support everything. We decide what to keep, remove or deprecate based on maintenance burden and how impactful it is for the community. The linked PRs and issues are from over 3 years ago, and to the best of my knowledge, the LSTM decoder models haven't been requested by other users since their deletion. 

The great thing about open source is that anyone can build upon this library! If you're interested in adding this capability, you're welcome to develop in your own fork and share it here or on the forums for other users to find and use. It's now also possible to [share models directly on the hub](https://huggingface.co/docs/transformers/custom_models). 

Hi @amyeroberts ,

Thanks for your reply!

> The great thing about open source is that anyone can build upon this library! If you're interested in adding this capability, you're welcome to develop in your own fork and share it here or on the forums for other users to find and use.

As mentioned, I have been looking into making the requisite changes myself with the hope of opening a PR for it. I opened this ticket first partly to see if others might express support/interest, and also in case there was some obvious or overt reason to not include such a model that might come up.

> It's now also possible to [share models directly on the hub](https://huggingface.co/docs/transformers/custom_models). 

Also a helpful suggestion, thank you.

@Ubadub At the moment, it's unlikely we'll merge this into the library unless we see a lot of demand from the community (which we'll measure through 👍's on this issue). In this case, it's still possible to develop on a fork and share your work by linking to it here, but it's not necessary to open a PR. 

You might also be interested in our recently [added RWKV model](https://huggingface.co/docs/transformers/model_doc/rwkv), which rework the traditional transformer attention so that it can be used as an RNN. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.