Mamba Models - Missing MambaForSequenceClassification
Author: Adibvafa
Pull request: no
Labels: name: Feature request

### Feature request

Many prominent transformers (e.g. BERT) under the transformers library have specific implementations for SequenceClassification. I believe implementing these could potentially save a huge amount of time in the community.

### Motivation

I have been using the transformers library for tasks in biology or healthcare that required elegant refactoring of the current transformers library which has made me quite familiar with the internal infrastructure used to connect everything. Recently, I came upon the Mamba model and it seems these sub-models are missing.

### Your contribution

I am currently implementing MambaforSequenceClassification using the standard transformers library design patterns and will make a PR. You could assign this issue to me!

Comments:
Hi @Adibvafa, thanks for opening this feature request! 

We tend to reserve creating these task-specific classes for when there are checkpoints available, or when we have only checkpoints for the foundation model, which when published was advertised as being used for these downstream tasks. 

cc @ArthurZucker 

Since I need to implement it for my use case, is there a possibility of accepting my PR afterward? I am following the HuggingFace code style and modularity practices.

@Adibvafa The great thing about developing on an open source project is that you can build upon it freely and share your work without needing it to be merged into the original repo! You can add this as a feature in your fork, and maintain that to be in-sync with the repo, or add the class in your own repo as needed. Feel free to share what you've built here for the community to see and use 🤗 

@Adibvafa take a look at <https://github.com/getorca/mamba_for_sequence_classification> it's based on a  previous pull request on Mamba for sequence classification. I have a few examples. It's probably best to get some usage, and than merge into transformers. 

@amyeroberts Great insights! I actually did as you say and implemented it locally for my project, available at:
https://github.com/VectorInstitute/odyssey/blob/main/odyssey/models/cehr_mamba/mamba_utils.py

@getorca Great work! I do see some flaws in your implementations that we could resolve, making it more efficient and HuggingFace friendly. You can check what I implemented for my project above!

@Adibvafa absolutely, I welcome your input or pull requests, and would gladly work with you on it more. The more focus we get the more likely we can merge it into transformers.


@ArthurZucker Our team has developed a cutting-edge foundation model on EHR data using the Mamba implementation in HuggingFace along with the new MambaForSequenceClassification module I developed on top of the existing HuggingFace infrastructure.

The paper is being submitted to NeurIPS, and we will open-source the pretrained/finetuned weights on HuggingFace. Once that happens, I will push the code I developed so we can merge it into Transformers and use the weights easily.

🔥 this is music to my ears! can' t wait for your update! 

Hi @ArthurZucker!
The paper, `EHRMamba: Towards Generalizable and Scalable Foundation Models for Electronic Health Records` is published! You can take a look here:
https://vectorinstitute.github.io/EHRMamba

I made a PR with the MambaForSequenceClassification implementation: https://github.com/huggingface/transformers/pull/31155
Please take a look!