Request for a Vision Transformer Model for Digital Image Segmentation
Author: hanshengzhu0001
Pull request: no
Labels: name: New model

### Model description

I would like to request the addition of a Vision Transformer (ViT) model specifically fine-tuned for digital image segmentation tasks. The model should leverage transformer-based architecture to effectively capture spatial relationships within images, improving performance in tasks such as medical image analysis, satellite image segmentation, or autonomous driving.

The Vision Transformer architecture has proven to be highly effective for various vision tasks by using self-attention mechanisms to capture long-range dependencies in images. This model would be particularly valuable for tasks requiring pixel-level classification, where traditional convolutional neural networks (CNNs) often struggle to capture global features effectively.

The ViT model should include the following key features:

- Pretrained on a large, diverse image segmentation dataset.
- Fine-tuned for pixel-level classification tasks such as medical image segmentation or semantic segmentation of everyday objects.
- Support for standard ViT variants such as ViT-B (Base), ViT-L (Large), and ViT-H (Huge) depending on the task's computational budget.
- Open-source weights and implementation.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

@amyeroberts, @qubvel

Comments:
@amyeroberts @qubvel I’m interested in working on this task. Could you please confirm if this model is required and would be a valuable addition?

Hi @hanshengzhu0001 and @Uvi-12, thanks for the feature request and interest in contributing! 

Indeed a powerful image segmentation model will be a great addition to the library, however, we do not have plans to train such a model by ourselves. In case you have a candidate model/repo it might be impactful to add it to the library, please provide links to repos/papers and we can discuss it here.

By the way `transformers` have several semantic segmentation models such as DPT (basically ViT with segmentation head), Segformer (also a transformer-based model), and UperNet (convolution-based). In addition MaskFormer, Mask2Former, and OneFormer models for instance/panoptic segmentation

Hi  @Uvi-12
<https://urldefense.com/v3/__https://github.com/Uvi-12__;!!IBzWLUs!VKhdH62zc_Ydi1rNq5BjSJseivUBkCiGN_BeyAiSuXfJa4du7mJGrdvcSy6CixpdrV76IY31sUH8j0el2GWzEwuTI1W0$>
,

Are you interested in moving this forward? If so I'm down for this open
source contribution opportunity.

Best regards,
Hans

On Fri, Jan 3, 2025 at 7:03 PM UV ***@***.***> wrote:

> @amyeroberts
> <https://urldefense.com/v3/__https://github.com/amyeroberts__;!!IBzWLUs!W6lbSBKKCn1COoOhWWKzxL1PujRvXFNQOPrYc3cC_ynZoaE_135gfTpvIHPD_Npsbi98GP8jubzwTyxyE8gkF3vfhSRw$>
> @qubvel
> <https://urldefense.com/v3/__https://github.com/qubvel__;!!IBzWLUs!W6lbSBKKCn1COoOhWWKzxL1PujRvXFNQOPrYc3cC_ynZoaE_135gfTpvIHPD_Npsbi98GP8jubzwTyxyE8gkF6xahPV6$>
> I’m interested in working on this task. Could you please confirm if this
> model is required and would be a valuable addition?
>
> —
> Reply to this email directly, view it on GitHub
> <https://urldefense.com/v3/__https://github.com/huggingface/transformers/issues/35477*issuecomment-2569051181__;Iw!!IBzWLUs!W6lbSBKKCn1COoOhWWKzxL1PujRvXFNQOPrYc3cC_ynZoaE_135gfTpvIHPD_Npsbi98GP8jubzwTyxyE8gkF1RCZ9M-$>,
> or unsubscribe
> <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ARVGW6W3LQ2ZZIABKIK5BYT2IZU7PAVCNFSM6AAAAABUPFXL3OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRZGA2TCMJYGE__;!!IBzWLUs!W6lbSBKKCn1COoOhWWKzxL1PujRvXFNQOPrYc3cC_ynZoaE_135gfTpvIHPD_Npsbi98GP8jubzwTyxyE8gkF1zsIskW$>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


Hi  @Uvi-12
<https://urldefense.com/v3/__https://github.com/Uvi-12__;!!IBzWLUs!VKhdH62zc_Ydi1rNq5BjSJseivUBkCiGN_BeyAiSuXfJa4du7mJGrdvcSy6CixpdrV76IY31sUH8j0el2GWzEwuTI1W0$>
,

Are you interested in moving this forward? If so I'm down for this open
source contribution opportunity.

Best regards,
Hans

On Tue, Jan 7, 2025 at 3:55 AM Pavel Iakubovskii ***@***.***>
wrote:

> Hi @hanshengzhu0001
> <https://urldefense.com/v3/__https://github.com/hanshengzhu0001__;!!IBzWLUs!VKhdH62zc_Ydi1rNq5BjSJseivUBkCiGN_BeyAiSuXfJa4du7mJGrdvcSy6CixpdrV76IY31sUH8j0el2GWzEzW5JRLZ$>
> and @Uvi-12
> <https://urldefense.com/v3/__https://github.com/Uvi-12__;!!IBzWLUs!VKhdH62zc_Ydi1rNq5BjSJseivUBkCiGN_BeyAiSuXfJa4du7mJGrdvcSy6CixpdrV76IY31sUH8j0el2GWzEwuTI1W0$>,
> thanks for the feature request and interest in contributing!
>
> Indeed a powerful image segmentation model will be a great addition to the
> library, however, we do not have plans to train such a model by ourselves.
> In case you have a candidate model/repo it might be impactful to add it to
> the library, please provide links to repos/papers and we can discuss it
> here.
>
> By the way transformers have several semantic segmentation models such as
> DPT (basically ViT with segmentation head), Segformer (also a
> transformer-based model), and UperNet (convolution-based). In addition
> MaskFormer, Mask2Former, and OneFormer models for instance/panoptic
> segmentation
>
> —
> Reply to this email directly, view it on GitHub
> <https://urldefense.com/v3/__https://github.com/huggingface/transformers/issues/35477*issuecomment-2573811649__;Iw!!IBzWLUs!VKhdH62zc_Ydi1rNq5BjSJseivUBkCiGN_BeyAiSuXfJa4du7mJGrdvcSy6CixpdrV76IY31sUH8j0el2GWzE8y7mAAP$>,
> or unsubscribe
> <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ARVGW6VMGLZO5RTM3AO6XX32JLNQTAVCNFSM6AAAAABUPFXL3OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZTHAYTCNRUHE__;!!IBzWLUs!VKhdH62zc_Ydi1rNq5BjSJseivUBkCiGN_BeyAiSuXfJa4du7mJGrdvcSy6CixpdrV76IY31sUH8j0el2GWzE_A5gEfI$>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


> 
> Are you interested in moving this forward? If so I'm down for this open source contribution opportunity.
> 

Hi @hanshengzhu0001, I’m interested in moving this forward! Would it be okay if I reached out to you if I needed any help?

Best regards,
Uvi-12


Yes, please! I’m an AI major student focusing on image learning and
diffusion, and I’d be happy to assist if needed.

Best regards,
Hans

On Tue, Jan 7, 2025 at 7:46 AM UV ***@***.***> wrote:

> Are you interested in moving this forward? If so I'm down for this open
> source contribution opportunity.
>
> Hi @hanshengzhu0001
> <https://urldefense.com/v3/__https://github.com/hanshengzhu0001__;!!IBzWLUs!RBgHEHt7Fm98Ly9VTy-fGP5GdwqU286xow9PJyM-zdYN5egP-F2lENDlw9fXBNgHaZqVkQTPt_hB42zXfhVTEPxTj8-g$>,
> I’m interested in moving this forward! Would it be okay if I reached out to
> you if I needed any help?
>
> Best regards,
> Uvi-12
>
> —
> Reply to this email directly, view it on GitHub
> <https://urldefense.com/v3/__https://github.com/huggingface/transformers/issues/35477*issuecomment-2574125248__;Iw!!IBzWLUs!RBgHEHt7Fm98Ly9VTy-fGP5GdwqU286xow9PJyM-zdYN5egP-F2lENDlw9fXBNgHaZqVkQTPt_hB42zXfhVTELyLxJ_7$>,
> or unsubscribe
> <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ARVGW6UP5KYJ6G6YF4MGYML2JMIWVAVCNFSM6AAAAABUPFXL3OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZUGEZDKMRUHA__;!!IBzWLUs!RBgHEHt7Fm98Ly9VTy-fGP5GdwqU286xow9PJyM-zdYN5egP-F2lENDlw9fXBNgHaZqVkQTPt_hB42zXfhVTEM-zjFBH$>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


> Yes, please! I’m an AI major student focusing on image learning and diffusion, and I’d be happy to assist if needed. 

That's amazing! Thank you so much for offering your help—I truly appreciate it.