Model card for vit_small_patch16_224.transpath_mocov3

A Vision Transformer (ViT) image classification model.
Trained on 15M histology patches from PAIP and TCGA.
Used the MoCo v3 self supervised learning method.

Model Details

• Model Type: Feature backbone
• Model Stats:
  • Params (M): 21.7
  • Image size: 224 x 224 x 3
• Papers:
  • Transformer-based unsupervised contrastive learning for histopathological image classification: https://www.sciencedirect.com/science/article/abs/pii/S1361841522002043
• Dataset: TCGA: https://portal.gdc.cancer.gov/
• Original: https://github.com/Xiyue-Wang/TransPath
• License: GPLv3

Model Usage

Image Embeddings

from urllib.request import urlopen
from PIL import Image
import timm

# get example histology image
img = Image.open(
  urlopen(
    "https://github.com/owkin/HistoSSLscaling/raw/main/assets/example.tif"
  )
)

# load model from the hub
model = timm.create_model(
  model_name="hf-hub:1aurent/vit_small_patch16_224.transpath_mocov3",
  pretrained=True,
  num_heads=12,
).eval()

# get model specific transforms (normalization, resize)
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

data = transforms(img).unsqueeze(0)  # input is (batch_size, num_channels, img_size, img_size) shaped tensor
output = model(data)  # output is (batch_size, num_features) shaped tensor

Citation

@article{WANG2022102559,
  title    = {Transformer-based unsupervised contrastive learning for histopathological image classification},
  journal  = {Medical Image Analysis},
  volume   = {81},
  pages    = {102559},
  year     = {2022},
  issn     = {1361-8415},
  doi      = {https://doi.org/10.1016/j.media.2022.102559},
  url      = {https://www.sciencedirect.com/science/article/pii/S1361841522002043},
  author   = {Xiyue Wang and Sen Yang and Jun Zhang and Minghui Wang and Jing Zhang and Wei Yang and Junzhou Huang and Xiao Han},
  keywords = {Histopathology, Transformer, Self-supervised learning, Feature extraction},
  abstract = {A large-scale and well-annotated dataset is a key factor for the success of deep learning in medical image analysis. However, assembling such large annotations is very challenging, especially for histopathological images with unique characteristics (e.g., gigapixel image size, multiple cancer types, and wide staining variations). To alleviate this issue, self-supervised learning (SSL) could be a promising solution that relies only on unlabeled data to generate informative representations and generalizes well to various downstream tasks even with limited annotations. In this work, we propose a novel SSL strategy called semantically-relevant contrastive learning (SRCL), which compares relevance between instances to mine more positive pairs. Compared to the two views from an instance in traditional contrastive learning, our SRCL aligns multiple positive instances with similar visual concepts, which increases the diversity of positives and then results in more informative representations. We employ a hybrid model (CTransPath) as the backbone, which is designed by integrating a convolutional neural network (CNN) and a multi-scale Swin Transformer architecture. The CTransPath is pretrained on massively unlabeled histopathological images that could serve as a collaborative local–global feature extractor to learn universal feature representations more suitable for tasks in the histopathology image domain. The effectiveness of our SRCL-pretrained CTransPath is investigated on five types of downstream tasks (patch retrieval, patch classification, weakly-supervised whole-slide image classification, mitosis detection, and colorectal adenocarcinoma gland segmentation), covering nine public datasets. The results show that our SRCL-based visual representations not only achieve state-of-the-art performance in each dataset, but are also more robust and transferable than other SSL methods and ImageNet pretraining (both supervised and self-supervised methods). Our code and pretrained model are available at https://github.com/Xiyue-Wang/TransPath.}
}