Package: histoencoder
HistoEncoderFoundation models for digital pathology.Description•What???•Installation•Usage•API Documentation•CitationDescriptionHistoEncoderCLI interface allows users toextractandclusteruseful features for
histological slide images. Thehistoencoderpython package also exposes some useful functions for using the encoder models, which
are described in theAPI docs.What is this?Here, I've trained foundation models with massive amounts of data (prostate tissue) on
Europe's largest supercomputerLUMI.
These foundation models are now freely shared to the public through this repository.
Let's go over some nice things these models are capable of...Duringself-supervisedpre-training, thesemodelslearn howproduce similar features
for images with similar histological patterns. This is a really nice thing that
previous models haven't reeeally been able to do, and we can leverage that to do cool things.When you feed in tile images (from histological slides), the model willencodethese
images into a feature vector. When we cluster these features, each cluster will contain
tile images with similar histological patterns (cuz that's what the model is good at)!
You can do this at a slide level or for a whole dataset to automatically annotate slide
images.In the figure above, you can see UMAP representation of epithelium tissue of the
Radboud split of thePANDAdataset. Here we can see that cancerous and benign epithelium form separate clusters,
and you can even rediscover theGleason grading
systemfrom the UMAP.Although UMAPs are pretty, they're kinda shitty for drawing any actual conclusions...
Thus, I also clustered all features in the PANDA dataset into 128 clusters. Then we can
calculate the propotion of tile images with cancerous or bengin tissue in each cluster.
Thislabel purityis visualised on the top right corner, and you can see that ~52% of
the 3.8 million tile images are contained in clusters with over 90% label purity!To learn more, you can check out myslidesfrom theECDP 2023seminar. We also have a preprint coming once I
finally finish writing it...What about [input your tissue type here]?Although the first models in this repository (prostate_small,prostate_medium) have
been trained on only prostate tissue, they will very likely be better than any natural
image pre-trained model. Next step of this project/repository is to take the current
models and fine-tune them for other tissues to create other foundation models for new
tissue types (eg.colon_*,breast_*and so on). This fine-tuning step does not
require that much computing power so it's doable even though you don't have access to
thousands of GPUs.If you're interest in fine-tuning the models for other tissue types, I'm happy to
help and share my training scripts/methods! Just send me anemail!InstallationpipinstallhistoencoderUsageHistoEncodercommand line interfaceCut histological slide images into small tile images withHistoPrep.HistoPrep--input"./slide_images/*.tiff"--output./tile_images--width512--overlap0.5--max-background0.5Extract features for each tile image.HistoEncoderextract--input_dir./tile_images--model-nameprostate_smallCluster extracted features.HistoEncodercluster--input_dir./tile_imagesNowtrain_tilescontains a directory for each slide with the following contents.train_tiles
└──slide_image├──clusters.parquet# Clusters for each tile image.├──features.parquet# Extracted features for each tile.├──metadata.parquet# Everything else is generated by HistoPrep.├──properties.json├──thumbnail.jpeg├──thumbnail_tiles.jpeg├──thumbnail_tissue.jpeg└──tiles[52473entriesexceedsfilelimit,notopeningdir]histoencodermoduleIf you already have tile images, or want to use your own programs for extracting tile
images, you can use thehistoencoderpython package! Check out theAPI
docsfor more information.importhistoencoder.functionalasFencoder=F.create_encoder("prostate_small")forimagesinyour_fancy_dataloader:features=F.extract_features(encoder,images,num_blocks=2,avg_pool=True)...CitationIf you useHistoEncodermodels or pipelines in your publication, please cite the
github repository. There's a preprint coming, but I wanted to share the model to the
public as quickly as possible! :)@misc{histoencoder,
  author = {Pohjonen, Joona},
  title = {HistoEncoder: Foundation models for digital pathology},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {https://github.com/jopo666/HistoEncoder},
}