DOI: 10.5281/zenodo.15340874
title: MIPHEI-ViT Dataset: Processed Restained H&E and mIF Data from Orion-CRC and HEMIT

description: Overview

This dataset is released alongside our paper:“MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models”,where we introduce a model for predicting mIF marker expression directly from H&E morphology using vision transformer (ViT) foundation models.

We provide a carefully preprocessed version of two existing public datasets, specifically tailored for the task of H&E-to-mIF image translation. The dataset contains aligned and restained Hematoxylin & Eosin (H&E) and multiplex immunofluorescence (mIF) image tiles. These preprocessed tiles, along with associated metadata, enable full reproducibility of our experiments—including model training, evaluation, and cell-level analysis—and can serve as a ready-to-use resource for future work in H&E-to-mIF translation and multimodal learning.

Source Datasets

The dataset is derived from the following open-source datasets, containing aligned restained H&E and mIF images:





ORION-CRCSource: labsyspharm/ORION-CRC – ZenodoCitation:Lin J. labsyspharm/ORION-CRC [dataset]. Zenodo. 2023. doi:10.5281/zenodo.7637988Lin, J., et al. "High-plex immunofluorescence imaging and traditional histology of the same tissue section for discovering image-based biomarkers," in Nature Cancer, vol. 4, no. 7, pp. 1036–1052, 2023.License: MIT Licence




HEMITSource: Mendeley DataCitation:Bian C, Philips B, Cootes T, et al. HEMIT: H&E to Multiplex-immunohistochemistry Image Translation with Dual-Branch Pix2pix Generator. arXiv preprint arXiv:2403.18501, 2024.License: CC BY 4.0



We also used data from the IMMUcan dataset for validation purposes; however, it is not redistributed here as the dataset remains private.

Preprocessing Pipeline

Our preprocessing steps include:



On ORION-CRC and HEMIT





Nucleus segmentation with Cellpose on the DAPI channel




Single-cell pseudo-labeling using Gaussian Mixture Models (GMM) based on mIF marker expression







On ORION-CRC only





WSI-to-tile extraction at 20× magnification (ORION-CRC)




Artifact filtering using:





Channel-based noise detection on mIF




Foundation model (H-optimus-0) feature clustering to remove H&E artifact tiles






Autofluorescence subtraction using a custom Napari tool with marker-specific correction formulas




Channel normalization via percentile clipping and log-transformation





These steps aim to produce high-confidence marker-positive cell annotations from noisy mIF data, enabling robust learning and evaluation on paired H&E images.

File Structure

The dataset is organized into the following archives and directories:





HEMIT_nuclei_analysis.zipPreprocessed HEMIT data containing:





Nuclei segmentation masks generated using Cellpose (40× resolution)




Corresponding single-cell data in .csv format (per-cell intensities and cell types)


⚠️ This archive does not include the raw images from the original HEMIT dataset (License under the Creative Commons Attribution 4.0 International License (CC BY 4.0)). You must download the original dataset separately from Mendeley Data. The internal structure of this archive is designed to match the original, allowing direct integration of our nuclei and single-cell annotations.





ORION_dataset_20x.zipORION tile dataset at 20× magnification. Folder structure:





he/ – JPEG tiles of H&E images




if/ – 8-bit TIFF cleaned mIF images




nuclei/ – Label TIFF nucleus masks




csv_nuclei_pos/ – Per-WSI CSV files containing single-cell data: Nucleus position and cell types




slide_dataframe.csvDataframe that maps each slide (identified by slide_name) to its corresponding H&E, mIF, and nuclei WSI and CSV names.Columns include:





slide_name: Unique slide IDs




he_path: Paths to the H&E WSI




if_path: Paths to the mIF WSI




nuclei_path: Paths to the nucleus label WSI


nuclei_csv_path: Paths to the CSV file containing single-cell data (nucleus positions and cell types)





train_dataframe.csv, val_dataframe.csv, test_dataframe.csvDataframes containing tile-level metadata for each dataset split. Each row corresponds to a tile used during model training or evaluation.Columns include:





slide_name: IDs of the associated slide




image_path: Paths to the H&E tile




target_path: Paths to the corresponding mIF tile




nuclei_path: Paths to the nucleus label tile








ORION_dataset_20x_he_norm.zipCycleGAN-normalized H&E images from ORION, transformed to match the staining style of IMMUcan data (20× resolution). These images can be used as augmentation during the training of MIPHEI-ViT.



To extract:Use the following command:

7z x <path.zip>

Code & Tools

The full preprocessing pipeline used to produce this dataset — including tile extraction, autofluorescence correction, artifact removal, and single-cell analysis — is available at:

👉 GitHub Repository

This code allows you to reproduce our results and adapt the workflow to new datasets.

Citation

Please cite the associated paper and Zenodo DOI when using this dataset:



G. Balezo, et al, "MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models," 2025 [DOI TO UPDATE]