Reviewerly/exHarmony huggingface dataset:
Unsupervised 3D Out-of-Distribution Detection with Latent Diffusion Models

Methods for out-of-distribution (OOD) detection that scale to 3D data are crucial components of any real-world clinical deep learning system. Classic denoising diffusion probabilistic models (DDPMs) have been recently proposed as a robust way to perform reconstruction-based OOD detection on 2D datasets, but do not trivially scale to 3D data. In this work, we propose to use Latent Diffusion Models (LDMs), which enable the scaling of DDPMs to high-resolution 3D medical data. We validate the proposed approach on near- and far-OOD datasets and compare it to a recently proposed, 3D-enabled approach using Latent Transformer Models (LTMs). Not only does the proposed LDM-based approach achieve statistically significant better performance, it also shows less sensitivity to the underlying latent representation, more favourable memory scaling, and produces better spatial anomaly maps. Code is available at https://github.com/marksgraham/ddpm-ood .
Investigating Transformer Encoding Techniques to Improve Data-Driven Volume-to-Surface Liver Registration for Image-Guided Navigation

Due to limited direct organ visualization, minimally invasive interventions rely extensively on medical imaging and image guidance to ensure accurate surgical instrument navigation and target tissue manipulation. In the context of laparoscopic liver interventions, intra-operative video imaging only provides a limited field-of-view of the liver surface, with no information of any internal liver lesions identified during diagnosis using pre-procedural imaging. Hence, to enhance intra-procedural visualization and navigation, the registration of pre-procedural, diagnostic images and anatomical models featuring target tissues to be accessed or manipulated during surgery entails a sufficient accurate registration of the pre-procedural data into the intra-operative setting. Prior work has demonstrated the feasibility of neural network-based solutions for nonrigid volume-to-surface liver registration. However, view occlusion, lack of meaningful feature landmarks, and liver deformation between the pre- and intra-operative settings all contribute to the difficulty of this registration task. In this work, we leverage some of the state-of-the-art deep learning frameworks to implement and test various network architecture modifications toward improving the accuracy and robustness of volume-to-surface liver registration. Specifically, we focus on the adaptation of a transformer-based segmentation network for the task of better predicting the optimal displacement field for nonrigid registration. Our results suggest that one particular transformer-based network architecture—UTNet—led to significant improvements over baseline performance, yielding a mean displacement error on the order of 4 mm across a variety of datasets.
Masked Vision and Language Pre-training with Unimodal and Multimodal Contrastive Losses for Medical Visual Question Answering

Medical visual question answering (VQA) is a challenging task that requires answering clinical questions of a given medical image, by taking consider of both visual and language information. However, due to the small scale of training data for medical VQA, pre-training fine-tuning paradigms have been a commonly used solution to improve model generalization performance. In this paper, we present a novel self-supervised approach that learns unimodal and multimodal feature representations of input images and text using medical image caption datasets, by leveraging both unimodal and multimodal contrastive losses, along with masked language modeling and image text matching as pre-training objectives. The pre-trained model is then transferred to downstream medical VQA tasks. The proposed approach achieves state-of-the-art (SOTA) performance on three publicly available medical VQA datasets with significant accuracy improvements of 2.2%, 14.7%, and 1.7% respectively. Besides, we conduct a comprehensive analysis to validate the effectiveness of different components of the approach and study different pre-training settings. Our codes and models are available at https://github.com/pengfeiliHEU/MUMC .
Multi-Target Domain Adaptation with Prompt Learning for Medical Image Segmentation

Domain shift is a big challenge when deploying deep learning models in real-world applications due to various data distributions. The recent advances of domain adaptation mainly come from explicitly learning domain invariant features (e.g., by adversarial learning, metric learning and self-training). While they cannot be easily extended to multi-domains due to the diverse domain knowledge. In this paper, we present a novel multi-target domain adaptation (MTDA) algorithm, i.e., prompt-DA, through implicit feature adaptation for medical image segmentation. In particular, we build a feature transfer module by simply obtaining the domain-specific prompts and utilizing them to generate the domain-aware image features via a specially designed simple feature fusion module. Moreover, the proposed prompt-DA is compatible with the previous DA methods (e.g., adversarial learning based) and the performance can be continuously improved. The proposed method is evaluated on two challenging domain-shift datasets, i.e., the Iseg2019 (domain shift in infant MRI of different ages), and the BraTS2018 dataset (domain shift between high-grade and low-grade gliomas). Experimental results indicate our proposed method achieves state-of-the-art performance in both cases, and also demonstrates the effectiveness of the proposed prompt-DA. The experiments with adversarial learning DA show our proposed prompt-DA can go well with other DA methods. Our code is available at https://github.com/MurasakiLin/prompt-DA .
Boundary Difference over Union Loss for Medical Image Segmentation

Medical image segmentation is crucial for clinical diagnosis. However, current losses for medical image segmentation mainly focus on overall segmentation results, with fewer losses proposed to guide boundary segmentation. Those that do exist often need to be used in combination with other losses and produce ineffective results. To address this issue, we have developed a simple and effective loss called the Boundary Difference over Union Loss (Boundary DoU Loss) to guide boundary region segmentation. It is obtained by calculating the ratio of the difference set of prediction and ground truth to the union of the difference set and the partial intersection set. Our loss only relies on region calculation, making it easy to implement and training stable without needing any additional losses. Additionally, we use the target size to adaptively adjust attention applied to the boundary regions. Experimental results using UNet, TransUNet, and Swin-UNet on two datasets (ACDC and Synapse) demonstrate the effectiveness of our proposed loss function. Code is available at https://github.com/sunfan-bvb/BoundaryDoULoss .
Topology Repairing of Disconnected Pulmonary Airways and Vessels: Baselines and a Dataset

Accurate segmentation of pulmonary airways and vessels is crucial for the diagnosis and treatment of pulmonary diseases. However, current deep learning approaches suffer from disconnectivity issues that hinder their clinical usefulness. To address this challenge, we propose a post-processing approach that leverages a data-driven method to repair the topology of disconnected pulmonary tubular structures. Our approach formulates the problem as a keypoint detection task, where a neural network is trained to predict keypoints that can bridge disconnected components. We use a training data synthesis pipeline that generates disconnected data from complete pulmonary structures. Moreover, the new Pulmonary Tree Repairing (PTR) dataset is publicly available, which comprises 800 complete 3D models of pulmonary airways, arteries, and veins, as well as the synthetic disconnected data. Our code and data are available at https://github.com/M3DV/pulmonary-tree-repairing .
Automated CT Lung Cancer Screening Workflow Using 3D Camera

Despite recent developments in CT planning that enabled automation in patient positioning, time-consuming scout scans are still needed to compute dose profile and ensure the patient is properly positioned. In this paper, we present a novel method which eliminates the need for scout scans in CT lung cancer screening by estimating patient scan range, isocenter, and Water Equivalent Diameter (WED) from 3D camera images. We achieve this task by training an implicit generative model on over 60,000 CT scans and introduce a novel approach for updating the prediction using real-time scan data. We demonstrate the effectiveness of our method on a testing set of 110 pairs of depth data and CT scan, resulting in an average error of 5 mm in estimating the isocenter, 13 mm in determining the scan range, 10 mm and 16 mm in estimating the AP and lateral WED respectively. The relative WED error of our method is $$4\%$$ , which is well within the International Electrotechnical Commission (IEC) acceptance criteria of $$10\%$$ .
Evolutionary Normalization Optimization Boosts Semantic Segmentation Network Performance

Semantic segmentation is an important task in medical imaging. Typically, encoder-decoder architectures, such as the U-Net, are used in various variants to approach this task. Normalization methods, such as Batch or Instance Normalization are used throughout the architectures to adapt to data-specific noise. However, it is barely investigated which normalization method is most suitable for a given dataset and if a combination of those is beneficial for the overall performance. In this work, we show that by using evolutionary algorithms we can fully automatically select the best set of normalization methods, outperforming any competitive single normalization method baseline. We provide insights into the selection of normalization and how this compares across imaging modalities and datasets. Overall, we propose that normalization should be managed carefully during the development of the most recent semantic segmentation models as it has a significant impact on medical image analysis tasks, contributing to a more efficient analysis of medical data. Our code is openly available at https://github.com/neuluna/evoNMS .
Deep Cellular Embeddings: An Explainable Plug and Play Improvement for Feature Representation in Histopathology

Weakly supervised classification of whole slide images (WSIs) in digital pathology typically involves making slide-level predictions by aggregating predictions from embeddings extracted from multiple individual tiles. However, these embeddings can fail to capture valuable information contained within the individual cells in each tile. Here we describe an embedding extraction method that combines tile-level embeddings with a cell-level embedding summary. We validated the method using four hematoxylin and eosin stained WSI classification tasks: human epidermal growth factor receptor 2 status and estrogen receptor status in primary breast cancer, breast cancer metastasis in lymph node tissue, and cell of origin classification in diffuse large B-cell lymphoma. For all tasks, the new method outperformed embedding extraction methods that did not include cell-level representations. Using the publicly available HEROHE Challenge data set, the method achieved a state-of-the-art performance of 90% area under the receiver operating characteristic curve. Additionally, we present a novel model explainability method that could identify cells associated with different classification groups, thus providing supplementary validation of the classification model. This deep learning approach has the potential to provide morphological insights that may improve understanding of complex underlying tumor pathologies.
Few-Shot Medical Image Segmentation via a Region-Enhanced Prototypical Transformer

Automated segmentation of large volumes of medical images is often plagued by the limited availability of fully annotated data and the diversity of organ surface properties resulting from the use of different acquisition protocols for different patients. In this paper, we introduce a more promising few-shot learning-based method named Region-enhanced Prototypical Transformer (RPT) to mitigate the effects of large intra-class diversity/bias. First, a subdivision strategy is introduced to produce a collection of regional prototypes from the foreground of the support prototype. Second, a self-selection mechanism is proposed to incorporate into the Bias-alleviated Transformer (BaT) block to suppress or remove interferences present in the query prototype and regional support prototypes. By stacking BaT blocks, the proposed RPT can iteratively optimize the generated regional prototypes and finally produce rectified and more accurate global prototypes for Few-Shot Medical Image Segmentation (FSMS). Extensive experiments are conducted on three publicly available medical image datasets, and the obtained results show consistent improvements compared to state-of-the-art FSMS methods. The source code is available at: https://github.com/YazhouZhu19/RPT .
QCResUNet: Joint Subject-Level and Voxel-Level Prediction of Segmentation Quality

Deep learning has achieved state-of-the-art performance in automated brain tumor segmentation from magnetic resonance imaging (MRI) scans. However, the unexpected occurrence of poor-quality outliers, especially in out-of-distribution samples, hinders their translation into patient-centered clinical practice. Therefore, it is important to develop automated tools for large-scale segmentation quality control (QC). However, most existing QC methods targeted cardiac MRI segmentation which involves a single modality and a single tissue type. Importantly, these methods only provide a subject-level segmentation-quality prediction, which cannot inform clinicians where the segmentation needs to be refined. To address this gap, we proposed a novel network architecture called QCResUNet that simultaneously produces segmentation-quality measures as well as voxel-level segmentation error maps for brain tumor segmentation QC. To train the proposed model, we created a wide variety of segmentation-quality results by using i) models that have been trained for a varying number of epochs with different modalities; and ii) a newly devised segmentation-generation method called SegGen. The proposed method was validated on a large public brain tumor dataset with segmentations generated by different methods, achieving high performance on the prediction of segmentation-quality metric as well as voxel-wise localization of segmentation errors. The implementation will be publicly available at https://github.com/peijie-chiu/QC-ResUNet .
Class-Aware Feature Alignment for Domain Adaptative Mitochondria Segmentation

Unsupervised domain adaptation (UDA) has gained great popularity in mitochondria segmentation, aiming to improve the adaptability of models from the labeled source domain to the unlabeled target domain via domain alignment. However, existing UDA methods only focus on aligning domains on the prediction level, while ignoring the feature space containing more adequate information than the predictions. In this paper, we propose a class-aware domain adaptation method for mitochondria segmentation on the feature level, which relies on the prototype representation to achieve more fine-grained alignment. In particular, we first extract the feature centroids of classes from the source domain as prototypes. Leveraging the extracted prototypes as a bridge, we constrain that features belonging to the same class but from different domains are pulled closer to each other, achieving the class-aware alignment. Meanwhile, we derive a segmentation prediction directly from feature space based on the distance between target features and source prototypes. By incorporating a pseudo label to supervise the learning of this prediction, the feature distribution gap across domains is further reduced. Furthermore, to take full advantage of the potential of target domain, we propose an intra-domain consistency constraint to maintain consistent predictions of samples perturbed differently from the target image. Extensive experiments on different datasets demonstrate the superiority of our proposed method over existing UDA methods. Code is available at https://github.com/Danyin813/CAFA.
Self-feedback Transformer: A Multi-label Diagnostic Model for Real-World Pancreatic Neuroendocrine Neoplasms Data

CAD is an emerging field, but most models are not equipped to handle missing and noisy data in real-world medical scenarios, particularly in the case of rare tumors like pancreatic neuroendocrine neoplasms (pNENs). Multi-label models meet the needs of real-world study, but current methods do not consider the issue of missing and noisy labels. This study introduces a multi-label model called Self-feedback Transformer (SFT) that utilizes a transformer to model the relationships between labels and images, and uses a ingenious self-feedback strategy to improve label utilization. We evaluated SFT on 11 clinical tasks using a real-world dataset of pNENs and achieved higher performance than other state-of-the-art multi-label models with mAUCs of 0.68 and 0.76 on internal and external datasets, respectively. Our model has four inference modes that utilize self-feedback and expert assistance to further increase mAUCs to 0.72 and 0.82 on internal and external datasets, respectively, while maintaining good performance even with input label noise ratios up to 40% in expert-assisted mode.
H-DenseFormer: An Efficient Hybrid Densely Connected Transformer for Multimodal Tumor Segmentation

Recently, deep learning methods have been widely used for tumor segmentation of multimodal medical images with promising results. However, most existing methods are limited by insufficient representational ability, specific modality number and high computational complexity. In this paper, we propose a hybrid densely connected network for tumor segmentation, named H-DenseFormer, which combines the representational power of the Convolutional Neural Network (CNN) and the Transformer structures. Specifically, H-DenseFormer integrates a Transformer-based Multi-path Parallel Embedding (MPE) module that can take an arbitrary number of modalities as input to extract the fusion features from different modalities. Then, the multimodal fusion features are delivered to different levels of the encoder to enhance multimodal learning representation. Besides, we design a lightweight Densely Connected Transformer (DCT) block to replace the standard Transformer block, thus significantly reducing computational complexity. We conduct extensive experiments on two public multimodal datasets, HECKTOR21 and PI-CAI22. The experimental results show that our proposed method outperforms the existing state-of-the-art methods while having lower computational complexity. The source code is available at https://github.com/shijun18/H-DenseFormer .
Morphology-Inspired Unsupervised Gland Segmentation via Selective Semantic Grouping

Designing deep learning algorithms for gland segmentation is crucial for automatic cancer diagnosis and prognosis. However, the expensive annotation cost hinders the development and application of this technology. In this paper, we make a first attempt to explore a deep learning method for unsupervised gland segmentation, where no manual annotations are required. Existing unsupervised semantic segmentation methods encounter a huge challenge on gland images. They either over-segment a gland into many fractions or under-segment the gland regions by confusing many of them with the background. To overcome this challenge, our key insight is to introduce an empirical cue about gland morphology as extra knowledge to guide the segmentation process. To this end, we propose a novel Morphology-inspired method via Selective Semantic Grouping. We first leverage the empirical cue to selectively mine out proposals for gland sub-regions with variant appearances. Then, a Morphology-aware Semantic Grouping module is employed to summarize the overall information about glands by explicitly grouping the semantics of their sub-region proposals. In this way, the final segmentation network could learn comprehensive knowledge about glands and produce well-delineated and complete predictions. We conduct experiments on the GlaS dataset and the CRAG dataset. Our method exceeds the second-best counterpart by over 10.56% at mIOU.
Full Image-Index Remainder Based Single Low-Dose DR/CT Self-supervised Denoising

Low-dose digital radiography (DR) and computed tomography (CT) play a crucial role in minimizing health risks during clinical examinations and diagnoses. However, reducing the radiation dose often leads to lower signal-to-noise ratio measurements, resulting in degraded image quality. Existing supervised and self-supervised reconstruction techniques have been developed with noisy and clean image pairs or noisy and noisy image pairs, implying they cannot be adapted to single DR and CT image denoising. In this study, we introduce the Full Image-Index Remainder (FIRE) method. Our method begins by dividing the entire high-dimensional image space into multiple low-dimensional sub-image spaces using a full image-index remainder technique. By leveraging the data redundancy present within these sub-image spaces, we identify similar groups of noisy sub-images for training a self-supervised denoising network. Additionally, we establish a sub-space sampling theory specifically designed for self-supervised denoising networks. Finally, we propose a novel regularization optimization function that effectively reduces the disparity between self-supervised and supervised denoising networks, thereby enhancing denoising training. Through comprehensive quantitative and qualitative experiments conducted on both clinical low-dose CT and DR datasets, we demonstrate the remarkable effectiveness and advantages of our FIRE method compared to other state-of-the-art approaches.
Structured State Space Models for Multiple Instance Learning in Digital Pathology

Multiple instance learning is an ideal mode of analysis for histopathology data, where vast whole slide images are typically annotated with a single global label. In such cases, a whole slide image is modelled as a collection of tissue patches to be aggregated and classified. Common models for performing this classification include recurrent neural networks and transformers. Although powerful compression algorithms, such as deep pre-trained neural networks, are used to reduce the dimensionality of each patch, the sequences arising from whole slide images remain excessively long, routinely containing tens of thousands of patches. Structured state space models are an emerging alternative for sequence modelling, specifically designed for the efficient modelling of long sequences. These models invoke an optimal projection of an input sequence into memory units that compress the entire sequence. In this paper, we propose the use of state space models as a multiple instance learner to a variety of problems in digital pathology. Across experiments in metastasis detection, cancer subtyping, mutation classification, and multitask learning, we demonstrate the competitiveness of this new class of models with existing state of the art approaches. Our code is available at https://github.com/MICS-Lab/s4_digital_pathology .