Reviewerly/exHarmony huggingface dataset:
Scaling up 3D Kernels with Bayesian Frequency Re-parameterization for Medical Image Segmentation

With the inspiration of vision transformers, the concept of depth-wise convolution revisits to provide a large Effective Receptive Field (ERF) using Large Kernel (LK) sizes for medical image segmentation. However, the segmentation performance might be saturated and even degraded as the kernel sizes scaled up (e.g., $$21\times 21\times 21$$ ) in a Convolutional Neural Network (CNN). We hypothesize that convolution with LK sizes is limited to maintain an optimal convergence for locality learning. While Structural Re-parameterization (SR) enhances the local convergence with small kernels in parallel, optimal small kernel branches may hinder the computational efficiency for training. In this work, we propose RepUX-Net, a pure CNN architecture with a simple large kernel block design, which competes favorably with current network state-of-the-art (SOTA) (e.g., 3D UX-Net, SwinUNETR) using 6 challenging public datasets. We derive an equivalency between kernel re-parameterization and the branch-wise variation in kernel convergence. Inspired by the spatial frequency in the human visual system, we extend to vary the kernel convergence into element-wise setting and model the spatial frequency as a Bayesian prior to re-parameterize convolutional weights during training. Specifically, a reciprocal function is leveraged to estimate a frequency-weighted value, which rescales the corresponding kernel element for stochastic gradient descent. From the experimental results, RepUX-Net consistently outperforms 3D SOTA benchmarks with internal validation (FLARE: 0.929 to 0.944), external validation (MSD: 0.901 to 0.932, KiTS: 0.815 to 0.847, LiTS: 0.933 to 0.949, TCIA: 0.736 to 0.779) and transfer learning (AMOS: 0.880 to 0.911) scenarios in Dice Score. Both codes and pre-trained models are available at: https://github.com/MASILab/RepUX-Net .
Mammo-Net: Integrating Gaze Supervision and Interactive Information in Multi-view Mammogram Classification

Breast cancer diagnosis is a challenging task. Recently, the application of deep learning techniques to breast cancer diagnosis has become a popular trend. However, the effectiveness of deep neural networks is often limited by the lack of interpretability and the need for significant amount of manual annotations. To address these issues, we present a novel approach by leveraging both gaze data and multi-view data for mammogram classification. The gaze data of the radiologist serves as a low-cost and simple form of coarse annotation, which can provide rough localizations of lesions. We also develop a pyramid loss better fitting to the gaze-supervised process. Moreover, considering many studies overlooking interactive information relevant to diagnosis, we accordingly utilize transformer-based attention in our network to mutualize multi-view pathological information, and further employ a bidirectional fusion learning (BFL) to more effectively fuse multi-view information. Experimental results demonstrate that our proposed model significantly improves both mammogram classification performance and interpretability through incorporation of gaze data and cross-view interactive information.
Devil is in Channels: Contrastive Single Domain Generalization for Medical Image Segmentation

Deep learning-based medical image segmentation models suffer from performance degradation when deployed to a new healthcare center. To address this issue, unsupervised domain adaptation and multi-source domain generalization methods have been proposed, which, however, are less favorable for clinical practice due to the cost of acquiring target-domain data and the privacy concerns associated with redistributing the data from multiple source domains. In this paper, we propose a Channel-level Contrastive Single Domain Generalization (C $$^2$$ SDG) model for medical image segmentation. In C $$^2$$ SDG, the shallower features of each image and its style-augmented counterpart are extracted and used for contrastive training, resulting in the disentangled style representations and structure representations. The segmentation is performed based solely on the structure representations. Our method is novel in the contrastive perspective that enables channel-wise feature disentanglement using a single source domain. We evaluated C $$^2$$ SDG against six SDG methods on a multi-domain joint optic cup and optic disc segmentation benchmark. Our results suggest the effectiveness of each module in C $$^2$$ SDG and also indicate that C $$^2$$ SDG outperforms the baseline and all competing methods with a large margin. The code is available at https://github.com/ShishuaiHu/CCSDG .
DAST: Differentiable Architecture Search with Transformer for 3D Medical Image Segmentation

Neural Architecture Search (NAS) has been widely used for medical image segmentation by improving both model performance and computational efficiency. Recently, the Visual Transformer (ViT) model has achieved significant success in computer vision tasks. Leveraging these two innovations, we propose a novel NAS algorithm, DAST, to optimize neural network models with transformers for 3D medical image segmentation. The proposed algorithm is able to search the global structure and local operations of the architecture with a GPU memory consumption constraint. The resulting architectures reveal an effective relationship between convolution and transformer layers in segmentation models. Moreover, we validate the proposed algorithm on large-scale medical image segmentation data sets, showing its superior performance over the baselines. The model achieves state-of-the-art performance in the public challenge of kidney CT segmentation (KiTS'19).
CT Kernel Conversion Using Multi-domain Image-to-Image Translation with Generator-Guided Contrastive Learning

Computed tomography (CT) image can be reconstructed by various types of kernels depending on what anatomical structure is evaluated. Also, even if the same anatomical structure is analyzed, the kernel being used differs depending on whether it is qualitative or quantitative evaluation. Thus, CT images reconstructed with different kernels would be necessary for accurate diagnosis. However, once CT image is reconstructed with a specific kernel, the CT raw data, sinogram is usually removed because of its large capacity and limited storage. To solve this problem, many methods have been proposed by using deep learning approach using generative adversarial networks in image-to-image translation for kernel conversion. Nevertheless, it is still challenging task that translated image should maintain the anatomical structure of source image in medical domain. In this study, we propose CT kernel conversion method using multi-domain image-to-image translation with generator-guided contrastive learning. Our proposed method maintains the anatomical structure of the source image accurately and can be easily utilized into other multi-domain image-to-image translation methods with only changing the discriminator architecture and without adding any additional networks. Experimental results show that our proposed method can translate CT images from sharp into soft kernels and from soft into sharp kernels compared to other image-to-image translation methods. Our code is available at https://github.com/cychoi97/GGCL .
SPR-Net: Structural Points Based Registration for Coronary Arteries Across Systolic and Diastolic Phases

Systolic and diastolic registration of coronary arteries is a critical yet challenging step in coronary artery disease analysis. Most existing methods ignore the important relationship between vascular geometric shape and image contextual information in the two phases, leading to limited performance. In this paper, we propose a novel structural point registration network, which comprehensively captures both point-level geometric features and image-level semantic features as enriched feature representations to assist coronary registration. Specifically, given the systolic and diastolic CCTA images, our method improves coronary artery registration from three aspects. First, the point cloud encoder learns the spatial geometric features of the points in the 3D coronary mask to effectively capture the vascular shape representation. Second, a vision transformer (ViT) is employed to extract the image semantic information as a complementary condition of the geometric features to identify the bi-phasic correspondence of different vascular branches. Third, we design a transformer module to fuse the features across points and images to obtain the corresponding structural points in the two phases and then use structural points to guide the coronary artery registration via the thin-plate spline (TPS) method. We evaluated our method on a real-clinical dataset. Extensive experiments show that our proposed method significantly outperforms the state-of-the-art methods in coronary artery registration.
3D Medical Image Segmentation with Sparse Annotation via Cross-Teaching Between 3D and 2D Networks

Medical image segmentation typically necessitates a large and precisely annotated dataset. However, obtaining pixel-wise annotation is a labor-intensive task that requires significant effort from domain experts, making it challenging to obtain in practical clinical scenarios. In such situations, reducing the amount of annotation required is a more practical approach. One feasible direction is sparse annotation, which involves annotating only a few slices, and has several advantages over traditional weak annotation methods such as bounding boxes and scribbles, as it preserves exact boundaries. However, learning from sparse annotation is challenging due to the scarcity of supervision signals. To address this issue, we propose a framework that can robustly learn from sparse annotation using the cross-teaching of both 3D and 2D networks. Considering the characteristic of these networks, we develop two pseudo label selection strategies, which are hard-soft confidence threshold and consistent label fusion. Our experimental results on the MMWHS dataset demonstrate that our method outperforms the state-of-the-art (SOTA) semi-supervised segmentation methods. Moreover, our approach achieves results that are comparable to the fully-supervised upper bound result. Our code is available at https://github.com/HengCai-NJU/3D2DCT .
Anti-adversarial Consistency Regularization for Data Augmentation: Applications to Robust Medical Image Segmentation

Modern deep learning methods for semantic segmentation require labor-intensive labeling for large-scale datasets with dense pixel-level annotations. Recent data augmentation methods such as dropping, mixing image patches, and adding random noises suggest effective ways to address the labeling issues for natural images. However, they can only be restrictively applied to medical image segmentation as they carry risks of distorting or ignoring the underlying clinical information of local regions of interest in an image. In this paper, we propose a novel data augmentation method for medical image segmentation without losing the semantics of the key objects (e.g., polyps). This is achieved by perturbing the objects with quasi-imperceptible adversarial noises and training a network to expand discriminative regions with a guide of anti-adversarial noises. Such guidance can be realized by a consistency regularization between the two contrasting data, and the strength of regularization is automatically and adaptively controlled considering their prediction uncertainty. Our proposed method significantly outperforms various existing methods with high sensitivity and Dice scores and extensive experiment results with multiple backbones on two datasets validate its effectiveness.
DiffuseIR: Diffusion Models for Isotropic Reconstruction of 3D Microscopic Images

Three-dimensional microscopy is often limited by anisotropic spatial resolution, resulting in lower axial resolution than lateral resolution. Current State-of-The-Art (SoTA) isotropic reconstruction methods utilizing deep neural networks can achieve impressive super-resolution performance in fixed imaging settings. However, their generality in practical use is limited by degraded performance caused by artifacts and blurring when facing unseen anisotropic factors. To address these issues, we propose DiffuseIR, an unsupervised method for isotropic reconstruction based on diffusion models. First, we pre-train a diffusion model to learn the structural distribution of biological tissue from lateral microscopic images, resulting in generating naturally high-resolution images. Then we use low-axial-resolution microscopy images to condition the generation process of the diffusion model and generate high-axial-resolution reconstruction results. Since the diffusion model learns the universal structural distribution of biological tissues, which is independent of the axial resolution, DiffuseIR can reconstruct authentic images with unseen low-axial resolutions into a high-axial resolution without requiring re-training. The proposed DiffuseIR achieves SoTA performance in experiments on EM data and can even compete with supervised methods.
A Self-supervised Approach for Detecting the Edges of Haustral Folds in Colonoscopy Video

Providing 3D navigation in colonoscopy can help decrease diagnostic miss rates in cancer screening by building a coverage map of the colon as the endoscope navigates the anatomy. However, this task is made challenging by the lack of discriminative localisation landmarks throughout the colon. While standard navigation techniques rely on sparse point landmarks or dense pixel registration, we propose edges as a more natural visual landmark to characterise the haustral folds of the colon anatomy. We propose a self-supervised methodology to train an edge detection method for colonoscopy imaging, demonstrating that it can effectively detect anatomy related edges while ignoring light reflection artifacts abundant in colonoscopy. We also propose a metric to evaluate the temporal consistency of estimated edges in the absence of real groundtruth. We demonstrate our results on video sequences from the public dataset HyperKvazir. Our code and pseudo-groundtruth edge labels are available at https://github.com/jwyhhh123/HaustralFold_Edge_Detector .
Geometry-Adaptive Network for Robust Detection of Placenta Accreta Spectrum Disorders

Placenta accreta spectrum (PAS) is a high-risk obstetric disorder associated with significant morbidity and mortality. Since the abnormal invasion usually occurs near the uteroplacental interface, there is a large geometry variation in the lesion bounding boxes, which considerably degrades the detection performance. In addition, due to the confounding visual representations of PAS, the diagnosis highly depends on the clinical experience of radiologists, which easily results in inaccurate bounding box annotations. In this paper, we propose a geometry-adaptive network for robust PAS detection. Specifically, to deal with the geometric prior missing problem, we design a Geometry-adaptive Label Assignment (GA-LA) strategy and a Geometry-adaptive RoI Fusion (GA-RF) module. The GA-LA strategy dynamically selects positive PAS candidates (RoIs) for each lesion according to its shape information. The GA-RF module aggregates the multi-scale RoI features based on the geometry distribution of proposals. Moreover, we develop a Lesion-aware Detection Head (LA-Head) to leverage high-quality predictions to iteratively refine inaccurate annotations with a novel multiple instance learning paradigm. Experimental results under both clean and noisy labels indicate that our method achieves state-of-the-art performance and demonstrate promising assistance for PAS diagnosis in clinical applications.
Dual Arbitrary Scale Super-Resolution for Multi-contrast MRI

Limited by imaging systems, the reconstruction of Magnetic Resonance Imaging (MRI) images from partial measurement is essential to medical imaging research. Benefiting from the diverse and complementary information of multi-contrast MR images in different imaging modalities, multi-contrast Super-Resolution (SR) reconstruction is promising to yield SR images with higher quality. In the medical scenario, to fully visualize the lesion, radiologists are accustomed to zooming the MR images at arbitrary scales rather than using a fixed scale, as used by most MRI SR methods. In addition, existing multi-contrast MRI SR methods often require a fixed resolution for the reference image, which makes acquiring reference images difficult and imposes limitations on arbitrary scale SR tasks. To address these issues, we proposed an implicit neural representations based dual-arbitrary multi-contrast MRI super-resolution method, called Dual-ArbNet. First, we decouple the resolution of the target and reference images by a feature encoder, enabling the network to input target and reference images at arbitrary scales. Then, an implicit fusion decoder fuses the multi-contrast features and uses an Implicit Decoding Function (IDF) to obtain the final MRI SR results. Furthermore, we introduce a curriculum learning strategy to train our network, which improves the generalization and performance of our Dual-ArbNet. Extensive experiments in two public MRI datasets demonstrate that our method outperforms state-of-the-art approaches under different scale factors and has great potential in clinical practice.
Adult-Like Phase and Multi-scale Assistance for Isointense Infant Brain Tissue Segmentation

Precise brain tissue segmentation is crucial for infant development tracking and early brain disorder diagnosis. However, it remains challenging to automatically segment the brain tissues of a 6-month-old infant (isointense phase), even for manual labeling, due to inherent ongoing myelination during the first postnatal year. The intensity contrast between gray matter and white matter is extremely low in isointense MRI data. To resolve this problem, in this study, we propose a novel network with multi-phase data and multi-scale assistance to accurately segment the brain tissues of the isointense phase. Specifically, our framework consists of two main modules, i.e., semantics-preserved generative adversarial network (SPGAN) and Transformer-based multi-scale segmentation network (TMSN). SPAGN bi-directionally transfers the brain appearance between the isointense phase and the adult-like phase. On the one hand, the synthesized isointense phase data augments the isointense dataset. On the other hand, the synthesized adult-like images provide prior knowledge to the ambiguous tissue boundaries in the paired isointense phase data. TMSN integrates features of multi-phase image pairs in a multi-scale manner, which exploits both the adult-like phase data, with much clearer boundaries as structural prior, and the surrounding tissues, with a larger receptive field to assist the isointense data tissue segmentation. Extensive experiments on the public dataset show that our proposed framework achieves significant improvement over the state-of-the-art methods quantitatively and qualitatively.
Procedurally Generated Colonoscopy and Laparoscopy Data for Improved Model Training Performance

The use of synthetic/simulated data can greatly improve model training performance, especially in areas such as image guided surgery, where real training data can be difficult to obtain, or of limited size. Procedural generation of data allows for large datasets to be rapidly generated and automatically labelled, while also randomising relevant parameters within the simulation to provide a wide variation in models and textures used in the scene. A method for procedural generation of both textures and geometry for IGS data is presented, using Blender Shader Graphs and Geometry Nodes, with synthetic datasets used to pre-train models for polyp detection (YoloV7) and organ segmentation (UNet), with performance evaluated on open-source datasets. Pre-training models with synthetic data significantly improves both model performance and generalisability (i.e. performance when evaluated on other datasets). Mean DICE score across all models for liver segmentation increased by 15% (p=0.02) after pre-training on synthetic data. For polyp detection, Precision increased by 11% (p=0.002), Recall by 9% (p=0.01), mAP@.5 by 10% (p=0.01) and mAP@[.5:95] by 8% (p-0.003). All synthetic data, as well as examples of different Shader Graph/Geometry Node operations can be downloaded at https://doi.org/10.5522/04/23843904 .
Category-Level Regularized Unlabeled-to-Labeled Learning for Semi-supervised Prostate Segmentation with Multi-site Unlabeled Data

Segmenting prostate from MRI is crucial for diagnosis and treatment planning of prostate cancer. Given the scarcity of labeled data in medical imaging, semi-supervised learning (SSL) presents an attractive option as it can utilize both limited labeled data and abundant unlabeled data. However, if the local center has limited image collection capability, there may also not be enough unlabeled data for semi-supervised learning to be effective. To overcome this issue, other partner centers can be consulted to help enrich the pool of unlabeled images, but this can result in data heterogeneity, which could hinder SSL that functions under the assumption of consistent data distribution. Tailoring for this important yet under-explored scenario, this work presents a novel Category-level regularized Unlabeled-to-Labeled (CU2L) learning framework for semi-supervised prostate segmentation with multi-site unlabeled MRI data. Specifically, CU2L is built upon the teacher-student architecture with the following tailored learning processes: (i) local pseudo-label learning for reinforcing confirmation of the data distribution of the local center; (ii) category-level regularized non-parametric unlabeled-to-labeled learning for robustly mining shared information by using the limited expert labels to regularize the intra-class features across centers to be discriminative and generalized; (iii) stability learning under perturbations to further enhance robustness to heterogeneity. Our method is evaluated on prostate MRI data from six different clinical centers and shows superior performance compared to other semi-supervised methods.
Unified Brain MR-Ultrasound Synthesis Using Multi-modal Hierarchical Representations

We introduce MHVAE, a deep hierarchical variational auto-encoder (VAE) that synthesizes missing images from various modalities. Extending multi-modal VAEs with a hierarchical latent structure, we introduce a probabilistic formulation for fusing multi-modal images in a common latent representation while having the flexibility to handle incomplete image sets as input. Moreover, adversarial learning is employed to generate sharper images. Extensive experiments are performed on the challenging problem of joint intra-operative ultrasound (iUS) and Magnetic Resonance (MR) synthesis. Our model outperformed multi-modal VAEs, conditional GANs, and the current state-of-the-art unified method (ResViT) for synthesizing missing images, demonstrating the advantage of using a hierarchical latent representation and a principled probabilistic fusion operation. Our code is publicly available ( https://github.com/ReubenDo/MHVAE ).