Reviewerly/exHarmony huggingface dataset:
Few Shot Medical Image Segmentation with Cross Attention Transformer

Medical image segmentation has made significant progress in recent years. Deep learning-based methods are recognized as data-hungry techniques, requiring large amounts of data with manual annotations. However, manual annotation is expensive in the field of medical image analysis, which requires domain-specific expertise. To address this challenge, few-shot learning has the potential to learn new classes from only a few examples. In this work, we propose a novel framework for few-shot medical image segme ntation, termed CAT-Net, based on cross masked attention Transformer. Our proposed network mines the correlations between the support image and query image, limiting them to focus only on useful foreground information and boosting the representation capacity of both the support prototype and query features. We further design an iterative refinement framework that refines the query image segmentation iteratively and promotes the support feature in turn. We validated the proposed method on three public datasets: Abd-CT, Abd-MRI, and Card-MRI. Experimental results demonstrate the superior performance of our method compared to state-of-the-art methods and the effectiveness of each component. Source code: https://github.com/hust-linyi/CAT-Net .
OpenAL: An Efficient Deep Active Learning Framework for Open-Set Pathology Image Classification

Active learning (AL) is an effective approach to select the most informative samples to label so as to reduce the annotation cost. Existing AL methods typically work under the closed-set assumption, i.e., all classes existing in the unlabeled sample pool need to be classified by the target model. However, in some practical clinical tasks, the unlabeled pool may contain not only the target classes that need to be fine-grainedly classified, but also non-target classes that are irrelevant to the clinical tasks. Existing AL methods cannot work well in this scenario because they tend to select a large number of non-target samples. In this paper, we formulate this scenario as an open-set AL problem and propose an efficient framework, OpenAL, to address the challenge of querying samples from an unlabeled pool with both target class and non-target class samples. Experiments on fine-grained classification of pathology images show that OpenAL can significantly improve the query quality of target class samples and achieve higher performance than current state-of-the-art AL methods. Code is available at https://github.com/miccaiif/OpenAL .
ModeT: Learning Deformable Image Registration via Motion Decomposition Transformer

The Transformer structures have been widely used in computer vision and have recently made an impact in the area of medical image registration. However, the use of Transformer in most registration networks is straightforward. These networks often merely use the attention mechanism to boost the feature learning as the segmentation networks do, but do not sufficiently design to be adapted for the registration task. In this paper, we propose a novel motion decomposition Transformer (ModeT) to explicitly model multiple motion modalities by fully exploiting the intrinsic capability of the Transformer structure for deformation estimation. The proposed ModeT naturally transforms the multi-head neighborhood attention relationship into the multi-coordinate relationship to model multiple motion modes. Then the competitive weighting module (CWM) fuses multiple deformation sub-fields to generate the resulting deformation field. Extensive experiments on two public brain magnetic resonance imaging (MRI) datasets show that our method outperforms current state-of-the-art registration networks and Transformers, demonstrating the potential of our ModeT for the challenging non-rigid deformation estimation problem. The benchmarks and our code are publicly available at https://github.com/ZAX130/SmileCode .
A Style Transfer-Based Augmentation Framework for Improving Segmentation and Classification Performance Across Different Sources in Ultrasound Images

Ultrasound imaging can vary in style/appearance due to differences in scanning equipment and other factors, resulting in degraded segmentation and classification performance of deep learning models for ultrasound image analysis. Previous studies have attempted to solve this problem by using style transfer and augmentation techniques, but these methods usually require a large amount of data from multiple sources and source-specific discriminators, which are not feasible for medical datasets with limited samples. Moreover, finding suitable augmentation methods for ultrasound data can be difficult. To address these challenges, we propose a novel style transfer-based augmentation framework that consists of three components: mixed style augmentation (MixStyleAug), feature augmentation (FeatAug), and mask-based style augmentation (MaskAug). MixStyleAug uses a style transfer network to transform the style of a training image into various reference styles, which enriches the information from different sources for the network. FeatAug augments the styles at the feature level to compensate for possible style variations, especially for small-size datasets with limited styles. MaskAug leverages segmentation masks to highlight the key regions in the images, which enhances the model's generalizability. We evaluate our framework on five ultrasound datasets collected from different scanners and centers. Our framework outperforms previous methods on both segmentation and classification tasks, especially on small-size datasets. Our results suggest that our framework can effectively improve the performance of deep learning models across different ultrasound sources with limited data.
PLD-AL: Pseudo-label Divergence-Based Active Learning in Carotid Intima-Media Segmentation for Ultrasound Images

Segmentation of the carotid intima-media (CIM) offers more precise morphological evidence for obesity and atherosclerotic disease compared to the method that measures its thickness and roughness during routine ultrasound scans. Although advanced deep learning technology has shown promise in enabling automatic and accurate medical image segmentation, the lack of a large quantity of high-quality CIM labels may hinder the model training process. Active learning (AL) tackles this issue by iteratively annotating the subset whose labels contribute the most to the training performance at each iteration. However, this approach substantially relies on the expert's experience, particularly when addressing ambiguous CIM boundaries that may be present in real-world ultrasound images. Our proposed approach, called pseudo-label divergence-based active learning (PLD-AL), aims to train segmentation models using a gradually enlarged and refined labeled pool. The approach has an outer and an inner loops: The outer loop calculates the Kullback-Leibler (KL) divergence of predictive pseudo-labels related to two consecutive AL iterations. It determines which portion of the unlabeled pool should be annotated by an expert. The inner loop trains two networks: The student network is fully trained on the current labeled pool, while the teacher network is weighted upon itself and the student one, ultimately refining the labeled pool. We evaluated our approach using both the Carotid Ultrasound Boundary Study dataset and an in-house dataset from Children's Hospital, Zhejiang University School of Medicine. Our results demonstrate that our approach outperforms state-of-the-art AL approaches. Furthermore, the visualization results show that our approach less over-estimates the CIM area than the rest methods, especially for severely ambiguous ultrasound images at the thickness direction.
FSDiffReg: Feature-Wise and Score-Wise Diffusion-Guided Unsupervised Deformable Image Registration for Cardiac Images

Unsupervised deformable image registration is one of the challenging tasks in medical imaging. Obtaining a high-quality deformation field while preserving deformation topology remains demanding amid a series of deep-learning-based solutions. Meanwhile, the diffusion model's latent feature space shows potential in modeling the deformation semantics. To fully exploit the diffusion model's ability to guide the registration task, we present two modules: Feature-wise Diffusion-Guided Module (FDG) and Score-wise Diffusion-Guided Module (SDG). Specifically, FDG uses the diffusion model's multi-scale semantic features to guide the generation of the deformation field. SDG uses the diffusion score to guide the optimization process for preserving deformation topology with barely any additional computation. Experiment results on the 3D medical cardiac image registration task validate our model's ability to provide refined deformation fields with preserved topology effectively. Code is available at: https://github.com/xmed-lab/FSDiffReg.git .
DISA: DIfferentiable Similarity Approximation for Universal Multimodal Registration

Multimodal image registration is a challenging but essential step for numerous image-guided procedures. Most registration algorithms rely on the computation of complex, frequently non-differentiable similarity metrics to deal with the appearance discrepancy of anatomical structures between imaging modalities. Recent Machine Learning based approaches are limited to specific anatomy-modality combinations and do not generalize to new settings. We propose a generic framework for creating expressive cross-modal descriptors that enable fast deformable global registration. We achieve this by approximating existing metrics with a dot-product in the feature space of a small convolutional neural network (CNN) which is inherently differentiable can be trained without registered data. Our method is several orders of magnitude faster than local patch-based metrics and can be directly applied in clinical settings by replacing the similarity measure with the proposed one. Experiments on three different datasets demonstrate that our approach generalizes well beyond the training data, yielding a broad capture range even on unseen anatomies and modality pairs, without the need for specialized retraining. We make our training code and data publicly available.
Learnable Query Initialization for Surgical Instrument Instance Segmentation

Surgical tool classification and instance segmentation are crucial for minimally invasive surgeries and related applications. Though most of the state-of-the-art for instance segmentation in natural images use transformer-based architectures, they have not been successful for medical instruments. In this paper, we investigate the reasons for the failure. Our analysis reveals that this is due to incorrect query initialization, which is unsuitable for fine-grained classification of highly occluded objects in a low data setting, typical for medical instruments. We propose a class-agnostic Query Proposal Network (QPN) to improve query initialization inputted to the decoder layers. Towards this, we propose a deformable-cross-attention-based learnable Query Proposal Decoder (QPD). The proposed QPN improves the recall rate of the query initialization by 44.89% at 0.9 IOU. This leads to an improvement in segmentation performance by 1.84% on Endovis17 and 2.09% on Endovis18 datasets, as measured by ISI-IOU. The source code can be accessed at https://aineurosurgery.github.io/learnableQPD .
Synthesising Rare Cataract Surgery Samples with Guided Diffusion Models

Cataract surgery is a frequently performed procedure that demands automation and advanced assistance systems. However, gathering and annotating data for training such systems is resource intensive. The publicly available data also comprises severe imbalances inherent to the surgical process. Motivated by this, we analyse cataract surgery video data for the worst-performing phases of a pre-trained downstream tool classifier. The analysis demonstrates that imbalances deteriorate the classifier's performance on underrepresented cases. To address this challenge, we utilise a conditional generative model based on Denoising Diffusion Implicit Models (DDIM) and Classifier-Free Guidance (CFG). Our model can synthesise diverse, high-quality examples based on complex multi-class multi-label conditions, such as surgical phases and combinations of surgical tools. We affirm that the synthesised samples display tools that the classifier recognises. These samples are hard to differentiate from real images, even for clinical experts with more than five years of experience. Further, our synthetically extended data can improve the data sparsity problem for the downstream task of tool classification. The evaluations demonstrate that the model can generate valuable unseen examples, allowing the tool classifier to improve by up to 10% for rare cases. Overall, our approach can facilitate the development of automated assistance systems for cataract surgery by providing a reliable source of realistic synthetic data, which we make available for everyone.
CycleSTTN: A Learning-Based Temporal Model for Specular Augmentation in Endoscopy

Feature detection and matching is a computer vision problem that underpins different computer assisted techniques in endoscopy, including anatomy and lesion recognition, camera motion estimation, and 3D reconstruction. This problem is made extremely challenging due to the abundant presence of specular reflections. Most of the solutions proposed in the literature are based on filtering or masking out these regions as an additional processing step. There has been little investigation into explicitly learning robustness to such artefacts with single-step end-to-end training. In this paper, we propose an augmentation technique (CycleSTTN) that adds temporally consistent and realistic specularities to endoscopic videos. Such videos can act as ground truth data with known texture occluded behind the added specularities. We demonstrate that our image generation technique produces better results than a standard CycleGAN model. Additionally, we leverage this data augmentation to re-train a deep-learning based feature extractor (SuperPoint) and show that it improves. CycleSTTN code is made available here .
Progressively Coupling Network for Brain MRI Registration in Few-Shot Situation

Segmentation-assisted registration models can leverage few available labels in exchange for large performance gains by their complementarity. Recent related works independently build the prediction branches of deformation field and segmentation label without any information interaction except for the joint supervision. They ignore underlying relationship between the two tasks, thereby failing to fully exploit their complementary nature. To this end, we propose a ProGressively Coupling Network (PGCNet) that relies on segmentation to regularize the correct projecting of registration. Our overall framework is a multi-task learning paradigm in which features are extracted by one shared encoder and then separate prediction branches are built for segmentation and registration. In the prediction phase, we utilize the bidirectional deformation fields as bridges to warp the features of moving and fixed images to each other's segmentation branches, thereby progressively and interactively supplementing additional context information at multiple levels for their segmentation. By establishing the entangled correspondence, segmentation supervision can indirectly regularize registration stream to accurately project semantic layout for segmentation branches. In addition, we design the position correlation calculation for registration to easier capture the spatial correlation of the images from the shared features. Experimental results on public 3D brain MRI datasets show that our work performs favorably against the state-of-the-art methods.
Continual Learning for Abdominal Multi-organ and Tumor Segmentation

The ability to dynamically extend a model to new data and classes is critical for multiple organ and tumor segmentation. However, due to privacy regulations, accessing previous data and annotations can be problematic in the medical domain. This poses a significant barrier to preserving the high segmentation accuracy of the old classes when learning from new classes because of the catastrophic forgetting problem. In this paper, we first empirically demonstrate that simply using high-quality pseudo labels can fairly mitigate this problem in the setting of organ segmentation. Furthermore, we put forward an innovative architecture designed specifically for continuous organ and tumor segmentation, which incurs minimal computational overhead. Our proposed design involves replacing the conventional output layer with a suite of lightweight, class-specific heads, thereby offering the flexibility to accommodate newly emerging classes. These heads enable independent predictions for newly introduced and previously learned classes, effectively minimizing the impact of new classes on old ones during the course of continual learning. We further propose incorporating Contrastive Language-Image Pretraining (CLIP) embeddings into the organ-specific heads. These embeddings encapsulate the semantic information of each class, informed by extensive image-text co-training. The proposed method is evaluated on both in-house and public abdominal CT datasets under organ and tumor segmentation tasks. Empirical results suggest that the proposed design improves the segmentation performance of a baseline model on newly-introduced and previously-learned classes along the learning trajectory.
Mining Specification Parameters for Multi-class Classification

We present a method for mining parameters of temporal specifications for signal classification. Given a parametric formula and a set of labeled traces, we find one parameter valuation for each class and use it to instantiate the specification template. The resulting formula characterizes the signals in a class by discriminating them from signals of other classes. We propose a two-step approach: first, for each class, we approximate its validity domain, which is the region of the valuations that render the formula satisfied. Second, we select from each validity domain the valuation that maximizes the distance from the validity domain of other classes. We provide a statistical guarantee that the selected parameter valuation is at a bounded distance from being optimal. Finally, we validate our approach on three case studies from different application domains.
Monitoring Algorithmic Fairness Under Partial Observations

As AI and machine-learned software are used increasingly for making decisions that affect humans, it is imperative that they remain fair and unbiased in their decisions. To complement design-time bias mitigation measures, runtime verification techniques have been introduced recently to monitor the algorithmic fairness of deployed systems. Previous monitoring techniques assume full observability of the states of the (unknown) monitored system. Moreover, they can monitor only fairness properties that are specified as arithmetic expressions over the probabilities of different events. In this work, we extend fairness monitoring to systems modeled as partially observed Markov chains (POMC), and to specifications containing arithmetic expressions over the expected values of numerical functions on event sequences. The only assumptions we make are that the underlying POMC is aperiodic and starts in the stationary distribution, with a bound on its mixing time being known. These assumptions enable us to estimate a given property for the entire distribution of possible executions of the monitored POMC, by observing only a single execution. Our monitors observe a long run of the system and, after each new observation, output updated PAC-estimates of how fair or biased the system is. The monitors are computationally lightweight and, using a prototype implementation, we demonstrate their effectiveness on several real-world examples.
SFusion: Self-attention Based N-to-One Multimodal Fusion Block

People perceive the world with different senses, such as sight, hearing, smell, and touch. Processing and fusing information from multiple modalities enables Artificial Intelligence to understand the world around us more easily. However, when there are missing modalities, the number of available modalities is different in diverse situations, which leads to an N-to-One fusion problem. To solve this problem, we propose a self-attention based fusion block called SFusion. Different from preset formulations or convolution based methods, the proposed block automatically learns to fuse available modalities without synthesizing or zero-padding missing ones. Specifically, the feature representations extracted from upstream processing model are projected as tokens and fed into self-attention module to generate latent multimodal correlations. Then, a modal attention mechanism is introduced to build a shared representation, which can be applied by the downstream decision model. The proposed SFusion can be easily integrated into existing multimodal analysis networks. In this work, we apply SFusion to different backbone networks for human activity recognition and brain tumor segmentation tasks. Extensive experimental results show that the SFusion block achieves better performance than the competing fusion strategies. Our code is available at https://github.com/scut-cszcl/SFusion .
Nonuniformly Spaced Control Points Based on Variational Cardiac Image Registration

Non-uniformly spaced control points located on the interface of different objects are beneficial for constructing an accurate displacement field for image registration. However, extracting features of non-uniformly spaced control points in images is challenging for convolutional neural networks (CNNs). We extend a probabilistic image registration model using uniformed-spaced control points by employing non-uniformly-spaced control points. We construct a network to extract the image and spatial features of non-uniformly-spaced control points. Moreover, a variational Bayesian (VB) model using a factorized prior is employed to estimate the distribution of latent variables. In theory, we analyze the KL divergence between the posterior and the two separated priors. We found that the factorized prior has the advantage of decreasing the KL divergence, but too more factorized priors, such as the standard normal, might deteriorate registration accuracy. Moreover, we analyze the relationship between the uncertainty of the displacement field and the spatial distribution of control points. Experimental results on four public datasets show that our network outperforms the state-of-arts registration networks and can provide registration uncertainty.
Deep Unsupervised Clustering for Conditional Identification of Subgroups Within a Digital Pathology Image Set

Consideration of subgroups or domains within medical image datasets is crucial for the development and evaluation of robust and generalizable machine learning systems. To tackle the domain identification problem, we examine deep unsupervised generative clustering approaches for representation learning and clustering. The Variational Deep Embedding (VaDE) model is trained to learn lower-dimensional representations of images based on a Mixture-of-Gaussians latent space prior distribution while optimizing cluster assignments. We propose the Conditionally Decoded Variational Deep Embedding (CDVaDE) model which incorporates additional variables of choice, such as the class labels, as conditioning factors to guide the clustering towards subgroup structures in the data which have not been known or recognized previously. We analyze the behavior of CDVaDE on multiple datasets and compare it to other deep clustering algorithms. Our experimental results demonstrate that the considered models are capable of separating digital pathology images into meaningful subgroups. We provide a general-purpose implementation of all considered deep clustering methods as part of the open source Python package DomId ( https://github.com/DIDSR/DomId ).