Reviewerly/exHarmony huggingface dataset:
Rectifying Noisy Labels with Sequential Prior: Multi-scale Temporal Feature Affinity Learning for Robust Video Segmentation

Noisy label problems are inevitably in existence within medical image segmentation causing severe performance degradation. Previous segmentation methods for noisy label problems only utilize a single image while the potential of leveraging the correlation between images has been overlooked. Especially for video segmentation, adjacent frames contain rich contextual information beneficial in cognizing noisy labels. Based on two insights, we propose a Multi-Scale Temporal Feature Affinity Learning (MS-TFAL) framework to resolve noisy-labeled medical video segmentation issues. First, we argue the sequential prior of videos is an effective reference, i.e., pixel-level features from adjacent frames are close in distance for the same class and far in distance otherwise. Therefore, Temporal Feature Affinity Learning (TFAL) is devised to indicate possible noisy labels by evaluating the affinity between pixels in two adjacent frames. We also notice that the noise distribution exhibits considerable variations across video, image, and pixel levels. In this way, we introduce Multi-Scale Supervision (MSS) to supervise the network from three different perspectives by re-weighting and refining the samples. This design enables the network to concentrate on clean samples in a coarse-to-fine manner. Experiments with both synthetic and real-world label noise demonstrate that our method outperforms recent state-of-the-art robust segmentation approaches. Code is available at https://github.com/BeileiCui/MS-TFAL .
A Semantic-Guided and Knowledge-Based Generative Framework for Orthodontic Visual Outcome Preview

Orthodontic treatment typically lasts for two years, and its outcome cannot be predicted intuitively in advance. In this paper, we propose a semantic-guided and knowledge-based generative framework to predict the visual outcome of orthodontic treatment from a single frontal photo. The framework involves four steps. Firstly, we perform tooth semantic segmentation and mouth cavity segmentation and extract category-specific teeth contours from frontal images. Secondly, we deform the established tooth-row templates to match the projected contours with the detected ones to reconstruct 3D teeth models. Thirdly, we apply a teeth alignment algorithm to simulate the orthodontic treatment. Finally, we train a semantic-guided generative adversarial network to predict the visual outcome of teeth alignment. Quantitative tests are conducted to evaluate the proposed framework, and the results are as follows: the tooth semantic segmentation model achieves a mean intersection of union of 0.834 for the anterior teeth, the average symmetric surface distance error of our 3D teeth reconstruction method is 0.626 mm on the test cases, and the image generation model has an average Fréchet inception distance of 6.847 over all the test images. These evaluation results demonstrate the practicality of our framework in orthodontics.
SHISRCNet: Super-Resolution and Classification Network for Low-Resolution Breast Cancer Histopathology Image

The rapid identification and accurate diagnosis of breast cancer, known as the killer of women, have become greatly significant for those patients. Numerous breast cancer histopathological image classification methods have been proposed. But they still suffer from two problems. (1) These methods can only hand high-resolution (HR) images. However, the low-resolution (LR) images are often collected by the digital slide scanner with limited hardware conditions. Compared with HR images, LR images often lose some key features like texture, which deeply affects the accuracy of diagnosis. (2) The existing methods have fixed receptive fields, so they can not extract and fuse multi-scale features well for images with different magnification factors. To fill these gaps, we present a Single Histopathological Image Super-Resolution Classification network (SHISRCNet), which consists of two modules: Super-Resolution (SR) and Classification (CF) modules. SR module reconstructs LR images into SR ones. CF module extracts and fuses the multi-scale features of SR images for classification. In the training stage, we introduce HR images into the CF module to enhance SHISRCNet's performance. Finally, through the joint training of these two modules, super-resolution and classified of LR images are integrated into our model. The experimental results demonstrate that the effects of our method are close to the SOTA methods with taking HR images as inputs.
Monitoring Hyperproperties with Prefix Transducers

Abstract Hyperproperties are properties that relate multiple execution traces. Previous work on monitoring hyperproperties focused on synchronous hyperproperties, usually specified in HyperLTL. When monitoring synchronous hyperproperties, all traces are assumed to proceed at the same speed. We introduce (multi-trace) prefix transducers and show how to use them for monitoring synchronous as well as, for the first time, asynchronous hyperproperties. Prefix transducers map multiple input traces into one or more output traces by incrementally matching prefixes of the input traces against expressions similar to regular expressions. The prefixes of different traces which are consumed by a single matching step of the monitor may have different lengths. The deterministic and executable nature of prefix transducers makes them more suitable as an intermediate formalism for runtime verification than logical specifications, which tend to be highly non-deterministic, especially in the case of asynchronous hyperproperties. We report on a set of experiments about monitoring asynchronous version of observational determinism.
Neural LerPlane Representations for Fast 4D Reconstruction of Deformable Tissues

Reconstructing deformable tissues from endoscopic stereo videos in robotic surgery is crucial for various clinical applications. However, existing methods relying only on implicit representations are computationally expensive and require dozens of hours, which limits further practical applications. To address this challenge, we introduce LerPlane, a novel method for fast and accurate reconstruction of surgical scenes under a single-viewpoint setting. LerPlane treats surgical procedures as 4D volumes and factorizes them into explicit 2D planes of static and dynamic fields, leading to a compact memory footprint and significantly accelerated optimization. The efficient factorization is accomplished by fusing features obtained through linear interpolation of each plane and enables using lightweight neural networks to model surgical scenes. Besides, LerPlane shares static fields, significantly reducing the workload of dynamic tissue modeling. We also propose a novel sample scheme to boost optimization and improve performance in regions with tool occlusion and large motions. Experiments on DaVinci robotic surgery videos demonstrate that LerPlane accelerates optimization by over 100 $$\times $$ while maintaining high quality across various non-rigid deformations, showing significant promise for future intraoperative surgery applications.
Transferability-Guided Multi-source Model Adaptation for Medical Image Segmentation

Unsupervised domain adaptation has drawn sustained attentions in medical image segmentation by transferring knowledge from labeled source data to unlabeled target domain. However, most existing approaches assume the source data are collected from a single client, which cannot be successfully applied to explore complementary transferable knowledge from multiple source domains with large distribution discrepancy. Moreover, they require access to source data during training, which is inefficient and unpractical due to privacy preservation and memory storage. To address these challenges, we study a novel and practical problem, named multi-source model adaptation (MSMA), which aims to transfer multiple source models to the unlabeled target domain without any source data. Since no target label and source data is provided to evaluate the transferability of each source model or domain gap between the source and the target domain, we may encounter negative transfer by those less related source domains, thus hurting target performance. To solve this problem, we propose a transferability-guided model adaptation (TGMA) framework to eliminate negative transfer. Specifically, 1) A label-free transferability metric (LFTM) is designed to evaluate transferability of source models without target annotations for the first time. 2) Based on the designed metric, we compute instance-level transferability matrix (ITM) for target pseudo label correction and domain-level transferability matrix (DTM) to achieve model selection for better target model initialization. Extensive experiments on multi-site prostate segmentation dataset demonstrate the superiority of our framework.
PIViT: Large Deformation Image Registration with Pyramid-Iterative Vision Transformer

Large deformation image registration is a challenging task in medical image registration. Iterative registration and pyramid registration are two common CNN-based methods for the task. However, these methods usually consume more parameters and time. Additionally, the existing CNN-based registration methods mainly focus on local feature extraction, limiting their ability to capture the long-distance correlation between image pairs. In this paper, we propose a fast and accurate learning-based algorithm, Pyramid-Iterative Vision Transformer (PIViT), for 3D large deformation medical image registration. Our method constructs a novel pyramid iterative composite structure to solve large deformation problem by using low-scale iterative registration with a Swin Transformer-based long-distance correlation decoder. Furthermore, we exploit pyramid structure to supplement the detailed information of the deformation field by using high-scale feature maps. Comprehensive experimental results implemented on brain MRI and liver CT datasets show that the proposed method is superior to the existing registration methods in terms of registration accuracy, training time and parameters, especially of a significant advantage in running time. Our code is available at https://github.com/Torbjorn1997/PIViT .
SAMConvex: Fast Discrete Optimization for CT Registration Using Self-supervised Anatomical Embedding and Correlation Pyramid

Estimating displacement vector field via a cost volume computed in the feature space has shown great success in image registration, but it suffers excessive computation burdens. Moreover, existing feature descriptors only extract local features incapable of representing the global semantic information, which is especially important for solving large transformations. To address the discussed issues, we propose SAMConvex, a fast coarse-to-fine discrete optimization method for CT registration that includes a decoupled convex optimization procedure to obtain deformation fields based on a self-supervised anatomical embedding (SAM) feature extractor that captures both local and global information. To be specific, SAMConvex extracts per-voxel features and builds 6D correlation volumes based on SAM features, and iteratively updates a flow field by performing lookups on the correlation volumes with a coarse-to-fine scheme. SAMConvex outperforms the state-of-the-art learning-based methods and optimization-based methods over two inter-patient registration datasets (Abdomen CT and HeadNeck CT) and one intra-patient registration dataset (Lung CT). Moreover, as an optimization-based method, SAMConvex only takes $${\sim }2$$ s ( $${\sim }5$$ s with instance optimization) for one paired images.
Weakly Supervised Medical Image Segmentation via Superpixel-Guided Scribble Walking and Class-Wise Contrastive Regularization

Deep learning-based segmentation typically requires a large amount of data with dense manual delineation, which is both time-consuming and expensive to obtain for medical images. Consequently, weakly supervised learning, which attempts to utilize sparse annotations such as scribbles for effective training, has garnered considerable attention. However, such scribble-supervision inherently lacks sufficient structural information, leading to two critical challenges: (i) while achieving good performance in overall overlap metrics such as Dice score, the existing methods struggle to perform satisfactory local prediction because no desired structural priors are accessible during training; (ii) the class feature distributions are inevitably less-compact due to sparse and extremely incomplete supervision, leading to poor generalizability. To address these, in this paper, we propose the SC-Net, a new scribble-supervised approach that combines Superpixel-guided scribble walking with Class-wise contrastive regularization. Specifically, the framework is built upon the recent dual-decoder backbone design, where predictions from two slightly different decoders are randomly mixed to provide auxiliary pseudo-label supervision. Besides the sparse and pseudo supervision, the scribbles walk towards unlabeled pixels guided by superpixel connectivity and image content to offer as much dense supervision as possible. Then, the class-wise contrastive regularization disconnects the feature manifolds of different classes to encourage the compactness of class feature distributions. We evaluate our approach on the public cardiac dataset ACDC and demonstrate the superiority of our method compared to recent scribble-supervised and semi-supervised learning methods with similar labeling efforts.
A Denoised Mean Teacher for Domain Adaptive Point Cloud Registration

Point cloud-based medical registration promises increased computational efficiency, robustness to intensity shifts, and anonymity preservation but is limited by the inefficacy of unsupervised learning with similarity metrics. Supervised training on synthetic deformations is an alternative but, in turn, suffers from the domain gap to the real domain. In this work, we aim to tackle this gap through domain adaptation. Self-training with the Mean Teacher is an established approach to this problem but is impaired by the inherent noise of the pseudo labels from the teacher. As a remedy, we present a denoised teacher-student paradigm for point cloud registration, comprising two complementary denoising strategies. First, we propose to filter pseudo labels based on the Chamfer distances of teacher and student registrations, thus preventing detrimental supervision by the teacher. Second, we make the teacher dynamically synthesize novel training pairs with noise-free labels by warping its moving inputs with the predicted deformations. Evaluation is performed for inhale-to-exhale registration of lung vessel trees on the public PVT dataset under two domain shifts. Our method surpasses the baseline Mean Teacher by 13.5/62.8%, consistently outperforms diverse competitors, and sets a new state-of-the-art accuracy (TRE = 2.31 mm). Code is available at https://github.com/multimodallearning/denoised_mt_pcd_reg .
Unsupervised 3D Registration Through Optimization-Guided Cyclical Self-training

State-of-the-art deep learning-based registration methods employ three different learning strategies: supervised learning, which requires costly manual annotations, unsupervised learning, which heavily relies on hand-crafted similarity metrics designed by domain experts, or learning from synthetic data, which introduces a domain shift. To overcome the limitations of these strategies, we propose a novel self-supervised learning paradigm for unsupervised registration, relying on self-training. Our idea is based on two key insights. Feature-based differentiable optimizers 1) perform reasonable registration even from random features and 2) stabilize the training of the preceding feature extraction network on noisy labels. Consequently, we propose cyclical self-training, where pseudo labels are initialized as the displacement fields inferred from random features and cyclically updated based on more and more expressive features from the learning feature extractor, yielding a self-reinforcement effect. We evaluate the method for abdomen and lung registration, consistently surpassing metric-based supervision and outperforming diverse state-of-the-art competitors. Source code is available at https://github.com/multimodallearning/reg-cyclical-self-train .
Adapter Learning in Pretrained Feature Extractor for Continual Learning of Diseases

Currently intelligent diagnosis systems lack the ability of continually learning to diagnose new diseases once deployed, under the condition of preserving old disease knowledge. In particular, updating an intelligent diagnosis system with training data of new diseases would cause catastrophic forgetting of old disease knowledge. To address the catastrophic forgetting issue, an Adapter-based Continual Learning framework called ACL is proposed to help effectively learn a set of new diseases at each round (or task) of continual learning, without changing the shared feature extractor. The learnable lightweight task-specific adapter(s) can be flexibly designed (e.g., two convolutional layers) and then added to the pretrained and fixed feature extractor. Together with a specially designed task-specific head which absorbs all previously learned old diseases as a single 'out-of-distribution' category, task-specific adapter(s) can help the pretrained feature extractor more effectively extract discriminative features between diseases. In addition, a simple yet effective fine-tuning is applied to collaboratively fine-tune multiple task-specific heads such that outputs from different heads are comparable and consequently the appropriate classifier head can be more accurately selected during model inference. Extensive empirical evaluations on three image datasets demonstrate the superior performance of ACL in continual learning of new diseases. The source code is available at https://github.com/GiantJun/CL_Pytorch .
StructuRegNet: Structure-Guided Multimodal 2D-3D Registration

Multimodal 2D-3D co-registration is a challenging problem with numerous clinical applications, including improved diagnosis, radiation therapy, or interventional radiology. In this paper, we present StructuRegNet, a deep-learning framework that addresses this problem with three novel contributions. First, we combine a 2D-3D deformable registration network with an adversarial modality translation module, allowing each block to benefit from the signal of the other. Second, we solve the initialization challenge for 2D-3D registration by leveraging tissue structure through cascaded rigid areas guidance and distance field regularization. Third, StructuRegNet handles out-of-plane deformation without requiring any 3D reconstruction thanks to a recursive plane selection. We evaluate the quantitative performance of StructuRegNet for head and neck cancer between 3D CT scans and 2D histopathological slides, enabling pixel-wise mapping of low-quality radiologic imaging to gold-standard tumor extent and bringing biological insights toward homogenized clinical guidelines. Additionally, our method can be used in radiation therapy by mapping 3D planning CT into the 2D MR frame of the treatment day for accurate positioning and dose delivery. Our framework demonstrates superior results to traditional methods for both applications. It is versatile to different locations or magnitudes of deformation and can serve as a backbone for any relevant clinical context.
SLPT: Selective Labeling Meets Prompt Tuning on Label-Limited Lesion Segmentation

Medical image analysis using deep learning is often challenged by limited labeled data and high annotation costs. Fine-tuning the entire network in label-limited scenarios can lead to overfitting and suboptimal performance. Recently, prompt tuning has emerged as a more promising technique that introduces a few additional tunable parameters as prompts to a task-agnostic pre-trained model, and updates only these parameters using supervision from limited labeled data while keeping the pre-trained model unchanged. However, previous work has overlooked the importance of selective labeling in downstream tasks, which aims to select the most valuable downstream samples for annotation to achieve the best performance with minimum annotation cost. To address this, we propose a framework that combines selective labeling with prompt tuning (SLPT) to boost performance in limited labels. Specifically, we introduce a feature-aware prompt updater to guide prompt tuning and a TandEm Selective LAbeling (TESLA) strategy. TESLA includes unsupervised diversity selection and supervised selection using prompt-based uncertainty. In addition, we propose a diversified visual prompt tuning strategy to provide multi-prompt-based discrepant predictions for TESLA. We evaluate our method on liver tumor segmentation and achieve state-of-the-art performance, outperforming traditional fine-tuning with only 6% of tunable parameters, also achieving 94% of full-data performance by labeling only 5% of the data.
Centroid-Aware Feature Recalibration for Cancer Grading in Pathology Images

Cancer grading is an essential task in pathology. The recent developments of artificial neural networks in computational pathology have shown that these methods hold great potential for improving the accuracy and quality of cancer diagnosis. However, the issues with the robustness and reliability of such methods have not been fully resolved yet. Herein, we propose a centroid-aware feature recalibration network that can conduct cancer grading in an accurate and robust manner. The proposed network maps an input pathology image into an embedding space and adjusts it by using centroids embedding vectors of different cancer grades via attention mechanism. Equipped with the recalibrated embedding vector, the proposed network classifiers the input pathology image into a pertinent class label, i.e., cancer grade. We evaluate the proposed network using colorectal cancer datasets that were collected under different environments. The experimental results confirm that the proposed network is able to conduct cancer grading in pathology images with high accuracy regardless of the environmental changes in the datasets.
SATTA: Semantic-Aware Test-Time Adaptation for Cross-Domain Medical Image Segmentation

Cross-domain distribution shift is a common problem for medical image analysis because medical images from different devices usually own varied domain distributions. Test-time adaptation (TTA) is a promising solution by efficiently adapting source-domain distributions to target-domain distributions at test time with unsupervised manners, which has increasingly attracted important attention. Previous TTA methods applied to medical image segmentation tasks usually carry out a global domain adaptation for all semantic categories, but global domain adaptation would be sub-optimal as the influence of domain shift on different semantic categories may be different. To obtain improved domain adaptation results for different semantic categories, we propose Semantic-Aware Test-Time Adaptation (SATTA), which can individually update the model parameters to adapt to target-domain distributions for each semantic category. Specifically, SATTA deploys an uncertainty estimation module to measure the discrepancies of semantic categories in domain shift effectively. Then, a semantic adaptive learning rate is developed based on the estimated discrepancies to achieve a personalized degree of adaptation for each semantic category. Lastly, semantic proxy contrastive learning is proposed to individually adjust the model parameters with the semantic adaptive learning rate. Our SATTA is extensively validated on retinal fluid segmentation based on SD-OCT images. The experimental results demonstrate that SATTA consistently improves domain adaptation performance on semantic categories over other state-of-the-art TTA methods.