Reviewerly/exHarmony huggingface dataset:
A Spatial-Temporal Deformable Attention Based Framework for Breast Lesion Detection in Videos

Detecting breast lesion in videos is crucial for computer-aided diagnosis. Existing video-based breast lesion detection approaches typically perform temporal feature aggregation of deep backbone features based on the self-attention operation. We argue that such a strategy struggles to effectively perform deep feature aggregation and ignores the useful local information. To tackle these issues, we propose a spatial-temporal deformable attention based framework, named STNet. Our STNet introduces a spatial-temporal deformable attention module to perform local spatial-temporal feature fusion. The spatial-temporal deformable attention module enables deep feature aggregation in each stage of both encoder and decoder. To further accelerate the detection speed, we introduce an encoder feature shuffle strategy for multi-frame prediction during inference. In our encoder feature shuffle strategy, we share the backbone and encoder features, and shuffle encoder features for decoder to generate the predictions of multiple frames. The experiments on the public breast lesion ultrasound video dataset show that our STNet obtains a state-of-the-art detection performance, while operating twice as fast inference speed. The code and model are available at https://github.com/AlfredQin/STNet .
CircleFormer: Circular Nuclei Detection in Whole Slide Images with Circle Queries and Attention

Both CNN-based and Transformer-based object detection with bounding box representation have been extensively studied in computer vision and medical image analysis, but circular object detection in medical images is still underexplored. Inspired by the recent anchor free CNN-based circular object detection method (CircleNet) for ball-shape glomeruli detection in renal pathology, in this paper, we present CircleFormer, a Transformer-based circular medical object detection with dynamic anchor circles. Specifically, queries with circle representation in Transformer decoder iteratively refine the circular object detection results, and a circle cross attention module is introduced to compute the similarity between circular queries and image features. A generalized circle IoU (gCIoU) is proposed to serve as a new regression loss of circular object detection as well. Moreover, our approach is easy to generalize to the segmentation task by adding a simple segmentation branch to CircleFormer. We evaluate our method in circular nuclei detection and segmentation on the public MoNuSeg dataset, and the experimental results show that our method achieves promising performance compared with the state-of-the-art approaches. The effectiveness of each component is validated via ablation studies as well. Our code is released at: https://github.com/zhanghx-iim-ahu/CircleFormer .
Improving Medical Image Classification in Noisy Labels Using only Self-supervised Pretraining

Noisy labels hurt deep learning-based supervised image classification performance as the models may overfit the noise and learn corrupted feature extractors. For natural image classification training with noisy labeled data, model initialization with contrastive self-supervised pretrained weights has shown to reduce feature corruption and improve classification performance. However, no works have explored: i) how other self-supervised approaches, such as pretext task-based pretraining, impact the learning with noisy label, and ii) any self-supervised pretraining methods alone for medical images in noisy label settings. Medical images often feature smaller datasets and subtle inter-class variations, requiring human expertise to ensure correct classification. Thus, it is not clear if the methods improving learning with noisy labels in natural image datasets such as CIFAR would also help with medical images. In this work, we explore contrastive and pretext task-based self-supervised pretraining to initialize the weights of a deep learning classification model for two medical datasets with self-induced noisy labels—NCT-CRC-HE-100K tissue histological images and COVID-QU-Ex chest X-ray images. Our results show that models initialized with pretrained weights obtained from self-supervised learning can effectively learn better features and improve robustness against noisy labels.
Multi-shot Prototype Contrastive Learning and Semantic Reasoning for Medical Image Segmentation

Despite the remarkable achievements made by deep convolutional neural networks in medical image segmentation, the limitation that they rely heavily on high-precision and intensively annotated samples makes it difficult to adapt to novel classes that have not been seen before. Few-shot learning is introduced to solve these challenges by learning the generalized representation of a semantic class from very few annotated support samples that can be used as a reference for unannotated query samples. In this paper, instead of averaging multiple support prototypes, we propose a multi-shot prototype contrastive learning and semantic reasoning network (MPSNet) for medical image segmentation. The multi-shot learning network exists independently within the support set, obtains effective semantic features for support images and gives priority to training the core segmentation model of prototype contrastive learning. We also propose a semantic reasoning network that takes the prior semantic features and prior segmentation model learned from the support set as the immediate and necessary conditions for the query image to deduce its segmentation mask. The proposed method is verified to be superior to the state-of-the-art methods on three public datasets, revealing its powerful segmentation and generalization abilities. Code: https://github.com/H51705/FSS_MPSNet .
NISF: Neural Implicit Segmentation Functions

Segmentation of anatomical shapes from medical images has taken an important role in the automation of clinical measurements. While typical deep-learning segmentation approaches are performed on discrete voxels, the underlying objects being analysed exist in a real-valued continuous space. Approaches that rely on convolutional neural networks (CNNs) are limited to grid-like inputs and not easily applicable to sparse or partial measurements. We propose a novel family of image segmentation models that tackle many of CNNs' shortcomings: Neural Implicit Segmentation Functions (NISF). Our framework takes inspiration from the field of neural implicit functions where a network learns a mapping from a real-valued coordinate-space to a shape representation. NISFs have the ability to segment anatomical shapes in high-dimensional continuous spaces. Training is not limited to voxelized grids, and covers applications with sparse and partial data. Interpolation between observations is learnt naturally in the training procedure and requires no post-processing. Furthermore, NISFs allow the leveraging of learnt shape priors to make predictions for regions outside of the original image plane. We go on to show the framework achieves dice scores of $$0.87 \pm 0.045$$ on a (3D+t) short-axis cardiac segmentation task using the UK Biobank dataset. We also provide a qualitative analysis on our frameworks ability to perform segmentation and image interpolation on unseen regions of an image volume at arbitrary resolutions.
Generating High-Resolution 3D CT with 12-Bit Depth Using a Diffusion Model with Adjacent Slice and Intensity Calibration Network

Since the advent of generative models, deep learning-based methods for generating high-resolution, photorealistic 2D images have made significant successes. However, it is still difficult to create precise 3D image data with 12-bit depth used in clinical settings that capture the anatomy and pathology of CT and MRI scans. Using a score-based diffusion model, we propose a slice-based method that generates 3D images from previous 2D CT slices along the inferior direction. We call this method stochastic differential equations with adjacent slice-based conditional iterative inpainting (ASCII). We also propose an intensity calibration network (IC-Net) that adjusts the among slices intensity mismatch caused by 12-bit depth image generation. As a result, Frechet Inception Distance (FIDs) scores of FID-Ax, FID-Cor and FID-Sag of ASCII(2) with IC-Net were 14.993, 19.188 and 19.698, respectively. Anatomical continuity of the generated 3D image along the inferior direction was evaluated by an expert radiologist with more than 15 years of experience. In the analysis of eight anatomical structures, our method was evaluated to be continuous for seven of the structures.
Accurate Multi-contrast MRI Super-Resolution via a Dual Cross-Attention Transformer Network

Magnetic Resonance Imaging (MRI) is a critical imaging tool in clinical diagnosis, but obtaining high-resolution MRI images can be challenging due to hardware and scan time limitations. Recent studies have shown that using reference images from multi-contrast MRI data could improve super-resolution quality. However, the commonly employed strategies, e.g., channel concatenation or hard-attention based texture transfer, may not be optimal given the visual differences between multi-contrast MRI images. To address these limitations, we propose a new Dual Cross-Attention Multi-contrast Super Resolution (DCAMSR) framework. This approach introduces a dual cross-attention transformer architecture, where the features of the reference image and the up-sampled input image are extracted and promoted with both spatial and channel attention in multiple resolutions. Unlike existing hard-attention based methods where only the most correlated features are sought via the highly down-sampled reference images, the proposed architecture is more powerful to capture and fuse the shareable information between the multi-contrast images. Extensive experiments are conducted on fastMRI knee data at high field and more challenging brain data at low field, demonstrating that DCAMSR can substantially outperform the state-of-the-art single-image and multi-contrast MRI super-resolution methods, and even remains robust in a self-referenced manner. The code for DCAMSR is avaliable at https://github.com/Solor-pikachu/DCAMSR .
Treasure in Distribution: A Domain Randomization Based Multi-source Domain Generalization for 2D Medical Image Segmentation

Although recent years have witnessed the great success of convolutional neural networks (CNNs) in medical image segmentation, the domain shift issue caused by the highly variable image quality of medical images hinders the deployment of CNNs in real-world clinical applications. Domain generalization (DG) methods aim to address this issue by training a robust model on the source domain, which has a strong generalization ability. Previously, many DG methods based on feature-space domain randomization have been proposed, which, however, suffer from the limited and unordered search space of feature styles. In this paper, we propose a multi-source DG method called Treasure in Distribution (TriD), which constructs an unprecedented search space to obtain the model with strong robustness by randomly sampling from a uniform distribution. To learn the domain-invariant representations explicitly, we further devise a style-mixing strategy in our TriD, which mixes the feature styles by randomly mixing the augmented and original statistics along the channel wise and can be extended to other DG methods. Extensive experiments on two medical segmentation tasks with different modalities demonstrate that our TriD achieves superior generalization performance on unseen target-domain data. Code is available at https://github.com/Chen-Ziyang/TriD .
Persons with Disabilities in COVID 19: Bangladesh Perspective

United Nations Convention on the Rights of Persons with Disabilities (UNCRPD)Persons with disabilities (PWD) remain vulnerable and susceptible to discrimination in crisis, whereas they are entitled to inclusive protection. During the COVID-19 pandemic, global health and other public services nearly collapsed, putting the disability needs on the backfoot. Hence, the World Health Organization (WHO) and the United Nations (UN) provided comprehensive and disability-inclusive disaster management guidelines for the member states to secure PWD. Even though Bangladesh adopted response plans in compliance, disability integration failed to reach the level of expectation. The cardinal disability rights instrument, the United Nations Convention on the Rights of Persons with Disabilities (UNCRPD) 2006, obliges the state parties to target and mainstream disability rights during risks and emergencies. Though Bangladesh has the Rights and Protection of Person with Disability Act (RPPDA) 2013 in compliance, the scope of the legislation during crises remains uncertain. This qualitative research investigates whether the current safeguard measures are adequate in protecting PWD of Bangladesh amid the COVID-19 pandemic. The chapter first critically examines the existing national disability safeguard measures and then explores the extent of the protection during the pandemic in compliance with international guidance. Relevant primary sources like national and international legislation, guidelines, and policies are analyzed. Secondary sources like authoritative journal articles, books, and newspaper reports are explored to investigate the disability situation. The pandemic management strategies in Bangladesh fall short in disability protection without comprehensive planning and adequate enforcement mechanism. Disability-inclusive disaster regulatory measures and proper monitoring are necessary to yield better protection in future crises.
Scribble-Based 3D Multiple Abdominal Organ Segmentation via Triple-Branch Multi-Dilated Network with Pixel- and Class-Wise Consistency

Multi-organ segmentation in abdominal Computed Tomography (CT) images is of great importance for diagnosis of abdominal lesions and subsequent treatment planning. Though deep learning based methods have attained high performance, they rely heavily on large-scale pixel-level annotations that are time-consuming and labor-intensive to obtain. Due to its low dependency on annotation, weakly supervised segmentation has attracted great attention. However, there is still a large performance gap between current weakly-supervised methods and fully supervised learning, leaving room for exploration. In this work, we propose a novel 3D framework with two consistency constraints for scribble-supervised multiple abdominal organ segmentation from CT. Specifically, we employ a Triple-branch multi-Dilated network (TDNet) with one encoder and three decoders using different dilation rates to capture features from different receptive fields that are complementary to each other to generate high-quality soft pseudo labels. For more stable unsupervised learning, we use voxel-wise uncertainty to rectify the soft pseudo labels and then supervise the outputs of each decoder. To further regularize the network, class relationship information is exploited by encouraging the generated class affinity matrices to be consistent across different decoders under multi-view projection. Experiments on the public WORD dataset show that our method outperforms five existing scribble-supervised methods.
MDA-SR: Multi-level Domain Adaptation Super-Resolution for Wireless Capsule Endoscopy Images

Super-resolution (SR) of wireless capsule endoscopy (WCE) images is challenging because paired high-resolution (HR) images are not available. An intuitive solution is to simulate paired low-resolution (LR) WCE images from HR electronic endoscopy images for supervised learning. However, the SR model obtained by this method cannot be well adapted to real WCE images due to the large domain gap between electronic endoscopy images and WCE images. To address this issue, we propose a Multi-level Domain Adaptation SR model (MDA-SR) in an unsupervised manner using arbitrary set of WCE images and HR electronic endoscopy images. Our approach implements domain adaptation at the image level and latent level during the degradation and SR processes, respectively. To the best of our knowledge, this is the first work to explore an unsupervised SR approach for WCE images. Furthermore, we design an Endoscopy Image Quality Evaluator (EIQE) based on the reference-free image evaluation metric NIQE, which is more suitable for evaluating WCE image quality. Extensive experiments demonstrate that our MDA-SR method outperforms state-of-the-art SR methods both quantitatively and qualitatively.
ConvFormer: Plug-and-Play CNN-Style Transformers for Improving Medical Image Segmentation

Transformers have been extensively studied in medical image segmentation to build pairwise long-range dependence. Yet, relatively limited well-annotated medical image data makes transformers struggle to extract diverse global features, resulting in attention collapse where attention maps become similar or even identical. Comparatively, convolutional neural networks (CNNs) have better convergence properties on small-scale training data but suffer from limited receptive fields. Existing works are dedicated to exploring the combinations of CNN and transformers while ignoring attention collapse, leaving the potential of transformers under-explored. In this paper, we propose to build CNN-style Transformers (ConvFormer) to promote better attention convergence and thus better segmentation performance. Specifically, ConvFormer consists of pooling, CNN-style self-attention (CSA), and convolutional feed-forward network (CFFN) corresponding to tokenization, self-attention, and feed-forward network in vanilla vision transformers. In contrast to positional embedding and tokenization, ConvFormer adopts 2D convolution and max-pooling for both position information preservation and feature size reduction. In this way, CSA takes 2D feature maps as inputs and establishes long-range dependency by constructing self-attention matrices as convolution kernels with adaptive sizes. Following CSA, 2D convolution is utilized for feature refinement through CFFN. Experimental results on multiple datasets demonstrate the effectiveness of ConvFormer working as a plug-and-play module for consistent performance improvement of transformer-based frameworks. Code is available at https://github.com/xianlin7/ConvFormer .
Pre-training with Simulated Ultrasound Images for Breast Mass Segmentation and Classification

We investigate the usefulness of formula-driven supervised learning (FDSL) for breast ultrasound (US) image analysis. Medical data are usually too scarce to develop a better performing deep learning model from scratch. Transfer learning with networks pre-trained on ImageNet is commonly applied to address this problem. FDSL techniques have been recently investigated as an alternative solution to ImageNet based approaches. In the FDSL setting, networks for transfer learning applications are developed using large amounts of synthetic images generated with mathematical formulas, possibly taking into account the characteristics of the target data. In this work, we use Field II to develop a large synthetic dataset of 100 000 US images presenting different contour objects, as shape features play an important role in breast mass characterization in US. Synthetic data are utilized to pre-train the ResNet50 classification model and various variants of the U-Net segmentation network. Next, the pre-trained models are fine-tuned on breast mass US images. Our results demonstrate that the proposed FDSL approach can provide good performance with respect to breast mass classification and segmentation.
MRIS: A Multi-modal Retrieval Approach for Image Synthesis on Diverse Modalities

Multiple imaging modalities are often used for disease diagnosis, prediction, or population-based analyses. However, not all modalities might be available due to cost, different study designs, or changes in imaging technology. If the differences between the types of imaging are small, data harmonization approaches can be used; for larger changes, direct image synthesis approaches have been explored. In this paper, we develop an approach, MRIS, based on multi-modal metric learning to synthesize images of diverse modalities. We use metric learning via multi-modal image retrieval, resulting in embeddings that can relate images of different modalities. Given a large image database, the learned image embeddings allow us to use k-nearest neighbor (k-NN) regression for image synthesis. Our driving medical problem is knee osteoarthritis (KOA), but our developed method is general after proper image alignment. We test our approach by synthesizing cartilage thickness maps obtained from 3D magnetic resonance (MR) images using 2D radiographs. Our experiments show that the proposed method outperforms direct image synthesis and that the synthesized thickness maps retain information relevant to downstream tasks such as progression prediction and Kellgren-Lawrence grading (KLG). Our results suggest that retrieval approaches can be used to obtain high-quality and meaningful image synthesis results given large image databases. Our code is available at https://github.com/uncbiag/MRIS .
CT-Guided, Unsupervised Super-Resolution Reconstruction of Single 3D Magnetic Resonance Image

Deep learning-based algorithms for single MR image (MRI) super-resolution have shown great potential in enhancing the resolution of low-quality images. However, many of these methods rely on supervised training with paired low-resolution (LR) and high-resolution (HR) MR images, which can be difficult to obtain in clinical settings. This is because acquiring HR MR images in clinical settings requires a significant amount of time. In contrast, HR CT images are acquired in clinical routine. In this paper, we propose a CT-guided, unsupervised MRI super-resolution reconstruction method based on joint cross-modality image translation and super-resolution reconstruction, eliminating the requirement of high-resolution MRI for training. The proposed approach is validated on two datasets respectively acquired from two different clinical sites. Well-established metrics including Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Metrics (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) are used to assess the performance of the proposed method. Our method achieved an average PSNR of 32.23, an average SSIM of 0.90 and an average LPIPS of 0.14 when evaluated on data of the first site. An average PSNR of 30.58, an average SSIM of 0.88, and an average LPIPS of 0.10 were achieved by our method when evaluated on data of the second site.