Reviewerly/exHarmony huggingface dataset:
Point Transformer for Coronary Artery Labeling

Coronary CT angiography (CCTA) scans are widely used for diagnosis of coronary artery diseases. An accurate and automatic vessel labeling algorithm for CCTA analysis can significantly improve the diagnostic efficiency and reduce the clinicians’ manual efforts. In this paper, we propose a simple vessel labeling method based on the Point Transformer [7], which only needs the coronary artery segmentation. Specifically, firstly, the coronary segmentation is transformed to point cloud. Then, these points are fed into the hierarchical transformer blocks to obtain the multi-level features, including local and global features. Finally, the network output the semantic classification points and map them to centerline labeling. This method is only based on the structure of coronary segmentation and need not other features, so it is easy to generalize to other vessel labeling tasks, e.g., head and neck vessel labeling. To evaluate the performance of our proposed method, CCTA scans of 53 subjects are collected in our experiment. The experimental results demonstrate the efficacy of this approach.
Unpaired H&amp;E to PR Stain Transfer with Self-Supervised Auxiliary Segmentation

Histochemical staining is a critical step in the diagnosis of cancer, where hematoxylin-eosin (H&E) stain is used most commonly in clinical practice. However, the H&E images often cannot be used for making accurate diagnoses. To this end, pathologists must perform immunohistochemical (IHC) stain, which is time-consuming and costly. In the field of computer-aided diagnosis, existing models can virtually generate IHC staining images, but they often require pixel-aligned data and annotations from pathologists, which are difficult to be obtained. To address this problem, we propose a self-supervised PR (a typical type of IHC) virtual staining model utilizing unpaired data without pathologists' annotations for the first time. Based on the observation that PR images are easy to be segmented, we introduce segmentation as the proxy task to make the virtual staining more accurate. Experimental results show that our model can generate PR images with the highest accuracy. Moreover, our model achieves the desired results on an external dataset.
Mask-Guided Data Augmentation for Multiparametric MRI Generation with a Rare Hepatocellular Carcinoma

Data augmentation is classically used to improve the overall performance of deep learning models. It is, however, challenging in the case of medical applications, and in particular for multiparametric datasets. For example, traditional geometric transformations used in several applications to generate synthetic images can modify in a non-realistic manner the patients’ anatomy. Therefore, dedicated image generation techniques are necessary in the medical field to, for example, mimic a given pathology in a realistic manner. This paper introduces a new data augmentation architecture that generates synthetic multiparametric (T1 arterial, T1 portal, and T2) magnetic resonance images (MRI) of massive macrotrabecular subtype hepatocellular carcinoma with their corresponding tumor masks through a generative deep learning approach. The proposed architecture creates liver tumor masks and abdominal edges used as input in a Pix2Pix network for synthetic data creation. The efficiency of the method is demonstrated by training it on a limited multiparametric dataset of MRI triplets from 89 patients with liver lesions to generate 1, 000 synthetic triplets and their corresponding liver tumor masks. The resulting Frechet Inception Distance score was 86.55. The proposed approach was among the winners of the 2021 data augmentation challenge organized by the French Society of Radiology.
Breast Lesion Diagnosis Using Static Images and Dynamic Video

Deep learning based Computer Aided Diagnosis (CAD) systems have been developed to treat breast ultrasound. Most of them focus on a single ultrasound imaging modality, either using representative static images or the dynamic video of a real-time scan. In fact, these two image modalities are complementary for lesion diagnosis. Dynamic videos provide detailed three-dimensional information about the lesion, while static images capture the typical sections of the lesion. In this work, we propose a multi-modality breast tumor diagnosis model to imitate the diagnosing process of radiologists, which learns the features of both static images and dynamic video and explores the potential relationship between the two modalities. Considering that static images are carefully selected by professional radiologists, we propose to aggregate dynamic video features under the guidance of domain knowledge from static images before fusing multi-modality features. Our work is validated on a breast ultrasound dataset composed of 897 sets of ultrasound images and videos. Experimental results show that our model boosts the performance of Benign/Malignant classification, achieving 90.0% in AUC and 81.7% in accuracy.
Learning Representations for MR Image Retrieval: Contrastive Models Versus Other Supervision Strategies

Content-Based Image Retrieval (CBIR) is an image search technique which consists of retrieving from a database the most similar images to a query image, based on their representations. In this paper, we propose to learn such representations in the context of MRI using a convolutional neural network trained in a self-supervised contrastive fashion, which requires a lower degree of supervision than state-of-the-art supervised approaches. Embedded in a CBIR pipeline for medical images, we apply this method to the BRaTS MR dataset. We also present an approach to evaluate the retrieval results using the annotations already available in the dataset. To capture the complementarity across different MR sequences, we investigate the benefit of information fusion in learning relevant representations. Our experiments show that a weakly supervised training may be advantageous not only in terms of requiring fewer annotations, but also because it may yield increased performance compared to its supervised counterparts.
Multiple Instance Learning with Critical Instance for Whole Slide Image Classification

Histopathological whole slide image (WSI) classification is a fundamental task in clinical diagnosis. The extremely high resolution of WSIs makes it laborious to obtain localized annotations. Hence, WSI classification is often cast as a multiple instance learning (MIL) problem when only slide-level labels are available. A major challenge for MIL approaches is that the proportion of malignant regions in a malignant WSI could be very small, resulting in the misclassification of these slides. In this work, we propose to find the critical instance in the projection space of instance representations. A novel critical instance pooling is proposed to aggregate instance representations with normalized distances between instances and the critical instance. Additionally, a feature magnitude loss is utilized to enforce the discrepancy between positive and negative instances in the projection space. Experimental results on existing benchmark datasets showed the effectiveness of our proposed method.
A Point in the Right Direction: Vector Prediction for Spatially-Aware Self-Supervised Volumetric Representation Learning

High annotation costs and limited labels for dense 3D medical imaging tasks have recently motivated an assortment of 3D self-supervised pretraining methods that improve transfer learning performance. However, these methods commonly lack spatial awareness despite its centrality in enabling effective 3D image analysis. More specifically, position, scale, and orientation are not only informative but also automatically available when generating image patches for training. Yet, to date, no work has proposed a pretext task that distills all key spatial features. To fulfill this need, we develop a new self-supervised method, VectorPOSE, which promotes better spatial understanding with two novel pretext tasks: Vector Prediction (VP) and Boundary-Focused Reconstruction (BFR). VP focuses on global spatial concepts (i.e., properties of 3D patches) while BFR addresses weaknesses of recent reconstruction methods to learn more effective local representations. We evaluate VectorPOSE on three 3D medical image segmentation tasks, showing that it often outperforms state-of-the-art methods, especially in limited annotation settings.
BEL: A Bag Embedding Loss for Transformer Enhances Multiple Instance Whole Slide Image Classification

Multiple Instance Learning (MIL) has become the predominant approach for classification tasks on gigapixel histopathology whole slide images (WSIs). Within the MIL framework, single WSIs (bags) are decomposed into patches (instances), with only WSI-level annotation available. Recent MIL approaches produce highly informative bag level representations by utilizing the transformer architecture’s ability to model the dependencies between instances. However, when applied to high magnification datasets, problems emerge due to the large number of instances and the weak supervisory learning signal. To address this problem, we propose to additionally train transformers with a novel Bag Embedding Loss (BEL). BEL forces the model to learn a discriminative bag-level representation by minimizing the distance between bag embeddings of the same class and maximizing the distance between different classes. We evaluate BEL with the Transformer architecture TransMIL on two publicly available histopathology datasets, BRACS and CAMELYON17. We show that with BEL, TransMIL outperforms the baseline models on both datasets, thus contributing to the clinically highly relevant AI-based tumor classification of histological patient material.
MFR-Net: Multi-Scale Feature Representation Module for 3D Cerebrovascular Segmentation

Cerebrovascular segmentation of Time-of-Flight magnetic resonance angiography (TOF-MRA) is a necessary step for computer-aided diagnosis. At present 3D U-Net is the most popular 3D medical image segmentation framework, but it can only capture the vascular features of single-size receptive field, and cannot distinguish different structural information of large, medium and small vessels. CNN-Transformer hybrid model requires more labelled datasets to learn effective segmentation, while 3D cerebrovascular annotation is difficult to obtain. In this work, we propose MFR-Net, novelly designed a Multi-scale Feature Representation module to make up for the defect that traditional convolution units only extract single scale features. At the same time, we introduce residual extraction path in skip connection to reduce the encoder-decoder semantic gap. In addition, due to the lack of public 3D cerebrovascular segmentation annotation dataset, we publish the 3D cerebrovascular annotation ground truth of public dataset TubeTK and official data annotation algorithm. Compared with numerous advanced 2D/3D segmentation models and the most advanced deep learning medical image segmentation benchmark nnU-Net , the proposed approach shows better performance. Code and 3D cerebrovascular annotation ground truth of public dataset TubeTK are available at: https://github.com/EllisLyu/TubeTK-Dateset-Annotation.
Domain Adaptive Multiple Instance Learning for Instance-Level Prediction of Pathological Images

Pathological image analysis is an important process for detecting abnormalities such as cancer from cell images. However, since the image size is generally very large, the cost of providing detailed annotations is high, which makes it difficult to apply machine learning techniques. One way to improve the performance of identifying abnormalities while keeping the annotation cost low is to use only labels for each slide, or to use information from another dataset that has already been labeled. However, such weak supervisory information often does not provide sufficient performance. In this paper, we proposed a new task setting to improve the classification performance of the target dataset without increasing annotation costs. And to solve this problem, we propose a pipeline that uses multiple instance learning (MIL) and domain adaptation (DA) methods. Furthermore, in order to combine the supervisory information of both methods effectively, we propose a method to create pseudo-labels with high confidence. We conducted experiments on the pathological image dataset we created for this study and showed that the proposed method significantly improves the classification performance compared to existing methods.
Using Forestnets for Partial Fine-Tuning Prior to Breast Cancer Detection in Ultrasounds

Ultrasound imaging is the most used technique for first investigating suspicions of breast lesions. In recent years, many deep learning techniques have been developed to analyze ultrasound imagery automatically. In this paper, we present results obtained on breast lesion detection using FasterRCNN with a ResNet18 as the backbone. We pre-trained the backbone on classifying benign and malignant lesions using an extremely sparse classification head called ForestNet instead of the usual fully-connected head. Results on classification showed that the inherent sparsity of ForestNet led to reduced overfitting and kept untouched roughly 7% of the millions of parameters in the ResNet18 backbone. Our experiments were run on two different datasets. On classification, our modified ResNet-18 architecture classified correctly 85.9% and 91.3%, while the usual architecture achieved 75.6% and 80.6%, respectively. Regarding detection, the average precision on the two datasets rises from 80.4% and 79.2% to respectively 87.3% and 87.0% using our approach.
Cluster Entropy: Active Domain Adaptation in Pathological Image Segmentation

The domain shift in pathological segmentation is an important problem, where a network trained by a source domain (collected at a specific hospital) does not work well in the target domain (from different hospitals) due to the different image features. Due to the problems of class imbalance and different class prior of pathology, typical unsupervised domain adaptation methods do not work well by aligning the distribution of source domain and target domain. In this paper, we propose a cluster entropy for selecting an effective whole slide image (WSI) that is used for semi-supervised domain adaptation. This approach can measure how the image features of the WSI cover the entire distribution of the target domain by calculating the entropy of each cluster and can significantly improve the performance of domain adaptation. Our approach achieved competitive results against the prior arts on datasets collected from two hospitals.
A Box-Supervised Instance Segmentation Method for Insulator Infrared Images Based on Shuffle Polarized Self-Attention

Infrared imaging technology, as an effective on-site condition monitoring means of equipment, is widely used in the maintenance task of insulators. Accurate segmentation of insulators in infrared images is of great significance for temperature data acquisition and fault diagnosis. The presence of multiple electrical devices in the infrared image substantially increases the difficulty of insulator instance segmentation. Moreover, segmentation methods based on pixel-level sample annotation consume a lot of time and resources. Therefore, a box-supervised instance segmentation method for insulator infrared images based on shuffle polarized self-attention is proposed. Firstly, an adaptive positive and negative sample matching mechanism is designed in an anchor-free detector, which adaptively adjusts the constraint range of sample matching according to the inherent characteristics of the insulator. Secondly, a shuffle polarized self-attention module is embedded in the mask branch to improve the segmentation accuracy of insulator infrared images by feature shuffling and polarized filtering. Thirdly, the insulator weakly supervised segmentation is realized with box annotations by introducing a pairwise loss term and a projection loss term in the training strategy. The experiments show that the proposed method displays superior segmentation performance compared with other advanced method algorithms, which can effectively improve the intelligence of insulator fault diagnosis in transmission lines.
O-2A: Outlier-Aware Compression for 8-bit Post-Training Quantization Model

Post Training Quantization (PTQ) is a practical and cost-effective technique that reduces main memory footprint of Deep Neural Networks (DNNs). However, the effectiveness of PTQ is limited by a notable decrease in accuracy when the precision falls below 8 bits. To overcome this limitation of PTQ, we present a new compression method called Outlier Aware Approximation (O-2A) that compresses 8-bit PTQ models into lower precision with minimal accuracy loss. In O-2A, parameters are classified into outliers and non-outliers. Critical bits of outliers are preserved, while unnecessary bits of non-outliers are removed, resulting in reduced compression error. Nevertheless, compressing outliers using O-2A below 6 bits precision poses challenges due to significant compression error. To achieve more aggressive compression, we introduce multiple levels of O-2A (mO-2A), where outliers are divided into different levels, and compression error is minimized. We evaluate our techniques on the ImageNet dataset using the Pytorch framework. The results demonstrate that our methods outperform previous works in term of accuracy with the same compression efficiency.
Cross-Parametric Generative Adversarial Network-Based Magnetic Resonance Image Feature Synthesis for Breast Lesion Classification

Dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) contains information on tumor morphology and physiology for breast cancer diagnosis and treatment. However, this technology requires contrast agent injection with more acquisition time than other parametric images, such as T2-weighted imaging (T2WI). Current image synthesis methods attempt to map the image data from one domain to another, whereas it is challenging or even infeasible to map the images with one sequence into images with multiple sequences. Here, we propose a new approach of cross-parametric generative adversarial network (GAN)-based feature synthesis (CPGANFS) to generate discriminative DCE-MRI features from T2WI with applications in breast cancer diagnosis. The proposed approach decodes the T2W images into latent cross-parameter features to reconstruct the DCE-MRI and T2WI features by balancing the information shared between the two. A Wasserstein GAN with a gradient penalty is employed to differentiate the T2WI-generated features from ground-truth features extracted from DCE-MRI. The synthesized DCE-MRI feature-based model achieved significantly (p = 0.036) higher prediction performance (AUC = 0.866) in breast cancer diagnosis than that based on T2WI (AUC = 0.815). Visualization of the model shows that our CPGANFS method enhances the predictive power by levitating attention to the lesion and the surrounding parenchyma areas, which is driven by the interparametric information learned from T2WI and DCE-MRI. Our proposed CPGANFS provides a framework for cross-parametric MR image feature generation from a single-sequence image guided by an information-rich, time-series image with kinetic information. Extensive experimental results demonstrate its effectiveness with high interpretability and improved performance in breast cancer diagnosis.
Query-Efficient Black-Box Adversarial Attacks on Automatic Speech Recognition

The susceptibility of Deep Neural Networks (DNNs) to adversarial attacks has raised concerns regarding their practical applications in real-world scenarios. Although the vulnerability of DNNs to adversarial attacks has been extensively studied in the image domain, research in the audio domain, particularly in the black-box setting with Automatic Speech Recognition (ASR) models, remains limited. While various black-box attacks have been proposed for ASR models, such as transfer attacks, hardware attacks, and query-based attacks, this study concentrates on query-based black-box attacks. The article introduces a new gradient estimation technique, Temporal Natural Evolution Strategies (T-NES), to generate adversarial audio samples more efficiently than existing attacks. T-NES leverages the temporal correlation present in audio to speed up gradient estimation based on the probability scores returned by the target model. The empirical results on benchmark datasets, LibriSpeech and TEDLIUM, and two state-of-the-art ASR models, DeepSpeech2 and Wav2Letter, demonstrate that T-NES can generate successful attacks with up to 30% fewer queries than existing attacks within 500 queries. T-NES could provide a robust baseline for evaluating the black-box adversarial vulnerability of ASR systems.
Evaluation of an HMD-Based Multisensory Virtual Museum Experience for Enhancing Sense of Presence

Advancements in digital technology have enabled the production of head-mounted display (HMD)-based digital cultural heritage virtual reality contents that provide immersive experiences through audiovisual interaction in a three-dimensional virtual space. However, these contents do not support tactile sensations and only provide audiovisual information using a controller, which limits users’ perception of cultural assets. In this study, we propose an HMD virtual reality-based multisensory digital cultural heritage content that increases the feeling of presence by giving tactile sensations in addition to auditory and visual senses. We present a virtual museum content that implements dummy objects and virtual objects that are 3D printed by high-precision scanning of artifacts and connects their locations with sensors so that users can touch and move them in various directions to feel their shape and size and enhance the sense of reality. Thirty-two participants were recruited to conduct quantitative and qualitative evaluations on realism, immersion, and preference, to compare the proposed multisensory method with the traditional audiovisual method. The experimental results showed that the HMD-based multisensory virtual museum enhanced realism, as users can naturally touch the dummy and feel the presence of artifacts through bare-hand interaction and that immersion and preference are enhanced through the novel multisensory experience.
Leveraging a Smartwatch for Activity Recognition in Salat

Salat, the most important worship of Muslims and the second pillar of Islam, is an integral part of the Muslim community. Being a complex activity, Salat involves a series of steady and transitional activities to be performed in a specific sequence. On top of that, Salat has variations based on time, priority, school of thought, etc., making activity recognition in Salat more challenging. Existing research studies related to recognizing individual activities in Salat either demand capturing images by a camera or carrying a smartphone (sometimes in inconvenient places) while praying. Both of the demands are not convenient or applicable in real cases. Besides, the existing studies lack user-independent accuracy analysis and fine-grained prediction. To address these gaps, in this study, we first assess the requirement and acceptability of technological solutions for activity recognition in Salat by conducting an exploratory study. Upon establishing the requirement, we propose an activity recognition methodology using a smartwatch to recognize different activities in Salat. We prepare a Salat activity dataset using a smartwatch and propose a new methodology using semantic rules and Dynamic Time Warping (DTW) that achieves a near-perfect accuracy (99.3%) in recognizing activities in Salat. Besides, our proposed methodology offers fine-grained recognition of the individual activities in Salat and is robust enough to overlook the extra transitional activities a person performs while praying, which does not nullify Salat. Therefore, this research is expected to lead to a comprehensive solution for monitoring Salat.