Reviewerly/exHarmony huggingface dataset:
Adaptively Multi-Objective Adversarial Training for Medical Image Report Generation

Medical Image Report Generation aims to automate the creation of medical reports from radiology images, alleviating the burden on radiologists. The challenge lies in producing text that aligns with the image content, highlights local anomalies, and adheres to professional norms. In this study, we restructure this task as a multi-objective optimization problem and introduce a novel adversarial training framework, MORGAN, with multiple discriminators to guide the generation of reports akin to those written by professional radiologists. MORGAN employs three discriminators to assess the alignment of image-text, focus on local anomalies, and adherence to medical norms when generating each token. These assessments serve as reinforcement learning rewards, guiding the generator to optimize these objectives in real time. We extensively test MORGAN on two prevalent chest-related disease datasets. The results demonstrate that MORGAN significantly enhances the generator’s performance on most metrics.
Parsing Clinical Trial Eligibility Criteria for Cohort Query by a Multi-Input Multi-Output Sequence Labeling Model

To enable electronic screening of eligible patients for clinical trials, free-text clinical trial eligibility criteria should be translated to a computable format. Natural language processing (NLP) techniques have the potential to automate this process. In this study, we explored a supervised multi-input multi-output (MIMO) sequence labelling model to parse eligibility criteria into combinations of fact and condition tuples. Our experiments on a small manually annotated training dataset showed that that the performance of the MIMO framework with a BERT-based encoder using all the input sequences achieved an overall lenient-level AUROC of 0.61. Although the performance is suboptimal, representing eligibility criteria into logical and semantically clear tuples can potentially make subsequent translation of these tuples into database queries more reliable.
UMCL: Unified Medical Image-Text-Label Contrastive Learning With Continuous Prompt

Contrastive Language-Image Pre-training (CLIP) can leverage large dataset of unlabeled Image-Text pairs, which have demonstrated impressive performance in various downstream tasks. Given that annotating medical data is timeconsuming and laborious, Image-Text Pre-training has promising applications in exploiting large-scale medical image and radiology report dataset. However, medical Image-Text Pre-training faces several challenges, as follows: (1) Due to privacy concerns, the amount of available medical data is relatively small compared to natural data, leading to weaker generalization ability of the model. (2) Medical images are highly similar with only finegrained differences in subtleties, resulting in a large number of false-negative sample pairs in comparison learning. (3) The hand-crafted Prompt usually differs from the natural medical image report, Subtle changes in wording can lead to significant differences in performance. In this paper, we propose a unified Image-Text-Label contrastive learning framework based on continuous prompts, with three main contributions. First, We unified the data of images, text, and labels, which greatly expanded the training data that the model could utilize. Second, we address the issue of data diversity and the impact of hand-crafted prompts on model performance by introducing continuous implicit prompts. Lastly, we propose a Image-Text-Label contrastive Training to mitigate the problem of too many false-negative samples. We demonstrate through sufficient experiments that the Unified Medical Contrastive Learning (UMCL) framework exhibits excellent performance on several downstream tasks.
A parameter adaptive tuning algorithm for medical image recognition model testing

Medical image recognition methods based on deep learning typically have extremely high requirements for the distribution consistency of source domain data and target domain data. In real-world deployment scenarios, there are often significant differences in target domain data generated by devices from different medical centers and manufacturers. When a medical image recognition model recognizes target domain data that is different from its source domain data distribution, its generalization performance will decrease, greatly reducing the credibility of the medical image recognition model in practical applications.In response to the above issues, this article proposes a parameter adaptive tuning algorithm for medical image recognition model testing. The algorithm uses parallel networks to learn the characteristics and distribution of target domain data, while keeping the parameters of the main network unchanged during adaptation. By optimizing the parameters of the parallel network through backpropagation and gradient descent, it can avoid catastrophic forgetting of the model while also learning the characteristics and distribution of target domain data to a certain extent. In the prediction stage, the fusion network is used to predict the target and improve the generalization performance of the model. This article uses YOLOv5 as the basic model and proposes a parameter adaptive tuning algorithm for medical image recognition model testing. The accuracy of the algorithm on datasets FPPD and DRCOLO has been improved to 94.2% and 97.8%, and the mAP on multiple Disease-Colo video datasets has been improved to 30.5% and 41.7%, verifying the effectiveness of the algorithm.
A novel few-shot learning approach for the classification of histological images of liver tumors

The liver is an internal organ that is most commonly affected by cancer metastasis. Hepatocellular carcinoma (HCC) is a major cause of cancer-related deaths worldwide. Identifying the histological category of this disease is a crucial step in determining the most appropriate treatment plan. Currently, the classification of HCC primarily relies on the observation of pathologists through a visual microscope, which is undoubtedly subjective. To address this issue, the present study conducted pathological examination of liver tumors based on few-shot learning. In order to achieve automatic identification of the three categories in HCC tissue, we propose a few-shot learning based HCC histological image classification method that combines contrastive learning (CL) using different backbone architectures with several well-known machine learning methods. We evaluate 12 backbone-classifier combinations, and the best performance is achieved by combining CL using DeiT-TinyR as the backbone architecture with support vector machine (SVM), with an accuracy of 74.42% and an F1 score of 74.58% in the 3-way-10-shot setting. Experimental results demonstrate the effectiveness of this method in NB histological image classification.
Infectious Disease Forecasting using Multivariate Incomplete Time-series: A Hybrid Architecture with Stacked Dilated Causal Convolutions

The COVID-19 pandemic brought into focus the importance of accurately predicting the future spread of an infectious disease. Hybrid neural networks combining standard convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have previously been utilized for infectious disease forecasting. This study enhances previous research by introducing a novel architecture using a Stacked-dilated-causal Convolutional Neural Network and Bidirectional Long Short-Term Memory (SCNN-BiLSTM) to forecast the spread of an infectious disease by accommodating incomplete multivariate temporal sequences. Unlike standard CNN, stacked-dilated-causal convolutions provide full coverage history, where all previous elements in the time-series input window are modelled to predict the next output.COVID-19 infection data in Ontario, Canada was selected to demonstrate the effectiveness of our approach in infectious-disease forecasting. The two main contributions of this study are the following: (1) proposing Stacked-dilated-causal Convolution (SCNN) along with Bidirectional Long Short-Term Memory to create a hybrid architecture (SCNN-BiLSTM) that can model full coverage history for infectious disease forecasting; (2) developing a multivariate model that is capable of incorporating multiple incomplete input sequences. The results obtained from this study show that the proposed architecture can successfully predict the spread of infectious disease with a long forecasting horizon by utilizing multivariate data even in the absence of complete temporal sequences for all variables.
MSCCNet: Multi-Scale Convolution-Capsule Network for Cervical Cell Classification

Cervical cancer is one of the fastest growing and most dangerous cancers, seriously threatening women’s health and lives. Cervical cytopathology image classification is a very important approach for diagnosing cervical cancer. The advent of the automatic computer-aided diagnosis system can tackles this issue. However, cervical cell images of different classes exhibit similar appearances, posing a challenge for accurate classification. To address this challenge, this work proposes a framework named MSCCNet. In our MSCCNet, the cross-layer attention-based feature fusion module is used to obtain multi-scale discriminative features. Meanwhile, the spatial relationship modeling module is utilized to encode the relative relationship between objects and capture more slight differences between cervical cells, further strengthening the representation ability of features. We also introduce the joint loss to enhance the penalty for misclassified samples. The model training and evaluation are performed on our developed DSCC dataset and publicly available SIPaKMeD datasets. The proposed MSCCNet achieves overall accuracies of 87.88% and 97.90% on these two datasets, respectively, outperforming several existing classification methods.
A Transformer-based Patient Clustering Method in Continuing Care Retirement Communities

Patient Clustering for elderly care in a continuing care retirement community (CCRC) is a very significant problem on management of diversified patient data. In recent years, deep clustering methods which learn a clustering-friendly representation result in a significant increase of clustering performance. In this paper, we present a Transformer-based patient clustering method based on tag graph embedding and binomial expectation-maximization (BEM) Transformer. First, we present a tag graph dimensionality reduction algorithm to represent a patient as an embedding vector. Then, we propose a patient clustering method based on BEM Transformer where we multiply the length of query vector and the length of key vector with the scaling factor of the original Transformer [1] to make the training process more stable. Our proposed Transformer-based patient clustering method is effective for clustering the patients in the CCRCs.
Enhanced Knowledge Injection for Radiology Report Generation

Automatic generation of radiology reports holds crucial clinical value, as it can alleviate substantial workload on radiologists and remind less experienced ones of potential anomalies. Despite the remarkable performance of various image captioning methods in the natural image field, generating accurate reports for medical images still faces challenges, i.e., disparities in visual and textual data, and lack of accurate domain knowledge. To address these issues, we propose an enhanced knowledge injection framework, which utilizes two branches to extract different types of knowledge. The Weighted Concept Knowledge (WCK) branch is responsible for introducing clinical medical concepts weighted by TF-IDF scores. The Multimodal Retrieval Knowledge (MRK) branch extracts triplets from similar reports, emphasizing crucial clinical information related to entity positions and existence. By integrating this finer-grained and well-structured knowledge with the current image, we are able to leverage the multi-source knowledge gain to ultimately facilitate more accurate report generation. Extensive experiments have been conducted on two public benchmarks, demonstrating that our method achieves superior performance over other state-of-the-art methods. Ablation studies further validate the effectiveness of two extracted knowledge sources.
A Latent Adversarial Cauchy-Schwarz Autoencoder for Medical Image Segmentation

Medical image segmentation plays a vital role in clinical diagnosis. However, previous methods cannot handle intrinsic ambiguities in extracting deep semantics of medical images. Moreover, they neglect fruitful semantic information in segmentation maps. To address the challenges, a latent adversarial Cauchy-Schwarz autoencoder is proposed, which defines image segmentation as a cross-modal translation task from medical images to segmentation maps. Specifically, a probabilistic graph model is defined to fit the conditional distribution of the image translation between modalities, which leverages the Cauchy-Schwarz divergence to alleviate approximation errors caused by ambiguities in extracting deep semantics. Then, a novel numerical solution is derived to optimize the probabilistic graph model, which explores semantics in segmentation maps to facilitate the segmentation. Afterwards, a dual-flow architecture is proposed with an adversarial encoding-decoding paradigm to implement the numerical solution. Finally, extensive experiments in two medical scenarios illustrate that the proposed method achieves the state-of-the-art performance compared with nine baseline methods.
MixUNet: Mix the 2D and 3D Models for Robust Medical Image Segmentation

Brain tumor segmentation is pivotal in the diagnosis and treatment of brain tumors. As functional imaging technologies like CT and MR advance, analyzing 3D medical image data becomes more time-consuming. Several challenges exist in 3D medical image segmentation: 1) 2D networks, when applied to 3D segmentation tasks, suffer from a lack of 3D structural information. 2) Pure 3D networks, due to their vast parameter count and smaller training sample, are susceptible to overfitting. 3) Current 2.5D networks do not fully leverage the available 3D structural information. In this study, we introduce the Mix-UNet, a multi-branch network that synergizes 2D and 3D networks. This design preserves essential 3D structural details for precise segmentation while ensuring computational efficiency. Our model comprises two main branches and a fusion module: a 2D branch for coarse segmentation without 3D structural information, a 3D branch to capture comprehensive 3D structural details, and a fusion module for pixel-level integration to produce the final segmentation. Experimental results demonstrate the model’s ability to reduce parameter count, increase robustness, and maintain high precision. When tested on the BraTS 2020 validation dataset, our model achieved mean dice coefficients of 90.4%, 80.7%, and 71.2% for the whole tumor, tumor core, and enhancing tumor, respectively, with only 2.2M parameters.
Stain Normalization and Augmentation in Frequency Space for Histology Analysis

Histopathology plays an indispensable role in cancer diagnosis. However, the frequent staining variations observed in tissue slides pose significant challenges to the effectiveness of computer-aided diagnostic systems. Stain Normalization (SN) and Stain Augmentation (SA) have been proposed to mitigate these issues. Traditional SN techniques focused on minimizing variations between slides, whereas SA sought to enhance data distribution without changing the inherent morphology, mainly within the image domain. Notably, many medical signals originate from frequency information, often processed in specialized facilities. Motivated by this, we introduce a novel approach: harnessing the frequency domain for SN and SA. Our proposed Fourier Stain Normalization (F-SN) and Fourier Stain Augmentation (FSA) methods innovatively mark a significant departure from conventional techniques by leveraging the focus on the frequency domain. Through the application of Fourier transformation, we transmute time-domain signals into the frequency domain, resulting in a streamlined and more efficient preprocessing phase compared to time-domain-based methodologies. Enhancing this approach, we incorporate a learnable mask prediction mechanism, adeptly merging source and target images, which bolsters its adaptability for a spectrum of downstream tasks. Comprehensive evaluations spanning medical image classification and segmentation using diverse network architectures, affirm the superiority of our F-SN and F-SA techniques over traditional time-domain methods, underscoring the frequency domain’s richer informational content. The code is available at https://github.com/Windyskys/FSNFSA.
Non-rigid Medical Image Registration Based on Unsupervised Self-driven Prior Fusion

Deformable image registration is a basic building block in intelligent bioinformatical analysis and biomedicine systems. With the rapid development of deep learning, the community has witnessed a great leap via this effective data-driven technique. Recently, the Vision Transformer, famous by its long-range modeling ability, has been successfully used in the field of medical image registration. However, the existing ViT based techniques are deemed to have certain limitations. Firstly, these methods often transplanted the transformer module directly into the networks, while did not dive deeper to explore its compatibility to practical registration tasks. Moreover, self-attention’s relatively rigid all-to-all patching strategy may cause undesirable discontinuity effect to the spatial calculation. To address these issues, we propose a novel medical image registration framework based on an efficient image prior learning and fusion mechanism. Unlike the existing prior-based registration methods, our model is capable of learning task-specific saliency priors, without the need of hand-crafted features, or heavy-loaded auxiliary tasks, or pre-acquired expensive annotations. Then, followed by a multi-scale patch embedding module, the self-driven saliency prior is integrated into a ViT block with an active feature fusion mechanism, to further expand our network’s structural learning abilities. Extensive experiments on multiple data sets have demonstrated the superior quality of the proposed framework. We believe this plug-and-play model will bring about more application potentials to the community (Project webpage: https://github.com/raincity212/SPF-Net).
Medical Extractive Question-Answering Based on Fusion of Hierarchical Features

With the combination of natural language processing and artificial intelligence techniques, medical extractive question-answering (Q&A) provides valuable insights and assists medical professionals in daily work and scientific research, answering medical questions rapidly and accurately, thus holding significant practical significance. Therefore, research on medical extractive Q&A holds significant practical significance. However, the current state of medical extractive question-answering lacks attention to the interaction and prediction layers in the model structure. To address these issues, this paper proposes the Integrating pre-trained multi-layer structural feature information based Bio-BERT (IPMF-Bio-BERT) approach. This method leverages the rich word vector representations generated by the pre-trained Bio-BERT model, incorporating semantic and syntactic structural information to obtain multi-dimensional and complementary interactive feature information. Additionally, we introduce a flexible guidance network based on interactive information, which combines iterative and pointer network techniques to enhance the predictive performance of the question-answering model. We evaluate our proposed model on the specialized biomedical extractive question-answering BioASQ corpus. Experimental results demonstrate that the IPMF-Bio-BERT training strategy enhances the recognition and predictive capabilities of medical extractive Q&A, we establish new state-of-the-art results by outperforming existing approaches.
ProBioRE: A Framework for Biomedical Causal Relation Extraction Based on Dual-head Prompt and Prototypical Network

Extracting relationships among events typically aims to recognize the precise relation between two given events. For the task of event causal relation extraction in the biomedical domain, it has been extremely challenging since the issue of class imbalance. Existing work proposes to address that challenge with certain traditional data augmentation methods, but this approach has been of limited help. Furthermore, existing work lacks fine-grained event relation identification for available datasets, but focuses only on determining whether two given events are causally related or not. With the development of pre-trained language models in recent years, inspired by prompt learning, we propose a framework for biomedical event causal relation extraction based on dual-head prompt augmentation and prototypical network filtering. By comparing the framework with existing methods, we validate the effectiveness of our method.
Integrating Automated Knowledge Extraction with Large Language Models for Explainable Medical Decision-Making

Large language models (LLMs) have demonstrated strong reasoning ability and inspired many previously unimaginable applications. In this paper, we aim to harness the strong reasoning capability of LLMs toward explainable medical diagnosis. As we know, deep learning has been widely adopted and shown improvement in medical diagnostics. However, it is often criticized for its lack of interpretability. To address this drawback, we propose the first method that innovatively combines Markov logic networks (MLNs) with external knowledge extracted using LLMs, aiming for improved both interpretability and accuracy. Specifically, our approach involves a new process, powered by LLMs and a search engine, to automatically collect and organize external medical knowledge. The outcome is a set of first-order logic (FOL) rules, which then become a key component for the following MLN-based diagnostic algorithm. The resulting MLN-based model can maintain the accuracy of deep networks while providing understandable reasoning for its decisions. By aiming to blend specific knowledge from the medical domain with LLM techniques, our work contributes towards the development of improved automatic diagnosis systems, with the potential for enhancing transparency and trust in medical diagnostics.
Lung Cancer Detection via Federated Learning

Lung cancer is one of the most common cancers worldwide. In 2020, there were an estimated 2.2 million new cases of lung cancer. It is often diagnosed at an advanced stage because in the early stages, it does not present particularly abnormal symptoms, such as cough and chest pain, which are often underestimated. Typically, it is detected through pronounced symptoms that lead to more in-depth diagnostic examinations or through other routine tests that reveal its presence almost by chance. The tools used for detection include chest X-rays, CT scans, and PET scans. Analyzing these allows the detection of abnormal formations in the early stages. From this perspective, artificial intelligence has set the goal, in recent years, of providing tools for image analysis capable of automatically, rigorously, and above all, promptly providing an initial classification of the detected formations. In particular, in the literature, it is possible to find various solutions that make use of Federated Learning. This method involves training deep learning models on expansive datasets distributed across multiple data centers. Significantly, it safeguards privacy by obviating the need to transmit sensitive patient data. The objective of this paper is to examine existing state-of-the-art solutions, emphasizing workflows and essential strategies. It considers the datasets employed and architectural decisions. The analysis extends to common issues encountered in the studied works, with a specific focus on challenges typical of the medical domain. The paper concludes by exploring potential future solutions to address these issues.