Reviewerly/exHarmony huggingface dataset:
Synthetic Monocular Depth Estimation Dataset for Cataract Surgery Assistance

In computer-assisted surgeries, monocular depth estimation plays an important role, which provides navigation for surgeons by computing precise depth information. In recent years, depth estimation has achieved significant breakthroughs with the application of deep learning. However, the lack of depth ground truth in the ophthalmology surgery scene has become an obstacle to the development of depth estimation in this scene. To resolve this problem, we built one synthetic dataset for cataract surgeries. The dataset contains information on RGB images, depth maps, and segmentation masks. We also adopt the state-of-the-art methods of depth estimation on this dataset as the baseline model to build the benchmark. We also analyze the generalization of the baseline models trained on the synthetic dataset to the real surgical scene.
IMAN: An Iterative Mutual-Aid Network for Breast Lesion Segmentation on Multi-modal Ultrasound Images

In the past decade, significant advancements have been made in utilizing deep learning for breast lesion segmentation. Recently, researchers have increasingly focused on harnessing the power of multiple modalities, recognizing its potential for enhancing segmentation performance. We observe that in clinical practice, many radiologists often rely on two types of ultrasound images, namely ultrasound (US) and contrast-enhanced ultrasound (CEUS) data for diagnosis. This motivates us to propose a multi-modal segmentation network, called as IMAN (Iterative Mutual-Aid Network), based on these two modalities. The architecture of IMAN adopts a novel hourglass shape, featuring two branches connected by an ‘X’ pathway. One branch is dedicated to processing CEUS data, while the other branch handles US data. Each branch generates segmentation results specific to its respective modality. The ’X’ pathway, realized by a margin mask generator module, serves as a bridge between these branches by forcing the segmentation results from one branch as additional input to the other. This head-to-tail pathway effectively facilitates mutual aid between the two modalities. In addition, we propose an iterative training policy during the training process to fully exploit the information from both US and CEUS data. Experimental results on a Breast-US-CEUS dataset comprising 169 samples demonstrate the effectiveness of IMAN, achieving Dice Similarity Coefficient of 83.96% and 81.16% for US images and CEUS videos, respectively. These scores surpass those obtained by many state-of-the-art segmentation methods. Furthermore, IMAN exhibits robust generalization capabilities across different segmentation structures.
Med-DualNet: Enhancing Medical Image Segmentation with Hard-Patch Mining and Joint Error Adjustment

Automated segmentation of organs and lesions is crucial for effective treatment design and care planning in the medical field. However, accurately segmenting organs and lesions is challenging due to the complex distribution of lesions within organs, such as cases with early diseases characterized by small volumes and blurred boundaries. In this work, we propose Med-DualNet, a dual-branch segmentation network that incorporates Hard-Patch Mining and Joint Error Adjustment strategies to address these challenges. Branch I focuses on organ segmentation, while Branch II targets challenging lesion segmentation using a two-stage network and dedicated strategies. The Hard-Patch Mining strategy in Stage-one dynamically generates a subset of image patches that highlight challenging aspects of lesion segmentation, including small volumes and blurred boundaries. Furthermore, the Joint Error Adjustment strategy in Stage-two facilitates the learning of challenging lesion features. Experimental results on three public medical datasets demonstrate the improved performance of the proposed model in organ and lesion segmentation, as measured by visual and quantitative metrics.
Implicit Feature Augmentation with Feature Transfer For Class-Imbalanced Medical Image Classification

The class imbalance problem, which is prevalent in medical image datasets, seriously affects the diagnostic effectiveness of deep learning-based network models. To alleviate this problem, data re-sampling and loss re-weighting techniques are often used to reshape the decision boundary of the classifier. However, these techniques still lead to biased decision boundary due to the lack of sufficient and diversified samples in the tail classes of medical image datasets. In this paper, we propose an implicit feature augmentation with feature transfer (FT-IFA) method which solves the class imbalance problem by expanding the feature space of tail classes to reshape the decision boundary of the classifier. FT-IFA utilizes prototype similarity to transfer the rich transformation information from the head classes to tail classes on the basis of the balanced feature space, enriching the intra-class diversity of tail classes. Experimental results on two class-imbalanced medical image datasets show that FT-IFA outperforms the current state-of-the-art methods and effectively solves the class imbalance problem.
Objects Affect Where We Look: Improving Human Attention Prediction Using Extra Data

Saliency Prediction aims to predict the attention distribution of human eyes given an RGB image, which has been widely considered in human-computer interaction. Most of the recent state-of-the-art methods are based on deep image feature representations from traditional CNNs. However, the traditional convolution could not capture the global features of the image well due to its small kernel size. Besides, the high-level factors which closely correlate to human visual perception, e.g., objects, color, light, etc., are not considered. Inspired by these, we propose a Transformer-based method with object segmentation as another learning objective. More global cues of the image could be captured by Transformer. In addition, simultaneously learning the object segmentation simulates the human visual causing factor. We build an extra decoder for the subtask and the multiple tasks share the same Transformer encoder, forcing it to learn from multiple feature spaces. We find in practice simply adding the subtask might confuse the main task learning, hence Multi-task Attention Module is proposed to deal with the feature interaction between the multiple learning targets. Our method boosts the performances on SALICON and CAT2000 benchmarks compared to other methods, which are two famous datasets for human attention prediction.
A Novel Approach for Cancer Image Segmentation

In this study, we present a novel approach to address the task of identifying nuclei in a diverse set of microscopy scan images containing slides from several distinct cancerous tissue types. Our approach is based on YOLOv5 algorithm. We thoroughly discuss the crucial training specifics that contributed to our outcomes. Additionally, we propose an intuitive strategy for combining multiple segmentation results to further improve the accuracy of nucleus identification. By employing this lightweight model, we successfully achieved results better or close to the state-of-the-art performance.
Prostate MRI Super-Resolution using Discrete Residual Diffusion Model

Prostate cancer (PCa) is one of the most common malignant tumors. High-resolution magnetic resonance imaging (HR MRI) is an effective tool for diagnosing PCa, but it requires patients to remain immobile for extended periods, increasing chances of image distortion due to motion. One solution is to utilize super-resolution (SR) techniques to create a higher-resolution MRI. However, existing medical SR models suffer from issues such as excessive smoothness and mode collapse. In this paper, we propose a novel generative model avoiding the problems, called Prostate MRI Super-Resolution using Discrete Residual Diffusion Model (DR-DM). First, the forward process of DR-DM gradually disrupts the input via a fixed Markov chain, producing a sequence of latent variables. The backward process optimizes a variant of the variational lower bound, training diffusion models effectively address the mode collapse. Second, to focus DR-DM on recovering high-frequency details, we synthesize residual images instead of synthesizing HR MRI directly. The residual image represents the difference between the HR and LR up-sampled MR image, and we convert residual image into discrete image tokens with a shorter sequence length by a vector quantized variational autoencoder (VQ-VAE), which reduced the computational complexity. Third, transformer architecture is integrated to model the relationship between LR MRI and residual image, which can capture the long-range dependencies between LR MRI and the synthesized imaging, thereby improving the fidelity of the reconstructed images. Our experiments on the Prostate-Diagnosis and PROSTATEx datasets demonstrate that the DR-DM model significantly improves image quality, resulting in greater clarity and improved diagnostic accuracy for patients.
Detection of Day-Based Health Evidence with Pretrained Large Language Models: A Case of COVID-19 Symptoms in Social Media Posts

Gathering the information pertaining to health evidence occurring on particular days can help understand the progression of a disease over time, and such evidence as symptoms was widely shared by the social media users during the COVID-19 pandemic. Identifying this type of evidence is challenging. In this work, we investigated pretrained large language models on their ability to identify the day-based COVID-19 symptom mentions in Twitter posts. Our results on a corpus of 635 tweets show that without any supervision and optimization both GPT-3.5 and GPT 4 models achieved impressive performance, much better than Web-based ChatGPT and Google Bard. In addition, we explored and utilized GPT-4’s ability to determine the number of matches between a list of predicted symptom expressions and a list of annotated symptom expressions, reducing the effort of designing a sophisticated algorithm for finding the matches.
CVFC: Attention-Based Cross-View Feature Consistency for Weakly Supervised Semantic Segmentation of Pathology Images

Histopathology image segmentation is the gold standard for diagnosing cancer, and can indicate cancer prognosis. However, histopathology image segmentation requires high-quality masks, so many studies now use image-level labels to achieve pixel-level segmentation to reduce the need for fine-grained annotation. To solve this problem, we propose an attention-based cross-view feature consistency end-to-end pseudo-mask generation framework named CVFC based on the attention mechanism. Specifically, CVFC is a three-branch joint framework composed of two Resnet38 and one Resnet50, and the independent branch multi-scale integrated feature map to generate a class activation map (CAM); in each branch, through down-sampling and The expansion method adjusts the size of the CAM; the middle branch projects the feature matrix to the query and key feature spaces, and generates a feature space perception matrix through the connection layer and inner product to adjust and refine the CAM of each branch; finally, through the feature consistency loss and feature cross loss to optimize the parameters of CVFC in co-training mode. After a large number of experiments, An IoU of 0.7122 and a fwIoU of 0.7018 are obtained on the WSSS4LUAD dataset, which outperforms HistoSegNet, SEAM, C-CAM, WSSS-Tissue, and OEEM, respectively.
Disease Diagnosis based on Multiple Semantic Relationship Prompt Subgraph

Existing disease diagnosis models are often trained on clinical data such as electronic medical records (EMRs), which are lack of knowledge guidance. In this paper, we propose a new disease diagnosis model based on Multiple Semantic Relationship Prompt Subgraph (MSRPS), which can efficiently incorporate medical knowledge and utilize pretrained language models (PLMs) to achieve reasonable and accurate diagnosis results. Firstly, the MSRPS is extracted from the medical knowledge graph based on the patient’s personalized condition. Then graph encoder is used to generate a continuous prompt learning template based on graph neural network (GNN). Finally, a disease diagnosis model with prompt learning template under pre-trained and prompt paradigm is constructed to predict diagnosis results. Compared with the fine-tuning approach, this method needs fewer trainable parameters and less training data but achieve better performance. Experiments are conduct on two multi-label disease diagnosis datasets: ChineseEMR-50 and MIMIC-III-50. Results demonstrate that our model can be used in various pre-trained models and achieve the state-of-the-art results.
Adversarial Multi-Sample Interpolation for Medical Image Segmentation

Deep learning-based medical segmentation methods suffer from the limited amount of data due to data privacy and ethical issues, which results in generalization degradation and hinders their applications in clinical scenarios. In this paper, we propose a novel data augmentation method named Adversarial Multi-Sample Interpolation (AMSI) based on adaptive interpolations among multiple latent codes to produce new data samples. AMSI first introduces an AutoDecoder to construct the latent codes for given data points and then augment the latent space through learning-based adaptive interpolations. Specifically, we design a multi-dimension interpolation among multiple samples with learnable coefficients to adaptively generate more diverse hard examples via adversarial training. Furthermore, our method generates both synthesized images and pseudo labels simultaneously, which is different against previous style-augmented methods. Additionally, the adjacent relationship between latent points will be considered for the sake of realistic appearance and smooth pseudo labels. The effectiveness and generalization ability of AMSI are validated by extensive experiments both on two segmentation tasks employing four publicly available datasets.
Simultaneous Super-Resolution and Denoising on MRI via Conditional Stochastic Normalizing Flow

Magnetic resonance imaging (MRI) scans often suffer from noise and low-resolution (LR), which affect the diagnosis and treatment results obtained for patients. LR images and noise come together with MRI, and the existing methods solve image super-resolution (SR) reconstruction and denoising tasks in a step-by-step manner, which influences the overall real distribution of the MRI data. In this paper, we present a simultaneous SR and denoising algorithm based on a stochastic normalizing flow (SNF), named the MR image SR and denoising model based on an SNF (SRDSNF). SRDSNF adds the encoded information of the input image as the conditional information to each reverse step of the stochastic normalizing flow, which realizes a consistent description of the spatial distribution between the reconstruction result and the input image. We introduce rangenull space decomposition and subsequence sampling strategies to enhance the consistency of the input and output data and increase the generation speed of the model. Simultaneous SR and denoising tasks experiment is carried out using the BrainWeb and NFBS datasets. The experimental results show that good SR and denoising results are obtained with fewer sampling steps, these results are consistent with the ground truths, and the structural similarity and peak signal-to-noise ratio of the results are also higher than those of the comparison methods. The proposed method demonstrates potential clinical promise.
SWT-UNet: Medical Image Segmentation Based on Multi-modality UNet with Sliding Window Transformer Block

Medical images are information carriers that visually reflect and record the anatomical structure of the human body, and play an important role in clinical diagnosis, teaching and research, etc. Modern medicine has become increasingly inseparable from the intelligent processing of medical images. In recent years, there have been more and more attempts to apply deep learning theory to medical image segmentation tasks, and it is imperative to explore a simple and efficient deep learning algorithm for medical image segmentation. In this paper, we conduct research on multi-modal medical image segmentation algorithms with a hybrid architecture of Convolutional Neural Networks and Vision Transformer. This paper proposes a multi-modal medical image segmentation model SWT-UNet based on the CNN-ViT hybrid framework. The self-attention mechanism and sliding window design of the visual Transformer are used to capture the global feature association and break the limitation of the receptive field caused by the inductive bias of the convolution operation. At the same time, a widened self-attentive vector is used to streamline the number of modules and compress the model size, so as to fit the characteristics of a small amount of medical data, which makes the model easy to be overfitted. Experiments on two multi-modal medical image datasets validate that the algorithm can achieve efficient medical image segmentation at a lightweight scale.
R2L-Net: Rapid Medical Image Segmentation Network Regularized by Self-Supervised Relative Localization Task

Medical image segmentation plays a crucial role in medical imaging, especially with advancements in techniques like magnetic resonance imaging (MRI) and computed tomography (CT). UNet, a widely used architecture, has shown promising results in medical image segmentation. Several variants based on UNet, and Transformer-based models, like TransUNet, have also exhibited potential for improving segmentation performance. However, these models often require substantial data and computational resources, making them less suitable for on-the-fly segmentation in medical scenarios. This paper proposes a fast medical image segmentation network called R2L-Net, which leverages self-supervised and supervised learning. R2L-Net introduces a self-supervised relative localization task as a regularization term during network training to enhance performance. Compared to UNeXt, our proposed R2L-Net achieves superior results on two public datasets (ISIC and BUSI), with an improved Intersection over Union (IoU) by 5.22 and 5.36, respectively. Moreover, R2L-Net offers several advantages over existing models, including a small number of parameters, low computational complexity, and fast image processing.
Steganography Platform

The Steganography Platform is an advanced data security solution that employs machine learning, particularly GANs, to deliver secure steganographic services by seamlessly embedding text within images and audio files without perceptible alterations. By surpassing the limitations of conventional techniques, this approach offers greater capacity and heightened security. This documentation presents an overview of the Steganography Platform, encompassing its motivation, objectives, and implementation specifics. The StegOBASE model builds upon the SteganoGAN architecture, yielding exceptional outcomes in terms of capacity, security, file size, and processing time. Accessible through an API, web app, and desktop client, the platform also provides a user manual for effortless utilization. In conclusion, the Steganography Platform propels data security forward by harnessing the power of machine learning and GAN-based models, thereby contributing to the advancement of data protection measures.
Multilingual Model Fine-tuning for Sentiment Analysis

Multilingual language models have decreased the barrier between languages, as it will be helpful overcoming many problems, such as sentiment analysis because the importance of this task is to make good decisions and customize products. Obtaining information from one language can help other languages generalize and understand a task more effectively. In this paper, we propose a general method for sentiment analysis of data that includes data from many languages, which enables all applications to use sentiment analysis results in a language-blind or language-independent manner. We performed experiments on two language combinations (English and Arabic) for sentence-level sentiment classification and found that the model with the final setup after adding translations from one language to another and fine-tuning the multilingual language model for Twitter, was the best setup, achieving for two languages and 71.2% and 68.1% f1-score for English and Arabic, respectively.
Ballbot Simulation System: Modeling, Verification, and Gym Environment Development

This paper introduces the development of a sophisticated simulation model for the ball-balancing robot (ballbot), implemented as a Gymnasium (Gym) environment in Python. The main purpose of this environment is to facilitate the application of Reinforcement Learning (RL) techniques to effectively control the ballbot system. Initially, a standard ballbot model is created using Solidworks, and a Unified Robot Description Format (URDF) file is generated to precisely capture the dynamics of the ballbot. To validate the model, simulations are performed within MATLAB/Simulink, comparing the ballbot URDF model against its nonlinear mathematical model with a simple PID controller. The simulation results demonstrate that the URDF model accurately represents the ballbot dynamics, exhibiting a comparable response to the mathematical model. Subsequently, the high-fidelity URDF model is integrated into the Gym environment using the Pybullet simulator. The ongoing objective of this research is to utilize the developed ballbot environment for RL-based control design.
Grab and Go: Leveraging Computer Vision for Intelligent Shopping

In an effort to address the challenges posed by long queues and slow payment processes in shopping, new technologies like Amazon's "Just Walk Out" have emerged to enable checkout-free purchases. Computer vision methods, specifically the application of On-Shelf Availability (OSA) monitoring, have been highlighted as crucial for efficient inventory management in grocery stores, using semi-supervised learning and YOLO deep learning architecture. However, the high cost associated with implementing such comprehensive smart market solutions, including the need for RFID sensors on every item, may hinder adoption in developing countries. This paper aims to provide a more affordable alternative by leveraging computer vision techniques, replacing sensors with cameras, and utilizing deep learning models. The proposed approach involved training YOLO V5 and various versions of YOLO V8 models, incorporating annotations, data augmentations, and exploring different techniques. The achieved results demonstrated a mean Average Precision (mAP) of 78.3%, precision of 64.2%, and recall of 61.4% using YOLO V8 with augmentations such as Mosaic, Horizontal Flip, Blur, etc. While the results were considered acceptable even with limited access to natural images, the performance is expected to improve further with increased access to a wider range of training images, especially natural ones.