Reviewerly/exHarmony huggingface dataset:
Residual Transformer YOLO for Detecting Multi-Scale Crowded Pedestrian

Crowding and occlusion pose significant challenges for pedestrian detection, which can easily lead to missed and false detections for small-scale and occluded pedestrian objects in dense pedestrian scenarios. To enhance dense pedestrian detection accuracy, we propose the Residual Transformer YOLO (RT-YOLO) algorithm in this paper. The RT-YOLO algorithm enhances the multi-scale fusion strategy based on YOLOv7 and introduces a dedicated detection layer for small-scale occluded targets. It also integrates Resnet and Transformer structures to improve the small-scale feature layer and detection head, enhancing feature extraction capabilities. Additionally, the RT-YOLO algorithm incorporates the Normalization-based Attention Module (NAM) into the backbone and neck networks to identify the region of interest. The experiments demonstrate that on the CrowdHuman and WiderPerson datasets, at IOU (Intersection over Union) = 0.5, the overall improvement in mAP50 is 3.8% and 3.4%. In the IOU range from 0.5 to 1, the improvement in mAP50: 95 is 5.1% and 4%. RT-YOLO achieves an FPS of 67, maintaining real-time performance. On the VOC2007 dataset, mAP50 has been enhanced by 5.1%, indicating higher effectiveness and robustness.
Cross-Domain Few-Shot Sparse-Quantization Aware Learning for Lymphoblast Detection in Blood Smear Images

Deep learning for medical image classification has enjoyed increased attention. However, a bottleneck that prevents it from widespread adoption is its dependency on very large, annotated datasets, a condition that cannot always be satisfied. Few-shot learning in the medical domain is still in its infancy but has the potential to overcome these challenges. Compression is a way for models to be deployed on resource-constrained machines. In an attempt to tackle some of the challenges imposed by limited data and high computational resources, we present a few-shot sparse-quantization aware meta-training framework (FS-SQAM). The proposed framework aims to exploit the role of sparsity and quantization for improved adaptability in a low-resource cross-domain setting for the classification of acute lymphocytic leukemia (ALL) in blood cell images. Combining these strategies enables us to approach two of the most common problems that encounter deep learning for medical images: the need for extremely large datasets and high computational resources. Extensive experiments have been conducted to evaluate the performance of the proposed framework on the ALL-IDB2 dataset in a cross-domain few-shot setting. Performance gains in terms of accuracy and compression have been demonstrated, thus serving to realize the suitability of meta-learning on resource-constrained devices. Future advancements in the domain of efficient deep learning computer-aided diagnosis systems will facilitate their adoption in clinical medicine.
Visualizing CNN: An Experimental Comparative Study

To intuitively and accurately understand the decision mechanism of Convolutional Neural Networks(CNN), CNN visualization, as an essential part of explainable deep learning, has gradually become a hot topic in artificial intelligence. There have been many achievements in CNN visualization research, such as Gradients, Deconvolution, Class Activation Maps(CAM), etc. But there has been no systematic comparative study on CNN visualization algorithms. The choice of visualization algorithm is critical for accurately explaining the decision process of CNNs. Therefore, an experimental evaluation research on representative CNN visualization algorithms is conducted in this paper for ResNet50 and VGG16 on Caltech101, ImageNet, and VOC2007. The visualization performance is assessed in four aspects: causality, anti-disturbance capability, usability, and computational complexity, and suggestions for selecting CNN visualization algorithms are proposed based on the experimental results.
Multi-discriminative Parts Mining for Fine-Grained Visual Classification

Fine-Grained Visual Classification (FGVC) aims to differentiate visually similar but subtly different subordinate categories of the same basic category. However, current methods primarily exploit deep-layer features to locate to the strong salient part of the network. This paper finds that some subtle but discernible parts and the rich details in shallow-layer features are also valuable for classification. Consequently, this paper proposes a fine-grained visual classification framework that integrates multiple discriminative parts and multi-layer features. Our framework consists of two modules: 1) The attention map based locate-mine module locates the most discriminate part and masks it, thereby encouraging the network to mine other discriminative parts. 2) The multi-layer feature fusion module combines shallow-layer and deep-layer features to enrich local details in discriminative features. We also introduce an adaptive label loss to distinguish categories with high similarity. Experimental results show that our approach achieves excellent performance on three widely used fine-grained benchmark datasets.
The Recommendation Algorithm Based on Improved Conditional Variational Autoencoder and Constrained Probabilistic Matrix Factorization

An improved recommendation algorithm based on Conditional Variational Autoencoder (CVAE) and Constrained Probabilistic Matrix Factorization (CPMF) is proposed to address the issues of poor recommendation performance in traditional user-based collaborative filtering algorithms caused by data sparsity and suboptimal feature extraction. Firstly, in the data preprocessing stage, a hidden layer is added to CVAE, and random noise is introduced into the hidden layer to constrain the data features, thereby obtaining more accurate latent features and improving the model’s robustness and generative capability. Secondly, the category of items is incorporated as auxiliary information in CVAE to supervise the encoding and decoding of item data. By learning the distribution characteristics of the data, missing values in the rating data can be effectively reconstructed, thereby reducing the sparsity of the rating matrix. Subsequently, the reconstructed data is processed using CPMF, which optimizes the feature extraction performance by imposing constraints on user features. Finally, the prediction rating of a user for an item can be obtained through the matrix product of user and item feature matrices. Experimental results on the MovieLens-100K and MovieLens-1M datasets demonstrate the effectiveness and superiority of the proposed algorithm over four comparative algorithms, as it exhibits significant advantages in terms of root mean square error and mean absolute error metrics.
MAREPVGG: Multi-attention RepPVGG to Facefake Detection

The threat posed by the increasing means of face forgery and the lowering of the threshold of use is increasing. Although the detection capability of current detection models is improving, most of them need to consume large computational resources and have complex model architectures. Therefore, in this paper, we propose a new deep learning detection framework MARepVGG, which uses RepVGG as the backbone, combines texture enhancement module and multi-attention module to strengthen the network to learn face forgery features through the idea of heavy parameterization to balance training performance and inference speed. We evaluate our method on the kaggle real and fake face detection dataset, which differs from the computer automatically generated images, where the fake faces are high quality images produced by Photoshop experts. Our method improves the accuracy by 14% on this dataset compared to a baseline of forgery detection by repvgg alone, while the number of parameters is only 8.75 M.
Frequency Mixup Manipulation Based Unsupervised Domain Adaptation for Brain Disease Identification

Unsupervised Domain Adaptation (UDA), which transfers the learned knowledge from a labeled source domain to an unlabeled target domain, has been widely utilized in various medical image analysis approaches. Recent advances in UDA have shown that manipulating the frequency domain between source and target distributions can significantly alleviate the domain shift problem. However, a potential drawback of these methods is the loss of semantic information in the low-frequency spectrum, which can make it difficult to consider semantic information across the entire frequency spectrum. To deal with this problem, we propose a frequency mixup manipulation that utilizes the overall semantic information of the frequency spectrum in brain disease identification. In the first step, we perform self-adversarial disentangling based on frequency manipulation to pretrain the model for intensity-invariant feature extraction. Then, we effectively align the distributions of both the source and target domains by using mixed-frequency domains. In the extensive experiments on ADNI and AIBL datasets, our proposed method achieved outstanding performance over other UDA-based approaches in medical image classification. Code is available at: https://github.com/ku-milab/FMM .
Geometric Distortion Correction for Extreme Scene Distortion of UAV-Acquired Images

Geometric distortion, especially in extreme cases of scene distortion, is a prevalent issue in images taken by Unmanned Aerial Vehicles (UAVs). This research focuses on tackling this issue using a proposed approach that combines cutting-edge computer vision and photogrammetry techniques. It combines precise feature matching and image-warping algorithms, resulting in a more straightforward approach. By utilizing a Graph Neural Networks-based strategy and fine-tuning hyperparameters to strengthen the algorithm, the feature detection technique is further improved. To rectify spatial deformation within the scene, the homography estimation, which merges the Direct Linear Transform (DLT) approach with Singular Value Decomposition (SVD), is employed. Additionally, at the image registration phase, outliers are rejected using the Random Sample Consensus (RANSAC) technique. The image quality of the rectified images was evaluated using both objective and subjective image quality parameters. Overall, this research produces visually appealing image output in both extreme scene distortion correction and pair-wise image stitching applications, achieving favorable assessment findings based on SSIM, PSNR, and PIQE image quality metrics.
A New Transformer-Based Approach for Text Detection in Shaky and Non-shaky Day-Night Video

Text detection in shaky and non-shaky videos is challenging because of variations caused by day and night videos. In addition, moving objects, vehicles, and humans in the video make the text detection problems more challenging in contrast to text detection in normal natural scene images. Motivated by the capacity of the transformer, we propose a new transformer-based approach for detecting text in both shaky and non-shaky day-night videos. To reduce the effect of object movement, poor quality, and other challenges mentioned above, the proposed work explores temporal frames for obtaining activation frames based on similarity and dissimilarity measures. For estimating similarity and dissimilarity, our method extracts luminance, contrast, and structural features. The activation frames are fed to the transformer which comprises an encoder, decoder, and feed-forward network for text detection in shaky and non-shaky day-night video. Since it is the first work, we create our own dataset for experimentation. To show the effectiveness of the proposed method, experiments are conducted on a standard dataset called the ICDAR-2015 video dataset. The results on our dataset and standard dataset show that the proposed model is superior to state-of-the-art methods in terms of recall, precision, and F-measure.
T5G2P: Multilingual Grapheme-to-Phoneme Conversion with Text-to-Text Transfer Transformer

In recent years, the Text-to-Text Transfer Transformer (T5) neural network has proved more powerful for many text-related tasks, including the grapheme-to-phoneme conversion (G2P). The paper describes the training process of T5-base models for several languages. It shows the advantages of training G2P models using that language-specific basis over the G2P models fine-tuned from the multilingual base model. The paper also explains the reasons for training G2P models on whole sentences (not a dictionary) and evaluates the trained G2P models on unseen sentences and words.
Transformer-Based Encoder-Encoder Architecture for Spoken Term Detection

The paper presents a method for spoken term detection based on the Transformer architecture. We propose the encoder-encoder architecture employing two BERT-like encoders with additional modifications, including attention masking, convolutional and upsampling layers. The encoders project a recognized hypothesis and a searched term into a shared embedding space, where the score of the putative hit is computed using the calibrated dot product. In the experiments, we used the Wav2Vec 2.0 speech recognizer. The proposed system outperformed a baseline method based on deep LSTMs on the English and Czech STD datasets based on USC Shoah Foundation Visual History Archive (MALACH).
Object Detection Algorithm Based on Bimodal Feature Alignment

A dual-modal feature alignment based object detection algorithm is proposed for the full fusion of visible and infrared image features. First, we propose a two stream detection model. The algorithm supports simultaneous input of visible and infrared image pairs. Secondly, a gated fusion network is designed, consisting of a dual-modal feature alignment module and a feature fusion module. Medium-term fusion is used, which will be used as the middle layer of the dual-stream backbone network. In particular, the dual-mode feature alignment module extracts detailed information of the dual-mode aligned features by computing a multi-scale dual-mode aligned feature vector. The feature fusion module recalibrates the bimodal fused features and then multiplies them with the bimodal aligned features to achieve cross-modal fusion with joint enhancement of the lower and higher level features. We validate the performance of the proposed algorithm using both the publicly available KAIST pedestrian dataset and a self-built GIR dataset. On the KAIST dataset, the algorithm achieves an accuracy of 77.1%, which is 17.3% and 5.6% better than the accuracy of the benchmark algorithm YOLOv5-s for detecting visible and infrared images alone; on the self-built GIR dataset, the detection accuracy is 91%, which is 1.2% and 14.2% better than the benchmark algorithm for detecting visible and infrared images alone respectively. And the speed meets the real time requirements.
Portrait Matting via Semantic and Detail Guidance

Portrait matting is a challenging computer vision task that aims to estimate the per-pixel opacity of the foreground human regions. To produce high-quality alpha mattes, the majority of available methods employ a user-supplied trimap as an auxiliary input. However, obtaining trimap is difficult, which restricts the application of these methods. Recently, trimap-free approaches have been developed, but the alpha quality is significantly lower than that of trimap-based methods due to the lack of guidance. In this paper, we propose a portrait matting framework via semantic and detail guidance. We use the mask obtained from portrait segmentation as semantic guidance to improve semantic understanding of portrait matting, and we present a detail encoder block to capture the low-level feature around the subject as detail guidance, which helps to refine alpha details, such as hair. The proposed approach achieves remarkable results on both composition and real-world benchmarks.
Fine-Grained Face Sketch-Photo Synthesis with Text-Guided Diffusion Models

Face sketch-photo synthesis involves generating face photos from input face sketches. However, existing Generative Adversarial Networks (GANs)-based methods struggle to produce high-quality images due to artifacts and lack of detail caused by training difficulties. Additionally, prior approaches exhibit fixed and monotonous image styles, limiting practical usability. Drawing inspiration from recent successes in Diffusion Probability Models (DPMs) for image generation, we present a novel DPMs-based framework. This framework produces detailed face photos from input sketches while allowing control over facial attributes using textual descriptions. Our framework employs a U-Net, a semantic sketch encoder for extracting information from input sketches, and a text encoder to convert textual descriptions into text features. Furthermore, we incorporate a cross-attention mechanism within the U-Net to integrate text features. Experimental results demonstrate the effectiveness of our model, showcasing its ability to generate high-fidelity face photos while surpassing alternative methods in qualitative and quantitative evaluations.
Anisotropic Operator Based on Adaptable Metric-Convolution Stage-Depth Filtering Applied to Depth Completion

Nowadays, depth maps are a crucial source of information for many applications based on artificial vision. Applications such as video games, 3D cinema, or unmanned autonomous vehicle control strongly depend on data extracted from depth maps. Depth maps can be estimated by algorithms or acquired by sensors (such as Kinect sensor, Time-of-Flight camera, or LiDAR sensor). Acquired depth data frequently contains holes or data with low confidence levels. An interpolation model is necessary to solve the problem of lack of data or complete these holes in depth maps. We constructed a manifold given the image domain and a metric whose parameters are learned from the data. The primary approach is to provide the parameterized metric enough flexibility to estimate the manifold’s shape correctly. Additionally, the proposal embedded depth estimation in a pipeline considering convolution stages and the anisotropic metric. We estimated the parameters’ proposal using the PSO algorithm. We assessed our proposal using the publicly available KITTI Depth Completion Suite dataset. Obtained results show that this proposal outperforms our previous implementation and other contemporary models. Additionally, we performed an ablation study of the model showing that the critical component of the model is the first convolution stage, meaning this stage that enforces the edges of the color image is a crucial component of the model.
Three-Dimensional Object Detection Network Based on Coordinate Attention and Overlapping Region Penalty Mechanisms

Three-dimensional target detection is a key technology in the fields of autonomous driving and robot control for applications such as self-driving cars and unmanned aircraft systems. In order to achieve high detection accuracy, this paper proposes a 3D target detection network with a coordinate attention training mechanism that generates voting feature points for better detection ability and an overlap region penalty mechanism that reduces false detection. In comparative experiments on public large-scale 3D datasets including the Scannet dataset and SUN-RGB-D dataset, the proposed method obtained an average detection accuracy mAP of 60.1% and 58.0% with an intersection ratio of 0.25, which demonstrates its superior effectiveness over the current main algorithms such as F-PointNet, VoxelNet and MV3D. The improved method is expected to achieve higher accuracy for 3D object detection relying only on point cloud information.
Hybrid Spatio-Temporal Network for Face Forgery Detection

Facial manipulation techniques have aroused increasing security concerns, leading to various methods to detect forgery videos. However, existing methods suffer from a significant performance gap compared to image manipulation methods, partially because the spatio-temporal information is not well explored. To address the issue, we introduce a Hybrid Spatio-Temporal Network (HSTNet) to integrate spatial and temporal information in the same framework. Specifically, our HSTNet utilizes a hybrid architecture, which consists of a 3D CNN branch and a transformer branch, to jointly learn short- and long-range relations in the spatio-temporal dimension. Due to the feature misalignment between the two branches, we design a Feature Alignment Block (FAB) to recalibrate and efficiently fuse heterogeneous features. Moreover, HSTNet introduces a Vector Selection Block (VSB) to combine the outputs of the two branches and fire important features for classification. Extensive experiments show that HSTNet obtains the best overall performance over state-of-the-art methods.
Voice-Interactive Learning Dialogue on a Low-Cost Device

Traditional offline learning approaches are reaching their limits in meeting the dynamic demands of specialized applications, such as real-time human-robot interaction. While high benchmark scores attained through offline fine-tuning large models on extensive data, offer a glimpse of their potential, the true functionality is validated when these models are deployed on target devices and utilized in real-life scenarios. This paper presents a method incorporating humans in an interactive learning loop, using their real-time feedback for online neural network retraining. By leveraging the power of transfer learning, we can proficiently adapt the model to suit the specific requirements of the target application through natural voice-based dialogue. The approach is evaluated on the image classification task utilizing a unique low-cost device and a practical example of the real-time dialogue is presented to demonstrate the functionality.
Replaying Styles for Continual Semantic Segmentation Across Domains

In the context of Domain Incremental Learning for Semantic Segmentation, catastrophic forgetting is a significant issue when a model learns new geographical domains. While replay-based approaches have been commonly used to mitigate this problem by allowing the model to review past knowledge, they require additional storage space for old data, which may not be feasible in real-world applications. To address this limitation, we propose a style replay method that leverages the characteristics of low-level representations in CNN to require only one style feature for each domain, leading to a significant reduction in storage overhead. By fusing the style features of past domains with the semantic features of current data, our method enables style transfer for new domain data, thereby improving the model’s generalization ability to the domain. Through extensive experimental evaluations on various autonomous driving datasets, we demonstrate the efficacy of our proposed method in addressing the challenges of continual semantic segmentation under both label and domain shift, outperforming the previous state-of-the-art methods.