Reviewerly/exHarmony huggingface dataset:
Research on an Effective Human Action Recognition Model Based on 3D CNN

Most of the human action recognition systems based on 3-Dimensional Convolutional Neural Network (3D CNN) architecture recognize human actions frame by frame in video streams, which need to be deployed on high-performance platforms such as cloud servers. Through the targeted optimization of the processing method of each frame of the video in the process of human action recognition, the computing power requirements and the total processing time of human action recognition are reduced. The optimization of human action recognition is tested and verified by the Kinetics-700 dataset, and the accuracy of action recognition is similar to that before optimization, and the total recognition time is only 14.1 % of the total time before optimization. It effectively reduces the performance requirements of the deployment platform, improves the real-time performance of action recognition, and increases the practicability of human action recognition based on deep learning in the application of low computing power platforms.
Dense view interpolation of 4D light fields for real-time augmented reality applications

Dense multi-view 3D displays appropriately combined with camera arrays may provide consistently augmented reality, where we enjoy a simple solution of occlusion caused by visual obstructions in the real world as if they were transparent for many users at the same time. In this paper, toward such multi-view systems electrically transmitting light fields beyond visual obstructions, we propose a novel method for real-time dense interpolation of entire 4D light fields acquired by sparse viewpoints. First, we discuss appearances of our proposed multiview system, while examining how dense 4D light fields should be. Then, we propose a novel method of real-time 4D light field interpolation to develop an inexpensive system by reducing the number of image sensors to acquire 4D light fields. By introducing simple depth estimation from equally-spaced multiview images without severe increase of computational cost, the novel method significantly improves reconstruction quality of the previously proposed one, that can only work for almost constant depths. Finally, experimental results show that our practical implementation on a GPGPU achieves real-time interpolation of entire 4D light fields while obtaining sufficient quality of reconstructed images. It is even comparable to that of various conventional methods for view interpolation, which depend on high computational cost not suitable for our real-time applications.
Image Segmentation for Colorectal cancer histopathological images analysis

Colorectal cancer (CRC) is the third most common malignancy and the second most deadly cancer. The most efficient way to determine CRC staging is to analyze whole slide digital pathology images; therefore, it is certainly important to ensure the accuracy of pathology slide analysis.We can obtain medical quantized data of pathological images by implementing deep learning methods. These methods not only can light pathologists’ load but also can provide accurate computing results.In this paper, we use U-2-NET as our backbone to perform Binary Image Segmentation on CRC pathology slides. CRC pathology slides have a variety of non-conforming shapes and colors which is an enormous challenge for detecting cancer areas. U-2-NET was originally used in the Salient Object Detection (SOD) task to find the most unique regions of human attention, which can be used to identify abnormal regions in pathological slices. Moreover, the RSU block of U-2-NET can handle long-term and short-term dependencies, which we believe helps maintain contextual information. With the large computational costs, U-2-NET is hard to implement for application. Our purposed method can use preprocessing, image-selecting mechanisms and transfer learning concepts to solve this problem.Our results show that the model trained with a small part of the data set and a modified small object function has the best results for Binary Image Segmentation of colorectal cancer pathology sections by U-2-NET, with the best IOU (0.77) and Dice Loss (0.83) compared with other models (MSRFCNN, FCN, SegNet, and Unet). Furthermore, after transferring learning using pre-trained weights from the SOD dataset, the results are improved compared to those of learning the network from scratch.
Edge-Guided Video Super-Resolution Network

In this paper, we propose an edge-guided video super-resolution (EGVSR) network that utilizes the edge information of the image to effectively recover high-frequency details for high-resolution frames. The reconstruction process consists of two stages. In the first stage, the Coarse Frame Reconstruction Network (CFRN) generates coarse SR frames. In addition, we propose the Edge-Prediction Network (EPN) to capture the edge details that help to supplement the missing high-frequency information. Unlike some prior SR works that tend to increase the depth of networks or use attention mechanisms to reconstruct large-size objects but ignore small-size objects, we propose the Attention Fusion Residual Block (AFRB) to process objects of different sizes. The AFRB, an enhanced version of the conventional residual block, performs fusion through a multi-scale channel attention mechanism and serves as the basic operation unit in the CFRN and the EPN. Then, in the second stage, we propose the Frame Refinement Network (FRN), which contains multiple convolution layers. Through the FRN, we fuse and refine the coarse SR frames and edge information learned from the first stage. Compared with the state-of-the-art methods, our SR model improves approximately 0.5% in PSNR and 1.8% in SSIM evaluation on the benchmark VID4 dataset when the number of parameters is reduced by 54%.
Small Object Detection for Intelligent Abnormal Event Detection in Power Distribution Networks

With the development of intelligent object detection technology, the traditional artificial monitoring system has been gradually replaced by the automatic monitoring system based on intelligent video analysis. In the daily operation of distribution network monitoring system, the monitoring of abnormal events associated with patrol inspectors is an important part of video analysis. Safety helmet compliance and the smoking behavior are key inspection points in these abnormal events in intelligent power distribution network monitor scenario. Due to small size of helmet and cigarette, it is hard to detect these small objects under standard object detection framework. To solve this problem, we designed a two-stage neural network framework to detect small objects attached on human body. At the first stage, we extract human body part from original image. Then, we use standard object detection framework to detect helmet and cigarette. With increased proportion of small object in the image, the proposed framework can detect small objects accurately and effectively. Experiment results and field test both demonstrated its effectiveness of the proposed algorithm.
MA<sup>2</sup>-FPN for Tiny Object Detection from Remote Sensing Images

Tiny object detection has been a challenging topic in computer vision recent years. Moreover, in remote sensing field, smaller and clustered tiny objects make its detection more difficult compared to ground-based images. This makes general detectors fail to achieve good performance when facing tiny objects in remote sensing images. In this paper, we propose a Mask Augmented Attention Feature Pyramid Network(MA <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> -FPN) to detect tiny objects in remote sensing images, which consists of two modules, Attention Enhancement Module(AEM) and Mask Supervision Module(MSM). Specifically, AEM aggregates tiny target context and spatial feature information by large kernel separable convolutional attention mechanism, and MSM supervises AEM through a segmentation attention loss to aggregate attention information more accurately while suppressing the influence of irrelevant background. Experiments based on the AI-TOD benchmark show that our MA <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> -FPN achieves state-of-the-art(SOTA) level.
Underwater Image Enhancement Using Realistic Dataset With Turbidity and Color Distortion

Underwater images suffer from the quality degradation caused by the light scattering and absorption in the water. Recent deep-learning based underwater image enhancement methods achieve limited performance due to the lack of proper training datasets. In this paper, we provide a realistic underwater image dataset reflecting the diverse characteristics of turbidity and color distortion in underwater environments. We also design an underwater image enhancement network to exploit the local and global features of the turbid underwater images with color distortion. The experimental results show that the proposed dataset is more effective than existing datasets to train deep neural networks, and the proposed underwater image enhancement network successfully removes the haze and color distortion from real underwater images while outperforming the existing methods.
Adversarial Speaker-Consistency Learning Using Untranscribed Speech Data for Zero-Shot Multi-Speaker Text-to-Speech

Several recently proposed text-to-speech (TTS) mod-els achieved to generate the speech samples with the human-level quality in the single-speaker and multi-speaker TTS scenarios with a set of pre-defined speakers. However, synthesizing a new speaker's voice with a single reference audio, commonly known as zero-shot multi-speaker text-to-speech (ZSM- TTS), is still a very challenging task. The main challenge of ZSM - TTS is the speaker domain shift problem upon the speech generation of a new speaker. To mitigate this problem, we propose adversarial speaker-consistency learning (ASCL). The proposed method first generates an additional speech of a query speaker using the external untranscribed datasets at each training iteration. Then, the model learns to consistently generate the speech sample of the same speaker as the corresponding speaker embedding vector by employing an adversarial learning scheme. The experimental results show that the proposed method is effective compared to the baseline in terms of the quality and speaker similarity in ZSM-TTS.
Effective ASR Error Correction Leveraging Phonetic, Semantic Information and N-best hypotheses

Automatic speech recognition (ASR) has recently achieved remarkable success and reached human parity, thanks to the synergistic breakthroughs in neural model architectures and training algorithms. However, the performance of ASR in many real-world use cases is still far from perfect. There has been a surge of research interest in designing and developing feasible post-processing modules to improve recognition performance by refining ASR output sentences, which fall roughly into two categories. The first category of methods is ASR N-best hypothesis reranking. ASR N-best hypothesis reranking aims to find the oracle hypothesis with the lowest word error rate from a given N-best hypothesis list. The other category of methods take inspiration from, for example, Chinese spelling correction (CSC) or English spelling correction (ESC), seeking to detect and correct text-level errors of ASR output sentences. In this paper, we attempt to integrate the above two methods into the ASR error correction (AEC) module and explore the impact of different kinds of features on AEC. Empirical experiments on the widely-used AISHELL-l dataset show that our proposed method can significantly reduce the word error rate (WER) of the baseline ASR transcripts in relation to some top-of-line AEC methods, thereby demonstrating its effectiveness and practical feasibility.
Studying Human-Based Speaker Diarization and Comparing to State-of-the-Art Systems

Human-based speaker diarization experiments were carried out on a five-minute extract of a typical AMI corpus meeting to see how much variance there is in human reviews based on hearing only and to compare with state-of-the-art diarization systems on the same extract. There are three distinct experiments: (a) one with no prior information; (b) one with the ground truth speech activity detection (GT-SAD); and (c) one with the blank ground truth labels (GT-labels). The results show that most human reviews tend to be quite similar, albeit with some outliers, but the choice of GT-labels can make a dramatic difference to scored performance. Using the GT-SAD provides a big advantage and improves human review scores substantially, though small differences in the GT-SAD used can have a dramatic effect on results. The use of forgiveness collars is shown to be unhelpful. The results show that state-of-the-art systems can outperform the best human reviews when no prior information is provided. However, the best human reviews still outperform state-of-the-art systems when starting from the GT-SAD.
Restoration of High-Frequency Components in Under Display Camera Images

Under-Display Camera (UDC) systems have been developed to remove noticeable camera holes or notches and entirely cover the front side with the screen. As the name implies, UDCs are placed under the display, generally organic light-emitting diode (OLED) these days. Since the OLED panel is not transparent and consists of circuits and display devices, the light reaching the camera experiences a loss of photons and a complicated point spread function (PSF). As a result, the obtained images through the UDC system usually experi-ence a color shift, decreased intensity, complex artifacts due to the PSF, and loss/distortion in high-frequency details. To overcome these degradations, we exploit the multi-stage image restoration network and frequency loss function. The network utilizes deformable convolutions to solve the spatially-variant degradations in UDC images based on the fact that the kernel of deformable convolutions is dynamic and adaptive to input. We also apply frequency reconstruction loss when training our models to better restore the lost high-frequency components due to the complicated PSF. We show that our method effectively removes the degradation caused by the UDC system and achieves state-of-the-art performance on a benchmark dataset.
F<sub>0</sub> Modification via PV-TSM Algorithm for Speaker Anonymization Across Gender

Speaker anonymization has been developed to protect personally identifiable information w hile r etaining other encapsulated information in speech. Datasets, metrics, and protocols for evaluating speaker anonymization have been defined in the Voice Privacy Challenge (VPC). However, existing privacy metrics focus on evaluating general speaker individuality anonymization, which is represented by an x-vector. This study aims to investigate the effect of anonymization on the perception of gender. Understanding how anonymization caused gender transformation is essential for various applications of speaker anonymization. We proposed speaker anonymization methods across genders based on phase-vocoder time-scale modification (PV-TSM). Subsequently, in addition to the VPC evaluation, we developed a gender classifier toe valuate a speaker's gender anonymization. The objective evaluation results showed that our proposed method can successfully anonymize gender. In addition, our proposed methods outperformed the signal processing-based baseline methods in anonymizing speaker individuality represented by the x-vector in ASVeval while maintaining speech intelligibility.
LP3DAM: Lightweight Parallel 3D Attention Module for Violence Detection

Recent studies have shown that the attention mechanism added to the deep convolutional neural network can effectively improve the network performance, but the attention mechanism applied to the field of violence detection has not been developed. The main reason is that violence detection uses 3D convolution network. At present, most attention modules are only suitable for 2D convolution, and these modules are designed as more complex modules to obtain better network performance, which inevitably increases the complexity of the network model. In order to overcome the trade-off between network performance and complexity, and explore the effectiveness and feasibility of attention mechanism in 3D convolutional network model, this paper proposes Lightweight Parallel 3D Attention Module (LP3DAM), which greatly improves the accuracy of the model by adding a small amount of parameters. Experiments show that LP3DAM has a positive effect on 3D lightweight convolutional networks, which makes the accuracy of the network (MiNet-3D) on the three datasets of Hockey, Crowd and RWF-2000 increase by 1.44%, 4.84% and 0.71%, respectively. The number of parameters added to the original network is controlled within 1K, and the increase of Flops is controlled at about 0.26M.
FAD-Net: Fake Images Detection and Generalization Based on Frequency Domain Transformation

With the continuous development of neural network technology, the generation methods of fake images are gradually improved. More and more faked photos and face changing videos appear on major social media platforms, causing people to pay attention to their reputation security, information security, and public opinion guidance. At present, the spatial detection model has excellent results. But most of them need a large number of training sets as support. Simultaneously, the frequency detection model primarily uses complex feature extraction operations, and most of the two detection models only detect a single fake image generation method. Given the above two points, this paper makes the following work: Based on the common issues and attention mechanism of fake image generation methods on the network, FAD-Net (Frequency-domain Attention Detection Network) is designed, which is suitable for most fake image generation methods. We use the frequency domain image as the network input to train the detector. Good detection results are obtained on 11 fake image generation methods such as Deepfakes and Gan series. Compared with the best spatial detection model, FAD-Net uses a smaller training set and shorter training time to get better detection generalization, which shows the superiority of frequency information in fake image detection generalization.
Research on Carotid Ultrasonic Image Detection Based on Convolutional Neural Network

Carotid ultrasound is a main and convenient method for diagnosing plaque, Therefore, accurately obtaining plaques information from ultrasound images is essential for further clinical diagnosis. Due to the interference of noise and the differences in artificial technical operations, the missed inspection of the plaques is likely to cause missed inspection. Therefore, a comparative experiment based on the constructing algorithm based on convolutional neural networks is performed to achieve more accurate detection and identification of cervical arterial tube cavity and plaques. First of all, we constructed the carotid artery data set, and then conducted the classification test of the carotid lumen and plaques through the YOLOv5 network based on migration and learning, and used the Faster R-CNN and SSD network for comparison experiments. Experiments show that the average accuracy obtained by YOLOv5 network reaches 0.928 when the IOU value is 0.5, and 0.659 when the IOU value is 0.75, and the average recall rate reaches 0.673, which are higher than the Faster R-CNN and SSD networks; The experiment shows that the average precision of the comprehensive comparison is also better than the other two comparison networks. At the same time, the model calculation speed meets the real-time needs. Therefore, the YOLOv5 network can improve the correctness and practical significance of the detection of the lumen and plaques in terms of carotid image detection.
Optimal Deep Multi-Route Self-Attention for Single Image Super-Resolution

Image restoration, such as single image super-resolution (SISR), is a long-established low-level vision issue that intends to regenerate high-resolution (HR) images from low-resolution (LR) input counterparts. While state-of-the-art image super-resolution models are based on the well-known convolutional neural network (CNN), many self-attention-based or transformer-based experiment attempts have been conducted and have shown promising performance on vision problems. A powerful baseline model based on the swin transformer adopts the shifted window approach. It enhances the capability by restricting the model to compute the self-attention function only on non-superimpose local windows while enabling cross-window relations. However, the architecture design is manually fixed. Therefore, the results are not achieving optimal performance. This paper presents an optimal deep multi-route self-attention network for single image super-resolution (ODMR-SASR). The genetic algorithm (GA) is introduced to discover the optimal number of filters and layers. Experimental results demonstrate that the proposed optimization technique can produce a progressive SR image quality.
DTUnet: A Transformer-based UNet Combined with DenseASPP Module for Pancreas Segmentation

Accurate pancreas segmentation is of great significance for the diagnosis and treatment of pancreatic cancer. Exploiting FCN, UNet or their variants, these CNN-based methods, to complete this task has become the de-facto standard and great success has been achieved. However, convolutional operation fails to build long-range dependency which is crucial for segmentation, hindering the further development of CNN-based methods. To address the difficulty, we propose the DTUnet network, which introduces the DenseASPP module and Transformer on the basis of UNet and stacks the two in a sequential manner. Transformer connects each pixel of the input feature maps to generate a global receptive field, thus capturing the global context information and realizing the construction of long-range dependency. Meanwhile, to alleviate the training challenges of Transformer's data-hungry, DTUnet employs DenseASPP module to generate rich and multi-scale high-level semantic feature maps as the input of Transformer, ensuring that Transformer can fully leverage global modeling capability even when applied to a small size pancreas segmentation dataset. Benefiting from the combination of the two, the proposed DTUnet generates more efficient and reliable global context information, and ultimately achieves an average Dice coefficient score of 84.77% ±4.65 on the public NIH pancreas segmentation dataset, which is 1.87% higher than UNet. The result is also higher than advanced methods in recent years, indicating that DTU net has the potential to assist doctors to segment the pancreas in clinical application.
DNN-Rule Hybrid Dyna-Q for Sample-Efficient Task-Oriented Dialog Policy Learning

Reinforcement learning (RL) is a powerful strategy for making a flexible task-oriented dialog agent, but it is weak in learning speed. Deep Dyna-Q augments the agent's experience to improve the learning efficiency by internally simulating the user's behavior. It uses a deep neural network (DNN) based learnable user model to predict user action, reward, and dialog termination from the dialog state and the agent's action. However, it still needs many agent-user interactions to train the user model. We propose a DNN-Rule hybrid user model for Dyna-Q, where the DNN only simulates the user action. Instead, a rule-based function infers the reward and the dialog termination. We also investigate the training with rollout to further enhance the learning efficiency. Experiments on a movie-ticket booking task demonstrate that our approach significantly improves learning efficiency.