Reviewerly/exHarmony huggingface dataset:
A Two-stage Cascading Method Based on Finetuning in Semi-supervised Domain Adaptation Semantic Segmentation

The traditional unsupervised domain adaptation (UDA) has achieved great success in many computer vision tasks, especially semantic segmentation, which requires high cost of pixel-wise annotations. However, the final performance of UDA method is still far behind that of supervised learning due to the lack of annotations. Researchers introduce the semi-supervised learning (SSL) and propose a more practical setting, semi-supervised domain adaptation (SSDA), that is, having labeled source domain data and a small number of labeled target domain data. To address the inter-domain gap, current researches focus on domain alignment by mixing annotated data from two domains, but we argue that adapting the target domain data distribution through model transfer is a better solution. In this paper, we propose a two-stage SSDA framework based on this assumption. Firstly, we adapt the model from the source domain to the labeled dataset in the target domain. To verify the assumption, we choose a basic transfer mode: finetuning. Then, to align the labeled subspace and the unlabeled subspace of the target domain, we choose teacher-student model with class-level data augmentation as the basis to realize online self-training. We also provide a deformation to solve overfitting on the target domain with a small number of annotated data. Extensive experiments on two synthetic-to-real benchmarks have demonstrated the correctness of our idea and the effectiveness of our method. In most SSDA scenarios, our approach can achieve supervised performance or even better.
Facial Attribute Editing based on Independent Selective Transfer Unit and Self-attention Mechanism

Facial attribute editing aims to change the facial attributes, which can be regarded as an image translation problem. Facial attribute editing is usually realized by combining encoder-decoder and Generative Adversarial Networks, but the generated image is not realistic enough, and the model has weak ability to control the fine granularity of face attributes of generated images. In this work, we propose a Generative Adversarial Network ISTSA-GAN based on Independent Selective Transfer Unit (ISTU) and Self-attention Mechanism. On the basis of STGAN, we use ISTU instead of Selective Transfer Unit (STU) to combine with encoder-decoder to selectively transfer the features of encoder. In addition, a self-attention mechanism is introduced into the transposed convolution layer of the decoder to establish long-distance dependence of the model across image regions. Finally, attribute interpolation loss and source domain adversarial loss are added to constrain the training of the model. Experimental results show that this method can improve the ability of editing attributes and saving much details, and enhance the ability of fine-grained control of editing attributes. It is superior to classical methods in attribute editing accuracy and image quality.
Deep Oral Cancer Lesion Segmentation with Heterogeneous Features

About 650,000 new cases of oral cavity cancer occur every year in the world, and cause more than 330,000 deaths. If oral cancer is diagnosed at an early stage, the overall 5-year survival rate is over 70%, while it drops to less than 40% if detected at a late stage. Thus, early detection of oral cancer is important. Visual non-invasive examination is an efficient and feasible approach for performing a preliminary diagnosis of oral cancer. In this paper, we propose a fully convolutional network (FCN) based model to segment cancer and precancer lesion regions in the oral cavity. In addition to the RGB channels of the input image, we append features of Gabor filter and wavelet filter that show strong response at cancer and precancer regions. We also propose a refine stage before the decision layer of FCN to preventing weight dominating problem when reducing high dimension features to small number of classes. In the experiments on oral cancer dataset, the IOU, sensitivity, and specificity of the proposed network achieves 0.586, 0.883, 0.726 respectively. The experimental results show the effectiveness of our method.
DCAN: Deep Consecutive Attention Network for Video Super Resolution

Slow motion is visually attractive in video applications and gets more attention in video super-resolution (VSR). To generate the high-resolution (HR) center frame with its neighbor HR frames from the low-resolution (LR) of two frames. Two sub-tasks are required, including video super-resolution (VSR) and video frame interpolation (VFI). However, the interpolation approach does not successfully extract low-level features to achieve the acceptable result of space-time video super-resolution. Therefore, the restoration performance of existing systems is constrained due to rarely considering the spatial-temporal correlation and the long-term temporal context concurrently. To this extent, we propose a deep consecutive attention network-based method to generate attentive features to get HR slow-motion frames. A channel attention module and an attentive temporal feature module are designed to improve the perceptual quality of predicted interpolation feature frames. The experimental results show the proposed method outperforms 0.17 dB in an average PSNR compared to the state-of-the-art baseline method.
Speech Enhancement-assisted Voice Conversion in Noisy Environments

Numerous voice conversion (VC) techniques have been proposed for the conversion of voices among different speakers. Although good quality of the converted speech can be observed when VC is applied in a clean environment, the quality degrades drastically when the system is run in noisy conditions. In order to address this issue, we propose a novel speech enhancement (SE)-assisted VC system that utilizes the SE techniques for signal pre-processing, where the VC and SE components are optimized in an joint training strategy with the aim to provide high-quality converted speech signals. We adopt a popular model, StarGAN, as the VC component and thus call the combined system as EStarGAN. We test the proposed EStarGAN system using a Mandarin speech corpus. The experimental results first verified the effectiveness of joint training strategy used in EStarGAN. Moreover, EStarGAN demonstrated performance robustness in various unseen noisy environments. The subjective listening test results further showed that EStarGAN can improve the sound quality of speech signals converted from noise-corrupted source utterances.
3D point cloud target recognition based on the Bi-LSTM and PointCNN network

Real traffic is complex and changeable scenes. How to accurately recognize targets in road scenes has always been one of the challenging problems in environmental perception. Lidar has been widely used in vehicle environment sensing because it has the ability of high precision ranging. To alleviate the influence of the point cloud disorder in the target recognition, most of the existing methods directly use a pooling strategy to the high dimensional- feature set to obtain local neighborhood features from the point cloud, which makes it difficult to learn global structure information. How to obtain semantic features of the point cloud is still a challenging problem. Aiming at the problem that the relationship between the features is not fully utilized, a point cloud recognition method based on Bi-LSTM (bidirectional long-term and short-term memory network) and convolution neural network is proposed in this paper. This method maps point clouds to high-dimensional space orderly and then learns the long-period dependencies between feature sequences. The experimental results show that the proposed method can achieve remarkable improvement in the case of high benchmark accuracy on the KITTI dataset.
A Multi-task Framework of Speaker Recognition with TTS Data Augmentation

Deep learning usually requires a lot of data, but we often have difficulties in collecting enough training data in many fields. In some limited resource application scenarios, data augmentation often plays a key role. A common method is adding background noise to the speech or changing the speed of speech to increase the number of utterances in the training dataset. For the training of ASV model, we propose a method to augment the training dataset through synthesizing large amounts of speech by VAE-based speech synthesis model, and we mitigate the problem of anti-spoofing detection performance degradation caused by the introduction of synthesized speech through the multi-task framework. Experiments on AISHELL-l, AISHELL-3 and ASVspoof2019LA databases show that our proposed method can improve the robustness of the speaker recognition model while also improving the anti-spoofing ability of the model.
Geometric Constraints for Establishing Point Correspondence Between Stereo Images

Finding point correspondence between stereo images of a 3D point set is a basic open problem in computer vision. It is usually the prerequisite for 3D reconstruction and object recognition. However, since geometric depth information is lost in 3D to 2D projection, there is no scalar geometric invariant among multiple images. So, there is no method based on pure geometric information to establish point correspondence among multiple images previously. In the literature, the epipolar constraint is widely applied to restrict the correspondence search space. In this paper we derive novel geometric constraints for nine corresponding points between stereo images. Using these constraints, we can design purely geometric method for establishing point correspondence. These constraints can also be used in conjunction with other methods to optimize the correspondence search algorithm.
Large-Scale Blind Face Super-Resolution via Edge Guided Frequency Aware Generative Facial Prior Networks

Large-scale blind face super-resolution is a significant image processing task with high practical value and wide applications. The task is more challenging than conventional image super-resolution since the degradation is more complex and the amount of useful information is very small inside the image. To address this problem, we propose Edge-guided Frequency-aware Generative Prior Network (EFGPN), a super-resolution method based on edge guidance, frequency constraint, and generative priors. The proposed edge guidance could aid the network in retaining more structural information. Frequency constraints suppress artifacts while allowing the network to generate good features across all frequency bands. The use of generative priors enables the network to produce photorealistic faces. The proposed EFGPN can blindly super-resolve low-resolution faces to high-quality faces. Experimental results indicate that our proposed EFGPN outperforms the state-of-the-arts under subjective and objective evaluations.
Eyeglass Frame Segmentation for Face Image Processing

Many people commonly wear eyeglasses on their face, masking the area around the eyes. The lens part of the eyeglasses can often be visible through the back area, while the frame part of the eyeglasses completely hides the back area, resulting in degrading the performance of face image processing. By taking the eyeglass frame into account in face image processing, we can not only improve the accuracy of recognition and analysis, but also apply it to automatic quality assessment in standardized photos such as passport photos. In this paper, we propose an eyeglass frame segmentation method using the combination of U-Net and PSPNet. We also propose a novel data augmentation method to increase the number of face images with eyeglasses. Through a set of experiments using CelebAMask-HQ, we demonstrate that the proposed method exhibits the efficient performance in the segmentation of eyeglass frames.
Learning to Sharpen Partially Blurred Image via Iterative Blurred Region Mining and Recovery

Despite progress on image deblurring, existing advanced deep-net models are difficult to directly process partially blurred images due to the mismatch in the distribution of training (full-scaled blurred) and test (partially blurred) images. To address this problem, we presents an iterative blurred region mining and recovery (iBRMR) approach, which can automatically and progressively mine and sharpen the blurred regions. Starting with a collaborative mechanism of two coupled deep-net models, our approach first learns a blur detector to locally discover evidence of blurred regions. We name this the blurred region mining process. Then, we construct a blur refiner to improve sharpness of the most blurred region. We name this the blurred region recovery process. We iterate the mining and recovery processes to sharpen the image. Our iBRMR approach takes advantage of the two processes' complementary nature. It functions as a bridge between the blur detector and the blur refiner, yielding the performance synergy in better solving the deblurring of partially blurred image. Experimental results on benchmark datasets demonstrate that the proposed iBRMR approach not only achieves superior deblurring performance in partially blurred images, but also performs well in full-scaled blurred images. The video demos can be found at https://sites.google.com/view/ibrmr.
An object detection method based on improved YOLOX

YOLOX is a representative algorithm of one-stage target detection. Its main network structure is based on the residual module composed of convolution neural network. Different from CNN, transformer adopts multi head attention mechanism to represent global information efficiently, which performs well in the field of computer vision. Therefore, it is worth considering to improve the original YOLOX combined with transformer. In this paper, the CSP residual module in the original network is replaced by the encoding module of transformer to excavate the potential of self attention mechanism in feature expression. At the same time, a Pyramid Split Attention module is used to optimize the original FPN module in the neck part of the network. In addition, in order to achieve better performance, weighted boxes fusion is used to replace the non maximum suppression processing in the original algorithm. Experiments show that through these improvements, the algorithm has a good improvement in both public data set and Transmission line power equipment data set.
Keypoints Representation of Density-aware and the Spatial-Channel-wise Decoder for 3D Object Detection

Current 3D object detection frameworks based on LiDAR mainly used sparse convolution as the backbone network after voxelization to extract features, and applied grids to further refine the proposal boxes. While these operations may limit the accuracy improvement of 3D object detection, because they destroyed the geometric characteristics of point clouds to a large extent, including density and object shape. Therefore, in this paper, we proposed a method to refine the proposals by estimating density-aware information in the second stage. A certain number of key points were sampled in each proposal, and then applied the self-attention module to study the relations between these key points. Then the designed spatial-channel-wise decoder fused channel-wise and spatial-wise features to obtain the global representation of the object. Finally, the global representation was fed into the detect head to obtain a more accurate box. The performance of our proposed 3D detection model was evaluated on the KITTI dataset, and the average accuracy of car class on the test set and validation split was 80.62% and 85.39% respectively, and the average accuracy of three classes in KITTI on the validation split was 72.41%.
Detection Method from 4K Images Using SSD300 without Retraining

In this paper, we propose a method to improve the accuracy of object detection from 4K images using SSD (single shot multibox detector) without retraining. When SSD is used for object detection, the input image will be resized to the size of images used for training the network. Because of this resizing of image, sizes of objects in the image are also affected so that the detection accuracy may possibly decrease. In the proposed method, the image is resized to one of the predefined sizes which is larger than usually used in the applications of SSD. This modification enables the improvement of the detection accuracy when the high-resolution images are inputted. The advantage of the proposed method is that it does not require retraining of the network parameters. The effectiveness of the proposed method is demonstrated by computer simulations.
Dense Labeling of Human Activity Recognition Using a U-Net++ Network Based on Inertial Sensor Data

Human activity recognition (HAR) is used to monitor and identify human activities intelligently and shows a good perspective in many applications. Traditional HAR methods in previous works confront with the multi-class-window problem, so it is difficult to achieve a high time-resolution in activity recognition. Herein, we present a dense labeling method based on a U-Net++ network to overcome the above problem. Inertial sensor data is fed into the network directly to realize activity classification of each sampling point. The architecture of U-Net++ adopts dense skip pathway and deep supervision, which makes the network pruning flexible. We tested our method on the dataset collected in our laboratory and a public dataset, WISDM. The results indicated that our method had a better performance over compared methods. Moreover, network pruning experiments were carried out. The results showed that in the laboratory dataset, the accuracy of a pruned network with 0.04 M parameters was more than 99%, and in WISDM, the accuracy of a pruned network with 0.18 M parameters could reach 97.3%.
Acoustic Model Adaption Using x-vectors for Improved Automatic Speech Recognition

Traditionally, i-vectors have been used for acoustic model adaptation and have been shown to capture the speaker and channel information quite robustly. Although x-vectors are known to be more robust for speaker verification, they have not, yet, been used successfully for large-scale ASR adaptation. This paper investigates the use of x-vectors for large vocabulary hybrid acoustic model adaptation. We keep our study limited to one x-vector per utterance due to the high computational cost of x-vector extraction. With ASR experiments conducted on Librispeech and TED-LIUM datasets using x-vector extractor trained on Voxceleb, we demonstrate three important findings. First, we show that only x-vectors concatenated with acoustic features, are not sufficient for improving performance compared to i-vectors. Second, a Word Error Rate improvement of 5.4% and 4.2% is observed in Librispeech and TED-LIUM databases respectively, when both x-vectors and i-vectors are concatenated with acoustic features and used for adaptation. Third, in order to enable streaming ASR, we propose an enrollment-test mechanism where a target speaker is enrolled and the enrollment x-vector is appended with the online i-vectors during inference. We show that such concatenation of x-vectors and i-vectors for adaptation improves the performance of speech recognition systems in both offline and streaming mode.
Single Image Raindrop Removal Using a Non-Local Operator and Feature Maps in the Frequency Domain

Taking a photo on a rainy day may result in a photo with raindrops. Images containing raindrops have a significant impact on the visual impression and accuracy when applied to image recognition systems. Thus, an automatic high-quality raindrop removal method is desired for outdoor image processing systems as well as for acquiring good-looking images. Several existing methods have been proposed to tackle this problem, but they often fail to keep global consistency and generate unnatural patterns. In this paper, we tackle this prob-lem by introducing a non-local operator. The non-local operator combines features in distant locations with matrix multiplication and enables consistency in distant locations. In addition, high-frequency components such as edges are more affected in images with raindrops. Inspired by the nature that high-frequency components can be separated from other components in the frequency domain, we also propose to process feature maps in the frequency domain, which are obtained by the fast Fourier transform operation and processed by several convolution layers. Experimental results show that our method effectively removes raindrops and achieves state-of-the-art performance.
Fast Asymmetric Bilateral Motion Estimation for Video Frame Interpolation

We propose a fast asymmetric bilateral motion estimation (f-ABME) network for video frame interpolation, which is based on fast flow estimation networks. First, we estimate symmetric bilateral motion fields, assuming that motion trajectories between two input frames are linear. Second, we use the symmetric fields to generate an anchor frame. Third, we estimate asymmetric bilateral motion fields from the anchor frame to the input frames. Finally, we use both the symmetric and asymmetric fields to reconstruct the intermediate frame. Experimental results demonstrate that the proposed f-ABME algorithm achieves competitive VFI results, while demanding relatively low complexity.
Adaptive End-to-End Text-to-Speech Synthesis Based on Error Correction Feedback from Humans

We propose an end-to-end text-to-speech (TTS) method that can intuitively correct accent errors in synthetic speech by using feedback from humans. State-of-the-art end-to-end TTS methods can synthesize high-quality speech, but humans can hardly interpret the black-boxed TTS system represented as a stack of neural networks. This reduced interpretability prevents humans from controlling the TTS system to correct errors in synthetic speech intuitively. In this paper, we focus on a method to involve human listeners in the process of accent error correction for synthetic speech generated by end-to-end TTS. Specifically, we build an end-to-end TTS model equipping a prosody predictor that estimates the change of pitch for each syllable from phoneme embeddings and context-aware word embedding. Then, we perform a human-in-the-loop (HITL) framework to correct errors of the prosody predictor using collective intelligence of human listeners. The results of Japanese TTS experiments show that our HITL framework can successfully correct accent errors and contribute to the quality of synthetic speech comparable to the conventional method requiring an accent dictionary for text analysis.
Multi-Modal Magnetic Resonance Images Segmentation Based on An Improved 3DUNet

The 3D U-Net model employs end-to-end training ways and does not demand pre-training process, but the limited acceptance range of convolutional kernel makes it difficult to establish an explicit long-range dependency, resulting in poor segmentation accuracy in magnetic resonance (MR) image. This paper presents an promoted 3D U-Net architecture that incorporates the Transformer in 3D U-Net (Trans3DUNet) to segment multi-modal MR images, called MMTrans3DUNet. Firstly, the tokenized image blocks from a convolutional neural network (CNN) feature mapping are encoded by Transformer as the input sequence to extract the global context. Then, the decoder up-sampling the encoded features and coalesce them in CNN feature mapping with high resolution to achieve exact positioning. Moreover, according to the characteristics of MR images with multiple imaging modes, the four modalities images (t l, t lce, t2, flair) are fused and put into the Trans3DUNet model for training, which can overcome the problem that the single-modal MR image cannot sufficiently subdivide the lesion in the relevant area. The experimental results on the BraTS2018 and BraTS2019 dataset show that MMTrans3DUNet model can further promote the efficiency and precision of segmentation due to the image information of multiple modes which can complement each other.