Reviewerly/exHarmony huggingface dataset:
Improving Dynamic Graph Convolutional Network with Fine-Grained Attention Mechanism

Graph convolutional network (GCN) is a novel framework that utilizes a pre-defined Laplacian matrix to learn graph data effectively. With its powerful nonlinear fitting ability, GCN can produce high-quality node embedding. However, generalized GCN can only handle static graphs, whereas a large number of graphs are dynamic and evolve over time, which limits the application field of GCN. Facing the challenge, GCN with recurrent neural network (e.g., RNN) is naturally combined to acquire dynamic graph changes through joint training. However, these methods must use the node information during the entire timeline and ignore two subtle factors: the influence of nodes change with time and are related to the frequency of events. Therefore, we propose a stable and scalable dynamic GCN method using a fine-grained attention mechanism named FADGC. We use GCN to obtain static node vectors at each timestep and integrate node influence factors with multi-head attention for graph time-series learning. Experiments on multiple datasets show that our approach can better capture the inherent special characteristics of different dynamic graphs and achieve higher performance compared with related approaches.
Incorporating Explicit Knowledge in Pre-trained Language Models for Passage Re-ranking

Passage re-ranking is to obtain a permutation over the candidate passage set from retrieval stage. Re-rankers have been boomed by Pre-trained Language Models (PLMs) due to their overwhelming advantages in natural language understanding. However, existing PLM based re-rankers may easily suffer from vocabulary mismatch and lack of domain specific knowledge. To alleviate these problems, explicit knowledge contained in knowledge graph is carefully introduced in our work. Specifically, we employ the existing knowledge graph which is incomplete and noisy, and first apply it in passage re-ranking task. To leverage a reliable knowledge, we propose a novel knowledge graph distillation method and obtain a knowledge meta graph as the bridge between query and passage. To align both kinds of embedding in the latent space, we employ PLM as text encoder and graph neural network over knowledge meta graph as knowledge encoder. Besides, a novel knowledge injector is designed for the dynamic interaction between text and knowledge encoder. Experimental results demonstrate the effectiveness of our method especially in queries requiring in-depth domain knowledge.
A Natural Language Processing based Intelligent Bot Application

In the modern day technology, superior models can be designed due to technology like Natural Language Processing, Open CV, python etc. and bot assistant is one among them. Personal robot assistants assist lowering the guide efforts being placed by humans for their daily tasks. In this paper, a voice-managed non-public assistant has been proposed. The human voice instructions are given to the robot assistant remotely and the bot can carry out distinct movements, turns, start/prevent operations and relocate an item from one area to another. The speech sign instructions transformed to textual content are communicated to the robot over a Bluetooth network. The non-public assistant robot is built primarily on a micro-controller based platform and is aware of the modern location. The effectiveness of the voice manipulated communication over a distance is measured via numerous experiments. Performance assessment is completed with encouraging effects of the preliminary experiments. Moreover, possible enhancements are mentioned towards the capability packages in home, hospitals and industries.
The Process of Using Face Detection Through Convolutional Neural Network

Face detection has become a technology in recognizing facial patterns. The presence of students in the learning process is essential to find out the number of student attendance accordingly. This study was conducted to create a face detection system in counting the number of students, that captures and detects the student faces accurately and conduct testing on the system that has been created. Neural Network is a Deep Learning method that can be used to recognize and classify an object in a digital image. The accuracy rate of the training process is 99%, and the testing is 98%. Testing the system that has been made using the Raspberry Pi, the first true label for facial recognition with a distance of 1-5 meters has been identified, whereas at a distance of 4 meters has not been identified, while to detect the number of students obtained with a distance of 1-6 meters can detect the number with accuracy.
Fairness-Aware Selective Sampling on Attributed Graphs

Selective sampling is an online learning framework where the learner tries to detect the data samples whose labels can boost the performance maximally, and only the labels of chosen data samples are queried. While the design of selective sampling algorithms is extensively studied for independent data samples, the area is rather under-explored in the context of graphs. Furthermore, the limited number of existing graph-based approaches do not take into consideration the nodal attributes that are available in attributed graphs. In this study, existing online learning and selective sampling algorithms are modified to be used with graphs that have nodal features. Additionally, the bias in the results of original algorithms is investigated, and a novel bias reduction strategy is proposed that can be embedded into the dimensionality reduction step without incurring a significant complexity cost. Experiments for node classification are carried out on real social networks to showcase the advantages of incorporating nodal features, as well as the fairness-enhancement framework.
An Investigation of Streaming Non-Autoregressive sequence-to-sequence Voice Conversion

Recent advances in sequence-to-sequence (S2S) models have improved the quality of voice conversion (VC), but it requires the entire sequence to perform inference, which prevents using it in real-time applications. To address this issue, this paper extends the non-autoregressive (NAR) S2S-VC model to enable us to perform streaming VC. We introduce streamable architectures such as causal convolution and self-attention with causal masking for the FastSpeech2-based NAR-S2S-VC model. The streamable architecture also tries to convert durations, which are kept as is in conventional real-time VC methods. To further improve the performance of the streaming VC model, we utilize an instant knowledge distillation with a dual-mode architecture, which performs non-causal and causal inference by sharing the network parameters. Through the experimental evaluation with Japanese parallel corpus, we investigate the impact on performance caused by the streamable architecture. The experimental results reveal that the use of future context frames increases latency, but it improves the conversion quality and that the difference in the speaking rate affects the performance of streaming inference.
A Simple Graph Neural Network via Layer Sniffer

Due to the success of Graph Neural Networks(GNNs) in graph-structure data, many efforts have been devoted to enhancing the propagation ability and alleviating the over-smoothing problem of GNNs. However, from the perspective of closeness extent of node representations, most existing GNNs pay less attention to the attributes of node representation space. In light of this, we design a Layer Sniffer module that can combine the effects of the local node-level representation closeness extent and the global layer-level information attention. On this basis, we propose a simple Layer Sniffer Graph Neural Network (LSGNN) with a propagation scheme that can fuse neighborhood information of different receptive fields densely and adaptively. Our extensive experiments on three public node classification datasets demonstrate the superior performance and stability of our proposed model.
Disentangled Speaker Embedding for Robust Speaker Verification

Entanglement of speaker features and redundant features may lead to poor performance when evaluating speaker verification systems on an unseen domain. To address this issue, we propose an InfoMax domain separation and adaptation network (InfoMax–DSAN) to disentangle the domain-specific features and domain-invariant speaker features based on domain adaptation techniques. A frame-based mutual information neural estimator is proposed to maximize the mutual information between frame-level features and input acoustic features, which can help retain more useful information. Furthermore, we propose adopting triplet loss based on the idea of self-supervised learning to overcome the label mismatch problem. Experimental results on VOiCES Challenge 2019 demonstrate that our proposed method can help learn more discriminative and robust speaker embeddings.
Social Welfare Maximization in Cross-Silo Federated Learning

As one of the typical settings of Federated Learning (FL), cross-silo FL allows organizations to jointly train an optimal Machine Learning (ML) model. In this case, some organizations may try to obtain the global model without contributing their local training, lowering the social welfare. In this paper, we model the interactions among organizations in cross-silo FL as a public goods game for the first time and theoretically prove that there exists a social dilemma where the maximum social welfare is not achieved in Nash equilibrium. To over-come this social dilemma, we employ the Multi-player Multi-action Zero-Determinant (MMZD) strategy to maximize the social welfare. With the help of the MMZD, an individual organization can unilaterally control the social welfare without extra cost. Experimental results validate that the MMZD strategy is effective in maximizing the social welfare.
A New Framework for Multiple Deep Correlation Filters Based Object Tracking

In recent years, Correlation Filter (CF) based tracking methods using Convolutional Neural Network (CNN) features have achieved the state-of-the-art performance for object tracking. However, how to design an efficient deep CF based tracking method has not been well studied in the literature. To address this issue, we first develop a generic framework, which breaks a deep CF based tracking method into five components, including motion model, CNN feature extractor, CF model, CF updater, and location model. According to this framework, we design each component step by step. Then we propose a novel deep CF based tracking method by combining five effective components together. The proposed method outperforms several state-of-the-art tracking methods on two tracking benchmarks. Then the ablative experiments are conducted to study the influence of each component. The results show that the CF model and the CNN feature extractor play the most important roles in a deep CF based tracking method. Moreover, the CF updater, the location model, and the motion model can also improve the performance substantially.
Attention Guided Invariance Selection for Local Feature Descriptors

To copy with the extreme variations of illumination and rotation in the real world, popular descriptors have captured more invariance recently, but more invariance makes descriptors less informative. So this paper designs a unique attention guided framework (named AISLFD) to select appropriate invariance for local feature descriptors, which boosts the performance of descriptors even in the scenes with extreme changes. Specifically, we first explore an efficient multi-scale feature extraction module that provides our local descriptors with more useful information. Besides, we propose a novel parallel self-attention module to get meta descriptors with the global receptive field, which guides the invariance selection more correctly. Compared with state-of-the-art methods, our method achieves competitive performance through sufficient experiments.
Text2Poster: Laying Out Stylized Texts on Retrieved Images

Poster generation is a significant task for a wide range of applications, which is often time-consuming and requires lots of manual editing and artistic experience. In this paper, we propose a novel data-driven framework, called Text2Poster, to automatically generate visually-effective posters from textual information. Imitating the process of manual poster editing, our framework leverages a large-scale pretrained visual-textual model to retrieve background images from given texts, lays out the texts on the images iteratively by cascaded autoencoders, and finally, stylizes the texts by a matching-based method. We learn the modules of the framework by weakly-and self-supervised learning strategies, mitigating the demand for labeled data. Both objective and subjective experiments demonstrate that our Text2Poster outperforms state-of-the-art methods, including academic research and commercial software, on the quality of generated posters.
Personalized Pagerank Graph Attention Networks

There has been a rising interest in graph neural networks (GNNs) for representation learning over the past few years. GNNs provide a general and efficient framework to learn from graph-structured data. However, GNNs typically only use the information of a very limited neighborhood for each node to avoid over-smoothing. A larger neighborhood would be desirable to provide the model with more information. In this work, we incorporate the limit distribution of Personalized PageRank (PPR) into graph attention networks (GATs) to reflect the larger neighbor information without introducing over-smoothing. Intuitively, message aggregation based on Personalized PageRank corresponds to infinitely many neighborhood aggregation layers. We show that our models outperform a variety of baseline models for four widely used benchmark datasets. Our implementation is publicly available online. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Inverse design of ultracompact multi-focal optical devices by diffractive neural networks

We propose an efficient inverse design approach for multifunctional optical elements based on adaptive deep diffractive neural networks (a-D2NNs). Specifically, we introduce a-D2NNs and design two-layer diffractive devices that can selectively focus incident radiation over two well-separated spectral bands at desired distances. We investigate focusing efficiencies at two wavelengths and achieve targeted spectral line shapes and spatial point-spread functions (PSFs) with optimal focusing efficiency. In particular, we demonstrate control of the spectral bandwidths at separate focal positions beyond the theoretical limit of single-lens devices with the same aperture size. Finally, we demonstrate devices that produce super-oscillatory focal spots at desired wavelengths. The proposed method is compatible with current diffractive optics and doublet metasurface technology for ultracompact multispectral imaging and lensless microscopy applications.
HiFi-SVC: Fast High Fidelity Cross-Domain Singing Voice Conversion

This paper presents HiFi-SVC, a small cross-domain singing voice conversion model for generating high-fidelity 22.05 kHz singing voices. Building on state-of-the-art neural vocoder HiFi-GAN and a convolution-based module for modeling F0, HiFi-SVC can be trained end-to-end with either speech or singing data, achieving better voice similarity on two of the datasets than FastSVC while using slightly smaller number of parameters. We also propose a pitch adjustment method for improving conversion quality.
Learning Domain-Invariant Transformation for Speaker Verification

Automatic speaker verification (ASV) faces domain shift caused by the mismatch of intrinsic and extrinsic factors such as recording device and speaking style in real-world applications, which leads to unsatisfactory performance. To this end, we propose the meta generalized transformation via meta-learning to build a domain-invariant embedding space. Specifically, the transformation module is motivated to learn the domain generalization knowledge by executing meta-optimization on the meta-train and meta-test sets which are designed to simulate domain shift. Furthermore, distribution optimization is incorporated to supervise the metric structure of embeddings. In terms of the transformation module, we investigate various instantiations and observe the multilayer perceptron with gating (gMLP) is the most effective given its extrapolation capability. The experimental results on cross-genre and cross-dataset settings demonstrate that the meta generalized transformation dramatically improves the robustness of ASV systems to domain shift, while outperforms the state-of-the-art methods.
Discourse-Level Prosody Modeling with a Variational Autoencoder for Non-Autoregressive Expressive Speech Synthesis

To address the issue of one-to-many mapping from phoneme sequences to acoustic features in expressive speech synthesis, this paper proposes a method of discourse-level prosody modeling with a variational autoencoder (VAE) based on the non-autoregressive architecture of FastSpeech. In this method, phone-level prosody codes are extracted from prosody features by combining VAE with FastSpeech, and are predicted using discourse-level text features together with BERT embeddings. The continuous wavelet transform (CWT) in FastSpeech2 for F0 representation is not necessary anymore. Experimental results on a Chinese audiobook dataset show that our proposed method can effectively take advantage of discourse-level linguistic information and has outperformed FastSpeech2 on the naturalness and expressiveness of synthetic speech.
No-Reference Quality Assessment of Variable Frame-Rate Videos Using Temporal Bandpass Statistics

Recent advances in mobile devices and cloud computing techniques have made it possible to capture, process, and share high resolution, high frame rate (HFR) videos across the Internet nearly instantaneously. Being able to monitor and control the quality of these streamed videos can enable the de-livery of many enjoyable content and perceptually optimized rate control. However, the development of no-reference (NR) VQA algorithms targeting frame rate variations has been little studied. Here, we propose a first-of-a-kind blind VQA model for evaluating HFR videos, which we dub the Framerate-Aware Videos Evaluator w/o Reference (FAVER). FAVER uses extended models of spatial natural scene statistics that encompass space-time wavelet-decomposed video signals, to conduct efficient frame rate sensitive quality prediction. Our extensive experiments on several HFR video quality datasets show that FAVER outperforms other blind VQA algorithms at a reasonable computational cost. The code will be released on https://github.com/uniqzheng/HFR-BVQA.
A Robust Object Segmentation Network for UnderWater Scenes

Underwater object segmentation is one of the key technologies in the fields of marine biology research and autonomous underwater vehicles. The challenges of underwater object segmentation originate from two aspects, 1) the complex underwater environment and 2) the camouflage characteristics of marine animals. In this paper, we propose WaterSNet, an underwater object segmentation network to address these challenges. Specified, we propose a random style adaption (RSA) module as well as a siamese structure to reduce the impact of water degradation diversity. We also extract multi-scale features via the receptive field block (RFB) module, and then fuses multi-level features to better utilize global context information via the attention fusion block (AFB) module. Experimental results on marine animal dataset MAS3K demonstrate that the proposed method outperforms other state-of-the-art methods significantly. The code will be available at: https://github.com/ruizhechen/WaterSNet/
Wavelet-Based Unsupervised Label-to-Image Translation

Semantic Image Synthesis (SIS) is a subclass of image-to-image translation where a semantic layout is used to generate a photorealistic image. State-of-the-art conditional Generative Adversarial Networks (GANs) need a huge amount of paired data to accomplish this task while generic un-paired image-to-image translation frameworks underperform in comparison, because they color-code semantic layouts and learn correspondences in appearance instead of semantic content. Starting from the assumption that a high quality generated image should be segmented back to its semantic layout, we propose a new Unsupervised paradigm for SIS (USIS) that makes use of a self-supervised segmentation loss and whole image wavelet based discrimination. Furthermore, in order to match the high-frequency distribution of real images, a novel generator architecture in the wavelet domain is proposed. We test our methodology on 3 challenging datasets and demonstrate its ability to bridge the performance gap between paired and unpaired models.
Duration Modeling of Neural TTS for Automatic Dubbing

Automatic dubbing (AD) addresses the problem of translating speech in a video with speech in another language while preserving the viewer experience. A most important requirement of AD is isochrony, i.e. dubbed speech has to closely match the timing of speech and pauses of the original audio. In our automatic dubbing system, isochrony is modeled by controlling the verbosity of machine translation; inserting pauses in the translations, a.k.a. prosodic alignment; and controlling the duration of text-to-speech (TTS) utterances. The latter two steps heavily rely on speech duration information, either to predict or control TTS duration. So far, duration prediction was based on a proxy method while duration control on linear warping of the TTS speech spectrogram. In this study, we propose novel duration models for neural TTS that can be leveraged both to predict and control TTS duration. Experimental results show that compared to previous work, the new models improve or match the performance of prosodic alignment and significantly enhance neural TTS speech quality for both slow and fast speaking rates.
Memory in Echo State Networks and the Controllability Matrix Rank

Echo State Networks (ESNs) are a variant of recurrent neural networks (RNNs). ESNs perform as nonlinear fading memory filters and excel in prediction of "chaotic" signals. Predictions are made using a forced nonlinear dynamical system called the "reservoir" which incorporates past information into new states. The length of memory is critical to a task effective ESN. We examine the rank behavior of minimal task-effective ESNs predicting the chaotic Lorenz 1963 system for single and multi-variable input/output. Relationships are observed between the rank of the controllability matrix, memory length, and attractor features. We find that reservoir memory varies dependent on input forcing and location in state space. Knowledge of controllability matrix rank indicates a signal specific range for memory length. This variability corresponds to the reservoir varying between stable and unstable. The controllability matrix rank can facilitate eﬃcient use of data and ESN construction.
Improve Image Captioning Via Relation Modeling

The performance of image captioning has been significantly improved recently through deep neural network architectures combining with attention mechanisms and reinforcement learning optimization. Exploring visual relationships and interactions between different objects appearing in the image, however, is far from being investigated. In this paper, we present a novel approach that combines scene graphs with Transformer, which we call SGT, to explicitly encode available visual relationships between detected objects. Specifically, we pretrain an scene graph generation model to predict graph representations for images. After that, for each graph node, a Graph Convolutional Network (GCN) is employed to acquire relationship knowledge by aggregating the information of its local neighbors. As we train the captioning model, we feed the potential relation-aware information into the Transformer to generate descriptive sentence. Experiments on the MS COCO dataset validate the superiority of our SGT model, which can realize state-of-the-art results in terms of all the standard evaluation metrics.