Reviewerly/exHarmony huggingface dataset:
VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised  Speech Representation Disentanglement for One-shot Voice Conversion

One-shot voice conversion (VC), which performs conversion across arbitrary speakers with only a single target-speaker utterance for reference, can be effectively achieved by speech representation disentanglement. Existing work generally ignores the correlation between different speech representations during training, which causes leakage of content information into the speaker representation and thus degrades VC performance. To alleviate this issue, we employ vector quantization (VQ) for content encoding and introduce mutual information (MI) as the correlation metric during training, to achieve proper disentanglement of content, speaker and pitch representations, by reducing their inter-dependencies in an unsupervised manner. Experimental results reflect the superiority of the proposed method in learning effective disentangled speech representations for retaining source linguistic content and intonation variations, while capturing target speaker characteristics. In doing so, the proposed approach achieves higher speech naturalness and speaker similarity than current state-of-the-art one-shot VC systems. Our code, pre-trained models and demo are available at https://github.com/Wendison/VQMIVC.
Zipf Matrix Factorization : Matrix Factorization with Matthew Effect  Reduction

Recommender system recommends interesting items to users based on users' past information history. Researchers have been paying attention to improvement of algorithmic performance such as MAE and precision@K. Major techniques such as matrix factorization and learning to rank are optimized based on such evaluation metrics. However, the intrinsic Matthew Effect problem poses great threat to the fairness of the recommender system, and the unfairness problem cannot be resolved by optimization of traditional metrics. In this paper, we propose a novel algorithm that incorporates Matthew Effect reduction with the matrix factorization framework. We demonstrate that our approach can boost the fairness of the algorithm and enhances performance evaluated by traditional metrics.
Multiscale Vision Transformers

We present Multiscale Vision Transformers (MViT) for video and image recognition, by connecting the seminal idea of multiscale feature hierarchies with transformer models. Multiscale Transformers have several channel-resolution scale stages. Starting from the input resolution and a small channel dimension, the stages hierarchically expand the channel capacity while reducing the spatial resolution. This creates a multiscale pyramid of features with early layers operating at high spatial resolution to model simple low-level visual information, and deeper layers at spatially coarse, but complex, high-dimensional features. We evaluate this fundamental architectural prior for modeling the dense nature of visual signals for a variety of video recognition tasks where it outperforms concurrent vision transformers that rely on large scale external pre-training and are 5-10x more costly in computation and parameters. We further remove the temporal dimension and apply our model for image classification where it outperforms prior work on vision transformers. Code is available at: https://github.com/facebookresearch/SlowFast
Uncertainty-aware Joint Salient Object and Camouflaged Object Detection

Visual salient object detection (SOD) aims at finding the salient object(s) that attract human attention, while camouflaged object detection (COD) on the contrary intends to discover the camouflaged object(s) that hidden in the surrounding. In this paper, we propose a paradigm of leveraging the contradictory information to enhance the detection ability of both salient object detection and camouflaged object detection. We start by exploiting the easy positive samples in the COD dataset to serve as hard positive samples in the SOD task to improve the robustness of the SOD model. Then, we introduce a similarity measure module to explicitly model the contradicting attributes of these two tasks. Furthermore, considering the uncertainty of labeling in both tasks' datasets, we propose an adversarial learning network to achieve both higher order similarity measure and network confidence estimation. Experimental results on benchmark datasets demonstrate that our solution leads to state-of-the-art (SOTA) performance for both tasks.
Improving Zero-shot Neural Machine Translation on Language-specific  Encoders-Decoders

Recently, universal neural machine translation (NMT) with shared encoder-decoder gained good performance on zero-shot translation. Unlike universal NMT, jointly trained language-specific encoders-decoders aim to achieve universal representation across non-shared modules, each of which is for a language or language family. The non-shared architecture has the advantage of mitigating internal language competition, especially when the shared vocabulary and model parameters are restricted in their size. However, the performance of using multiple encoders and decoders on zero-shot translation still lags behind universal NMT. In this work, we study zero-shot translation using language-specific encoders-decoders. We propose to generalize the non-shared architecture and universal NMT by differentiating the Transformer layers between language-specific and interlingua. By selectively sharing parameters and applying cross-attentions, we explore maximizing the representation universality and realizing the best alignment of language-agnostic information. We also introduce a denoising auto-encoding (DAE) objective to jointly train the model with the translation task in a multi-task manner. Experiments on two public multilingual parallel datasets show that our proposed model achieves a competitive or better results than universal NMT and strong pivot baseline. Moreover, we experiment incrementally adding new language to the trained model by only updating the new model parameters. With this little effort, the zero-shot translation between this newly added language and existing languages achieves a comparable result with the model trained jointly from scratch on all languages.
Integrating Multimodal Electronic Health Records for Diagnosis Prediction.

Diagnosis prediction aims to predict the patient's future diagnosis based on their Electronic Health Records (EHRs). Most existing works adopt recurrent neural networks (RNNs) to model the sequential EHR data. However, they mainly utilize medical codes and ignore other useful information such as patients' clinical features and demographics. We proposed a new model called MDP to augment the prediction performance by integrating the multimodal clinical data. MDP learns the clinical feature representation by adjusting the weights of clinical features based on a patient's current health condition and demographics. Also, the clinical feature representation, diagnosis codes representation and the demographic embedding are integrated to perform the prediction task. Experiments on a real-world dataset demonstrate that MDP outperforms the state-of-the-art methods.
Towards Society 5.0 in Perspective of Agile Society

Both explicitly and implicitly, the central role of my writing is given to social trust, especially to its role in sustainable social futuring as a form of agile society in the twenty-first century. I consider the characterisation of trust a crucial notion as a theoretical frame of my paper. In the first part of my writing, I plan to introduce trust as social capital opposed to financial capital and other forms of human capital, its role in agile society building, focussing on its function as a social connective tissue. The theoretical part will be followed by a retrospection to the twentieth century Industry 4.0 peculiarities as a way that leads societal actors to the creation of Society 5.0 on a scientific, technological and political level. I intend to prove or confute statements regarding the relationship between social trust and the space for inclusive and sustainable social futuring, and Industry 4.0 peculiarities and inventions as a way of creating Society 5.0 and generating greater trust towards it. Finally, I plan to draw conclusions of the lessons of social futuring activities as a tool of creating agile society. I find this paper an especially important one in addressing the challenges of social trust building, of finding the appropriate steps towards the creation of an agile and responsive society and in describing the characterisations of Society 5.0.
Rethinking the Heatmap Regression for Bottom-up Human Pose Estimation

Heatmap regression has become the most prevalent choice for nowadays human pose estimation methods. The ground-truth heatmaps are usually constructed via covering all skeletal keypoints by 2D gaussian kernels. The standard deviations of these kernels are fixed. However, for bottom-up methods, which need to handle a large variance of human scales and labeling ambiguities, the current practice seems unreasonable. To better cope with these problems, we propose the scale-adaptive heatmap regression (SAHR) method, which can adaptively adjust the standard deviation for each keypoint. In this way, SAHR is more tolerant of various human scales and labeling ambiguities. However, SAHR may aggravate the imbalance between fore-background samples, which potentially hurts the improvement of SAHR. Thus, we further introduce the weight-adaptive heatmap regression (WAHR) to help balance the fore-background samples. Extensive experiments show that SAHR together with WAHR largely improves the accuracy of bottom-up human pose estimation. As a result, we finally outperform the state-of-the-art model by +1.5AP and achieve 72.0AP on COCO test-dev2017, which is com-arable with the performances of most top-down methods. Source codes are available at https://github.com/greatlog/SWAHR-HumanPose.
Mitigating Sybil Attacks on Differential Privacy based Federated  Learning

In federated learning, machine learning and deep learning models are trained globally on distributed devices. The state-of-the-art privacy-preserving technique in the context of federated learning is user-level differential privacy. However, such a mechanism is vulnerable to some specific model poisoning attacks such as Sybil attacks. A malicious adversary could create multiple fake clients or collude compromised devices in Sybil attacks to mount direct model updates manipulation. Recent works on novel defense against model poisoning attacks are difficult to detect Sybil attacks when differential privacy is utilized, as it masks clients' model updates with perturbation. In this work, we implement the first Sybil attacks on differential privacy based federated learning architectures and show their impacts on model convergence. We randomly compromise some clients by manipulating different noise levels reflected by the local privacy budget epsilon of differential privacy on the local model updates of these Sybil clients such that the global model convergence rates decrease or even leads to divergence. We apply our attacks to two recent aggregation defense mechanisms, called Krum and Trimmed Mean. Our evaluation results on the MNIST and CIFAR-10 datasets show that our attacks effectively slow down the convergence of the global models. We then propose a method to keep monitoring the average loss of all participants in each round for convergence anomaly detection and defend our Sybil attacks based on the prediction cost reported from each client. Our empirical study demonstrates that our defense approach effectively mitigates the impact of our Sybil attacks on model convergence.
COMET: Convolutional Dimension Interaction for Collaborative Filtering

Representation learning-based recommendation models play a dominant role among recommendation techniques. However, most of the existing methods assume both historical interactions and embedding dimensions are independent of each other, and thus regrettably ignore the high-order interaction information among historical interactions and embedding dimensions. In this article, we propose a novel representation learning-based model called COMET ( CO nvolutional di M E nsion in T eraction), which simultaneously models the high-order interaction patterns among historical interactions and embedding dimensions. To be specific, COMET stacks the embeddings of historical interactions horizontally at first, which results in two “embedding maps”. In this way, internal interactions and dimensional interactions can be exploited by convolutional neural networks (CNN) with kernels of different sizes simultaneously. A fully connected multi-layer perceptron (MLP) is then applied to obtain two interaction vectors. Lastly, the representations of users and items are enriched by the learnt interaction vectors, which can further be used to produce the final prediction. Extensive experiments and ablation studies on various public implicit feedback datasets clearly demonstrate the effectiveness and rationality of our proposed method.
RaSE: Random Subspace Ensemble Classification

We propose a flexible ensemble classification framework, Random Subspace Ensemble (RaSE), for sparse classification. In the RaSE algorithm, we aggregate many weak learners, where each weak learner is a base classifier trained in a subspace optimally selected from a collection of random subspaces. To conduct subspace selection, we propose a new criterion, ratio information criterion (RIC), based on weighted Kullback-Leibler divergence. The theoretical analysis includes the risk and Monte-Carlo variance of the RaSE classifier, establishing the screening consistency and weak consistency of RIC, and providing an upper bound for the misclassification rate of the RaSE classifier. In addition, we show that in a high-dimensional framework, the number of random subspaces needs to be very large to guarantee that a subspace covering signals is selected. Therefore, we propose an iterative version of the RaSE algorithm and prove that under some specific conditions, a smaller number of generated random subspaces are needed to find a desirable subspace through iteration. An array of simulations under various models and real-data applications demonstrate the effectiveness and robustness of the RaSE classifier and its iterative version in terms of low misclassification rate and accurate feature ranking. The RaSE algorithm is implemented in the R package RaSEn on CRAN.
Motion Planning of Automatic Underwater Vehicle Based on 3D Dipolar Vector Field

This paper investigates a 3-dimensional (3D) motion planning problem of an automatic underwater vehicle (AUV) in ocean environment with multiple stationary/moving spherical obstacles. To solve the 3D motion planning problem, a family of novel 3D analytic vector fields are employed, the representation of which depends on the selection of the integral curve according to the parameter λ. More specifically, when the value of λ is known, the vector field has a unique singularity of the dipole type and its integral curves are able to attract an AUV to a predefined target point. Furthermore, the vector field integral curve with continuous singular points enables the AUV to repel spherical obstacles. By blending the above two kinds of vector fields, a mixed vector field is generated in 3D space such that an AUV can simultaneously reach the target point and avoid spherical obstacles. The efficacy of the proposed 3D motion planning approach is demonstrated via simulation results.
Network Traffic Classification Based on Domain Adaptive Migration for Multimedia Services in Smart City Networks

Network traffic classification is a key technology for multimedia service management in smart city networks, which is of great significance to broadcast applications to smart cities. According to the characteristics of different network traffic to ensure the full utilization of broadband resources and improve the quality of network service. Due to the wide application of dynamic port number and stream encryption technology, the traditional network traffic classification method based on port number and deep packet inspection is no longer effective. In recent years, machine learning methods can be used to accurately identify categories of network traffic applications by labeled data. However, collecting enough labeled network traffic data is a time-consuming and difficult task, and the distribution of traffic data collected in various networks is different. In order to solve the problem of insufficient labeled samples and domain distribution differences, a network traffic classification method based on domain adaptive migration is proposed in this paper. This paper designs a new domain adaptive method, which reduces distribution differences between domains, makes the distribution of samples under the same category more compact and different application categories more separated. Then migrate the feature extraction module to the target domain. And the unlabeled data and few labeled samples are utilized to jointly classify network traffic applications using a siamese sparse denoising stacked autoencoder. Experimental results show that compared with other algorithms, the network traffic classification algorithm proposed in this paper achieves the best result.
Temporal Generalization for Spoken Language Understanding

Judith Gaspers, Anoop Kumar, Greg Ver Steeg, Aram Galstyan. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track. 2022.
CausalML: Python Package for Causal Machine Learning

CausalML is a Python implementation of algorithms related to causal inference and machine learning. Algorithms combining causal inference and machine learning have been a trending topic in recent years. This package tries to bridge the gap between theoretical work on methodology and practical applications by making a collection of methods in this field available in Python. This paper introduces the key concepts, scope, and use cases of this package.
Recent Advances of Generative Adversarial Networks

Generative adversarial networks (GANs) have received great attention recently. GAN is an unsupervised learning method that learns representations without highly relying on annotations. GAN comprises two networks called generator and discriminator that training in a competitive process. GANs have made notable progress and promising performance in various applications including image inpainting, image super-resolution, and face synthesis. This paper aims to introduce an overview of GAN, including fundamentals, several up-to-date variants, applications, and challenges of GAN. Finally, we also provide readers with some solutions to mitigate issues existing in GANs.
YNU-HPCC at SemEval-2022 Task 5: Multi-Modal and Multi-label Emotion Classification Based on LXMERT

This paper describes our system used in the SemEval-2022 Task5 Multimedia Automatic Misogyny Identification (MAMI).This task is to use the provided text-image pairs to classify emotions.In this paper, We propose a multi-label emotion classification model based on pre-trained LXMERT.We use Faster-RCNN to extract visual representation and utilize LXMERT's cross-attention for multimodal alignment.Then we use the Bilinearinteraction layer to fuse these features.Our experimental results surpass the F 1 score of baseline.For Sub-task A, our F 1 score is 0.662 and Sub-task B's F 1 score is 0.633.The code of this study is available on GitHub 1 .
Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models

Distilling state-of-the-art transformer models into lightweight student models is an effective way to reduce computation cost at inference time.The student models are typically compact transformers with fewer parameters, while expensive operations such as selfattention persist.Therefore, the improved inference speed may still be unsatisfactory for real-time or high-volume use cases.In this paper, we aim to further push the limit of inference speed by distilling teacher models into bigger, sparser student models -bigger in that they scale up to billions of parameters; sparser in that most of the model parameters are ngram embeddings.Our experiments on six single-sentence text classification tasks show that these student models retain 97% of the RoBERTa-Large teacher performance on average, and meanwhile achieve up to 600x speedup on both GPUs and CPUs at inference time.Further investigation reveals that our pipeline is also helpful for sentence-pair classification tasks, and in domain generalization settings.
Benchmarking Language-agnostic Intent Classification for Virtual Assistant Platforms

Current virtual assistant (VA) platforms are beholden to the limited number of languages they support. Every component, such as the tokenizer and intent classifier, is engineered for specific languages in these intricate platforms. Thus, supporting a new language in such platforms is a resource-intensive operation requiring expensive re-training and re-designing. In this paper, we propose a benchmark for evaluating language-agnostic intent classification, the most critical component of VA platforms. To ensure the benchmarking is challenging and comprehensive, we include 29 public and internal datasets across 10 low-resource languages and evaluate various training and testing settings with consideration of both accuracy and training time. The benchmarking result shows that Watson Assistant, among 7 commercial VA platforms and pre-trained multilingual language models (LMs), demonstrates close-to-best accuracy with the best accuracy-training time trade-off.
Capturing Conversational Interaction for Question Answering via Global History Reasoning

Conversational Question Answering (ConvQA) is required to answer the current question, conditioned on the observable paragraph-level context and conversation history.Previous works have intensively studied history-dependent reasoning.They perceive and absorb topic-related information of prior utterances in the interactive encoding stage.It yielded significant improvement compared to history-independent reasoning.This paper further strengthens the ConvQA encoder by establishing long-distance dependency among global utterances in multiturn conversation.We use multi-layer transformers to resolve long-distance relationships, which potentially contribute to the reweighting of attentive information in historical utterances.Experiments on QuAC show that our method obtains a substantial improvement (1%), yielding the F1 score of 73.7%.
Physically-based sound synthesis software for Computer-Aided-Design of piano soundboards

The design of pianos is mainly based on empirical knowledge due to the lack of a simple tool that could predict sound changes induced by modifications of the geometry and/or the mechanical properties of the soundboard. We introduce the concept of Sound Computer-Aided Design through the framework of a program that is intended to simulate the acoustic results of virtual pianos. The calculation of the sound is split into four modules that compute respectively the modal basis of the stiffened soundboard, the string dynamics excited by the hammer, the soundboard dynamics excited by the string vibration, and the sound radiation. The exact resemblance between synthesis and natural sounds is not the primary purpose of the software. However, sound synthesis of real and modified pianos are used as reference tests to assess our main objective, namely to reflect faithfully structural modifications in the produced sound, and thus to make this tool helpful for both instrument makers and researchers of the musical acoustics community.