Reviewerly/exHarmony huggingface dataset:
Clicker: Attention-Based Cross-Lingual Commonsense Knowledge Transfer

Recent advances in cross-lingual commonsense reasoning (CSR) are facilitated by the development of multilingual pre-trained models (mPTMs). While mPTMs show the potential to encode commonsense knowledge for different languages, transferring commonsense knowledge learned in large-scale English corpus to other languages is challenging. To address this problem, we propose the attention-based Cross-LIngual Commonsense Knowledge transfER (CLICKER) framework, which minimizes the performance gaps between English and non-English languages in commonsense question-answering tasks. CLICKER effectively improves commonsense reasoning for non-English languages by differentiating non-commonsense knowledge from commonsense knowledge. Experimental results on public benchmarks demonstrate that CLICKER achieves remarkable improvements in the cross-lingual CSR task for languages other than English.
CLMAE: A Liter and Faster Masked Autoencoders

Self-supervised pre-training has been widely utilized on various vision tasks and gains a great success. However, pre-training on big datasets suffers a lengthy training schedule and large memory consumption. To alleviate these problems, we propose a light-weighted model called Convolutional Lite Masked AutoEncoder (CLMAE). To improve the convergence speed of the transformer during pre-training. We introduce two-stage convolutional progressive patch embedding and an additional convolution in the feed-forward layer, which promote better correlation among patches in the spatial dimensions. The most important design is called cross-layer parameter sharing mechanism, which reduces model parameters with little impact on the performance. We find that sharing parameters among layers not only improves the parameter efficiency, but also acts as a form of regularization that stabilizes the training. Experimental results on downstream tasks show the effectiveness and generalization ability of CLMAE, which accelerates the training process significantly (by 5× for ViT-B and MAE) and reduces a quarter of parameters (by 25M fewer for ViT-B), with a competitive accuracy (82.8% on ImageNet-1K).
Enhancement of Text-Predicting Style Token With Generative Adversarial Network for Expressive Speech Synthesis

This work proposes an advanced text-predicting style embedding for expressive speech synthesis. Text-predicting global style token (TPGST) predicts style embedding from text instead of reference speech and uses it to condition a text-to-speech synthesis (TTS) model, resulting in style TTS without reference speech. Although this minimizes the style embedding’s L1-loss between that extracted from reference speech and that predicted during training, predicted embedding tends to be over-smoothed. To overcome this issue, the proposed method uses the generative adversarial network (GAN) in training style predictors. This not only improves style reproduction, but also aims to reduce style conditioning mismatch during TTS model training. We also utilize TTS text embeddings as in other related work, as well as word information via BERT in order to find better style distributions in GAN. An evaluation of subjective style reproduction demonstrates that 1) the proposed method outperforms conventional TPGST, and 2) the use of words yielded by BERT provides even better performance. Our style predictor is also effective in attaining unseen style TTS for "seen" and "unseen" speakers.
Lightweight Feature Encoder for Wake-Up Word Detection Based on Self-Supervised Speech Representation

Self-supervised learning method that provides generalized speech representations has recently received increasing attention. Wav2vec 2.0 is the most famous example, showing remarkable performance in numerous downstream speech processing tasks. Despite its success, it is challenging to use it directly for wake-up word detection on mobile devices due to its expensive computational cost. In this work, we propose LiteFEW, a lightweight feature encoder for wake-up word detection that preserves the inherent ability of wav2vec 2.0 with a minimum scale. In the method, the knowledge of the pre-trained wav2vec 2.0 is compressed by introducing an auto-encoder-based dimensionality reduction technique and distilled to LiteFEW. Experimental results on the open-source "Hey Snips" dataset show that the proposed method applied to various model structures significantly improves the performance, achieving over 20% of relative improvements with only 64k parameters.
Image Sharing Chain Detection VIA Sequence-To-Sequence Model

Image sharing chain detection aims to recover the sharing history of an image downloaded from online social networks (OSNs), including the ever-shared OSNs and their orders, which is an important task in the multimedia forensics community. Most of the existing algorithms directly treat the sharing chain detection as a classification problem by simply assigning a unique label to each sharing chain. Such a strategy though seems straightforward, it ignores the inherent properties of the sharing chain which can be regarded as a time sequence that carries the sharing history of an online image. In this paper, we suggest a new sharing chain detection framework via Sequence-to-Sequence (Seq2Seq) model. Different from previous classification based approaches, our model detects the sharing chain of online image progressively via a decoder. This progressive manner can fully utilize the decoded chain, which is embedded into a series of learned representations. Experimental results show that our method can detect sharing chains involving up to three OSNs, and exhibits much better performance than conventional ones.
Stereoscopic Video Retargeting Based on Camera Motion Classification

The existing stereo video retargeting algorithms commonly use a same methodology to perform resizing without considering different videos with various features, leading to the low quality of reconstructed videos. To address this issue, we propose a stereo video retargeting method based on camera motion classification, which employs different retargeting strategies to rescale stereo videos. We also design an adaptive stereo video classification method which determines the types of camera motion according to the distribution of motion vectors extracted from the left view of stereo videos. Besides, we develop a motion saliency detection method to eliminate the jittering of moving objects during video resizing. Experimental results show that the qualities of retargeted videos produced by our method are significantly superior to those of existing methods.
A Sentiment and Syntactic-Aware Graph Convolutional Network for Aspect-Level Sentiment Classification

Aspect-level sentiment classification (ASC) is a significant problem in fine-grained sentiment analysis, which automatically predicts the sentiment polarity of a given aspect in a sentence. Dependency tree-based graph convolutional networks have been widely studied for their ability to effectively capture the dependencies of aspect words with other words. However, constructing more accurate syntactic trees by introducing external knowledge has limited improvement on ungrammatical informal texts and has led to over-parameterization of the model. To alleviate this problem, we propose a sentiment and syntactic-aware graph convolutional network (SaS-GCN) that combines syntactic and sentiment relations. We use an attention mechanism and the Sparsemax activation function to construct a sparse sentiment-dependent graph. Compared with existing methods that use LSTM or CNN to obtain semantics from text directly, this graph, combined with a GCN, contains more semantic features. Moreover, we redesign the network structure of GCN, calling it EN-GCN, to make it sensitive to node dimensional features and hence to have a strong feature mining ability. The experimental results indicate that our model outperforms state-of-the-art methods. In particular, when evaluated on the Rest15 and Rest16 datasets, the F1 scores of the proposed lightweight model are 4.15% and 3.77% better than BERT respectively.
Dialogue Context Modelling for Action Item Detection: Solution for ICASSP 2023 Mug Challenge Track 5

Action item detection aims at recognizing sentences containing information about actionable tasks, which can help people quickly grasp core tasks in the meeting without going through the redundant meeting contents. Therefore, in this paper, we thoroughly describe our carefully designed solution for the Action Item Detection Track of the General Meeting Understanding and Generation (MUG) challenge in the ICASSP 2023 Signal Processing Grand Challenge. Specifically, we systematically analyze the task instances provided by MUG and find that the key ingredient for successful action item detection is leveraging the dialogue context information into consideration. To this end, we design a simple and effective method for modelling context and utterance information concurrently. The experimental results show our method achieves remarkable improvements over baseline models, with an absolute increase of 0.62 of the F <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf> score on the validation set. The stable generalizability of our method is further verified by our score on the final test set <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Batch-Ensemble Stochastic Neural Networks for Out-of-Distribution Detection

Out-of-distribution (OOD) detection has recently received much attention from the machine learning community because it is important for deploying machine learning models in real-world applications. In this paper we propose an uncertainty quantification approach by modeling data distributions in feature spaces. We further incorporate an efficient ensemble mechanism, namely batch-ensemble, to construct the batch-ensemble stochastic neural networks (BE-SNNs) and overcome the feature collapse problem. We compare the performance of the proposed BE-SNNs with the other state-of-the-art approaches and show that BE-SNNs yield superior performance on several OOD detection benchmarks, such as the Two-Moons dataset, the FashionMNIST vs MNIST dataset, Fashion-MNIST vs NotMNIST dataset, and the CIFAR10 vs SVHN dataset.
Optimal Kernel for Real-Time Arbitrary-Shaped Text Detection

Recently, segmentation-based text detection methods develop rapidly, which achieve competitive accuracy and detection speed. However, these methods are hard to fit text instances accurately, which leads to the decrease of model performance. Meanwhile, the poor perception of the text center by the boundary pixels further affects the detection accuracy. We follow the issues and design an efficient framework for arbitrary-shaped text detection, which is constructed based on Optimal Kernel Representation (OKR) and Pixel Enhancement Module (PEM). Specifically, OKR is proposed to fit texts with optimal kernels. It erodes texts according to the corresponding geometric characteristics, which is simpler and more accurate compared with previous methods. PEM is used to enhance the perception of boundary pixels to the virtual character centers of text, thus improving the cohesion of the whole instance. Particularly, PEM only participates in the training process, which brings no extra computation costs to inference. Ablation experiments show the effectiveness of OKR and PEM. Comparisons on serveral benchmarks verify that our efficient detector is superior to the existing state-of-the-art (SOTA) methods.
Unrestricted Anchor Graph Based GCN for Incomplete Multi-View Clustering

In recent years, the task of multi-view clustering(MVC) has attracted more and more attention. Meanwhile, the graph convolution network(GCN) based MVC method has made consistent achievements in processing graph-structured data. However, real world data often suffers from missing some instances in each view, leading to the problem of incomplete multi-view clustering. It’s a really challenge to capture the graph structure of incomplete views for GCN to process, especially in the high missing-rate situation. To address this is-sue, this paper proposes a novel and effective graph construct method called unrestricted anchor graph(UAG). Moreover, an Unrestricted Anchor Graph based GCN framework(UAGCN) is designed for incomplete multi-view clustering. Specifically, our method employs the unrestricted anchor to reconstruct the relationship in high missing-rate data to describe the graph structure, and then integrates GCN to obtain the graph embedding of incomplete data for clustering. The experimental results on multiple data sets show that our method is superior to comparison methods.
Global Matching-Optimization Network for Stereo Depth Estimation

Recently, iterative optimization-based approaches have gained tremendous progress in the field of stereo matching. However, it still remains a challenge to accurately estimate disparity for occlusion and textureless regions. To address this challenge, we present the Global Matching-Optimization Stereo Network (GMOStereo), which contains three components: Conv-Trans Feature Extraction Module (C-TFEM), Global Matching Module (GMM), and scene-aware disparity optimization. Before iterative optimization, attention-based GMM builds stable interdependence across distinct views. The C-TFEM, which extracts features through a two-branch network based of convolution blocks and transformer blocks, is designed to obtain global representations of features while preserving fine-grained information. The scene self-similarity adopted in disparity optimization provides supplement for matching information. Finally, a Matching-Optimization loss is designed to guide the training by imposing a direct constraint on the correlation volume. Evaluation demonstrates that GMOStereo achieves superior cross-dataset generalization performance and outperforms typical methods in the foreground and challenging regions on KITTI-2015 benchmarks.
An Empirical Study of Backdoor Attacks on Masked Auto Encoders

Large-scale unlabeled data has spurred recent progress in self-supervised learning methods for learning rich visual representations. Masked autoencoders (MAE), a recently proposed self-supervised method, has exhibited exemplary performance on many vision tasks by masking and reconstructing random patches of input. However, as a representation learning method, the backdoor pitfall of MAE, and its impact on downstream tasks, have not been fully investigated. In this paper, we use several common triggers to perform backdoor attacks on the pre-training phase of MAE and test them on downstream tasks. We explore some key factors such as trigger patterns and the number of poisoned samples. Some interesting results can be obtained. The pre-training process of MAE can be used to enhance the memory of the encoder for the trigger mode. The global trigger is easier than local triggers to attack the encoder. The blended ratio and patch size of the triggers have a great impact on MAE. Code is available at https://github.com/zhuangshuli/MAE-Backdoor.
Wavsyncswap: End-To-End Portrait-Customized Audio-Driven Talking Face Generation

Audio-driven talking face with portrait customization enhances the flexibility of avatar applications for different scenarios, such as on-line meetings, mixed reality, and data generation. Among the existing methods, audio-driven talking face and face swapping are typically viewed as separate tasks that are cascaded to achieve the objective. Using state-of-the-art methods Wav2Lip and SimSwap for this purpose, we meet some issues: affected mouth synchronization, lost texture information, and slow inference speed. To resolve these issues, we propose an end-to-end model that combines the advantages of both approaches. Our approach generates highly-synchronized mouth with the aid of a pre-trained lip-sync discriminator. And identity information is provided by ArcFace and the ID injection module in the model because of its strong correlation with facial texture. Experimental results demonstrate that our method achieves lip-sync accuracy comparable to real synced videos, preserves more texture details than cascade methods, and alleviates the blurring of Wav2Lip. Also, our approach improves the inference speed. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
AdapITN: A Fast, Reliable, and Dynamic Adaptive Inverse Text Normalization

Inverse text normalization (ITN) is the task that transforms text in spoken-form into written-form. While automatic speech recognition (ASR) produces text in spoken-form, human and natural language understanding systems prefer to consume text in written-form. ITN generally deals with semiotic phrases (e.g., numbers, date, time). However, lack of studies to deal with phonetization phrases, which is ASR’s output when it handles unseen data (e.g., foreign-named entities, domain names), although these exist in the same form in the spoken-form text. The reason is that phonetization phrases are infinite patterns and language-dependent. In this study, we introduce a novel end2end model that can handle both semiotic phrases (SEP) and phonetization phrases (PHP), named AdapITN. We call it "Adap" because it allows for handling unseen PHP. The model performs only when necessary by providing a mechanism to narrow normalized regions and external query knowledge, reducing the runtime significantly.
SR-init: An Interpretable Layer Pruning Method

Despite the popularization of deep neural networks (DNNs) in many fields, it is still challenging to deploy state-of-the-art models to resource-constrained devices due to high computational overhead. Model pruning provides a feasible solution to the aforementioned challenges. However, the interpretation of existing pruning criteria is always overlooked. To counter this issue, we propose a novel layer pruning method by exploring the Stochastic Re-initialization. Our SR-init method is inspired by the discovery that the accuracy drop due to stochastic re-initialization of layer parameters differs in various layers. On the basis of this observation, we come up with a layer pruning criterion, i.e., those layers that are not sensitive to stochastic re-initialization (low accuracy drop) produce less contribution to the model and could be pruned with acceptable loss. Afterward, we experimentally verify the interpretability of SR-init via feature visualization. The visual explanation demonstrates that SR-init is theoretically feasible, thus we compare it with state-of-the-art methods to further evaluate its practicability. As for ResNet56 on CIFAR-10 and CIFAR-100, SR-init achieves a great reduction in parameters (63.98% and 37.71%) with an ignorable drop in top-1 accuracy (-0.56% and 0.8%). With ResNet50 on ImageNet, we achieve a 15.59% FLOPs reduction by removing 39.29% of the parameters, with only a drop of 0.6% in top-1 accuracy. Our code is available at https://github.com/huitang-zjut/SR-init.
Improved Acoustic-to-Articulatory Inversion Using Representations from Pretrained Self-Supervised Learning Models

In this work, we investigate the effectiveness of pretrained Self-Supervised Learning (SSL) features for learning the mapping for acoustic to articulatory inversion (AAI). Signal processing-based acoustic features such as MFCCs have been predominantly used for the AAI task with deep neural networks. With SSL features working well for various other speech tasks such as speech recognition, emotion classification, etc., we experiment with its efficacy for AAI. We train on SSL features with transformer neural networks-based AAI models of 3 different model complexities and compare its performance with MFCCs in subject-specific (SS), pooled and fine-tuned (FT) configurations with data from 10 subjects, and evaluate with correlation coefficient (CC) score on the unseen sentence test set. We find that acoustic feature reconstruction objective-based SSL features such as TERA and DeCoAR work well for AAI, with SS CCs of these SSL features reaching close to the best FT CCs of MFCC. We also find the results consistent across different model sizes.
LQGNET: Hybrid Model-Based and Data-Driven Linear Quadratic Stochastic Control

Stochastic control deals with finding an optimal control signal for a dynamical system in a setting with uncertainty, playing a key role in numerous applications. The linear quadratic Gaussian (LQG) is a widely-used setting, where the system dynamics is represented as a linear Gaussian state-space (SS) model, and the objective function is quadratic. For this setting, the optimal controller is obtained in closed form by the separation principle. However, in practice, the underlying system dynamics often cannot be faithfully captured by a fully known linear Gaussian SS model, limiting its performance. Here, we present LQGNet, a stochastic controller that leverages data to operate under partially known dynamics. LQGNet augments the state tracking module of separation-based control with a dedicated trainable algorithm. The resulting system preserves the operation of classic LQG control while learning to cope with partially known SS models without having to fully identify the dynamics. We empirically show that LQGNet outperforms classic stochastic control by overcoming mismatched SS models.
Counterfactual Two-Stage Debiasing For Video Corpus Moment Retrieval

Video Corpus Moment Retrieval aims to select a temporal video moment pertinent to a given language query from a large video corpus. Existing systems are prone to rely on a retrieval bias as a shortcut, which hinders the systems from accurately learning vision-language association. The retrieval bias is spurious correlations between query and scene. For a given query, systems tend to retrieve incorrectly correlated scenes due to biased annotations that have predominant binding in a dataset. To this end, we present a Counterfactual Two-stage Debiasing Learning (CTDL), which incorporates a counterfactual bias network that intentionally learns the retrieval bias by providing a shortcut to learn the spurious correlation between keyword and scene, and performs two-stage debiasing learning that mitigates the bias via contrasting factual retrievals with counterfactually biased retrievals. Extensive experiments show the effectiveness of CTDL paradigm.
Asymptotic Distribution of Stochastic Mirror Descent Iterates in Average Ensemble Models

The stochastic mirror descent (SMD) algorithm is a general class of training algorithms that utilizes a mirror potential to influence the implicit bias of the training algorithm and includes stochastic gradient descent (SGD) as a special case. In this paper, we explore the performance of the SMD on mean-field ensemble models and generalize earlier results obtained for SGD. The evolution of the distribution of parameters is mapped to a continuous time process in the space of probability distributions. Our main result gives a nonlinear partial differential equation (PDE) to which the continuous time process converges in the asymptotic of large networks. The impact of the mirror potential appears through a multiplicative term that is equal to the inverse of its Hessian and defines a gradient flow over an appropriate Riemannian manifold. We provide numerical simulations which allow us to study and characterize the effect of the mirror potential on the performance of networks trained with SMD for some binary classification problems.