Reviewerly/exHarmony huggingface dataset:
Robust Video Hashing Based on Local Fluctuation Preserving for Tracking Deep Fake Videos

With the rapid development of deepfake techniques, massive face manipulation videos appeared on social networks. These deepfake videos not only violated the original video of the author’s privacy, but also seriously threatened the security of the video database. Robust video hashing can map videos with similar visual content into similar hash codes, which is beneficial for tracking fake video material in social networks. In this paper, a robust video hashing algorithm based on local fluctuation preserving is proposed. The algorithm uses a shot segmentation model and local statistical descriptors, which is robust to many commonly-used digital operations and can accurately track the original version of these fake videos from a huge video database. An essential contribution is a shot segmentation model reconstruction from input video with image hashing and discrete wavelet transform, reaching initial data compression and against noise attack. In addition, as local statistical descriptors are content-based and local preserving features, the hash generated by local statistical descriptors can achieve good discrimination and ensure that the proposed hash has good tracking ability to fake videos from original videos.
A Model for Assessor Bias in Automatic Pronunciation Assessment

In pronunciation assessment, the assessor’s perception is influenced by a particular pronunciation template. This assessor may hold a bias towards certain variations in pronunciation which do not necessarily impact communication, yet they may be penalized during the assessment. This work proposes a model for pronunciation assessment as the combination of an assessor independent (A) and an assessor specific (B) component. The latter could be interpreted as the assessor bias. The resulting assessment function was implemented as a dual model trained to detect mispronounced speech segments. The models incorporate Long-Short Memory and saliency region selection using attention. An experiment was performed using recordings from young Dutch learners of English as second language, which were annotated for mispronunciation by three trained phoneticians (a1, a2, a3). The models combined were able to detect mispronunciations given the assessor identity achieving F1 scores of 0.77, 0.68 and 0.86 for a1, a2, a3 respectively on the Train set and 0.66, 0.53 and 0.81 on the Test set. Additionally, the attention weights of the B model were able to illustrate disagreements between assessors related to the bias.
Tempo: Improving Training Performance in Cross-Silo Federated Learning

Different from its commonly studied scenario to centrally store clients’ data in institutions, which implicitly neglects clients’ data privacy, we study cross-silo federated learning in a preferable setting to keep private data on clients, and train the global model with a three-layer structure, where the institutions aggregate model updates from their clients for several rounds before sending their aggregated updates to the central server. In this context, we mathematically prove that the number of clients’ local training epochs affects the global model performance and thus propose a new approach, Tempo, to adaptively tune the epoch number of each client through training. The results of our evaluation conducted under real network environments show that Tempo can not only improve training performance in terms of global model accuracy and communication efficiency, but also the elapsed training time.
HIRL: Hybrid Image Restoration Based on Hierarchical Deep Reinforcement Learning via Two-Step Analysis

The restoration of hybrid distorted images in real-world scenarios is still a difficult problem due to the fact that the degrading types and degrees are always unknown. Previous studies typically utilize multiple recovery tools to restore images. However, each tool adopted inevitably introduces additional noise and will affect the subsequent recovery results. To address this issue, in this paper, we propose a hierarchical deep reinforcement learning framework (HIRL), which balance both benefits and noises brought by each tool and select the appropriate type and degree tools. The proposed method endeavors to find a long-term optimal tool sequence, which is better than the greedy strategy that employs the tools with the largest short-term returns. Meanwhile, it benefits from a hierarchical design to reduce time consumption and complexity compared to a brute force strategy. Experiments demonstrate the superiority of the proposed method over other state-of-the-art methods. Furthermore, our framework is highly scalable and can be easily extended to the other recovery pipelines.
Wassertrain: An Adversarial Training Framework Against Wasserstein Adversarial Attacks

This paper presents an adversarial training framework WasserTrain for improving model robustness against the adversarial attacks in terms of the Wasserstein distance. First, an effective attack method WasserAttack is introduced with a novel encoding of the optimization problem, which directly finds the worst point within the Wasserstein ball while keeping the relaxation error of the Wasserstein transformation as small as possible. The proposed adversarial training frame-work utilizes these high-quality adversarial examples to train robust models. Experiments on MNIST show that the adversarial loss arising from adversarial examples found by our method is about three times as much as that found by the PGD-based attack method. Furthermore, within the Wasserstein ball with a radius of 0.5, the WasserTrain model achieves 31% adversarial robustness against WasserAttack, which is 22% higher than that on the PGD-based training model.
Run-and-Back Stitch Search: Novel Block Synchronous Decoding For Streaming Encoder-Decoder ASR

A streaming style inference of encoder–decoder automatic speech recognition (ASR) systems is important for reducing latency, which is essential for interactive use cases. To this end, we propose a novel blockwise synchronous decoding algorithm with a hybrid approach that combines endpoint prediction and endpoint post-determination. In the endpoint prediction, we compute the expectation of the number of tokens that are yet to be emitted in the encoder features of the current blocks using the CTC posterior. Based on the expectation value, the decoder predicts the endpoint to realize continuous block synchronization, as a running stitch. Meanwhile, end-point post-determination probabilistically detects backward jump of the source–target attention, which is caused by the misprediction of endpoints. Then it resumes decoding by discarding those hypotheses, as back stitch. We combine these methods into a hybrid approach, namely run-and-back stitch search, which reduces the computational cost and latency. Evaluations of various ASR tasks show the efficiency of our proposed decoding algorithm, which achieves a latency reduction, for instance in the Librispeech test set from 1487 ms to 821 ms at the 90th percentile, while maintaining a high recognition accuracy.
Quantized Winograd Acceleration for CONV1D Equipped ASR Models on Mobile Devices

The intensive computation of Automatic Speech Recognition (ASR) models obstructs them from being deployed on mobile devices. In this work, we present a novel quantized Wino-grad optimization framework, combining quantization and fast convolution to achieve efficient inference acceleration for ASR models on mobile devices. To avoid the information loss due to the combination of quantization and Winograd convolution, a Range-Scaled Quantization (RSQ) training method is proposed, integrating integer-range scaling and quantization noise minimization. Moreover, the Conv1D equipped DFSMN (ConvDFSMN) model is designed for mobile applications and experimental verification. We conduct extensive experiments on ConvDFSMN and Wav2letter models, demonstrating that the models can be effectively optimized with the proposed optimization framework. Especially, the optimized Wav2letter model achieves 1.48× speedup for end-to-end inference and 1.92× speedup for model backbone inference on ARMv7-based mobile devices, with only an approximate 0.07% decrease in WER on AIShell-1.
Synthesis of Adversarial Samples in Two-Stage Classifiers

Adversarial attacks can drastically reduce the accuracy and confidence level of classifiers while being imperceptible. Existing studies on the topic have largely focused on one-stage classifiers. In this paper, we study the robustness of two Two-Stage Hierarchical Classifier models, the flat and top-down hierarchical classifiers, termed FHC and TDHC respectively, to targeted and confidence reduction attacks. We formulate feasibility programs based on similarity and distance measures for the one-shot synthesis of adversarial examples, and devise a generative approach to the solution. In this approach, the adjustable parameters of a generative network are iteratively updated by optimizing loss functions for the dual objective of (i) low attack perceptibility and (ii) small distance from desired soft predictions. We demonstrate the performance of the proposed approach in terms of imperceptibilty and measures of attack success, and show it compares favorably with state-of-the-art techniques.
Point Cloud Attribute Compression Via Chroma Subsampling

We introduce chroma subsampling for 3D point cloud attribute compression by proposing a novel technique to sample points irregularly placed in 3D space. While most current video compression standards use chroma subsampling, these chroma subsampling methods cannot be directly applied to 3D point clouds, given their irregularity and sparsity. In this work, we develop a framework to incorporate chroma subsampling into geometry-based point cloud encoders, such as region adaptive hierarchical transform (RAHT) and region adaptive graph Fourier transform (RAGFT). We propose different sampling patterns on a regular 3D grid to sample the points at different rates. We use a simple graph-based nearest neighbor interpolation technique to reconstruct the full resolution point cloud at the decoder end. Experimental results demonstrate that our proposed method provides significant coding gains with negligible impact on the reconstruction quality. For some sequences, we observe a bitrate reduction of 10-15% under the Bjontegaard metric. More generally, perceptual masking makes it possible to achieve larger bitrate reductions without visible changes in quality.
Investigating Robustness of Biological vs. Backprop Based Learning

Robustness of learning algorithms remains an important problem to be solved from both the perspective of adversarial attacks and improving generalization. In this work, we investigate the robustness of biologically inspired Hebbian learning algorithm in depth. We find that Hebbian learning based algorithms outperform conventional learning algorithms like CNNs by a huge margin of upto 18% on the CIFAR-10 dataset under the addition of noise. We highlight that an important reason for this is the underlying representations that are being learnt by the learning algorithms. Specifically, we find that the Hebbian method learns the most robust representations compared to other methods that helps it to generalize better. We also conduct ablations on the Hebbian network and showcase that robustness of the model drops by upto 16% on the CIFAR-10 dataset if the representation capacity of the network is deteriorated. Hence, we find that the representations learnt play an important role in the resultant robustness of the models. We conduct experiments on multiple datasets and show that the results hold on all the datasets and at various noise levels.
Tracking the Dimensions of Latent Spaces of Gaussian Process Latent Variable Models

Determining the number of latent variables, or the dimensions of latent states, is a ubiquitous problem in dimension reduction. In this paper, we introduce a novel sequential method that relies on the Bayesian approach to estimate the dimension of a latent space of a Gaussian process latent variable model. The proposed method also considers settings where the number of latent variables varies with time. To evaluate our methodology, we compared the estimated dimensions with the true dimensions as they vary with time. Results on synthetic data demonstrate that our method has a very good performance.
ER-PIQA: A Task-Guided Pedestrian Image Quality Assessment Via Embedding Reconstruction

Image quality is an important factor for pedestrian recognition systems. Pedestrian image quality assessment aims at evaluating images in order to provide more reliable and stable images for the following analysis process. Previous work proposed supervised solutions that require artificially or manually labelled quality values. However, due to the lack of a clear quality definition and the variety of recognition tasks, there are still some difficulties in subjective labeling methods. In this paper, a novel task-guided method is proposed to measure pedestrain image quality based on embedding reconstruction without the involvement of subjective labels. Considering the various attention area in different tasks, the pedestrian image quality can be estimated by comparing the similarity between the semantics representation and tasks embedding after reconstruction. Experiments on both person re-identification and pedestrian attribute recognition show advantages of the proposed ER-PIQA. Meanwhile, our approach can be integrated into current recognition systems and adaptively modified for other tasks beyond pedestrian recognition.
Federated Stochastic Gradient Descent Begets Self-Induced Momentum

Federated learning (FL) is an emerging machine learning method that can be applied in mobile edge systems, in which a server and a host of clients collaboratively train a statistical model utilizing the data and computation resources of the clients without directly exposing their privacy-sensitive data. We show that running stochastic gradient descent (SGD) in such a setting can be viewed as adding a momentum-like term to the global aggregation process. Based on this finding, we further analyze the convergence rate of a federated learning system by accounting for the effects of parameter staleness and communication resources. These results advance the understanding of the Federated SGD algorithm, and also forges a link between staleness analysis and federated computing systems, which can be useful for systems designers.
Accurate Inference of Unseen Combinations of Multiple Rootcauses with Classifier Ensemble

Root cause analysis (RCA) of network faults is crucial to wireless network operation and management. It, however, is challenging, due to diverse feature types, diverse lengths of time slices, simultaneous occurrences of multiple root causes, and lack of training samples. In this paper, we present our solutions for these problems in ICASSP-SPGC-2022 AIOps Challenge in Communication Networks. We first design specific feature engineering method to represent the provided spatial features. Secondly, we conduct time series analysis on training data and propose an efficient method to infer whether a sample includes multiple root causes. Thirdly, in order to solve the lack of training data for unseen combinations of multiple root causes, we propose to ensemble multiple single-root-cause classifiers. Fourthly, we introduce TextCNN into multivariate time series classification to obtain high accuracy. Our approach achieved score 0.93 and ranked 4th place on the leader board. Code is available at https://github.com/zxuan000/SPGC_aiops_bjtu
Integrating Pretrained Language Model for Dialogue Policy Evaluation

Reinforcement Learning (RL) has been witnessed its potential for training a dialogue policy agent towards maximizing the accumulated rewards given from users. However, the reward can be very sparse for it is usually only provided at the end of a dialog session, which causes unaffordable interaction requirements for an acceptable dialog agent. Distinguished from many efforts dedicated to optimizing the policy and recovering the reward alternatively which suffers from easily getting stuck in local optima and model collapse, we decompose the adversarial training into two steps: 1) we integrate a pre-trained language model as a discriminator to judge whether the current system action is good enough for the last user action (i.e., next action prediction); 2) the discriminator gives and extra local dense reward to guide the agent’s exploration. The experimental result demonstrates that our method significantly improves the complete rate (4.4%) and success rate ( 8.0%) of the dialogue system.
Stacked Multi-Scale Attention Network for Image Colorization

Deep convolutional networks (CNNs) show their potential in image colorization for producing plausible results. Recently, the attention mechanism further boosts the performances of CNNs by constructing channel and spatial interactions. However, existing attention methods are performed in a single-scale manner, which is hard to capture multi-scale information interactions in a limited computational cost. This can limit the performance of the network to reconstruct color channels. In this paper, we propose a stacked multi-scale attention network (SMSANet) for image colorization. The core idea is to perform the attentions of a feature map in a multi-scale manner so that sufficient interactions are conducted to efficiently and adaptively capture multi-scale and long-range dependencies. By stacking the multi-scale attention layers in different convolutional layers, the SMSANet can focus on more discriminative features to reconstruct color channels. Moreover, a salient loss is designed to further refine the generated image at both pixel-level and object-level. Extensive experiments on the ImageNet dataset have demonstrated that SMSANet outperforms the state-of-the-art automatic colorization methods.
Inferring Camera Intrinsics Based on Surfaces of Revolution: A Single Image Geometric Network Approach for Camera Calibration

Camera calibration is a necessary prerequisite in many applications of robotics, especially in robot vision in order to obtain metric reconstruction from a 2D image. In this paper, we address the problem of calibrating from a single image of a surface of revolution (SOR) based on deep learning, in order to determine the camera intrinsic parameters. Geometric constraints based on the symmetry properties of the SOR structure are deployed to our proposed learning-based camera calibration framework. To enable the calibration from a single view, we also propose a learning-based conics detection model fitting the geometric primitive of a cylinder. The calibration from a single view can be completed by minimizing the geometric constraints of two conics detected by the learning-based model with cylinder images as input. Objects with a surface of revolution are commonly visible in daily life, such as cans, bottles, and bowls, making this research both significant and practical. Finally, traditional calibration techniques are compared against our single image calibration. Experiments conducted on newly generated dataset demonstrate the effectiveness and robustness of the proposed method.
Self Supervised Representation Learning with Deep Clustering for Acoustic Unit Discovery from Raw Speech

The automatic discovery of acoustic sub-word units from raw speech, without any text or labels, is a growing field of research. The key challenge is to derive representations of speech that can be categorized into a small number of phoneme-like units which are speaker invariant and can broadly capture the content variability of speech. In this work, we propose a novel neural network paradigm that uses the deep clustering loss along with the autoregressive contrastive predictive coding (CPC) loss. Both the loss functions, the CPC and the clustering loss, are self-supervised. The clustering cost involves the loss function using the phoneme-like labels generated with an iterative k-means algorithm. The inclusion of this loss ensures that the model representations can be categorized into a small number of automatic speech units. We experiment with several sub-tasks described as part of the Zerospeech 2021 challenge to illustrate the effectiveness of the framework. In these experiments, we show that proposed representation learning approach improves significantly over the previous self-supervision based models as well as the wav2vec family of models on a range of word-level similarity tasks and language modeling tasks.
W-ART: Action Relation Transformer for Weakly-Supervised Temporal Action Localization

Weakly-supervised temporal action localization (WTAL) is a long-standing and challenging research problem in video signal analysis. It is to localize the action segments in the video given only video-level labels. The key to this task is understanding how the diverse actions interact. In this paper, we propose W-ART, a relation Transformer to explicitly capture the relationships between action segments. We devise a new effective Transformer architecture and construct new training loss functions for WTAL. Further, we propose a dedicated query mechanism to satisfy the different feature preferences between classification and localization. Thanks to these designs, our W-ART can accurately localize the diverse actions even in weakly-supervised setting. Extensive evaluation and empirical analysis show that our method outperforms the state of the arts on two challenging benchmarks, Charades and THUMOS14.
Subspace Clustering Using Unsupervised Data Augmentation

Subspace clustering is an unsupervised approach for determining the union of multiple subspaces that best fits a collection of high-dimensional samples. Self-expressive representation, that is, expressing samples as linear combination of other samples, is the core of most state-of-the-art subspace clustering approaches. However, existence of sufficiently well-spread samples within each subspace is crucial for precise representation, which might not always be available in real-world scenarios. Inspired by the remarkable influence of data augmentation on the performance of neural networks, we propose a scalable approach that employs data augmentation within subspace clustering. Benefiting from the increased diversity in data, we use augmented samples as an enlarged dictionary and combine the self-expressive representations based on the assumption that augmentation does not alter the labels of the samples. Significant improvement of the clustering performance on two real-world datasets demonstrates the effectiveness of the proposed approach.
Part-of-Speech Models Compression Methods for on-Device Grapheme-to-Phoneme Conversion

The paper investigates methods of compressing part-of-speech models that are developed for an on-device grapheme-to-phoneme conversion module. The performance of part-of-speech models is analyzed under different compression regimes. The evaluation is done with respect to French, German and Italian datasets that consist of TTS input prompts. The study shows that a proper selection of a compression method reduces the model size significantly without deteriorating the grapheme-to-phoneme conversion performance.