Reviewerly/exHarmony huggingface dataset:
Wekws: A Production First Small-Footprint End-to-End Keyword Spotting Toolkit

Keyword spotting (KWS) enables speech-based user interaction and gradually becomes an indispensable component of smart devices. Recently, end-to-end (E2E) methods have be-come the most popular approach for on-device KWS tasks. However, there is still a gap between the research and deployment of E2E KWS methods. In this paper, we introduce WeKws, a production-quality, easy-to-build, and convenient-to-be-applied E2E KWS toolkit. WeKws contains the implementations of several state-of-the-art backbone networks, making it achieve highly competitive results on three publicly available datasets. To make WeKws a pure E2E toolkit, we utilize a refined max-pooling loss to make the model learn the ending position of the keyword by itself, which significantly simplifies the training pipeline and makes WeKws very efficient to be applied in real-world scenarios. The toolkit is publicly available at https://github.com/wenet-e2e/wekws.
Understandable Relu Neural Network For Signal Classification

ReLU neural networks suffer from a problem of explainability because they partition the input space into a lot of polyhedrons. This paper proposes a constrained neural network model that replaces polyhedrons by orthotopes: each hidden neuron processes only a single component of the input signal. When the number of hidden neurons is large, we show that our neural network is equivalent to a logistic regression whose input is a non-linear transformation of the processed signal. Hence, the training of our neural network always converges to a unique solution. Numerical simulations show that the loss of performance with respect to state-of-the-art methods is negligible even though our neural network is strongly constrained on robustness and explainability.
Modify: Model-Driven Face Stylization Without Style Images

Existing face stylization methods always acquire the presence of the target (style) domain during the translation process, which violates privacy regulations and limits their applicability in real-world systems. To address this issue, we propose a new method called MODel-drIven Face stYlization (MODIFY), which relies on the generative model to bypass the dependence of the target images. Briefly, MODIFY first trains a generative model in the target domain and then translates a source input to the target domain via the provided style model. To preserve the multimodal style information, MODIFY further introduces an additional remapping network, mapping a known continuous distribution into the encoder’s embedding space. During translation in the source domain, MODIFY fine-tunes the encoder module within the target style-persevering model to capture the content of the source input as precisely as possible. Our method is extremely simple and satisfies versatile training modes for face stylization. Experimental results on several different datasets validate the effectiveness of MODIFY for unsupervised face stylization. Code will be released at https://github.com/YuheD/MODIFY.
Multi-Agent Adversarial Training Using Diffusion Learning

This work focuses on adversarial learning over graphs. We propose a general adversarial training framework for multi-agent systems using diffusion learning. We analyze the convergence properties of the proposed scheme for convex optimization problems, and illustrate its enhanced robustness to adversarial attacks.
Pseudo-Inverted Bottleneck Convolution for Darts Search Space

Differentiable Architecture Search (DARTS) has attracted considerable attention as a gradient-based neural architecture search method. Since the introduction of DARTS, there has been little work done on adapting the action space based on state-of-art architecture design principles for CNNs. In this work, we aim to address this gap by incrementally augmenting the DARTS search space with micro-design changes inspired by ConvNeXt and studying the trade-off between accuracy, evaluation layer count, and computational cost. We introduce the Pseudo-Inverted Bottleneck Conv (PIBConv) block intending to reduce the computational footprint of the inverted bottleneck block proposed in ConvNeXt. Our proposed architecture is much less sensitive to evaluation layer count and outperforms a DARTS network with similar size significantly, at layer counts as small as 2. Furthermore, with less layers, not only does it achieve higher accuracy with lower computational footprint (measured in GMACs) and parameter count, GradCAM comparisons show that our network can better detect distinctive features of target objects compared to DARTS. Code is available from https://github.com/mahdihosseini/PIBConv.
Adapting Exploratory Behaviour in Active Inference for Autonomous Driving

Active inference is a probabilistic framework for modeling intelligent agent behaviours, which drives by the principle of minimizing free energy. In this paper, we integrate the imitation learning method with active inference to minimize the expected free energy under the supervision of an expert model. The proposed approach affords explainable decision-making as a combination of self-information and novelty-seeking or exploratory behavior in a hierarchical generative model. A lane-changing driving scenario is demonstrated to verify the efficiency of the proposed framework that outperforms conventional Reinforcement learning methods.
SYNTACC : Synthesizing Multi-Accent Speech By Weight Factorization

Conventional multi-speaker text-to-speech synthesis (TTS) is known to be capable of synthesizing speech for multiple voices, yet it cannot generate speech in different accents. This limitation has motivated us to develop SYNTACC (Synthesizing speech with accents) which adapts conventional multi-speaker TTS to produce multi-accent speech. Our method uses the YourTTS model and involves a novel multi-accent training mechanism. The method works by decomposing each weight matrix into a shared component and an accent-dependent component, with the former being initialized by the pretrained multi-speaker TTS model and the latter being factorized into vectors using rank-1 matrices to reduce the number of training parameters per accent. This weight factorization method proves to be effective in fine-tuning the SYNTACC on multi-accent data sets in a low-resource condition. Our SYNTACC model eventually allows speech synthesis in not only different voices but also in different accents.
Security Auditing Methodology Using Crowdsourced Testing

The term cyber is derived from French, meaning "the art of governing". Cyber security as we know is the process of securing software and electronic networks from malicious attacks. Cyber security is a collective phrase and can be divided into different common categories and domains like information security. Most importantly cyber security has become essential for maintaining confidentiality and integrity. Vulnerabilities briefly defined are the backdoors, a weakness in the developed software or IT application that can be exploited to harm our software's integrity and disclose our confidentiality. Cybersecurity deals with these vulnerabilities as they seriously threaten critical information. As for how the security software resolves such issues is that they maintain a database, and a record of vulnerabilities and threats which were previously addressed and resolved, using them as a reference we scrutinize the software thoroughly to know if there are any known vulnerabilities or any other malware of similar origin.
Decomposition, Interaction, Reconstruction Meets Global Context Learning In Visual Tracking

Tensor decomposition and reconstruction attention is a promising global context learning approach because it can remain efficient while avoiding feature compression. To exploit its potential even further in visual tracking, we redesign a 3D tensor modeling paradigm, namely tensor Decomposition, Interaction, Reconstruction attention (DIR), respectively corresponding to three function components, Tensor Decomposition Module (TDM), Tensor Interaction Module (TIM) and Context Reconstruction Module (CRM). Specifically, TDM decomposes a 3D tensor feature into rank-1 context fragments in different dimension views. The ingenuity here lies in the introduction of Circular Convolution for processing features at arbitrary scales and channel-sharing segments to enhance the interaction of the two branches in the Siamese network architecture. TIM obtains the tensor planes of each dimension by the Cross-Similarity operation of rank-1 tensors and fused cubic features, which brings more interactions between all feature dimensions. CRM reconstructs 3D context representations with the outputs of the above modules. In experiments, DIR is embedded into the tracker to verify its effectiveness.
Self-Supervised Learning-Based Source Separation for Meeting Data

Source separation can improve automatic speech recognition (ASR) under multi-party meeting scenarios by extracting single-speaker signals from overlapped speech. Despite the success of self-supervised learning models in single-channel source separation, most studies have focused on simulated setups. In this paper, seven SSL models were compared on both simulated and real-world corpora. Then, we propose to integrate the best-performing model WavLM into an automatic transcription system through a novel iterative source selection method. To improve real-world performance, time-domain unsupervised mixture invariant training was adapted to the time-frequency domain. Experiments showed that in the transcription system when source separation was inserted before an ASR model fine-tuned on separated speech, absolute reductions of 1.9% and 1.5% in concatenated minimum-permutation word error rate for an unknown number of speakers (cpWER-us) were observed on the AMI dev and test sets.
Joint Modeling for ASR Correction and Dialog State Tracking

In spoken dialog system, transcription errors in Automated Speech Recognition (ASR) impact downstream task, especially dialog state tracking (DST). Approaches to alleviate such errors involve using richer information such as word-lattices and word confusion networks. However, in some cases, this information may not be easily obtained. In addition, the large pre-trained language model is trained on plain text, leading to the gap between spoken DST and original pretrained model. In this paper, we propose a multi-task method which performs DST jointly with ASR correction to improve the performance of both tasks. To do so, we build a MultiWOZ-ASR dataset containing ASR noise in DST and mitigate the gap by utilizing a multi-task pre-training framework. Moreover, curriculum learning is adopted to alleviate the phenomenon that the correction task is difficult to converge at the initial stage of pre-training. Experimental results show that our model achieves significant improvements on DSTC2 and MultiWOZ-ASR dataset.
Domain Generalized Fundus Image Segmentation via Dual-Level Mixing

Single domain generalization plays a vital role in signal processing tasks, which is capable of extracting domain-invariant knowledge from a single source domain such that the learned model can be generalizable to unseen domains. However, many existing methods require the help of auxiliary tasks, bringing extra computation cost. Aiming at this pitfall, this study proposes Dual-Level Mixing (DLM) to boost the diversity of the single source domain and enhance the generalization performance. Specifically, at the input level, we apply different image augmentations to get different variants of the same image. Patches from augmented images are spatially mixed to get a perturbed image as the input, which enlarges the scale of training data and boosts the diversity. Meanwhile, at the feature level, we characterize feature statistics as Gaussian distributions. Then, we resample the feature statistics to renormalize the latent features, which simulates the potential style variance of different domains. In summary, the proposed DLM method synergies input-level and feature-level mixing strategies, leading to enhanced generalization performance. Experimental results of cross-domain fundus image segmentation demonstrate that either input-level mixing or feature-level mixing can effectively promote the performance of domain generalization. Moreover, the collaboration of dual-level mixing strategies leads to superior or comparable performance to domain adaptation counterparts that rely on target data.
Multiple Domain-Adversarial Ensemble Learning for Domain Generalization

Domain generalization (DG) aims to train a model on multiple source domains which can be well generalized to the unseen target domains. Currently, most DG techniques in vision tasks mainly focus on one of the three research lines (i.e., data manipulation, representation learning, and learning strategy). The direction of combining multiple DG research lines still remains to be studied and explored. In this paper, we propose a unified framework for DG that combines multiple research lines to enhance the generalization ability. To be specific, we combine representative strategies of three research lines: ensemble learning, domain alignment, and data augmentation. A basic framework is built with an ensemble learning strategy to train an expert model in a domain unit. Besides, the ability to learn domain-independent features is enhanced with an adversarial-based domain alignment strategy. Furthermore, a domain transformation network (DoTNet) is introduced, which is expected to generate additional deviated images and further enhance the generalization capability. Extensive experiments on DG datasets demonstrate the effectiveness of our approach.
Virtual Dressing Room Application Using GANs

In recent years, there has been a lot of interest in the idea of a virtual clothing try-on, which entails placing an image of a desired outfit on a mannequin. In order to give the user the idea that they are trying on the item, the standard try-on task attempts to provide a natural fit between the target clothing and the provided body. However, in real life, people may also care about how they "try on" in various stances. We offer a novel try- on setting that enables modification of both the garment and the model's posture in order to address this problem. To this purpose, we present a generative adversarial networks–based pose- guided virtual try-on technique (GANs). By initially predicting how the semantic layout of the reference image will change throughout the try- on process, it is possible to create a photorealistic try-on with a plethora of garment details. There are three individual parts to this. Initially, a reference image's semantic segmentation is used by a semantic layout generation module to progressively forecast the tried-on garment's desired semantic layout. A garments warping module then applies a second-order difference constraint to the created semantic layout in order to change clothing pictures while preserving the transformation's consistency over the course of training. Third, a module that can adapt to produce each semantic component of the human body from a variety of inputs (such as reference image, semantic layout, and distorted clothing). It outperforms cutting-edge generative models in the image-based virtual try-on task, according to our tests on the newly acquired dataset.
Advancing the Dimensionality Reduction of Speaker Embeddings for Speaker Diarisation: Disentangling Noise and Informing Speech Activity

The objective of this work is to train noise-robust speaker embeddings adapted for speaker diarisation. Speaker embeddings play a crucial role in the performance of diarisation systems, but they often capture spurious information such as noise, adversely affecting performance. Our previous work has proposed an auto-encoder-based dimensionality reduction module to help remove the redundant information. However, they do not explicitly separate such information and have also been found to be sensitive to hyper-parameter values. To this end, we propose two contributions to overcome these issues: (i) a novel dimensionality reduction framework that can disentangle spurious information from the speaker embeddings; (ii) the use of speech activity vector to prevent the speaker code from representing the background noise. Through a range of experiments conducted on four datasets, our approach consistently demonstrates the state-of-the-art performance among models without system fusion.
Accelerating Matrix Trace Estimation by Aitken’s Δ<sup>2</sup> Process

We present an algorithm to estimate the trace of symmetric matrices that are available only via Matrix-Vector multiplication. The proposed algorithm constructs a series of trace estimates by applying the probing technique with an increasing number of vectors. These estimates are then treated as a converging sequence whose limit is the sought matrix trace, and we apply Aitken’s Δ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> process to accelerate its convergence to the trace limit. Numerical experiments performed on covariance matrices demonstrate the competitiveness of the proposed scheme versus probing and randomized trace estimators.
Code-Enhanced Fine-Grained Semantic Matching For Tag Recommendation In Software Information Sites

Tag recommendation in software information sites is a significant task to help developers make distinctions among software objects. Most existing methods usually ignore the semantic information of code snippets in software information sites. To tackle this issue, we regard the code as a semantic enhancement signal, and propose a novel Code-Enhanced fine-grained semantic matching method for Tag Recommendation in software information sites (CETR) to learn the matching score between tags and software objects. In our CETR, code-enhanced semantic interaction is designed to capture fine-grained semantic relevance between tags and software objects. Specifically, code snippets and text descriptions in software objects are first encoded in two specialized ways to capture semantic information from the perspective of natural language and structural language respectively. The semantic interaction is then learned between code snippets and text descriptions. Besides, a hierarchy-aware semantic learning block attentively combines the features from different layers as the final feature to capture semantic information of different levels. Lastly, a pairwise sentence matching computation block is exploited to predict matching probability. Experimental results on four software information site datasets have demonstrated the effectiveness of our proposed CETR for tag recommendation compared with the state-of-the-art methods.
Ultimate Negative Sampling for Contrastive Learning

Unsupervised learning has received more attention due to the superior performance of contrastive learning methods. Most contrastive methods use data augmentation techniques to construct positive and negative pairs. The augmented view of the same sample is regarded as a positive sample, while the rest are negative samples. This negative sampling strategy has strong randomness and ignores samples that are semantically similar to anchors, namely sampling bias. This problem has been addressed by weighting the similarity of negative samples. In this paper, we propose a novel ultimate negative sampling for contrastive learning. Unlike random sampling, we set a more extreme negative sample selection mechanism based on the ideal representation of the sample. Furthermore, we constrain the consistency between samples across the space. Experiment results demonstrate the proposed method’s superiority on multiple benchmark datasets.
A Two-Branch Network for Video Anomaly Detection with Spatio-Temporal Feature Learning

Video anomaly detection is very challenging, as most anomalies are rare and inconclusive. Previous weakly supervised learning approaches utilize the classifier trained with video-level labels to locate anomalous clips from the video. However, the anomalous clips often contain both anomalies and numerous irrelevant background behaviors, increasing the difficulty of localization. In this work, we propose a two-branch network to obtain the global and each local object’s action information of the clip respectively, where the local objects are extracted by a pre-trained object detector. This local-cum-global perception highlights the anomalous features from the background noise. We further propose a spatio-temporal relationship network, which is based on the attention mechanism to model the spatial relations of different objects and the temporal correlations among different clips to efficiently capture the spatio-temporal distribution of anomalies in the video. Extensive experiments on two benchmarks show that our method achieves significant performance gains.
Memory-Augmented Contrastive Learning for Talking Head Generation

Given one reference facial image and a piece of speech as input, talking head generation aims to synthesize a realistic-looking talking head video. However, generating a lip-synchronized video with natural head movements is challenging. The same speech clip can generate multiple possible lip and head movements, that is, there is no one-to-one mapping relationship between them. To overcome this problem, we propose a Speech Feature Extractor (SFE) based on memory-augmented self-supervised contrastive learning, which introduces the memory module to store multiple different speech mapping results. In addition, we introduce the Mixed Density Networks (MDN) into the landmark regression task to generate multiple predicted facial landmarks. Extensive qualitative and quantitative experiments show that the quality of our facial animation is significantly superior to that of the state-of-the-art (SOTA). The code has been released at https://github.com/Yaxinzhao97/MACL.git.
Real-Time Human Reconstruction Based on Human Pose Prior and Epipolar Refinement

Current vision-based human-robot collaboration systems use a human skeleton model instead of human physical boundary for collision-free trajectory planning, which could lead to high security risk in applications. In this paper, we propose a real-time human performance capture framework to generate high-quality surface geometry from multi-view RGB inputs. To achieve run-time efficiency and robust reconstruction, we decompose the human pose prior to human keypoint detection, joint radius estimation as well as human segmentation and fuse them into our multi-task model YOLO-DKS to estimate parameters for coarse reconstruction. After that, we exploit the coarse-to-fine reconstruction paradigm, starting with coarsely reconstructing human physical boundary and following by refinement using masks and RGB images from different viewpoints to obtain accurate surface topology. Experiments demonstrate the effectiveness of our approach in achieving high-quality geometry and efficient computation in comparison with state-of-the-art methods.
Artificial Synthesis of Single Person Videos through Motion Transfer using Cycle Generative Adversarial Networks and Machine Learning

In this work, we address the problem of motion transfer from the source video of one person to another target person’s video. This basically involves human pose synthesis, whereby the source person’s poses in a video are superimposed frame-by-frame onto the target. This results into an artificially generated video of the target, where his/her movement is determined by the given source video. In this work, we use global and local edge detectors as well as pose or skeletal keypoints ex- tractors for synthesizing an artificial human dance video. In this paper, we explore two different Generative Adversarial Network (GAN) architectures for pose detection and superimposition from source to target object. We present experiments and results to validate the proposed models’ outputs, as well as compare their performances.