Reviewerly/exHarmony huggingface dataset:
Backdoor Defense via Suppressing Model Shortcuts

Recent studies have demonstrated that deep neural networks (DNNs) are vulnerable to backdoor attacks during the training process. Specifically, the adversaries intend to embed hidden backdoors in DNNs so that malicious model predictions can be activated through pre-defined trigger patterns. In this paper, we explore the backdoor mechanism from the angle of the model structure. We select the skip connection for discussions, inspired by the understanding that it helps the learning of model ‘shortcuts’ where backdoor triggers are usually easier to be learned. Specifically, we demonstrate that the attack success rate (ASR) decreases significantly when reducing the outputs of some key skip connections. Based on this observation, we design a simple yet effective backdoor removal method by suppressing the skip connections in critical layers selected by our method. We also implement fine-tuning on these layers to recover high benign accuracy and to further reduce ASR. Extensive experiments on benchmark datasets verify the effectiveness of our method.
Efficient Data Loading with Quantum Autoencoder

Faithfully loading classical data into a quantum system is a core problem in quantum machine learning and various quantum information processing tasks. In this work, we propose an efficient quantum autoencoder architecture that can construct a quantum state approximating the unknown classical distribution with high precision and with only linear circuit depth. Simulation experiments show that our proposed method substantially outperforms state-of-the-art methods on a wide range of datasets by evaluating divergences between the loaded distributions and the target distribution, and it also enjoys a faster convergence rate and stability. Moreover, the proposed scheme can be efficiently implemented on near-term hybrid classical-quantum systems with very shallow circuit depths.
LiNuIQA: Lightweight No-Reference Image Quality Assessment Based on Non-Uniform Weighting

No-Reference Image Quality Assessment (NR-IQA) techniques have shown improved performance with the help of deep-learning but lightweight architectures have not received attention. In this paper, we propose an NR-IQA network named Lightweight Non-uniform Weighting-based NR-IQA (LiNuIQA) that adopts an efficient network as a feature extractor for a resource constraint environment and harnesses non-uniformly self-weighted local (from each patch) and global information (from all patches) to overcome the inherent problem of low performance stemming from use of lightweight feature extractor. This non-uniform weighting technique is designed to utilize combinations of local and global information with very low resources unlike conventional weighting techniques. The experimental results show that our network outperforms several recently popular NR-IQA networks in terms of both PLCC and SRCC while having the smallest number of parameters and multiply-adds (MAdd) operations. In addition, it can be seen from our experiments that appropriate weighting method plays an important role in IQA and can be implemented with extremely low resources.
Classification via Subspace Learning Machine (SLM): Methodology and Performance Evaluation

Inspired by the decision learning process of multilayer per-ceptron (MLP) and decision tree (DT), a new classification model, named the subspace learning machine (SLM), is proposed in this work. SLM first identifies a discriminant subspace, S <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sup> , by examining the discriminant power of each input feature. Then, it learns projections of features in S <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sup> to yield 1D subspaces and finds the optimal partition for each. A criterion is developed to choose the best q partitions that yield 2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">q</sup> partitioned subspaces. The partitioning process is recursively applied at each child node to build an SLM tree. When the samples at a child node are sufficiently pure, the partitioning process stops, and each leaf node makes a prediction. The ensembles of SLM trees can yield a stronger predictor. Extensive experiments are conducted for performance benchmarking among SLM trees, ensembles and classical classifiers.
Matrix Low-Rank Approximation for Policy Gradient Methods

Estimating a policy that maps states to actions is a central problem in reinforcement learning. Traditionally, policies are inferred from the so called value functions (VFs), but exact VF computation suffers from the curse of dimensionality. Policy gradient (PG) methods bypass this by learning directly a parametric stochastic policy. Typically, the parameters of the policy are estimated using neural networks (NNs) tuned via stochastic gradient descent. However, finding adequate NN architectures can be challenging, and convergence issues are common as well. In this paper, we put forth low-rank matrix-based models to estimate efficiently the parameters of PG algorithms. We collect the parameters of the stochastic policy into a matrix, and then, we leverage matrix-completion techniques to promote (enforce) low rank. We demonstrate via numerical studies how low-rank matrix-based policy models reduce the computational and sample complexities relative to NN models, while achieving a similar aggregated reward.
MADI: Inter-Domain Matching and Intra-Domain Discrimination for Cross-Domain Speech Recognition

End-to-end automatic speech recognition (ASR) usually suffers from performance degradation when applied to a new domain due to domain shift. Unsupervised domain adaptation (UDA) aims to improve the performance on the unlabeled target domain by transferring knowledge from the source to the target domain. To improve transferability, existing UDA approaches mainly focus on matching the distributions of the source and target domains globally and/or locally, while ignoring the model discriminability. In this paper, we propose a novel UDA approach for ASR via inter-domain MAtching and intra-domain DIscrimination (MADI), which improves the model transferability by fine-grained inter-domain matching and discriminability by intra-domain contrastive discrimination simultaneously. Evaluations on the Libri-Adapt dataset demonstrate the effectiveness of our approach. MADI reduces the relative word error rate (WER) on cross-device and cross-environment ASR by 17.7% and 22.8%, respectively.
Image Adversarial Steganography Based on Joint Distortion

Image steganography is the technique of concealing secret messages into digital images without arousing suspicion from detectors. Recently, adversarial steganography has received much attention from the research community, since it is effective in deceiving target deep-learning-based steganalysis (DLS) and designing more secure embedding distortion. However, how to combine adversarial steganography with handcrafted adjustment strategies to design adversarial steganography based on joint distortion has not been discussed yet. In this paper, incorporating adversarial steganography and joint distortion assignment, we present a novel adversarial steganographic scheme named JAS (Joint Adversarial Steganography). We compute joint distortion and adjust it based on joint gradient, which is a vector consisting of the gradients of two adjacent pixels, until the resulting stego image could deceive the target DLS. Furthermore, by combining JAS with synchronizing modification directions profile, we enhance the steganography security more desirably. Experiments demonstrate that the proposed method effectively enhances the anti-detection ability of joint distortion steganography.
Distributionally Robust Multiclass Classification and Applications in Deep Image Classifiers

We develop a Distributionally Robust Optimization (DRO) formulation for Multiclass Logistic Regression (MLR), which could tolerate data contaminated by outliers. The DRO framework uses a probabilistic ambiguity set defined as a ball of distributions that are close to the empirical distribution of the training set in the sense of the Wasserstein metric. We relax the DRO formulation into a regularized learning problem whose regularizer is a norm of the coefficient matrix. We establish out-of-sample performance guarantees for the solutions to our model, offering insights on the role of the regularizer in controlling the prediction error. We apply the proposed method in rendering deep Vision Transformer (ViT)-based [1] image classifiers robust to random and adversarial attacks. Specifically, using the MNIST and CIFAR-10 datasets, we demonstrate reductions in test error rate by up to 83.5% and loss by up to 91.3% compared with baseline methods, by adopting a novel random training method.
A Magnetic Framelet-Based Convolutional Neural Network for Directed Graphs

Spectral Graph Convolutional Networks (spectral GCNNs), a powerful tool for analyzing and processing graph data, typically apply frequency filtering via Fourier transform to obtain representations with selective information. Although research shows that spectral GCNNs can be enhanced by framelet-based filtering, the massive majority of such research only considers undirected graphs. In this paper, we introduce Framelet-MagNet, a magnetic framelet-based spectral GCNN for directed graphs (digraphs). The model applies the framelet transform to digraph signals to form a more sophisticated representation for filtering. Digraph framelets are constructed with the complex-valued magnetic Laplacian, simultaneously leading to signal processing in both real and complex domains. We empirically validate the predictive power of Framelet-MagNet over a range of state-of-the-art models in node classification, link prediction, and denoising.
Space-Time Graph Neural Networks with Stochastic Graph Perturbations

Space-time graph neural networks (ST-GNNs) are recently developed architectures that learn efficient graph representations of time-varying data. ST-GNNs are particularly useful in multi-agent systems, due to their stability properties and their ability to respect communication delays between the agents. In this paper we revisit the stability properties of ST-GNNs and prove that they are stable to stochastic graph perturbations. Our analysis suggests that ST-GNNs are suitable for transfer learning on time-varying graphs and enables the design of generalized convolutional architectures that jointly process time-varying graphs and time-varying signals. Numerical experiments on decentralized control systems validate our theoretical results and showcase the benefits of traditional and generalized ST-GNN architectures.
RD-NAS: Enhancing One-Shot Supernet Ranking Ability Via Ranking Distillation From Zero-Cost Proxies

Neural architecture search (NAS) has made tremendous progress in the automatic design of effective neural network structures but suffers from a heavy computational burden. One-shot NAS significantly alleviates the burden through weight sharing and improves computational efficiency. Zero-shot NAS further reduces the cost by predicting the performance of the network from its initial state, which conducts no training. Both methods aim to distinguish between "good" and "bad" architectures, i.e., ranking consistency of predicted and true performance. In this paper, we propose Ranking Distillation one-shot NAS (RD-NAS) to enhance ranking consistency, which utilizes zero-cost proxies as the cheap teacher and adopts the margin ranking loss to distill the ranking knowledge. Specifically, we propose a margin subnet sampler to distill the ranking knowledge from zero-shot NAS to one-shot NAS by introducing Group distance as margin. Our evaluation of the NAS-Bench-201 and ResNet-based search space demonstrates that RD-NAS achieve 10.7% and 9.65% improvements in ranking ability, respectively. Our codes are available at https://github.com/pprp/CVPR2022-NAS-competition-Track1-3th-solution
Joint Multi-Level Feature Network for Lightweight Person Re-Identification

Learning fine-grained features is crucial to the performance improvement of person re-identification (Re-ID). Although existing methods have made significant progress, utilizing multi-level information to obtain fine-grained features has not been explored in this field. To alleviate this issue, we propose a lightweight person Re-ID method named Joint Multi-Level Feature Network (JMLFNet) to obtain robust feature representation for the Re-ID task. Specifically, we design a Multi-Attention Block (MAB) and embed it into the lightweight backbone network to improve performance, which can make the network focus on the key parts of pedestrian images. Meanwhile, we propose a Multi-Level Feature Extraction (MLFE) method to extract multi-granularity features of high-level semantic information and low-level detail information, which can effectively capture the feature diversity of pedestrian images. Furthermore, we design a Feature Fusion Block (FFB), which is fused the fine-grained features of high-level and low-level information to better obtain the discriminative feature representation of pedestrian images. Extensive experiments conducted on popular datasets Market1501 and DukeMTMC-reID demonstrate that the proposed JMLFNet has competitive performance compared with the state-of-the-art methods.
Sparse Mixture Once-for-all Adversarial Training for Efficient in-situ Trade-off between Accuracy and Robustness of DNNs

Existing deep neural networks (DNNs) that achieve state-of-the-art (SOTA) performance on both clean and adversarially-perturbed images rely on either activation or weight conditioned convolution operations. However, such conditional learning costs additional multiply-accumulate (MAC) or addition operations, increasing inference memory and compute costs. To that end, we present a sparse mixture once for all adversarial training (SMART), that allows a model to train once and then in-situ trade-off between accuracy and robustness, that too at a reduced compute and parameter overhead. In particular, SMART develops two expert paths, for clean and adversarial images, respectively, that are then conditionally trained via respective dedicated sets of binary sparsity masks. Extensive evaluations on multiple image classification datasets across different models show SMART to have up to 2.72× fewer non-zero parameters costing proportional reduction in compute overhead, while yielding SOTA accuracy-robustness trade-off. Additionally, we present insightful observations in designing sparse masks to successfully condition on both clean and perturbed images.
Higher-Order Spatio-Temporal Neural Networks for Covid-19 Forecasting

Coronavirus Disease 2019 (COVID-19) pneumonia started in December 2019 and cases have been reported in 240 countries/regions with more than 570 million confirmed cases and more than 6 million deaths which caused large casualties and huge economic losses. To enhance the understanding of the levels of COVID-19 transmission and infection, and the effects of treatments and interventions, high-quality spatio-temporal COVID-19 datasets and accurate multivariate time-series forecasting models for COVID-19 case prediction play crucial roles. In this paper, we present the COVID-19 spatio-temporal graph (COV19-STG) datasets, i.e., spatio-temporal United States COVID-19 graph datasets on the county-level. By using these datasets, we propose Higher-order Spatio-temporal Neural Networks (HOST-NETs) to further improve the accuracy of predicting COVID-19 trends. Specifically, we incorporate higher-order structure to build a simplicial complex representation learning module, and integrate it into a spatio-temporal neural network architecture, thus leveraging both global and local topological information. Experimental results show that our model consistently outperforms previous state-of-the-art models.
Fast and Accurate Factorized Neural Transducer for Text Adaption of End-to-End Speech Recognition Models

Neural transducer is now the most popular end-to-end model for speech recognition, due to its naturally streaming ability. However, it is challenging to adapt it with text-only data. Factorized neural transducer (FNT) model was proposed to mitigate this problem. The improved adaptation ability of FNT on text-only adaptation data came at the cost of lowered accuracy compared to the standard neural transducer model. We propose several methods to improve the performance of the FNT model. They are: adding CTC criterion during training, adding KL divergence loss during adaptation, using a pre-trained language model to seed the vocabulary predictor, and an efficient adaptation approach by interpolating the vocabulary predictor with the n-gram language model. A combination of these approaches results in a relative word-error-rate reduction of 9.48% from the standard FNT model. Furthermore, n-gram interpolation with the vocabulary predictor improves the adaptation speed hugely with satisfactory adaptation performance.
MUG: A General Meeting Understanding and Generation Benchmark

Listening to long video/audio recordings from video conferencing and online courses for acquiring information is extremely inefficient. Even after ASR systems transcribe recordings into long-form spoken language documents, reading ASR transcripts only partly speeds up seeking information. It has been observed that a range of NLP applications, such as keyphrase extraction, topic segmentation, and summarization, significantly improve users’ efficiency in grasping important information. The meeting scenario is among the most valuable scenarios for deploying these spoken language processing (SLP) capabilities. However, the lack of large-scale public meeting datasets annotated for these SLP tasks severely hinders their advancement. To prompt SLP advancement, we establish a large-scale general Meeting Understanding and Generation Benchmark (MUG) to benchmark the performance of a wide range of SLP tasks, including topic segmentation, topic-level and session-level extractive summarization and topic title generation, keyphrase extraction, and action item detection. To facilitate the MUG benchmark, we construct and release a large-scale meeting dataset for comprehensive long-form SLP development, the AliMeeting4MUG Corpus, which consists of 654 recorded Mandarin meeting sessions with diverse topic coverage, with manual annotations for SLP tasks on manual transcripts of meeting recordings. To the best of our knowledge, the AliMeeting4MUG Corpus is so far the largest meeting corpus in scale and facilitates most SLP tasks. In this paper, we provide a detailed introduction of this corpus, SLP tasks and evaluation methods, baseline systems and their performance <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval

Multilingual text-video retrieval methods have improved significantly in recent years, but the performance for languages other than English still lags. We propose a Cross-Lingual Cross-Modal Knowledge Distillation method to improve multilingual text-video retrieval. Inspired by the fact that English text-video retrieval outperforms other languages, we train a student model using input text in different languages to match the cross-modal predictions from teacher models using input text in English. We propose a cross entropy based objective which forces the distribution over the student’s text-video similarity scores to be similar to those of the teacher models. We introduce a new multilingual video dataset, Multi-YouCook2, by translating the English captions in the YouCook2 video dataset to 8 other languages. Our method improves multilingual text-video retrieval performance on Multi-YouCook2 and several other datasets such as Multi-MSRVTT and VATEX. We also conducted an analysis on the effectiveness of different multilingual text models as teachers.
SPG: Structure-Private Graph Database via SqueezePIR

Many relational data in our daily life are represented as graphs, making graph application an important workload. Because of the large scale of graph datasets, moving graph data to the cloud becomes a popular option. To keep the confidential and private graph secure from an untrusted cloud server, many cryptographic techniques are leveraged to hide the content of the data. However, protecting only the data content is not enough for a graph database. Because the structural information of the graph can be revealed through the database accessing track. In this work, we study the graph neural network (GNN), an important graph workload to mine information from a graph database. We find that the server is able to infer which node is processing during the edge retrieving phase and also learn its neighbor indices during GNN's aggregation phase. This leads to the leakage of the information of graph structure data. In this work, we present SPG, a structure-private graph database with SqueezePIR. Our SPG is built on top of Private Information Retrieval (PIR), which securely hides which nodes/neighbors are accessed. In addition, we propose SqueezePIR, a compression technique to overcome the computation overhead of PIR. Based on our evaluation, our SqueezePIR achieves 11.85× speedup on average with less than 2% accuracy loss when compared to the state-of-the-art FastPIR protocol.
Mission Impact Analysis by Measuring the Effect on Physical Combat Operations Associated With Cyber Asset Damage

As operations previously undertaken only in physical space in the past have changed to operations that include cyberspace, it is crucial to define the concept of ''cyber missions'' clearly.In this study, ''cyber mission'' refers to any military operation or process that utilizes cyber systems to perform actions in accordance with orders delivered to them.Because a weapon system that utilizes a cyber system executes actions based on the commands transmitted to the cyber system, it is necessary to analyze how attacks from cyberspace affects such a weapon system.To this end, it would be meaningful to analyze the tools used to analyze the mission impact of physical weapon systems linked to cyber-attacks.The US military's Joint Munitions Effectiveness Manual (JMEM), which contains the results of analyzing the effects of weapon systems, does not include analysis results for the effects of weapon systems on cyber-attacks.In this study, based on the analysis of the effectiveness of physical warfare, the damage to cyber assets was quantified and associated to calculate the cyber index for the analysis of operational efficiency.In connection with JMEM, the results of combat in cyberspace and the effects of physical operations were compared and analyzed to propose a framework to judge the impact of missions, and the performance was tested.To verify the effectiveness of the proposed framework, domestic and foreign operational scenarios were analyzed and designed, assets were defined, and experiments were conducted.These experiments showed that a greater decrease in the cyber mission effect value was related to a greater effect on physical operations.This framework could be used in a variety of operations to predict the physical impact of a cyber-attack and will help determine the next step in an operation.