OpenRaiser/Intern-Atlas huggingface dataset:
Reduced Robust Random Cut Forest for Out-Of-Distribution detection in machine learning models
Venue: arXiv cs.LG 2022
Year: 2022

Most machine learning-based regressors extract information from data collected via past observations of limited length to make predictions in the future. Consequently, when input to these trained models is data with significantly different statistical properties from data used for training, there is no guarantee of accurate prediction. Consequently, using these models on out-of-distribution input data may result in a completely different predicted outcome from the desired one, which is not only erroneous but can also be hazardous in some cases. Successful deployment of these machine learning models in any system requires a detection system, which should be able to distinguish between out-of-distribution and in-distribution data (i.e. similar to training data). In this paper, we introduce a novel approach for this detection process using a Reduced Robust Random Cut Forest (RRRCF) data structure, which can be used on both small and large data sets. Similar to the Robust Random Cut Forest (RRCF), RRRCF is a structured, but a reduced representation of the training data sub-space in form of cut trees. Empirical results of this method on both low and high-dimensional data showed that inference about data being in/out of training distribution can be made efficiently and the model is easy to train with no difficult hyper-parameter tuning. The paper discusses two different use-cases for testing and validating results.
GMM based multi-stage Wiener filtering for low SNR speech enhancement
Venue: arXiv cs.SD 2022
Year: 2022

This paper proposes a single-channel speech enhancement method to reduce the noise and enhance speech at low signal-to-noise ratio (SNR) levels and non-stationary noise conditions. Specifically, we focus on modeling the noise using a Gaussian mixture model (GMM) based on a multi-stage process with a parametric Wiener filter. The proposed noise model estimates a more accurate noise power spectral density (PSD), and allows for better generalization under various noise conditions compared to traditional Wiener filtering methods. Simulations show that the proposed approach can achieve better performance in terms of speech quality (PESQ) and intelligibility (STOI) at low SNR levels.
Vehicle-to-Vehicle Charging Coordination over Information Centric Networking
Venue: arXiv cs.NI 2022
Year: 2022

Cities around the world are increasingly promoting electric vehicles (EV) to reduce and ultimately eliminate greenhouse gas emissions. For example, the city of San Francisco aims to increase the number of EVs from tens of thousands to over quarter of a million by 2025. This huge number of EVs will put unprecedented stress on the power grid. To efficiently serve the increased charging load, these EVs need to be charged in a coordinated fashion. One promising coordination strategy is vehicle-to-vehicle (V2V) charging coordination, enabling EVs to sell their surplus energy in an ad-hoc, peer to peer manner. Enabling V2V charging coordination requires new communication network protocols that can facilitate such charging coordination in a peer-to-peer fashion. This paper introduces an Information Centric Networking (ICN)-based protocol to support ad-hoc V2V charging coordination (V2V-CC). Our evaluations demonstrate that V2V-CC can provide added flexibility, fault tolerance, and reduced communication latency than a conventional centralized cloud based approach. We show that V2V-CC can achieve a 93\% reduction in protocol completion time compared to a conventional approach. We also show that V2V-CC also works well under extreme packet loss, making it ideal for V2V charging coordination.
Laziness, Barren Plateau, and Noise in Machine Learning
Venue: arXiv cs.LG 2022
Year: 2022

We define \emph{laziness} to describe a large suppression of variational parameter updates for neural networks, classical or quantum. In the quantum case, the suppression is exponential in the number of qubits for randomized variational quantum circuits. We discuss the difference between laziness and \emph{barren plateau} in quantum machine learning created by quantum physicists in \cite{mcclean2018barren} for the flatness of the loss function landscape during gradient descent. We address a novel theoretical understanding of those two phenomena in light of the theory of neural tangent kernels. For noiseless quantum circuits, without the measurement noise, the loss function landscape is complicated in the overparametrized regime with a large number of trainable variational angles. Instead, around a random starting point in optimization, there are large numbers of local minima that are good enough and could minimize the mean square loss function, where we still have quantum laziness, but we do not have barren plateaus. However, the complicated landscape is not visible within a limited number of iterations, and low precision in quantum control and quantum sensing. Moreover, we look at the effect of noises during optimization by assuming intuitive noise models, and show that variational quantum algorithms are noise-resilient in the overparametrization regime. Our work precisely reformulates the quantum barren plateau statement towards a precision statement and justifies the statement in certain noise models, injects new hope toward near-term variational quantum algorithms, and provides theoretical connections toward classical machine learning. Our paper provides conceptual perspectives about quantum barren plateaus, together with discussions about the gradient descent dynamics in \cite{together}.
Characterizing and Mitigating the Difficulty in Training Physics-informed Artificial Neural Networks under Pointwise Constraints
Venue: arXiv cs.LG 2022
Year: 2022

Neural networks can be used to learn the solution of partial differential equations (PDEs) on arbitrary domains without requiring a computational mesh. Common approaches integrate differential operators in training neural networks using a structured loss function. The most common training algorithm for neural networks is backpropagation which relies on the gradient of the loss function with respect to the parameters of the network. In this work, we characterize the difficulty of training neural networks on physics by investigating the impact of differential operators in corrupting the back propagated gradients. Particularly, we show that perturbations present in the output of a neural network model during early stages of training lead to higher levels of noise in a structured loss function that is composed of high-order differential operators. These perturbations consequently corrupt the back-propagated gradients and impede convergence. We mitigate this issue by introducing auxiliary flux parameters to obtain a system of first-order differential equations. We formulate a non-linear unconstrained optimization problem using the augmented Lagrangian method that properly constrains the boundary conditions and adaptively focus on regions of higher gradients that are difficult to learn. We apply our approach to learn the solution of various benchmark PDE problems and demonstrate orders of magnitude improvement over existing approaches.
Fairness-aware Model-agnostic Positive and Unlabeled Learning
Venue: arXiv cs.LG 2022
Year: 2022

With the increasing application of machine learning in high-stake decision-making problems, potential algorithmic bias towards people from certain social groups poses negative impacts on individuals and our society at large. In the real-world scenario, many such problems involve positive and unlabeled data such as medical diagnosis, criminal risk assessment and recommender systems. For instance, in medical diagnosis, only the diagnosed diseases will be recorded (positive) while others will not (unlabeled). Despite the large amount of existing work on fairness-aware machine learning in the (semi-)supervised and unsupervised settings, the fairness issue is largely under-explored in the aforementioned Positive and Unlabeled Learning (PUL) context, where it is usually more severe. In this paper, to alleviate this tension, we propose a fairness-aware PUL method named FairPUL. In particular, for binary classification over individuals from two populations, we aim to achieve similar true positive rates and false positive rates in both populations as our fairness metric. Based on the analysis of the optimal fair classifier for PUL, we design a model-agnostic post-processing framework, leveraging both the positive examples and unlabeled ones. Our framework is proven to be statistically consistent in terms of both the classification error and the fairness metric. Experiments on the synthetic and real-world data sets demonstrate that our framework outperforms state-of-the-art in both PUL and fair classification.
Scalable Neural Data Server: A Data Recommender for Transfer Learning
Venue: arXiv cs.LG 2022
Year: 2022

Absence of large-scale labeled data in the practitioner's target domain can be a bottleneck to applying machine learning algorithms in practice. Transfer learning is a popular strategy for leveraging additional data to improve the downstream performance, but finding the most relevant data to transfer from can be challenging. Neural Data Server (NDS), a search engine that recommends relevant data for a given downstream task, has been previously proposed to address this problem. NDS uses a mixture of experts trained on data sources to estimate similarity between each source and the downstream task. Thus, the computational cost to each user grows with the number of sources. To address these issues, we propose Scalable Neural Data Server (SNDS), a large-scale search engine that can theoretically index thousands of datasets to serve relevant ML data to end users. SNDS trains the mixture of experts on intermediary datasets during initialization, and represents both data sources and downstream tasks by their proximity to the intermediary datasets. As such, computational cost incurred by SNDS users remains fixed as new datasets are added to the server. We validate SNDS on a plethora of real world tasks and find that data recommended by SNDS improves downstream task performance over baselines. We also demonstrate the scalability of SNDS by showing its ability to select relevant data for transfer outside of the natural image setting.
Towards Adversarial Attack on Vision-Language Pre-training Models
Venue: arXiv cs.LG 2022
Year: 2022

While vision-language pre-training model (VLP) has shown revolutionary improvements on various vision-language (V+L) tasks, the studies regarding its adversarial robustness remain largely unexplored. This paper studied the adversarial attack on popular VLP models and V+L tasks. First, we analyzed the performance of adversarial attacks under different settings. By examining the influence of different perturbed objects and attack targets, we concluded some key observations as guidance on both designing strong multimodal adversarial attack and constructing robust VLP models. Second, we proposed a novel multimodal attack method on the VLP models called Collaborative Multimodal Adversarial Attack (Co-Attack), which collectively carries out the attacks on the image modality and the text modality. Experimental results demonstrated that the proposed method achieves improved attack performances on different V+L downstream tasks and VLP models. The analysis observations and novel attack method hopefully provide new understanding into the adversarial robustness of VLP models, so as to contribute their safe and reliable deployment in more real-world scenarios. Code is available at https://github.com/adversarial-for-goodness/Co-Attack.
Are your dependencies code reviewed?: Measuring code review coverage in dependency updates
Venue: arXiv cs.SE 2022
Year: 2022

As modern software extensively uses free open source packages as dependencies, developers have to regularly pull in new third-party code through frequent updates. However, without a proper review of every incoming change, vulnerable and malicious code can sneak into the codebase through these dependencies. The goal of this study is to aid developers in securely accepting dependency updates by measuring if the code changes in an update have passed through a code review process. We implement Depdive, an update audit tool for packages in Crates.io, npm, PyPI, and RubyGems registry. Depdive first (i) identifies the files and the code changes in an update that cannot be traced back to the package's source repository, i.e., \textit{phantom artifacts}; and then (ii) measures what portion of changes in the update, excluding the phantom artifacts, has passed through a code review process, i.e., \textit{code review coverage}. Using Depdive, we present an empirical study across the latest ten updates of the most downloaded 1000 packages in each of the four registries. We further evaluated our results through a maintainer agreement survey. We find the updates are typically only partially code-reviewed (52.5\% of the time). Further, only 9.0\% of the packages had all their updates in our data set fully code-reviewed, indicating that even the most used packages can introduce non-reviewed code in the software supply chain. We also observe that updates either tend to have high \textit{CRC} or low \textit{CRC}, suggesting that packages at the opposite end of the spectrum may require a separate set of treatments.
Meta-Analysis of the Accuracy of Syndromic Surveillance
Venue: arXiv cs.SI 2022
Year: 2022

We present the first meta-analysis of co-evolutionary learning networks for digital disease surveillance research over last 10 years. In doing so, we show the co-evolution and dynamical changes that occurred in academic research related to digital disease surveillance for improving accuracy, approach and results. Using dynamic network analysis, we are able to show the incorporation of social media-based analytics and algorithms which have been proposed and later improved by other researchers as co-evolutionary learning networks. This essentially demonstrates how we improve our research and increase accuracy through feedback loop for correcting the behaviour of an open system and perhaps infer learning patterns, reliability and validity using 10 years scientific research in digital disease surveillance.
SNN2ANN: A Fast and Memory-Efficient Training Framework for Spiking Neural Networks
Venue: arXiv cs.NE 2022
Year: 2022

Spiking neural networks are efficient computation models for low-power environments. Spike-based BP algorithms and ANN-to-SNN (ANN2SNN) conversions are successful techniques for SNN training. Nevertheless, the spike-base BP training is slow and requires large memory costs. Though ANN2NN provides a low-cost way to train SNNs, it requires many inference steps to mimic the well-trained ANN for good performance. In this paper, we propose a SNN-to-ANN (SNN2ANN) framework to train the SNN in a fast and memory-efficient way. The SNN2ANN consists of 2 components: a) a weight sharing architecture between ANN and SNN and b) spiking mapping units. Firstly, the architecture trains the weight-sharing parameters on the ANN branch, resulting in fast training and low memory costs for SNN. Secondly, the spiking mapping units ensure that the activation values of the ANN are the spiking features. As a result, the classification error of the SNN can be optimized by training the ANN branch. Besides, we design an adaptive threshold adjustment (ATA) algorithm to address the noisy spike problem. Experiment results show that our SNN2ANN-based models perform well on the benchmark datasets (CIFAR10, CIFAR100, and Tiny-ImageNet). Moreover, the SNN2ANN can achieve comparable accuracy under 0.625x time steps, 0.377x training time, 0.27x GPU memory costs, and 0.33x spike activities of the Spike-based BP model.
The Game of Tumbleweed is PSPACE-complete
Venue: arXiv cs.CC 2022
Year: 2022

Tumbleweed is a popular two-player perfect-information new territorial game played at the prestigious Mind Sport Olympiad. We define a generalized version of the game, where the board size is arbitrary and so is the possible number of neutral stones. Our result: the complexity of deciding for a given configuration which of the players has a winning strategy is PSPACE-complete. The proof is by a log-space reduction from a Boolean formula game of T.J. Schaefer, known to be PSPACE-complete. We embed the non-planar Schaefer game within the planar Tumbleweed board without using proper "bridges", that are impossible due to the board's topology. Instead, our new technique uses a one-move tight race that forces the players to move only according to the protocol of playing the embedded 4-CNF game.
An Analysis of the Admissibility of the Objective Functions Applied in Evolutionary Multi-objective Clustering
Venue: arXiv cs.NE 2022
Year: 2022

A variety of clustering criteria has been applied as an objective function in Evolutionary Multi-Objective Clustering approaches (EMOCs). However, most EMOCs do not provide detailed analysis regarding the choice and usage of the objective functions. Aiming to support a better choice and definition of the objectives in the EMOCs, this paper proposes an analysis of the admissibility of the clustering criteria in evolutionary optimization by examining the search direction and its potential in finding optimal results. As a result, we demonstrate how the admissibility of the objective functions can influence the optimization. Furthermore, we provide insights regarding the combinations and usage of the clustering criteria in the EMOCs.
Walking to Hide: Privacy Amplification via Random Message Exchanges in Network
Venue: arXiv cs.CR 2022
Year: 2022

The *shuffle model* is a powerful tool to amplify the privacy guarantees of the *local model* of differential privacy. In contrast to the fully decentralized manner of guaranteeing privacy in the local model, the shuffle model requires a central, trusted shuffler. To avoid this central shuffler, recent work of Liew et al. (2022) proposes shuffling locally randomized data in a decentralized manner, via random walks on the communication network constituted by the clients. The privacy amplification bound it thus provides depends on the topology of the underlying communication network, even for infinitely long random walks. It does not match the state-of-the-art privacy amplification bound for the shuffle model (Feldman et al., 2021). In this work, we prove that the output of~$n$ clients' data, each perturbed by an $ε_0$-local randomizer, and shuffled by random walks with a logarithmic number of steps, is $( {O} ( (1 - e^{-ε_0} ) \sqrt{ ( e^{ε_0} / n ) \ln (1 / δ) } ), O(δ) )$-differentially private. Importantly, this bound is independent of the topology of the communication network, and asymptotically closes the gap between the privacy amplification bounds for the network shuffle model (Liew et al., 2022) and the shuffle model (Feldman et al., 2021). Our proof is based on a reduction to the shuffle model, and an analysis of the distribution of random walks of finite length. Building on this, we further show that if each client is sampled independently with probability~$p$, the privacy guarantee of the network shuffle model can be further improved to $( {O} ( (1 - e^{-ε_0} ) \sqrt{p ( e^{ε_0} / n ) \ln (1 / δ) } ) , O(δ) )$. Importantly, the subsampling is also performed in a fully decentralized manner that does not require a trusted central entity; compared with related bounds in prior work, our bound is stronger.
Temporal Link Prediction via Adjusted Sigmoid Function and 2-Simplex Sructure
Venue: arXiv cs.SI 2022
Year: 2022

Temporal network link prediction is an important task in the field of network science, and has a wide range of applications in practical scenarios. Revealing the evolutionary mechanism of the network is essential for link prediction, and how to effectively utilize the historical information for temporal links and efficiently extract the high-order patterns of network structure remains a vital challenge. To address these issues, in this paper, we propose a novel temporal link prediction model with adjusted sigmoid function and 2-simplex structure (TLPSS). The adjusted sigmoid decay mode takes the active, decay and stable states of edges into account, which properly fits the life cycle of information. Moreover, the latent matrix sequence is introduced, which is composed of simplex high-order structure, to enhance the performance of link prediction method since it is highly feasible in sparse network. Combining the life cycle of information and simplex high-order structure, the overall performance of TLPSS is achieved by satisfying the consistency of temporal and structural information in dynamic networks. Experimental results on six real-world datasets demonstrate the effectiveness of TLPSS, and our proposed model improves the performance of link prediction by an average of 15% compared to other baseline methods.
Eliminating The Impossible, Whatever Remains Must Be True
Venue: arXiv cs.AI 2022
Year: 2022

The rise of AI methods to make predictions and decisions has led to a pressing need for more explainable artificial intelligence (XAI) methods. One common approach for XAI is to produce a post-hoc explanation, explaining why a black box ML model made a certain prediction. Formal approaches to post-hoc explanations provide succinct reasons for why a prediction was made, as well as why not another prediction was made. But these approaches assume that features are independent and uniformly distributed. While this means that "why" explanations are correct, they may be longer than required. It also means the "why not" explanations may be suspect as the counterexamples they rely on may not be meaningful. In this paper, we show how one can apply background knowledge to give more succinct "why" formal explanations, that are presumably easier to interpret by humans, and give more accurate "why not" explanations. In addition, we show how to use existing rule induction techniques to efficiently extract background information from a dataset, and also how to report which background information was used to make an explanation, allowing a human to examine it if they doubt the correctness of the explanation.
Asymptotic Nash Equilibrium for the $M$-ary Sequential Adversarial Hypothesis Testing Game
Venue: arXiv cs.IT 2022
Year: 2022

In this paper, we consider a novel $M$-ary sequential hypothesis testing problem in which an adversary is present and perturbs the distributions of the samples before the decision maker observes them. This problem is formulated as a sequential adversarial hypothesis testing game played between the decision maker and the adversary. This game is a zero-sum and strategic one. We assume the adversary is active under \emph{all} hypotheses and knows the underlying distribution of observed samples. We adopt this framework as it is the worst-case scenario from the perspective of the decision maker. The goal of the decision maker is to minimize the expectation of the stopping time to ensure that the test is as efficient as possible; the adversary's goal is, instead, to maximize the stopping time. We derive a pair of strategies under which the asymptotic Nash equilibrium of the game is attained. We also consider the case in which the adversary is not aware of the underlying hypothesis and hence is constrained to apply the same strategy regardless of which hypothesis is in effect. Numerical results corroborate our theoretical findings.