OpenRaiser/Intern-Atlas huggingface dataset:
Muti-view Mouse Social Behaviour Recognition with Deep Graphical Model
Venue: arXiv cs.CV 2021
Year: 2021

Home-cage social behaviour analysis of mice is an invaluable tool to assess therapeutic efficacy of neurodegenerative diseases. Despite tremendous efforts made within the research community, single-camera video recordings are mainly used for such analysis. Because of the potential to create rich descriptions of mouse social behaviors, the use of multi-view video recordings for rodent observations is increasingly receiving much attention. However, identifying social behaviours from various views is still challenging due to the lack of correspondence across data sources. To address this problem, we here propose a novel multiview latent-attention and dynamic discriminative model that jointly learns view-specific and view-shared sub-structures, where the former captures unique dynamics of each view whilst the latter encodes the interaction between the views. Furthermore, a novel multi-view latent-attention variational autoencoder model is introduced in learning the acquired features, enabling us to learn discriminative features in each view. Experimental results on the standard CRMI13 and our multi-view Parkinson's Disease Mouse Behaviour (PDMB) datasets demonstrate that our model outperforms the other state of the arts technologies and effectively deals with the imbalanced data problem.
Hints and Principles for Computer System Design
Venue: arXiv cs.DC 2021
Year: 2021

This new long version of my 1983 paper suggests the goals you might have for your system -- Simple, Timely, Efficient, Adaptable, Dependable, Yummy (STEADY) -- and techniques for achieving them -- Approximate, Incremental, Divide & Conquer (AID). It also gives some principles for system design that are more than just hints, and many examples of how to apply the ideas.
HILONet: Hierarchical Imitation Learning from Non-Aligned Observations
Venue: arXiv cs.LG 2021
Year: 2021

It is challenging learning from demonstrated observation-only trajectories in a non-time-aligned environment because most imitation learning methods aim to imitate experts by following the demonstration step-by-step. However, aligned demonstrations are seldom obtainable in real-world scenarios. In this work, we propose a new imitation learning approach called Hierarchical Imitation Learning from Observation(HILONet), which adopts a hierarchical structure to choose feasible sub-goals from demonstrated observations dynamically. Our method can solve all kinds of tasks by achieving these sub-goals, whether it has a single goal position or not. We also present three different ways to increase sample efficiency in the hierarchical structure. We conduct extensive experiments using several environments. The results show the improvement in both performance and learning efficiency.
Deep tree-ensembles for multi-output prediction
Venue: arXiv cs.LG 2021
Year: 2021

Recently, deep neural networks have expanded the state-of-art in various scientific fields and provided solutions to long standing problems across multiple application domains. Nevertheless, they also suffer from weaknesses since their optimal performance depends on massive amounts of training data and the tuning of an extended number of parameters. As a countermeasure, some deep-forest methods have been recently proposed, as efficient and low-scale solutions. Despite that, these approaches simply employ label classification probabilities as induced features and primarily focus on traditional classification and regression tasks, leaving multi-output prediction under-explored. Moreover, recent work has demonstrated that tree-embeddings are highly representative, especially in structured output prediction. In this direction, we propose a novel deep tree-ensemble (DTE) model, where every layer enriches the original feature set with a representation learning component based on tree-embeddings. In this paper, we specifically focus on two structured output prediction tasks, namely multi-label classification and multi-target regression. We conducted experiments using multiple benchmark datasets and the obtained results confirm that our method provides superior results to state-of-the-art methods in both tasks.
Pitfalls in Machine Learning Research: Reexamining the Development Cycle
Venue: arXiv cs.LG 2021
Year: 2021

Machine learning has the potential to fuel further advances in data science, but it is greatly hindered by an ad hoc design process, poor data hygiene, and a lack of statistical rigor in model evaluation. Recently, these issues have begun to attract more attention as they have caused public and embarrassing issues in research and development. Drawing from our experience as machine learning researchers, we follow the machine learning process from algorithm design to data collection to model evaluation, drawing attention to common pitfalls and providing practical recommendations for improvements. At each step, case studies are introduced to highlight how these pitfalls occur in practice, and where things could be improved.
Depth Self-Optimized Learning Toward Data Science
Venue: arXiv cs.LG 2021
Year: 2021

We propose a two-stage model called Depth Self-Optimized Learning (DSOL), which aims to realize ANN depth self-configuration, self-optimization as well as ANN training without manual intervention. In the first stage of DSOL, it will configure ANN of specific depth according to a specific dataset. In the second stage, DSOL will continuously optimize ANN based on Reinforcement Learning (RL). Finally, the optimal depth is returned to the first stage of DSOL for training, so that DSOL can configure the appropriate ANN depth and perform more reasonable optimization when processing similar datasets again. In the experiment, we ran DSOL on the Iris and Boston housing datasets, and the results showed that DSOL performed well. We have uploaded the experiment records and code to our Github.
This Looks Like That, Because ... Explaining Prototypes for Interpretable Image Recognition
Venue: arXiv cs.CV 2021
Year: 2021

Image recognition with prototypes is considered an interpretable alternative for black box deep learning models. Classification depends on the extent to which a test image "looks like" a prototype. However, perceptual similarity for humans can be different from the similarity learned by the classification model. Hence, only visualising prototypes can be insufficient for a user to understand what a prototype exactly represents, and why the model considers a prototype and an image to be similar. We address this ambiguity and argue that prototypes should be explained. We improve interpretability by automatically enhancing visual prototypes with textual quantitative information about visual characteristics deemed important by the classification model. Specifically, our method clarifies the meaning of a prototype by quantifying the influence of colour hue, shape, texture, contrast and saturation and can generate both global and local explanations. Because of the generality of our approach, it can improve the interpretability of any similarity-based method for prototypical image recognition. In our experiments, we apply our method to the existing Prototypical Part Network (ProtoPNet). Our analysis confirms that the global explanations are generalisable, and often correspond to the visually perceptible properties of a prototype. Our explanations are especially relevant for prototypes which might have been interpreted incorrectly otherwise. By explaining such 'misleading' prototypes, we improve the interpretability and simulatability of a prototype-based classification model. We also use our method to check whether visually similar prototypes have similar explanations, and are able to discover redundancy. Code is available at https://github.com/M-Nauta/Explaining_Prototypes .
Towards Dark Jargon Interpretation in Underground Forums
Venue: arXiv cs.CR 2021
Year: 2021

Dark jargons are benign-looking words that have hidden, sinister meanings and are used by participants of underground forums for illicit behavior. For example, the dark term "rat" is often used in lieu of "Remote Access Trojan". In this work we present a novel method towards automatically identifying and interpreting dark jargons. We formalize the problem as a mapping from dark words to "clean" words with no hidden meaning. Our method makes use of interpretable representations of dark and clean words in the form of probability distributions over a shared vocabulary. In our experiments we show our method to be effective in terms of dark jargon identification, as it outperforms another related method on simulated data. Using manual evaluation, we show that our method is able to detect dark jargons in a real-world underground forum dataset.
Neural Stochastic Contraction Metrics for Learning-based Control and Estimation
Venue: arXiv cs.LG 2021
Year: 2021

We present Neural Stochastic Contraction Metrics (NSCM), a new design framework for provably-stable robust control and estimation for a class of stochastic nonlinear systems. It uses a spectrally-normalized deep neural network to construct a contraction metric, sampled via simplified convex optimization in the stochastic setting. Spectral normalization constrains the state-derivatives of the metric to be Lipschitz continuous, thereby ensuring exponential boundedness of the mean squared distance of system trajectories under stochastic disturbances. The NSCM framework allows autonomous agents to approximate optimal stable control and estimation policies in real-time, and outperforms existing nonlinear control and estimation techniques including the state-dependent Riccati equation, iterative LQR, EKF, and the deterministic neural contraction metric, as illustrated in simulation results.
Beyond Marginal Uncertainty: How Accurately can Bayesian Regression Models Estimate Posterior Predictive Correlations?
Venue: arXiv cs.LG 2021
Year: 2021

While uncertainty estimation is a well-studied topic in deep learning, most such work focuses on marginal uncertainty estimates, i.e. the predictive mean and variance at individual input locations. But it is often more useful to estimate predictive correlations between the function values at different input locations. In this paper, we consider the problem of benchmarking how accurately Bayesian models can estimate predictive correlations. We first consider a downstream task which depends on posterior predictive correlations: transductive active learning (TAL). We find that TAL makes better use of models' uncertainty estimates than ordinary active learning, and recommend this as a benchmark for evaluating Bayesian models. Since TAL is too expensive and indirect to guide development of algorithms, we introduce two metrics which more directly evaluate the predictive correlations and which can be computed efficiently: meta-correlations (i.e. the correlations between the models correlation estimates and the true values), and cross-normalized likelihoods (XLL). We validate these metrics by demonstrating their consistency with TAL performance and obtain insights about the relative performance of current Bayesian neural net and Gaussian process models.
Channel Pruning via Multi-Criteria based on Weight Dependency
Venue: arXiv cs.CV 2021
Year: 2021

Channel pruning has demonstrated its effectiveness in compressing ConvNets. In many related arts, the importance of an output feature map is only determined by its associated filter. However, these methods ignore a small part of weights in the next layer which disappears as the feature map is removed. They ignore the phenomenon of weight dependency. Besides, many pruning methods use only one criterion for evaluation and find a sweet spot of pruning structure and accuracy in a trial-and-error fashion, which can be time-consuming. In this paper, we proposed a channel pruning algorithm via multi-criteria based on weight dependency, CPMC, which can compress a pre-trained model directly. CPMC defines channel importance in three aspects, including its associated weight value, computational cost, and parameter quantity. According to the phenomenon of weight dependency, CPMC gets channel importance by assessing its associated filter and the corresponding partial weights in the next layer. Then CPMC uses global normalization to achieve cross-layer comparison. Finally, CPMC removes less important channels by global ranking. CPMC can compress various CNN models, including VGGNet, ResNet, and DenseNet on various image classification datasets. Extensive experiments have shown CPMC outperforms the others significantly.
A New Inference algorithm of Dynamic Uncertain Causality Graph based on Conditional Sampling Method for Complex Cases
Venue: arXiv cs.AI 2021
Year: 2021

Dynamic Uncertain Causality Graph(DUCG) is a recently proposed model for diagnoses of complex systems. It performs well for industry system such as nuclear power plants, chemical system and spacecrafts. However, the variable state combination explosion in some cases is still a problem that may result in inefficiency or even disability in DUCG inference. In the situation of clinical diagnoses, when a lot of intermediate causes are unknown while the downstream results are known in a DUCG graph, the combination explosion may appear during the inference computation. Monte Carlo sampling is a typical algorithm to solve this problem. However, we are facing the case that the occurrence rate of the case is very small, e.g. $10^{-20}$, which means a huge number of samplings are needed. This paper proposes a new scheme based on conditional stochastic simulation which obtains the final result from the expectation of the conditional probability in sampling loops instead of counting the sampling frequency, and thus overcomes the problem. As a result, the proposed algorithm requires much less time than the DUCG recursive inference algorithm presented earlier. Moreover, a simple analysis of convergence rate based on a designed example is given to show the advantage of the proposed method. % In addition, supports for logic gate, logic cycles, and parallelization, which exist in DUCG, are also addressed in this paper. The new algorithm reduces the time consumption a lot and performs 3 times faster than old one with 2.7% error ratio in a practical graph for Viral Hepatitis B.
Detecting and Counting Small Subgraphs, and Evaluating a Parameterized Tutte Polynomial: Lower Bounds via Toroidal Grids and Cayley Graph Expanders
Venue: arXiv cs.CC 2021
Year: 2021

Given a graph property $Φ$, we consider the problem $\mathtt{EdgeSub}(Φ)$, where the input is a pair of a graph $G$ and a positive integer $k$, and the task is to decide whether $G$ contains a $k$-edge subgraph that satisfies $Φ$. Specifically, we study the parameterized complexity of $\mathtt{EdgeSub}(Φ)$ and of its counting problem $\#\mathtt{EdgeSub}(Φ)$ with respect to both approximate and exact counting. We obtain a complete picture for minor-closed properties $Φ$: the decision problem $\mathtt{EdgeSub}(Φ)$ always admits an FPT algorithm and the counting problem $\#\mathtt{EdgeSub}(Φ)$ always admits an FPTRAS. For exact counting, we present an exhaustive and explicit criterion on the property $Φ$ which, if satisfied, yields fixed-parameter tractability and otherwise $\#\mathsf{W[1]}$-hardness. Additionally, most of our hardness results come with an almost tight conditional lower bound under the so-called Exponential Time Hypothesis, ruling out algorithms for $\#\mathtt{EdgeSub}(Φ)$ that run in time $f(k)\cdot|G|^{o(k/\log k)}$ for any computable function $f$. As a main technical result, we gain a complete understanding of the coefficients of toroidal grids and selected Cayley graph expanders in the homomorphism basis of $\#\mathtt{EdgeSub}(Φ)$. This allows us to establish hardness of exact counting using the Complexity Monotonicity framework due to Curticapean, Dell and Marx (STOC'17). Our methods can also be applied to a parameterized variant of the Tutte Polynomial $T^k_G$ of a graph $G$, to which many known combinatorial interpretations of values of the (classical) Tutte Polynomial can be extended. As an example, $T^k_G(2,1)$ corresponds to the number of $k$-forests in the graph $G$. Our techniques allow us to completely understand the parametrized complexity of computing the evaluation of $T^k_G$ at every pair of rational coordinates $(x,y)$.
Extending Equational Monadic Reasoning with Monad Transformers
Venue: arXiv cs.LO 2021
Year: 2021

There is a recent interest for the verification of monadic programs using proof assistants. This line of research raises the question of the integration of monad transformers, a standard technique to combine monads. In this paper, we extend Monae, a Coq library for monadic equational reasoning, with monad transformers and we explain the benefits of this extension. Our starting point is the existing theory of modular monad transformers, which provides a uniform treatment of operations. Using this theory, we simplify the formalization of models in Monae and we propose an approach to support monadic equational reasoning in the presence of monad transformers. We also use Monae to revisit the lifting theorems of modular monad transformers by providing equational proofs and explaining how to patch a known bug using a non-standard use of Coq that combines impredicative polymorphism and parametricity.
Non-local convolutional neural networks (nlcnn) for speaker recognition
Venue: arXiv cs.SD 2021
Year: 2021

Speaker recognition is the process of identifying a speaker based on the voice. The technology has attracted more attention with the recent increase in popularity of smart voice assistants, such as Amazon Alexa. In the past few years, various convolutional neural network (CNN) based speaker recognition algorithms have been proposed and achieved satisfactory performance. However, convolutional operations are building blocks that typically perform on a local neighborhood at a time and thus miss to capture global, long-range interactions at the feature level which are critical for understanding the pattern in a speaker's voice. In this work, we propose to apply Non-local Convolutional Neural Networks (NLCNN) to improve the capability of capturing long-range dependencies at the feature level, therefore improving speaker recognition performance. Specifically, we introduce non-local blocks where the output response of a position is computed as a weighted sum of the input features at all positions. Combining non-local blocks with pre-defined CNN networks, we investigate the effectiveness of NLCNN models. Without extensive tuning, the proposed NLCNN models outperform state-of-the-art speaker recognition algorithms on the public Voxceleb dataset. What's more, we investigate different types of non-local operations applied to the frequency-time domain, time domain, frequency domain and frame-level respectively. Among them, time domain is the most effective one for speaker recognition applications.
Sampling Constraint Satisfaction Solutions in the Local Lemma Regime
Venue: arXiv cs.DS 2021
Year: 2021

We give a Markov chain based algorithm for sampling almost uniform solutions of constraint satisfaction problems (CSPs). Assuming a canonical setting for the Lovász local lemma, where each constraint is violated by a small number of forbidden local configurations, our sampling algorithm is accurate in a local lemma regime, and the running time is a fixed polynomial whose dependency on $n$ is close to linear, where $n$ is the number of variables. Our main approach is a new technique called state compression, which generalizes the "mark/unmark" paradigm of Moitra (Moitra, JACM, 2019), and can give fast local-lemma-based sampling algorithms. As concrete applications of our technique, we give the current best almost-uniform samplers for hypergraph colorings and for CNF solutions.
Time Synchronized State Estimation for Incompletely Observed Distribution Systems Using Deep Learning Considering Realistic Measurement Noise
Venue: arXiv cs.LG 2021
Year: 2021

Time-synchronized state estimation is a challenge for distribution systems because of limited real-time observability. This paper addresses this challenge by formulating a deep learning (DL)-based approach to perform unbalanced three-phase distribution system state estimation (DSSE). Initially, a data-driven approach for judicious measurement selection to facilitate reliable state estimation is provided. Then, a deep neural network (DNN) is trained to perform DSSE for systems that are incompletely observed by synchrophasor measurement devices (SMDs). Robustness of the proposed methodology is demonstrated by considering realistic measurement error models for SMDs. A comparative study of the DNN-based DSSE with classical linear state estimation indicates that the DL-based approach gives better accuracy with a significantly smaller number of SMDs.
Ordered $k$-Median with Outliers and Fault-Tolerance
Venue: arXiv cs.DS 2021
Year: 2021

In this paper, we study two natural generalizations of ordered $k$-median, named robust ordered $k$-median and fault-tolerant ordered $k$-median. In ordered $k$-median, given a finite metric space $(X,d)$, we seek to open $k$ facilities $S\subseteq X$ which induce a service cost vector $\vec{c}=\{d(j,S):j\in X\}$, and minimize the ordered objective $w^\top\vec{c}^\downarrow$. Here $d(j,S)=\min_{i\in S}d(j,i)$ is the minimum distance between $j$ and facilities in $S$, $w\in\mathbb{R}^{|X|}$ is a given non-increasing non-negative vector, and $\vec{c}^\downarrow$ is the non-increasingly sorted version of $\vec{c}$. The current best result is a $(5+ε)$-approximation [CS19]. We first consider robust ordered $k$-median, a.k.a. ordered $k$-median with outliers, where the input consists of an ordered $k$-median instance and parameter $m\in\mathbb{Z}_+$. The goal is to open $k$ facilities $S$, select $m$ clients $T\subseteq X$ and assign the nearest open facility to each $j\in T$. The service cost vector is $\vec{c}=\{d(j,S):j\in T\}$ and $w$ is in $\mathbb{R}^m$. We introduce a novel yet simple objective function that enables linear analysis of the non-linear ordered objective, apply an iterative rounding framework [KLS18] and obtain a constant-factor approximation. We devise the first constant-approximations for ordered matroid median and ordered knapsack median using the same method. We also consider fault-tolerant ordered $k$-median, where besides the same input as ordered $k$-median, we are also given additional client requirements $\{r_j\in\mathbb{Z}_+:j\in X\}$ and need to assign $r_j$ distinct open facilities to each client $j\in X$. The service cost of $j$ is the sum of distances to its assigned facilities, and the objective is the same. We obtain a constant-factor approximation using a novel LP relaxation with constraints created via a new sparsification technique.
Sparsely constrained neural networks for model discovery of PDEs
Venue: arXiv cs.LG 2021
Year: 2021

Sparse regression on a library of candidate features has developed as the prime method to discover the partial differential equation underlying a spatio-temporal data-set. These features consist of higher order derivatives, limiting model discovery to densely sampled data-sets with low noise. Neural network-based approaches circumvent this limit by constructing a surrogate model of the data, but have to date ignored advances in sparse regression algorithms. In this paper we present a modular framework that dynamically determines the sparsity pattern of a deep-learning based surrogate using any sparse regression technique. Using our new approach, we introduce a new constraint on the neural network and show how a different network architecture and sparsity estimator improve model discovery accuracy and convergence on several benchmark examples. Our framework is available at \url{https://github.com/PhIMaL/DeePyMoD}