OpenRaiser/Intern-Atlas huggingface dataset:
Adaptive Submodular Meta-Learning
Venue: arXiv cs.LG 2021
Year: 2021

Meta-Learning has gained increasing attention in the machine learning and artificial intelligence communities. In this paper, we introduce and study an adaptive submodular meta-learning problem. The input of our problem is a set of items, where each item has a random state which is initially unknown. The only way to observe an item's state is to select that item. Our objective is to adaptively select a group of items that achieve the best performance over a set of tasks, where each task is represented as an adaptive submodular function that maps sets of items and their states to a real number. To reduce the computational cost while maintaining a personalized solution for each future task, we first select an initial solution set based on previously observed tasks, then adaptively add the remaining items to the initial solution set when a new task arrives. As compared to the solution where a brand new solution is computed for each new task, our meta-learning based approach leads to lower computational overhead at test time since the initial solution set is pre-computed in the training stage. To solve this problem, we propose a two-phase greedy policy and show that it achieves a $1/2$ approximation ratio for the monotone case. For the non-monotone case, we develop a two-phase randomized greedy policy that achieves a $1/32$ approximation ratio.
Theory-guided hard constraint projection (HCP): a knowledge-based data-driven scientific machine learning method
Venue: arXiv cs.LG 2021
Year: 2021

Machine learning models have been successfully used in many scientific and engineering fields. However, it remains difficult for a model to simultaneously utilize domain knowledge and experimental observation data. The application of knowledge-based symbolic AI represented by an expert system is limited by the expressive ability of the model, and data-driven connectionism AI represented by neural networks is prone to produce predictions that violate physical mechanisms. In order to fully integrate domain knowledge with observations, and make full use of the prior information and the strong fitting ability of neural networks, this study proposes theory-guided hard constraint projection (HCP). This model converts physical constraints, such as governing equations, into a form that is easy to handle through discretization, and then implements hard constraint optimization through projection. Based on rigorous mathematical proofs, theory-guided HCP can ensure that model predictions strictly conform to physical mechanisms in the constraint patch. The performance of the theory-guided HCP is verified by experiments based on the heterogeneous subsurface flow problem. Due to the application of hard constraints, compared with fully connected neural networks and soft constraint models, such as theory-guided neural networks and physics-informed neural networks, theory-guided HCP requires fewer data, and achieves higher prediction accuracy and stronger robustness to noisy observations.
OpenHoldem: A Benchmark for Large-Scale Imperfect-Information Game Research
Venue: arXiv cs.LG 2021
Year: 2021

Owning to the unremitting efforts by a few institutes, significant progress has recently been made in designing superhuman AIs in No-limit Texas Hold'em (NLTH), the primary testbed for large-scale imperfect-information game research. However, it remains challenging for new researchers to study this problem since there are no standard benchmarks for comparing with existing methods, which seriously hinders further developments in this research area. In this work, we present OpenHoldem, an integrated toolkit for large-scale imperfect-information game research using NLTH. OpenHoldem makes three main contributions to this research direction: 1) a standardized evaluation protocol for thoroughly evaluating different NLTH AIs, 2) four publicly available strong baselines for NLTH AI, and 3) an online testing platform with easy-to-use APIs for public NLTH AI evaluation. We have released OpenHoldem at holdem.ia.ac.cn, hoping it facilitates further studies on the unsolved theoretical and computational issues in this area and cultivate crucial research problems like opponent modeling and human-computer interactive learning.
Writer Identification and Writer Retrieval Based on NetVLAD with Re-ranking
Venue: arXiv cs.CV 2021
Year: 2021

This paper addresses writer identification and writer retrieval which is considered as a challenging problem in the document analysis and recognition field. In this work, a novel pipeline is proposed for the problem at hand by employing a unified neural network architecture consisting of the ResNet-20 as a feature extractor and an integrated NetVLAD layer, inspired by the vector of locally aggregated descriptors (VLAD), in the head of the latter part. Having defined this architecture, the triplet semi-hard loss function is used to directly learn an embedding for individual input image patches. Subsequently, generalized max-pooling technique is employed for the aggregation of embedded descriptors of each handwritten image. Also, a novel re-ranking strategy is introduced for the task of identification and retrieval based on $k$-reciprocal nearest neighbors, and it is shown that the pipeline can benefit tremendously from this step. Experimental evaluation has been done on the three publicly available datasets: the ICDAR 2013, CVL, and KHATT datasets. Results indicate that while we perform comparably to the state-of-the-art on the KHATT, our writer identification and writer retrieval pipeline achieves superior performance on the ICDAR 2013 and CVL datasets in terms of mAP.
Comparison of Anomaly Detectors: Context Matters
Venue: arXiv cs.LG 2021
Year: 2021

Deep generative models are challenging the classical methods in the field of anomaly detection nowadays. Every new method provides evidence of outperforming its predecessors, often with contradictory results. The objective of this comparison is twofold: to compare anomaly detection methods of various paradigms with focus on deep generative models, and identification of sources of variability that can yield different results. The methods were compared on popular tabular and image datasets. We identified the main sources of variability to be experimental conditions: i) the type data set (tabular or image) and the nature of anomalies (statistical or semantic), and ii) strategy of selection of hyperparameters, especially the number of available anomalies in the validation set. Different methods perform the best in different contexts, i.e. combination of experimental conditions together with computational time. This explains the variability of the previous results and highlights the importance of careful specification of the context in the publication of a new method. All our code and results are available for download.
Advanced Algorithms in Heterogeneous and Uncertain Networking Environments
Venue: arXiv cs.NI 2021
Year: 2021

Communication networks are used today everywhere and on every scale: starting from small Internet of Things (IoT) networks at home, via campus and enterprise networks, and up to tier-one networks of Internet providers. Accordingly, network devices should support a plethora of tasks with highly heterogeneous characteristics in terms of processing time, bandwidth energy consumption, deadlines and so on. Evaluating these characteristics and the amount of currently available resources for handling them requires analyzing all the arriving inputs, gathering information from numerous remote devices, and integrating all this information. Performing all these tasks in real time is very challenging in today's networking environments, which are characterized by tight bounds on the latency, and always-increasing data rates. Hence, network algorithms should typically make decisions under uncertainty. This work addresses optimizing performance in heterogeneous and uncertain networking environments. We begin by detailing the sources of heterogeneity and uncertainty and show that uncertainty appears in all layers of network design, including the time required to perform a task; the amount of available resources; and the expected gain from successfully completing a task. Next, we survey current solutions and show their limitations. Based on these insights we develop general design concepts to tackle heterogeneity and uncertainty, and then use these concepts to design practical algorithms. For each of our algorithms, we provide rigorous mathematical analysis, thus showing worst-case performance guarantees. Finally, we implement and run the suggested algorithms on various input traces, thus obtaining further insights as to our algorithmic design principles.
RegulaTor: A Straightforward Website Fingerprinting Defense
Venue: arXiv cs.CR 2021
Year: 2021

Website Fingerprinting (WF) attacks are used by local passive attackers to determine the destination of encrypted internet traffic by comparing the sequences of packets sent to and received by the user to a previously recorded data set. As a result, WF attacks are of particular concern to privacy-enhancing technologies such as Tor. In response, a variety of WF defenses have been developed, though they tend to incur high bandwidth and latency overhead or require additional infrastructure, thus making them difficult to implement in practice. Some lighter-weight defenses have been presented as well; still, they attain only moderate effectiveness against recently published WF attacks. In this paper, we aim to present a realistic and novel defense, RegulaTor, which takes advantage of common patterns in web browsing traffic to reduce both defense overhead and the accuracy of current WF attacks. In the closed-world setting, RegulaTor reduces the accuracy of the state-of-the-art attack, Tik-Tok, against comparable defenses from 66% to 25.4%. To achieve this performance, it requires limited added latency and a bandwidth overhead 39.3% less than the leading moderate-overhead defense. In the open-world setting, RegulaTor limits a precision-tuned Tik-Tok attack to an F-score of .135, compared to .625 for the best comparable defense.
Clustered Sparse Channel Estimation for Massive MIMO Systems by Expectation Maximization-Propagation (EM-EP)
Venue: arXiv cs.IT 2021
Year: 2021

We study the problem of downlink channel estimation in multi-user massive multiple input multiple output (MIMO) systems. To this end, we consider a Bayesian compressive sensing approach in which the clustered sparse structure of the channel in the angular domain is employed to reduce the pilot overhead. To capture the clustered structure, we employ a conditionally independent identically distributed Bernoulli-Gaussian prior on the sparse vector representing the channel, and a Markov prior on its support vector. An expectation propagation (EP) algorithm is developed to approximate the intractable joint distribution on the sparse vector and its support with a distribution from an exponential family. The approximate distribution is then used for direct estimation of the channel. The EP algorithm assumes that the model parameters are known a priori. Since these parameters are unknown, we estimate these parameters using the expectation maximization (EM) algorithm. The combination of EM and EP referred to as EM-EP algorithm is reminiscent of the variational EM approach. Simulation results show that the proposed EM-EP algorithm outperforms several recently-proposed algorithms in the literature.
Are We Ready For Learned Cardinality Estimation?
Venue: arXiv cs.DB 2021
Year: 2021

Cardinality estimation is a fundamental but long unresolved problem in query optimization. Recently, multiple papers from different research groups consistently report that learned models have the potential to replace existing cardinality estimators. In this paper, we ask a forward-thinking question: Are we ready to deploy these learned cardinality models in production? Our study consists of three main parts. Firstly, we focus on the static environment (i.e., no data updates) and compare five new learned methods with eight traditional methods on four real-world datasets under a unified workload setting. The results show that learned models are indeed more accurate than traditional methods, but they often suffer from high training and inference costs. Secondly, we explore whether these learned models are ready for dynamic environments (i.e., frequent data updates). We find that they cannot catch up with fast data up-dates and return large errors for different reasons. For less frequent updates, they can perform better but there is no clear winner among themselves. Thirdly, we take a deeper look into learned models and explore when they may go wrong. Our results show that the performance of learned methods can be greatly affected by the changes in correlation, skewness, or domain size. More importantly, their behaviors are much harder to interpret and often unpredictable. Based on these findings, we identify two promising research directions (control the cost of learned models and make learned models trustworthy) and suggest a number of research opportunities. We hope that our study can guide researchers and practitioners to work together to eventually push learned cardinality estimators into real database systems.
Knowledge Capture and Replay for Continual Learning
Venue: arXiv cs.LG 2021
Year: 2021

Deep neural networks have shown promise in several domains, and the learned data (task) specific information is implicitly stored in the network parameters. Extraction and utilization of encoded knowledge representations are vital when data is no longer available in the future, especially in a continual learning scenario. In this work, we introduce {\em flashcards}, which are visual representations that {\em capture} the encoded knowledge of a network as a recursive function of predefined random image patterns. In a continual learning scenario, flashcards help to prevent catastrophic forgetting and consolidating knowledge of all the previous tasks. Flashcards need to be constructed only before learning the subsequent task, and hence, independent of the number of tasks trained before. We demonstrate the efficacy of flashcards in capturing learned knowledge representation (as an alternative to the original dataset) and empirically validate on a variety of continual learning tasks: reconstruction, denoising, task-incremental learning, and new-instance learning classification, using several heterogeneous benchmark datasets. Experimental evidence indicates that: (i) flashcards as a replay strategy is { \em task agnostic}, (ii) performs better than generative replay, and (iii) is on par with episodic replay without additional memory overhead.
Digital Twins Are Not Monozygotic -- Cross-Replicating ADAS Testing in Two Industry-Grade Automotive Simulators
Venue: arXiv cs.SE 2021
Year: 2021

The increasing levels of software- and data-intensive driving automation call for an evolution of automotive software testing. As a recommended practice of the Verification and Validation (V&V) process of ISO/PAS 21448, a candidate standard for safety of the intended functionality for road vehicles, simulation-based testing has the potential to reduce both risks and costs. There is a growing body of research on devising test automation techniques using simulators for Advanced Driver-Assistance Systems (ADAS). However, how similar are the results if the same test scenarios are executed in different simulators? We conduct a replication study of applying a Search-Based Software Testing (SBST) solution to a real-world ADAS (PeVi, a pedestrian vision detection system) using two different commercial simulators, namely, TASS/Siemens PreScan and ESI Pro-SiVIC. Based on a minimalistic scene, we compare critical test scenarios generated using our SBST solution in these two simulators. We show that SBST can be used to effectively and efficiently generate critical test scenarios in both simulators, and the test results obtained from the two simulators can reveal several weaknesses of the ADAS under test. However, executing the same test scenarios in the two simulators leads to notable differences in the details of the test outputs, in particular, related to (1) safety violations revealed by tests, and (2) dynamics of cars and pedestrians. Based on our findings, we recommend future V&V plans to include multiple simulators to support robust simulation-based testing and to base test objectives on measures that are less dependant on the internals of the simulators.
Learning Symbolic Expressions via Gumbel-Max Equation Learner Networks
Venue: arXiv cs.LG 2021
Year: 2021

Most of the neural networks (NNs) learned via state-of-the-art machine learning techniques are black-box models. For a widespread success of machine learning in science and engineering, it is important to develop new NN architectures to effectively extract high-level mathematical knowledge from complex datasets. Motivated by this understanding, this paper develops a new NN architecture called the Gumbel-Max Equation Learner (GMEQL) network. Different from previously proposed Equation Learner (EQL) networks, GMEQL applies continuous relaxation to the network structure via the Gumbel-Max trick and introduces two types of trainable parameters: structure parameters and regression parameters. This paper also proposes a two-stage training process with new techniques to train structure parameters in both online and offline settings based on an elite repository. On 8 benchmark symbolic regression problems, GMEQL is experimentally shown to outperform several cutting-edge machine learning approaches.
Multi-Interactive Attention Network for Fine-grained Feature Learning in CTR Prediction
Venue: arXiv cs.IR 2021
Year: 2021

In the Click-Through Rate (CTR) prediction scenario, user's sequential behaviors are well utilized to capture the user interest in the recent literature. However, despite being extensively studied, these sequential methods still suffer from three limitations. First, existing methods mostly utilize attention on the behavior of users, which is not always suitable for CTR prediction, because users often click on new products that are irrelevant to any historical behaviors. Second, in the real scenario, there exist numerous users that have operations a long time ago, but turn relatively inactive in recent times. Thus, it is hard to precisely capture user's current preferences through early behaviors. Third, multiple representations of user's historical behaviors in different feature subspaces are largely ignored. To remedy these issues, we propose a Multi-Interactive Attention Network (MIAN) to comprehensively extract the latent relationship among all kinds of fine-grained features (e.g., gender, age and occupation in user-profile). Specifically, MIAN contains a Multi-Interactive Layer (MIL) that integrates three local interaction modules to capture multiple representations of user preference through sequential behaviors and simultaneously utilize the fine-grained user-specific as well as context information. In addition, we design a Global Interaction Module (GIM) to learn the high-order interactions and balance the different impacts of multiple features. Finally, Offline experiment results from three datasets, together with an Online A/B test in a large-scale recommendation system, demonstrate the effectiveness of our proposed approach.
Fine-Grained Lineage for Safer Notebook Interactions
Venue: arXiv cs.SE 2021
Year: 2021

Computational notebooks have emerged as the platform of choice for data science and analytical workflows, enabling rapid iteration and exploration. By keeping intermediate program state in memory and segmenting units of execution into so-called "cells", notebooks allow users to execute their workflows interactively and enjoy particularly tight feedback. However, as cells are added, removed, reordered, and rerun, this hidden intermediate state accumulates in a way that is not necessarily correlated with the notebook's visible code, making execution behavior difficult to reason about, and leading to errors and lack of reproducibility. We present NBSafety, a custom Jupyter kernel that uses runtime tracing and static analysis to automatically manage lineage associated with cell execution and global notebook state. NBSafety detects and prevents errors that users make during unaided notebook interactions, all while preserving the flexibility of existing notebook semantics. We evaluate NBSafety's ability to prevent erroneous interactions by replaying and analyzing 666 real notebook sessions. Of these, NBSafety identified 117 sessions with potential safety errors, and in the remaining 549 sessions, the cells that NBSafety identified as resolving safety issues were more than $7\times$ more likely to be selected by users for re-execution compared to a random baseline, even though the users were not using NBSafety and were therefore not influenced by its suggestions.
Joint Hardware Design and Capacity Analysis for Intelligent Reflecting Surface Enabled Terahertz MIMO Communications
Venue: arXiv cs.IT 2021
Year: 2021

Terahertz (THz) communications have been envisioned as a promising enabler to provide ultra-high data transmission for sixth generation (6G) wireless networks. To tackle the blockage vulnerability brought by severe path attenuation and poor diffraction of THz waves, an intelligent reflecting surface (IRS) is put forward to smartly control the incident THz waves by adjusting the phase shifts. In this paper, we firstly design an efficient hardware structure of graphene-based IRS with phase response up to 306.82 degrees. Subsequently, to characterize the capacity of the IRS-enabled THz multiple-input multiple-output (MIMO) system, an adaptive gradient descent (A-GD) algorithm is developed by dynamically updating the step size during the iterative process, which is determined by the second-order Taylor expansion formulation. In contrast with conventional gradient descent (C-GD) algorithm with fixed step size, the A-GD algorithm evidently improves the achievable rate performance. However, both A-GD algorithm and C-GD algorithm inherit the unacceptable complexity. Then a low complexity alternating optimization (AO) algorithm is proposed by alternately optimizing the precoding matrix by a column-by-column (CBC) algorithm and the phase shift matrix of the IRS by a linear search algorithm. Ultimately, the numerical results demonstrate the effectiveness of the designed hardware structure and the considered algorithms.
Statistical CSI-based Design for Reconfigurable Intelligent Surface-aided Massive MIMO Systems with Direct Links
Venue: arXiv cs.IT 2021
Year: 2021

This paper investigates the performance of reconfigurable intelligent surface (RIS)-aided massive multiple-input multiple-output (MIMO) systems with direct links, and the phase shifts of the RIS are designed based on the statistical channel state information (CSI). We first derive the closed-form expression of the uplink ergodic data rate. Then, based on the derived expression, we use the genetic algorithm (GA) to solve the sum data rate maximization problem. With low-complexity maximal-ratio combination (MRC) and low-overhead statistical CSI-based scheme, we validate that the RIS can still bring significant performance gains to traditional massive MIMO systems.