OpenRaiser/Intern-Atlas huggingface dataset:
DataSist: A Python-based library for easy data analysis, visualization and modeling
Venue: arXiv cs.LG 2020
Year: 2020

A large amount of data is produced every second from modern information systems such as mobile devices, the world wide web, Internet of Things, social media, etc. Analysis and mining of this massive data requires a lot of advanced tools and techniques. Therefore, big data analytics and mining is currently an active and trending area of research because of the enormous benefits businesses and organizations derive from it. Numerous tools like Pandas, Numpy, STATA, SPSS, have been created to help analyze and mine these huge outburst of data and some have become so popular and widely used in the field. This paper presents a new python-based library, DataSist, which offers high level, intuitive and easy to use functions, and methods that helps data scientists/analyst to quickly analyze, mine and visualize big data sets. The objectives of this project were to (i) design a python library to aid data analysis process by abstracting low level syntax, (ii) increase productivity of data scientist by making them focus on what to do rather than how to do it. This project shows that data analysis can be automated and much faster when we abstract certain functions, and will serve as an important tool in the workflow of data scientists.
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT
Venue: arXiv cs.CL 2020
Year: 2020

We present a novel way of injecting factual knowledge about entities into the pretrained BERT model (Devlin et al., 2019): We align Wikipedia2Vec entity vectors (Yamada et al., 2016) with BERT's native wordpiece vector space and use the aligned entity vectors as if they were wordpiece vectors. The resulting entity-enhanced version of BERT (called E-BERT) is similar in spirit to ERNIE (Zhang et al., 2019) and KnowBert (Peters et al., 2019), but it requires no expensive further pretraining of the BERT encoder. We evaluate E-BERT on unsupervised question answering (QA), supervised relation classification (RC) and entity linking (EL). On all three tasks, E-BERT outperforms BERT and other baselines. We also show quantitatively that the original BERT model is overly reliant on the surface form of entity names (e.g., guessing that someone with an Italian-sounding name speaks Italian), and that E-BERT mitigates this problem.
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning
Venue: arXiv cs.CL 2020
Year: 2020

Recently, large-scale pre-trained language models have demonstrated impressive performance on several commonsense-reasoning benchmark datasets. However, building machines with commonsense to compose realistically plausible sentences remains challenging. In this paper, we present a constrained text generation task, CommonGen associated with a benchmark dataset, to explicitly test machines for the ability of generative commonsense reasoning. Given a set of common concepts (e.g., {dog, frisbee, catch, throw}); the task is to generate a coherent sentence describing an everyday scenario using these concepts (e.g., "a man throws a frisbee and his dog catches it"). The CommonGen task is challenging because it inherently requires 1) relational reasoning with background commonsense knowledge, and 2) compositional generalization ability to work on unseen concept combinations. Our dataset, constructed through a combination of crowdsourced and existing caption corpora, consists of 79k commonsense descriptions over 35k unique concept-sets. Experiments show that there is a large gap between state-of-the-art text generation models (e.g., T5) and human performance. Furthermore, we demonstrate that the learned generative commonsense reasoning capability can be transferred to improve downstream tasks such as CommonsenseQA by generating additional context.
Estimation in Poisson Noise: Properties of the Conditional Mean Estimator
Venue: arXiv cs.IT 2020
Year: 2020

This paper considers estimation of a random variable in Poisson noise with signal scaling coefficient and dark current as explicit parameters of the noise model. Specifically, the paper focuses on properties of the conditional mean estimator as a function of the scaling coefficient, the dark current parameter, the distribution of the input random variable and channel realizations. With respect to the scaling coefficient and the dark current, several identities in terms of derivatives are established. For example, it is shown that the gradient of the conditional mean estimator with respect to the scaling coefficient and dark current parameter is proportional to the conditional variance. Moreover, a score function is proposed and a Tweedie-like formula for the conditional expectation is recovered. With respect to the distribution, several regularity conditions are shown. For instance, it is shown that the conditional mean estimator uniquely determines the input distribution. Moreover, it is shown that if the conditional expectation is close to a linear function in terms of mean squared error, then the input distribution is approximately gamma in the Lévy distance. Furthermore, sufficient and necessary conditions for linearity are found. Interestingly, it is shown that the conditional mean estimator cannot be linear when the dark current parameter of the Poisson noise is non-zero.
Distilling Knowledge Learned in BERT for Text Generation
Venue: arXiv cs.CL 2020
Year: 2020

Large-scale pre-trained language model such as BERT has achieved great success in language understanding tasks. However, it remains an open question how to utilize BERT for language generation. In this paper, we present a novel approach, Conditional Masked Language Modeling (C-MLM), to enable the finetuning of BERT on target generation tasks. The finetuned BERT (teacher) is exploited as extra supervision to improve conventional Seq2Seq models (student) for better text generation performance. By leveraging BERT's idiosyncratic bidirectional nature, distilling knowledge learned in BERT can encourage auto-regressive Seq2Seq models to plan ahead, imposing global sequence-level supervision for coherent text generation. Experiments show that the proposed approach significantly outperforms strong Transformer baselines on multiple language generation tasks such as machine translation and text summarization. Our proposed model also achieves new state of the art on IWSLT German-English and English-Vietnamese MT datasets. Code is available at https://github.com/ChenRocks/Distill-BERT-Textgen.
Cooperative Pathfinding based on memory-efficient Multi-agent RRT*
Venue: arXiv cs.MA 2020
Year: 2020

In cooperative pathfinding problems, no-conflicts paths that bring several agents from their start location to their destination need to be planned. This problem can be efficiently solved by Multi-agent RRT*(MA-RRT*) algorithm, which is still state-of-the-art in the field of coupled methods. However, the implementation of this algorithm is hindered in systems with limited memory because the number of nodes in the tree grows indefinitely as the paths get optimized. This paper proposes an improved version of MA-RRT*, called Multi-agent RRT* Fixed Node(MA-RRT*FN), which limits the number of nodes stored in the tree by removing the weak nodes on the path which are not likely to reach the goal. The results show that MA-RRT*FN performs close to MA-RRT* in terms of scalability and solution quality while the memory required is much lower and fixed.
A Nonexistence Certificate for Projective Planes of Order Ten with Weight 15 Codewords
Venue: arXiv cs.DM 2020
Year: 2020

Using techniques from the fields of symbolic computation and satisfiability checking we verify one of the cases used in the landmark result that projective planes of order ten do not exist. In particular, we show that there exist no projective planes of order ten that generate codewords of weight fifteen, a result first shown in 1973 via an exhaustive computer search. We provide a simple satisfiability (SAT) instance and a certificate of unsatisfiability that can be used to automatically verify this result for the first time. All previous demonstrations of this result have relied on search programs that are difficult or impossible to verify---in fact, our search found partial projective planes that were missed by previous searches due to previously undiscovered bugs. Furthermore, we show how the performance of the SAT solver can be dramatically increased by employing functionality from a computer algebra system (CAS). Our SAT+CAS search runs significantly faster than all other published searches verifying this result.
Meta Answering for Machine Reading
Venue: arXiv cs.CL 2020
Year: 2020

We investigate a framework for machine reading, inspired by real world information-seeking problems, where a meta question answering system interacts with a black box environment. The environment encapsulates a competitive machine reader based on BERT, providing candidate answers to questions, and possibly some context. To validate the realism of our formulation, we ask humans to play the role of a meta-answerer. With just a small snippet of text around an answer, humans can outperform the machine reader, improving recall. Similarly, a simple machine meta-answerer outperforms the environment, improving both precision and recall on the Natural Questions dataset. The system relies on joint training of answer scoring and the selection of conditioning information.
Sticky polymatroids on at most five elements
Venue: arXiv cs.IT 2020
Year: 2020

The sticky polymatroid conjecture states that any two extensions of the polymatroid have an amalgam if and only if the polymatroid has no non-modular pairs of flats. We show that the conjecture holds for polymatroids on five or less elements.
Non-Cooperative Inverse Reinforcement Learning
Venue: arXiv cs.GT 2020
Year: 2020

Making decisions in the presence of a strategic opponent requires one to take into account the opponent's ability to actively mask its intended objective. To describe such strategic situations, we introduce the non-cooperative inverse reinforcement learning (N-CIRL) formalism. The N-CIRL formalism consists of two agents with completely misaligned objectives, where only one of the agents knows the true objective function. Formally, we model the N-CIRL formalism as a zero-sum Markov game with one-sided incomplete information. Through interacting with the more informed player, the less informed player attempts to both infer, and act according to, the true objective function. As a result of the one-sided incomplete information, the multi-stage game can be decomposed into a sequence of single-stage games expressed by a recursive formula. Solving this recursive formula yields the value of the N-CIRL game and the more informed player's equilibrium strategy. Another recursive formula, constructed by forming an auxiliary game, termed the dual game, yields the less informed player's strategy. Building upon these two recursive formulas, we develop a computationally tractable algorithm to approximately solve for the equilibrium strategies. Finally, we demonstrate the benefits of our N-CIRL formalism over the existing multi-agent IRL formalism via extensive numerical simulation in a novel cyber security setting.
Data Efficient Direct Speech-to-Text Translation with Modality Agnostic Meta-Learning
Venue: arXiv cs.CL 2020
Year: 2020

End-to-end Speech Translation (ST) models have several advantages such as lower latency, smaller model size, and less error compounding over conventional pipelines that combine Automatic Speech Recognition (ASR) and text Machine Translation (MT) models. However, collecting large amounts of parallel data for ST task is more difficult compared to the ASR and MT tasks. Previous studies have proposed the use of transfer learning approaches to overcome the above difficulty. These approaches benefit from weakly supervised training data, such as ASR speech-to-transcript or MT text-to-text translation pairs. However, the parameters in these models are updated independently of each task, which may lead to sub-optimal solutions. In this work, we adopt a meta-learning algorithm to train a modality agnostic multi-task model that transfers knowledge from source tasks=ASR+MT to target task=ST where ST task severely lacks data. In the meta-learning phase, the parameters of the model are exposed to vast amounts of speech transcripts (e.g., English ASR) and text translations (e.g., English-German MT). During this phase, parameters are updated in such a way to understand speech, text representations, the relation between them, as well as act as a good initialization point for the target ST task. We evaluate the proposed meta-learning approach for ST tasks on English-German (En-De) and English-French (En-Fr) language pairs from the Multilingual Speech Translation Corpus (MuST-C). Our method outperforms the previous transfer learning approaches and sets new state-of-the-art results for En-De and En-Fr ST tasks by obtaining 9.18, and 11.76 BLEU point improvements, respectively.
GRASS: Graph Spectral Sparsification Leveraging Scalable Spectral Perturbation Analysis
Venue: arXiv cs.DS 2020
Year: 2020

Spectral graph sparsification aims to find ultra-sparse subgraphs whose Laplacian matrix can well approximate the original Laplacian eigenvalues and eigenvectors. In recent years, spectral sparsification techniques have been extensively studied for accelerating various numerical and graph-related applications. Prior nearly-linear-time spectral sparsification methods first extract low-stretch spanning tree from the original graph to form the backbone of the sparsifier, and then recover small portions of spectrally-critical off-tree edges to the spanning tree to significantly improve the approximation quality. However, it is not clear how many off-tree edges should be recovered for achieving a desired spectral similarity level within the sparsifier. Motivated by recent graph signal processing techniques, this paper proposes a similarity-aware spectral graph sparsification framework that leverages efficient spectral off-tree edge embedding and filtering schemes to construct spectral sparsifiers with guaranteed spectral similarity (relative condition number) level. An iterative graph densification scheme is also introduced to facilitate efficient and effective filtering of off-tree edges for highly ill-conditioned problems. The proposed method has been validated using various kinds of graphs obtained from public domain sparse matrix collections relevant to VLSI CAD, finite element analysis, as well as social and data networks frequently studied in many machine learning and data mining applications. For instance, a sparse SDD matrix with 40 million unknowns and 180 million nonzeros can be solved (1E-3 accuracy level) within two minutes using a single CPU core and about 6GB memory.
A many-to-many assignment game and stable outcome algorithm to evaluate collaborative Mobility-as-a-Service platforms
Venue: arXiv cs.CY 2020
Year: 2020

As Mobility as a Service (MaaS) systems become increasingly popular, travel is changing from unimodal trips to personalized services offered by a platform of mobility operators. Evaluation of MaaS platforms depends on modeling both user route decisions as well as operator service and pricing decisions. We adopt a new paradigm for traffic assignment in a MaaS network of multiple operators using the concept of stable matching to allocate costs and determine prices offered by operators corresponding to user route choices and operator service choices without resorting to nonconvex bilevel programming formulations. Unlike our prior work, the proposed model allows travelers to make multimodal, multi-operator trips, resulting in stable cost allocations between competing network operators to provide MaaS for users. An algorithm is proposed to efficiently generate stability conditions for the stable outcome model. Extensive computational experiments demonstrate the use of the model to handling pricing responses of MaaS operators in technological and capacity changes, government acquisition, consolidation, and firm entry, using the classic Sioux Falls network. The proposed algorithm replicates the same stability conditions as explicit path enumeration while taking only 17 seconds compared to explicit path enumeration timing out over 2 hours.
Matchings in 1-planar graphs with large minimum degree
Venue: arXiv cs.DM 2020
Year: 2020

In 1979, Nishizeki and Baybars showed that every planar graph with minimum degree 3 has a matching of size $\frac{n}{3}+c$ (where the constant $c$ depends on the connectivity), and even better bounds hold for planar graphs with minimum degree 4 and 5. In this paper, we investigate similar matching-bounds for {\em 1-planar} graphs, i.e., graphs that can be drawn such that every edge has at most one crossing. We show that every 1-planar graph with minimum degree 3 has a matching of size at least $\frac{1}{7}n+\frac{12}{7}$, and this is tight for some graphs. We provide similar bounds for 1-planar graphs with minimum degree 4 and 5, while the case of minimum degree 6 and 7 remains open.
Object-Centric Task and Motion Planning in Dynamic Environments
Venue: arXiv cs.RO 2020
Year: 2020

We address the problem of applying Task and Motion Planning (TAMP) in real world environments. TAMP combines symbolic and geometric reasoning to produce sequential manipulation plans, typically specified as joint-space trajectories, which are valid only as long as the environment is static and perception and control are highly accurate. In case of any changes in the environment, slow re-planning is required. We propose a TAMP algorithm that optimizes over Cartesian frames defined relative to target objects. The resulting plan then remains valid even if the objects are moving and can be executed by reactive controllers that adapt to these changes in real time. We apply our TAMP framework to a torque-controlled robot in a pick and place setting and demonstrate its ability to adapt to changing environments, inaccurate perception, and imprecise control, both in simulation and the real world.
Online Stochastic Matching with Edge Arrivals
Venue: arXiv cs.DS 2020
Year: 2020

Online bipartite matching with edge arrivals remained a major open question for a long time until a recent negative result by [Gamlath et al. FOCS 2019], who showed that no online policy is better than the straightforward greedy algorithm, i.e., no online algorithm has a worst-case competitive ratio better than $0.5$. In this work, we consider the bipartite matching problem with edge arrivals in a natural stochastic framework, i.e., Bayesian setting where each edge of the graph is independently realized according to a known probability distribution. We focus on a natural class of prune & greedy online policies motivated by practical considerations from a multitude of online matching platforms. Any prune & greedy algorithm consists of two stages: first, it decreases the probabilities of some edges in the stochastic instance and then runs greedy algorithm on the pruned graph. We propose prune & greedy algorithms that are $0.552$-competitive on the instances that can be pruned to a $2$-regular stochastic bipartite graph, and $0.503$-competitive on arbitrary bipartite graphs. The algorithms and our analysis significantly deviate from the prior work. We first obtain analytically manageable lower bound on the size of the matching, which leads to a non linear optimization problem. We further reduce this problem to a continuous optimization with a constant number of parameters that can be solved using standard software tools.
Machine Intelligence at the Edge with Learning Centric Power Allocation
Venue: arXiv cs.IT 2020
Year: 2020

While machine-type communication (MTC) devices generate considerable amounts of data, they often cannot process the data due to limited energy and computational power. To empower MTC with intelligence, edge machine learning has been proposed. However, power allocation in this paradigm requires maximizing the learning performance instead of the communication throughput, for which the celebrated water-filling and max-min fairness algorithms become inefficient. To this end, this paper proposes learning centric power allocation (LCPA), which provides a new perspective on radio resource allocation in learning driven scenarios. By employing 1) an empirical classification error model that is supported by learning theory and 2) an uncertainty sampling method that accounts for different distributions at users, LCPA is formulated as a nonconvex nonsmooth optimization problem, and is solved using a majorization minimization (MM) framework. To get deeper insights into LCPA, asymptotic analysis shows that the transmit powers are inversely proportional to the channel gains, and scale exponentially with the learning parameters. This is in contrast to traditional power allocations where quality of wireless channels is the only consideration. Last but not least, a large-scale optimization algorithm termed mirror-prox LCPA is further proposed to enable LCPA in large-scale settings. Extensive numerical results demonstrate that the proposed LCPA algorithms outperform traditional power allocation algorithms, and the large-scale optimization algorithm reduces the computation time by orders of magnitude compared with MM-based LCPA but still achieves competing learning performance.
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB
Venue: arXiv cs.CL 2020
Year: 2020

We show that margin-based bitext mining in a multilingual sentence space can be applied to monolingual corpora of billions of sentences. We are using ten snapshots of a curated common crawl corpus (Wenzek et al., 2019) totalling 32.7 billion unique sentences. Using one unified approach for 38 languages, we were able to mine 4.5 billions parallel sentences, out of which 661 million are aligned with English. 20 language pairs have more then 30 million parallel sentences, 112 more then 10 million, and most more than one million, including direct alignments between many European or Asian languages. To evaluate the quality of the mined bitexts, we train NMT systems for most of the language pairs and evaluate them on TED, WMT and WAT test sets. Using our mined bitexts only and no human translated parallel data, we achieve a new state-of-the-art for a single system on the WMT'19 test set for translation between English and German, Russian and Chinese, as well as German/French. In particular, our English/German system outperforms the best single one by close to 4 BLEU points and is almost on pair with best WMT'19 evaluation system which uses system combination and back-translation. We also achieve excellent results for distant languages pairs like Russian/Japanese, outperforming the best submission at the 2019 workshop on Asian Translation (WAT).
Persistent Graphs and Cyclic Polytope Triangulations
Venue: arXiv cs.DM 2020
Year: 2020

We prove a bijection between the triangulations of the 3-dimensional cyclic polytope C(n+2, 3) and persistent graphs with n vertices. We show that under this bijection the Stasheff-Tamari orders on triangulations naturally translate to subgraph inclusion between persistent graphs. Moreover, we describe a connection to the second higher Bruhat order B(n, 2). We additionally give an algorithm to efficiently enumerate all persistent graphs on n vertices and thus all triangulations of C(n+2, 3).