OpenRaiser/Intern-Atlas huggingface dataset:
Local Connectivity in Centroid Clustering
Venue: arXiv cs.LG 2020
Year: 2020

Clustering is a fundamental task in unsupervised learning, one that targets to group a dataset into clusters of similar objects. There has been recent interest in embedding normative considerations around fairness within clustering formulations. In this paper, we propose 'local connectivity' as a crucial factor in assessing membership desert in centroid clustering. We use local connectivity to refer to the support offered by the local neighborhood of an object towards supporting its membership to the cluster in question. We motivate the need to consider local connectivity of objects in cluster assignment, and provide ways to quantify local connectivity in a given clustering. We then exploit concepts from density-based clustering and devise LOFKM, a clustering method that seeks to deepen local connectivity in clustering outputs, while staying within the framework of centroid clustering. Through an empirical evaluation over real-world datasets, we illustrate that LOFKM achieves notable improvements in local connectivity at reasonable costs to clustering quality, illustrating the effectiveness of the method.
A BERT-based Distractor Generation Scheme with Multi-tasking and Negative Answer Training Strategies
Venue: arXiv cs.CL 2020
Year: 2020

In this paper, we investigate the following two limitations for the existing distractor generation (DG) methods. First, the quality of the existing DG methods are still far from practical use. There is still room for DG quality improvement. Second, the existing DG designs are mainly for single distractor generation. However, for practical MCQ preparation, multiple distractors are desired. Aiming at these goals, in this paper, we present a new distractor generation scheme with multi-tasking and negative answer training strategies for effectively generating \textit{multiple} distractors. The experimental results show that (1) our model advances the state-of-the-art result from 28.65 to 39.81 (BLEU 1 score) and (2) the generated multiple distractors are diverse and show strong distracting power for multiple choice question.
New P$c$N and AP$c$N functions over finite fields
Venue: arXiv cs.IT 2020
Year: 2020

Functions with low $c$-differential uniformity were proposed in $2020$ and attracted lots of attention, especially the P$c$N and AP$c$N functions, due to their applications in cryptography. The objective of this paper is to study P$c$N and AP$c$N functions. As a consequence, we propose a class of P$c$N functions and four classes of AP$c$N functions by using the cyclotomic technique and the switch method. In addition, four classes of P$c$N or AP$c$N functions are presented by virtue of (generalized) AGW criterion.
Unweighted linear congruences with distinct coordinates and the Varshamov--Tenengolts codes
Venue: arXiv cs.IT 2020
Year: 2020

In this paper, we first give explicit formulas for the number of solutions of unweighted linear congruences with distinct coordinates. Our main tools are properties of Ramanujan sums and of the discrete Fourier transform of arithmetic functions. Then, as an application, we derive an explicit formula for the number of codewords in the Varshamov--Tenengolts code $VT_b(n)$ with Hamming weight $k$, that is, with exactly $k$ $1$'s. The Varshamov--Tenengolts codes are an important class of codes that are capable of correcting asymmetric errors on a $Z$-channel. As another application, we derive Ginzburg's formula for the number of codewords in $VT_b(n)$, that is, $|VT_b(n)|$. We even go further and discuss connections to several other combinatorial problems, some of which have appeared in seemingly unrelated contexts. This provides a general framework and gives new insight into all these problems which might lead to further work.
Spacetime Autoencoders Using Local Causal States
Venue: arXiv cs.LG 2020
Year: 2020

Local causal states are latent representations that capture organized pattern and structure in complex spatiotemporal systems. We expand their functionality, framing them as spacetime autoencoders. Previously, they were only considered as maps from observable spacetime fields to latent local causal state fields. Here, we show that there is a stochastic decoding that maps back from the latent fields to observable fields. Furthermore, their Markovian properties define a stochastic dynamic in the latent space. Combined with stochastic decoding, this gives a new method for forecasting spacetime fields.
PAST-AI: Physical-layer Authentication of Satellite Transmitters via Deep Learning
Venue: arXiv cs.CR 2020
Year: 2020

Physical-layer security is regaining traction in the research community, due to the performance boost introduced by deep learning classification algorithms. This is particularly true for sender authentication in wireless communications via radio fingerprinting. However, previous research efforts mainly focused on terrestrial wireless devices while, to the best of our knowledge, none of the previous work took into consideration satellite transmitters. The satellite scenario is generally challenging because, among others, satellite radio transducers feature non-standard electronics (usually aged and specifically designed for harsh conditions). Moreover, the fingerprinting task is specifically difficult for Low-Earth Orbit (LEO) satellites (like the ones we focus in this paper) since they orbit at about 800Km from the Earth, at a speed of around 25,000Km/h, thus making the receiver experiencing a down-link with unique attenuation and fading characteristics. In this paper, we propose PAST-AI, a methodology tailored to authenticate LEO satellites through fingerprinting of their IQ samples, using advanced AI solutions. Our methodology is tested on real data -- more than 100M I/Q samples -- collected from an extensive measurements campaign on the IRIDIUM LEO satellites constellation, lasting 589 hours. Results are striking: we prove that Convolutional Neural Networks (CNN) and autoencoders (if properly calibrated) can be successfully adopted to authenticate the satellite transducers, with an accuracy spanning between 0.8 and 1, depending on prior assumptions. The proposed methodology, the achieved results, and the provided insights, other than being interesting on their own, when associated to the dataset that we made publicly available, will also pave the way for future research in the area.
A Sentiment-Controllable Topic-to-Essay Generator with Topic Knowledge Graph
Venue: arXiv cs.CL 2020
Year: 2020

Generating a vivid, novel, and diverse essay with only several given topic words is a challenging task of natural language generation. In previous work, there are two problems left unsolved: neglect of sentiment beneath the text and insufficient utilization of topic-related knowledge. Therefore, we propose a novel Sentiment-Controllable topic-to-essay generator with a Topic Knowledge Graph enhanced decoder, named SCTKG, which is based on the conditional variational autoencoder (CVAE) framework. We firstly inject the sentiment information into the generator for controlling sentiment for each sentence, which leads to various generated essays. Then we design a Topic Knowledge Graph enhanced decoder. Unlike existing models that use knowledge entities separately, our model treats the knowledge graph as a whole and encodes more structured, connected semantic information in the graph to generate a more relevant essay. Experimental results show that our SCTKG can generate sentiment controllable essays and outperform the state-of-the-art approach in terms of topic relevance, fluency, and diversity on both automatic and human evaluation.
Automatic Quantification of Settlement Damage using Deep Learning of Satellite Images
Venue: arXiv cs.CV 2020
Year: 2020

Humanitarian disasters and political violence cause significant damage to our living space. The reparation cost to homes, infrastructure, and the ecosystem is often difficult to quantify in real-time. Real-time quantification is critical to both informing relief operations, but also planning ahead for rebuilding. Here, we use satellite images before and after major crisis around the world to train a robust baseline Residual Network (ResNet) and a disaster quantification Pyramid Scene Parsing Network (PSPNet). ResNet offers robustness to poor image quality and can identify areas of destruction with high accuracy (92\%), whereas PSPNet offers contextualised quantification of built environment damage with good accuracy (84\%). As there are multiple damage dimensions to consider (e.g. economic loss and fatalities), we fit a multi-linear regression model to quantify the overall damage. To validate our combined system of deep learning and regression modeling, we successfully match our prediction to the ongoing recovery in the 2020 Beirut port explosion. These innovations provide a better quantification of overall disaster magnitude and inform intelligent humanitarian systems of unfolding disasters.
Carbon to Diamond: An Incident Remediation Assistant System From Site Reliability Engineers' Conversations in Hybrid Cloud Operations
Venue: arXiv cs.CL 2020
Year: 2020

Conversational channels are changing the landscape of hybrid cloud service management. These channels are becoming important avenues for Site Reliability Engineers (SREs) %Subject Matter Experts (SME) to collaboratively work together to resolve an incident or issue. Identifying segmented conversations and extracting key insights or artefacts from them can help engineers to improve the efficiency of the incident remediation process by using information retrieval mechanisms for similar incidents. However, it has been empirically observed that due to the semi-formal behavior of such conversations (human language) they are very unique in nature and also contain lot of domain-specific terms. This makes it difficult to use the standard natural language processing frameworks directly, which are popularly used in standard NLP tasks. %It is important to identify the correct keywords and artefacts like symptoms, issue etc., present in the conversation chats. In this paper, we build a framework that taps into the conversational channels and uses various learning methods to (a) understand and extract key artefacts from conversations like diagnostic steps and resolution actions taken, and (b) present an approach to identify past conversations about similar issues. Experimental results on our dataset show the efficacy of our proposed method.
Social Commonsense Reasoning with Multi-Head Knowledge Attention
Venue: arXiv cs.CL 2020
Year: 2020

Social Commonsense Reasoning requires understanding of text, knowledge about social events and their pragmatic implications, as well as commonsense reasoning skills. In this work we propose a novel multi-head knowledge attention model that encodes semi-structured commonsense inference rules and learns to incorporate them in a transformer-based reasoning cell. We assess the model's performance on two tasks that require different reasoning skills: Abductive Natural Language Inference and Counterfactual Invariance Prediction as a new task. We show that our proposed model improves performance over strong state-of-the-art models (i.e., RoBERTa) across both reasoning tasks. Notably we are, to the best of our knowledge, the first to demonstrate that a model that learns to perform counterfactual reasoning helps predicting the best explanation in an abductive reasoning task. We validate the robustness of the model's reasoning capabilities by perturbing the knowledge and provide qualitative analysis on the model's knowledge incorporation capabilities.
The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?
Venue: arXiv cs.CL 2020
Year: 2020

There is a recent surge of interest in using attention as explanation of model predictions, with mixed evidence on whether attention can be used as such. While attention conveniently gives us one weight per input token and is easily extracted, it is often unclear toward what goal it is used as explanation. We find that often that goal, whether explicitly stated or not, is to find out what input tokens are the most relevant to a prediction, and that the implied user for the explanation is a model developer. For this goal and user, we argue that input saliency methods are better suited, and that there are no compelling reasons to use attention, despite the coincidence that it provides a weight for each input. With this position paper, we hope to shift some of the recent focus on attention to saliency methods, and for authors to clearly state the goal and user for their explanations.
A Unified Framework for Generic, Query-Focused, Privacy Preserving and Update Summarization using Submodular Information Measures
Venue: arXiv cs.LG 2020
Year: 2020

We study submodular information measures as a rich framework for generic, query-focused, privacy sensitive, and update summarization tasks. While past work generally treats these problems differently ({\em e.g.}, different models are often used for generic and query-focused summarization), the submodular information measures allow us to study each of these problems via a unified approach. We first show that several previous query-focused and update summarization techniques have, unknowingly, used various instantiations of the aforesaid submodular information measures, providing evidence for the benefit and naturalness of these models. We then carefully study and demonstrate the modelling capabilities of the proposed functions in different settings and empirically verify our findings on both a synthetic dataset and an existing real-world image collection dataset (that has been extended by adding concept annotations to each image making it suitable for this task) and will be publicly released. We employ a max-margin framework to learn a mixture model built using the proposed instantiations of submodular information measures and demonstrate the effectiveness of our approach. While our experiments are in the context of image summarization, our framework is generic and can be easily extended to other summarization settings (e.g., videos or documents).
From Time Series to Euclidean Spaces: On Spatial Transformations for Temporal Clustering
Venue: arXiv cs.LG 2020
Year: 2020

Unsupervised clustering of temporal data is both challenging and crucial in machine learning. In this paper, we show that neither traditional clustering methods, time series specific or even deep learning-based alternatives generalise well when both varying sampling rates and high dimensionality are present in the input data. We propose a novel approach to temporal clustering, in which we (1) transform the input time series into a distance-based projected representation by using similarity measures suitable for dealing with temporal data,(2) feed these projections into a multi-layer CNN-GRU autoencoder to generate meaningful domain-aware latent representations, which ultimately (3) allow for a natural separation of clusters beneficial for most important traditional clustering algorithms. We evaluate our approach on time series datasets from various domains and show that it not only outperforms existing methods in all cases, by up to 32%, but is also robust and incurs negligible computation overheads.
Cybersecurity Dynamics: A Foundation for the Science of Cybersecurity
Venue: arXiv cs.CR 2020
Year: 2020

Cybersecurity Dynamics is new concept that aims to achieve the modeling, analysis, quantification, and management of cybersecurity from a holistic perspective, rather than from a building-blocks perspective. It is centered at modeling and analyzing the attack-defense interactions in cyberspace, which cause a ``natural'' phenomenon -- the evolution of the global cybersecurity state. In this Chapter, we systematically introduce and review the Cybersecurity Dynamics foundation for the Science of Cybersecurity. We review the core concepts, technical approaches, research axes, and results that have been obtained in this endeavor. We outline a research roadmap towards the ultimate research goal, including a systematic set of technical barriers.
On the Complementary Nature of Knowledge Graph Embedding, Fine Grain Entity Types, and Language Modeling
Venue: arXiv cs.CL 2020
Year: 2020

We demonstrate the complementary natures of neural knowledge graph embedding, fine-grain entity type prediction, and neural language modeling. We show that a language model-inspired knowledge graph embedding approach yields both improved knowledge graph embeddings and fine-grain entity type representations. Our work also shows that jointly modeling both structured knowledge tuples and language improves both.
Security and Privacy Considerations for Machine Learning Models Deployed in the Government and Public Sector (white paper)
Venue: arXiv cs.CR 2020
Year: 2020

As machine learning becomes a more mainstream technology, the objective for governments and public sectors is to harness the power of machine learning to advance their mission by revolutionizing public services. Motivational government use cases require special considerations for implementation given the significance of the services they provide. Not only will these applications be deployed in a potentially hostile environment that necessitates protective mechanisms, but they are also subject to government transparency and accountability initiatives which further complicates such protections. In this paper, we describe how the inevitable interactions between a user of unknown trustworthiness and the machine learning models, deployed in governments and public sectors, can jeopardize the system in two major ways: by compromising the integrity or by violating the privacy. We then briefly overview the possible attacks and defense scenarios, and finally, propose recommendations and guidelines that once considered can enhance the security and privacy of the provided services.
Embedded methods for feature selection in neural networks
Venue: arXiv cs.LG 2020
Year: 2020

The representational capacity of modern neural network architectures has made them a default choice in various applications with high dimensional feature sets. But these high dimensional and potentially noisy features combined with the black box models like neural networks negatively affect the interpretability, generalizability, and the training time of these models. Here, I propose two integrated approaches for feature selection that can be incorporated directly into the parameter learning. One of them involves adding a drop-in layer and performing sequential weight pruning. The other is a sensitivity-based approach. I benchmarked both the methods against Permutation Feature Importance (PFI) - a general-purpose feature ranking method and a random baseline. The suggested approaches turn out to be viable methods for feature selection, consistently outperform the baselines on the tested datasets - MNIST, ISOLET, and HAR. We can add them to any existing model with only a few lines of code.
Differentially Private Secure Multi-Party Computation for Federated Learning in Financial Applications
Venue: arXiv cs.CR 2020
Year: 2020

Federated Learning enables a population of clients, working with a trusted server, to collaboratively learn a shared machine learning model while keeping each client's data within its own local systems. This reduces the risk of exposing sensitive data, but it is still possible to reverse engineer information about a client's private data set from communicated model parameters. Most federated learning systems therefore use differential privacy to introduce noise to the parameters. This adds uncertainty to any attempt to reveal private client data, but also reduces the accuracy of the shared model, limiting the useful scale of privacy-preserving noise. A system can further reduce the coordinating server's ability to recover private client information, without additional accuracy loss, by also including secure multiparty computation. An approach combining both techniques is especially relevant to financial firms as it allows new possibilities for collaborative learning without exposing sensitive client data. This could produce more accurate models for important tasks like optimal trade execution, credit origination, or fraud detection. The key contributions of this paper are: We present a privacy-preserving federated learning protocol to a non-specialist audience, demonstrate it using logistic regression on a real-world credit card fraud data set, and evaluate it using an open-source simulation platform which we have adapted for the development of federated learning systems.
Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
Venue: arXiv cs.CL 2020
Year: 2020

Massively multilingual models subsuming tens or even hundreds of languages pose great challenges to multi-task optimization. While it is a common practice to apply a language-agnostic procedure optimizing a joint multilingual task objective, how to properly characterize and take advantage of its underlying problem structure for improving optimization efficiency remains under-explored. In this paper, we attempt to peek into the black-box of multilingual optimization through the lens of loss function geometry. We find that gradient similarity measured along the optimization trajectory is an important signal, which correlates well with not only language proximity but also the overall model performance. Such observation helps us to identify a critical limitation of existing gradient-based multi-task learning methods, and thus we derive a simple and scalable optimization procedure, named Gradient Vaccine, which encourages more geometrically aligned parameter updates for close tasks. Empirically, our method obtains significant model performance gains on multilingual machine translation and XTREME benchmark tasks for multilingual language models. Our work reveals the importance of properly measuring and utilizing language proximity in multilingual optimization, and has broader implications for multi-task learning beyond multilingual modeling.
Nearly Minimax Optimal Reward-free Reinforcement Learning
Venue: arXiv cs.LG 2020
Year: 2020

We study the reward-free reinforcement learning framework, which is particularly suitable for batch reinforcement learning and scenarios where one needs policies for multiple reward functions. This framework has two phases. In the exploration phase, the agent collects trajectories by interacting with the environment without using any reward signal. In the planning phase, the agent needs to return a near-optimal policy for arbitrary reward functions. We give a new efficient algorithm, \textbf{S}taged \textbf{S}ampling + \textbf{T}runcated \textbf{P}lanning (\algoname), which interacts with the environment at most $O\left( \frac{S^2A}{ε^2}\text{poly}\log\left(\frac{SAH}ε\right) \right)$ episodes in the exploration phase, and guarantees to output a near-optimal policy for arbitrary reward functions in the planning phase. Here, $S$ is the size of state space, $A$ is the size of action space, $H$ is the planning horizon, and $ε$ is the target accuracy relative to the total reward. Notably, our sample complexity scales only \emph{logarithmically} with $H$, in contrast to all existing results which scale \emph{polynomially} with $H$. Furthermore, this bound matches the minimax lower bound $Ω\left(\frac{S^2A}{ε^2}\right)$ up to logarithmic factors. Our results rely on three new techniques : 1) A new sufficient condition for the dataset to plan for an $ε$-suboptimal policy; 2) A new way to plan efficiently under the proposed condition using soft-truncated planning; 3) Constructing extended MDP to maximize the truncated accumulative rewards efficiently.