OpenRaiser/Intern-Atlas huggingface dataset:
Layer-stacked Attention for Heterogeneous Network Embedding
Venue: arXiv cs.LG 2020
Year: 2020

The heterogeneous network is a robust data abstraction that can model entities of different types interacting in various ways. Such heterogeneity brings rich semantic information but presents nontrivial challenges in aggregating the heterogeneous relationships between objects - especially those of higher-order indirect relations. Recent graph neural network approaches for representation learning on heterogeneous networks typically employ the attention mechanism, which is often only optimized for predictions based on direct links. Furthermore, even though most deep learning methods can aggregate higher-order information by building deeper models, such a scheme can diminish the degree of interpretability. To overcome these challenges, we explore an architecture - Layer-stacked ATTention Embedding (LATTE) - that automatically decomposes higher-order meta relations at each layer to extract the relevant heterogeneous neighborhood structures for each node. Additionally, by successively stacking layer representations, the learned node embedding offers a more interpretable aggregation scheme for nodes of different types at different neighborhood ranges. We conducted experiments on several benchmark heterogeneous network datasets. In both transductive and inductive node classification tasks, LATTE can achieve state-of-the-art performance compared to existing approaches, all while offering a lightweight model. With extensive experimental analyses and visualizations, the framework can demonstrate the ability to extract informative insights on heterogeneous networks.
CARONTE: Crawling Adversarial Resources Over Non-Trusted, High-Profile Environments
Venue: arXiv cs.CR 2020
Year: 2020

The monitoring of underground criminal activities is often automated to maximize the data collection and to train ML models to automatically adapt data collection tools to different communities. On the other hand, sophisticated adversaries may adopt crawling-detection capabilities that may significantly jeopardize researchers' opportunities to perform the data collection, for example by putting their accounts under the spotlight and being expelled from the community. This is particularly undesirable in prominent and high-profile criminal communities where entry costs are significant (either monetarily or for example for background checking or other trust-building mechanisms). This paper presents CARONTE, a tool to semi-automatically learn virtually any forum structure for parsing and data-extraction, while maintaining a low profile for the data collection and avoiding the requirement of collecting massive datasets to maintain tool scalability. We showcase the tool against four underground forums, and compare the network traffic it generates (as seen from the adversary's position, i.e. the underground community's server) against state-of-the-art tools for web-crawling as well as human users.
Extensible Data Skipping
Venue: arXiv cs.DB 2020
Year: 2020

Data skipping reduces I/O for SQL queries by skipping over irrelevant data objects (files) based on their metadata. We extend this notion by allowing developers to define their own data skipping metadata types and indexes using a flexible API. Our framework is the first to natively support data skipping for arbitrary data types (e.g. geospatial, logs) and queries with User Defined Functions (UDFs). We integrated our framework with Apache Spark and it is now deployed across multiple products/services at IBM. We present our extensible data skipping APIs, discuss index design, and implement various metadata indexes, requiring only around 30 lines of additional code per index. In particular we implement data skipping for a third party library with geospatial UDFs and demonstrate speedups of two orders of magnitude. Our centralized metadata approach provides a x3.6 speed up even when compared to queries which are rewritten to exploit Parquet min/max metadata. We demonstrate that extensible data skipping is applicable to broad class of applications, where user defined indexes achieve significant speedups and cost savings with very low development cost.
Holistic Filter Pruning for Efficient Deep Neural Networks
Venue: arXiv cs.LG 2020
Year: 2020

Deep neural networks (DNNs) are usually over-parameterized to increase the likelihood of getting adequate initial weights by random initialization. Consequently, trained DNNs have many redundancies which can be pruned from the model to reduce complexity and improve the ability to generalize. Structural sparsity, as achieved by filter pruning, directly reduces the tensor sizes of weights and activations and is thus particularly effective for reducing complexity. We propose "Holistic Filter Pruning" (HFP), a novel approach for common DNN training that is easy to implement and enables to specify accurate pruning rates for the number of both parameters and multiplications. After each forward pass, the current model complexity is calculated and compared to the desired target size. By gradient descent, a global solution can be found that allocates the pruning budget over the individual layers such that the desired target size is fulfilled. In various experiments, we give insights into the training and achieve state-of-the-art performance on CIFAR-10 and ImageNet (HFP prunes 60% of the multiplications of ResNet-50 on ImageNet with no significant loss in the accuracy). We believe our simple and powerful pruning approach to constitute a valuable contribution for users of DNNs in low-cost applications.
DSC IIT-ISM at SemEval-2020 Task 6: Boosting BERT with Dependencies for Definition Extraction
Venue: arXiv cs.CL 2020
Year: 2020

We explore the performance of Bidirectional Encoder Representations from Transformers (BERT) at definition extraction. We further propose a joint model of BERT and Text Level Graph Convolutional Network so as to incorporate dependencies into the model. Our proposed model produces better results than BERT and achieves comparable results to BERT with fine tuned language model in DeftEval (Task 6 of SemEval 2020), a shared task of classifying whether a sentence contains a definition or not (Subtask 1).
Deploying machine learning to assist digital humanitarians: making image annotation in OpenStreetMap more efficient
Venue: arXiv cs.CV 2020
Year: 2020

Locating populations in rural areas of developing countries has attracted the attention of humanitarian mapping projects since it is important to plan actions that affect vulnerable areas. Recent efforts have tackled this problem as the detection of buildings in aerial images. However, the quality and the amount of rural building annotated data in open mapping services like OpenStreetMap (OSM) is not sufficient for training accurate models for such detection. Although these methods have the potential of aiding in the update of rural building information, they are not accurate enough to automatically update the rural building maps. In this paper, we explore a human-computer interaction approach and propose an interactive method to support and optimize the work of volunteers in OSM. The user is asked to verify/correct the annotation of selected tiles during several iterations and therefore improving the model with the new annotated data. The experimental results, with simulated and real user annotation corrections, show that the proposed method greatly reduces the amount of data that the volunteers of OSM need to verify/correct. The proposed methodology could benefit humanitarian mapping projects, not only by making more efficient the process of annotation but also by improving the engagement of volunteers.
Vax-a-Net: Training-time Defence Against Adversarial Patch Attacks
Venue: arXiv cs.CV 2020
Year: 2020

We present Vax-a-Net; a technique for immunizing convolutional neural networks (CNNs) against adversarial patch attacks (APAs). APAs insert visually overt, local regions (patches) into an image to induce misclassification. We introduce a conditional Generative Adversarial Network (GAN) architecture that simultaneously learns to synthesise patches for use in APAs, whilst exploiting those attacks to adapt a pre-trained target CNN to reduce its susceptibility to them. This approach enables resilience against APAs to be conferred to pre-trained models, which would be impractical with conventional adversarial training due to the slow convergence of APA methods. We demonstrate transferability of this protection to defend against existing APAs, and show its efficacy across several contemporary CNN architectures.
Accelerated solving of coupled, non-linear ODEs through LSTM-AI
Venue: arXiv cs.LG 2020
Year: 2020

The present project aims to use machine learning, specifically neural networks (NN), to learn the trajectories of a set of coupled ordinary differential equations (ODEs) and decrease compute times for obtaining ODE solutions by using this surragate model. As an example system of proven biological significance, we use an ODE model of a gene regulatory circuit of cyanobacteria related to photosynthesis \cite{original_biology_Kehoe, Sundus_math_model}. Using data generated by a numeric solution to the exemplar system, we train several long-short-term memory neural networks. We stopping training when the networks achieve an accuracy of of 3\% on testing data resulting in networks able to predict values in the ODE time series ranging from 0.25 minutes to 6.25 minutes beyond input values. We observed computational speed ups ranging from 9.75 to 197 times when comparing prediction compute time with compute time for obtaining the numeric solution. Given the success of this proof of concept, we plan on continuing this project in the future and will attempt to realize the same computational speed-ups in the context of an agent-based modeling platfom.
Low-Rank Matrix Recovery from Noise via an MDL Framework-based Atomic Norm
Venue: arXiv cs.CV 2020
Year: 2020

The recovery of the underlying low-rank structure of clean data corrupted with sparse noise/outliers is attracting increasing interest. However, in many low-level vision problems, the exact target rank of the underlying structure and the particular locations and values of the sparse outliers are not known. Thus, the conventional methods cannot separate the low-rank and sparse components completely, especially in the case of gross outliers or deficient observations. Therefore, in this study, we employ the minimum description length (MDL) principle and atomic norm for low-rank matrix recovery to overcome these limitations. First, we employ the atomic norm to find all the candidate atoms of low-rank and sparse terms, and then we minimize the description length of the model in order to select the appropriate atoms of low-rank and the sparse matrices, respectively. Our experimental analyses show that the proposed approach can obtain a higher success rate than the state-of-the-art methods, even when the number of observations is limited or the corruption ratio is high. Experimental results utilizing synthetic data and real sensing applications (high dynamic range imaging, background modeling, removing noise and shadows) demonstrate the effectiveness, robustness and efficiency of the proposed method.
Evolutionary Selective Imitation: Interpretable Agents by Imitation Learning Without a Demonstrator
Venue: arXiv cs.NE 2020
Year: 2020

We propose a new method for training an agent via an evolutionary strategy (ES), in which we iteratively improve a set of samples to imitate: Starting with a random set, in every iteration we replace a subset of the samples with samples from the best trajectories discovered so far. The evaluation procedure for this set is to train, via supervised learning, a randomly initialised neural network (NN) to imitate the set and then execute the acquired policy against the environment. Our method is thus an ES based on a fitness function that expresses the effectiveness of imitating an evolving data subset. This is in contrast to other ES techniques that iterate over the weights of the policy directly. By observing the samples that the agent selects for learning, it is possible to interpret and evaluate the evolving strategy of the agent more explicitly than in NN learning. In our experiments, we trained an agent to solve the OpenAI Gym environment Bipedalwalker-v3 by imitating an evolutionarily selected set of only 25 samples with a NN with only a few thousand parameters. We further test our method on the Procgen game Plunder and show here as well that the proposed method is an interpretable, small, robust and effective alternative to other ES or policy gradient methods.
Competitiveness of MAP-Elites against Proximal Policy Optimization on locomotion tasks in deterministic simulations
Venue: arXiv cs.AI 2020
Year: 2020

The increasing importance of robots and automation creates a demand for learnable controllers which can be obtained through various approaches such as Evolutionary Algorithms (EAs) or Reinforcement Learning (RL). Unfortunately, these two families of algorithms have mainly developed independently and there are only a few works comparing modern EAs with deep RL algorithms. We show that Multidimensional Archive of Phenotypic Elites (MAP-Elites), which is a modern EA, can deliver better-performing solutions than one of the state-of-the-art RL methods, Proximal Policy Optimization (PPO) in the generation of locomotion controllers for a simulated hexapod robot. Additionally, extensive hyper-parameter tuning shows that MAP-Elites displays greater robustness across seeds and hyper-parameter sets. Generally, this paper demonstrates that EAs combined with modern computational resources display promising characteristics and have the potential to contribute to the state-of-the-art in controller learning.
The Next Big Thing(s) in Unsupervised Machine Learning: Five Lessons from Infant Learning
Venue: arXiv cs.LG 2020
Year: 2020

After a surge in popularity of supervised Deep Learning, the desire to reduce the dependence on curated, labelled data sets and to leverage the vast quantities of unlabelled data available recently triggered renewed interest in unsupervised learning algorithms. Despite a significantly improved performance due to approaches such as the identification of disentangled latent representations, contrastive learning, and clustering optimisations, the performance of unsupervised machine learning still falls short of its hypothesised potential. Machine learning has previously taken inspiration from neuroscience and cognitive science with great success. However, this has mostly been based on adult learners with access to labels and a vast amount of prior knowledge. In order to push unsupervised machine learning forward, we argue that developmental science of infant cognition might hold the key to unlocking the next generation of unsupervised learning approaches. Conceptually, human infant learning is the closest biological parallel to artificial unsupervised learning, as infants too must learn useful representations from unlabelled data. In contrast to machine learning, these new representations are learned rapidly and from relatively few examples. Moreover, infants learn robust representations that can be used flexibly and efficiently in a number of different tasks and contexts. We identify five crucial factors enabling infants' quality and speed of learning, assess the extent to which these have already been exploited in machine learning, and propose how further adoption of these factors can give rise to previously unseen performance levels in unsupervised learning.
Intelligent Reflecting Surface Aided Pilot Contamination Attack and Its Countermeasure
Venue: arXiv cs.IT 2020
Year: 2020

Pilot contamination attack (PCA) in a time division duplex wireless communication system is considered, where an eavesdropper (Eve) attacks the reverse pilot transmission phase in order to wiretap the data transmitted from a transmitter, Alice, to a receiver, Bob. We propose a new PCA scheme for Eve, wherein Eve does not emit any signal by itself but uses an intelligent reflecting surface (IRS) to reflect the pilot sent by Bob to Alice. The proposed new PCA scheme, referred to as IRS-PCA, increases the signal leakage from Alice to the IRS during the data transmission phase, which is then reflected by the IRS to Eve in order to improve the wiretapping capability of Eve. The proposed IRS-PCA scheme disables many existing countermeasures on PCA due to the fact that with IRS-PCA, Eve no longer needs to know the pilot sequence of Bob, and therefore, poses severe threat to the security of the legitimate wireless communication system. In view of this, the problems of 1) IRS-PCA detection and 2) secure transmission under IRSPCA are considered in this paper. For IRS-PCA detection, a generalized cumulative sum (GCUSUM) detection procedure is proposed based on the framework of quickest detection, aiming at detecting the occurrence of IRS-PCA as soon as possible once it occurs. For secure transmission under IRS-PCA, a cooperative channel estimation scheme is proposed to estimate the channel of the IRS, based on which zero-forcing beamforming is designed to reduce signal leakage.
Ensemble of Binary Classifiers Combined Using Recurrent Correlation Associative Memories
Venue: arXiv cs.LG 2020
Year: 2020

An ensemble method should cleverly combine a group of base classifiers to yield an improved classifier. The majority vote is an example of a methodology used to combine classifiers in an ensemble method. In this paper, we propose to combine classifiers using an associative memory model. Precisely, we introduce ensemble methods based on recurrent correlation associative memories (RCAMs) for binary classification problems. We show that an RCAM-based ensemble classifier can be viewed as a majority vote classifier whose weights depend on the similarity between the base classifiers and the resulting ensemble method. More precisely, the RCAM-based ensemble combines the classifiers using a recurrent consult and vote scheme. Furthermore, computational experiments confirm the potential application of the RCAM-based ensemble method for binary classification problems.
NEU at WNUT-2020 Task 2: Data Augmentation To Tell BERT That Death Is Not Necessarily Informative
Venue: arXiv cs.CL 2020
Year: 2020

Millions of people around the world are sharing COVID-19 related information on social media platforms. Since not all the information shared on the social media is useful, a machine learning system to identify informative posts can help users in finding relevant information. In this paper, we present a BERT classifier system for W-NUT2020 Shared Task 2: Identification of Informative COVID-19 English Tweets. Further, we show that BERT exploits some easy signals to identify informative tweets, and adding simple patterns to uninformative tweets drastically degrades BERT performance. In particular, simply adding 10 deaths to tweets in dev set, reduces BERT F1- score from 92.63 to 7.28. We also propose a simple data augmentation technique that helps in improving the robustness and generalization ability of the BERT classifier.
Decentralized Game-Theoretic Control for Dynamic Task Allocation Problems for Multi-Agent Systems
Venue: arXiv cs.MA 2020
Year: 2020

We propose a decentralized game-theoretic framework for dynamic task allocation problems for multi-agent systems. In our problem formulation, the agents' utilities depend on both the rewards and the costs associated with the successful completion of the tasks assigned to them. The rewards reflect how likely is for the agents to accomplish their assigned tasks whereas the costs reflect the effort needed to complete these tasks (this effort is determined by the solution of corresponding optimal control problems). The task allocation problem considered herein corresponds to a dynamic game whose solution depends on the states of the agents in contrast with classic static (or single-act) game formulations. We propose a greedy solution approach in which the agents negotiate with each other to find a mutually agreeable (or individually rational) task assignment profile based on evaluations of the task utilities that reflect their current states. We illustrate the main ideas of this work by means of extensive numerical simulations.
A Social Network of Russian "Kompromat"
Venue: arXiv cs.SI 2020
Year: 2020

"Kompromat" (the Russian word for "compromising material") has been efficiently used to harass Russian political and business elites since the days of the USSR. Online crowdsourcing projects such as "RuCompromat" made it possible to catalog and analyze kompromat using quantitative techniques -- namely, social network analysis. In this paper, we constructed a social network of 11,000 Russian and foreign nationals affected by kompromat in Russia in 1991 -- 2020. The network has an excellent modular structure with 62 dense communities. One community contains prominent American officials, politicians, and entrepreneurs (including President Donald Trump) and appears to concern Russia's controversial interference in the 2016 U.S. presidential elections. Various network centrality measures identify seventeen most central kompromat figures, with President Vladimir Putin solidly at the top. We further reveal four types of communities dominated by entrepreneurs, politicians, bankers, and law enforcement officials ("siloviks"), the latter disjointed from the first three.
Gateway Controller with Deep Sensing: Learning to be Autonomic in Intelligent Internet of Things
Venue: arXiv cs.SE 2020
Year: 2020

The Internet of Things(IoT) will revolutionize the Future Internet through ubiquitous sensing. One of the challenges of having the hundreds of billions of devices that are estimated to be deployed would be rise of an enormous amount of data, along with the devices ability to manage. This paper presents an approach as a controller solution and designed specifically for autonomous management, connectivity and data interoperability in an IoT gateway. The approach supports distributed IoT nodes with both management and data interoperability with other cloud-based solutions. The concept further allows gateways to easily collect and process interoperability of data from IoT devices. We demonstrated the feasibility of the approach and evaluate its advantages regarding deep sensing and autonomous enabled gateway as an edge computational intelligence.
Monte Carlo Tree Search Based Tactical Maneuvering
Venue: arXiv cs.AI 2020
Year: 2020

In this paper we explore the application of simultaneous move Monte Carlo Tree Search (MCTS) based online framework for tactical maneuvering between two unmanned aircrafts. Compared to other techniques, MCTS enables efficient search over long horizons and uses self-play to select best maneuver in the current state while accounting for the opponent aircraft tactics. We explore different algorithmic choices in MCTS and demonstrate the framework numerically in a simulated 2D tactical maneuvering application.