OpenRaiser/Intern-Atlas huggingface dataset:
Streaming Temporal Graphs: Subgraph Matching
Venue: arXiv cs.PL 2020
Year: 2020

We investigate solutions to subgraph matching within a temporal stream of data. We present a high-level language for describing temporal subgraphs of interest, the Streaming Analytics Language (SAL). SAL programs are translated into C++ code that is run in parallel on a cluster. We call this implementation of SAL the Streaming Analytics Machine (SAM). SAL programs are succinct, requiring about 20 times fewer lines of code than using the SAM library directly, or writing an implementation using Apache Flink. To benchmark SAM we calculate finding temporal triangles within streaming netflow data. Also, we compare SAM to an implementation written for Flink. We find that SAM is able to scale to 128 nodes or 2560 cores, while Apache Flink has max throughput with 32 nodes and degrades thereafter. Apache Flink has an advantage when triangles are rare, with max aggregate throughput for Flink at 32 nodes greater than the max achievable rate of SAM. In our experiments, when triangle occurrence was faster than five per second per node, SAM performed better. Both frameworks may miss results due to latencies in network communication. SAM consistently reported an average of 93.7% of expected results while Flink decreases from 83.7% to 52.1% as we increase to the maximum size of the cluster. Overall, SAM can obtain rates of 91.8 billion netflows per day.
Efficient Implementation of Multi-Channel Convolution in Monolithic 3D ReRAM Crossbar
Venue: arXiv cs.AR 2020
Year: 2020

Convolutional neural networks (CNNs) demonstrate promising accuracy in a wide range of applications. Among all layers in CNNs, convolution layers are the most computation-intensive and consume the most energy. As the maturity of device and fabrication technology, 3D resistive random access memory (ReRAM) receives substantial attention for accelerating large vector-matrix multiplication and convolution due to its high parallelism and energy efficiency benefits. However, implementing multi-channel convolution naively in 3D ReRAM will either produce incorrect results or exploit only partial parallelism of 3D ReRAM. In this paper, we propose a 3D ReRAM-based convolution accelerator architecture, which efficiently maps multi-channel convolution to monolithic 3D ReRAM. Our design has two key principles. First, we exploit the intertwined structure of 3D ReRAM to implement multi-channel convolution by using a state-of-the-art convolution algorithm. Second, we propose a new approach to efficiently implement negative weights by separating them from non-negative weights using configurable interconnects. Our evaluation demonstrates that our mapping scheme in 16-layer 3D ReRAM achieves a speedup of 5.79X, 927.81X, and 36.8X compared with a custom 2D ReRAM baseline and state-of-the-art CPU and GPU. Our design also reduces energy consumption by 2.12X, 1802.64X, and 114.1X compared with the same baseline.
Self-Augmentation: Generalizing Deep Networks to Unseen Classes for Few-Shot Learning
Venue: arXiv cs.LG 2020
Year: 2020

Few-shot learning aims to classify unseen classes with a few training examples. While recent works have shown that standard mini-batch training with a carefully designed training strategy can improve generalization ability for unseen classes, well-known problems in deep networks such as memorizing training statistics have been less explored for few-shot learning. To tackle this issue, we propose self-augmentation that consolidates self-mix and self-distillation. Specifically, we exploit a regional dropout technique called self-mix, in which a patch of an image is substituted into other values in the same image. Then, we employ a backbone network that has auxiliary branches with its own classifier to enforce knowledge sharing. Lastly, we present a local representation learner to further exploit a few training examples for unseen classes. Experimental results show that the proposed method outperforms the state-of-the-art methods for prevalent few-shot benchmarks and improves the generalization ability.
Statistically Model Checking PCTL Specifications on Markov Decision Processes via Reinforcement Learning
Venue: arXiv cs.LG 2020
Year: 2020

Probabilistic Computation Tree Logic (PCTL) is frequently used to formally specify control objectives such as probabilistic reachability and safety. In this work, we focus on model checking PCTL specifications statistically on Markov Decision Processes (MDPs) by sampling, e.g., checking whether there exists a feasible policy such that the probability of reaching certain goal states is greater than a threshold. We use reinforcement learning to search for such a feasible policy for PCTL specifications, and then develop a statistical model checking (SMC) method with provable guarantees on its error. Specifically, we first use upper-confidence-bound (UCB) based Q-learning to design an SMC algorithm for bounded-time PCTL specifications, and then extend this algorithm to unbounded-time specifications by identifying a proper truncation time by checking the PCTL specification and its negation at the same time. Finally, we evaluate the proposed method on case studies.
Probing Channel Balances in the Lightning Network
Venue: arXiv cs.CR 2020
Year: 2020

As Lightning network payments are neither broadcasted nor publicly stored. Thus LN has been seen not only as scalability but also as privacy solution for Bitcoin. The protocol guarantees that only the latest channel state can be confirmed on channel closure. LN nodes gossip about channels available for routing and their total capacities. To issue a (multi-hop) payment, the sender creates a route based on its local knowledge of the graph. As local channel balances are not public, payments often fail due to insufficient balance at an intermediary hop. In that case, the payment is attempted along multiple routes until it succeeds. This constitutes a privacy-efficiency tradeoff: hidden balances improve privacy but hinder routing efficiency. In this work, we show that an attacker can easily discover channel balances using probing. This takes under a minute per channel and requires moderate capital commitment and no expenditures. We describe the algorithm and test our proof-of-concept implementation on Bitcoin's testnet. We argue that LN's balance between privacy and routing efficiency is suboptimal: channel balances are neither well protected nor utilized. We outline two ways for LN to evolve in respect to this issue. To emphasize privacy, we propose a modification of error handling that hides details of the erring channel from the sending node. This would break our probing technique but make routing failures more common, as the sender would not know which channel from the attempted route has failed. To improve efficiency, we propose a new API call that would let the sender query balances of channels that it is not a party of. We argue that combining these approaches can help LN take the best of both worlds: hide private data when feasible, and utilize public data for higher routing efficiency.
Multi-Class classification of vulnerabilities in Smart Contracts using AWD-LSTM, with pre-trained encoder inspired from natural language processing
Venue: arXiv cs.IR 2020
Year: 2020

Vulnerability detection and safety of smart contracts are of paramount importance because of their immutable nature. Symbolic tools like OYENTE and MAIAN are typically used for vulnerability prediction in smart contracts. As these tools are computationally expensive, they are typically used to detect vulnerabilities until some predefined invocation depth. These tools require more search time as the invocation depth increases. Since the number of smart contracts is increasing exponentially, it is difficult to analyze the contracts using these traditional tools. Recently a machine learning technique called Long Short Term Memory (LSTM) has been used for binary classification, i.e., to predict whether a smart contract is vulnerable or not. This technique requires nearly constant search time as the invocation depth increases. In the present article, we have shown a multi-class classification, where we classify a smart contract in Suicidal, Prodigal, Greedy, or Normal categories. We used Average Stochastic Gradient Descent Weight-Dropped LSTM (AWD-LSTM), which is a variant of LSTM, to perform classification. We reduced the class imbalance (a large number of normal contracts as compared to other categories) by considering only the distinct opcode combination for normal contracts. We have achieved a weighted average Fbeta score of 90.0%. Hence, such techniques can be used to analyze a large number of smart contracts and help to improve the security of these contracts.
Multi-Connectivity in 5G terrestrial-Satellite Networks: the 5G-ALLSTAR Solution
Venue: arXiv cs.NI 2020
Year: 2020

The 5G-ALLSTAR project is aimed at integrating Terrestrial and Satellite Networks for satisfying the highly challenging and demanding requirements of the 5G use cases. The integration of the two networks is a key feature to assure the service continuity in challenging communication situations (e.g., emergency cases, marine, railway, etc.) by avoiding service interruptions. The 5G-ALLSTAR project proposes to develop Multi-Connectivity (MC) solutions in order to guarantee network reliability and improve the throughput and latency for each connection between User Equipment (UE) and network. In the 5G-ALLSTAR vision, we divide the gNB in two entities: 1) gNB-CU (Centralized Unit) and 2) gNB-DU (Distributed Unit) The gNB-CU integrates an innovative Traffic Flow Control algorithm able to optimize the network resources by coordinating the controlled gNB-DUs resources, while implementing MC solutions. The MC permits to connect each UE with simultaneous multiple access points (even different radio access technologies). This solution leads to have independent gNB-DU/s that contain the RLC, MAC and PHY layers. The 5G-ALLSTAR MC algorithms offer advanced functionalities to RRC layer (in the gNB-CU) that is, in turn, able to set up the SDAP, the PDCP and the lower layers in gNB-DU. In this regard, the AI-based MC algorithms, implemented in gNB-CU, by considering the network performances in the UE surrounding environment as well as the UE QoS requirements, will dynamically select the most promising access points able to guarantee the fulfilment of the requirements also enabling the optimal traffic splitting to cope with the connection reliability. In this paper, we present also an innovative AI-based framework, included within the Traffic Flow Control, able to address the MC objectives, by implementing a Reinforcement Learning algorithm in charge of solving the network control problem.
Demonstrating Immersive Media Delivery on 5G Broadcast and Multicast Testing Networks
Venue: arXiv cs.NI 2020
Year: 2020

This work presents eight demonstrators and one showcase developed within the 5G-Xcast project. They experimentally demonstrate and validate key technical enablers for the future of media delivery, associated with multicast and broadcast communication capabilities in 5th Generation (5G). In 5G-Xcast, three existing testbeds: IRT in Munich (Germany), 5GIC in Surrey (UK), and TUAS in Turku (Finland), have been developed into 5G broadcast and multicast testing networks, which enables us to demonstrate our vision of a converged 5G infrastructure with fixed and mobile accesses and terrestrial broadcast, delivering immersive audio-visual media content. Built upon the improved testing networks, the demonstrators and showcase developed in 5G-Xcast show the impact of the technology developed in the project. Our demonstrations predominantly cover use cases belonging to two verticals: Media & Entertainment and Public Warning, which are future 5G scenarios relevant to multicast and broadcast delivery. In this paper, we present the development of these demonstrators, the showcase, and the testbeds. We also provide key findings from the experiments and demonstrations, which not only validate the technical solutions developed in the project, but also illustrate the potential technical impact of these solutions for broadcasters, content providers, operators, and other industries interested in the future immersive media delivery.
Deep Learning on Knowledge Graph for Recommender System: A Survey
Venue: arXiv cs.IR 2020
Year: 2020

Recent advances in research have demonstrated the effectiveness of knowledge graphs (KG) in providing valuable external knowledge to improve recommendation systems (RS). A knowledge graph is capable of encoding high-order relations that connect two objects with one or multiple related attributes. With the help of the emerging Graph Neural Networks (GNN), it is possible to extract both object characteristics and relations from KG, which is an essential factor for successful recommendations. In this paper, we provide a comprehensive survey of the GNN-based knowledge-aware deep recommender systems. Specifically, we discuss the state-of-the-art frameworks with a focus on their core component, i.e., the graph embedding module, and how they address practical recommendation issues such as scalability, cold-start and so on. We further summarize the commonly-used benchmark datasets, evaluation metrics as well as open-source codes. Finally, we conclude the survey and propose potential research directions in this rapidly growing field.
A hybrid optimization procedure for solving a tire curing scheduling problem
Venue: arXiv cs.AI 2020
Year: 2020

This paper addresses a lot-sizing and scheduling problem variant arising from the study of the curing process of a tire factory. The aim is to find the minimum makespan needed for producing enough tires to meet the demand requirements on time, considering the availability and compatibility of different resources involved. To solve this problem, we suggest a hybrid approach that consists in first applying a heuristic to obtain an estimated value of the makespan and then solving a mathematical model to determine the minimum value. We note that the size of the model (number of variables and constraints) depends significantly on the estimated makespan. Extensive numerical experiments over different instances based on real data are presented to evaluate the effectiveness of the hybrid procedure proposed. From the results obtained we can note that the hybrid approach is able to achieve the optimal makespan for many of the instances, even large ones, since the results provided by the heuristic allow to reduce significantly the size of the mathematical model.
Ensemble Forecasting of Monthly Electricity Demand using Pattern Similarity-based Methods
Venue: arXiv cs.LG 2020
Year: 2020

This work presents ensemble forecasting of monthly electricity demand using pattern similarity-based forecasting methods (PSFMs). PSFMs applied in this study include $k$-nearest neighbor model, fuzzy neighborhood model, kernel regression model, and general regression neural network. An integral part of PSFMs is a time series representation using patterns of time series sequences. Pattern representation ensures the input and output data unification through filtering a trend and equalizing variance. Two types of ensembles are created: heterogeneous and homogeneous. The former consists of different type base models, while the latter consists of a single-type base model. Five strategies are used for controlling a diversity of members in a homogeneous approach. The diversity is generated using different subsets of training data, different subsets of features, randomly disrupted input and output variables, and randomly disrupted model parameters. An empirical illustration applies the ensemble models as well as individual PSFMs for comparison to the monthly electricity demand forecasting for 35 European countries.
Handling Concept Drifts in Regression Problems -- the Error Intersection Approach
Venue: arXiv cs.LG 2020
Year: 2020

Machine learning models are omnipresent for predictions on big data. One challenge of deployed models is the change of the data over time, a phenomenon called concept drift. If not handled correctly, a concept drift can lead to significant mispredictions. We explore a novel approach for concept drift handling, which depicts a strategy to switch between the application of simple and complex machine learning models for regression tasks. We assume that the approach plays out the individual strengths of each model, switching to the simpler model if a drift occurs and switching back to the complex model for typical situations. We instantiate the approach on a real-world data set of taxi demand in New York City, which is prone to multiple drifts, e.g. the weather phenomena of blizzards, resulting in a sudden decrease of taxi demand. We are able to show that our suggested approach outperforms all regarded baselines significantly.
Spatio-temporal Tubelet Feature Aggregation and Object Linking in Videos
Venue: arXiv cs.CV 2020
Year: 2020

This paper addresses the problem of how to exploit spatio-temporal information available in videos to improve the object detection precision. We propose a two stage object detector called FANet based on short-term spatio-temporal feature aggregation to give a first detection set, and long-term object linking to refine these detections. Firstly, we generate a set of short tubelet proposals containing the object in $N$ consecutive frames. Then, we aggregate RoI pooled deep features through the tubelet using a temporal pooling operator that summarizes the information with a fixed size output independent of the number of input frames. On top of that, we define a double head implementation that we feed with spatio-temporal aggregated information for spatio-temporal object classification, and with spatial information extracted from the current frame for object localization and spatial classification. Furthermore, we also specialize each head branch architecture to better perform in each task taking into account the input data. Finally, a long-term linking method builds long tubes using the previously calculated short tubelets to overcome detection errors. We have evaluated our model in the widely used ImageNet VID dataset achieving a 80.9% mAP, which is the new state-of-the-art result for single models. Also, in the challenging small object detection dataset USC-GRAD-STDdb, our proposal outperforms the single frame baseline by 5.4% mAP.
Scheduling for Cellular Federated Edge Learning with Importance and Channel Awareness
Venue: arXiv cs.IT 2020
Year: 2020

In cellular federated edge learning (FEEL), multiple edge devices holding local data jointly train a neural network by communicating learning updates with an access point without exchanging their data samples. With very limited communication resources, it is beneficial to schedule the most informative local learning updates. In this paper, a novel scheduling policy is proposed to exploit both diversity in multiuser channels and diversity in the "importance" of the edge devices' learning updates. First, a new probabilistic scheduling framework is developed to yield unbiased update aggregation in FEEL. The importance of a local learning update is measured by its gradient divergence. If one edge device is scheduled in each communication round, the scheduling policy is derived in closed form to achieve the optimal trade-off between channel quality and update importance. The probabilistic scheduling framework is then extended to allow scheduling multiple edge devices in each communication round. Numerical results obtained using popular models and learning datasets demonstrate that the proposed scheduling policy can achieve faster model convergence and higher learning accuracy than conventional scheduling policies that only exploit a single type of diversity.
Beyond privacy regulations: an ethical approach to data usage in transportation
Venue: arXiv cs.CY 2020
Year: 2020

With the exponential advancement of business technology in recent years, data-driven decision making has become the core of most industries. With the rise of new privacy regulations such as the General Data Protection Regulation in the European Union and the California Consumer Privacy Act in the United States, companies dealing with personal data had to conform to these changes and adapt their processes accordingly. This obviously included the transportation industry with their use of location data. At the other side of the spectrum, users still expect a form of personalization, without having to compromise on their privacy. For this reason, companies across the industries started applying privacy-enhancing or preserving technologies at scale in their products as a competitive advantage. In this paper, we describe how Federated Machine Learning can be applied to the transportation sector. We present use-cases for which Federated Learning is beneficial in transportation and the new product lifecycle that is required for using such a technology. We see Federated Learning as a method that enables us to process privacy-sensitive data, while respecting customer's privacy and one that guides us beyond privacy-regulations and into the world of ethical data-usage.
Unique Chinese Linguistic Phenomena
Venue: arXiv cs.CL 2020
Year: 2020

Linguistics holds unique characteristics of generality, stability, and nationality, which will affect the formulation of extraction strategies and should be incorporated into the relation extraction. Chinese open relation extraction is not well-established, because of the complexity of Chinese linguistics makes it harder to operate, and the methods for English are not compatible with that for Chinese. The diversities between Chinese and English linguistics are mainly reflected in morphology and syntax.
Exploration in Action Space
Venue: arXiv cs.LG 2020
Year: 2020

Parameter space exploration methods with black-box optimization have recently been shown to outperform state-of-the-art approaches in continuous control reinforcement learning domains. In this paper, we examine reasons why these methods work better and the situations in which they are worse than traditional action space exploration methods. Through a simple theoretical analysis, we show that when the parametric complexity required to solve the reinforcement learning problem is greater than the product of action space dimensionality and horizon length, exploration in action space is preferred. This is also shown empirically by comparing simple exploration methods on several toy problems.
Deep Entity Matching with Pre-Trained Language Models
Venue: arXiv cs.DB 2020
Year: 2020

We present Ditto, a novel entity matching system based on pre-trained Transformer-based language models. We fine-tune and cast EM as a sequence-pair classification problem to leverage such models with a simple architecture. Our experiments show that a straightforward application of language models such as BERT, DistilBERT, or RoBERTa pre-trained on large text corpora already significantly improves the matching quality and outperforms previous state-of-the-art (SOTA), by up to 29% of F1 score on benchmark datasets. We also developed three optimization techniques to further improve Ditto's matching capability. Ditto allows domain knowledge to be injected by highlighting important pieces of input information that may be of interest when making matching decisions. Ditto also summarizes strings that are too long so that only the essential information is retained and used for EM. Finally, Ditto adapts a SOTA technique on data augmentation for text to EM to augment the training data with (difficult) examples. This way, Ditto is forced to learn "harder" to improve the model's matching capability. The optimizations we developed further boost the performance of Ditto by up to 9.8%. Perhaps more surprisingly, we establish that Ditto can achieve the previous SOTA results with at most half the number of labeled data. Finally, we demonstrate Ditto's effectiveness on a real-world large-scale EM task. On matching two company datasets consisting of 789K and 412K records, Ditto achieves a high F1 score of 96.5%.