OpenRaiser/Intern-Atlas huggingface dataset:
Emerging Relation Network and Task Embedding for Multi-Task Regression Problems
Venue: arXiv cs.LG 2020
Year: 2020

Multi-task learning (mtl) provides state-of-the-art results in many applications of computer vision and natural language processing. In contrast to single-task learning (stl), mtl allows for leveraging knowledge between related tasks improving prediction results on the main task (in contrast to an auxiliary task) or all tasks. However, there is a limited number of comparative studies on applying mtl architectures for regression and time series problems taking recent advances of mtl into account. An interesting, non-linear problem is the forecast of the expected power generation for renewable power plants. Therefore, this article provides a comparative study of the following recent and important mtl architectures: Hard parameter sharing, cross-stitch network, sluice network (sn). They are compared to a multi-layer perceptron model of similar size in an stl setting. Additionally, we provide a simple, yet effective approach to model task specific information through an embedding layer in an multi-layer perceptron, referred to as task embedding. Further, we introduce a new mtl architecture named emerging relation network (ern), which can be considered as an extension of the sluice network. For a solar power dataset, the task embedding achieves the best mean improvement with 14.9%. The mean improvement of the ern and the sn on the solar dataset is of similar magnitude with 14.7% and 14.8%. On a wind power dataset, only the ern achieves a significant improvement of up to 7.7%. Results suggest that the ern is beneficial when tasks are only loosely related and the prediction problem is more non-linear. Contrary, the proposed task embedding is advantageous when tasks are strongly correlated. Further, the task embedding provides an effective approach with reduced computational effort compared to other mtl architectures.
Human-Like Summaries from Heterogeneous and Time-Windowed Software Development Artefacts
Venue: arXiv cs.SE 2020
Year: 2020

Automatic text summarisation has drawn considerable interest in the area of software engineering. It is challenging to summarise the activities related to a software project, (1) because of the volume and heterogeneity of involved software artefacts, and (2) because it is unclear what information a developer seeks in such a multi-document summary. We present the first framework for summarising multi-document software artefacts containing heterogeneous data within a given time frame. To produce human-like summaries, we employ a range of iterative heuristics to minimise the cosine-similarity between texts and high-dimensional feature vectors. A first study shows that users find the automatically generated summaries the most useful when they are generated using word similarity and based on the eight most relevant software artefacts.
Models of Homotopy Type Theory with an Interval Type
Venue: arXiv cs.LO 2020
Year: 2020

In this short note, we construct a class of models of an extension of homotopy type theory, which we call homotopy type theory with an interval type.
Specifying a Cryptographical Protocol in Lustre and SCADE
Venue: arXiv cs.CR 2020
Year: 2020

We present SCADE and Lustre models of the Message Authenticator Algorithm (MAA), which is one of the first cryptographic functions for computing a message authentication code. The MAA was adopted between 1987 and 2001, in international standards (ISO 8730 and ISO 8731-2), to ensure the authenticity and integrity of banking transactions. This paper discusses the choices and the challenges of our MAA implementations. Our SCADE and Lustre models validate 201 official test vectors for the MAA.
Batch Normalization in Quantized Networks
Venue: arXiv cs.LG 2020
Year: 2020

Implementation of quantized neural networks on computing hardware leads to considerable speed up and memory saving. However, quantized deep networks are difficult to train and batch~normalization (BatchNorm) layer plays an important role in training full-precision and quantized networks. Most studies on BatchNorm are focused on full-precision networks, and there is little research in understanding BatchNorm affect in quantized training which we address here. We show BatchNorm avoids gradient explosion which is counter-intuitive and recently observed in numerical experiments by other researchers.
The computational framework for continuum-kinematics-inspired peridynamics
Venue: arXiv cs.CE 2020
Year: 2020

Peridynamics (PD) is a non-local continuum formulation. The original version of PD was restricted to bond-based interactions. Bond-based PD is geometrically exact and its kinematics are similar to classical continuum mechanics (CCM). However, it cannot capture the Poisson effect correctly. This shortcoming was addressed via state-based PD, but the kinematics are not accurately preserved. Continuum-kinematics-inspired peridynamics (CPD) provides a geometrically exact framework whose underlying kinematics coincide with that of CCM and captures the Poisson effect correctly. In CPD, one distinguishes between one-, two- and three-neighbour interactions. One-neighbour interactions are equivalent to the bond-based interactions of the original PD formalism. However, two- and three-neighbour interactions are fundamentally different from state-based interactions as the basic elements of continuum kinematics are preserved precisely. The objective of this contribution is to elaborate on computational aspects of CPD and present detailed derivations that are essential for its implementation. Key features of the resulting computational CPD are elucidated via a series of numerical examples. These include three-dimensional problems at large deformations. The proposed strategy is robust and the quadratic rate of convergence associated with the Newton--Raphson scheme is observed.
Lifted codes and the multilevel construction for constant dimension codes
Venue: arXiv cs.IT 2020
Year: 2020

Constant dimension codes are e.g. used for error correction and detection in random linear network coding, so that constructions for these codes have achieved wide attention. Here, we improve over 150 lower bounds by describing better constructions for subspace distance 4.
Exploring the Suitability of Semantic Spaces as Word Association Models for the Extraction of Semantic Relationships
Venue: arXiv cs.CL 2020
Year: 2020

Given the recent advances and progress in Natural Language Processing (NLP), extraction of semantic relationships has been at the top of the research agenda in the last few years. This work has been mainly motivated by the fact that building knowledge graphs (KG) and bases (KB), as a key ingredient of intelligent applications, is a never-ending challenge, since new knowledge needs to be harvested while old knowledge needs to be revised. Currently, approaches towards relation extraction from text are dominated by neural models practicing some sort of distant (weak) supervision in machine learning from large corpora, with or without consulting external knowledge sources. In this paper, we empirically study and explore the potential of a novel idea of using classical semantic spaces and models, e.g., Word Embedding, generated for extracting word association, in conjunction with relation extraction approaches. The goal is to use these word association models to reinforce current relation extraction approaches. We believe that this is a first attempt of this kind and the results of the study should shed some light on the extent to which these word association models can be used as well as the most promising types of relationships to be considered for extraction.
Actor-Critic Reinforcement Learning for Control with Stability Guarantee
Venue: arXiv cs.RO 2020
Year: 2020

Reinforcement Learning (RL) and its integration with deep learning have achieved impressive performance in various robotic control tasks, ranging from motion planning and navigation to end-to-end visual manipulation. However, stability is not guaranteed in model-free RL by solely using data. From a control-theoretic perspective, stability is the most important property for any control system, since it is closely related to safety, robustness, and reliability of robotic systems. In this paper, we propose an actor-critic RL framework for control which can guarantee closed-loop stability by employing the classic Lyapunov's method in control theory. First of all, a data-based stability theorem is proposed for stochastic nonlinear systems modeled by Markov decision process. Then we show that the stability condition could be exploited as the critic in the actor-critic RL to learn a controller/policy. At last, the effectiveness of our approach is evaluated on several well-known 3-dimensional robot control tasks and a synthetic biology gene network tracking task in three different popular physics simulation platforms. As an empirical evaluation on the advantage of stability, we show that the learned policies can enable the systems to recover to the equilibrium or way-points when interfered by uncertainties such as system parametric variations and external disturbances to a certain extent.
TUNIZI: a Tunisian Arabizi sentiment analysis Dataset
Venue: arXiv cs.CL 2020
Year: 2020

On social media, Arabic people tend to express themselves in their own local dialects. More particularly, Tunisians use the informal way called "Tunisian Arabizi". Analytical studies seek to explore and recognize online opinions aiming to exploit them for planning and prediction purposes such as measuring the customer satisfaction and establishing sales and marketing strategies. However, analytical studies based on Deep Learning are data hungry. On the other hand, African languages and dialects are considered low resource languages. For instance, to the best of our knowledge, no annotated Tunisian Arabizi dataset exists. In this paper, we introduce TUNIZI a sentiment analysis Tunisian Arabizi Dataset, collected from social networks, preprocessed for analytical studies and annotated manually by Tunisian native speakers.
Real-Time Resource Allocation for Wireless Powered Multiuser Mobile Edge Computing With Energy and Task Causality
Venue: arXiv cs.IT 2020
Year: 2020

This paper considers a wireless powered multiuser mobile edge computing (MEC) system, in which a multi-antenna hybrid access point (AP) wirelessly charges multiple users, and each user relies on the harvested energy to execute computation tasks. We jointly optimize the energy beamforming and remote task execution at the AP, as well as the local computing and task offloading, aiming to minimize the total system energy consumption over a finite time horizon, subject to causality constraints for both energy harvesting and task arrival at the users. In particular, we consider a practical scenario with casual task state information (TSI) and channel state information (CSI), i.e., only the current and previous TSI and CSI are available, but the future TSI and CSI can only be predicted subject to certain errors. To solve this real-time resource allocation problem, we propose an offline-optimization inspired online design approach. First, we consider the offline optimization case by assuming that the TSI and CSI are perfectly known a-priori. In this case, the energy minimization problem corresponds to a convex problem, for which the semi-closed-form optimal solution is obtained via the Lagrange duality method. Next, inspired by the optimal offline solution, we propose a sliding-window based online resource allocation design in practical cases by integrating with the sequential optimization. Finally, numerical results show that the proposed joint wireless powered MEC designs significantly improve the system's energy efficiency, as compared with the benchmark schemes that consider a sliding window of size one or without such joint optimization.
Learning to Learn to Disambiguate: Meta-Learning for Few-Shot Word Sense Disambiguation
Venue: arXiv cs.CL 2020
Year: 2020

The success of deep learning methods hinges on the availability of large training datasets annotated for the task of interest. In contrast to human intelligence, these methods lack versatility and struggle to learn and adapt quickly to new tasks, where labeled data is scarce. Meta-learning aims to solve this problem by training a model on a large number of few-shot tasks, with an objective to learn new tasks quickly from a small number of examples. In this paper, we propose a meta-learning framework for few-shot word sense disambiguation (WSD), where the goal is to learn to disambiguate unseen words from only a few labeled instances. Meta-learning approaches have so far been typically tested in an $N$-way, $K$-shot classification setting where each task has $N$ classes with $K$ examples per class. Owing to its nature, WSD deviates from this controlled setup and requires the models to handle a large number of highly unbalanced classes. We extend several popular meta-learning approaches to this scenario, and analyze their strengths and weaknesses in this new challenging setting.
Transfer Learning for Thermal Comfort Prediction in Multiple Cities
Venue: arXiv cs.LG 2020
Year: 2020

HVAC (Heating, Ventilation and Air Conditioning) system is an important part of a building, which constitutes up to 40% of building energy usage. The main purpose of HVAC, maintaining appropriate thermal comfort, is crucial for the best utilisation of energy usage. Besides, thermal comfort is also crucial for well-being, health, and work productivity. Recently, data-driven thermal comfort models have got better performance than traditional knowledge-based methods (e.g. Predicted Mean Vote Model). An accurate thermal comfort model requires a large amount of self-reported thermal comfort data from indoor occupants which undoubtedly remains a challenge for researchers. In this research, we aim to tackle this data-shortage problem and boost the performance of thermal comfort prediction. We utilise sensor data from multiple cities in the same climate zone to learn thermal comfort patterns. We present a transfer learning based multilayer perceptron model from the same climate zone (TL-MLP-C*) for accurate thermal comfort prediction. Extensive experimental results on ASHRAE RP-884, the Scales Project and Medium US Office datasets show that the performance of the proposed TL-MLP-C* exceeds the state-of-the-art methods in accuracy, precision and F1-score.
To Reduce Gross NPA and Classify Defaulters Using Shannon Entropy
Venue: arXiv cs.LG 2020
Year: 2020

Non Performing Asset(NPA) has been in a serious attention by banks over the past few years. NPA cause a huge loss to the banks hence it becomes an extremely critical step in deciding which loans have the capabilities to become an NPA and thereby deciding which loans to grant and which ones to reject. In this paper which focuses on the exact crux of the matter we have proposed an algorithm which is designed to handle the financial data very meticulously to predict with a very high accuracy whether a particular loan would be classified as a NPA in future or not. Instead of the conventional less accurate classifiers used to decide which loans can turn to be NPA we build our own classifier model using Entropy as the base. We have created an entropy based classifier using Shannon Entropy. The classifier model categorizes our data points in two categories accepted or rejected. We make use of local entropy and global entropy to help us determine the output. The entropy classifier model is then compared with existing classifiers used to predict NPAs thereby giving us an idea about the performance.
Mainlining Databases: Supporting Fast Transactional Workloads on Universal Columnar Data File Formats
Venue: arXiv cs.DB 2020
Year: 2020

The proliferation of modern data processing tools has given rise to open-source columnar data formats. The advantage of these formats is that they help organizations avoid repeatedly converting data to a new format for each application. These formats, however, are read-only, and organizations must use a heavy-weight transformation process to load data from on-line transactional processing (OLTP) systems. We aim to reduce or even eliminate this process by developing a storage architecture for in-memory database management systems (DBMSs) that is aware of the eventual usage of its data and emits columnar storage blocks in a universal open-source format. We introduce relaxations to common analytical data formats to efficiently update records and rely on a lightweight transformation process to convert blocks to a read-optimized layout when they are cold. We also describe how to access data from third-party analytical tools with minimal serialization overhead. To evaluate our work, we implemented our storage engine based on the Apache Arrow format and integrated it into the DB-X DBMS. Our experiments show that our approach achieves comparable performance with dedicated OLTP DBMSs while enabling orders-of-magnitude faster data exports to external data science and machine learning tools than existing methods.
Arc Routing with Time-Dependent Travel Times and Paths
Venue: arXiv cs.DS 2020
Year: 2020

Vehicle routing algorithms usually reformulate the road network into a complete graph in which each arc represents the shortest path between two locations. Studies on time-dependent routing followed this model and therefore defined the speed functions on the complete graph. We argue that this model is often inadequate, in particular for arc routing problems involving services on edges of a road network. To fill this gap, we formally define the time-dependent capacitated arc routing problem (TDCARP), with travel and service speed functions given directly at the network level. Under these assumptions, the quickest path between locations can change over time, leading to a complex problem that challenges the capabilities of current solution methods. We introduce effective algorithms for preprocessing quickest paths in a closed form, efficient data structures for travel time queries during routing optimization, as well as heuristic and exact solution approaches for the TDCARP. Our heuristic uses the hybrid genetic search principle with tailored solution-decoding algorithms and lower bounds for filtering moves. Our branch-and-price algorithm exploits dedicated pricing routines, heuristic dominance rules and completion bounds to find optimal solutions for problem counting up to 75 services. Based on these algorithms, we measure the benefits of time-dependent routing optimization for different levels of travel-speed data accuracy.
KGClean: An Embedding Powered Knowledge Graph Cleaning Framework
Venue: arXiv cs.DB 2020
Year: 2020

The quality assurance of the knowledge graph is a prerequisite for various knowledge-driven applications. We propose KGClean, a novel cleaning framework powered by knowledge graph embedding, to detect and repair the heterogeneous dirty data. In contrast to previous approaches that either focus on filling missing data or clean errors violated limited rules, KGClean enables (i) cleaning both missing data and other erroneous values, and (ii) mining potential rules automatically, which expands the coverage of error detecting. KGClean first learns data representations by TransGAT, an effective knowledge graph embedding model, which gathers the neighborhood information of each data and incorporates the interactions among data for casting data to continuous vector spaces with rich semantics. KGClean integrates an active learning-based classification model, which identifies errors with a small seed of labels. KGClean utilizes an efficient PRO-repair strategy to repair errors using a novel concept of propagation power. Extensive experiments on four typical knowledge graphs demonstrate the effectiveness of KGClean in practice.
A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT
Venue: arXiv cs.CL 2020
Year: 2020

We present a novel supervised word alignment method based on cross-language span prediction. We first formalize a word alignment problem as a collection of independent predictions from a token in the source sentence to a span in the target sentence. As this is equivalent to a SQuAD v2.0 style question answering task, we then solve this problem by using multilingual BERT, which is fine-tuned on a manually created gold word alignment data. We greatly improved the word alignment accuracy by adding the context of the token to the question. In the experiments using five word alignment datasets among Chinese, Japanese, German, Romanian, French, and English, we show that the proposed method significantly outperformed previous supervised and unsupervised word alignment methods without using any bitexts for pretraining. For example, we achieved an F1 score of 86.7 for the Chinese-English data, which is 13.3 points higher than the previous state-of-the-art supervised methods.
An Empirical Study of Pre-trained Transformers for Arabic Information Extraction
Venue: arXiv cs.CL 2020
Year: 2020

Multilingual pre-trained Transformers, such as mBERT (Devlin et al., 2019) and XLM-RoBERTa (Conneau et al., 2020a), have been shown to enable the effective cross-lingual zero-shot transfer. However, their performance on Arabic information extraction (IE) tasks is not very well studied. In this paper, we pre-train a customized bilingual BERT, dubbed GigaBERT, that is designed specifically for Arabic NLP and English-to-Arabic zero-shot transfer learning. We study GigaBERT's effectiveness on zero-short transfer across four IE tasks: named entity recognition, part-of-speech tagging, argument role labeling, and relation extraction. Our best model significantly outperforms mBERT, XLM-RoBERTa, and AraBERT (Antoun et al., 2020) in both the supervised and zero-shot transfer settings. We have made our pre-trained models publicly available at https://github.com/lanwuwei/GigaBERT.
Secrecy Analysis of Ambient Backscatter NOMA Systems under I/Q Imbalance
Venue: arXiv cs.IT 2020
Year: 2020

We investigate the reliability and security of the ambient backscatter (AmBC) non-orthogonal multiple access (NOMA) systems, where the source aims to communication with two NOMA users in the presence of an eavesdropper. We consider a more practical case that nodes and backscatter device (BD) suffer from in-phase and quadrature-phase imbalance (IQI). More specifically, exact analytical expressions for the outage probability (OP) and the intercept probability (IP) are derived in closedform. Moreover, the asymptotic behaviors and corresponding diversity orders for the OP are discussed. Numerical results show that: 1) Although IQI reduces the reliability, it can enhance the security. 2) Compared with the traditional orthogonal multiple access (OMA) system, the AmBC-NOMA system can obtain better reliability when the signal-to-noise (SNR) ratio is low; 3) There are error floors for the OP because of the reflection coefficient \b{eta} .
Capsule-Transformer for Neural Machine Translation
Venue: arXiv cs.CL 2020
Year: 2020

Transformer hugely benefits from its key design of the multi-head self-attention network (SAN), which extracts information from various perspectives through transforming the given input into different subspaces. However, its simple linear transformation aggregation strategy may still potentially fail to fully capture deeper contextualized information. In this paper, we thus propose the capsule-Transformer, which extends the linear transformation into a more general capsule routing algorithm by taking SAN as a special case of capsule network. So that the resulted capsule-Transformer is capable of obtaining a better attention distribution representation of the input sequence via information aggregation among different heads and words. Specifically, we see groups of attention weights in SAN as low layer capsules. By applying the iterative capsule routing algorithm they can be further aggregated into high layer capsules which contain deeper contextualized information. Experimental results on the widely-used machine translation datasets show our proposed capsule-Transformer outperforms strong Transformer baseline significantly.