OpenRaiser/Intern-Atlas huggingface dataset:
Normalization of Input-output Shared Embeddings in Text Generation Models
Venue: arXiv cs.CL 2020
Year: 2020

Neural Network based models have been state-of-the-art models for various Natural Language Processing tasks, however, the input and output dimension problem in the networks has still not been fully resolved, especially in text generation tasks (e.g. Machine Translation, Text Summarization), in which input and output both have huge sizes of vocabularies. Therefore, input-output embedding weight sharing has been introduced and adopted widely, which remains to be improved. Based on linear algebra and statistical theories, this paper locates the shortcoming of existed input-output embedding weight sharing method, then raises methods for improving input-output weight shared embedding, among which methods of normalization of embedding weight matrices show best performance. These methods are nearly computational cost-free, can get combined with other embedding techniques, and show good effectiveness when applied on state-of-the-art Neural Network models. For Transformer-big models, the normalization techniques can get at best 0.6 BLEU improvement compared to the original version of model on WMT'16 En-De dataset, and similar BLEU improvements on IWSLT 14' datasets. For DynamicConv models, 0.5 BLEU improvement can be attained on WMT'16 En-De dataset, and 0.41 BLEU improvement on IWSLT 14' De-En translation task is achieved.
Graph Generators: State of the Art and Open Challenges
Venue: arXiv cs.DB 2020
Year: 2020

The abundance of interconnected data has fueled the design and implementation of graph generators reproducing real-world linking properties, or gauging the effectiveness of graph algorithms, techniques and applications manipulating these data. We consider graph generation across multiple subfields, such as Semantic Web, graph databases, social networks, and community detection, along with general graphs. Despite the disparate requirements of modern graph generators throughout these communities, we analyze them under a common umbrella, reaching out the functionalities, the practical usage, and their supported operations. We argue that this classification is serving the need of providing scientists, researchers and practitioners with the right data generator at hand for their work. This survey provides a comprehensive overview of the state-of-the-art graph generators by focusing on those that are pertinent and suitable for several data-intensive tasks. Finally, we discuss open challenges and missing requirements of current graph generators along with their future extensions to new emerging fields.
Benchmarking Symbolic Execution Using Constraint Problems -- Initial Results
Venue: arXiv cs.SE 2020
Year: 2020

Symbolic execution is a powerful technique for bug finding and program testing. It is successful in finding bugs in real-world code. The core reasoning techniques use constraint solving, path exploration, and search, which are also the same techniques used in solving combinatorial problems, e.g., finite-domain constraint satisfaction problems (CSPs). We propose CSP instances as more challenging benchmarks to evaluate the effectiveness of the core techniques in symbolic execution. We transform CSP benchmarks into C programs suitable for testing the reasoning capabilities of symbolic execution tools. From a single CSP P, we transform P depending on transformation choice into different C programs. Preliminary testing with the KLEE, Tracer-X, and LLBMC tools show substantial runtime differences from transformation and solver choice. Our C benchmarks are effective in showing the limitations of existing symbolic execution tools. The motivation for this work is we believe that benchmarks of this form can spur the development and engineering of improved core reasoning in symbolic execution engines.
Preventive and Reactive Cyber Defense Dynamics with Ergodic Time-dependent Parameters Is Globally Attractive
Venue: arXiv cs.CR 2020
Year: 2020

Cybersecurity dynamics is a mathematical approach to modeling and analyzing cyber attack-defense interactions in networks. In this paper, we advance the state-of-the-art in characterizing one kind of cybersecurity dynamics, known as preventive and reactive cyber defense dynamics, which is a family of highly nonlinear system models. We prove that this dynamics in its general form with time-dependent parameters is globally attractive when the time-dependent parameters are ergodic, and is (almost) periodic when the time-dependent parameters have the stronger properties of being (almost) periodic. Our results supersede the state-of-the-art ones, including that the same type of dynamics but with time-independent parameters is globally convergent.
A Neural Architecture for Person Ontology population
Venue: arXiv cs.AI 2020
Year: 2020

A person ontology comprising concepts, attributes and relationships of people has a number of applications in data protection, didentification, population of knowledge graphs for business intelligence and fraud prevention. While artificial neural networks have led to improvements in Entity Recognition, Entity Classification, and Relation Extraction, creating an ontology largely remains a manual process, because it requires a fixed set of semantic relations between concepts. In this work, we present a system for automatically populating a person ontology graph from unstructured data using neural models for Entity Classification and Relation Extraction. We introduce a new dataset for these tasks and discuss our results.
Subjective Knowledge and Reasoning about Agents in Multi-Agent Systems
Venue: arXiv cs.MA 2020
Year: 2020

Though a lot of work in multi-agent systems is focused on reasoning about knowledge and beliefs of artificial agents, an explicit representation and reasoning about the presence/absence of agents, especially in the scenarios where agents may be unaware of other agents joining in or going offline in a multi-agent system, leading to partial knowledge/asymmetric knowledge of the agents is mostly overlooked by the MAS community. Such scenarios lay the foundations of cases where an agent can influence other agents' mental states by (mis)informing them about the presence/absence of collaborators or adversaries. In this paper, we investigate how Kripke structure-based epistemic models can be extended to express the above notion based on an agent's subjective knowledge and we discuss the challenges that come along.
Depthwise Non-local Module for Fast Salient Object Detection Using a Single Thread
Venue: arXiv cs.CV 2020
Year: 2020

Recently deep convolutional neural networks have achieved significant success in salient object detection. However, existing state-of-the-art methods require high-end GPUs to achieve real-time performance, which makes them hard to adapt to low-cost or portable devices. Although generic network architectures have been proposed to speed up inference on mobile devices, they are tailored to the task of image classification or semantic segmentation, and struggle to capture intra-channel and inter-channel correlations that are essential for contrast modeling in salient object detection. Motivated by the above observations, we design a new deep learning algorithm for fast salient object detection. The proposed algorithm for the first time achieves competitive accuracy and high inference efficiency simultaneously with a single CPU thread. Specifically, we propose a novel depthwise non-local moudule (DNL), which implicitly models contrast via harvesting intra-channel and inter-channel correlations in a self-attention manner. In addition, we introduce a depthwise non-local network architecture that incorporates both depthwise non-local modules and inverted residual blocks. Experimental results show that our proposed network attains very competitive accuracy on a wide range of salient object detection datasets while achieving state-of-the-art efficiency among all existing deep learning based algorithms.
Experiments on Manual Thesaurus based Query Expansion for Ad-hoc Monolingual Gujarati Information Retrieval Tasks
Venue: arXiv cs.IR 2020
Year: 2020

In this paper, we present the experimental work done on Query Expansion (QE) for retrieval tasks of Gujarati text documents. In information retrieval, it is very difficult to estimate the exact user need, query expansion adds terms to the original query, which provides more information about the user need. There are various approaches to query expansion. In our work, manual thesaurus based query expansion was performed to evaluate the performance of widely used information retrieval models for Gujarati text documents. Results show that query expansion improves the recall of text documents.
Learning to Correct 3D Reconstructions from Multiple Views
Venue: arXiv cs.CV 2020
Year: 2020

This paper is about reducing the cost of building good large-scale 3D reconstructions post-hoc. We render 2D views of an existing reconstruction and train a convolutional neural network (CNN) that refines inverse-depth to match a higher-quality reconstruction. Since the views that we correct are rendered from the same reconstruction, they share the same geometry, so overlapping views complement each other. We take advantage of that in two ways. Firstly, we impose a loss during training which guides predictions on neighbouring views to have the same geometry and has been shown to improve performance. Secondly, in contrast to previous work, which corrects each view independently, we also make predictions on sets of neighbouring views jointly. This is achieved by warping feature maps between views and thus bypassing memory-intensive 3D computation. We make the observation that features in the feature maps are viewpoint-dependent, and propose a method for transforming features with dynamic filters generated by a multi-layer perceptron from the relative poses between views. In our experiments we show that this last step is necessary for successfully fusing feature maps between views.
Accelerating supply chains with Ant Colony Optimization across range of hardware solutions
Venue: arXiv cs.NE 2020
Year: 2020

Ant Colony algorithm has been applied to various optimization problems, however most of the previous work on scaling and parallelism focuses on Travelling Salesman Problems (TSPs). Although, useful for benchmarks and new idea comparison, the algorithmic dynamics does not always transfer to complex real-life problems, where additional meta-data is required during solution construction. This paper looks at real-life outbound supply chain problem using Ant Colony Optimization (ACO) and its scaling dynamics with two parallel ACO architectures - Independent Ant Colonies (IAC) and Parallel Ants (PA). Results showed that PA was able to reach a higher solution quality in fewer iterations as the number of parallel instances increased. Furthermore, speed performance was measured across three different hardware solutions - 16 core CPU, 68 core Xeon Phi and up to 4 Geforce GPUs. State of the art, ACO vectorization techniques such as SS-Roulette were implemented using C++ and CUDA. Although excellent for TSP, it was concluded that for the given supply chain problem GPUs are not suitable due to meta-data access footprint required. Furthermore, compared to their sequential counterpart, vectorized CPU AVX2 implementation achieved 25.4x speedup on CPU while Xeon Phi with its AVX512 instruction set reached 148x on PA with Vectorized (PAwV). PAwV is therefore able to scale at least up to 1024 parallel instances on the supply chain network problem solved.
Iterated Type Partitions
Venue: arXiv cs.CC 2020
Year: 2020

This paper deals with the complexity of some natural graph problems when parametrized by {measures that are restrictions of} clique-width, such as modular-width and neighborhood diversity. The main contribution of this paper is to introduce a novel parameter, called iterated type partition, that can be computed in polynomial time and nicely places between modular-width and neighborhood diversity. We prove that the Equitable Coloring problem is W[1]-hard when parametrized by the iterated type partition. This result extends to modular-width, answering an open question about the possibility to have FPT algorithms for Equitable Coloring when parametrized by modular-width. Moreover, we show that the Equitable Coloring problem is instead FTP when parameterized by neighborhood diversity. Furthermore, we present simple and fast FPT algorithms parameterized by iterated type partition that provide optimal solutions for several graph problems; in particular, this paper presents algorithms for the Dominating Set, the Vertex Coloring and the Vertex Cover problems. While the above problems are already known to be FPT with respect to modular-width, the novel algorithms are both simpler and more efficient: For the Dominating set and Vertex Cover problems, our algorithms output an optimal set in time $O(2^t+poly(n))$, while for the Vertex Coloring problem, our algorithm outputs an optimal set in time $O(t^{2.5t+o(t)}\log n+poly(n))$, where $n$ and $t$ are the size and the iterated type partition of the input graph, respectively.
On the Capacity of Waveform Channels Under Square-Law Detection of Time-Limited Signals
Venue: arXiv cs.IT 2020
Year: 2020

Capacity bounds for waveform channels under square-law detection of time-limited complex-valued signals are derived. The upper bound is the capacity of the channel under (complex-valued) coherent detection. The lower bound is one bit less, per dimension, than the upper bound.
Hybrid Coded Replication in LoRa Networks
Venue: arXiv cs.NI 2020
Year: 2020

Low Power Wide Area Networks (LPWAN) are wireless connectivity solutions for Internet-of-Things (IoT) applications, including industrial automation. Among the several LPWAN technologies, LoRaWAN has been extensively addressed by the research community and the industry. However, the reliability and scalability of LoRaWAN are still uncertain. One of the techniques to increase the reliability of LoRaWAN is message replication, which exploits time diversity. This paper proposes a novel hybrid coded message replication scheme that interleaves simple repetition and a recently proposed coded replication method. We analyze the optimization of the proposed scheme under minimum reliability requirements and show that it enhances the network performance without requiring additional transmit power compared to the competing replication techniques.
Causality based Feature Fusion for Brain Neuro-Developmental Analysis
Venue: arXiv cs.CV 2020
Year: 2020

Human brain development is a complex and dynamic process that is affected by several factors such as genetics, sex hormones, and environmental changes. A number of recent studies on brain development have examined functional connectivity (FC) defined by the temporal correlation between time series of different brain regions. We propose to add the directional flow of information during brain maturation. To do so, we extract effective connectivity (EC) through Granger causality (GC) for two different groups of subjects, i.e., children and young adults. The motivation is that the inclusion of causal interaction may further discriminate brain connections between two age groups and help to discover new connections between brain regions. The contributions of this study are threefold. First, there has been a lack of attention to EC-based feature extraction in the context of brain development. To this end, we propose a new kernel-based GC (KGC) method to learn nonlinearity of complex brain network, where a reduced Sine hyperbolic polynomial (RSP) neural network was used as our proposed learner. Second, we used causality values as the weight for the directional connectivity between brain regions. Our findings indicated that the strength of connections was significantly higher in young adults relative to children. In addition, our new EC-based feature outperformed FC-based analysis from Philadelphia neurocohort (PNC) study with better discrimination of the different age groups. Moreover, the fusion of these two sets of features (FC + EC) improved brain age prediction accuracy by more than 4%, indicating that they should be used together for brain development studies.
Automatic phantom test pattern classification through transfer learning with deep neural networks
Venue: arXiv cs.CV 2020
Year: 2020

Imaging phantoms are test patterns used to measure image quality in computer tomography (CT) systems. A new phantom platform (Mercury Phantom, Gammex) provides test patterns for estimating the task transfer function (TTF) or noise power spectrum (NPF) and simulates different patient sizes. Determining which image slices are suitable for analysis currently requires manual annotation of these patterns by an expert, as subtle defects may make an image unsuitable for measurement. We propose a method of automatically classifying these test patterns in a series of phantom images using deep learning techniques. By adapting a convolutional neural network based on the VGG19 architecture with weights trained on ImageNet, we use transfer learning to produce a classifier for this domain. The classifier is trained and evaluated with over 3,500 phantom images acquired at a university medical center. Input channels for color images are successfully adapted to convey contextual information for phantom images. A series of ablation studies are employed to verify design aspects of the classifier and evaluate its performance under varying training conditions. Our solution makes extensive use of image augmentation to produce a classifier that accurately classifies typical phantom images with 98% accuracy, while maintaining as much as 86% accuracy when the phantom is improperly imaged.
Talek: Private Group Messaging with Hidden Access Patterns
Venue: arXiv cs.CR 2020
Year: 2020

Talek is a private group messaging system that sends messages through potentially untrustworthy servers, while hiding both data content and the communication patterns among its users. Talek explores a new point in the design space of private messaging; it guarantees access sequence indistinguishability, which is among the strongest guarantees in the space, while assuming an anytrust threat model, which is only slightly weaker than the strongest threat model currently found in related work. Our results suggest that this is a pragmatic point in the design space, since it supports strong privacy and good performance: we demonstrate a 3-server Talek cluster that achieves throughput of 9,433 messages/second for 32,000 active users with 1.7-second end-to-end latency. To achieve its security goals without coordination between clients, Talek relies on information-theoretic private information retrieval. To achieve good performance and minimize server-side storage, Talek introduces new techniques and optimizations that may be of independent interest, e.g., a novel use of blocked cuckoo hashing and support for private notifications. The latter provide a private, efficient mechanism for users to learn, without polling, which logs have new messages.
To schedule or not to schedule: when no-scheduling can beat the best-known flow scheduling algorithm in datacenter networks
Venue: arXiv cs.NI 2020
Year: 2020

Conventional wisdom for minimizing the average flow completion time (AFCT) in the datacenter network (DCN), where flow sizes are highly variable, would suggest scheduling every individual flow. However, we show that considering scheduling delay (including scheduler's computational and communication delays), serving most of the flows without any scheduling and only in first-come-first-served (FCFS) manner significantly improves their performance even when it is compared to the shortest remaining processing time (SRPT)-known as optimum algorithm when scheduling delay is zero. To do so, we only require to have two coarse classes of flows categorized based on flows' sizes (1st-class including flows smaller than a threshold, H, and 2nd-class including others) and serve 1st-class flows always before serving 2nd-class ones. To show that, we take SRPT scheduling algorithm accompanied by the global knowledge of flows, formulate impact of scheduling delay on its performance, and prove that for any flow size distribution and network load (<1), there is always a threshold, H, which guarantees 1st-class flows achieve lower AFCT under FCFS compared to SRPT. Our numerically calculated results and extensive flow-level simulations show that on average, more than 90% of flows could be in 1st-class and consequently do not require any scheduling.
A Probabilistic Framework for Imitating Human Race Driver Behavior
Venue: arXiv cs.RO 2020
Year: 2020

Understanding and modeling human driver behavior is crucial for advanced vehicle development. However, unique driving styles, inconsistent behavior, and complex decision processes render it a challenging task, and existing approaches often lack variability or robustness. To approach this problem, we propose Probabilistic Modeling of Driver behavior (ProMoD), a modular framework which splits the task of driver behavior modeling into multiple modules. A global target trajectory distribution is learned with Probabilistic Movement Primitives, clothoids are utilized for local path generation, and the corresponding choice of actions is performed by a neural network. Experiments in a simulated car racing setting show considerable advantages in imitation accuracy and robustness compared to other imitation learning algorithms. The modular architecture of the proposed framework facilitates straightforward extensibility in driving line adaptation and sequencing of multiple movement primitives for future research.
Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case
Venue: arXiv cs.LG 2020
Year: 2020

In this paper, we present a new approach to time series forecasting. Time series data are prevalent in many scientific and engineering disciplines. Time series forecasting is a crucial task in modeling time series data, and is an important area of machine learning. In this work we developed a novel method that employs Transformer-based machine learning models to forecast time series data. This approach works by leveraging self-attention mechanisms to learn complex patterns and dynamics from time series data. Moreover, it is a generic framework and can be applied to univariate and multivariate time series data, as well as time series embeddings. Using influenza-like illness (ILI) forecasting as a case study, we show that the forecasting results produced by our approach are favorably comparable to the state-of-the-art.
BLK-REW: A Unified Block-based DNN Pruning Framework using Reweighted Regularization Method
Venue: arXiv cs.LG 2020
Year: 2020

Accelerating DNN execution on various resource-limited computing platforms has been a long-standing problem. Prior works utilize l1-based group lasso or dynamic regularization such as ADMM to perform structured pruning on DNN models to leverage the parallel computing architectures. However, both of the pruning dimensions and pruning methods lack universality, which leads to degraded performance and limited applicability. To solve the problem, we propose a new block-based pruning framework that comprises a general and flexible structured pruning dimension as well as a powerful and efficient reweighted regularization method. Our framework is universal, which can be applied to both CNNs and RNNs, implying complete support for the two major kinds of computation-intensive layers (i.e., CONV and FC layers). To complete all aspects of the pruning-for-acceleration task, we also integrate compiler-based code optimization into our framework that can perform DNN inference in a real-time manner. To the best of our knowledge, it is the first time that the weight pruning framework achieves universal coverage for both CNNs and RNNs with real-time mobile acceleration and no accuracy compromise.