OpenRaiser/Intern-Atlas huggingface dataset:
Rethinking High Performance Computing Platforms: Challenges, Opportunities and Recommendations
Venue: arXiv cs.DC 2017
Year: 2017

A new class of Second generation high-performance computing applications with heterogeneous, dynamic and data-intensive properties have an extended set of requirements, which cover application deployment, resource allocation, -control, and I/O scheduling. These requirements are not met by the current production HPC platform models and policies. This results in a loss of opportunity, productivity and innovation for new computational methods and tools. It also decreases effective system utilization for platform providers due to unsupervised workarounds and rogue resource management strategies implemented in application space. In this paper we critically discuss the dominant HPC platform model and describe the challenges it creates for second generation applications because of its asymmetric resource view, interfaces and software deployment policies. We present an extended, more symmetric and application-centric platform model that adds decentralized deployment, introspection, bidirectional control and information flow and more comprehensive resource scheduling. We describe cHPC: an early prototype of a non-disruptive implementation based on Linux Containers (LXC). It can operate alongside existing batch queuing systems and exposes a symmetric platform API without interfering with existing applications and usage modes. We see our approach as a viable, incremental next step in HPC platform evolution that benefits applications and platform providers alike. To demonstrate this further, we layout out a roadmap for future research and experimental evaluation.
"SHORT"er Reasoning About Larger Requirements Models
Venue: arXiv cs.SE 2017
Year: 2017

When Requirements Engineering(RE) models are unreasonably complex, they cannot support efficient decision making. SHORT is a tool to simplify that reasoning by exploiting the "key" decisions within RE models. These "keys" have the property that once values are assigned to them, it is very fast to reason over the remaining decisions. Using these "keys", reasoning about RE models can be greatly SHORTened by focusing stakeholder discussion on just these key decisions. This paper evaluates the SHORT tool on eight complex RE models. We find that the number of keys are typically only 12% of all decisions. Since they are so few in number, keys can be used to reason faster about models. For example, using keys, we can optimize over those models (to achieve the most goals at least cost) two to three orders of magnitude faster than standard methods. Better yet, finding those keys is not difficult: SHORT runs in low order polynomial time and terminates in a few minutes for the largest models.
An Online Secretary Framework for Fog Network Formation with Minimal Latency
Venue: arXiv cs.IT 2017
Year: 2017

Fog computing is seen as a promising approach to perform distributed, low-latency computation for supporting Internet of Things applications. However, due to the unpredictable arrival of available neighboring fog nodes, the dynamic formation of a fog network can be challenging. In essence, a given fog node must smartly select the set of neighboring fog nodes that can provide low-latency computations. In this paper, this problem of fog network formation and task distribution is studied considering a hybrid cloud-fog architecture. The goal of the proposed framework is to minimize the maximum computational latency by enabling a given fog node to form a suitable fog network, under uncertainty on the arrival process of neighboring fog nodes. To solve this problem, a novel approach based on the online secretary framework is proposed. To find the desired set of neighboring fog nodes, an online algorithm is developed to enable a task initiating fog node to decide on which other nodes can be used as part of its fog network, to offload computational tasks, without knowing any prior information on the future arrivals of those other nodes. Simulation results show that the proposed online algorithm can successfully select an optimal set of neighboring fog nodes while achieving a latency that is as small as the one resulting from an ideal, offline scheme that has complete knowledge of the system. The results also show how, using the proposed approach, the computational tasks can be properly distributed between the fog network and a remote cloud server.
Brain Inspired Cognitive Model with Attention for Self-Driving Cars
Venue: arXiv cs.CV 2017
Year: 2017

Perception-driven approach and end-to-end system are two major vision-based frameworks for self-driving cars. However, it is difficult to introduce attention and historical information of autonomous driving process, which are the essential factors for achieving human-like driving into these two methods. In this paper, we propose a novel model for self-driving cars named brain-inspired cognitive model with attention (CMA). This model consists of three parts: a convolutional neural network for simulating human visual cortex, a cognitive map built to describe relationships between objects in complex traffic scene and a recurrent neural network that combines with the real-time updated cognitive map to implement attention mechanism and long-short term memory. The benefit of our model is that can accurately solve three tasks simultaneously:1) detection of the free space and boundaries of the current and adjacent lanes. 2)estimation of obstacle distance and vehicle attitude, and 3) learning of driving behavior and decision making from human driver. More significantly, the proposed model could accept external navigating instructions during an end-to-end driving process. For evaluation, we build a large-scale road-vehicle dataset which contains more than forty thousand labeled road images captured by three cameras on our self-driving car. Moreover, human driving activities and vehicle states are recorded in the meanwhile.
One-Pass Error Bounded Trajectory Simplification
Venue: arXiv cs.DB 2017
Year: 2017

Nowadays, various sensors are collecting, storing and transmitting tremendous trajectory data, and it is known that raw trajectory data seriously wastes the storage, network band and computing resource. Line simplification (LS) algorithms are an effective approach to attacking this issue by compressing data points in a trajectory to a set of continuous line segments, and are commonly used in practice. However, existing LS algorithms are not sufficient for the needs of sensors in mobile devices. In this study, we first develop a one-pass error bounded trajectory simplification algorithm (OPERB), which scans each data point in a trajectory once and only once. We then propose an aggressive one-pass error bounded trajectory simplification algorithm (OPERB-A), which allows interpolating new data points into a trajectory under certain conditions. Finally, we experimentally verify that our approaches (OPERB and OPERB-A) are both efficient and effective, using four real-life trajectory datasets.
Scalable Spectrum Allocation for Large Networks Based on Sparse Optimization
Venue: arXiv cs.IT 2017
Year: 2017

Joint allocation of spectrum and user association is considered for a large cellular network. The objective is to optimize a network utility function such as average delay given traffic statistics collected over a slow timescale. A key challenge is scalability: given $n$ Access Points (APs), there are $O(2^n)$ ways in which the APs can share the spectrum. The number of variables is reduced from $O(2^n)$ to $O(nk)$, where $k$ is the number of users, by optimizing over local overlapping neighborhoods, defined by interference conditions, and by exploiting the existence of sparse solutions in which the spectrum is divided into $k+1$ segments. We reformulate the problem by optimizing the assignment of subsets of active APs to those segments. An $\ell_0$ constraint enforces a one-to-one mapping of subsets to spectrum, and an iterative (reweighted $\ell_1$) algorithm is used to find an approximate solution. Numerical results for a network with 100 APs serving several hundred users show the proposed method achieves a substantial increase in total throughput relative to benchmark schemes.
Computational Complexity of Atomic Chemical Reaction Networks
Venue: arXiv cs.CC 2017
Year: 2017

Informally, a chemical reaction network is "atomic" if each reaction may be interpreted as the rearrangement of indivisible units of matter. There are several reasonable definitions formalizing this idea. We investigate the computational complexity of deciding whether a given network is atomic according to each of these definitions. Our first definition, primitive atomic, which requires each reaction to preserve the total number of atoms, is to shown to be equivalent to mass conservation. Since it is known that it can be decided in polynomial time whether a given chemical reaction network is mass-conserving, the equivalence gives an efficient algorithm to decide primitive atomicity. Another definition, subset atomic, further requires that all atoms are species. We show that deciding whether a given network is subset atomic is in $\textsf{NP}$, and the problem "is a network subset atomic with respect to a given atom set" is strongly $\textsf{NP}$-$\textsf{Complete}$. A third definition, reachably atomic, studied by Adleman, Gopalkrishnan et al., further requires that each species has a sequence of reactions splitting it into its constituent atoms. We show that there is a $\textbf{polynomial-time algorithm}$ to decide whether a given network is reachably atomic, improving upon the result of Adleman et al. that the problem is $\textbf{decidable}$. We show that the reachability problem for reachably atomic networks is $\textsf{Pspace}$-$\textsf{Complete}$. Finally, we demonstrate equivalence relationships between our definitions and some special cases of another existing definition of atomicity due to Gnacadja.
Zoom Out-and-In Network with Recursive Training for Object Proposal
Venue: arXiv cs.CV 2017
Year: 2017

In this paper, we propose a zoom-out-and-in network for generating object proposals. We utilize different resolutions of feature maps in the network to detect object instances of various sizes. Specifically, we divide the anchor candidates into three clusters based on the scale size and place them on feature maps of distinct strides to detect small, medium and large objects, respectively. Deeper feature maps contain region-level semantics which can help shallow counterparts to identify small objects. Therefore we design a zoom-in sub-network to increase the resolution of high level features via a deconvolution operation. The high-level features with high resolution are then combined and merged with low-level features to detect objects. Furthermore, we devise a recursive training pipeline to consecutively regress region proposals at the training stage in order to match the iterative regression at the testing stage. We demonstrate the effectiveness of the proposed method on ILSVRC DET and MS COCO datasets, where our algorithm performs better than the state-of-the-arts in various evaluation metrics. It also increases average precision by around 2% in the detection system.
Lessons Learnt in Conducting Survey Research
Venue: arXiv cs.SE 2017
Year: 2017

Context: Surveys constitute an valuable tool to capture a large-scale snapshot of the state of the practice. Apparently trivial to adopt, surveys hide, however, several pitfalls that might hinder rendering the result valid and, thus, useful. Goal: We aim at providing an overview of main pitfalls in software engineering surveys and report on practical ways to deal with them. Method: We build on the experiences we collected in conducting many studies and distill the main lessons learnt. Results: The eight lessons learnt we report cover different aspects of the survey process ranging from the design of initial research objectives to the design of a questionnaire. Conclusions: Our hope is that by sharing our lessons learnt, combined with a disciplined application of the general survey theory, we contribute to improving the quality of the research results achievable by employing software engineering surveys.
Fast, Warped Graph Embedding: Unifying Framework and One-Click Algorithm
Venue: arXiv cs.SI 2017
Year: 2017

What is the best way to describe a user in a social network with just a few numbers? Mathematically, this is equivalent to assigning a vector representation to each node in a graph, a process called graph embedding. We propose a novel framework, GEM-D that unifies most of the past algorithms such as LapEigs, DeepWalk and node2vec. GEM-D achieves its goal by decomposing any graph embedding algorithm into three building blocks: node proximity function, warping function and loss function. Based on thorough analysis of GEM-D, we propose a novel algorithm, called UltimateWalk, which outperforms the most-recently proposed state-of-the-art DeepWalk and node2vec. The contributions of this work are: (1) The proposed framework, GEM-D unifies the past graph embedding algorithms and provides a general recipe of how to design a graph embedding; (2) the nonlinearlity in the warping function contributes significantly to the quality of embedding and the exponential function is empirically optimal; (3) the proposed algorithm, UltimateWalk is one-click (no user-defined parameters), scalable and has a closed-form solution.
Evaluation of Lane Departure Correction Systems Using a Stochastic Driver Model
Venue: arXiv cs.SY 2017
Year: 2017

Evaluating the effectiveness and benefits of driver assistance systems is crucial for improving the system performance. In this paper, we propose a novel framework for testing and evaluating lane departure correction systems at a low cost by using lane departure events reproduced from naturalistic driving data. First, 529,096 lane departure events were extracted from the Safety Pilot Model Deployment (SPMD) database collected by the University of Michigan Transportation Research Institute. Second, a stochastic lane departure model consisting of eight random key variables was developed to reduce the dimension of the data description and improve the computational efficiency. As such, we used a bounded Gaussian mixture model (BGM) model to describe drivers' stochastic lane departure behaviors. Then, a lane departure correction system with an aim point controller was designed, and a batch of lane departure events were reproduced from the learned stochastic driver model. Finally, we assessed the developed evaluation approach by comparing lateral departure areas of vehicles between with and without correction controller. The simulation results show that the proposed method can effectively evaluate lane departure correction systems.
Network-based Anomaly Detection for Insider Trading
Venue: arXiv cs.SI 2017
Year: 2017

Insider trading is one of the numerous white collar crimes that can contribute to the instability of the economy. Traditionally, the detection of illegal insider trades has been a human-driven process. In this paper, we collect the insider tradings made available by the US Securities and Exchange Commissions (SEC) through the EDGAR system, with the aim of initiating an automated large-scale and data-driven approach to the problem of identifying illegal insider tradings. The goal of the study is the identification of interesting patterns, which can be indicators of potential anomalies. We use the collected data to construct networks that capture the relationship between trading behaviors of insiders. We explore different ways of building networks from insider trading data, and argue for a need of a structure that is capable of capturing higher order relationships among traders. Our results suggest the discovery of interesting patterns.
Sprites and State Channels: Payment Networks that Go Faster than Lightning
Venue: arXiv cs.CR 2017
Year: 2017

Bitcoin, Ethereum and other blockchain-based cryptocurrencies, as deployed today, cannot scale for wide-spread use. A leading approach for cryptocurrency scaling is a smart contract mechanism called a payment channel which enables two mutually distrustful parties to transact efficiently (and only requires a single transaction in the blockchain to set-up). Payment channels can be linked together to form a payment network, such that payments between any two parties can (usually) be routed through the network along a path that connects them. Crucially, both parties can transact without trusting hops along the route. In this paper, we propose a novel variant of payment channels, called Sprites, that reduces the worst-case "collateral cost" that each hop along the route may incur. The benefits of Sprites are two-fold. 1) In Lightning Network, a payment across a path of $\ell$ channels requires locking up collateral for $Θ(\ellΔ)$ time, where $Δ$ is the time to commit an on-chain transaction. Sprites reduces this cost to $O(\ell + Δ)$. 2) Unlike prior work, Sprites supports partial withdrawals and deposits, during which the channel can continue to operate without interruption. In evaluating Sprites we make several additional contributions. First, our simulation-based security model is the first formalism to model timing guarantees in payment channels. Our construction is also modular, making use of a generic abstraction from folklore, called the "state channel," which we are the first to formalize. We also provide a simulation framework for payment network protocols, which we use to confirm that the Sprites construction mitigates against throughput-reducing attacks.
Compressive Embedding and Visualization using Graphs
Venue: arXiv cs.LG 2017
Year: 2017

Visualizing high-dimensional data has been a focus in data analysis communities for decades, which has led to the design of many algorithms, some of which are now considered references (such as t-SNE for example). In our era of overwhelming data volumes, the scalability of such methods have become more and more important. In this work, we present a method which allows to apply any visualization or embedding algorithm on very large datasets by considering only a fraction of the data as input and then extending the information to all data points using a graph encoding its global similarity. We show that in most cases, using only $\mathcal{O}(\log(N))$ samples is sufficient to diffuse the information to all $N$ data points. In addition, we propose quantitative methods to measure the quality of embeddings and demonstrate the validity of our technique on both synthetic and real-world datasets.
A Platform for Automating Chaos Experiments
Venue: arXiv cs.SE 2017
Year: 2017

The Netflix video streaming system is composed of many interacting services. In such a large system, failures in individual services are not uncommon. This paper describes the Chaos Automation Platform, a system for running failure injection experiments on the production system to verify that failures in non-critical services do not result in system outages.
Memory Efficient Max Flow for Multi-label Submodular MRFs
Venue: arXiv cs.DS 2017
Year: 2017

Multi-label submodular Markov Random Fields (MRFs) have been shown to be solvable using max-flow based on an encoding of the labels proposed by Ishikawa, in which each variable $X_i$ is represented by $\ell$ nodes (where $\ell$ is the number of labels) arranged in a column. However, this method in general requires $2\,\ell^2$ edges for each pair of neighbouring variables. This makes it inapplicable to realistic problems with many variables and labels, due to excessive memory requirement. In this paper, we introduce a variant of the max-flow algorithm that requires much less storage. Consequently, our algorithm makes it possible to optimally solve multi-label submodular problems involving large numbers of variables and labels on a standard computer.
Why Do Cascade Sizes Follow a Power-Law?
Venue: arXiv cs.SI 2017
Year: 2017

We introduce random directed acyclic graph and use it to model the information diffusion network. Subsequently, we analyze the cascade generation model (CGM) introduced by Leskovec et al. [19]. Until now only empirical studies of this model were done. In this paper, we present the first theoretical proof that the sizes of cascades generated by the CGM follow the power-law distribution, which is consistent with multiple empirical analysis of the large social networks. We compared the assumptions of our model with the Twitter social network and tested the goodness of approximation.
SurvivalNet: Predicting patient survival from diffusion weighted magnetic resonance images using cascaded fully convolutional and 3D convolutional neural networks
Venue: arXiv cs.CV 2017
Year: 2017

Automatic non-invasive assessment of hepatocellular carcinoma (HCC) malignancy has the potential to substantially enhance tumor treatment strategies for HCC patients. In this work we present a novel framework to automatically characterize the malignancy of HCC lesions from DWI images. We predict HCC malignancy in two steps: As a first step we automatically segment HCC tumor lesions using cascaded fully convolutional neural networks (CFCN). A 3D neural network (SurvivalNet) then predicts the HCC lesions' malignancy from the HCC tumor segmentation. We formulate this task as a classification problem with classes being "low risk" and "high risk" represented by longer or shorter survival times than the median survival. We evaluated our method on DWI of 31 HCC patients. Our proposed framework achieves an end-to-end accuracy of 65% with a Dice score for the automatic lesion segmentation of 69% and an accuracy of 68% for tumor malignancy classification based on expert annotations. We compared the SurvivalNet to classical handcrafted features such as Histogram and Haralick and show experimentally that SurvivalNet outperforms the handcrafted features in HCC malignancy classification. End-to-end assessment of tumor malignancy based on our proposed fully automatic framework corresponds to assessment based on expert annotations with high significance (p>0.95).
On the Spectral Efficiency and Fairness in Full-Duplex Cellular Networks
Venue: arXiv cs.IT 2017
Year: 2017

To increase the spectral efficiency of wireless networks without requiring full-duplex capability of user devices, a potential solution is the recently proposed three-node full-duplex mode. To realize this potential, networks employing three-node full-duplex transmissions must deal with self-interference and user-to-user interference, which can be managed by frequency channel and power allocation techniques. Whereas previous works investigated either spectral efficient or fair mechanisms, a scheme that balances these two metrics among users is investigated in this paper. This balancing scheme is based on a new solution method of the multi-objective optimization problem to maximize the weighted sum of the per-user spectral efficiency and the minimum spectral efficiency among users. The mixed integer non-linear nature of this problem is dealt by Lagrangian duality. Based on the proposed solution approach, a low-complexity centralized algorithm is developed, which relies on large scale fading measurements that can be advantageously implemented at the base station. Numerical results indicate that the proposed algorithm increases the spectral efficiency and fairness among users without the need of weighting the spectral efficiency. An important conclusion is that managing user-to-user interference by resource assignment and power control is crucial for ensuring spectral efficient and fair operation of full-duplex networks.