OpenRaiser/Intern-Atlas huggingface dataset:
LSH Ensemble: Internet-Scale Domain Search
Venue: arXiv cs.DB 2016
Year: 2016

We study the problem of domain search where a domain is a set of distinct values from an unspecified universe. We use Jaccard set containment, defined as $|Q \cap X|/|Q|$, as the relevance measure of a domain $X$ to a query domain $Q$. Our choice of Jaccard set containment over Jaccard similarity makes our work particularly suitable for searching Open Data and data on the web, as Jaccard similarity is known to have poor performance over sets with large differences in their domain sizes. We demonstrate that the domains found in several real-life Open Data and web data repositories show a power-law distribution over their domain sizes. We present a new index structure, Locality Sensitive Hashing (LSH) Ensemble, that solves the domain search problem using set containment at Internet scale. Our index structure and search algorithm cope with the data volume and skew by means of data sketches (MinHash) and domain partitioning. Our index structure does not assume a prescribed set of values. We construct a cost model that describes the accuracy of LSH Ensemble with any given partitioning. This allows us to formulate the partitioning for LSH Ensemble as an optimization problem. We prove that there exists an optimal partitioning for any distribution. Furthermore, for datasets following a power-law distribution, as observed in Open Data and Web data corpora, we show that the optimal partitioning can be approximated using equi-depth, making it efficient to use in practice. We evaluate our algorithm using real data (Canadian Open Data and WDC Web Tables) containing up over 262 M domains. The experiments demonstrate that our index consistently outperforms other leading alternatives in accuracy and performance. The improvements are most dramatic for data with large skew in the domain sizes. Even at 262 M domains, our index sustains query performance with under 3 seconds response time.
Push- and Pull-based Epidemic Spreading in Networks: Thresholds and Deeper Insights
Venue: arXiv cs.CR 2016
Year: 2016

Understanding the dynamics of computer virus (malware, worm) in cyberspace is an important problem that has attracted a fair amount of attention. Early investigations for this purpose adapted biological epidemic models, and thus inherited the so-called homogeneity assumption that each node is equally connected to others. Later studies relaxed this often-unrealistic homogeneity assumption, but still focused on certain power-law networks. Recently, researchers investigated epidemic models in {\em arbitrary} networks (i.e., no restrictions on network topology). However, all these models only capture {\em push-based} infection, namely that an infectious node always actively attempts to infect its neighboring nodes. Very recently, the concept of {\em pull-based} infection was introduced but was not treated rigorously. Along this line of research, the present paper investigates push- and pull-based epidemic spreading dynamics in arbitrary networks, using a Non-linear Dynamical Systems approach. The paper advances the state of the art as follows: (1) It presents a more general and powerful sufficient condition (also known as epidemic threshold in the literature) under which the spreading will become stable. (2) It gives both upper and lower bounds on the global mean infection rate, regardless of the stability of the spreading. (3) It offers insights into, among other things, the estimation of the global mean infection rate through localized monitoring of a small {\em constant} number of nodes, {\em without} knowing the values of the parameters.
A General Framework for Estimating Graphlet Statistics via Random Walk
Venue: arXiv cs.SI 2016
Year: 2016

Graphlets are induced subgraph patterns and have been frequently applied to characterize the local topology structures of graphs across various domains, e.g., online social networks (OSNs) and biological networks. Discovering and computing graphlet statistics are highly challenging. First, the massive size of real-world graphs makes the exact computation of graphlets extremely expensive. Secondly, the graph topology may not be readily available so one has to resort to web crawling using the available application programming interfaces (APIs). In this work, we propose a general and novel framework to estimate graphlet statistics of "any size". Our framework is based on collecting samples through consecutive steps of random walks. We derive an analytical bound on the sample size (via the Chernoff-Hoeffding technique) to guarantee the convergence of our unbiased estimator. To further improve the accuracy, we introduce two novel optimization techniques to reduce the lower bound on the sample size. Experimental evaluations demonstrate that our methods outperform the state-of-the-art method up to an order of magnitude both in terms of accuracy and time cost.
Going Out of Business: Auction House Behavior in the Massively Multi-Player Online Game
Venue: arXiv cs.CY 2016
Year: 2016

The in-game economies of massively multi-player online games (MMOGs) are complex systems that have to be carefully designed and managed. This paper presents the results of an analysis of auction house data from the MMOG Glitch, across a 14 month time period, the entire lifetime of the game. The data comprise almost 3 million data points, over 20,000 unique players and more than 650 products. Furthermore, an interactive visualization, based on Sankey flow diagrams, is presented which shows the proportion of the different clusters across each time bin, as well as the flow of players between clusters. The diagram allows evaluation of migration of players between clusters as a function of time, as well as churn analysis. The presented work provides a template analysis and visualization model for progression-based or temporal-based analysis of player behavior broadly applicable to games.
Multi-Object Tracking with Multiple Birth, Death, and Spawn Scenarios Using A Randomized Hypothesis Generation Technique (R-FISST)
Venue: arXiv cs.SY 2016
Year: 2016

In multi-object tracking one may encounter situations were at any time step the number of possible hypotheses is too large to generate exhaustively. These situations generally occur when there are multiple ambiguous measurement returns that can be associated to many objects. This paper contains a newly developed approach that keeps the aforementioned situations computationally tractable. Utilizing a hypothesis level derivation of the Finite Set Statistics (FISST) Bayesian recursions for multi-object tracking we are able to propose a randomized method called randomized FISST (R-FISST). Like our previous methods, this approach utilizes Markov Chain Monte Carlo (MCMC) methods to sample highly probable hypotheses, however, the newly developed (R-FISST) can account for hypotheses containing multiple births and death within the MCMC sampling. This alleviates the burden of having to exhaustively enumerate all birth and death hypotheses and makes the method more equipped to handle spawn scenarios. We test our method on Space Situational Awareness (SSA) scenarios with spawn events.
Online Budgeted Allocation with General Budgets
Venue: arXiv cs.DS 2016
Year: 2016

We study the online budgeted allocation (also called ADWORDS) problem, where a set of impressions arriving online are allocated to a set of budget-constrained advertisers to maximize revenue. Motivated by connections to Internet advertising, several variants of this problem have been studied since the seminal work of Mehta, Saberi, Vazirani, and Vazirani (FOCS 2005). However, this entire body of work focuses on a single budget for every advertising campaign, whereas in order to fully represent the actual agenda of an advertiser, an advertising budget should be expressible over multiple tiers of user-attribute granularity. A simple example is an advertising campaign that is constrained by an overall budget but is also accompanied by a set of sub-budgets for each target demographic. In such a contract scheme, an advertiser can specify their true user-targeting goals, allowing the publisher to fulfill them through relevant allocations. In this paper, we give a complete characterization of the ADWORDS problem for general advertising budgets. In the most general setting, we show that, unlike in the single-budget ADWORDS problem, obtaining a constant competitive ratio is impossible and give asymptotically tight upper and lower bounds. However for our main result, we observe that in many real-world scenarios (as in the above example), multi-tier budgets have a laminar structure, since most relevant consumer or product classifications are hierarchical. For laminar budgets, we obtain a competitive ratio of e/(e-1) in the small bids case, which matches the best known ADWORDS result for single budgets. Our algorithm has a primal-dual structure and generalizes the primal-dual analysis for single- budget ADWORDS first given by Buchbinder, Jain, and Naor (ESA 2007).
Quadratic Projection Based Feature Extraction with Its Application to Biometric Recognition
Venue: arXiv cs.CV 2016
Year: 2016

This paper presents a novel quadratic projection based feature extraction framework, where a set of quadratic matrices is learned to distinguish each class from all other classes. We formulate quadratic matrix learning (QML) as a standard semidefinite programming (SDP) problem. However, the con- ventional interior-point SDP solvers do not scale well to the problem of QML for high-dimensional data. To solve the scalability of QML, we develop an efficient algorithm, termed DualQML, based on the Lagrange duality theory, to extract nonlinear features. To evaluate the feasibility and effectiveness of the proposed framework, we conduct extensive experiments on biometric recognition. Experimental results on three representative biometric recogni- tion tasks, including face, palmprint, and ear recognition, demonstrate the superiority of the DualQML-based feature extraction algorithm compared to the current state-of-the-art algorithms
Training-Free Synthesized Face Sketch Recognition Using Image Quality Assessment Metrics
Venue: arXiv cs.CV 2016
Year: 2016

Face sketch synthesis has wide applications ranging from digital entertainments to law enforcements. Objective image quality assessment scores and face recognition accuracy are two mainly used tools to evaluate the synthesis performance. In this paper, we proposed a synthesized face sketch recognition framework based on full-reference image quality assessment metrics. Synthesized sketches generated from four state-of-the-art methods are utilized to test the performance of the proposed recognition framework. For the image quality assessment metrics, we employed the classical structured similarity index metric and other three prevalent metrics: visual information fidelity, feature similarity index metric and gradient magnitude similarity deviation. Extensive experiments compared with baseline methods illustrate the effectiveness of the proposed synthesized face sketch recognition framework. Data and implementation code in this paper are available online at www.ihitworld.com/WNN/IQA_Sketch.zip.
An end-to-end convolutional selective autoencoder approach to Soybean Cyst Nematode eggs detection
Venue: arXiv cs.CV 2016
Year: 2016

This paper proposes a novel selective autoencoder approach within the framework of deep convolutional networks. The crux of the idea is to train a deep convolutional autoencoder to suppress undesired parts of an image frame while allowing the desired parts resulting in efficient object detection. The efficacy of the framework is demonstrated on a critical plant science problem. In the United States, approximately $1 billion is lost per annum due to a nematode infection on soybean plants. Currently, plant-pathologists rely on labor-intensive and time-consuming identification of Soybean Cyst Nematode (SCN) eggs in soil samples via manual microscopy. The proposed framework attempts to significantly expedite the process by using a series of manually labeled microscopic images for training followed by automated high-throughput egg detection. The problem is particularly difficult due to the presence of a large population of non-egg particles (disturbances) in the image frames that are very similar to SCN eggs in shape, pose and illumination. Therefore, the selective autoencoder is trained to learn unique features related to the invariant shapes and sizes of the SCN eggs without handcrafting. After that, a composite non-maximum suppression and differencing is applied at the post-processing stage.
Helenos: A Realistic Benchmark for Distributed Transactional Memory
Venue: arXiv cs.DC 2016
Year: 2016

Transactional Memory (TM) is an approach to concurrency control that aims to make writing parallel programs both effective and simple. The approach is started in non-distributed multiprocessor systems, but is gaining popularity in distributed systems to synchronize tasks at large scales. Efficiency and scalability are often the key issues in TM research, so performance benchmarks are an important part of it. However, while standard TM benchmarks like the STAMP suite and STMBench7 are available and widely accepted, they do not translate well into distributed systems. Hence, the set of benchmarks usable with distributed TM systems is very limited, and must be padded with microbenchmarks, whose simplicity and artificial nature often makes them uninformative or misleading. Therefore, this paper introduces Helenos, a realistic, complex, and comprehensive distributed TM benchmark based on the problem of the Facebook inbox, an application of the Cassandra distributed store.
An Incremental Learner for Language-Based Anomaly Detection in XML
Venue: arXiv cs.CR 2016
Year: 2016

The Extensible Markup Language (XML) is a complex language, and consequently, XML-based protocols are susceptible to entire classes of implicit and explicit security problems. Message formats in XML-based protocols are usually specified in XML Schema, and as a first-line defense, schema validation should reject malformed input. However, extension points in most protocol specifications break validation. Extension points are wildcards and considered best practice for loose composition, but they also enable an attacker to add unchecked content in a document, e.g., for a signature wrapping attack. This paper introduces datatyped XML visibly pushdown automata (dXVPAs) as language representation for mixed-content XML and presents an incremental learner that infers a dXVPA from example documents. The learner generalizes XML types and datatypes in terms of automaton states and transitions, and an inferred dXVPA converges to a good-enough approximation of the true language. The automaton is free from extension points and capable of stream validation, e.g., as an anomaly detector for XML-based protocols. For dealing with adversarial training data, two scenarios of poisoning are considered: a poisoning attack is either uncovered at a later time or remains hidden. Unlearning can therefore remove an identified poisoning attack from a dXVPA, and sanitization trims low-frequent states and transitions to get rid of hidden attacks. All algorithms have been evaluated in four scenarios, including a web service implemented in Apache Axis2 and Apache Rampart, where attacks have been simulated. In all scenarios, the learned automaton had zero false positives and outperformed traditional schema validation.
OptEx: A Deadline-Aware Cost Optimization Model for Spark
Venue: arXiv cs.DC 2016
Year: 2016

We present OptEx, a closed-form model of job execution on Apache Spark, a popular parallel processing engine. To the best of our knowledge, OptEx is the first work that analytically models job completion time on Spark. The model can be used to estimate the completion time of a given Spark job on a cloud, with respect to the size of the input dataset, the number of iterations, the number of nodes comprising the underlying cluster. Experimental results demonstrate that OptEx yields a mean relative error of 6% in estimating the job completion time. Furthermore, the model can be applied for estimating the cost optimal cluster composition for running a given Spark job on a cloud under a completion deadline specified in the SLO (i.e., Service Level Objective). We show experimentally that OptEx is able to correctly estimate the cost optimal cluster composition for running a given Spark job under an SLO deadline with an accuracy of 98%.
A Scheduler for Unsolicited Grant Service (UGS) in IEEE 802.16e Mobile WiMAX Networks
Venue: arXiv cs.NI 2016
Year: 2016

Most of the IEEE 802.16e Mobile WiMAX scheduling proposals for real-time traffic using Unsolicited Grant Service (UGS) focus on the throughput and the guaranteed latency. The delay variation or delay jitter and the effect of burst overhead have not yet been investigated. This paper introduces a new technique called Swapping Min-Max (SWIM) for UGS scheduling that not only meets the delay constraint with optimal throughput, but also minimizes the delay jitter and burst overhead.
A generalized flow for multi-class and binary classification tasks: An Azure ML approach
Venue: arXiv cs.CV 2016
Year: 2016

The constant growth in the present day real-world databases pose computational challenges for a single computer. Cloud-based platforms, on the other hand, are capable of handling large volumes of information manipulation tasks, thereby necessitating their use for large real-world data set computations. This work focuses on creating a novel Generalized Flow within the cloud-based computing platform: Microsoft Azure Machine Learning Studio (MAMLS) that accepts multi-class and binary classification data sets alike and processes them to maximize the overall classification accuracy. First, each data set is split into training and testing data sets, respectively. Then, linear and nonlinear classification model parameters are estimated using the training data set. Data dimensionality reduction is then performed to maximize classification accuracy. For multi-class data sets, data centric information is used to further improve overall classification accuracy by reducing the multi-class classification to a series of hierarchical binary classification tasks. Finally, the performance of optimized classification model thus achieved is evaluated and scored on the testing data set. The classification characteristics of the proposed flow are comparatively evaluated on 3 public data sets and a local data set with respect to existing state-of-the-art methods. On the 3 public data sets, the proposed flow achieves 78-97.5% classification accuracy. Also, the local data set, created using the information regarding presence of Diabetic Retinopathy lesions in fundus images, results in 85.3-95.7% average classification accuracy, which is higher than the existing methods. Thus, the proposed generalized flow can be useful for a wide range of application-oriented "big data sets".
Mining DNA Sequences Based on Spatially Coded Technique Using Spatial Light Modulator
Venue: arXiv cs.IT 2016
Year: 2016

In this paper, we present an optical computing method for string data alignment applicable to genome information analysis. By applying moire technique to spatial encoding patterns of deoxyribonucleic acid (DNA) sequences, association information of the genome and the expressed phenotypes could more effectively be extracted. Such moire fringes reveal occurrence of matching, deletion and insertion between DNA sequences providing useful visualized information for prediction of gene function and classification of species. Furthermore, by applying a cylindrical lens, a new technique is proposed to map two-dimensional (2D) association information to a one-dimensional (1D) column of pixels, where each pixel in the column is representative of superposition of all bright and dark pixels in the corresponding row. By such a time-consuming preprocessing, local similarities between two intended patterns can readily be found by just using a 1D array of photodetectors and postprocessing could be performed on specified parts in the initial 2D pattern. We also evaluate our proposed circular encoding adapted for poor data alignment condition. Our simulation results together with experimental implementation verify the effectiveness of our dynamic proposed methods which significantly improve system parameters such as processing gain and signal to noise ratio (SNR).
Nonrigid Optical Flow Ground Truth for Real-World Scenes with Time-Varying Shading Effects
Venue: arXiv cs.CV 2016
Year: 2016

In this paper we present a dense ground truth dataset of nonrigidly deforming real-world scenes. Our dataset contains both long and short video sequences, and enables the quantitatively evaluation for RGB based tracking and registration methods. To construct ground truth for the RGB sequences, we simultaneously capture Near-Infrared (NIR) image sequences where dense markers - visible only in NIR - represent ground truth positions. This allows for comparison with automatically tracked RGB positions and the formation of error metrics. Most previous datasets containing nonrigidly deforming sequences are based on synthetic data. Our capture protocol enables us to acquire real-world deforming objects with realistic photometric effects - such as blur and illumination change - as well as occlusion and complex deformations. A public evaluation website is constructed to allow for ranking of RGB image based optical flow and other dense tracking algorithms, with various statistical measures. Furthermore, we present an RGB-NIR multispectral optical flow model allowing for energy optimization by adoptively combining featured information from both the RGB and the complementary NIR channels. In our experiments we evaluate eight existing RGB based optical flow methods on our new dataset. We also evaluate our hybrid optical flow algorithm by comparing to two existing multispectral approaches, as well as varying our input channels across RGB, NIR and RGB-NIR.
Binary search trees and rectangulations
Venue: arXiv cs.DS 2016
Year: 2016

We revisit the classical problem of searching in a binary search tree (BST) using rotations, and present novel connections of this problem to a number of geometric and combinatorial structures. In particular, we show that the execution trace of a BST that serves a sequence of queries is in close correspondence with the flip-sequence between two rectangulations. (Rectangulations are well-studied combinatorial objects also known as mosaic floorplans.) We also reinterpret Small Manhattan Network, a problem with known connections to the BST problem, in terms of flips in rectangulations. We apply further transformations to the obtained geometric model, to arrive at a particularly simple view of the BST problem that resembles sequences of edge-relaxations in a shortest path algorithm. Our connections yield new results and observations for all structures concerned. In this draft we present some preliminary findings. BSTs with rotations are among the most fundamental and most thoroughly studied objects in computer science, nonetheless they pose long-standing open questions, such as the dynamic optimality conjecture of Sleator and Tarjan (STOC 1983). Our hope is that the correspondences presented in this paper provide a new perspective on this old problem and bring new tools to the study of dynamic optimality.