pinecone/core-2020-05-10-deduplication huggingface dataset:
Competitive On-line Scheduling of Continuous-Media Streams

AbstractMultimedia applications require a guaranteed level of service for accessing continuous-media data. To obtain such guarantees, the database server where the data are residing must employ an admission control scheme to limit the number of clients that can be served concurrently. We investigate the problem of on-line admission control, where the decision of whether to accept or reject a request must be made without any knowledge about future requests. Employing competitive analysis techniques, we address the problem in its most general form with the following key contributions: (1) We prove a tight upper bound on the competitive ratio of the conventional Work-Conserving (WC) policy, showing that it is within a factor 1+Δ1−ρ of the optimal clairvoyant strategy, where Δ is the ratio of the maximum to minimum request length (i.e., time duration), and ρ is the maximum fraction of the server's bandwidth that a request can demand; (2) we prove a lower bound of Ω(logΔ1−ρ) on the competitive ratio of any deterministic or randomized admission control scheme, demonstrating an exponential gap between greedy and optimal on-line solutions; (3) we propose simple deterministic schemes based on the idea of bandwidth prepartitioning that guarantee competitive ratios within a small constant factor of logΔ (i.e., they are provably near-optimal) if ρ<1/⌈logΔ⌉; and (4) we introduce a novel admission control policy that partitions the server bandwidth based on the expected popularities of different request lengths and experimentally demonstrate its benefits compared to WC
Formal Software Specification with Refinements and Modules of Typed Graph Transformation Systems

AbstractGraph transformation systems are a formal specification technique for software systems that support the rule based specification of the dynamic behaviour of a system.Their main advantages are the intuitive visual representation of states and state transformations as graphs on the one hand, and the fully formal semantics on the other hand, that allow precise statements about the specification and tool support. In this paper we introduce refinements and modules for typed graph transformation systems to support the software specification development in both dimensions: modules for the horizontal structuring of a specification, i.e., its composition from feasible parts, and refinements for the development over time
Minimality of an Automaton Cascade Decomposition for Learning System Environments

AbstractWe already know how to decompose any finite automaton with a strongly connected state diagram into a strongly connected version of what we call a synchronizable cascade decomposition. This is a two component cascade decomposition whose first component has a synchronizer and whose second component is a permutation automaton. Here, we give a simpler procedure for constructing such a decomposition and show that the constructed decomposition is a homomorphic image of a subautomaton of any other synchronizable cascade decomposition. The constructed decomposition is then of minimal size, and all minimal synchronizable cascade decompositions are isomorphic, including all decompositions constructed by the old procedure. This means that their first components are isomorphic, but their second components need not be. In analyzing learning systems, we can use a synchronizable cascade decomposition to model a finite automaton environment, and in these analytical applications, the second component has equiprobable states and can often be ignored in analysis. There are many ways of constructing synchronizable cascade decompositions and we will want to use the construction that is easiest for the analytical application. The isomorphism result says that two different construction methods produce isomorphic decompositions provided the decompositions produced have the minimal number of states, and it is often easy to show this by a simple counting argument. This paper confines itself to giving the simpler procedure and proving the homomorphism result. It does not discuss analytical applications
Rewriting of Regular Expressions and Regular Path Queries

AbstractRecent work on semi-structured data has revitalized the interest in path queries, i.e., queries that ask for all pairs of objects in the database that are connected by a path conforming to a certain specification, in particular to a regular expression. Also, in semi-structured data, as well as in data integration, data warehousing, and query optimization, the problem of view-based query rewriting is receiving much attention: Given a query and a collection of views, generate a new query which uses the views and provides the answer to the original one. In this paper we address the problem of view-based query rewriting in the context of semi-structured data. We present a method for computing the rewriting of a regular expression E in terms of other regular expressions. The method computes the exact rewriting (the one that defines the same regular language as E) if it exists, or the rewriting that defines the maximal language contained in the one defined by E, otherwise. We present a complexity analysis of both the problem and the method, showing that the latter is essentially optimal. Finally, we illustrate how to exploit the method for view-based rewriting of regular path queries in semi-structured data. The complexity results established for the rewriting of regular expressions apply also to the case of regular path queries
Inherent Complexity of Recursive Queries

AbstractWe give lower bounds on the complexity of certain Datalog queries. Our notion of complexity applies to compile-time optimization techniques for Datalog; thus, our results indicate limitations of these techniques. The main new tool is linear first-order formulas, whose depth (respectively, number of variables) matches the sequential (respectively, parallel) complexity of Datalog programs. We define a combinatorial game (a variant of Ehrenfeucht–Fraı̈ssé games) that can be used to prove nonexpressibility by linear formulas. We thus obtain lower bounds for the sequential and parallel complexity of Datalog queries. We prove syntactically tight versions of our results, by exploiting uniformity and invariance properties of Datalog queries
Maximizing Sharing of Protected Information

AbstractDespite advances in recent years in the area of mandatory access control in database systems, today's information repositories remain vulnerable to inference and data association attacks that can result in serious information leakage. Without support for coping against these attacks, sensitive information can be put at risk because of release of other (less sensitive) related information. The ability to protect information diclosure against such improper leakage would be of great benefit to governmental, public, and private institutions, which are, today more than ever, required to make portions of their data available for external release. In this paper we address the problem of classifying information by enforcing explicit data classification as well as inference and association constraints. We formulate the problem of determining a classification that ensures satisfaction of the constraints, while at the same time guaranteeing that information will not be overclassified. We present an approach to the solution of this problem and give an algorithm implementing it which is linear in simple cases, and quadratic in the general case. We also analyze a variant of the problem that is NP-complete
A Framework for Measuring Differences in Data Characteristics

AbstractA data mining algorithm builds a model that captures interesting aspects of the underlying data. We develop a framework for quantifying the difference, called the deviation, between two datasets in terms of the models they induce. In addition to being a quantitative, intuitively interpretable measure of difference, the deviation between two datasets can also be computed very fast. Our framework covers a wide variety of models including frequent itemsets, decision tree classifiers, and clusters, and captures standard measures of deviation such as the misclassification rate and the chi-squared metric as special cases. We also show how statistical techniques can be applied to the deviation measure to assess whether the difference between two models is significant (i.e., whether the underlying datasets have statistically significant differences in their characteristics), and discuss several practical applications
Aggregate Operators in Constraint Query Languages

AbstractWe investigate the problem of how to extend constraint query languages with aggregate operators. We deal with standard relational aggregation, and also with aggregates specific to spatial data, such as volume. We study several approaches, including the addition of a new class of approximate aggregate operators which allow an error tolerance in the computation. We show how techniques of M. Karpinski and A. Macintyre (in “Structures in Logic and Computer Science: A Selection of Essays in Honor of A. Ehrenfeucht,” Springer Lecture Notes on Computer Science 1261, pp. 162–173, Springer-Verlag, Berlin, 1997) and P. Koiran (in “FOCS '95,” pp. 134–141) based on VC-dimension can be used to give languages with approximation operators, but also show that these languages have a number of shortcomings. We then give a set of results showing that it is impossible to get constraint-based languages that admit definable aggregation operators, both for exact operators and for approximate ones. These results are quite robust, in that they show that closure under aggregation is problematic even when the class of functions permitted in constraints is expanded. This motivates a different approach to the aggregation problem. We introduce a language FO+Poly+Sum, which permits standard discrete aggregation operators to be applied to the outputs of range-restricted constraint queries. We show that this language has a number of attractive closure and expressivity properties, and that it can compute volumes of linear-constraint databases
Querying Incomplete Information in Semistructured Data

AbstractSemistructured data occur in situations where information lacks a homogeneous structure and is incomplete. Yet, up to now the incompleteness of information has not been reflected by special features of query languages. Our goal is to investigate the principles of queries that allow for incomplete answers. We do not present, however, a concrete query language. Queries over classical structured data models contain a number of variables and constraints on these variables. An answer is a binding of the variables by elements of the database such that the constraints are satisfied. In the present paper, we loosen this concept in so far as we allow also answers that are partial; that is, not all variables in the query are bound by such an answer. Partial answers make it necessary to refine the model of query evaluation. The first modification relates to the satisfaction of constraints: in some circumstances we consider constraints involving unbound variables as satisfied. Second, in order to prevent a proliferation of answers, we only accept answers that are maximal in the sense that there are no assignments that bind more variables and satisfy the constraints of the query. Our model of query evaluation consists of two phases, a search phase and a filter phase. Semistructured databases are essentially labeled directed graphs. In the search phase, we use a query graph containing variables to match a maximal portion of the database graph. We investigate three different semantics for query graphs, which give rise to three variants of matching. For each variant, we provide algorithms and complexity results. In the filter phase, the maximal matchings resulting from the search phase are subjected to constraints, which may be weak or strong. Strong constraints require all their variables to be bound, while weak constraints do not. We describe a polynomial algorithm for evaluating a special type of queries with filter constraints, and assess the complexity of evaluating other queries for several kinds of constraints. In the final part, we investigate the containment problem for queries consisting only of search constraints under the different semantics
Polymorphic Type Inference for the Relational Algebra

AbstractWe give a polymorphic account of the relational algebra. We introduce a formalism of “type formulas” specifically tuned for relational algebra expressions, and present an algorithm that computes the “principal” type for a given expression. The principal type of an expression is a formula that specifies, in a clear and concise manner, all assignments of types (sets of attributes) to relation names, under which a given relational algebra expression is well-typed, as well as the output type that expression will have under each of these assignments. Topics discussed include complexity and polymorphic expressive power
Tracking Join and Self-Join Sizes in Limited Storage

AbstractThis paper presents algorithms for tracking (approximate) join and self-join sizes in limited storage, in the presence of insertions and deletions to the data set(s). Such algorithms detect changes in join and self-join sizes without an expensive recomputation from the base data, and without the large space overhead required to maintain such sizes exactly. Query optimizers rely on fast, high-quality estimates of join sizes in order to select between various join plans, and estimates of self-join sizes are used to indicate the degree of skew in the data. For self-joins, we consider two approaches proposed in N. Alon et al. (The space complexity of approximating the frequency moments, J. Comput. System Sci.58 (1999), 137–147), which we denote tug-of-war and sample-count . We present fast algorithms for implementing these approaches, and extensions to handle deletions as well as insertions. We also report on the first experimental study of the two approaches, on a range of synthetic and real-world data sets. Our study shows that tug-of-war provides more accurate estimates for a given storage limit than sample-count, which in turn is far more accurate than a standard sampling-based approach. For example, tug-of-war needed only 4-256 memory words, depending on the data set, in order to estimate the self-join size to within a 15% relative error; on average, this is over 4 times (50 times) fewer memory words than needed by sample-count (standard sampling, resp.) to obtain a similar accuracy. For joins, we propose schemes based on maintaining a small signature of each relation independently, such that join sizes can be quickly and accurately estimated between any pair of relations using only these signatures. We show that taking random samples for join signatures can lead to an inaccurate estimation unless the sample size is quite large; moreover, we show that no other signature scheme can significantly improve upon sampling without further assumptions. These negative results are shown to hold even in the presence of sanity bounds. On the other hand, we present a fast join signature scheme based on tug-of-war signatures that provides guarantees on join size estimation as a function of the self-join sizes of the joining relations; this scheme can significantly improve upon the sampling scheme
Reducing the Number of Solutions of NP Functions

AbstractWe study whether one can prune solutions from NP functions. Though it is known that, unless surprising complexity class collapses occur, one cannot reduce the number of accepting paths of NP machines, we nonetheless show that it often is possible to reduce the number of solutions of NP functions. For finite cardinality types, we give a sufficient condition for such solution reduction. We also give absolute and conditional necessary conditions for solution reduction, and in particular we show that in many cases solution reduction is impossible unless the polynomial hierarchy collapses
Output String Languages of Compositions of Deterministic Macro Tree Transducers

AbstractThe composition of total deterministic macro tree transducers gives rise to a proper hierarchy with respect to their output string languages (these are the languages obtained by taking the yields of the output trees). There is a language not in this hierarchy which can be generated by a (quite restricted) nondeterministic string transducer, namely, a two-way generalized sequential machine. Similar results hold for attributed tree transducers, for controlled EDT0L systems, and for YIELD mappings (which proves properness of the IO-hierarchy). Witnesses for the properness of the macro tree transducer hierarchy can already be found in the latter three hierarchies
LTL Is Expressively Complete for Mazurkiewicz Traces

AbstractA long standing open problem in the theory of (Mazurkiewicz) traces has been the question whether LTL (linear temporal logic) is expressively complete with respect to the first order theory. We solve this problem posi- tively for finite and infinite traces and for the simplest temporal logic, which is based only on next and until modalities. Similar results were established previously, but they were all weaker, since they used additional past or future modalities. Another feature of our work is that our proof is direct and does not use any reduction to the word case
Lower Bounds for Linearly Transformed OBDDs and FBDDs

AbstractLinearly transformed ordered binary decision diagrams (LTOBDDs) have been suggested as a generalization of OBDDs for the representation and manipulation of Boolean functions. Instead of variables as in the case of OBDDs, parities of variables may be tested at the nodes of an LTOBDD. By this extension it is possible to represent functions in polynomial size that do not have polynomial size OBDDs, e.g., the characteristic functions of linear codes. In this paper lower bound methods for LTOBDDs and some generalizations of LTOBDDs are presented and applied to explicitly defined functions. By the lower bound results it is possible to compare the set of functions with polynomial size LTOBDDs and their generalizations with the set of functions with polynomial size representations for many other restrictions of BDDs
Interval Consistency of Asynchronous Distributed Computations

AbstractAn interval of a sequential process is a sequence of consecutive events of this process. The set of intervals defined on a distributed computation defines an abstraction of this distributed computation, and the traditional causality relation on events induces a relation on the set of intervals that we call I-precedence. An important question is then, “Is the interval-based abstraction associated with a distributed computation consistent?” To answer this question, this paper introduces a consistency criterion named interval consistency (IC). Intuitively, this criterion states that an interval-based abstraction of a distributed computation is consistent if its I-precedence relation does not contradict the sequentiality of each process. More formally, IC is defined as a property of a precedence graph. Interestingly, the IC criterion can be operationally characterized in terms of timestamps (whose values belong to a lattice). The paper uses this characterization to design a versatile protocol that, given intervals defined by a daemon whose behavior is unpredictable, breaks them (in a nontrivial manner) in order to produce an abstraction satisfying the IC criterion. Applications to communication-induced checkpointing are suggested
Reducibility between Classes of Port Graph Grammar

AbstractThis paper introduces a general notion of static-port graph grammar (PGG) that encompasses existing formalisms that have been independently proposed, such as linear graph grammars, interaction nets and partial sharing graphs. These formalisms provide a computational framework for asynchronous computation that respects a very rigorous notion of local interaction, and have proven a suitable basis for the internal representation of program and data in the compilation of high-level programming languages for distributed execution. The general class of PGGs introduced in this paper provides a more expressive computational framework, still possessing asynchronous concurrency but which has a reduction mechanism that is complex and non-local. The increased expressiveness of this calculus is shown by means of a motivating example of concurrent pattern matching. However, the non-locality of the calculus renders it unsuitable for direct implementation. The paper introduces a class of simple PGGs, a local rewriting calculus suitable for distributed implementation. The central result of the paper is to show how the general class of PGGs may be reduced to the simple PGGs, in a manner that fully preserves all the potential for concurrent rewriting present in the original
Online Learning of Binary and n-ary Relations over Clustered Domains

AbstractWe consider the online learning problem for binary relations defined over two finite sets, each clustered into a relatively small number k,l of types (such a relation is termed a (k,l)-binary relation), extending the models of S. Goldman, R. Rivest, and R. Schapire (1993, SIAM J. Comput.22, 1006–1034). We investigate the learning complexity of (k,l)-binary relations with respect to both the self-directed and adversary-directed learning models. We also generalize this problem to the learning problem for (k1,…,kd)-d-ary relations. In the self-directed model, we exhibit an efficient learning algorithm which makes at most kl+(n−k)log k+(m−l)log l mistakes, where n and m are the number of rows and columns, roughly twice the lower bound we show for this problem, 14⌊log k⌋⌊log l⌋+12(n−k) ⌊log k⌋+12(m−l) ⌊log l⌋. In the adversary-directed model, we exhibit an efficient algorithm for the (2,2)-binary relations, which makes at most n+m+2 mistakes, only two more than the lower bound we show for this problem, n+m. As for (k1,…,kd)-d-ary relations, we obtain lower bounds and upper bounds on the number of mistakes in the self-directed model, teacher-directed model, and adversary-directed model. Finally we show that, although the sample consistency problem for (2,2)-binary relations is solvable in polynomial time, the same problem for (2,2,2)-ternary relations is already NP-complete
Determinism versus Nondeterminism for Linear Time RAMs with Memory Restrictions

AbstractOur computational model is a random access machine with n read only input registers each containing clogn bits of information and a read and write memory. We measure the time by the number of accesses to the input registers. We show that for all k there is an ε>0 so that if n is sufficiently large then the elements distinctness problem cannot be solved in time kn with εn bits of read and write memory; that is, there is no machine with this value of the parameters which decides whether there are two different input registers whose contents are identical
Finding Similar Regions in Many Sequences

AbstractAlgorithms for finding similar, or highly conserved, regions in a group of sequences are at the core of many molecular biology problems. Assume that we are given n DNA sequences s1, …, sn. The Consensus Patterns problem, which has been widely studied in bioinformatics research, in its simplest form, asks for a region of length L in each si, and a median string s of length L so that the total Hamming distance from s to these regions is minimized. We show that the problem is NP-hard and give a polynomial time approximation scheme (PTAS) for it. We then present an efficient approximation algorithm for the consensus pattern problem under the original relative entropy measure. As an interesting application of our analysis, we further obtain a PTAS for a restricted (but still NP-hard) version of the important consensus alignment problem allowing at most constant number of gaps, each of arbitrary length, in each sequence