ENFrame = Programs + QueriesProbabilistic Data Highlights on published work ( with a bit of vision )

This paper overviews ENFrame, a framework for processing probabilistic data. In addition to relational query processing supported by existing probabilistic database management systems, ENFrame allows programming with loops, assignments, list comprehension, and aggregates to encode complex tasks such as clustering and classification of data retrieved via queries from probabilistic databases. We explain the design choices behind ENFrame, some distilled from the wealth of work on probabilistic databases and some new. We also highlight a few challenges lying ahead. 1. MOTIVATION AND SCOPE Probabilistic data management has gone a long, fruitful way in the last decade [8]: We currently have a good understanding of the space of possible relational and hierarchical data models and its implication on query tractability; the community already delivered several open-source systems that exploit the first-order structure of database queries for scalable inference, e.g., MystiQ, Trio, and MayBMS to name very few, and applications in the space of web data management, e.g., SPROUT2 [3] with more to come in near future from Google Knowledge World. Significantly less effort has been spent on supporting complex data processing beyond mere querying, such as general-purpose programming or even data mining tasks. There is a growing need for computing platforms that would allow users to build applications feeding on uncertain data without worrying about the underlying uncertain nature of such data or the computationally-hard inference task that comes along with it. For tasks that only need to query probabilistic data, existing systems do offer a viable solution. For more complex tasks, however, successful development requires a high level of expertise in probabilistic databases and this hinders the adoption of the existing technology as well as communication between potential users and experts. A similar observation has been recently made in the areas of machine learning [1] and programming languages [4], with the goal of developing a probabilistic programming framework that allows models to be expressed concisely, yet with random variables and probabilities as first-class citizens. This follows a line of work on, e.g., marrying first-order logic with Bayes nets in BLOG [6] and probabilistic extensions of logic programming [5]. Copyright is held by the author/owner(s). BUDA ’14 (SIGMOD/PODS), ACM . The thesis of this paper is that one can build powerful and useful systems for processing probabilistic data that leverage aforementioned existing work by the data management community. ENFrame, our recent probabilistic data processing framework [9], fits this vision. Following the design of the MayBMS/SPROUT probabilistic database system, ENFrame adheres to a sound semantics (the possible-worlds semantics) for its whole processing pipeline; it exploits where possible the structure of queries for scalable processing; and it uses a rich language of probabilistic events to express input correlations, to trace the correlations introduced during task computation, and to enable sensitivity analysis, incremental maintenance, and knowledge compilation-based approaches for approximate inference. A key property inherited from database systems principles is that it separates between the physical and logical representations of probabilistic data [7]: Whereas the physical representation may be that of, e.g., Bayes nets for some sources and (variants of) probabilistic c-tables for others, the users are exposed to a unified relational view of the underlying data. Finally, the users are oblivious to the probabilistic nature of the data: They program as if the input data is plain relational, with no uncertainty or layout intricacy; it is the ENFrame’s job to make sense of the underlying data formats, probabilities, and input correlations, thus relieving the users from requiring expert knowledge on inference, query tractability, and probabilistic models and interpretation of their programs. ENFrame then adds a programming language with rich constructs that allow to express complex tasks such as clustering or nearest neighbour classification intermixed with structured querying: User programs are written in a fragment of Python that so far supports bounded-range loops, if-then-else statements, list comprehension, aggregates, and query calls to external database engines. This addition has non-trivial effects on the other aspects of the system. Firstly, the event language needed to support such tasks is powerful. While propositional formulas over Boolean random variables are expressive enough to capture program computation traces, they can be extremely large and expensive to manage. ENFrame’s event language extends algebraic formalisms based on semirings and semi-modules for probabilistic data [2] that can succinctly encode program events (they can be exponentially more succinct than equivalent propositional formulas or Bayesian nets). Secondly, ENFrame needs to scale up inference of networks of highly interconnected events. It does so using distributed approximate inference algorithms that work for a bulk of events at a time. 1 (O, n) = loadData () # list and number of objects 2 (k, iter) = loadParams () # number of clusters and iterations 3 M = init () # initialise medoids