System and method for querying an append-only database-system

Abstract: To produce a continuous query for an append-only database, a client defined query first is converted into its minimal bounding monotone query if not already monotonic non-decreasing (hereinafter referred to as "monotonic increasing"). This monotonic query, in turn, is converted into an incremental query. The resulting monotonically increasing incremental query (Q') then is installed on the database as a stored procedure that takes two date/time parameters (hereinafter referred to as "time" parameters), one of which (τ) identifies the last time the procedure was executed, and the other of which (t) identifies the current time. All database records are timestamped as of the time that they are entered into the database. Thus, in operation, more or less standard procedure calls periodically invoke to each of the stored query procedures, thereby periodically executing the incremental queries over database records that have timestamps spanning successive time slots.

Description: This invention relates to append-only databases and, more particularly,to an efficient query mechanism for extracting selected information from such a database more or less continuously. As is known, records are added to "append-only" databases as they arrive, and those records thereafter are neither deleted nor modified. Typically, the records are assembled into tables within such a database for indexing the underlying source documents by various values, such as by author, date, keywords, and title for a database of news messages, or by sender, recipient, subject, copy distribution, date, and "in reply to" for a database of mail messages. "Continuous queries" are issued once to henceforth run "continually" over the database until they are modified or deleted . As will be appreciated, this is a useful class of queries for filtering streams of electronic documents, such as mail messages or news articles, in situations where there is a need or desire to identify documents that are of special interest to particular clients (i.e., particular users and/or particular application programs). Among the challenges that are involved in implementing continuous queries are: avoiding nondeterministic results (i.e., the results returned to the user should be independent of the time and frequency at which the query is executed), minimizing the return of duplicate results, and avoiding inefficiencies in the execution of the query. As will be evident, the wastefulness of returning to the clients duplicate results can be avoided by having the database system maintain a record of all of the results that have been returned to each client in response to each query, so that the only results that would be subsequently returned to that same client, when the same query is re-executed, would be those that cannot be found in that record. Clearly, however, this solution to the problem inherently is inefficient because a significant part of the computational cost of executing the query is likely to be devoted to the selection of records that are subsequently discarded. Some active databases, such as the Alert system that is described in V. Schreier et al., "Alert: An Architecture for Transforming a Passive DBMS into an Active DBMS," Proceedings 17th International Conference on Very Large Databases (VLDB) , Barcelona, Spain, 1991, pp. 469-478, address the efficiency issue by using triggers to execute queries over new data as it arrives. However, there still is a need for an efficient and reliable technique for implementing more or less arbitrary continuous queries in standard SQL (Standard Query Language) because that is the query language of choice for many existing relational databases. Flexibility in formulating these continuous queries is important because it often is desirable to be able to filter a database based on the relationship of documents to each other, or based on the age of the records, or based on annotations that users may have attached to messages, etc. According to the present invention there is provided an append-only database system having a database manager that is installed on a digital computer for constructing at least one database of timestamped records for indexing documents arriving over time in a document stream, said records being timestamped in accordance with the time at which they are appended to said database; characterised by storing means for storing client-specified incremental monotone queries in said system as callable procedures, each of said queries being parametrized by a current time value and by a recorded time value indicating when the query was last executed; means for periodically calling said queries to execute each query over the database records having timestamps falling into the time span between when the query was last run and the current time; and means for returning query results to the users specifying the respective queries. In another aspect of this invention, a method is provided of producing a continuous query for an append-only database, wherein a client defined query is first converted into its minimal bounding monotone query if not already monotonic non-decreasing (hereinafter referred to as "monotonic increasing"). This monotonic query, in turn, is converted into an incremental query, The resulting monotonically increasing incremental query then is installed on the database as a stored procedure that takes two date/time parameters (hereinafter referred to as "time" parameters), one of which (τ) identifies the last time the procedure was executed, and the other of which (t) identifies the current time. All database records are timestamped as of the time that they are entered into the database. Thus, in operation, more or less standard procedure calls periodically invoke to each of the stored query procedures, thereby periodically executing the incremental queries over database records that have timestamps spanning successive time slots. The present invention will be described further, by way of example, with reference to the accompanying drawings, in which: Fig. 1 is a simplified schematic diagram of a distributed computing environment having an append-only database that is suitable for filtering by the continous query mechanism of this invention; Fig. 2 is a simplified flow chart illustrating the execution of a continuous query; and Fig. 3 is a simplified server/client model for a representative implementation of this invention. While the invention is described in some detail hereinbelow with reference to an illustrated embodiment, it is to be understood that there is no intent to limit it to that embodiment. On the contrary, the intent is to cover all modifications, alternatives and equivalents falling within the scope of the invention as defined by the appended claims. I. Overview Turning now to the drawings, and at this point especially to Fig. 1, the present invention suitably is implemented in a generally standard distributed computing environment 19 having a dedicated or non-dedicated database server 21 that is linked to one or more nearby computers 22a-22n and to a gateway 23 by a local area network (LAN) 24. As shown, the computers 22a-22n (or their users) are "clients" for the database services that are offered by the server 21. However, they also may be data sources if, for example, the server 21 constructs a database from the electronic mail messages that are carried by the LAN 24. The gateway 23, in turn, links the LAN 24 to remote clients and/or data sources by dedicated and/or dial-up lines, as at 25 and 26, respectively. For instance, the server 21 might construct a database from a stream of news messages supplied by a remote source (not shown) that is coupled to the gateway 23 or to the server 21 by a dedicated communication channel. In keeping with standard practices, the server 21 typically runs a generally conventional relational database manager 31 on an appropriate digital computer 32 that is equipped with a conventional mass memory 33. Queries issued by the local clients 22a-22n and/or by the remote clients (not shown) are registered with the database manager 31, and the results of the searches that are performed in response to those queries are returned to the appropriate clients under the control of the database manager 31. One-time, ad hoc queries (i.e., those that are executed only once) are handled in accordance with the ordinary query handling mechanisms of the database manager 31, but continuous queries are stored by the database manager 31 as callable processes that can be periodically invoked. II. Continuous Semantics In accordance with this invention, it has been recognized that continuous semantics are required to prevent the return of nondeterministic results when a query is run periodically over an append-only database to which new records can be added over time. "Continuous semantics" exist if the result of a continuous query is the set of data that would be returned if the query were executed at every instant in time. Clearly, however, it is neither possible nor practical to run a query at every instant of time, so this invention implements an equivalent behavior for ensuring that the results that are returned to the client are well-defined and time-independent. More particularly, if Q(t) is the set of records (i.e., the result) returned by the execution of a query Q over a database as it exists at some instant in time t, a fully equivalent result Q M (t) is obtained by executing the query Q as a "continuous" query and accumulating the results up until time t, such that Q M (t) = ∪ Q (s) s≦t Thus, a query that is executed with continuous semantics returns Q M (t), not Q(t). Equation (1) is valid for all monotone increasing queries (i.e., queries that are never matched or, if once matched by a database record, are then matched by that record in perpetuity). As discussed hereinbelow, some interesting queries are not inherently monotone, so they are converted into a "minimum bounding" monotone query Q M , which is the smallest monotone query that returns all of the results of the query Q. A query Q M is monotone only if Q M (t 1 ) ⊆ Q M (t 2 ) whenever t 1 <t 2 . Given a monotone query Q M as defined above, the key to executing the query efficiently is to convert the query into an incremental query Q' that is parametrized by two times; the time τ that the query was last executed, and the current time t. As will be appreciated Q' is only an approximation of Q m (t) - Q M (τ) because of the overlap that occurs between successive instance of the incremental query Q' at the boundary time τ and t. However, the incremental query Q' always returns all new results and only occasionally returns prior results. Accordingly, as shown in Fig. 2, a continuous query is executed by initializing the τ start time parameter τ for the incremental monotone query Q' to a value of - ∞ as at 51 and by then periodically looping through a run routine 52. During this run routine 52, the end time parameter t is set to the current time as at 54 and the incremental query Q' (τ, t) then is executed as at 55. Subsequently, the query results are queued for return to the client that owns the query as at 56, and the time parameter τ is incremented to the current time t as at 57. Thereafter, the routine 52 pauses for some predetermined period of time as at 58, prior to recycling through that same loop. A psuedo-code description of the process illustrated in Fig. 2 follows: Set τ = -∞ FOREVER DO sett : = current time Execute query Q'(τ, t) Return result to user set τ: = t Sleep for some period of time ENDLOOP III. Monotone Queries No query is monotone unless the database in which it is run is append-only. Within that constraint, however, many common SQL queries are monotone. For example, SQL queries that are merely boolean predicates over the column values of a single table are monotone. These monotone queries can include the comparison operators(=, <,>,...) and the boolean operators (AND, OR, and NOT). Queries involving joins also inherently are monotone if the tables that are joined are append-only. There are, however, certain constructs that can result in non-monotone queries, including (1) functions that read the current time, and (2) subqueries prefaced by "NOT EXISTS". Such queries, therefore, are rewritten or otherwise converted into equivalent incremental minimum bounding monotone queries before they are loaded into the database system. It can be shown that a query containing a time-valued expression E is monotone if its only references to time are in subexpressions of the form E< E Now() or ≤ Now(), where Now () is an operator that returns the current time. On the other hand, a query may be non-monotone if it includes any subexpressions of the form E > Now (), E ≥ Now (), E=Now (), or E≠Now(). To assist in computing the minimum bounding monotone equivalent queries for those of the queries that contain subexpressions that are at risk of being non-monotonic, each of the database tables is configured to include a timestamp column, so that each row or "record" that is added to the table is identified by the time (tbl.ts) at which it was added. This permits a tbl.ts< t term, where t is the current time, to be conjunctively combined (i.e., ANDed) with every subexpression of each query (or at least with every time dependent subexpression). This conjunctive tbl.ts<t term does not alter the query because it merely encodes the fact that a record is not in the database prior to the timestamp value that is applied thereto (i.e., until t=tbl.ts). It does, however, timebound the query so that the query only applies to those records that are in the database at the time that the query is executed. This timebounding causes the query to be monotone when applied to an append-only database because once any match is found for the query, that match will always exist. For monotonizing queries containing subexpressions, R, that are prefaced by NOT EXISTS, provision is made for quantifying those subexpressions over time as represented by the timestamp, tbl.ts, values that are applied to the respective database records. As is known, standard SQL does not permit quantifying over t, where t is the current time. It does, however, permit quantifying over the timestamps, tbl.ts, on the database records. If the subexpression R is itself a monotone, the NOT EXISTS preface can be converted into a monotone by using the aforementioned conjunctive tbl.ts<t terms disjunctively to prevent the subexpression R from being tested over any records that were added to the database prior to the earliest time at which the subexpression R might become true. Even more generally, for monotonizing a NOT EXISTS with an arbitrary time-dependent subexpression R(t), the subexpression R(t) is quantified over the timestamped times on the database records to convert NOT EXISTS (...t.. .)into∃tNOT EXISTS(...t...). IV. The Architecture of an Implementation Fig. 3 illustrates the flow of documents through the major components of a client/server model that is suitable for implementing this invention. In this particular implementation, documents, annotations and continuous "filter" queries are stored in a relational database system that is offered commercially by Sybase, Inc. under the name "Transact-SQL". It is to be understood, however, that the functional partitioning of the client/server model can be modified. More particularly, as shown in Fig. 3, documents from local or remote external sources are loaded into an append-only document store 71 at the server by an indexer 72. The indexer 72 reads the incoming documents to parse them into a set of indexed fields that can be referenced in queries. A filterer 73 runs a batch of continuous queries over the documents that are stored in the document store 71 and/or the documents that are stored in an annotation store 74. These queries are substantively defined by, and are run on behalf of, the local and/or remote client or clients. The results of the query or queries that are executed on behalf of any one client, such as the client 22a (Fig. 1) are loaded into an output store 76 that in dedicated to that particular client. The client may browse the results that are accumulated in this output store 76 through the use of a browser 77. Moreover, if desired, an appraiser 78 may be included for facilitating the browsing of the results by applying client specified classifications to them, such as to prioritize and/or categorize them. Alternatively, the client may retrieve the results of its queries by electronic mail through the use of remailer daemon 81 and a conventional mail reader 82. Again, an appraiser 83 may be employed to prioritize and categorize the results that are returned to the client by electronic mail. The reader/browser 84 is the client interface to the database, so it is to be understood that it includes facilities for adding/deleting/editing continuous filter queries under the control of the client, retrieving and displaying query results (including documents that match the client defined queries), organizing the query matching documents into folders, returning client annotated documents to the annotation store 74, causing the database system to run ad hoc queries, and performing other client initiated interactions with the database. Suitably, these interactions are invoked by the client 22a by sending by mail documents to the server 21. In view of the foregoing, it will be understood that the continuous query mechanism that is provided by this invention permits persistent filters to be implemented on append-only databases for extracting information that is of particular interest to any given client or set of clients from a more or less continuous stream of incoming data. The query mechanism that has been provided supports arbitrarily complex queries, including joins of multiple database tables, calls to read the current time, and existential subqueries. The continuous semantics of these queries causes the results to be well-specified and time-independent. A variety of applications can benefit from the use of the continuous query mechanism of this invention. These queries may be used for many different purposes, including information filtering, data monitoring, and supplying active reminders. Indeed, many existing databases, such as those that store payroll data, billing information, stock market reports, reservations or reminder, are append-only and, therefore, are potential hosts for continuous queries.

Claims: An append-only database system having a database manager (31) that is installed on a digital computer (32) for constructing at least one database of timestamped records for indexing documents arriving over time in a document stream, said records being timestamped in accordance with the time at which they are appended to said database; characterised by a client-means for converting monotone queries (Q M ) into incremental queries (Q') by parametrizing said incremental queries by two time parameters, wherein the first time parameter (τ) identifies the last time the incremental query (Q') was executed and the second time parameter (t) identifies the current time; a storing means for storing client-means-specified incremental queries (Q') in said system as callable procedures, each of said procedures being parametrized by said two time parameters; means for periodically calling said procedures to execute each client-means-specified incremental query (Q') over the database records having timestamps falling into the time span between when the query was last run and the current time; and means for returning query results to the users specifying the respective queries. The append-only database system of claim 1, characterised in that said client means further comprises a converting means for converting arbitrary queries (Q) into monotone queries (Q M ). The append-only database system of claim 2, characterised in that said converting means rewrites subqueries prefaced by "NOT EXISTS" into equivalent monotone subqueries. The append-only database system of claim 2, characterised in that said converting means rewrites subqueries of the form E>Now(), E≥Now(), E=Now() and E≠Now() into subqueries only comprising subqueries of the form E<Now() or E≤Now(), wherein Now() is an operator returning the current time. A method of providing a continuous query for use in a database on digital computer, wherein all database records being timestamped as of the time that they are entered into the database, comprising the steps of: converting a monotone query (Q M ) into an incremental query (Q'), wherein said incremental query (Q') is parametrized by two time parameters, wherein the first time parameter (τ) identifies the last time the procedure was executed and the second time parameter (t) identifies the current time; and installing said incremental query (Q') on said database as a stored procedure that takes said first and second time parameter. The method of claim 5, characterised in that the method comprises the further step of converting an arbitrary query (Q) into its minimal bounding monotone query (Q M ). The method of claim 6, characterised in that during the converting step of an arbitrary query (Q) into its minimal bounding monotone query (Q M ) subqueries prefaced by "NOT EXISTS" statement are rewritten into monotone subqueries. The method of claim 6, characterised in that during the converting step of an arbitrary query (Q) into its minimal bounding monotone query (Q M ) subqueries of the form E>Now(), E≥Now(), E=Now() and E≠Now() are rewritten into queries only comprising subqueries of the form E<Now() or E≤Now(), wherein Now() is an operator returning the current time.

Inventors: GOLDBERG DAVID, NICHOLS DAVID A, OKI BRIAN M, TERRY DOUGLAS B, GOLDBERG, DAVID, NICHOLS, DAVID A., OKI,BRIAN M., TERRY, DOUGLAS B
Applicants: XEROX CORP, XEROX CORPORATION
Application year: 1993
Main classification: G06F17
IPC codes: G06F17