QUERY PREVIEWS IN NETWORKED INFORMATION SYSTEMS Khoa Doan , Catherine Plaisant and Ben Shneidermany Human-Computer Interaction Laboratory & Department of Computer Sciencey , Institute for Systems Researchy University of Maryland College Park , MD20742 e-mail : fdoan,plaisant,beng@cs.umd.edu phone : -LRB-301-RRB- 405 2725 ; fax : -LRB-301-RRB- 405 6707 CAR-TR-788 CS-TR-3524 ISR-TR-95-90 Oct. 1995 ABSTRACT In a networked information system , there are three major obstacles facing users in a querying process : network performance , data volume and data complexity . In order to overcome these obstacles , we propose a two-phase approach to dynamic query formulation by volume preview . The two phases are the Query Preview and Query Refinement . In the Query Preview phase , users formulate an initial query by selecting desired attribute values . The volume of matching data sets is shown graphically on preview bars which aid users to rapidly eliminate undesired data sets , and focus on a manageable number of relevant data sets . Query previews also prevent wasted steps by eliminating zero-hit queries . When the estimated number of data sets is low enough , the initial query is submitted to the network , which returns the metadata of the data sets for further refinement in the Query Refinement phase . The two-phase approach to query formulation overcomes slow network performance , and reduces the data volume and data complexity problems . This approach is especially appropriate for users who prefer the exploratory method to discover data patterns and exceptions during the query formulation process . Using this approach , we have developed dynamic query user interfaces to allow users to formulate their queries across a networked environment . INTRODUCTION Dynamic Queries For the past several years , research at the Human-Computer Interaction Laboratory has focused on creating dynamic query user interfaces that apply the principles of direct manipulation to the database environment : ffl visual representation of the query 's components ; ffl visual representation of results ; ffl rapid , incremental , and reversible control of the query ; ffl selection by pointing , not typing ; and immediate and continuous feedback . Dynamic queries involve the interactive control by a user of visual query parameters that generate a rapid -LRB- 100 ms update -RRB- , animated , visual display of database search results . The dynamic query approach lets users rapidly , safely , and even playfully explore a database . They can quickly discover where there are clusters , exceptions , gaps , or outliers , and what trends ordinal data reveal -LSB- Shn94 -RSB- . An experiment was conducted comparing the dynamic query interface with a form-based interface and a natural language interface -LSB- WS92 -RSB- . This experiment demonstrated the strengths of dynamic queries for complex queries , trend analysis and exceptions . Information Retrieval Exploration of large networked information resources becomes increasingly difficult as the volume grows . There are three major problems : ffl Network Performance : Current network technology does not support rapid extraction of data , therefore retrieving large amounts of information on the network is often considerably slow , especially when the network traffic is high . ffl Data Volume : The total amount of data sets , which are available from many different data servers on the network , is large . The data volume , which can reach hundred thousands or millions of data points , is too large for users to make decisions on which portions they want to extract . ffl Data Complexity : The large number of attributes of a data set poses a challenge to users as it can be difficult to remember the type and name of all data set attributes , and is even harder to understand the relationship between the data set attributes during the querying process . Traditionally , there are two strategies for information seekers to quickly and efficiently obtain the data in large information retrieval systems -LSB- Mar95 -RSB- . Analytical strategies depend on careful planning , the recall of query terms , relevant iterative query formulation and examination of results . Browsing strategies are heuristic and opportunistic and depend on recognizing relevant information . The analytical strategies require users to have an intensive knowledge of application domains , and to be skillful in reasoning . The browsing strategies are difficult to use when the data volume is extremely large . Our information seeking strategies apply dynamic queries in a two-phase approach to query formulation by volume preview to combine the advantages of the analytical and browsing strategies . For example , dynamic queries are used to reduce the cognitive loads required in the analytical strategies . A two-phase approach reduces the number of undesired data sets and focuses on a manageable number of relevant data sets , which overcomes slow network performance , data volume and data complexity problems in the browsing strategies . In the next section , we present a short survey of related work . Next , we demonstrate how we perform queries by a sequence of volume previews in a simple application called the Restaurant Finder . We then introduce new concepts and foundations for the two-phase approach to query formulation by volume preview , and describe a dynamic query user interface to the EOSDIS -LRB- Earth Observing System ? Data and Information Systems -RRB- , which assists users to formulate their queries in a very large networked information system . Finally , conclusions and future work are presented . RELATED WORK At present , extracting information from the network is performed using World-Wide-Web browsers such as Netscape or Mosaic . The querying technique is primarily based on keywords . Using this traditional technique , users may specify how many data a query should return -LRB- e.g. 20 -RRB- but they never can estimate how many data were ignored , and how representative all the available data are . Querying is time consuming for it frequently retrieves undesired data , or gets zero-hit queries . Users also often fail to find the data if keywords can not be guessed/found . This technique also suffers from slow network performance . The Butterfly system was developed for simultaneously exploring multiple DIALOG bibliographic databases across the Internet using 3D interactive animation techniques -LSB- MRC95 -RSB- . The key technique used by Butterfly is to create a virtual environment that grows under user control as asynchronous query processes link bibliographic records to form citation graphs . Asynchronous query processes reduce the overhead associated with accessing networked databases , and automatically formulated link-generating queries reduce the number of queries that must be formulated by the user . However , the authors confirm that Butterfly is hard to use without the support of a visual language after the experiment -LSB- MRC95 -RSB- . The Attribute Explorer is a graphical interactive tool for visualizing the relationships within multi-attribute data sets -LSB- TSWB94 -RSB- . In the Attribute Explorer , each attribute is mapped to a single dimensional representation -LRB- interactive histograms -RRB- . Sections of an attribute 's histogram can be selected by a variety of means -LRB- e.g. buttons , sliders , etc -RRB- . The effect of one attribute on the others can be explored by selecting values of interest , and viewing the changes in the histograms . The Attribute Explorer is useful for perceiving trends and outliers in the multi-attribute data sets . However , there is neither discussions of applying the technique for querying data in a networked environment nor on how to handle complex data sets . The Aggregate Manipulator -LRB- AM -RRB- allows users to create and decompose aggregates , which are groupings of data , and see their derived properties . In -LSB- GR94 -RSB- , a combination of the Aggregate Manipulator and Dynamic Query provide a highly useful tool for data manipulation functionalities in exploring large data sets such as : controlling scope , selecting focus of attention , and choosing level of detail . The method has been used to implement a data exploration interface to a large realestate application . However , the system does n't deal with querying data in a networked environment . THE RESTAURANT FINDER The Restaurant Finder is designed to help users identify restaurants that fulfill their desires . Users first specify a few criteria of the restaurants that they want , reduce incrementally the number of the available restaurants to a manageable size , and then submit their requests to the network to retrieve further information , and continue refining their queries with additional criteria . Figure 1 : Display of the Restaurant Finder Preview Panel . Initially , there are approximately 50,000 restaurants available for selection in the North East area . The Restaurant Finder aids users to reduce the number of the selected restaurants to under 100 , so that users can retrieve more detailed information from the network . The Restaurant Finder 's user interface provides sliders and buttons for selecting the relevant cuisine , range of cost , range of hours , geographic regions , rating , and charge cards -LRB- see figure 1 -RRB- . As selections are made , the preview bar on the bottom displays the volume of the restaurants in the database that satisfy the users ' request . The preview bar allows users to explore safely through the database , and eliminates the chance of requesting information that is not available . To allow volume preview updates within a tenth of a second , the attribute values must be kept in the high speed storage . Users can quickly see if there are any Chinese restaurants open after midnight . Users may discover that there are more Chinese restaurants than Italian restaurants , but more Italian restaurants than Chinese restaurants , that are open after midnight . When the size of the volume preview bar is below the recommended level , users can click on the retrieve button . Detailed information is retrieved from the network . The map then becomes local , showing each restaurant as a dot , and more parameters become available , for instance , parking space , number of tables , meeting rooms , disabled access , etc. . The query can then be further refined by selecting more precise values . Details on demand for each restaurant remaining after the query refinement -LRB- e.g. the full menus , reviews , photographs , etc -RRB- can be obtained from the network . TWO-PHASE QUERY FORMULATION PROCESS Our approach is based on the volume preview table , which is used to update preview bars during the Query Preview Phase and the Query Refinement Phase . The goal of this approach is to reduce the volume of the data sets to a manageable size , and to prevent zero-hit queries before submitting queries to the network . The reduction process is performed incrementally by selecting rough ranges of values of a few attributes in the Query Preview Phase to selecting more precise values or exact values of more attributes in the Query Refinement Phase -LRB- figure 2 -RRB- . The reduction of the number of the available data sets in the second phase gives users more control over the attribute values . Only a few attributes are displayed in the Query Preview Panel but in the Query Refinement Panel , an exhaustive list of all common attributes are presented for further selection . A complete list of the attributes of any data set can also be obtained for details on demand . QUERY PREVIEW QUERY REFINEMENT Number of Data Sets Number of Attributes Selection of Attr ibute Values Very large Limited Rough Ranges More Precise or Exact Values Exhaustive List of Common Attributes Manageable -LRB- each one is selectable for details-on-demand -RRB- Figure 2 : A comparison table of the two phases of the query formulation process . The architecture of the two-phase approach to query formulation is illustrated in figure 4 . In these two phases , the query is created by direct manipulations such as adjusting the sliders , pressing buttons or using the pointing devices -LRB- e.g. a mouse or a trackball -RRB- . This approach depends on the network data centers being willing to produce and publish tables of contents for relevant sets of categories . Alternatively , web browsers could extract this information . The tables of content should be small enough to load into the high speed storage to support dynamic queries . The idea of using table of contents in the Query Preview Panel is analogous to using the table of contents/index while looking for the relevant information in a new book . Using the table of contents/index helps to estimate the types and size of the available data without reading the book . In a networked information system , the volume preview table is produced by intersection on multiple tables of content . For example , a data center might have N documents -LRB- in the millions -RRB- , and two tables of contents with cardinalities n1 and n2 , for example 40 years and 8 languages . The volume preview table would have n1 * n2 values -LRB- 320 entries for our example -RRB- and its size is independent of N . Such a volume preview table would enable users to discover that there were no Japanese papers before 1965 without even going to the data center -LRB- figure 3 -RRB- . The volume preview table has to be updated periodically -LRB- for example , daily -RRB- . This is a limitation of query previews but the advantages are substantial if frequent queries are anticipated . In the Query Preview Panel , the volume preview table is visualized using bars or a combination of bars , shaded maps , pie charts -LRB- figure 5 -RRB- , which are called the preview bars . A preview bar is used to display the estimated number of the data set hits for an attribute value -LRB- called attribute preview bar -RRB- , or for the query -LRB- called query preview bar -LRB- figure 1 -RRB- -RRB- . In a preview bar -LRB- either an attribute preview bar or the query preview bar -RRB- , the two colors , such as gray and white , may be used to indicate a selection or a non-selection . The width and length -LRB- or even the area -RRB- of the preview bars is proportional to the size of the volume it represents . All the preview bars are tightly-coupled in the Query Preview Panel -LSB- AS94 -RSB- . When an attribute value or range is modified , all the preview bars are updated and immediately visible -LRB- see figures 6,7,8 and 9 -RRB- . The query preview bar also has a recommended level which can be set by users in the Query Preview Phase . When the number of the data set hits exceeds the currently selected recommended level , the preview bar displays a message to . . . . . . . . . ... ... . . . . . Year Lang L2 L8 X11 X12 X13 X139 X140 X239 X240 X839X83X82X81 L1 X21 X22 X23 X840 Y40Y391965Y3Y2Y1 ... . . Xij Xij : the number of the documents in language Li published in year Yj Japanese ... ... ... ... ... 200 135 Figure 3 : The volume preview table produced by the two tables of contents with 40 years and 8 languages . This table is used to update the preview bars users warning them that the number of hits will result in delays and slow operations in the Query Refinement Phase . The volume preview table is especially a useful starting point for the query formulation process when users do n't have an extensive knowledge about the data . In summary , the benefits of the volume preview in the Query Preview Phase are : ffl Provide users useful statistical information about the metadata of the data sets without having to retrieve the data sets from the network ; ffl Aid users to eliminate rapidly undesired data sets , and guide them to focus on the data sets of interest and of a manageable size ; ffl Prevent retrieving undesired data sets via the network which is costly and time-consuming ; ffl Eliminate the chance of retrieving with zero hits ; ffl Support dynamic queries ; and ffl Aid users to discover data set patterns and exceptions . DYNAMIC QUERY USER INTERFACES TO THE EOSDIS Figure 5 : Initial Display of the Query Preview Panel . The Earth Observing System -LRB- EOS -RRB- Data and Information System -LRB- EOSDIS -RRB- is a comprehensive data and information system , developed by NASA under the Mission to Planet Figure 6 : Display of the Query Preview Panel after selecting a parameter group : Atmospheric Dynamics . Earth -LRB- MTPE -RRB- Program . EOSDIS will manage data from NASA 's past and current Earth science research satellites and field measurement programs , providing data archive , distribution , and information management services . Currently , the V0 IMS -LRB- Information Management System -RRB- is the sole user interface that provides access to the EOSDIS so that EOS scientists and users can use it to search and study the EOS data -LSB- Doa95 -RSB- . The V0 IMS is difficult for EOSDIS users without a specific knowledge of the science data to find the right data sets due to the extremely large volume of the available data sets in the EOSDIS . In the VO IMS , users may specify how many data sets a query should return -LRB- e.g. 20 -RRB- but they never can estimate how many data sets were ignored , and how representative all the available data sets are . In addressing the limitations of the V0 IMS , we present a Dynamic Query User Interface to the EOSDIS consisting of the Query Preview Panel and Query Refinement Panel as illustrated in figure 5 and 10 respectively . The Query Preview Panel Figure 8 : Display of the Query Preview Panel after selecting a specific year : 1992 . A Visual Basic prototype of the Query Preview Panel is described in this section . There are three selected attributes displayed in the interface which are the parameter , spatial ... . WS1 WS2 WS3 WS4 WS5 WSnWSm ... ... . Query Preview Panel Query Refinement Panel WEBSITES on the Internet update regularly attribute values select rough data sets refine on demands details Local Databasesubmit initial query return initial query 's metadata QUERY REFINEMENT QUERY PREVIEW Volume Preview Table Figure 4 : Architecture of Two-phase Dynamic Query Approach for Networked Information Systems . Figure 7 : Display of the Query Preview Panel after selecting a parameter value : Sea Surface Temp . and temporal coverage . The parameters of EOS data sets are classified into 9 groups in terms of the types of the data sets they represent -LRB- e.g. Atmospheric Composition , Atmospheric Dynamics , etc -RRB- . The spatial coverage are defined by the continents -LRB- e.g. Africa , Asia , etc -RRB- , oceans -LRB- e.g. Pacific , Atlantic , etc -RRB- or a selectable grid map , and the temporal coverage is measured in terms of years -LRB- e.g. 1986 , 1987 , etc -RRB- . When the Query Preview Panel starts off , it first displays the number of data sets for each parameter group , selected region and year respectively , in the form of the attribute preview bars , as shown in figure 5 . The size of the data sets for each attribute value is proportional to either the area -LRB- e.g. the attribute preview bars of the continents -RRB- , or to the length -LRB- e.g. the attribute preview bars of the years and parameters -RRB- of the corresponding rectangular bars . The query preview bar , which is on the bottom of the Query Preview Panel , displays graphically the total number of the selected data sets in the gray part on the left section of the bar , and the increasingly red parts on the right section of the bar represent the excessive region -LRB- above the recommended level , which is 1000 in figure 7 -RRB- . When the number of the selected data sets exceeds the recommended level -LRB- which results in the overlapping of the gray part over the red parts -RRB- , a warning message is displayed to users . An initial query in the Query Preview Panel may be formulated by first selecting the parameter group of interest , which results in the display of all the available parameters in that group and its corresponding preview bars . As a result of the parameter group selection , the attribute preview bars for each continent and year are also updated to display the corresponding number of data sets that contain one or more parameters of the selected parameter group . For example , if users might be interested in the temperature of US Coastal Waters . Since the US Coast is in the `` North America '' area , they may select the `` By Continents '' option from the Geographical Selection 's popup menu . Using the Data Dictionary facility , users discover that the parameter `` Sea Surface Temp '' that is used to study the temperature of coastal waters is in both the `` Atmospheric Dynamics '' and `` Ocean Dynamic '' parameter groups . The pie chart of the parameter groups shows that there are more data sets in the `` Atmospheric Dynamics '' than in the `` Ocean Dynamic '' . Hence , users may select the `` Atmospheric Dynamics '' in order to get more data . The result of the parameter group selection is illustrated in figure 6 , in which there are 13993 data sets in the query preview bar -LRB- the bottom rectangular bar in the Query Preview Panel -RRB- . Users then select the parameter `` Sea Surface Temp '' , which results in the change of the attribute preview bars representing each continent and year -LRB- figure 7 -RRB- . The updated preview bars represent the number of data sets that contain the parameter `` Sea Surface Temp '' , and the total number of the selected data sets in the query preview bar is now reduced to 909 as illustrated in figure 7 . Users then further reduce the number of the data set hits by choosing a specific year . The attribute preview bars of the non-selected attribute values disclose to users the total of the data set hits they might get if selected . For example , users would n't select the years 1984 or 1985 since the correspondingpreview bars indicate that there are zero data sets in the year 1984 and 1985 -LRB- figure 7 -RRB- . It also reveals that there are the most data sets on the `` Sea Surface Temp '' in the year 1992 -LRB- hence , it was selected -RRB- . The total number of the selected data sets is now reduced to 276 , and users can continue to reduce the volume of the relevant data sets to 91 by selecting `` North America '' which contains the US Coast -LRB- figure 8 -RRB- . Finally , users submit the initial query to the DAACs -LRB- Data Acquisition Archive Centers -RRB- for the extraction of metadata of the selected data sets . The Query Refinement Panel The Query Refinement Panel supports dynamic queries over a local database that stores the metadata of the data sets extracted from the Query Preview Panel . The metadata contains the information of all the attributes of the data sets such as the parameter , sensor , platform , project , data archive centers , processing data level , time , location which are also visually represented in the interface -LRB- figure 10 -RRB- . The main function of the Query Refinement Panel is to support further refinement for the data sets in the first step . Each data set is now represented as a line in the starfield display -LSB- AS94 -RSB- , which is referenced by the two axes representing the size -LRB- vertical axis -RRB- and the time period -LRB- horizontal axis -RRB- of the data sets respectively . By randomly selecting the regions in the `` North America '' map , users may discover that there are more data sets in the US West Coast -LRB- hence it was selected -RRB- . Users further refine the query by selecting more precise values for the parameter , sensor , platform , project , data archive centers , processing data level , etc. . When the query is completely refined , users select the number of returned granules 1 per data set . Users may want to access to details on demand by clicking on a specific data set in the starfield display . The image of the granules and full details of the selected data set are retrieved from DAACs . Subsequently , these graphical and detailed information of the data set are displayed at the bottom right of the Query Refinement Panel , as shown in figure 11 . Users can use the `` timeline '' slider to eliminate the data sets of undesired periods from the starfield display . In both the Query Preview Panel and Query Refinement Panel , the system also supports multiple selection of the attribute values and going back and forth between the two phases . However , the system has several limitations . The attribute preview bar only gives the conjunction of currently selected data sets . It is rather time consuming to go back and forth between the two phases due to slow network performance . Our solution to network transfer problems in some sense defeats the power of relevance feedback and query reformulation but this will need to be tested . CONCLUSIONS AND FUTURE WORK The two-phase approach to query formulationby volume preview appears to be an efficient method to extract or query a very large and complex database . This approach also demonstrates how dynamic queries can be used in a networked environment via the development of a user interface to the EOSDIS . Future work includes : ffl Extend the Query Preview Panel with capabilities to provide users an option to select the attributes of their choice , and to support both generic and specific and user-defined visualization schema -LRB- for the selected attributes -RRB- for presentation . ffl Integrate the Query Preview Panel into multiple platforms -LRB- eg . Unix , Windows -RRB- and information systems -LRB- e.g. NASA 's VO IMS -RRB- . ffl Research on data structures and algorithms to support rapid multi-dimensional search in a large or very large database . ffl Conduct a comparative query performance experiment between the traditional query approach and the approach described here in the context of a networked query environment , 1In the EOSDIS , a data set may consist of thousands or millions granules which is a smallest unit that can managed and displayed in the interface . Hence , only a modest number of granules are retrieved for browsing Figure 9 : Display of the Query Preview Panel after selecting North America . Figure 10 : Display of the Query Refinement Panel in the Data Set Refinement Step Figure 11 : Display of the Query Refinement Panel in the Details-on-Demand Step . with different categories of users , query tasks and application domains . ACKNOWLEDGEMENTS This work is supported in part byNASA -LRB- NAG 52895 -RRB- and by the NSF grant NSFEEC 94-02384 . We thank Teresa Cronnell for her graphic design of the Restaurant Finder prototype . Our thanks also go to Gary Marchionini , Robin Pfister , and Chris Rouff for reviewing the draft paper . REFERENCES AS94 . C. Ahlberg and B. Shneiderman . Visual Information Seeking : Tight Coupling of Dynamic Query Filters with Starfield Displays . In The Proc . Human Factors in Computing Systems Conf , pages 313 ? 319 , 1994 . Doa95 . Khoa Doan . Initial Review of the V0 IMS GUI . Technical report , HCIL , Unisersity of Maryland , 1995 . GR94 . J. Goldstein and S. F. Roth . Using Aggregation and Dynamic Queries for Exploring Large Data Sets . In The Proc . Human Factors in Computing Systems Conf , pages 23 ? 29 , 1994 . Mar95 . G. Marchionini . Information Seeking in Electronic Environments . Cambridge Series on HCI , 1995 . MRC95 . J. D. Mackinlay , R. Rao , and S. K. Card . An Organic User Interface for Searching Citation Links . In The Proc . Human Factors in Computing Systems Conf , pages 67 ? 75 , 1995 . Shn94 . B. Shneiderman . Dynamic Queries for Visual Information Seeking . IEEE Software , pages 70 ? 77 , 1994 . TSWB94 . L. Tweedie , B. Spence , D. Williams , and R. Bhogal . The Attribute Explorer . In The Video Proc . Human Factors in Computing Systems Conf , pages 435 ? 436 , 1994 . WS92 . C. Williamson and B. Shneiderman . The Dynamic HomeFinder : Evaluating dynamic queries in a real-estate information exploration system . In Proc . ACM SIGIR Conf. , pages 339 ? 346 , 1992 .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O B I O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

direct manipulation
dynamic query
information system
network
preview bar
query preview
science data
user interface
volume preview


SLDMagic

An

Improved

Magic

Set

Technique

Stefan

Brass

Institut

f?ur

Informatik

Universit?at

Hannover

Lange

Laube

Hannover

Fed

Rep.

Germany

E-mail

sb@informatik.uni-hannover.de

Abstract

We

present

a

rewriting

method

for

Datalog-programs

which

simulates

SLD-resolution

more

closely

than

the

ordinary

magic

set

method

does

This

is

especially

advantageous

in

the

case

of

tail-recursive

programs

but

already

in

nonrecursive

programs

we

can

often

save

a

number

of

joins

In

contrast

to

the

method

of

Ross

-LSB-

-RSB-

we

do

not

only

solve

the

problem

of

tail-recursion

but

try

to

simulate

SLD-resolution

as

fully

as

possible

Based

on

an

idea

of

Bry

-LSB-

-RSB-

our

method

can

be

described

in

an

elegant

way

by

means

of

a

meta-interpreter

An

especially

nice

feature

of

our

approach

is

that

we

get

many

other

known

optimizations

for

free

when

we

simulate

SLD-resolution

in

this

way

In

contrast

to

our

earlier

method

-LSB-

-RSB-

we

can

now

handle

the

full

range

from

a

direct

simulation

of

SLD-resolution

to

the

standard

magic

set

technique

-LRB-

selectable

by

annotations

within

the

program

-RRB-

We

compare

the

efficiency

of

SLD-resolution

and

magic

sets

within

a

common

framework

explain

their

main

differences

and

give

hints

for

choosing

between

them

Introduction

It

has

been

noted

that

bottom-up

evaluation

of

a

program

rewritten

by

the

magic

set

method

can

be

much

less

efficient

than

SLD-resolution

if

there

are

tail-recursive

rules

-LSB-

-RSB-

This

already

applies

to

one

of

the

most

often

used

example

programs

namely

the

transitive

closure

For

instance

the

ancestor

relation

can

be

described

as

follows

anc

-LRB-

X

Y

-RRB-

parent

-LRB-

X

Y

-RRB-

anc

-LRB-

X

Z

-RRB-

parent

-LRB-

X

Y

-RRB-

anc

-LRB-

Y

Z

-RRB-

To

make

the

example

as

simple

as

possible

let

us

assume

that

we

know

only

the

father

within

some

ancient

dynasty

of

kings

so

the

parent-relation

is

one

straight

line

of

length

n

parent

-LRB-

i

i

-RRB-

i

n

In

B.

Novikov

J.

W.

Schmidt

-LRB-

Eds

-RRB-

ADBIS

Proceedings

of

the

Third

International

Workshop

on

Advances

in

Databases

and

Information

Systems

-LCB-

Moscow

Sept.

MEPhI

Publishing

Then

the

SLD-tree

for

the

query

anc

-LRB-

X

-RRB-

has

nodes

With

an

index

over

the

first

argument

of

parent

any

reasonable

implementation

of

SLD-resolution

should

have

the

running

time

O

n

log

-LRB-

n

-RRB-

However

in

this

particular

ex

ample

the

magic

set

transformation

does

not

improve

the

program

at

all

We

can

derive

m

ancbf

-LRB-

i

-RRB-

corresponding

to

the

query

anc

-LRB-

i

X

-RRB-

for

all

i

n

This

is

intuitively

understandable

since

when

we

want

to

determine

the

ancestors

of

person

number

we

have

to

determine

recursively

the

ancestors

of

person

number

and

so

on

But

when

we

have

m

ancbf

-LRB-

i

-RRB-

we

also

get

anc

-LRB-

i

j

-RRB-

for

all

i

j

n

-LRB-

i.e.

the

answer

facts

to

the

above

queries

-RRB-

Thus

the

behaviour

is

at

least

O

-LRB-

-RRB-

-LRB-

not

counting

the

computations

of

joins

duplicate

elimination

and

so

on

-RRB-

So

the

original

magic

set

method

is

not

always

as

efficient

as

SLD

resolution

not

even

asymptotically

This

does

not

conflict

with

a

well-known

result

of

Ullman

-LSB-

-RSB-

because

he

has

used

QRGT-resolution

for

comparison

not

SLD-resolution

The

problem

is

that

QRGT

resolution

as

well

as

OLDT-resolution

materialize

the

IDB-facts

-LRB-

lemmas

-RRB-

proven

in

an

SLD-tree

But

in

case

of

tail-recursive

programs

-LRB-

and

only

there

-LSB-

-RSB-

-RRB-

a

single

node

in

an

SLD-tree

can

complete

the

proof

an

unbounded

number

of

lemmas

This

problem

has

been

solved

by

Ross

who

introduced

a

variant

of

magic

sets

with

tail-recursion

optimization

-LSB-

-RSB-

The

idea

was

to

add

to

a

tail-recursive

literal

A

the

literal

B

which

is

really

proven

when

the

proof

of

A

is

finished

Ramakrishnan

and

Sudarshan

have

compared

in

-LSB-

-RSB-

the

efficiency

of

Ross

's

method

with

Prolog-evaluation

and

at

least

on

an

abstract

level

the

result

was

that

this

method

is

as

efficient

as

Prolog

This

is

also

suggested

by

-LSB-

-RSB-

where

it

is

shown

that

for

programs

without

tail-recursion

magic

sets

are

as

efficient

as

SLD-resolution

-LRB-

using

a

very

high

level

cost

measure

-RRB-

However

we

will

show

here

that

it

is

possible

to

go

further

in

simulating

SLD-resolution

The

goal

of

Ross

was

to

solve

the

problem

of

tail-recursion

our

goal

is

to

implement

SLD-resolution

on

top

of

a

deductive

database

The

tail-recursion

optimization

is

only

a

consequence

of

our

approach

and

we

get

also

a

number

of

other

known

optimizations

for

free

Furthermore

our

method

can

be

described

simply

and

elegantly

by

adapting

Bry

's

idea

of

using

a

meta-interpreter

to

derive

the

magic

set

method

-LSB-

-RSB-

The

meta-interpreter

given

by

Bry

corresponds

to

the

QRGT

algorithm

therefore

he

got

the

standard

magic-set

transformation

In

fact

-LSB-

-RSB-

also

contains

a

meta-interpreter

which

corresponds

more

directly

to

SLD-resolution

but

it

still

im

plicitly

uses

the

critical

tabulation

of

derived

lemmas

So

we

present

a

bottom-up

meta-interpreter

which

corresponds

exactly

to

SLD-resolution

This

is

in

itself

simple

but

the

partial

evaluation

of

this

meta-interpreter

needs

some

thoughts

We

then

explain

the

main

differences

between

this

implementation

of

SLD-resolution

and

magic

sets

In

short

SLD-resolution

gives

the

called

literals

more

context

information

namely

how

the

proof

will

continue

after

the

literal

is

proven

In

contrast

magic

sets

usually

need

a

join

to

get

this

context

back

The

disadvantage

of

SLD-resolution

is

that

it

might

happen

that

the

same

literal

is

proven

more

than

once

if

it

is

called

in

different

contexts

This

is

avoided

by

magic

sets

but

they

might

introduce

new

recursions

by

merging

different

computations

So

both

methods

have

advantages

of

their

own

and

the

real

optimum

will

be

some

combination

Thus

we

extend

the

meta-interpreter

in

such

a

way

that

it

can

do

both

either

simulate

SLD-resolution

or

magic

sets

-LRB-

selectable

for

every

literal

by

annotations

in

the

program

-RRB-

The

idea

for

combining

the

two

methods

is

that

we

get

magic

sets

from

SLD-resolution

if

we

evaluate

IDB-literals

in

a

subproof

and

explicitly

get

back

the

answer

Something

similar

happens

with

negative

literals

in

SLDNF-resolution

they

are

also

evaluated

by

a

recursive

call

to

SLDNF-resolution

Alternatively

we

can

say

that

the

extended

meta-interpreter

describes

SLD-resolution

with

tabulation

for

some

literals

The

currently

fastest

deductive

database

system

namely

the

XSB-system

-LSB-

-RSB-

does

not

use

bottom-up

evaluation

with

magic

sets

but

a

variant

of

SLD-resolution

with

tabulation

This

means

that

it

has

the

same

problems

with

tail-recursive

programs

and

yet

it

is

faster

than

bottom-up

evaluation

with

magic

sets

So

probably

any

contribution

to

simulate

SLD-resolution

more

directly

could

help

to

make

real

deductive

database

systems

catch

up

with

the

XSB-system

and

fully

utilize

the

theoretical

advantages

of

bottom-up

evaluation

A

Bottom-Up

Simulation

of

SLD-Resolution

Our

goal

in

this

section

is

to

present

a

bottom-up

metainterpreter

which

describes

SLD-resolution

We

consider

only

positive

Datalog-programs

with

range-restricted

rules

As

usual

for

meta-interpreters

the

rules

of

the

given

program

are

treated

as

input

data

so

they

are

stored

in

the

database

We

represent

rules

as

facts

about

a

binary

predicate

rule

where

the

first

argument

is

the

head

literal

and

the

second

argument

is

the

list

of

body

literals

For

instance

anc

-LRB-

X

Z

-RRB-

parent

-LRB-

X

Y

-RRB-

anc

-LRB-

Y

Z

-RRB-

is

stored

as

rule

anc

-LRB-

X

Z

-RRB-

-LSB-

parent

-LRB-

X

Y

-RRB-

anc

-LRB-

Y

Z

-RRB-

-RSB-

Of

course

this

fact

is

neither

Datalog

-LRB-

the

old

predicates

are

now

function

symbols

-RRB-

nor

range-restricted

-LRB-

it

contains

variables

but

does

not

have

a

rule

body

-RRB-

Meta-interpreters

always

need

a

more

general

rule

language

but

this

is

only

temporary

because

we

will

later

try

to

remove

this

by

partial

evaluation

Besides

the

rules

of

the

program

of

course

also

the

facts

in

the

database

are

input

to

our

meta-interpreter

We

assume

that

the

tuples

in

the

database

predicates

are

represented

in

a

common

relation

db

namely

in

the

form

db

p

-LRB-

cn

-RRB-

In

this

way

the

meta-interpreter

does

not

depend

on

the

set

of

EDB-predicates

Finally

the

query

Q

-LRB-

a

single

literal

-RRB-

is

represented

by

the

fact

query

-LRB-

Q

-RRB-

Our

goal

now

is

to

compute

the

nodes

of

the

SLD-tree

more

precisely

the

goals

attached

to

these

nodes

But

we

need

of

course

also

the

computed

substitution

for

the

answer

variables

We

solve

this

problem

by

attaching

to

each

node

the

given

query

literal

It

shares

variables

with

the

goal

at

the

node

and

all

substitutions

are

also

applied

to

the

query

literal

So

when

we

have

derived

the

empty

goal

the

corresponding

instance

of

the

query

has

been

proven

For

example

the

following

node-fact

might

be

derivable

node

anc

-LRB-

X

-RRB-

-LSB-

parent

-LRB-

X

-RRB-

-RSB-

This

means

that

a

parent

X

of

person

number

is

also

an

ancestor

of

person

number

There

are

of

course

also

other

possibilities

to

remember

the

substitution

for

the

answer

variables

but

this

encoding

of

the

nodes

is

especially

useful

when

we

later

have

to

manage

more

than

one

SLD-tree

The

meta-interpreter

consists

of

four

rules

First

we

need

to

create

the

root

node

of

the

SLD-tree

Of

course

the

goal

in

the

root

node

is

the

given

query

literal

node

-LRB-

Query

-LSB-

Query

-RSB-

-RRB-

query

-LRB-

Query

-RRB-

The

second

rule

describes

the

SLD-resolution

step

It

is

the

main

rule

of

the

meta-interpreter

which

does

the

real

work

As

usual

in

SLD-resolution

we

replace

the

first

goal

literal

by

the

body

of

a

rule

with

a

matching

head

literal

node

-LRB-

Query

Child

-RRB-

node

-LRB-

Query

-LSB-

Lit

jRest

-RSB-

-RRB-

rule

-LRB-

Lit

Body

-RRB-

append

-LRB-

Body

Rest

Child

-RRB-

Note

that

bottom-up

evaluation

with

non-ground

facts

does

the

necessary

unification

and

renames

the

variables

of

the

used

facts

before

that

in

order

to

avoid

name

clashes

The

variables

in

the

derived

facts

are

again

normalized

so

that

facts

which

differ

only

in

the

variable

names

are

detected

as

duplicates

This

is

important

for

the

termination

The

third

rule

does

the

evaluation

of

EDB-literals

by

finding

matching

facts

in

the

database

node

-LRB-

Query

Rest

-RRB-

node

-LRB-

Query

-LSB-

Lit

jRest

-RSB-

-RRB-

db

-LRB-

Lit

-RRB-

The

last

rule

derives

the

ground

instances

of

the

query

which

are

proven

once

the

empty

goal

is

reached

answer

-LRB-

Query

-RRB-

node

Query

-LSB-

-RSB-

The

complete

meta-interpreter

is

shown

in

Figure

It

computes

exactly

the

nodes

in

the

SLD-tree

Example

This

meta-interpreter

can

be

executed

by

deductive

database

systems

like

which

allow

structured

terms

and

non-ground

facts

For

instance

consider

again

the

transitive

closure

program

anc

-LRB-

X

Y

-RRB-

parent

-LRB-

X

Y

-RRB-

anc

-LRB-

X

Z

-RRB-

parent

-LRB-

X

Y

-RRB-

anc

-LRB-

Y

Z

-RRB-

available

from

Initialization

-LRB-

Root

Node

-RRB-

node

-LRB-

Query

-LSB-

Query

-RSB-

-RRB-

query

-LRB-

Query

-RRB-

SLD-Resolution

node

-LRB-

Query

Child

-RRB-

node

-LRB-

Query

-LSB-

Lit

jRest

-RSB-

-RRB-

rule

-LRB-

Lit

Body

-RRB-

append

-LRB-

Body

Rest

Child

-RRB-

Evaluation

of

DB-Literal

node

-LRB-

Query

Rest

-RRB-

node

-LRB-

Query

-LSB-

Lit

jRest

-RSB-

-RRB-

db

-LRB-

Lit

-RRB-

Turn

Proven

Query

into

Answer

answer

-LRB-

Query

-RRB-

node

Query

-LSB-

-RSB-

Figure

Bottom-Up

Meta-Interpreter

for

SLD-Resolution

For

simplicity

let

the

parent-relation

contain

the

single

tuple

-LRB-

-RRB-

The

SLD-tree

for

the

query

anc

-LRB-

X

-RRB-

then

looks

as

follows

anc

-LRB-

X

-RRB-

parent

-LRB-

X

-RRB-

X

parent

-LRB-

Y

-RRB-

anc

-LRB-

Y

X

-RRB-

anc

-LRB-

X

-RRB-

parent

-LRB-

X

-RRB-

parent

-LRB-

Y

-RRB-

anc

-LRB-

Y

X

-RRB-

CORAL

answers

the

query

node

-LRB-

Q

N

-RRB-

with

Q

anc

-LRB-

-RRB-

N

-LSB-

anc

-LRB-

-RRB-

-RSB-

Q

anc

-LRB-

-RRB-

N

-LSB-

parent

-LRB-

-RRB-

-RSB-

Q

anc

-LRB-

-RRB-

N

-LSB-

parent

-LRB-

-RRB-

anc

-LRB-

-RRB-

-RSB-

Q

anc

-LRB-

-RRB-

N

-RSB-

Q

anc

-LRB-

-RRB-

N

-LSB-

anc

-LRB-

-RRB-

-RSB-

Q

anc

-LRB-

-RRB-

N

-LSB-

parent

-LRB-

-RRB-

-RSB-

Q

anc

-LRB-

-RRB-

N

-LSB-

parent

-LRB-

-RRB-

anc

-LRB-

-RRB-

-RSB-

-LRB-

Number

of

Answers

-RRB-

The

query

answer

-LRB-

A

-RRB-

returns

A

anc

-LRB-

-RRB-

So

this

approach

directly

simulates

SLD-resolution

Since

we

do

not

compute

the

nodes

themselves

but

only

the

goals

attached

to

them

the

termination

behaviour

is

better

than

that

of

SLD-resolution

For

instance

the

rule

p

-LRB-

X

-RRB-

p

-LRB-

X

-RRB-

poses

no

problem

at

all

since

it

does

not

yield

new

goals

In

general

we

can

guarantee

the

termination

for

all

tail-recursive

Datalog-programs

using

only

finite

database

predicates

We

do

not

suggest

to

simulate

SLD

resolution

for

predicates

which

are

recursive

but

not

tailrecursive

For

programs

containing

different

kinds

of

recursion

we

will

later

present

a

combined

method

which

allows

to

use

the

magic

set

behaviour

-LRB-

tabulation

-RRB-

for

calling

some

literals

Example

But

let

us

have

a

look

at

a

non

tail-recursive

program

for

instance

the

same

generation

example

sg

-LRB-

X

X

-RRB-

parent

-LRB-

X

-RRB-

sg

-LRB-

X

X

-RRB-

parent

-LRB-

X

-RRB-

sg

-LRB-

X

Y

-RRB-

parent

-LRB-

X

Xp

-RRB-

sg

-LRB-

Xp

Yp

-RRB-

parent

-LRB-

Y

Yp

-RRB-

Let

the

query

be

sg

-LRB-

a

-RRB-

If

the

parent-relation

should

be

cyclic

e.g.

consist

of

the

single

tuple

parent

-LRB-

a

a

-RRB-

longer

and

longer

goals

of

the

form

sg

-LRB-

a

Yn

-RRB-

parent

-LRB-

Yn

Yn

-RRB-

parent

-LRB-

-RRB-

are

constructed

This

problem

also

occurs

with

the

counting

method

-LSB-

-RSB-

One

can

say

that

this

method

also

tries

to

simulate

SLD

resolution

in

its

limited

application

area

It

only

uses

the

special

encoding

sg

down

-LRB-

a

n

-RRB-

for

the

above

node

of

the

SLD-tree

In

fact

many

known

optimizations

can

be

understood

by

the

simple

idea

to

simulate

SLD-resolution

-LRB-

see

below

-RRB-

As

mentioned

above

we

can

guarantee

termination

if

all

recursions

are

tail-recursions

-LRB-

and

the

database

is

finite

-RRB-

Definition

-LRB-

Tail-Recursive

Program

-RRB-

A

program

is

at

most

tail-recursive

iff

for

every

rule

A

Bm

the

predicates

of

Bi

i

m

do

not

depend

on

the

predicate

of

A

i.e.

no

body

literal

except

possibly

the

last

is

recursive

Note

that

this

class

of

programs

is

much

larger

than

the

class

for

which

the

right

recursion

optimization

of

-LSB-

-RSB-

is

applicable

Since

the

only

requirement

is

that

the

recursive

literal

is

the

last

literal

we

could

of

course

make

the

same

generation

program

tail-recursive

by

moving

the

recursive

body

literal

to

the

end

of

the

rule

But

for

a

query

of

the

form

sg

-LRB-

a

Y

-RRB-

SLD-resolution

becomes

very

inefficient

with

this

sequence

of

body

literals

-LRB-

parent

is

called

with

both

arguments

free

-RRB-

so

we

would

not

want

the

behaviour

of

SLD-resolution

in

this

case

Lemma

Let

P

be

an

at

most

tail-recursive

program

and

Q

be

a

query

Then

there

is

an

n

IN

such

that

for

every

DB

no

goal

in

the

SLD-tree

for

P

-LSB-

f

Qg

wrt

DB

has

more

than

n

literals

Proof

In

at

most

tail-recursive

programs

a

rule

can

only

applied

recursively

on

its

last

body

literal

so

iterative

recursive

rule

applications

can

not

make

the

stack

of

to

be

proven

literals

longer

Therefore

the

length

-LRB-

number

of

literals

-RRB-

of

nodes

in

the

SLD-tree

is

bounded

An

upper

bound

would

be

the

sum

of

the

lengths

of

all

rule

bodies

So

if

there

are

only

finitely

many

possible

constants

occurring

in

the

goals

-LRB-

e.g.

because

the

program

contains

only

finite

database

predicates

-RRB-

then

the

number

of

possible

goals

is

bounded

and

we

can

guarantee

termination

Efficiency

Comparison

An

important

difference

between

SLD-resolution

and

magic

sets

is

that

SLD-resolution

passes

more

context

into

the

subcomputations

For

instance

when

a

predicate

p

is

called

in

two

different

rules

then

the

two

computations

for

p

are

kept

disjoint

in

SLD-resolution

while

they

are

merged

in

the

magic

set

method

If

the

arguments

for

which

p

is

called

are

more

or

less

disjoint

then

SLD-resolution

is

superior

because

we

do

not

need

to

filter

the

appropriate

results

later

by

means

of

a

join

If

however

p

is

called

for

the

same

arguments

at

different

places

then

SLD-resolution

repeats

the

computation

and

magic

sets

are

superior

-LRB-

at

least

if

the

computation

of

p

is

difficult

-RRB-

Example

Consider

for

instance

the

following

program

computing

grandparent

relation

where

we

prefer

to

use

the

shorthand

p

for

parent

gp

-LRB-

X

Z

-RRB-

p

-LRB-

X

Y

-RRB-

p

-LRB-

Y

Z

-RRB-

p

-LRB-

X

Y

-RRB-

parent

-LRB-

X

Y

-RRB-

Suppose

that

the

query

is

gp

-LRB-

a

Z

-RRB-

The

magic

set

transformation

leads

to

the

following

program

gp

-LRB-

X

Z

-RRB-

m

gpbf

-LRB-

X

-RRB-

p

-LRB-

X

Y

-RRB-

p

-LRB-

Y

Z

-RRB-

m

pbf

-LRB-

X

-RRB-

m

gpbf

-LRB-

X

-RRB-

m

pbf

-LRB-

Y

-RRB-

m

gpbf

-LRB-

X

-RRB-

p

-LRB-

X

Y

-RRB-

p

-LRB-

X

Y

-RRB-

m

pbf

-LRB-

X

-RRB-

parent

-LRB-

X

Y

-RRB-

m

gpbf

-LRB-

a

-RRB-

-LRB-

Since

there

is

only

one

binding

pattern

for

gp

and

p

we

supressed

the

adornments

-RRB-

Now

if

parent

does

not

contain

-LRB-

a

a

-RRB-

the

two

rules

about

m

pbf

yield

in

fact

disjoint

bindings

for

p

But

these

are

collected

together

in

one

set

and

when

we

get

the

resulting

p-tupels

back

we

need

two

joins

to

match

them

with

the

calling

body

literal

In

total

four

joins

are

needed

to

evaluate

this

program

-LRB-

and

probably

even

more

see

below

-RRB-

In

contrast

SLD-resolution

computes

the

following

sequence

of

goals

-LRB-

encoded

in

our

node-facts

-RRB-

ffl

node

gp

-LRB-

a

Z

-RRB-

-LSB-

gp

-LRB-

a

Z

-RRB-

-RSB-

ffl

node

gp

-LRB-

a

Z

-RRB-

-LSB-

p

-LRB-

a

Y

-RRB-

p

-LRB-

Y

Z

-RRB-

-RSB-

ffl

node

gp

-LRB-

a

Z

-RRB-

-LSB-

parent

-LRB-

a

Y

-RRB-

p

-LRB-

Y

Z

-RRB-

-RSB-

ffl

For

every

b

with

parent

-LRB-

a

b

-RRB-

we

get

node

gp

-LRB-

a

Z

-RRB-

-LSB-

p

-LRB-

b

Z

-RRB-

-RSB-

ffl

node

gp

-LRB-

a

Z

-RRB-

-LSB-

parent

-LRB-

b

Z

-RRB-

-RSB-

ffl

For

every

c

with

parent

-LRB-

b

c

-RRB-

we

get

node

gp

-LRB-

a

c

-RRB-

-LSB-

-RSB-

So

the

results

of

the

calls

to

p

appear

directly

at

the

place

where

they

are

needed

no

additional

filtering

is

necessary

Only

when

parent

is

evaluated

a

join

or

a

selection

is

needed

which

gives

two

joins

in

total

We

will

see

later

how

the

intermediate

steps

can

be

optimized

away

If

parent

contained

-LRB-

a

a

-RRB-

SLD-resolution

would

repeat

the

computation

of

p

-LRB-

a

Y

-RRB-

once

with

continuation

-LSB-

p

-LRB-

Y

Z

-RRB-

-RSB-

and

once

with

continuation

-LSB-

-RSB-

Of

course

since

the

computation

of

p

is

easy

this

is

no

problem

But

if

it

would

take

a

long

time

to

compute

p

magic

sets

might

be

better

The

merging

of

computations

done

by

the

magic

set

method

can

sometimes

introduce

recursions

in

the

magically

rewritten

program

and

indeed

this

happens

in

the

above

example

-LRB-

m

pbf

depends

on

p

and

p

depends

on

m

pbf

-RRB-

Since

the

overhead

for

evaluating

recursions

is

relatively

large

p

must

be

rather

complicated

in

order

to

justify

the

effort

for

avoiding

its

duplicate

computation

Example

Another

source

of

multiple

contexts

for

one

goal

literal

are

bindings

for

variables

not

occurring

in

this

literal

For

instance

consider

the

rule

uncle

-LRB-

X

Y

-RRB-

p

-LRB-

X

Z

-RRB-

brother

-LRB-

Z

Y

-RRB-

where

p

is

again

a

computed

parent-relation

-LRB-

e.g.

the

union

of

mother

and

father

-RRB-

Suppose

that

uncle

is

called

with

both

arguments

bound

The

magic

set

method

translates

this

to

uncle

-LRB-

X

Y

-RRB-

m

unclebb

-LRB-

X

Y

-RRB-

p

-LRB-

X

Z

-RRB-

brother

-LRB-

Z

Y

-RRB-

m

pbf

-LRB-

X

-RRB-

m

unclebb

-LRB-

X

Y

-RRB-

So

when

p

is

called

the

given

values

of

Y

are

projected

away

but

after

p

is

computed

we

must

join

the

result

again

with

m

unclebb

in

order

to

get

Y

back

SLD-resolution

retains

Y

during

the

computation

of

p

and

thus

avoids

this

join

Of

course

there

is

a

tradeoff

If

there

are

many

different

Y

values

for

a

single

X-value

magic

sets

can

be

faster

It

is

interesting

to

note

that

we

might

get

both

advantages

if

we

use

and

group

the

-LRB-

X

Y

-RRB-

tuples

by

X

when

we

execute

p

Such

a

nesting

operation

is

not

much

more

expensive

than

the

projection

-LRB-

with

duplicate

elimination

-RRB-

done

by

the

magic

set

method

There

seem

to

be

many

cases

in

which

would

allow

a

more

compact

representation

and

fewer

copying

operations

resulting

in

a

more

efficient

query

evaluation

Let

us

also

note

that

our

approach

yields

the

following

optimizations

as

a

byproduct

ffl

Duplicate

variables

in

body

literals

pose

no

problem

since

we

always

have

the

complete

information

about

the

structure

of

the

goals

In

contrast

the

magic

set

method

needs

an

extra

rectification

phase

ffl

Values

for

anonymous

variables

are

never

represented

explicitly

ffl

Constants

from

the

rules

are

pushed

into

the

called

rules

corresponding

to

a

kind

of

compile

time

magic

set

transformation

ffl

If

the

given

program

is

non-recursive

the

resulting

program

is

also

non-recursive

Of

course

each

of

these

optimizations

is

already

known

in

the

field

of

deductive

databases

but

they

have

been

introduced

separately

Our

common

framework

can

explain

the

deeper

reasons

behind

them

Adding

Magic

Set

Behaviour

So

both

magic

sets

and

our

bottom-up

simulation

of

SLD

resolution

have

advantages

of

their

own

and

it

depends

on

the

application

which

one

is

superior

Therefore

we

introduce

in

this

section

a

parameterized

approach

which

allows

to

choose

between

both

possibilities

on

a

single

literal

basis

In

order

to

simulate

the

magic

set

behaviour

we

allow

that

SLD-resolution

can

call

itself

recursively

for

evaluating

certain

literals

Standard

SLD-resolution

does

a

similar

thing

for

negative

literals

We

believe

that

this

idea

of

starting

a

subproof

and

getting

back

the

explicitly

represented

result

is

the

key

to

understand

the

difference

between

magic

sets

and

the

real

SLD-resolution

For

our

meta-interpreter

we

assume

that

the

user

has

included

literals

to

be

evaluated

via

magic

sets

in

a

special

predicate

call

-LRB-

:-RRB-

-LRB-

not

otherwise

used

in

the

program

-RRB-

Another

possibility

would

be

to

have

table

declarations

for

the

predicates

to

be

evaluated

in

subproofs

In

fact

our

meta-interpreter

can

also

be

understood

as

describing

a

form

of

SLD-resolution

with

tabulation

Of

course

the

best

option

would

be

an

intelligent

optimizer

which

automatically

determines

-LRB-

guesses

-RRB-

where

it

is

better

to

use

SLD-resolution

and

where

magic

sets

are

superior

Our

meta-interpreter

is

generalized

by

adding

two

additional

rules

to

handle

call

-LRB-

:-RRB-

literals

First

such

a

literal

creates

a

new

query

query

-LRB-

Lit

-RRB-

node

-LSB-

call

-LRB-

Lit

-RRB-

j

-RSB-

So

we

now

construct

not

a

single

SLD-tree

but

a

set

of

SLD

trees

one

for

each

recursive

call

Since

every

derived

nodefact

contains

the

root

of

its

tree

as

the

first

argument

we

can

always

identify

the

tree

-LRB-

s

-RRB-

to

which

a

node

belongs

If

the

subproof

started

by

the

above

rule

is

successful

one

or

more

instances

of

the

query

are

derived

in

the

answer

relation

Now

the

following

rule

gets

these

results

back

into

the

main

proof

by

resolving

the

call

-LRB-

:-RRB-

literal

with

the

derived

answer

literals

node

-LRB-

Query

Rest

-RRB-

node

Query

-LSB-

call

-LRB-

Lit

-RRB-

jRest

-RSB-

answer

-LRB-

Lit

-RRB-

The

complete

meta-interpreter

is

shown

in

Figure

Initialization

-LRB-

Root

Node

-RRB-

node

-LRB-

Query

-LSB-

Query

-RSB-

-RRB-

query

-LRB-

Query

-RRB-

SLD-Resolution

node

-LRB-

Query

Child

-RRB-

node

-LRB-

Query

-LSB-

Lit

jRest

-RSB-

-RRB-

rule

-LRB-

Lit

Body

-RRB-

append

-LRB-

Body

Rest

Child

-RRB-

Set

up

Recursive

Call

-LRB-

Derive

Magic

Literal

-RRB-

query

-LRB-

Lit

-RRB-

node

-LSB-

call

-LRB-

Lit

-RRB-

j

-RSB-

Get

Result

of

Recursive

Call

node

-LRB-

Query

Rest

-RRB-

node

Query

-LSB-

call

-LRB-

Lit

-RRB-

jRest

-RSB-

answer

-LRB-

Lit

-RRB-

Evaluation

of

DB

Literals

node

-LRB-

Query

Rest

-RRB-

node

Query

-LSB-

Lit

jRest

-RSB-

db

-LRB-

Lit

-RRB-

Turn

Proven

Query

into

Answer

answer

-LRB-

Query

-RRB-

node

Query

-LSB-

-RSB-

Figure

Meta-Interpreter

for

SLD-Resolution/Magic

Sets

If

no

IDB-literals

is

classified

as

call

-LRB-

:-RRB-

we

obviously

compute

exactly

the

nodes

of

the

single

SLD-tree

as

in

the

above

meta-interpreter

If

all

literals

are

evaluated

in

subproofs

we

get

something

very

similar

to

Bry

's

meta-interpreter

-LSB-

-RSB-

-LRB-

only

the

representation

of

conjunctions

is

a

little

different

-RRB-

In

fact

we

will

argue

below

that

partial

evaluation

yields

exactly

the

magic

set

method

with

supplementary

predicates

where

query-facts

correspond

to

magic

facts

answer

facts

correspond

to

derived

IDB-facts

and

node-facts

correspond

to

facts

of

the

supplementary

predicates

Note

that

the

first

new

rule

can

also

be

understood

as

entering

a

goal

literal

into

a

table

query

and

the

second

rule

then

allows

resolution

with

proven

lemmas

from

the

table

answer

In

this

way

our

meta-interpreter

also

describes

some

form

of

SLD-resolution

with

tabulation

Computation

of

Node

Types

The

above

meta-interpreter

can

be

executed

in

deductive

database

systems

which

allow

bottom-up

evaluation

with

non-ground

facts

Of

course

the

use

of

lists

and

non-ground

facts

significantly

decreases

the

performance

If

we

are

given

a

Datalog-program

consisting

of

range-restricted

rules

without

function

symbols

we

of

course

want

that

the

result

of

the

transformation

also

has

these

nice

properties

The

approach

suggested

by

-LSB-

-RSB-

is

partial

evaluation

with

respect

to

the

given

rules

and

the

query

However

simply

unfolding

the

call

to

the

rule-predicate

does

not

help

much

The

problem

is

of

course

the

relation

node

we

need

more

information

about

the

derivable

node-facts

in

order

to

continue

the

partial

evaluation

To

this

end

we

have

introduced

in

-LSB-

-RSB-

the

notion

of

a

node

type

It

results

from

a

node-fact

by

abstracting

from

the

occuring

constants

and

the

names

of

the

variables

Definition

-LRB-

Node

Type

-RRB-

Let

Xi

and

Ci

-LRB-

i

-RRB-

be

disjoint

sequences

of

variables

We

will

call

the

Ci

parameters

Let

ak

be

the

constants

appearing

in

the

given

program

P

or

query

Q

A

node

type

N

is

a

node-literal

of

the

form

node

Q

-LSB-

An

-RSB-

where

ffl

Q

is

a

literal

and

the

Ai

are

possibly

call

-LRB-

:-RRB-

enclosed

literals

ffl

Q

and

the

Ai

contain

only

variables

from

g

-LSB-

g

and

constants

from

akg

ffl

If

Xi

appears

somewhere

in

N

then

Xi

appear

already

to

the

left

-LRB-

and

the

same

for

Ci

-RRB-

A

node-literal

L

has

type

N

if

there

is

a

mapping

from

the

Ci

to

constants

and

from

the

Xi

to

variables

such

that

N

L.

Beside

node

types

there

are

also

query

types

and

answer

types

which

are

defined

completely

analogously

Example

Consider

again

the

transitive

closure

program

anc

-LRB-

X

Y

-RRB-

parent

-LRB-

X

Y

-RRB-

anc

-LRB-

X

Z

-RRB-

parent

-LRB-

X

Y

-RRB-

anc

-LRB-

Y

Z

-RRB-

In

case

of

the

query

anc

-LRB-

X

-RRB-

the

following

node

types

can

appear

-LRB-

for

any

possible

database

-RRB-

node

anc

-LRB-

-RRB-

-LSB-

anc

-LRB-

-RRB-

-RSB-

node

anc

-LRB-

-RRB-

-LSB-

parent

-LRB-

-RRB-

-RSB-

node

anc

-LRB-

-RRB-

-LSB-

parent

-LRB-

-RRB-

anc

-LRB-

-RRB-

-RSB-

node

anc

-LRB-

-RRB-

-LSB-

true

-RSB-

node

anc

-LRB-

-RRB-

-LSB-

anc

-LRB-

-RRB-

-RSB-

node

anc

-LRB-

-RRB-

-LSB-

parent

-LRB-

-RRB-

-RSB-

node

anc

-LRB-

-RRB-

-LSB-

parent

-LRB-

-RRB-

anc

-LRB-

-RRB-

-RSB-

This

set

of

node

types

corresponds

to

the

nodes

computed

in

Example

However

these

seven

node

types

suffice

for

any

possible

parent-relation

since

can

be

any

ancestor

of

person

number

Of

course

it

is

possible

to

replace

the

constant

from

the

query

by

a

parameter

This

is

usually

done

in

the

magic

set

method

since

in

this

way

the

program

does

not

have

to

be

recompiled

when

the

same

query

is

posed

later

with

a

different

constant

However

if

we

use

the

constants

from

the

query

and

the

rules

the

resulting

program

can

be

slightly

faster

The

computation

of

the

node-types

can

be

done

by

executing

the

meta-interpreter

bottom-up

but

with

the

following

differences

ffl

Since

the

actual

database

is

unknown

at

compile

time

we

use

a

generic

EDB-fact

p

-LRB-

Cn

-RRB-

when

we

have

to

resolve

with

an

EDB-literal

Here

the

parameters

Ci

act

like

variables

but

if

we

have

to

unify

a

real

variable

Xj

and

a

parameter

Ci

we

replace

Xj

by

Ci

and

not

vice

versa

Thus

we

know

at

what

places

constants

appear

ffl

The

variables

in

derived

literals

are

normalized

to

and

the

parameters

to

In

this

way

we

can

detect

when

we

have

derived

the

same

node

type

again

It

is

optionally

possible

to

replace

known

constants

-LRB-

from

the

query

or

the

rules

-RRB-

by

parameters

Ci

This

results

in

the

algorithm

of

Figure

For

simplicity

we

have

only

shown

how

the

second

rule

of

the

meta-interpreter

is

used

to

generate

node

types

but

the

application

of

the

other

rules

is

similar

The

procedures

called

in

this

algorithm

do

the

following

ffl

normalize

lit

and

normalize

goal

rename

variables

and

parameters

as

required

in

Definition

ffl

max

var

index

returns

one

above

the

maximal

index

i

of

a

variable

Xi

occurring

in

a

node

type

ffl

fresh

var

adds

the

given

offset

to

all

numbers

of

variables

occurring

in

a

rule

This

ensures

the

disjointness

of

variables

before

unification

ffl

unify

is

the

standard

unification

procedure

however

variable-to-variable

bindings

are

done

in

such

a

way

that

never

a

Ci

is

replaced

by

an

Xj

ffl

append

is

the

list

concatenation

As

explained

above

it

is

possible

that

longer

and

longer

goals

occur

during

SLD-resolution

In

that

case

our

algorithm

will

not

terminate

However

we

can

guarantee

termination

if

every

recursive

body

literal

which

is

not

the

last

Input

Program

P

Query

Q.

Output

Node

Types

N

Query

Types

Q

Answer

Types

A.

Initialization

N

No

node

types

so

far

A

No

answer

types

so

far

Q

normalize

lit

-LRB-

Q

-RRB-

changed

true

Bottom-Up

Computation

Loop

while

changed

do

changed

false

Meta-Interpreter

Rule

Meta-Interpreter

Rule

foreach

node

type

in

N

do

let

node

Q

-LSB-

An

-RSB-

foreach

rule

A

B

in

P

do

i

max

var

index

-LRB-

-RRB-

fresh

var

-LRB-

i

A

B

-RRB-

unify

-LRB-

-RRB-

if

then

G

append

-LSB-

An

-RSB-

node

-LRB-

Q

G

-RRB-

N

normalize

node

-LRB-

-RRB-

if

N

N

then

N

N

-LSB-

fN

changed

true

Further

Meta-Interpreter

Rules

Figure

Algorithm

to

Compute

Node

Types

literal

of

the

rule

is

evaluated

in

a

subproof

-LRB-

In

fact

it

suffices

if

every

recursive

loop

containing

a

non-tail-recursive

literal

is

broken

somewhere

by

requesting

a

subproof

-RRB-

As

stated

in

Lemma

in

this

case

the

number

of

nodes

in

a

goal

occurring

in

the

SLD-tree

is

bounded

-LRB-

literals

evaluated

in

a

subproof

are

treated

like

EDB-literals

-RRB-

But

since

node

types

can

contain

only

the

finitely

many

constants

occurring

the

program

or

the

query

and

since

the

variables/parameters

are

normalized

there

is

only

a

finite

number

of

possible

node

types

Partial

Evaluation

When

we

know

the

structure

of

the

occurring

node

query

and

answer

literals

it

is

a

simple

task

to

partially

evaluate

the

rules

of

the

meta-interpreter

In

fact

the

specialized

rules

can

already

be

constructed

during

the

computation

of

the

node

types

since

this

computation

simply

simulates

the

rule

application

In

the

rewritten

program

the

node

types

become

predicates

with

the

actual

values

of

the

parameters

Ci

as

arguments

In

this

way

the

Ci

and

Xi

generalize

the

usual

b

f-binding

patterns

For

instance

the

node

type

node

anc

-LRB-

-RRB-

-LSB-

parent

-LRB-

-RRB-

anc

-LRB-

-RRB-

-RSB-

is

used

as

a

predicate

of

arity

one

-LRB-

because

it

contains

one

foreach

node

type

in

N

do

let

node

Q

-LSB-

An

-RSB-

node

params

-LRB-

-RRB-

foreach

rule

A

B

in

P

do

i

max

var

index

-LRB-

-RRB-

fresh

var

-LRB-

i

A

B

-RRB-

unify

-LRB-

-RRB-

if

then

G

append

-LSB-

An

-RSB-

node

-LRB-

Q

G

-RRB-

node

params

-LRB-

-RRB-

N

normalize

node

-LRB-

-RRB-

R

R

-LSB-

nodehN

-LRB-

-RRB-

-LRB-

-RRB-

if

N

N

then

N

N

-LSB-

fN

changed

true

Figure

Specialization

of

Meta-Interpreter

Rule

parameter

-RRB-

We

denote

the

predicate

corresponding

to

node

Q

-LSB-

An

-RSB-

by

nodehQ

Ani

Then

the

fact

node

anc

-LRB-

-RRB-

parent

-LRB-

-RRB-

anc

-LRB-

-RRB-

-LRB-

-RRB-

corresponds

to

the

node

parent

-LRB-

Y

-RRB-

anc

-LRB-

Y

X

-RRB-

in

the

SLD

tree

with

root

anc

-LRB-

X

-RRB-

It

is

actually

very

simple

to

modify

the

program

shown

in

Figure

in

such

a

way

that

it

additionally

computes

the

specialized

rule

instances

All

we

need

is

a

procedure

node

params

which

constructs

an

argument

list

consisting

of

the

parameters

Ci

contained

in

a

given

node

As

an

example

the

program

to

partially

evaluate

the

second

rule

SLD-Resolution

node

-LRB-

Query

Child

-RRB-

node

-LRB-

Query

-LSB-

Lit

jRest

-RSB-

-RRB-

rule

-LRB-

Lit

Body

-RRB-

append

-LRB-

Body

Rest

Child

-RRB-

of

the

meta-interpreter

is

shown

in

Figure

There

are

two

points

worth

noting

ffl

It

is

important

to

select

the

parameters

of

the

given

node

type

N

before

the

unification

with

the

head

literal

and

to

apply

later

the

unifier

Otherwise

if

replaces

a

parameter

Ci

by

a

constant

a

from

the

head

literal

the

argument

list

will

not

be

correctly

constructed

ffl

The

selection

of

the

parameters

of

the

resulting

node

type

must

be

done

before

the

normalization

otherwise

we

would

loose

the

connection

between

head

and

body

of

the

rule

has

to

work

also

with

non-normalized

node

types

and

creates

the

argument

list

in

the

sequence

of

occurrence

not

in

the

sequence

of

their

indices

-LRB-

of

course

every

parameter

is

represented

only

once

in

this

list

-RRB-

Example

Let

us

for

example

consider

the

node

type

node

p

-LRB-

-RRB-

-LSB-

q

-LRB-

-RRB-

-RSB-

and

the

rule

-LRB-

with

fresh

variables

-RRB-

q

-LRB-

a

-RRB-

r

-LRB-

-RRB-

The

argument

list

of

the

given

node

type

is

-LRB-

-RRB-

The

unifier

replaces

by

a

by

then

by

-LRB-

or

the

other

way

round

-RRB-

and

finally

by

-LRB-

or

vice

versa

-RRB-

Before

normalization

the

resulting

node

type

is

node

p

-LRB-

-RRB-

-LSB-

r

-LRB-

-RRB-

-RSB-

Its

parameter

list

is

-LRB-

-RRB-

After

normalization

the

resulting

node

type

is

node

p

-LRB-

-RRB-

-LSB-

r

-LRB-

-RRB-

-RSB-

So

we

finally

get

the

following

rule

node

p

-LRB-

-RRB-

r

-LRB-

-RRB-

-LRB-

-RRB-

node

p

-LRB-

-RRB-

q

-LRB-

-RRB-

-LRB-

a

-RRB-

It

is

optionally

possible

to

normalize

also

the

argument

lists

i.e.

to

replace

as

a

last

step

in

the

argument

lists

by

But

this

is

only

cosmetics

Example

For

the

transitive

closure

example

the

complete

program

resulting

from

our

transformation

is

shown

in

Figure

Obviously

some

optimizations

are

needed

ffl

The

first

four

rules

are

propositional

and

can

already

be

evaluated

during

the

compilation

phase

since

they

do

not

depend

on

the

given

data

ffl

Rules

number

and

are

pure

copying

rules

and

can

be

eliminated

It

is

always

advantageous

to

unfold

a

call

to

a

predicate

p

if

this

is

the

only

call

to

p

and

there

is

only

one

rule

about

p

In

the

example

node

anc

-LRB-

-RRB-

parent

-LRB-

-RRB-

is

defined

only

in

rule

and

called

only

in

rule

and

the

same

happens

with

rules

and

The

state

after

these

optimizations

is

node

anc

-LRB-

-RRB-

true

-LRB-

-RRB-

parent

-LRB-

-RRB-

node

anc

-LRB-

-RRB-

anc

-LRB-

-RRB-

-LRB-

-RRB-

parent

-LRB-

-RRB-

node

anc

-LRB-

-RRB-

true

-LRB-

-RRB-

node

anc

-LRB-

-RRB-

anc

-LRB-

-RRB-

-LRB-

-RRB-

parent

-LRB-

-RRB-

node

anc

-LRB-

-RRB-

anc

-LRB-

-RRB-

-LRB-

-RRB-

node

anc

-LRB-

-RRB-

anc

-LRB-

-RRB-

-LRB-

-RRB-

parent

-LRB-

-RRB-

anc

-LRB-

-RRB-

node

anc

-LRB-

-RRB-

true

-LRB-

-RRB-

Now

the

predicates

node

anc

-LRB-

-RRB-

true

-LRB-

Fact

for

given

Query

-RRB-

query

anc

-LRB-

-RRB-

-LRB-

Root

Node

-RRB-

node

anc

-LRB-

-RRB-

anc

-LRB-

-RRB-

query

anc

-LRB-

-RRB-

-LRB-

SLD-Resolution

with

First

Rule

-RRB-

node

anc

-LRB-

-RRB-

parent

-LRB-

-RRB-

node

anc

-LRB-

-RRB-

anc

-LRB-

-RRB-

-LRB-

SLD-Resolution

with

Second

Rule

-RRB-

node

anc

-LRB-

-RRB-

parent

-LRB-

-RRB-

anc

-LRB-

-RRB-

node

anc

-LRB-

-RRB-

anc

-LRB-

-RRB-

-LRB-

Evaluation

of

DB-Literal

-RRB-

node

anc

-LRB-

-RRB-

true

-LRB-

-RRB-

node

anc

-LRB-

-RRB-

parent

-LRB-

-RRB-

parent

-LRB-

-RRB-

-LRB-

Evaluation

of

DB-Literal

-RRB-

node

anc

-LRB-

-RRB-

anc

-LRB-

-RRB-

-LRB-

-RRB-

node

anc

-LRB-

-RRB-

parent

-LRB-

-RRB-

anc

-LRB-

-RRB-

parent

-LRB-

-RRB-

-LRB-

SLD-Resolution

with

First

Rule

-RRB-

node

anc

-LRB-

-RRB-

parent

-LRB-

-RRB-

-LRB-

-RRB-

node

anc

-LRB-

-RRB-

anc

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

SLD-Resolution

with

Second

Rule

-RRB-

node

anc

-LRB-

-RRB-

parent

-LRB-

-RRB-

anc

-LRB-

-RRB-

-LRB-

-RRB-

node

anc

-LRB-

-RRB-

anc

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

Evaluation

of

DB-Literal

-RRB-

node

anc

-LRB-

-RRB-

true

-LRB-

-RRB-

node

anc

-LRB-

-RRB-

parent

-LRB-

-RRB-

-LRB-

-RRB-

parent

-LRB-

-RRB-

-LRB-

Evaluation

of

DB-Literal

-RRB-

node

anc

-LRB-

-RRB-

anc

-LRB-

-RRB-

-LRB-

-RRB-

node

anc

-LRB-

-RRB-

parent

-LRB-

-RRB-

anc

-LRB-

-RRB-

-LRB-

-RRB-

parent

-LRB-

-RRB-

-LRB-

Turn

Proven

Query

into

Answer

-RRB-

anc

-LRB-

-RRB-

node

anc

-LRB-

-RRB-

true

-LRB-

-RRB-

Figure

Rewritten

Program

-LRB-

Without

Optimizations

-RRB-

and

node

anc

-LRB-

-RRB-

anc

-LRB-

-RRB-

have

identical

definitions

We

can

assume

that

some

form

of

common

subexpression

elimination

detects

this

Let

us

replace

the

two

synonyms

by

a

predicate

With

the

help

of

these

optimizations

we

get

a

very

reasonable

program

for

computing

the

reachable

nodes

-LRB-

-RRB-

parent

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

parent

-LRB-

-RRB-

anc

-LRB-

-RRB-

-LRB-

-RRB-

This

seems

to

be

even

slightly

superior

to

the

standard

optimized

version

of

this

program

-LSB-

-RSB-

m

ancbf

-LRB-

-RRB-

m

ancbf

-LRB-

Z

-RRB-

m

ancbf

-LRB-

X

-RRB-

parent

-LRB-

X

Z

-RRB-

ans

-LRB-

Y

-RRB-

m

ancbf

-LRB-

X

-RRB-

parent

-LRB-

X

Y

-RRB-

But

even

without

any

optimizations

bottom-up

evaluation

of

the

transformed

program

in

Figure

is

already

as

efficient

as

SLD-resolution

in

the

sense

that

any

applicable

rule

instance

corresponds

to

an

edge

in

the

SLD-tree

And

since

we

have

seen

how

much

we

really

still

can

optimize

there

are

good

chances

that

bottom-up

evaluation

really

can

beat

SLD-resolution

Magic

Sets

with

Supplementary

Predicates

Let

us

now

consider

the

special

case

that

all

IDB-literals

are

evaluated

in

subproofs

A

small

inefficiency

in

the

above

meta-interpreter

is

that

we

first

must

turn

the

query

into

a

goal

and

only

then

can

apply

the

rules

In

order

to

improve

this

we

can

treat

already

query

-LRB-

Query

-RRB-

as

representing

the

root

node

This

is

done

by

replacing

the

first

rule

of

the

meta-interpreter

by

the

following

optimized

version

Initialization

-LRB-

Select

Applicable

Rules

for

Query

-RRB-

node

-LRB-

Query

Body

-RRB-

query

-LRB-

Query

-RRB-

rule

-LRB-

Query

Body

-RRB-

Now

all

node

types

have

the

form

node

A

-LSB-

Bi

Bn

-RSB-

where

A

Bn

is

a

rule

of

the

given

program

and

replaces

variables

which

are

bound

in

the

call

to

A

or

in

Bi

by

parameters

-LRB-

and

normalizes

the

variable

names

-RRB-

Note

that

only

values

of

variables

occurring

in

Bi

Bn

or

A

are

represented

by

parameters

So

the

arguments

correspond

exactly

to

the

arguments

of

the

i-th

supplementary

predicate

of

this

rule

Furthermore

if

we

compare

the

resulting

rules

with

the

magic

set

transformation

of

-LSB-

-RSB-

we

see

a

striking

similarity

The

rules

for

the

zeroth

supplementary

predicates

are

generated

by

the

first

rule

of

our

meta-interpreter

the

rules

for

the

other

supplementary

predicates

are

generated

by

the

fourth

and

fifth

rule

the

rules

for

the

magic

predicates

are

generated

by

the

third

rule

and

the

rules

for

IDB-predicates

correspond

to

our

last

rule

To

be

precise

there

are

only

two

very

minor

differences

ffl

As

presented

in

-LSB-

-RSB-

there

is

no

n-th

supplementary

predicate

for

a

rule

with

n

body

literals

This

would

correspond

to

nodes

of

the

form

node

A

-LSB-

-RSB-

It

is

a

simple

optimization

to

eliminate

it

ffl

Second

if

two

rules

have

the

same

head

A

and

body

rest

Bi

Bn

we

merge

the

two

supplementary

predicates

But

this

can

only

improve

the

performance

Conclusions

This

work

has

tried

to

clarify

the

relation

between

SLD

resolution

and

magic

sets

and

has

introduced

a

joint

generalization

through

a

meta-interpreter

It

was

the

goal

of

the

magic

set

transformation

to

combine

all

advantages

of

bottom-up

and

top-down

evaluation

-LRB-

i.e.

SLD-resolution

-RRB-

However

this

goal

was

not

fully

reached

and

our

approach

goes

further

in

this

direction

Our

rewriting

method

allows

to

decide

between

SLD

resolution

and

magic

sets

for

every

literal

and

choose

whatever

seems

better

If

there

is

only

one

call

of

a

predicate

or

we

can

see

that

the

arguments

are

disjoint

or

the

computation

is

so

simple

that

it

does

no

harm

to

repeat

it

then

we

should

choose

SLD-resolution

Also

the

potential

savings

for

tail-recursive

literals

are

so

big

that

they

probably

outweigh

the

cost

of

duplicate

computations

On

the

other

hand

recursive

calls

which

are

not

tail-recursive

can

only

be

processed

via

magic

sets

A

first

prototype

implementation

of

the

computation

of

node

types

and

the

partially

evaluated

meta-interpreter

is

available

from

ftp

ftp.informatik.uni-hannover

de

An

especially

nice

feature

of

our

method

is

that

it

gives

many

known

optimizations

for

free

namely

ffl

No

extra

rectification

is

needed

ffl

Values

for

anonymous

variables

are

never

explicitly

represented

ffl

Nonrecursive

programs

are

transformed

into

nonrecursive

programs

ffl

Already

at

compile-time

constants

are

pushed

downward

into

the

called

rules

ffl

Our

method

pushes

also

non-equality

conditions

such

as

X

into

the

called

literals

These

conditions

are

represented

in

the

context

or

continuation

of

a

literal

and

a

good

selection

function

would

evaluate

them

as

soon

as

X

becomes

bound

In

fact

we

should

check

the

consistency

of

the

goals

already

at

compiletime

-LRB-

as

far

as

possible

-RRB-

In

the

standard

magic

set

transformation

such

optimizations

are

not

easy

-LSB-

-RSB-

At

the

moment

we

have

either

passed

the

complete

context

with

the

called

literal

or

no

context

at

all

This

seemed

reasonable

since

as

soon

as

we

have

not

the

complete

context

we

need

a

join

to

get

back

to

the

caller

However

incomplete

contexts

can

still

help

to

delimit

the

computations

within

the

called

literal

For

instance

we

would

like

to

pass

the

condition

X

even

to

a

left-recursive

literal

where

we

can

not

represent

all

possible

continuations

completely

Let

us

also

note

that

many

deductive

database

systems

have

some

procedure

call

mechanism

for

the

predicates

and

do

not

use

the

pure

magic

set

transformation

This

leads

to

some

double

computations

but

avoids

the

merging

of

different

calls

to

a

predicate

Our

approach

can

be

understood

as

going

further

in

this

direction

It

seems

that

future

improvements

need

to

look

at

the

internal

data

structures

We

still

have

considered

only

a

very

abstract

level

and

talked

about

SLD-trees

and

applicable

rule

instances

Now

the

time

has

come

to

look

at

the

lower

implementation

levels

Currently

Prolog

implementations

are

still

much

faster

than

deductive

databases

-LRB-

for

instance

the

ECLiPSe

Prolog-System

is

times

faster

than

the

CORAL

system

if

the

problems

mentioned

above

do

not

occur

-RRB-

This

probably

can

be

improved

or

else

it

would

be

useful

to

know

the

theoretical

reasons

why

setoriented

evaluation

is

inherently

slower

than

tuple-oriented

evaluation

-LRB-

what

I

do

not

believe

-RRB-

Acknowledgement

I

would

like

thank

Udo

Lipeck

for

leading

me

the

way

to

the

research

presented

here

and

for

numerous

helpful

suggestions

on

an

earlier

version

of

this

paper

I

also

profited

from

discussions

with

Klaus-Dieter

Schewe

Torsten

Polle

and

Ulrich

Zukowski

References

-LSB-

-RSB-

F.

Bancilhon

and

R.

Ramakrishnan

An

amateur

's

introduction

to

recursive

query

processing

In

C.

Zaniolo

editor

Proc

of

SIGMOD

pages

-LCB-

-LSB-

-RSB-

S.

Brass

Magic

sets

vs.

SLD-resolution

In

J.

Eder

and

L.

A.

Kalinichenko

editors

Advances

in

Databases

and

Information

Systems

-LRB-

ADBIS

-RRB-

pages

-LCB-

Springer

-LSB-

-RSB-

F.

Bry

Query

evaluation

in

recursive

databases

bottom-up

and

top-down

reconciled

Data

Knowledge

Engineering

-LCB-

-LSB-

-RSB-

D.

B.

Kemp

K.

Ramamohanarao

and

Z.

Somogyi

Right

left

and

multi-linear

rule

transformations

that

maintain

context

information

In

D.

McLeod

R.

SacksDavis

and

H.

Schek

editors

Proc

Very

Large

Data

Bases

Int

Conf

-LRB-

VLDB

-RRB-

pages

-LCB-

Morgan

Kaufmann

Publishers

-LSB-

-RSB-

I.

S.

Mumick

S.

J.

Finkelstein

H.

Pirahesh

and

R.

Ramakrishnan

Magic

conditions

In

Proc

of

the

Ninth

ACM

SIGACT-SIGMOD-SIGART

Symposium

on

Principles

of

Database

Systems

-LRB-

PODS

-RRB-

pages

-LCB-

-LSB-

-RSB-

R.

Ramakrishnan

and

S.

Sudarshan

Top-down

vs.

bottom-up

revisited

In

V.

Saraswat

and

K.

Ueda

editors

Proc

of

the

Int

Symposium

on

Logic

Programming

pages

-LCB-

MIT

Press

-LSB-

-RSB-

K.

A.

Ross

Modular

acyclicity

and

tail

recursion

in

logic

programs

In

Proc

of

the

Tenth

ACM

SIGACT

SIGMOD-SIGART

Symp

on

Princ

of

Database

Systems

-LRB-

PODS

-RRB-

pages

-LCB-

-LSB-

-RSB-

K.

Sagonas

T.

Swift

and

D.

S.

Warren

XSB

as

an

efficient

deductive

database

engine

In

R.

T.

Snodgrass

and

M.

Winslett

editors

Proc

of

the

ACM

SIGMOD

Int

Conf

on

Management

of

Data

-LRB-

SIG

MOD

-RRB-

pages

-LCB-

-LSB-

-RSB-

S.

Sudarshan

Optimizing

Bottom-Up

Query

Evaluation

for

Deductive

Databases

PhD

thesis

University

of

Wisconsin

Madison

-LSB-

-RSB-

J.

D.

Ullman

Bottom-up

beats

top-down

for

Datalog

In

Proc

of

the

Eighth

ACM

SIGACT-SIGMOD

SIGART

Symposium

on

Principles

of

Database

Systems

-LRB-

PODS

-RRB-

pages

-LCB-

-LSB-

-RSB-

J.

D.

Ullman

Principles

of

Database

and

KnowledgeBase

Systems

Vol

Computer

Science

Press

deductive databases
magic sets
query evaluation