Transaction Scheduling in Dynamic Composite Multidatabase Systems ? Dexter P. Bradshaw Per - ? Ake Larson Jacob Slonim y Department of Computer Science University of Waterloo , Waterloo , Ontario N2L 3G1 Technical Report : CS-94-11 Abstract Multidatabase systems based on a single monolithic multidatabase server are not realistic . Their performance and administration do not scale with increases in the radius of service or the number of component databases under their control . We propose that a composite multidatabase architecture that consists of multiple , possibly heterogeneous , peer multidatabase servers distributed on a communications network is inevitable . Global transactions should be able to span multiple multidatabase servers , sometimes forcing multidatabase servers to act as component database systems . Particular focus is given to the problem of guaranteeing the correct execution of interleaving global transactions across multiple multidatabase systems . Correctness is based on global serializability . Three algorithms for maintaining global serializability through transaction ordering during dynamic multidatabase composition are proposed . We examine the restrictions of these algorithms and the scalability of their performance . 1 Introduction A multidatabase system -LRB- MDBS -RRB- is a service that provides access to data stored in multiple autonomous and possibly heterogeneous database systems distributed on a communications network . ? This research is supported by IBM Canada Ltd. and by the Natural Sciences and Engineering Research Council -LRB- NSERC -RRB- of Canada . yJacob Slonim is an adjunct Professor at the University of Waterloo . His mailing address is at the Centre for Advanced Studies , IBM Toronto Labs , 844 Don Mills , North York , Ontario , M3C 1V7 . To client applications multidatabase systems provide an integrated view of the data stored at component database systems under their control . The abstraction gives applications the illusion that they are accessing a single database system . This removes the complexity of distribution , heterogeneity , integration , transaction management , and administration from the application to the multidatabase system . A centralized view of a single multidatabase server defeats the transparency aspect of the multidatabase abstraction . With a single centralized server , applications are exposed to the nuances of accessing data sources that cross multidatabase domains . We propose to maintain transparency by configuring a multidatabase system as dynamic sets of cooperating heterogeneous peer servers distributed on a communications network . We refer to this configuration as a composite multidatabase system . Also , we call each multidatabase system participating in a composition a multidatabase cell , or simply a cell . During composition , global data requests can span multiple cells composing multidatabase servers , dynamically and arbitrarily , as they are delegated from one multidatabase cell to the next . Multidatabase composition provides a flexible framework for configuring and planning the evolution of component database systems in a network domain . Issues such as administration and control , security , reliability through redundancy , performance and load balancing , and system scalability also motivate concept of multidatabase composition . We claim that decentralized composite multidatabase architectures are inevitable since closed centralized architectures restrict functionality , transparency , reliability , and performance . A Composition Example We further motivate the usefulness and practicality of multidatabase composition through a travel example . Consider a traveler in Trinidad booking a flight to Vancouver with a stop-over in Toronto . In Vancouver , the traveler simply needs to stay at a hotel . However , in Toronto , the traveler needs a hotel room and the convenience of a rented vehicle . Figure 1 illustrates the flow of requests generated by the transaction , T , that books the trip itinerary . The access pattern of T dynamically composes multidatabase cells at the Trinidad branch of Agency A , the Toronto branch of Agency A , and Agency B. T is submitted to the multidatabase server for Agency A in Trinidad . The original request is decomposed into subtransactions T1 , T2 , and T4 that are submitted to component databases to update client data , book an airline seat and bill the customer 's VISA account , respectively . Since the local branch does not have direct access to Canadian databases , all bookings in Canada are forwarded to the multidatabase server at the branch office in Toronto through transaction T3 . The multidatabase server at Agency A , in Toronto , treats the forwarded subtransaction like T T T 35 32 4 1 T T T T T : schedule trip Application T -LRB- Trinidad -RRB- Agency B -LRB- Vancouver -RRB- Agency A -LRB- Toronto -RRB- Agency A AgencyCDB VISA CDB CDB BillingCDB CDB 34 CDB AgencyCDB : Charge VISA account 4TT 3 : Canadian Bookings T 2 : book flight : update client1T T 32 : schedule car rental : book hotel room 31T 33T : customer billing : Vancouver Bookings 34T : book flight 35T T 342 : update customer and billing data T 341 : book hotel room BWIACDB Avis Rental Air Canada Hilton Sheraton CDB 31T T341 2T 342 33 3 Figure 1 : An example of multidatabase composition . any other global transaction from an application using the service . The forwarded transaction T3 is broken down into subtransactions T31 , T32 , T33 , and T35 . These transactions represent a hotel room booking , a car rental reservation , a customer billing and an airline reservation at component databases accessible from the Toronto branch office . The other subtransaction , T34 , is forwarded to Agency B in Vancouver where hotel reservations and customer updates are made directly to component systems through the subtransactions T341 and T342 , respectively . Objective and Overview This work focuses on concurrency control algorithms for guaranteeing the correctness of interleaved global transactions in a composite multidatabase environment . Our correctness criterion is based on serializability . We show that by implicitly or explicitly imposing ordering constraints on global transactions that span more than one multidatabase cell , we can still guarantee global serializability . The concurrency control algorithms need no changes to underlying multidatabase concurrency control algorithms . However , they may impose some conditions on the histories at component database systems within participating cells . These algorithms permit multidatabase cells to act as peers and be composed dynamically at runtime according to the access patterns of global transactions . The performance of the algorithms is compared with those presented in related work . We begin , in Section 2 , with a review of previous work on multidatabase concurrency control and composition , and show how this work differs . In Section 3 , we present a reference architecture for composite multidatabase transaction management , some terminology , and basic assumptions . The problem of scheduling global transactions in a composite multidatabase environment is introduced in Section 4 . Section 5 outlines a set of algorithms for dealing with the scheduling problem . Section 6 highlights the effect of multi-cell transaction ordering on performance in composite systems . We conclude , in Section 7 , with a summary of this work and its significance . 2 Related Work The only known published work on multidatabase composition is on superdatabases -LSB- 18 -RSB- . A superdatabase is analogous to a composite multidatabase system . Superdatabases form a static hierarchy with a master superdatabase at the root composed of other superdatabases and component database systems . Each intermediate superdatabase has its own superdatabase hierarchy with component database systems occupying the leaves . Concurrency control is handled optimistically by certifying the serialization orders of the subtransactions of a global transaction with that of the subtransactions previously committed global transactions at the root . If the orders are consistent , the transaction is committed , otherwise , certification fails and it is aborted . Transactions orders at any intermediate superdatabase is maintained through a data structure called an order vector , or O-vector . Before a transaction can be committed its order vectors must all be shipped from leaf superdatabases to the root through intermediate superdatabases for certification . Because the concurrency control algorithm is optimistic the serialization point of a global transaction is only known at the end of its execution . Hence , detection of a violation in the global serialization order is only known at end of a transaction , and later global aborts can cause an excessive waste of resources . The algorithms we present stress dynamic composition and determine serialization points much earlier in their execution . Usually , the correctness of multidatabase algorithms is based on the classical notion of serializability . Several algorithms have been proposed in the literature to guarantee serializability and usually fall into one of three classes of mechanisms : exclusion , strong recoverability -LRB- or commitment ordering -RRB- -LSB- 6 , 9 , 21 -RSB- and rigorous scheduling -LSB- 9 , 13 -RSB- . Exclusion prevents more than one transaction from executing at two or more sites , inherently preventing indirect conflicts through local transactions -LSB- 1 , 8 , 23 -RSB- . Concurrency control schedulers based on strong recoverability ensure that the serialization and commit orders at component database sites are equivalent . Certification of consistent serialization orders before atomic commitment guarantees serializability -LSB- 19 , 20 , 21 -RSB- . Rigorousness is a subclass of strong recoverability in which component databases guarantee that a transaction can not execute a conflicting operation on data items until the transaction that last operated on them is committed or aborted . Rigorousness combined with an atomic commit protocol guarantees serializability -LSB- 9 , 13 -RSB- . In Bradshaw -LSB- 5 -RSB- , work on concurrency control in open nested transaction systems -LSB- 3 -RSB- was combined with the notion of multidatabase serializability -LSB- 17 -RSB- to derive the concept of open nested multidatabase serializability -LRB- ONMSR -RRB- . ONMSR is not a new serializability class . It generalizes two-level multidatabase serializability to asymmetric execution hierarchies of arbitrary depths . A composite multidatabase is ONMSR if all leaf component databases guarantee serializable histories and all multidatabases , participating in the composition , are multidatabase serializable . ONMSR combined with upward and downward ordering compatibilities in an execution hierarchy guarantees global serializability . 3 Composite Multidatabase Model The model for transaction management in composite multidatabase comprises a transaction processing architecture and assumptions . The former describes abstract modules , their functions and interfaces for transaction processing , while the former describes expected characteristics at module interfaces . Transaction Processing Architecture A multidatabase cell consists of a multidatabase -LRB- MDB -RRB- server and a finite set of component database systems -LRB- CDB -RRB- under its control . Multidatabase servers and component database systems run as separate processes , or as a separate group of processes , on nodes -LRB- or sites -RRB- distributed on a communications network . Multidatabase cells can communicate directly , or indirectly through other intermediate multidatabase servers . Component database systems can only communicate with their multidatabase server through a multidatabase processing agent -LRB- MPA -RRB- . A multidatabase transaction processing system consists of a global transaction manager -LRB- GTM -RRB- , a set of local transaction managers -LRB- LTM -RRB- belonging to the component systems , and multidatabase transaction processing agents -LRB- MTA -RRB- running as part of the MPA . The GTM coordinates the execution of global transactions at component sites through the MTA . The LTM schedules all LTM MPA LTM ... . T T T T L CDBCDB MTA T MDB ServerGTM T MDB ServerGTM Tij Global Transactions GTM i k i k l i k MDB Server MPAMTA Figure 2 : Composite multidatabase transaction processing model . local transactions and global subtransactions local to a component database system . The MTA is responsible for mapping global subtransactions into local subtransactions that are scheduled for execution by an underlying LTM . Interleaving concurrent global transactions are scheduled by a component of the GTM called the global concurrency controller -LRB- GCC -RRB- . The GCC cooperates with an global currency control agent -LRB- GCCA -RRB- in the MTA to guarantee correct global execution schedules at component database systems . This transaction processing architecture is illustrated in Figure 2 . During composition , global subtransactions are forwarded from one multidatabase cell to another to complete the work of a global transaction . At a multidatabase cell , subtransactions from other cells are treated like any other transaction from an application attempting to access the multidatabase . Therefore , composition is transparent to a multidatabase server that may act in isolation , or participate in a composite environment . In general , we distinguish four types of transactions : simple global transactions , multi-cellular global transactions , delegated subtransactions , and local transactions . A simple global transaction executes is a global transaction that executes within one database cell . A multi-cellular global transaction begins in a single cell but may span multiple multidatabase cells through delegated subtransactions . Simple and multi-cellular global transactions are collectively called global transactions . Delegated subtransactions are global subtransactions forwarded from a global transaction at one cell to execute at another . Local transactions execute solely at a single component database system . From Figure 2 , Ti and Tk are multi-cellular and simple global transactions , respectively . Tij is a delegated subtransaction , while Ll is a local transaction . Assumptions We assume that multidatabase cell servers communicate with each other and can participate in an atomic commit protocol . The composite multidatabase system architecture is peer-to-peer in that the client/server relationship between multidatabase cell servers can reverse depending on the multidatabase cell forwarding the transactional work . Hence , multidatabase composition is not static , but depends on the run-time structure of active multidatabase transactions . No two multidatabase cells share the same component database system . We restrict multidatabase access to a component database system through a single multidatabase server . Applications from other multidatabase cells can only access a component database in a another multidatabase cell through delegated subtransactions to its multidatabase cell server . We further restrict the environment by assuming no shared or replicated data and data structures exist among multidatabase cells . We refer to this assumption as the shared-nothing assumption . It implies that distributed or replicated component databases are treated as single component database systems . The catalogue in a multidatabase server may import mappings to views and base tables managed by another multidatabase server . All multidatabase cells guarantee multidatabase serializable histories -LSB- 17 -RSB- . Since multidatabase serializability is a more restrictive form of two-level serializability , this subsumes locally serializable schedules at all component database systems . In particular , we assume that local schedules are strict and global atomicity between a multidatabase server and its component sites is guaranteed with the 2PC protocol -LSB- 4 , 14 , 15 -RSB- . We also assume that local deadlock is resolved by deadlock resolution mechanisms at component database sites while global deadlock is resolved through timeouts . Other assumptions about the relationship between multidatabase servers and their component database systems depends on global concurrency control algorithm in use . The algorithms presented in this work guarantee that none of these assumptions would be violated unless otherwise stated . Finally , no two subtransactions at any multidatabase server or component database system share a common ancestor transaction -LRB- or subtransaction -RRB- . This assumption guarantees that the runtime composition lattice for any global transaction always takes the form of a directed acyclic graph -LRB- DAG -RRB- . 4 Composite Transaction Scheduling Problem In a composite multidatabase environment , guaranteeing serializability in each multidatabase cell is not enough for ensuring global serializability . This condition generates globally serializable schedules only if there are no global transactions that span more than one multidatabase cell . However , this case represents a degenerate composition . In this work , we focus only on nondegenerate multidatabase composition and show in our algorithms that little overhead is incurred in the degenerate case . Let us consider the non-degenerate composition CM = fM1 ; M2 ; M3g illustrated in Figure 3 . There are three multidatabase cells under the control of servers M1 , M2 , and M3 . Each server Mi having a set of component databases , Cij -LRB- j = 1 ; : : : ; ni -RRB- , under its control . T1 and T2 are two global transactions that begin execution at M1 and M3 , respectively . These transactions are decomposed into subtransactions , Tkl -LRB- k = 1 ; 2 -RRB- , that run at a component database or get delegated to run in another multidatabase cell . Delegated subtransactions are further decomposed into subtransactions Tklm that run at component databases in the foreign cell . Although M1 , M2 , and M3 are all multidatabase serializable , it is possible to generate non-serializable global schedules as follows . Let a , b , c , and d be data items stored at C12 . Also , let data items e and f , and data items g and h be stored at C13 and C21 , respectively . Finally , let L1 be a local transaction running at C12 . The following histories are possible at C12 , C13 , and C21 : HC12 : rT12 -LRB- a -RRB- wT12 -LRB- b -RRB- rL1 -LRB- c -RRB- wL1 -LRB- a -RRB- cT1 cL1 rT261 -LRB- d -RRB- wT261 -LRB- c -RRB- cT2 HC13 : rT14 -LRB- e -RRB- cT1 rT262 -LRB- f -RRB- wT262 -LRB- e -RRB- cT2 HC21 : rT251 -LRB- g -RRB- wT251 -LRB- h -RRB- cT2 wT131 -LRB- g -RRB- cT1 We ignore the histories at other component databases since single transactions run at these sites and there are no conflicts . The resulting serialization orders at the component databases follow . C11 : T11 C12 : T12 ; T261 C13 : T14 ; T262 C21 : T251 ; T131 C22 : T252 C31 : T21 C32 : T22 C33 : T23 C34 : T24 At component databases the Ti ; Tj means that Ti is serialized before Tj , and there may be other local transactions , Li -LRB- i = 1 ; 2 ; : : : ; n -RRB- , serialized between Ti and Tj such that , Ti ?! L1 ?! L2 ?! ? ? ? ?! Ln ?! Tj : Since each multidatabase cell is multidatabase serializable , the following serialization orders are induced on multidatabase transactions : M1 : T1 ?! T26 M2 : T25 ?! T13 M3 : T2 T261 262T 14 T T13 T T T 131 T T C 2 11 C12 C13 C21 C22 C31 C32 C33 C34 3MM 2 M 1 25 251 252 26 T 21 T22 T23 T24T 11 T 12 T1 T Figure 3 : An illustration of the scheduling problem in a composite MDBS . Therefore , T1 ! T2 ! T1 is a possible global serialization order and the global committed history is not serializable . 5 Scheduling Transactions in Composite Multidatabase Systems Ordering constraints need to be imposed on the multi-cellular transactions to guarantee globally serializable histories . In the previous section , no restrictions were imposed on the multi-cellular global transactions T1 and T2 , in M1 and M2 , resulting in inconsistent serialization orders at the global level . We describe three mechanisms for guaranteeing global consistent orders and prove their correctness . These mechanisms impose consistent global ordering constraints on underlying multidatabase cells , leaving their concurrency control schedulers unaffected . 5.1 Composite Rigorous Scheduling Algorithm The algorithm described in this section extends the rigorous multidatabase scheduling algorithm -LSB- 7 , 13 , 22 -RSB- to composite multidatabase environments . Recall a component database system , C , is said to have a rigorous history if when two transactions Ti and Tj have operations such that oj -LRB- x -RRB- conflicts with oi -LRB- x -RRB- on a data item x in C , Ti must commit or abort before oj -LRB- x -RRB- executes . Rigorous histories are a subclass of strongly recoverable histories . Therefore , they inherit the property that the serialization order of any two transactions is equivalent to their commitment order -LSB- 9 -RSB- . We use this property to guarantee consistent global transaction orders across multidatabase cells . The following theorem is an application of the rigorous scheduling algorithm for simple multi - database systems to composite multidatabase environments . Theorem 1 -LRB- Composite Rigorous Scheduling Algorithm -LRB- CRS -RRB- -RRB- Let CM be a composite multidatabase system consisting of n multidatabase cells M1 ; M2 ; : : : ; Mn . If -LRB- i -RRB- all component databases of each Mi -LRB- i = 1 ; : : : ; n -RRB- guarantee rigorous serializable histories , and -LRB- ii -RRB- global atomicity at and among the Mi is guaranteed through the two-phase commit -LRB- 2PC -RRB- protocol then the global committed history of CM is serializable . Proof Let HCM be a global committed history of CM = fM1 ; M2 ; : : : ; Mng . Assume that HCM is not serializable . Then , the serialization graph SGCM has a cycle , T1 Moe1 ?! T2 Moe2 ?! ? ? ? Moek ? 1 ?! Tk Moek ?! T1 where Ti Moel ?! Tj refers to the serialization order of Ti and Tj imposed by cell Moel . It is possible that Moel ? Moem . If Moe1 ? Moe1 ? ? ? ? ? Moek then we have a contradiction because we assume in Section 3 that Mi -LRB- i = 1 ; 2 ; : : : ; n -RRB- are all multidatabase serializable and therefore can not contain cyclic serialization graphs . So , there must be Moel and Moem such that Moel 6 ? Moem . Since Moel -LRB- l = 1 ; 2 ; : : : ; k -RRB- are all multidatabase serializable , Ti Moek ?! Tj implies that Ti Coelm ?! Tj at every component database of Coelm of Moel where Ti and Tj conflict . But all local histories at the Coelm are rigorous , so the commit order of Ti and Tj is equivalent to their serialization order . This means that every transaction in any cycle of SGCM has committed before itself . This contradicts the atomicity property of the 2PC protocol . Therefore our assumption is false and HCM is serializable . 2 Let us apply the algorithm to the composition in Figure 3 . All the component databases Cij guarantee rigorous serializable histories and the atomicities of the global transactions T1 and T2 are guaranteed by the repeated application of the 2PC protocol . The proof shows that under these conditions either C12 : T12 ; T261 C13 : T14 ; T262 C21 : T131 ; T251 or , C12 : T261 ; T12 C13 : T262 ; T14 C21 : T251 ; T131 : Both cases lead to serializable global histories . 5.2 Composite Forced Conflicts Algorithm The rigorous history condition , -LRB- i -RRB- in Theorem 1 , has restricted applicability because some component database systems may not support rigorous serializable histories . Also , some multidatabase cells may use mechanisms other than rigorous scheduling to guarantee serializable cell histories . We relax the rigorousness condition by requiring that all multi-cellular global transactions conflict at rigorous component database in cells where they execute . This requires that all multidatabase cells have at least one component database , R , that supports rigorous serializable histories and each multi-cellular global transaction reads and updates a designated data item , or ticket at R . We outline the composite forced conflicts algorithm and prove its correctness below . Algorithm : Composite Forced Conflict -LRB- CFC -RRB- Let CM = fM1 ; M2 ; : : : ; Mng be a composite multidatabase environment where each Mi -LRB- i = 1 ; 2 ; : : : ; n -RRB- has a rigorous component database Ri . 8 global transactions T executing at Mi 2 CM SubT rasactionSet = decompose -LRB- T -RRB- if globalT ransaction -LRB- T -RRB- is a multi-cell transaction or 9 Tj 2 f Tk j Tk 2 SubT ransactionSet and Tk executes at Mj < > Mig create subtransaction , Tticket , to update the ticket at Ri SubT ransactionSet = SubT ransactionSet S fTticketg endif 8 Tj 2 SubT ransactionSet submit -LRB- Tj ; Site -LRB- Tj -RRB- -RRB- end8 end8 3 The Composite Forced Conflict -LRB- CFC -RRB- algorithm assumes multidatabase serializability is guaranteed in each cell and at least one component database is rigorous . All multi-cellular global transactions are forced to read and update a ticket data item at a designated rigorous component database system by creating a dummy subtransaction , Tticket , to perform the update and adding it to the decomposition set of the multi-cellular global transaction . The ticket is read and accessed by multi-cellular global transactions only 1 . Simple global transactions do not read the ticket . Notice 1Permitting other global transactions and local transactions to access the ticket does not affect the correctness of the algorithm , but does affect performance . that the algorithm assumes that a multi-cellular global transaction can be distinguished from a simple global transaction . This assumption is reasonable since it is possible to include the address of a parent multidatabase cell server as part of the delegation protocol between two multidatabase cell servers . The following theorem guarantees the correctness of the CFC algorithm . Corollary 1 -LRB- CFC Correctness -RRB- Let CM be a composite multidatabase system consisting of n multidatabase cells M1 ; M2 ; : : : ; Mn . If -LRB- i -RRB- every multi-cellular transaction T that executes at Mi -LRB- i = 1 ; 2 ; : : : ; n -RRB- is forced to conflict with all other concurrent multi-cellular transactions at a rigorous component database system , Ri , and -LRB- ii -RRB- the atomicity of global transactions are guaranteed through the 2PC protocol then the global committed history of CM is serializable . Proof The proof of the corollary follows from the proof of Theorem 1 . Assume that the global committed history HCM is not serializable . Then , the serialization graph SGCM contains a cycle , T1 Moe1 ?! T2 Moe2 ?! ? ? ? Moek ? 1 ?! Tk Moek ?! T1 : Since Moel -LRB- l = 1 ; 2 ; : : : ; k -RRB- are all multidatabase serializable , Ti Moek ?! Tj implies that Ti Coelm ?! Tj at every component database of Coelm of Moek where Ti and Tj conflict . In particular , Ti Roek ?! Tj . But all local histories at Roek are rigorous , so the commit order of Ti and Tj is equivalent to their serialization order . This means that every transaction in any cycle of SGCM has committed before itself . This contradicts the atomicity property of the 2PC protocol . Therefore , our assumption is false and HCM is serializable . 2 If we apply CFC to the composition CM = f M1 ; M2 ; M3 g in Figure 3 , we need to augment M1 , M2 , and M3 with the rigorous component database systems R1 , R2 , and R3 respectively . The dummy transactions TticketT1 and TticketT26 are forced to conflict at R1 , while the ticket transactions TticketT13 and TticketT25 conflict at R2 . The proof shows that either R1 : TticketT1 ?! TticketT26 C12 : T12 ; T261 C13 : T14 ; T262 R2 : TticketT13 ?! TticketT25 C21 : T131 ; T251 or , R1 : TticketT26 ?! TticketT1 C12 : T261 ; T12 C13 : T262 ; T14 R2 : TticketT25 ?! TticketT13 C21 : T251 ; T131 : Both sets of serialization orders lead to serializable schedules in the global committed history . 5.3 Composite Timestamp Ordering Algorithm The Composite Timestamp Ordering -LRB- CTO -RRB- imposes an a priori total order on all multi-cellular transactions . This algorithm does not have any special requirements of component databases as in the CRS and CFC algorithms , and may be used as an alternative in compositions where some multidatabase cells have no component databases that enforce rigorous histories . The CTO algorithm assigns and validates timestamp orders as follows . All delegated subtransactions are tagged as multi-cellular and inherit the global timestamp from their parent transaction . Untagged transactions that enter a multidatabase cell are new transactions submitted directly from an application . New transactions are first decomposed . If any of the resulting subtransactions are delegated to other multidatabase cells , then both they and their parent are tagged as multi-cellular and assigned the same unique timestamp . Tagged transactions are delegated from other multidatabase cells . These transactions are first validated by checking that their timestamp is not older than that of the last committed transaction , Tlast . If it is older , it is aborted . Otherwise , it is decomposed . All resulting subtransactions inherit the tags and the unique timestamp from the parent and are submitted to component databases for execution . The CTO algorithm ensures that new and preparing delegated subtransactions execute and commit in timestamp order . The the global cocurrency control scheduler of each cell maintains a multi-cellular commit order list -LRB- MCOL -RRB- . The MCOL is never empty because it always maintains Tlast as a node . Every multi-cellular transaction is added to the MCOL sorted on their timestamp value and placed in a running state . When any subtransaction on the list is ready to prepare , it first checks if it is older than Tlast . If it is older it is aborted . Otherwise , it checks if any younger transactions are prepared . If so , it is placed in a waiting state and rechecks after a set interval , or after a prepared transaction is forced to abort through the 2PC protocol . Once a transaction passes the validation step , it prepares , enters a prepare state , and waits for a commit signal from its 2PC coordinator . On receiving a commit , a prepared transaction aborts all older transactions in the running and waiting states then commits . If it receives an abort signal it is removed from MCOL and signals all older transactions in the waiting state . The algorithm is outlined below . Algorithm : Composite Timestamp Ordering -LRB- CTO -RRB- START : Let Mi be a multidatabase cell in a composition , CM = f M1 ; M2 ; : : : ; Mn g. NEW : 8 transactions , Ti , starting at Mi : SubT ransactionSetTi = Decompose -LRB- Ti -RRB- if Untagged -LRB- Ti -RRB- and 9 Tik 2 f Tij j Tij 2 SubT ransactionSetTi and Tk executes at Mj < > Mi g SetTag -LRB- Ti -RRB- SetTimeStamp -LRB- Ti ; NewTimeStamp -LRB- -RRB- -RRB- endif if Tagged -LRB- Ti -RRB- if TimeStamp -LRB- Ti -RRB- < TimeStamp -LRB- Tlast -RRB- abort Ti Exit -LRB- -RRB- endif Insert -LRB- Ti ; MCOL -RRB- SetState -LRB- Ti ; RUNNING -RRB- 8 Tik 2 SubT ransactionSetTi SetTag -LRB- Tik -RRB- SetTimeStamp -LRB- Tik ; TimeStamp -LRB- Ti -RRB- -RRB- Submit -LRB- Tik ; Site -LRB- Tik -RRB- -RRB- end8 endif end8 PREPARE : if TimeStamp -LRB- Ti -RRB- < TimeStamp -LRB- Tlast -RRB- abort Ti Exit -LRB- -RRB- endif if 9 Tk 2 f Tj j Tj 2 MCOL and TimeStamp -LRB- Tj -RRB- > TimeStamp -LRB- Ti -RRB- and P repared -LRB- Tj -RRB- g SetState -LRB- Ti ; WAIT ING -RRB- else prepare Ti with 2PC coordinator SetState -LRB- Ti ; PREPARED -RRB- endif COMMIT : 8 Tk 2 f Tj j Tj 2 MCOL and TimeStamp -LRB- Tj -RRB- < TimeStamp -LRB- Ti -RRB- and Running -LRB- Tj -RRB- or W aiting -LRB- Tj -RRB- g abort Tk end8 commit Ti and all subtransactions ABORT : Delete -LRB- Ti ; MCOL -RRB- abort Ti and all subtransactions 3 In CTO , we assume that all timestamps within a composition are globally unique and are related in a total order . Timestamps with these properties can be generated by concatenating the global time -LRB- from some global distributed time service -RRB- with a unique site-dependent identifier at each site . Given that the timestamps are a total order the following theorem guarantees the correctness of CTO . Theorem 2 -LRB- CTO Correctness -RRB- For a composition CM = fM1 ; M2 ; : : : ; Mng if -LRB- i -RRB- each Mi -LRB- i = 1 ; 2 ; : : : ; n -RRB- executes the CTO algorithm , and -LRB- ii -RRB- global atomicity at and among the Mi is guaranteed through the two-phase commit -LRB- 2PC -RRB- protocol then the global committed history of CM is serializable . Proof The inheritance of timestamps by subtransactions from parent transactions coupled with the validations in the NEW , PREPARE and COMMIT steps of CTO guarantee that multi-cellular transactions are serialized and committed in timestamp order . The rest of this proof follows the proof by contradiction of TO in -LSB- 4 -RSB- . If we assume the global committed history HCM is not serializable , then its serialization graph SGCM has a cycle . Therefore every transaction Ti in the cycle has TimeStamp -LRB- Ti -RRB- > TimeStamp -LRB- Ti -RRB- , since the serialization order corresponds to the timestamp order . But this contradicts the total ordering of timestamps . Thus our assumption is false and the global committed history HCM is serializable . 2 In applying CTO to the example in Figure 3 , if TimeStamp -LRB- T1 -RRB- > TimeStamp -LRB- T2 -RRB- then C12 : T12 ; T261 C13 : T14 ; T262 C21 : T131 ; T251 otherwise , if TimeStamp -LRB- T2 -RRB- < TimeStamp -LRB- T1 -RRB- then C12 : T261 ; T12 C13 : T262 ; T14 C21 : T251 ; T131 : Note TimeStamp -LRB- Ti -RRB- represents only the timestamp of the commited transaction . Both cases lead to serializable global histories . 6 Implementation Issues Our investigation in transaction management for multidatabase composition falls under the CORDS Multidatabase project . We have built a multidatabase prototype that provides transparent transactional access to DB2/60002 and Oracle3 V7 component database systems running under AIX4 3.2.x a set of RS/6000s . Although we use DB2/6000 and Oracle V7 in particular , nothing restricts us from using other X/Open5 XA compliant databases . For details on the CORDS Multidatabase architecture and prototypes , see -LSB- 10 , 12 -RSB- . Multidatabase cells are accessed through multidatabase servers that export an ODBC6 interface to applications and other multidatabase servers . Global distributed transaction processing functionality is provided by the Encina7 transaction toolkit facility . The transaction and resource access interfaces at component databases are exactly the same as those exported by multidatabase servers . Each component database system has set of agents that run as applications to provide the necessary interface functionality . Both the agents and the multidatabase cell servers run as Encina TP monitor applications . The monitor environment tightly integrates Encina distributed transaction management , 2PC through the X/Open XA standard -LSB- 11 -RSB- , distributed deployment of the system , and fault tolerance . The agents and cell servers are all multi-threaded and communicate via OSF8 DCE remote procedure calls -LRB- RPC -RRB- . Details on the transaction management architecture are described in -LSB- 2 -RSB- . Currently , we support only the composite rigorous scheduling algorithm . Work on adding the CFC and CTO algorithms to the the GCC schedulers of our multidatabase cell servers will soon be completed . Initial tests validate both the correctness and viability of the algorithms . These tests also validate the following performance issues . 2DB2/6000 is a trademark of the IBM Corporation . 3Oracle is a trademark of the Oracle Corporation . 4AIX is a trademark of the IBM Corporation . 5X/Open is a trademark of the X/OPEN company . 6Open Database Connect from the Microsoft Corporation . 7Encina is a trademark of Transarc Corporation . 8OSF is a trademark of the Open Software Foundation . Dynamic Composition and Scalability In our prototype , multidatabase composition is dynamic and depends on collective runtime access patterns of all global transactions . This is unlike the static hierarchies required by Schek and Weikum in -LSB- 24 -RSB- , and by Pu in -LSB- 18 -RSB- . This feature allows the dynamic removal and addition of multidatabase cell servers from a composite environment . Transaction access patterns are also dynamically controlled at runtime by manipulating catalogue data at active multidatabase cell servers . Dynamic composition facilitates scalability . Our concurrency control algorithms do not limit the number of multidatabase cells that participate in a composition nor the number of active transactions . However , the scale of a composite multidatabase environment may be limited by the latency 2PC protocol and the timeout interval at component database systems . Message Complexity The message complexity of our algorithms is equivalent to that of the 2PC protocol . In each of our algorithms , control information is piggy-backed as transaction properties on 2PC protocol messages . Our algorithms do not suffer from the communication limitations of explicitly communicating global ordering data structures like O-vectors in -LSB- 16 , 18 -RSB- 9 . Global Aborts Each of the algorithms guarantees serializable global transaction orders through the preemption of non-serializable transactions by global aborts . The CRS and CFC algorithms work by creating deadlock among non-serializable multi-cellular transactions . The deadlock is resolved by forcing a transaction in the deadlock cycle to rollback through timeouts at component databases , timeouts in the 2PC coordinator , or deadlock resolution algorithms at component database systems . The CTO algorithm maintains the total ordering of the timestamps of committed transactions by explicitly preempting any multi-cellular transaction whose commitment will violate the total order . Global aborts degrade system performance by reducing both global and local transaction throughput and wasting valuable system resources . We expect abort frequencies to increase significantly as the locality of multi-cellular transactions increases and their read/write ratios decrease . We believe that a smart recovery/resubmission algorithm can avert these effects . However , the design of the recovery algorithm may be limited by the underlying properties of the underlying multidatabase concurrency control mechanism . 9Pu cited this as a major reason maintaining a static hierarchy of databases . 7 Conclusion The significance of issues such as location transparency , scalability , performance , and administration make multidatabase composition inevitable . We have shown that scheduling transactions that span multiple multidatabase cells is not simply a matter of guaranteeing transaction atomicity and serializability at each multidatabase cell . To guarantee global serializability in a composite multidatabase environment , ordering constraints must be imposed on multi-cellular transactions . We presented and proved the correctness three multi-cellular transaction scheduling algorithms : Composite Rigorous Scheduling -LRB- CRS -RRB- , Composite Forced Conflicts -LRB- CFC -RRB- , and Composite Timestamp Ordering -LRB- CTO -RRB- . CRS and CFC enforce ordering constraints implicitly through rigorous histories at component database . While CTO does so explicitly through a global total order on timestamps . Our algorithms scalable and support dynamic multidatabase composition . They incur no communications overhead since all control information is superimposed on 2PC messages . However , we expect abort rates to increase as multi-cellular transaction locality increases and read/write ratios decrease . This implies that composite multidatabase recovery must be smart and efficient . References -LSB- 1 -RSB- R. Alonso , H. Garcia-Molina , and K. Salem . Concurrency Control and Recovery for Global Procedures in Federated Database Systems . In Proceedings of the IEEE Conference on Data Engineering , pages 5 -LCB- 11 . IEEE Computer Society Press , September 1987 . -LSB- 2 -RSB- G. Attaluri and D. P. Bradshaw . Architecture for Transaction Management in the CORDS Multidatabase System . In A. Gawman , W. Morven Gentleman , E. Kidd , P. - ? A. Larson , and J. Slonim , editors , `` Proceedings of CASCON '93 Volume II : Distributed Computing '' , pages 873 -LCB- 887 , Toronto , Canada , October 1993 . IBM Toronto Labs , IBM Centre for Advanced Studies . To appear in Proceedings of the 1993 CAS Conference . -LSB- 3 -RSB- C. Beeri , P. A. Bernstein , and N. Goodman . A Model for Concurrency Control in Nested Transaction Systems . JACM , 36:230 -LCB- 269 , 1989 . -LSB- 4 -RSB- P. A. Bernstein , V. Hadzilacos , and N. Goodman . Concurrency Control and Recovery in Database Systems . Addison-Wesley Series in Computer Science . Addison-Wesley , United States of America , 1987 . -LSB- 5 -RSB- D. P. Bradshaw . Open Nested Serializability in Multidatabase Systems . In M. Bauer , J. Botsford , P. - ? A. Larson , and J. Slonim , editors , Proceedings of the 1992 CAS Conference , pages 93 -LCB- 109 , Toronto , Canada , November 1992 . IBM Toronto Labs , IBM Centre for Advanced Studies . -LSB- 6 -RSB- Y. Breitbart and H. Garcia-Molina . Overview of Multidatabase Transaction Management . Technical Report TR-92-21 , Department of Computer Services , University of Texas , Austin , Texas 78712 , May 1992 . -LSB- 7 -RSB- Y. Breitbart , D. Georgakopoulos , M. Rusinkiewicz , and A. Silberschatz . On Rigorous Transaction Scheduling . IEEE Transactions on Software Engineering , 17 -LRB- 4 -RRB- :954 -LCB- 960 , September 1991 . -LSB- 8 -RSB- Y. Breitbart and A. Silberschatz . Multidatabase Update Issues . In Proceedings of the ACM SIGMOD Conference on the Management of Data , pages 135 -LCB- 142 . ACM , ACM Press , June 1988 . -LSB- 9 -RSB- Y. Breitbart and A. Silberschatz . Strong Recoverability in Multidatabase Systems . In P. S. Yu , editor , Second International Workshop on Research Issues on Data Engineering : Transaction and Query Processing , pages 170 -LCB- 175 , Los Alamitos , Ca , February 1992 . IEEE Computer Society Technical Committee on Data Engineering , IEEE Computer Society Press . -LSB- 10 -RSB- N. Coburn and P. - ? A. Larson . Multidatabase Services : Issues and Architectural Design . In M. Bauer , J. Botsford , P. - ? A. Larson , and J. Slonim , editors , Proceedings of the 1992 CAS Conference , pages 57 -LCB- 66 , Toronto , Canada , November 1992 . IBM Toronto Labs , IBM Centre for Advanced Studies . -LSB- 11 -RSB- X/Open Company . CAE Specification . Distributed Transaction Processing : The XA Specification . X/Open Company Limited , United Kingdom , 1991 . -LSB- 12 -RSB- D. L. Erickson , P. J. Finnigan , G. K. Attaluri , M. A. Bauer , D. P. Bradshaw , N. Coburn , M. P. Consens , M. Z. Hasan , J. W. Hong , K. A. Lyons , T. P. Martin , G. W. Neufeld , W. Powley , D. Rappaport , D. J. Taylor , T. J. Teorey , and Y. Yemini . CORDS : An Update to Prototypes . Technical Report TR-74 .120 , Centre for Advanced Studies , IBM Toronto Labs , August 1993 . -LSB- 13 -RSB- D. Georgakopoulos , M. Rusinkiewicz , and A. Sheth . On Serializability of Multidatabase Transactions Through Forced Local Conflicts . In Proceedings of the Seventh International Conference on Data Engineering , pages 314 -LCB- 323 , Los Alamitos , Ca , April 1991 . IEEE , IEEE Computer Society Press . -LSB- 14 -RSB- J. N. Gray . Notes on Database Operating Systems . In R. Bayer , R. M. Graham , and G. Seegmuller , editors , Operating Systems : An Advanced Course , pages 393 -LCB- 481 . Springer-Verlag , Berlin , Germany , 1978 . -LSB- 15 -RSB- J. N. Gray and A. Reuter . Transaction Processing : Concepts and Techniques . Morgan Kaufmann Series in Data Management Systems . Morgan Kaufmann Publishers , San Mateo , Ca , 1993 . -LSB- 16 -RSB- A. Leff and C. Pu . Classification of Transaction Processing Systems . IEEE Computer , 24 -LRB- 6 -RRB- :63 -LCB- 76 , June 1991 . -LSB- 17 -RSB- S. Mehrotra , R. Rastogi , Y. Breitbart , H. F. Korth , and A. Silberschatz . Ensuring Transaction Atomicity in Multidatabase Systems . Technical Report TR-92-12 , Department of Computer Sciences , University of Texas at Austin , Austin , Texas , 78712 , June 1992 . -LSB- 18 -RSB- C. Pu . Superdatabases for Composition of Heterogeneous Databases . In Proceedings of the Fourth International Conference on Data Engineering , pages 548 -LCB- 555 , Los Alamitos , Ca , May 1988 . IEEE , IEEE Computer Society Press . -LSB- 19 -RSB- Y. Raz . Extended Commitment Ordering . Technical Report DEC-TR 842 , Digital Equipment Corporation , 151 Taylor St , Littleton , Ma 01460 , December 1991 . -LSB- 20 -RSB- Y. Raz . Guaranteeing global serializability via Commitment Ordering . Technical Report DEC-TR 843 , Digital Equipment Corporation , 151 Taylor St , Littleton , Ma 01460 , December 1991 . -LSB- 21 -RSB- Y. Raz . Principle of Commitment Ordering . Technical Report DEC-TR 841 , Digital Equipment Corporation , 151 Taylor St , Littleton , Ma 01460 , November 1991 . -LSB- 22 -RSB- Y. Raz . Locking Based Strict Commitment Ordering . Technical Report DEC-TR 844 , Digital Equipment Corporation , 151 Taylor St , Littleton , Ma 01460 , February 1992 . -LSB- 23 -RSB- K. Salem , H. Garcia-Molina , and R. Alonso . Altruistic Locking : A Startegy for Coping with Long-Lived Transactions . In D. Gawlick , M. Haynie , and A. Reuter , editors , Lecture Notes in Computer Sciences , High Performance Transaction Processing Systems , volume 359 , pages 175 -LCB- 199 . Springer-Verlag , 1989 . -LSB- 24 -RSB- H-J . Schek , G. Weikum , and W. Schaad . A Multi-Level Transaction Approach to Federated DBMS Transaction Management . In Proceedings of the First International Workshop on Interoperability in Multidatabase Systems , pages 280 -LCB- 287 . IEEE Computer Society Press , April 1991 .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

concurrency control
multidatabase composition
rigorous scheduling
serializability
timestamp ordering