Abstract:
A method and system improves efficiency of highly concurrent aggregate summaries updates by delaying the updates to as late as possible in the transaction, while maintaining an accurate in-progress aggregate summary for use by transaction in progress. The system uses a temporary table to store updates to aggregate summaries and consolidates the temporary table with the aggregate summary to create a view of the accurate in-progress data for use by the transaction. Prior to the transaction commit, the system converts the contents of the temporary delta table into a single-statement consolidated update of the inventory summary table, reducing throughput delays caused by write locks early in the transaction.

Description:
RELATED APPLICATIONS  
       [0001]     This application claims priority to Provisional U.S. Patent Application Ser. No. 60/528,971, filed on Dec. 8, 2003, entitled “Efficient Aggregate Summary Views of Massive Numbers of Items in Highly Concurrent Update Environments,” by Z. Chai and A. Alcock. 
     
    
     FIELD OF THE INVENTION  
       [0002]     The present invention relates to updates of information stored in a database, and more particularly, to highly concurrent updates of aggregate summary data.  
       BACKGROUND  
       [0003]     A standard database table maintains information about of a large set of uniquely identified items, one row corresponding to each item, with columns corresponding to various attributes of the items, and fields containing values of the attributes of the items. Attributes could include various information that may change over time.  
         [0004]     When processing large numbers of items with frequent updates, typically both a large database table with one row for each item and a smaller table for aggregate summaries of the item data are used. Dealing with a high volume of transactions with this scheme causes no locking or conflict issues with the large asset table, but can be problematic for the smaller aggregate table.  
         [0005]     One context in which such tables are used is management of asset changes, including asset movement, creation, and destruction. Information about assets in various locations is stored in the tables, which are frequently updated upon movement of the assets from place to place.  
         [0006]     Referring now to  FIG. 1 , there is illustrated an example of assets at different locations in an asset management system  800  as depicted in  FIG. 8 . The example shows Locations A-D  105 - 120  including Assets  1 - 4  ( 125 - 140 ). Referring also now to  FIG. 2 , it shows an initial Asset Table  205 . The Asset Table  205  data corresponds to the Assets  125 - 140  depicted in  FIG. 1 . The rows  210 - 225 , respectively, correspond to the Assets  1 - 4  ( 125 - 140 ), and may include additional rows, e.g.,  230 , for greater numbers of assets. The columns  235 - 250  are for various attributes relating to the Assets  125 - 140 . In this example, the attributes shown are Type  235 , Location (Loc)  240 , and Status  245 , however, various other attributes  250  could be used depending on the business need. The types  235  listed in this example include Locomotive and Carriage. The type  235  generally describes a large container in this example, but could refer to other asset types, such as individual items or cartons, depending on business needs. The locations  240  (A, C, D) correspond to the locations of the respective assets  125 - 140  in  FIG. 1 . The status  245 , shown as available to use (ATU) indicates that the asset is intact and available. All of the assets in this example are in available to use status to simplify the example, however, assets could also have a different status  245 , for example, damaged, for assets damaged in transit.  
         [0007]     In a simple asset management system  100  for updates to such tables, each message or update is about a unique “atom” or object. Thus, the transaction processing a single message needs to lock only the single object that the message refers to. However, for a system managing a large number of assets with various attributes, a message may be about an asset, but the asset may be related (possibly by containment or a physical linkage) to other assets; those assets that are related to one other are known as “an aggregation.” An aggregate summary may contain several transactions and may correspond to one or more threads. For example, Assets  2  ( 130 ) and  3  ( 135 ) are assets that can form an aggregate summary, as the data displayed in  FIG. 2  indicates that these assets are related to each other because they share the same location (A), type (Locomotive), and status (ATU). While it is not necessary that assets share all attributes to be part of an aggregate summary, in this example they must share at least the same location.  
         [0008]     Aggregate summaries are groups of items characterized by common attribute-value combinations. For example, the number of items available-to-promise at each location. In applications in which such aggregate summaries must be queried frequently over large number of items satisfying the aggregate criteria, it is inefficient to re-compute the aggregate summary each time, even with appropriate database indexes on the attributes.  
         [0009]     Referring now to  FIG. 3 , there is shown an inventory summary table  305   a . The term summary table, as used herein, refers to any table that includes aggregate data. An initial Inventory Summary Table  305   a  summarizes the aggregate summary information from  FIG. 2 . For example, in  FIG. 2  Assets  2  and  3  are both of the type locomotive, are at location A, and are available to use. Thus, in the Inventory Summary Table  305   a  these assets  215 ,  220  are grouped (or aggregated) by their common attributes, in this example, into a single row  310 . Note that the columns in the Inventory Summary Table  305   a  are similar to  FIG. 2 , including Type  325  and Loc  330 , however, the third column  335  represents available-to-promise (ATP) assets (e.g., number of assets having any available status). In this example, only a few rows and columns are shown for clarity, however, each such table  305  may have multiple rows and columns as appropriate for the business need and data contained therein.  
         [0010]     Conceptually, therefore, messages about different assets may in fact be messages about the same aggregate summary. Locking and ordering within the system must take these aggregations into account to prevent data integrity failures.  
         [0011]     One prior art solution to this problem is to maintain a separate aggregate summary table, such as the Inventory Summary Table  305   a  of  FIG. 3 , by incremental atomic transaction updates to the attributes of the individual items. For example, if an available-to-promise item, e.g., Asset  2  ( 130 ), at a first location, e.g., Location A, is moved to another location, e.g., Location B, in addition to the update of each individual item&#39;s location, the aggregate available-to-promise summaries corresponding to the old and new locations respectively are decremented and incremented to account for the change. In this example, row  215  of the Asset Table  205  would reflect a change in the Loc column  240  from A to B. In addition, row  310  of the Inventory Summary Table  305   a  would change to reflect  1  asset in the ATP column  335 , and a new row would be added to reflect the new location (B).  
         [0012]     In environments in which the concurrency of item attribute update is low this scheme is effective in supporting highly frequent queries of such aggregate summaries. However, to process a very large number of updates per second, such as with concurrent movement of assets, the updates are processed in many concurrent threads. A thread executes a series of sequential transactions, for example transactions in an aggregation. A transaction corresponds to movement, in this example, of a single asset. When separate threads are processing information concurrently, it becomes critical to ensure that no two threads attempt to update the same data record at the same time; if this condition occurs, the data will almost certainly become corrupt causing the system to behave incorrectly. The general approach to resolving the concurrent update problem is for each transaction to lock the rows it needs to update for the duration of the transaction. Locking may take the form of pessimistic or optimistic locking.  
         [0013]     In pessimistic locking, a transaction locking a row locks out access to that row to all other transactions that want to read, modify, or lock that row. All other transactions wanting to access the row in any way must wait until the transaction that owns the lock releases it. This scheme can lead to a state known as “deadlock.” Consider an example in which a first transaction needs rows 1 &amp; 2, and a second transaction needs rows 1 &amp; 2. If the first transaction gets a lock on row 1 and the second transaction gets a lock on row 2, both transactions will wait indefinitely for the row each transaction is missing. Avoiding this sort of deadlock in a pessimistic locking scheme requires complicated, custom, hard-to-debug code in a system. One known approach is to ensure that threads always take out locks on rows in the same order. However, in a message driven system, doing so can be difficult if not impossible when there exist complex, dynamic relationships between the objects that need to be locked. In addition, pessimistic locking reduces system throughput when many transactions are queued up waiting for a lock on the same row.  
         [0014]     In optimistic locking, many transactions can have a lock on the same row at once, on the theory that potential problems are detected when updates are issued to the database. This process is implemented by logic to detect when a row with a lock on it has changed. At this stage in the transaction, one can back out of the transaction and restart it. In a system in which conflict for rows is low, this process will maximize throughput. However, whenever there is a concurrent update to an optimistic locking system, all but one transaction is guaranteed to fail. Failure can be expensive, especially in terms of CPU and resource cost. Not only does each failed transaction have to roll back all the updates that it made up to backing out, which is typically several times the cost of making the update, but the transaction also has to be performed for a second time.  
         [0015]     Thus, highly concurrent updates of item attributes within extended transactions cannot be supported well by either a pessimistic or optimistic locking scheme. For example, in the available-to-promise at location application, many transactions may be of the form of an available-to-promise sufficiency check query, followed by an update items to promised status, followed by consequential business logic updates based on the available-to-promise end results. Such transactions will take write locks on the aggregate summary table very early leading to poor concurrency.  
         [0016]     Concurrency issues are exacerbated when aggregate summary updates are double-entry as in the available-to-promise at location aggregate summary example. This issue potentially leads to transaction contention chains on the aggregate summary table, which can close resulting in deadlocks.  
         [0017]     Referring again to  FIGS. 1-3 , the concurrent movement  145  of Asset  2  ( 130 ) and Asset  3  ( 135 ) from Location A  105  to Location B  110  would cause a single change in each of rows  215  and  220  of the initial Asset Table  205  of  FIG. 2 , from Location A to Location B. In addition, the change would cause at least two changes to the initial Inventory Summary Table  305   a  of  FIG. 3 . For example, row  310  currently indicates that two ATP Locomotives are at Location A. Thus, moving one of the two assets would cause row  310  to change to one available to promise Locomotive at Location A and cause row  312  to change to one available to promise Locomotive at Location B. A similar change would be required for the second asset. Because the asset movement for each asset affects the same rows in the Inventory Summary Table  305   a , the movement could cause possible conflicts between locks on the rows needed for each change. Using this simple example of just two assets moving concurrently, it is easy to see how concurrent movement of many assets to and from various locations, as is common in large shipments of assets, could cause major locking conflicts and likely deadlocks.  
         [0018]     In addition, some business logic is dependent on accurate post-update data. For example, as a transaction processes, certain business logic may look to the data update in progress to for changes in the aggregate summary data, for example, for purposes of determining changes in the status of assets in route from what has been promised. Thus, the above locking schemes fail to accommodate such business logic because the transaction must complete before any data is available.  
       SUMMARY OF THE INVENTION  
       [0019]     In accordance with one aspect of the present invention, there is provided a method and system of updating highly concurrent aggregate summaries. The method comprises modifying the above-described scheme to delay update of the aggregate summary table, typically to about as late as possible in the transaction, while maintaining an accurate in-progress aggregate summary for use by post-update dependent business logic.  
         [0020]     Specifically, the system converts all intended updates to an inventory summary table for each transaction to updates to a temporary in-database or in-memory delta table containing either the unconsolidated updates or net deltas according to application need. It then constructs a view table, which is a consolidation of the inventory summary table and the temporary delta table for use by the in-progress transaction. At the end of the transaction, typically just prior to the transaction commit, the system converts the contents of the temporary delta table into a single-statement consolidated update of the inventory summary table.  
         [0021]     This process bypasses the above-described locking issue entirely by separating updates to the summary table as inserts to another table. Because this process is not functionally equivalent to updating rows on a database, the problems associated with prior art methods are reduced or eliminated. By placing the ultimate update late in the transaction timeline and locking rows in a predetermined sequential order, the duration of any write locks on the inventory summary table is minimized, thus maximizing concurrency, improving throughput, and avoiding deadlocks. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0022]      FIG. 1  is a schematic diagram illustrating an example of assets at different locations in an asset management system.  
         [0023]      FIG. 2  is an initial asset table including data from the schematic of  FIG. 1 .  
         [0024]      FIG. 3  is an inventory summary table including aggregate data from the initial asset table of  FIG. 2 .  
         [0025]      FIG. 4  is a flow chart illustrating a method of updating highly concurrent aggregate summaries according to one embodiment of the present invention.  
         [0026]      FIGS. 5A &amp; 5B  are delta tables including changes intended for an inventory summary table according to one embodiment of the present invention.  
         [0027]      FIGS. 6A &amp; 6B  are view tables including information from a consolidation of an inventory summary table and a delta table according to one embodiment of the present invention.  
         [0028]      FIGS. 7A &amp; 7B  are updated inventory summary tables according to one embodiment of the present invention.  
         [0029]      FIG. 8  is a flow diagram illustrating an example of a method of updating highly concurrent aggregate summaries according to one embodiment of the present invention.  
         [0030]      FIG. 9  is a block diagram illustrating a system for updating highly concurrent aggregate summaries according to one embodiment of the present invention. 
     
    
     DETAILED DESCRIPTION OF THE EMBODIMENTS  
       [0031]     Referring now to  FIG. 4 , there is shown a flow chart illustrating a method of updating highly concurrent aggregate summaries according to one embodiment of the present invention. Note that the right side of the  FIG. 405, 430 ,  440 - 455  corresponds to actions taken by a single transaction in progress, the left side of the  FIG. 410-425 ,  435  corresponds to actions taken by a system, for example system  800  of  FIG. 8 , for updating highly concurrent aggregate summaries, and the dotted lines  407 ,  412 ,  417  connecting the two sides indicate alignment between the timing of the respective actions by the transaction and system. To begin the process, a transaction initiates  405  relative to an inventory summary table, e.g.,  305   a , to update data stored in the inventory summary table stored in a database, e.g., Database  810  of  FIG. 8 . Once the transaction request is received  410  by the system, it applies  415  the request instead to a temporary delta table, e.g., Temporary Delta Table  505   a  of  FIG. 5A , stored in the database or in memory (not shown). The delta table is a staging table containing only the changes (or deltas) intended for the inventory summary table, e.g.,  305   a , and may contain multiple rows incrementing or decrementing values for each row in the inventory summary table. The delta table is specific to, and cleared at the end of, each transaction. Thus, transactions need not have access to delta tables other than their own. In addition, because the delta table rows will be deleted, removed, or otherwise inactivated before the transaction commits, no later-occurring transactions can see the rows in the delta table for previous transactions. In some embodiments, the information in the delta table is consolidated  420  before moving forward. For example, if several separate changes affecting a single attribute are contained in the delta table in different rows, those rows first are consolidated into a single row. In other words, if Row 1 reduced the number of ATP Locomotives at Location X by 5, Row 2 increased the number of Locomotives at Location X by 2, and Row 3 reduced the number of ATP Locomotives at Location X by 1, these three rows would be consolidated to a single row indicating that ATP Locomotives at Location X had been reduced by 4 (−5 (Row1)+2 (Row 2)−1 (Row 3)=−4).  
         [0032]     Next, the inventory summary and delta tables, e.g.,  305   a  &amp;  505   a , are consolidated to create  425  a view table, e.g.,  605   a , in the database. The view table is a logical construct that is stable and always is used  430  by the in-process transaction (rather than the inventory summary table). The data in the view table is inaccurate at this point with respect to what is contained in the inventory summary table because the transaction has not yet committed  455 . However, the view table is an accurate in-progress summary useful for the transaction, e.g., in step  430 . As a result, if the view table indicates that inventory level has dropped below the level required, system alerts may be issued at this stage, triggering other business processes. As a result, the view table can be thought of as an in-the-future view with respect to what data will exist in the inventory summary table following the transaction commit. Thus, other transactions processing at about the same time as the in-process transaction see the (accurate) information in the inventory summary table rather than the view table.  
         [0033]     Just before the transaction commit  455 , the system performs a single consolidated update  435  of the inventory summary table using the contents of the delta table. As part of this update, the transaction first locks  440  the row(s) of the inventory summary table affected by the transaction. Next, the locked rows of the inventory summary table affected by the deltas are updated  445  with those changes. In one embodiment, the single update is a single Structured Query Language (SQL) statement, subgrouped by data that affects each row in the inventory summary table. Because the transaction manager  825  (of  FIG. 9 ) limits row locking to near the end of the transaction, the row is locked for short period of time, compared to if the same lock was taken early in the transaction. Locks taken earlier in the transaction would require other transactions to wait, resulting in some of the locking issues discussed with respect to pessimistic and optimistic locking methods in the background section. In addition, locks on the inventory table are taken out in a specific, algorithmic order, guaranteeing that a deadlock cannot occur. Then, the deltas are inactivated  450  from the delta table prior to the transaction commit  455 . In one embodiment, inactivation may include deletion or removal of rows. Because the view table is a logical construct, inactivation of the delta table rows concurrently inactivates or deletes the changes to the view table.  
         [0034]     The single statement update of this method is advantageous over a traditional update when used for assert movement in terms of simplicity and efficiency; a traditional update would require two statements to update the inventory summary table: one to decrement the asset count at the sending location and one to increment the asset count at the receiving location. In addition, the placement of the update late in the transaction results in higher system throughput, while the view table allows the business logic to operate unaffected by transaction processing.  
         [0035]     The above-described aspects of the invention are advantageous for use in asset management contexts, such as for asset movement, as detailed in the following example, as well as asset creation and destruction. In addition, the method is useful for maintaining real-time financial account balances in accounting systems dealing with high-volume line-item amounts, as well as in other applications.  
         [0036]     The following is an example of one embodiment of the method of  FIG. 4 , with reference again to the simple movement  145  of Assets  2  ( 130 ) and  3  ( 135 ) from Location A  105  to Location B  110  as depicted in  FIG. 1 . Once a transaction request for Asset  2  (Transaction  1 ) is received  410  by the system, it applies  415  the request to a temporary delta table in the database instead of the initial Inventory Summary Table  305   a . Referring also now to  FIG. 5A , it shows an example of a Delta Table  505   a  according to this example. The Delta Table  505   a  includes one row for each change ( 510 - 515 ), and columns for the type of asset involved in the change ( 530 ), the location (LOC) of the asset ( 535 ), the change in ATP assets ( 540 ), and for which transaction the change applies ( 545 ). Thus, in this example, the Delta Table  505   a  contains two rows  510 - 515 : one row representing the decrement of ATP by one asset for Assets  2  at Location A ( 510 ) and one row representing the increment of ATP by one asset for Assets  2  at Location B ( 515 ).  
         [0037]     Next, the Inventory Summary  305   a  and Delta  505   a  Tables are consolidated to create  425   a  a View Table  605   a  in the database, e.g.,  810  of  FIG. 9 . Referring now also to  FIG. 6   a , a View Table  605   a  contains the information from the initial Inventory Summary Table  305   a  updated  425   a  to include the Delta Table  505   a  information. Continuing with the example, the View Table  605   a  includes four rows  607 - 620 . Note that rows  615  and  620  have not changed from the initial Inventory Summary Table  305   a . However, row  607  now shows that one locomotive type asset is available at Location A and row  610  now shows that one locomotive type asset is available at Location B.  
         [0038]     As a final step before the transaction commit  455 , the system performs a single consolidated update  435  of the Inventory Summary Table  305   a  using the contents of the Delta Table  505   a . As part of this update, the transaction first locks  440  the row(s) of the inventory summary table  305   a  affected, in this example row  310  of  FIG. 3 . Next, the locked rows are updated  445   a  with those changes. The result is the updated Inventory Summary Table  305   b , shown as  FIG. 7A . Initial Inventory Summary Table  305   a  and updated Inventory Summary Table  305   b  represent instances of data contained in the Inventory Summary Table  305  at two different points in time. Note that the content of the updated Inventory Summary Table  305   b  is identical to that of the View Table  605   a . Then, the changes are inactivated  450  (or deleted) from the Delta Table  505   a  (and thus from View Table  605   a ) prior to the transaction commit  455 .  
         [0039]     This process then repeats for Asset  3  (Transaction  2 ), as shown in  FIGS. 7A, 5B ,  6 B, and  7 B. Referring now to  FIG. 8 , it summarizes the process of  FIGS. 3A-7B , showing the simple movement  145  of Assets  2  ( 130 , T 1 ) and  3  ( 135 , T 2 ) from Location A  105  to Location B  110  as depicted in  FIG. 1 . Briefly, the Initial Inventory Summary Table  305   a  and Delta Table  505   a  merge  425   a  to form View Table  605   a . Then, prior to the transaction (T 1 ) commit, the Inventory Summary Table is updated  445   a  with the data from Delta Table  505   a  to create updated Inventory Summary Table  305   b . Next, the (now updated) Inventory Summary Table  305   b  and Delta Table  505   b  merge  425   b  to form View Table  605   b . Then, prior to the transaction (T 2 ) commit, the Inventory Summary Table is updated  445   b  with the data from Delta Table  505   b  to create updated Inventory Summary Table  305   c.    
         [0040]     Note that the ordering here of Transaction  1  and  2  is arbitrary. The first transaction to commit gets first priority; thus the transaction ordering may be considered “accidental.” 
         [0041]     Referring now to  FIG. 9 , it shows a block diagram illustrating a system for updating highly concurrent aggregate summaries according to one embodiment for implementing the present invention. The system includes an application server  805  attached to a database  810 ; the application server also may be attached to various business components  815 . The application server  805  controls data updates, for example as described in the above method, and may be any application server that interacts between various business components  815  and a database  810 . In one embodiment, the application server  805  is a Weblogic J2EE application server implemented by BEA. The database  810  may be any data repository that persists data in tables  830 . In one example, the database  810  is an Oracle  9   i  relational database. In one example, the tables  830  include the various types of tables described herein ( 205 ,  305 ,  505 ,  605 ). The business components  815  may be various components that transmit or receive data to/from the business engine  820 . In one embodiment in which the system monitors updates to asset movement using RFID tags, one of the components  815  is an RFID tag reader.  
         [0042]     The application server  805  further comprises a business engine  820 , a transaction manager  825 , and an inventory module  827 . The business engine  820  receives messages to move assets, create assets, dispose of assets or change assets&#39; properties and states from the various business components  815 . The business engine  820  communicates with the transaction module  825  and inventory module  827  to execute transactions to adjust tables  830 , for example in response to movement, creation, or destruction of assets.  
         [0043]     The business engine  820  further includes one or more application modules  835 . The application modules  835  check the inventory position by sending instructions to the view module  845  of the inventory module  827 , then modify asset information in the tables  830  in response to changes in asset state and send instructions to the increment/decrement module  840  of the inventory module  827 . In one embodiment, the invention requires at least two application modules  835  because changes in asset state take at least two forms, for example, movement and change of status type. The delta table may be one of the tables  830  or may be stored in memory (not shown). After the application modules  835  have been executed, the business engine  820  instructs the transaction manager  825  to commit the transaction. This causes the consolidation of the inventory delta and inventory tables to occur as late as possible in the transaction. The transaction manager  825  interacts with the consolidation module  850  of inventory module  827 , and database  810  to process updates to the tables  830 . Specifically, the transaction manager  825  delegates the consolidation function to the consolidation module  850  of the inventory module  827 .  
         [0044]     The inventory module  827  controls the inventory positions of assets. The inventory module  827  further includes an increment/decrement module  840 , a view module  845 , and a consolidation module  850 . The increment/decrement module  840  acts to increase and decrease the inventory positions of assets in the tables  830 . The view module  845  understands the view table and provides access to the view table for the transaction in progress. The consolidation module  850 , in response to delegation form the transaction manager, consolidates data contained in the delta table into the inventory summary tables at or towards the end of the transaction. The consolidation module  850  performs the inventory summary table update using the data from the delta table. Specifically, the consolidation module  850  facilitates the locking of rows to be updated, updates the information in the inventory summary table, and inactivates or deletes the changes to the delta table. At the end of these steps, the transaction manager  825  to commits the transaction to the database  810 .  
         [0045]     Finally, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.