Persisting of a low latency in-memory database

Processing is provided for operating an in-memory database, wherein transaction data is stored by a persistence buffer in an FIFO queue, and update processor subsequently: waits for a trigger; extracts the last transactional data associated with a single transaction of the in-memory database from the FIFO memory queue; determines if the transaction data includes updates to data fields in the in-memory database which were already processed; and if not, then stores the extracted transaction data to a store queue, remembering the fields updated in the in-memory database, or otherwise updates the store queue with the extracted transaction data. The process continues until the extracting is complete, and the content of the store queue is periodically written into a persistent storage device.

PRIOR FOREIGN APPLICATION

This application claims priority from European patent application number EP11162181.9, filed Apr. 13, 2011, which is hereby incorporated herein by reference in its entirety.

BACKGROUND

The present invention relates in general to the field of real-time processing of data streams. More specifically, this invention pertains to a system and method for efficiently writing data from an in-memory database to a persistent memory.

Many applications need the fast response and high throughput that is typical of in-memory databases, but also need the reliability and recoverability of traditional disk based relational database management systems. What these applications require is a method for copying data from the in-memory database to a persistent memory, such that the data can be easily recovered in the event of a machine crash.

Minimizing the lag time between the data written to the in-memory database and subsequently written to the persistent memory is important. This lag should be as short as possible to minimize the loss of data in the event of a machine crash. In addition, if the data copied to the persistent memory is time stamped then the persistent memory effectively represents the state of the in-memory database over time. This allows for the possibility of recreating the in-memory database from the persistent memory for any point in time.

Conventional technologies that store streaming data in a buffer memory and then transfer this data to an on-disk database have relatively long lag times between arrival of the data and the transfer to the database.

The U.S. Pat. No. 7,113,953 B2 describes an efficient system, where an in-memory database is synchronized with a relational database management system with a lag time on the order of seconds. But this system requires that the operations of the in-memory database are limited to insert only operations such that update operations are not allowed. This restricts the class of real-time applications significantly. Examples are in-memory databases for financial market data.

For applications such as in-memory game database management systems for massively multiplayer online games (MMOG) this restriction is not feasible as update operations cannot be omitted. In a MMOG the game database needs to store information about all the objects and players on the game and hosts business critical information therefore. As a MMOG needs to support hundreds or even thousands of players simultaneously, the game database can require a huge amount of space. The scalability of a MMOG in terms of numbers of players and game objects depends mostly on the scalability of its object model and the game transaction rate that must be visible to all players in the same part of the world of the game. Therefore, the performance of the game database management system determines the overall performance and responsiveness of the online game.

State of the art MMOGs use game database management systems which partition the game users into disjoint groups such that members of different groups can never meet, or which partition the game world into disjoint spaces, or which use both approaches. The partitioning is achieved by using multiple small, cheap and unreliable server machines and by splitting the data of the game database between these machines. However, this adds latency to the database operations due to the additional overhead for the operation and control of the machines.

BRIEF SUMMARY

According to one embodiment of the present invention, a method to operate a volatile in-memory database is presented which comprises: the in-memory database subsequently performing: a) receiving a transaction to modify content of the in-memory database; b) storing transaction data associated with the transaction in the in-memory database; c) determining if an active FIFO memory queue in a persistence buffer is full; d) if the active FIFO memory queue is full, setting a trigger for an update processor and selecting another FIFO memory queue as active; e) storing the associated transaction data in the active FIFO memory queue; f) continuing with the receiving a); wherein steps b) to e) are performed as part of the commit function of the in-memory database, and: the update processor in parallel to the in-memory database subsequently performs: g) waiting for the trigger; h) extracting the last transaction data associated to a single transaction of the in-memory database from the FIFO memory queue; i) determining if the transaction data comprises updates to data fields in the in-memory database which were already processed since step g) was performed; j) if not then storing the extracted transaction data to a store queue and remembering the fields updated in the in-memory database; otherwise, updating the store queue with the extracted transaction data; and k) continuing with step h).

According to another embodiment of the invention, a data processing system is proposed, which comprises a volatile in-memory database and an update processor, wherein the in-memory database and the update processor comprise means to implement the method described above.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

FIG. 1illustrates a data processing system100in accordance with the present invention. An application110exchanges data with an in-memory database120. The in-memory database120uses a persistence buffer130to persistently store data in a backing store150. The stored data can also be retrieved by the in-memory database120from the backing store150. The data is stored from the persistence buffer130to the backing store by an update processor140.

In one embodiment of the invention, the application110is an MMOG supporting multiple players. For example, a possible MMOG is an application110which allows a player to buy a virtual drink for e.g. $3 from another player at a virtual bar using virtual money accounts for the players. The application110calls the in-memory database120to process a transaction to move $3 from the buyers account to purchasers account within the application110. The in-memory database120then processes the transaction and stores the new values.

In pseudocode this transaction can be described as:

begin transaction;set a.x=a.x−3;set b.x=b.x+3;commit;
If the buyer had $15 and the purchaser had $14 before the transaction, then the transaction will result in a.x=12 and b.x=17.

In one embodiment of the invention, the in-memory database120is an object-oriented database. This allows to easily manage the game data, which can be represented in form of data objects. Each player can be represented as a data object to which comprises further objects. In the example above, the players are represented by the objects “a” and “b” which comprise an object “x” each, which represents the respective accounts of the players. But the in-memory database120can also be a relational database management system (RDBMS) for example, which does not support persistent data.

The “commit” function of the in-memory database120is adapted such that upon success instead of just returning to the calling application110it will remember the update. This is done by intercepting the commit function and writing a “transaction record” from the in-memory database120to the persistence buffer130. In pseudocode this transaction record can be described as:

In one embodiment, with an object-oriented in-memory database120the transaction record could comprise the entire data objects that are modified during the transaction. But this would consume too much memory. Therefore, at least the data fields from the modified objects that are changed during the transaction need to be comprised within the transaction record in order to save as much space as possible. In that case, those updated data fields are stored together with an object identifier in the transaction record to ensure that these data fields can be associated to the respective object.

FIG. 3shows the creation of a new transaction of the application110in step300. The new transaction is then stored in the in-memory database120in step310. When the corresponding transaction record is written by the in-memory database120to the persistence buffer130as part of the interception of the commit function, the persistence buffer130inserts the current timestamp to the transaction record and stores it in one of its buffer storage areas, e.g., by appending the transaction record to the end of a sequential file. The timestamp is assumed to be unique which can be guaranteed by well-known methods.

Once this is completed and the record is safe the persistence buffer130signals the successful storage of the transaction record to the in-memory database120. The in-memory database120then notifies the application110that the update transaction completed successfully, i.e., the commit function returns without errors. In case of failures by writing the transaction record to the persistence buffer130, the commit function fails, the usual “unrolling” of the transaction is triggered by the in-memory database120and the application110is informed that the transaction failed.

The persistence buffer130stores the transaction records in one of its FIFO (First-In First-Out) buffer storage areas. In the simplest case, there is one active FIFO buffer to which all incoming transaction records are appended. There may be one or more FIFO buffers that are “complete”. Each of these FIFO buffers contains all of the transaction records between two unique timestamps. The FIFO buffers do not overlap as far as the timestamps of the transaction records are concerned. For example,FIG. 2shows such buffers200,210, and220in the persistence buffer130. FIFO buffer200represents the transaction record from the example above, wherein the amount of money owned by the purchaser and the buyer is adapted as a result of the sale of the virtual drink.

It is possible that the capacity of a FIFO buffer is exceeded. In that case, no further transaction records can be stored in this full FIFO buffer. Therefore, in step320shown inFIG. 3the persistence buffer130determines if the active FIFO buffer is already full. If that is the case, then the persistence buffer130will switch to another FIFO buffer in step330. This FIFO buffer is then marked as active. Otherwise, the persistence buffer will store the transaction record in-order in the active FIFO buffer in step340.

The oldest of the unprocessed data sets in the persistence buffer130will be processed now by the update processor140. The newest timestamp in the dataset is t0. The data set may be large, so it will not be physically copied, but accessed on a per transaction record basis. The data set will be sorted into a data structure set such that for each value changed during the transaction processing timeframe represented by this data set the newest value is kept.

FIG. 2shows an oldest data set230and a newer data set240. Both data sets contain an identifier (TR) for the associated transaction record and the time stamp (TS) of the transaction record. The newer data set240represents the transaction that immediately preceded the example above, wherein the current amount of money owned by the purchaser and buyer is defined. The oldest data set230is sorted in data structure set250by the update processor140. Then the newer data set240is processed by the update processor140, which results in the data structure set250as shown inFIG. 2.

A possible implementation for a persistence buffer data set is a sequential file, which can be maintained in a main memory of a computer system. During transaction processing (while the buffer/data set is active), new transaction records are simply appended. The update processor140reads the file sequentially from beginning to end. The “sorting-in” step340becomes trivial then: If a newly read transaction record updates a field already recorded, its timestamp is checked. If it is newer then the one associated with the recorded update, then the new value and timestamp is remembered. If it is older, then it is ignored. Therefore, one sequential read is enough, no matter in what order the records are stored in or retrieved from the data set.

When the persistence buffer130switches to another FIFO buffer in step330, then also a full queue process is triggered. The update processor140will then extract all the transaction records stored in the full FIFO buffer. This is shown inFIG. 4. In step400the update processor140will determine if the queue of the FIFO buffer is already empty. If that is the case, then the update processor140stops its transaction processing in step440. Otherwise, the last transaction record is extracted from the FIFO buffer in step410. Then it will be determined in step420if the extracted transaction record was already processed before during the extraction of the queue. If that is not the case, then the transaction record is stored in a store queue of the update processor (140) and the execution is then continued with step400. Otherwise, the store queue is updated with the transaction record in step440. After the store queue was updated, the update processor (140) continues with step400.

The update of the store queue in step440can be implemented easily for those embodiments that store the entire modified objects within the transaction records. In that case the extracted transaction record can be ignored in case its timestamps indicates that it is older than the one already stored in the store queue. For other embodiments it is required to update the fields within the objects only, that are affected by the transaction records. An implementation is shown inFIG. 5, which is an adaptation of the method shown inFIG. 4. In this example, the objects are related to fields within database objects, which can therefore be stored in a temporary empty in-memory database. The transaction records are then stored in the persistence buffer as a sequence of objects relating to the updated fields.

In step500ofFIG. 4, the update processor140will determine if the queue of the FIFO buffer is already empty. If that is the case, then the update processor140stops its transaction processing in step510after it stored the modified objects from the store queue in the persistent database on the backing store150. Otherwise, the last queue field is extracted from the FIFO buffer in step520. Then it will be determined in step530if the extracted object was already processed before during the extraction of the queue. If that is not the case, then the respective fields in the object are updated and stored in a store queue of the update processor (140) and the execution is then continued with step500. Otherwise, the object is retrieved from the persistent database in step440. After the store queue was updated, the update processor (140) continues with step400.

The content of the store queue is periodically written by the update processor (140) to the backing store (150). In the simplest case, the backing store150is a standard database management system with persistent storage devices, which maintains a database and the new values for the changed fields are just updated in this database. So the next time this database is loaded in the in-memory database120it represents a consistent overall state for a certain point in time t0. Advantageous embodiments of the invention use computer systems with multiple logical partitions. One of these partitions can then execute the application110and the in-memory database120. Another partition can execute the persistence buffer130and the update processor140. This partition can also execute the database management system for the backing store150. In the preferred embodiment of the invention, the application110and the in-memory database120are executed on the same computer system, whereas the persistence buffer130and the update processor140are executed on a different physical computer system. The in-memory database120and the persistence buffer130communicate via a network connection. In a special embodiment, the persistence buffer130and the update processor140could be executed twice on two different computer systems in order to obtain redundancy to improve the system reliability.

In a different embodiment of the invention recovery for different points in time is possible. One embodiment just stores the set of fields/values for t0, for example, in a file that is associated with t0. The original data is not updated. At a convenient point in time, old update files, i.e., all files representing updates before a user-specified point in time—are eliminated by applying them to the original data in chronological order thereby creating a new original dataset for the processing to continue as described above.

The details of how many update files to keep, whether or when to apply them, etc. has to be part of the overall solution for managing the data processing system100and can be derived from the needs of its users in terms of recoverability.

The completion of the update to the backing store150is acknowledged. The update processor140can now clean up all data structures related to the previously processed data set for t0. It may fetch the next, finished data set and continue processing. The update processing is completely asynchronous to the transaction processing.

When the application110or the in-memory database120fails, or the system administrator of the data processing system100stops transaction processing in order to reset to a previous stage then the persistence buffer130will deactivate the current buffer data set and stop processing further transactions. Then the buffers in the persistence buffer130will be marked as inactive and handled by the update processor140as described above. All updates will be reflected in the backing store150, again as described above.

The in-memory database120is reloaded with the data from the backing store150either with the newest possible state or an administrator-defined level some time back, by selecting one of the recoverable states held in the backing store. Now the application110can be restarted. Additional administrative tasks may be necessary. For example, if the restart is required because of an inconsistency, a reset to a state corresponding to t1may make it necessary to remove all newer snapshots t1+i from the backing store—otherwise the timeline would fork.

An example for a realistic scenario of the workload for the application110could be a mixture of action and strategy game, which can be characterized as follows:1 million subscribed users;100,000 concurrently active users;100 objects per user (which can participate in transactions);100 bytes per object;20% of the users show high activity (flying, shooting, . . . ) generating 10 transactions per second;80% of the users show low activity (thinking, trading, socializing, . . . ) generating 0.1 transactions per second;an average of 2 objects modified per transaction.

For this example a database size of at least 10 GB is necessary:
1 million users*100 objects/user=100 million objects
100 million objects*100 bytes/object=10 GB.

When it is assumed that full objects are recorded upon change, then transaction volumes are in the range of 208 k transactions/second in this example:
100 k users*20%*10 transactions/second+100 k users*80%*0.1 transactions/second

This results in a data rate of 41.600M Bytes/second:
208 k transactions/second*2 objects/transaction*100 bytes/object
So in one embodiment of the invention, the in-memory database120and the persistence buffer130can be connected with a single network connection using state of the art network technology.

When it is further assumed that a single buffer records the transactions of 1 hour of gaming, then the buffer contains:
41.600M Bytes/second*3600 seconds=149,760M Bytes˜150 GB 208 k transactions/second*2 objects/transaction*3600 seconds=1,497.600M object updates.

When it is also assumed that during one hour 200,000 players are active at least once, then during that hour 200,000 users*100 objects/user=20M objects may potentially be touched.

Assuming in the worst case for the invention that the modification of objects is uniformly distributed over time, then each object will be modified ˜75 times during one hour:
1,497.6M objects modified/20M total objects.

Since the update processor140only needs to actually store the last update for each object, one can save ˜99% of the updates to the backing store150, which in fact would make it feasible now to use a standard RDBMS to implement the backing store150.

Computer program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). The present invention is described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

FIG. 6illustrates a block diagram of a computer system1300in which certain embodiments may be implemented. The system1300may include a circuitry1302that may in certain embodiments include a microprocessor1304. The computer system1300may also include a memory1306(e.g., a volatile memory device), and storage1308. The storage1308may include a non-volatile memory device (e.g., EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, firmware, programmable logic, etc.), magnetic disk drive, optical disk drive, tape drive, etc. The storage1308may comprise an internal storage device, an attached storage device and/or a network accessible storage device. The system1300may include a program logic1310including code1312that may be loaded into the memory1306and executed by the microprocessor1304or circuitry1302. In certain embodiments, the program logic1310including code1312may be stored in the storage1308. In certain other embodiments, the program logic1310may be implemented in the circuitry1302. Therefore, whileFIG. 6shows the program logic1310separately from the other elements, the program logic1310may be implemented in the memory1306and/or the circuitry1302.