Log consolidation

Consolidating a set of tenant log streams from separate user databases into a consolidated log stream. The method includes receiving a plurality of tenant log streams from separate user data bases. The method further includes recording the plurality of tenant log streams as a consolidated log stream. The method further includes maintaining metadata about the consolidated log stream to map log records from the plurality of tenant log streams to their location in the consolidated log stream.

BACKGROUND

Background and Relevant Art

Further, computing system functionality can be enhanced by a computing system's ability to be interconnected to other computing systems via network connections. Network connections may include, but are not limited to, connections via wired or wireless Ethernet, cellular connections, or even computer to computer connections through serial, parallel, USB, or other connections. The connections allow a computing system to access services at other computing systems and to quickly and efficiently receive application data from other computing systems.

Interconnection of computing systems has facilitated distributed computing systems, such as so-called “cloud” computing systems. In this description, “cloud computing” may be systems or resources for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, services, etc.) that can be provisioned and released with reduced management effort or service provider interaction. A cloud model can be composed of various characteristics (e.g., on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, etc.), service models (e.g., Software as a Service (“SaaS”), Platform as a Service (“PaaS”), Infrastructure as a Service (“IaaS”), and deployment models (e.g., private cloud, community cloud, public cloud, hybrid cloud, etc.).

Computing systems often store data using databases. Databases are typically associated with a database log which contains a history of actions executed by a database management system. The log can be used to recover from database crashes or hardware failures. In particular, if a database is found to be inconsistent, the database log can be used to put the database back into a last known correct state. The database log is typically maintained by streaming the history of actions to storage.

For optimal performance, databases exploit the sequential nature of access to the log by keeping the log on a dedicated disk. Cloud environments typically have a large number of tenants and thus a large number of user databases. To control costs, it may be useful in cloud environments to consolidate a large number of user databases on a single node (i.e. machine). However, because the node has far fewer disks than the number of databases it hosts, it is impractical to have a dedicated disk for each database log.

BRIEF SUMMARY

One embodiment illustrated herein is directed to a method of consolidating a set of tenant log streams from separate user databases into a consolidated log stream. The method includes receiving a plurality of tenant log streams from separate user databases. The method further includes recording the plurality of tenant log streams as a consolidated log stream. The method further includes maintaining metadata about the consolidated log stream to map log records from the plurality of tenant log streams to their location in the consolidated log stream.

DETAILED DESCRIPTION

Some embodiments described herein are able to consolidate the logs of various user databases into a single consolidated log. An example of this is illustrated inFIG. 1, illustrating three databases102-1,102-2, and102-3implemented in a remote service provider system104. Each of the databases includes a logging subsystem106-1,106-2, in106-3respectively. Each of the logging subsystems is coupled to a consolidated log108. The consolidated log108is able to create a consolidated log file110. Thus, log data from a plurality of different databases can be consolidated into a consolidated log file110that can be stored on a single disk112. While not shown here, separate metadata is maintained including information about the database from which log data originally originated from, such that a view can be generated for each of the different databases of the log information pertinent to the particular databases. As illustrated herein, this functionality is accomplished by implementing various features now described.

Embodiments may implement virtualization at the I/O subsystem. This allows physical log streams from different databases to be virtualized into a consolidated log stream that can be stored in the physical log file110. Embodiments may provide asynchronous I/O abstraction to tenant file I/O mechanisms illustrated herein as File Control Blocks (FCBs). This allows individual tenant databases (referred to herein generically at102) to write log streams into the consolidated log108. Embodiments may group multiple I/Os to retain benefits of consolidation. For example, as illustrated inFIG. 1, log streams from a number of different databases are consolidated into a consolidated log108.

Some embodiments illustrated herein implement mapping of data and the physical log file110to allow the data to be correlated with its originating database102. The mapping is performed in a way such that the mapping is efficient while still achieving persistence and recovery of the mapping through the same log stream.

Some embodiments illustrated herein implement functionality for managing the consolidated stream. This may be accomplished through sector-remapping and the move-to-data strategies discussed in more detail below.

Some embodiments illustrated herein may make use of abstraction. In particular, embodiments may facilitate using existing components and composing them into a working solution for solving the log consolidation problem.

The following illustrates a system configured to append multiple user database log streams to a shared physical sequential log while maintaining the identity of each stream. This may allow each stream to be independently provisioned, backed up and exported for high availability.

Such embodiments may be able to ensure sequential I/O to the log in the common case.

Additionally or alternatively, embodiments may be implemented such that the number of individual log streams on a node is independent of the number of disks available.

Additionally or alternately, embodiments may be implemented such that the log based functionality that depends on having a separate log (e.g. physical log based high availability, log backup, log shipping, database snapshots, transparent database encryption, etc.) works seamlessly.

Additionally or alternatively, embodiments may be implemented such that existing on-disk format is preserved so that a database may be exported (using database backup and restore functionality) from a consolidated environment to one that uses individual log files and vice versa.

Alternatively or additionally, embodiments may be implemented such that to the extent possible, embodiments use existing remote based (e.g. cloud based) database infrastructure.

To understand how a consolidated log108can be managed, it is first helpful to understand how individual database logging functions.FIG. 2illustrates an example of a SQL Server® database available from Microsoft® Corporation of Redmond Washington.FIG. 2illustrates a logging subsystem106. In the example illustrated, an I/O subsystem114(illustrated as an FCB component) is used to write data to the file system stack116. The I/O subsystem is not bypassed by any higher level component that needs to access the contents of a log file. Therefore embodiments consolidate logs at the I/O subsystem.

Each log manager118operates on its log stream as though it were an independent file. This preserves the structure of each stream thereby having no impact on existing log based functionality. As illustrated inFIG. 3, the consolidated log108intercepts write requests to the individual streams from the I/O subsystems114-1,114-2and114-3(referred to herein generically as114) and linearizes them into the physical log file110.

The consolidated log provides (1) virtualization functionality, (2) consolidation functionality, and (3) mapping functionality.

To implement virtualization, the consolidated log exposes file system like APIs over the consolidated log stream to the database engine subsystems114. In a very specific example, embodiments may expose NTFS-like APIs to a SQL Server® Engine FCB layer. Embodiments do not need to support all of the file system APIs, but rather only the ones needed for log files. In the NTFS and SQL Server® example, embodiments may support the following:WriteFile to write N sectors at a specified offsetReadFile to read N sectors starting at a specified offsetZeroFileSetFileSize (Grow/Shrink)Support for asynchronous I/Os

To intercept the I/O subsystem specific I/Os embodiments may include another thin layer of I/O interfaces at the I/O subsystem level that mimic the I/O level interfaces as static virtual methods, then create a new class of I/O subsystem that derives from the I/O subsystem and override these virtuals for log consolidation.

To implement the consolidation functionality, the consolidated log108performs group writes from multiple log streams and linearizes them into a sequential stream. Multiple writes are grouped into a consolidated I/O to the consolidated stream. The consolidated log108does the necessary caching to facilitate such grouping.

Some embodiments maintain a mapping to map from an offset within a tenant log stream of a logging subsystem104of a database102to the corresponding location in the consolidated stream stored in the physical log file110of the consolidated log108. This mapping should be persistent and recoverable. The mapping structure may not always fit in memory, so it may be spooled to disk while caching the frequently accessed portions in memory. In particular, the mapping may map a stream identifier and an offset in a local stream to an offset in a consolidated log stream.

With reference toFIG. 4, the following now illustrates the architecture of a consolidated log108. The consolidated log108may be a standard database. For example, the consolidated log108may be itself a SQL Server® Database. The consolidated log108may be referred to herein as “log consolidation host” or simply “host”. The consolidated log108includes a physical log file110and a data file120. The log manager122of the host manages the physical consolidated stream. Log files of a given user database102(hereafter referred to as a tenant) are virtualized with their offsets mapped to the specific locations in the consolidated log file110where they are eventually located.

Each sector in the user database log file is written as a separate log record in the consolidated log stream. A new log record type (illustrated herein as LOP_TENANT_LOG) is introduced whose fixed size fields include metadata about the tenant and the variable portion includes the data from the tenant log stream. This abstraction ensures that the on-disk structure of the consolidated log108is unchanged and the log manager122and log scanners124of the host can operate on it like any other log file.

The mapping is stored in the host108in internal tables (referred to as LCMap126herein) which are suitably indexed for fast access. Updates to the map internal tables126are logged as regular log records in the consolidated log. Since the consolidated log stream also contains log records for updates to the LCMap126, the durability of the contents of tenant102log streams and the corresponding mapping is ensured through a consolidated I/O. The LCMap126effectively stores mapping from: a stream identifier and a sector number (or other unit of granularity in a given stream) to: a log sequence number (or other identifier that can be used to identify a log record) of the tenant log record that has data for the sector.

As shown in theFIG. 4, this novel architecture facilitates re-use of many existing standard database components while only developing a thin virtualization layer128to intercept I/O and serve them using the consolidated stream.

The following now illustrates tenant log read and write requests. Writes to the tenant log are now illustrated. Log writes are sector aligned, so each write (and read) includes an integral number of sectors. For each sector:(1) Start a transaction.(2) Generate LOP_TENANT_LOG record that includes the contents of the sector as the payload and obtain its log sequence number.(3) Insert/Update the mapping table to map (StreamId, Sector#) to the log sequence number in Step 2.(4) Commit the transaction without flushing the log.(5) The transaction can optionally be scoped for the complete write request (which includes N sectors).

During this process, the data is still in the cache of the consolidated log manager122, when sufficient writes have accumulated, the consolidated log is written to disk. Only after the log has been written to disk, the tenant102is notified of I/O completion.

Reads are handled similarly. For each sector embodiments first access the mapping table LCMap126and retrieve the map log sequence number. Embodiments then position a log scanner on the map log sequence number and read the log record and copy the sector data to the read buffer.

The tenant I/O thread simply submits I/O requests to a queue. The actual reads and writes are handled by a pool of threads in the background thereby facilitating asynchronous I/O from the tenant's perspective. This also provides opportunity to re-use existing data structures such as log scanners which can be expensive to setup.

Zeroing a tenant file is achieved by simply updating the map to indicate that the corresponding sector is zero. Similarly tenant file size changes are achieved by growing and shrinking the LCMap126without actually affecting the physical log stream.

The following now illustrates details related to database recovery. From the perspective of recovery, the consolidated database108is just another database. The mapping structure is recovered like any other table. If the transaction that was started to write to a log on behalf of a tenant was incomplete, the mapping entry will not reflect the update and the corresponding write is considered to have failed. This does not affect correctness, as the tenant is notified of write completion only after the transaction was durably committed.

The recovery of the tenants102needs to access the consolidated database108, therefore the consolidated database108is recovered before any of the tenants102. Embodiments therefore ensure that the consolidated database108recovery does not have any dependence on the tenants102. Any cross database transactions that involve the consolidated108and the tenant102should always choose the consolidated108as the coordinator.

The following illustrates details with respect to log truncation. The tenant log streams maintain the oldest log sequence number of interest to gate the truncation of the consolidated stream. When the tenant log truncates any portion of its log, it frees up the corresponding log records in the consolidated stream. The consolidated stream ensures that any log records that are required by any tenant are retained.

Idle databases that do not advance their log may hold up the truncation of the log. Two ways that this can be handled by the truncation include (1) sector re-mapping and (2) moving the log to data.

The following illustrates details regarding sector re-mapping. As the LCMap126provides a layer of in-direction, the contents in the consolidated stream can be moved without affecting the tenant102. The background truncation task looks for old log records in the host that may be holding up truncation and re-maps the corresponding sectors by generating new log records at the end of the stream and updating the map to point to the new record instead.

The following illustrates details regarding moving a log to data. The LCMap126has an additional nullable blob field. The background task pushes the contents of the sector to the blob thereby freeing up the log records in the host. Any attempts to read these sectors will be served directly from LCMap126without having to go to the consolidated log at all.

Referring now toFIG. 5, a method500is illustrated. The method includes acts for consolidating a set of tenant log streams from separate user databases into a consolidated log stream. The method includes receiving a plurality of tenant log streams from separate user data bases (act502).

The method500further includes recording the plurality of tenant log streams as a consolidated log stream (act504).

The method further includes maintaining metadata about the consolidated log stream to map log records from the plurality of tenant log streams to their location in the consolidated log stream.

The method500may be practiced where maintaining metadata comprises updating metadata in mapping tables to identify where data in the consolidated log stream came from. For example,FIG. 4illustrates an LCMap that can be updated to identify what logs data comes from.

The method500may be practiced where recording the plurality of physical log streams as a consolidated log stream is performed by virtualizing the plurality of physical log streams into the consolidated log stream. For example, this may be done by providing asynchronous I/O abstractions to the tenant log files to facilitate the virtualization.

The method500may be practiced where maintaining metadata comprises using native indexing methods of a backing database to perform efficient and persistent, and recoverable mapping. For example, existing database infrastructure may be modified to implement the consolidated log.

The method500may further include recovering one or more of the separate user databases, wherein recovering one or more of the separate user databases comprises first recovering the database for the consolidated log stream and then recovering log streams for individual tenants.

Further, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures can be transferred automatically from transmission computer readable media to physical computer readable storage media (or vice versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., a “NIC”), and then eventually transferred to computer system RAM and/or to less volatile computer readable physical storage media at a computer system. Thus, computer readable physical storage media can be included in computer system components that also (or even primarily) utilize transmission media.