Patent Description:
One way to improve data access times in a relational database system is to pre-load into volatile memory either an entire database object, or portions thereof. For example, operations that involve a table T1 may be performed faster if a copy of table T1 has been preloaded into volatile memory. Techniques for pre-loading database objects, or portions thereof, into volatile memory are described, for example, in <CIT> (the "Mirroring Patent").

Within volatile memory, in-memory copies of database objects (or portions thereof) are stored in In-memory Units ("IMUs"). The data stored in an IMU is referred to as a chunk. Any given chunk may include all data from a database object or a subset of the data from a database object. For example, data from a table T1 may be divided into four chunks, which may be respectively loaded into four distinct IMUs.

As explained in the Mirroring Patent, the format of data in an IMU may be different than the format in which the same data is stored on persistent storage. For example, the format used to store data from table T on disk (the "persistent-format") may be row-major, while the format used to store the same data in volatile memory (the "mirror-format") may be column-major. In addition, some or all of the data within an IMU may be compressed. When some or all of the data within an IMU is compressed, the IMU may be referred to as an In-memory Compression Unit ("IMCU").

The amount of data that can be loaded into IMUs is constrained by the amount of available volatile memory. Consequently, to effectively leverage the IMUs, it is necessary to intelligently select which elements (e.g. tables, partitions, columns, etc.) are loaded into the IMUs. The database elements that are selected for loading into IMUs are referred to herein as the "to-be-mirrored elements".

Ideally, at any given time, the volatile memory that is available for storing IMUs will be filled with chunks from the database elements that are currently being accessed the most heavily. Unfortunately, which elements are most heavily accessed changes over time. For example, during a period in which table T1 is being accessed heavily, mirroring chunks of table T1 in volatile memory may achieve the greatest benefit. However, at a later time when table T1 is not being accessed, it may be more beneficial to use the volatile memory that was occupied by the chunks of table T1 to cache chunks of a different table T2 that is being more heavily accessed. Techniques for dynamically changing which database elements are mirrored in volatile memory are described in <CIT>, published as <CIT>.

Regardless of how the system determines the to-be-mirrored elements, it is possible that elements that were previously selected as to-be-mirrored elements are no longer selected as to-be-mirrored elements. Such elements are evicted from volatile memory to free up space for newly-selected to-be-mirrored elements. Once an element has been evicted from volatile memory, subsequent requests for data items in that element must obtain the data items in the persistent-format. Obtaining the data items in the persistent-format may involve reading the data items from disk, or accessing a copy of a disk block that has previously been loaded into a buffer cache. Accessing a mirror copy of the data items, in the mirror format, is no longer possible because the mirror copy has been evicted from volatile memory to make room for the currently selected to-be-mirrored elements.

Thus, using conventional techniques, the data items of an element are typically either available in a mirror format from very fast storage (volatile memory), or only available in the persistent-format (from disk or cache). This all-or-nothing approach to mirroring data is inefficient for systems that have one or more tiers of storage that have performance characteristics between volatile memory and magnetic disks.

<CIT> discloses a technique for accelerating queries using dynamically generated columnar data in a flash cache. In an example, a method comprises a storage device receiving a first request for data that is stored in the storage device in a base major format in one or more primary storage devices. The storage device comprises a cache. The base major format is any one of: a row-major format, a column-major format and a hybrid-columnar format. Based on first one or more criteria, it is determined whether to rewrite the data into rewritten data in a rewritten major format. In response to determining to rewrite the data into rewritten data in a rewritten major format, the storage device rewrites at least a portion of the data into particular rewritten data in the rewritten major format. The rewritten data is stored in the cache.

<CIT> discloses a multi-processor data storage system having a fast storage tier and a slow storage tier. Blocks of data may be transferred from the slow data storage tier to the fast data storage tier, or from the fast data storage tier to the slow data storage tier, as data access patterns change. Furthermore, a multi-processor data storage system may include a tiered cache architecture wherein the system includes a fast cache and a slow cache, wherein a cache stores a subset of data also contained in a persistent data storage such as the fast data storage tier and the slow data storage tier. Blocks of data may be transferred from the slow cache to the fast cache, or from the fast cache to the slow cache, as data access patterns change.

<CIT> discloses techniques for maintaining data persistently in one format, but making that data available to a database server in more than one format. One of the formats in which the data is made available for query processing is based on an on-disk format, such as row-major disk blocks. Another format in which the data is made available for query processing is an in-memory format independent of the on-disk format, such as a column-major format.

The dependent claims concern optional elements of some embodiments of the present invention. For the purpose of determining the extent of protection, due account is to be taken of any element which is equivalent to an element specified in the claims.

Rather than employ an all-or-nothing approach to mirroring data, the techniques described herein involve storing mirror-format data at different tiers of storage. As time passes and it becomes necessary to evict an IMU that stores mirror-format data from a particular chunk from a particular level of storage, the IMU is moved to a lower level tier of storage rather than being deleted entirely. Thus, a copy of the mirror-format IMU continues to exist, but overhead associated with obtaining data from the IMU increases. However, the increased overhead is still significantly less than what would be required to rebuild from scratch, in volatile memory, the mirror-format IMU of the chunk when the corresponding chunk begins to be heavily accessed again.

In addition to moving mirror-format IMUs between levels of storage, techniques are provided for initially pre-loading IMUs into storage levels that are below DRAM. For example, when a load-triggering event occurs to cause an IMU to be initially constructed and loaded, the IMU may be created in both the DRAM level of storage and in a NVRAM level of storage. The DRAM-level copy of the particular IMU may be used until evicted. After that IMU is evicted from DRAM, the NVRAM copy of the IMU remains. The next time the IMU is needed by the database server, the IMU may simply be copied from NVRAM to DRAM. The process of copying the IMU from NVRAM to DRAM is several magnitudes faster than rebuilding the IMU in DRAM based on persistent-format data obtained from disk.

By pre-loading mirror-format IMUs into storage levels below (i.e. slower than) the DRAM level, a significantly higher number of mirror-format IMUs may be maintained within a database system. This is particularly true because such lower tiers of storage can be significantly less expensive than DRAM. Consequently, it is economically feasible for database systems to have significantly more storage available at those lower levels.

Techniques are also provided for pushing the functionality of creating and using IMUs to the storage system. The availability of storage-system-maintained IMUs may significantly increase performance of query processing when database servers push work to the storage system using techniques such as those described in <CIT>.

In particular, the storage system may pre-load chunks of database objects into IMUs in any one of the storage system's own tiers of storage. Database server instances may then communicate higher-level database operations (including but not limited to scan/filter and aggregate operations) to the storage system, and have the storage system perform some or all of the work required by the operations using IMUs loaded in its own memory. For example, assume that the on-disk version of table T1 stores data in row-major format. When table T1 is heavily used, the storage system may preload, into its own volatile memory, an IMU containing values from a column c1, of a table T1, in column-major format. A database server instance may then instruct the storage system to process a filter condition involving column c1. To process the filter condition, the storage system may make use of the IMU, rather than performing the operation based on row-major data read from the on-disk copy of table T1.

When the storage system needs to evict the IMU from the storage system's volatile memory, the storage system may first copy the IMU to a lower-tier of storage within the storage system, such as NVMe FLASH. Consequently, when that mirror-format IMU is needed again, the storage system may simply copy the IMU from its NVMe FLASH to its DRAM, rather than reconstruct the IMU from scratch.

Using the techniques described herein, the larger slower storage tiers may be used to cover against sudden performance degradation in the presence of selective column caching. For example, if a database administrator (DBA) has optimized the DRAM resources of a system by selectively caching hot columns, a query that uses other less frequently used columns can use the columnar data, retrieved temporarily into DRAM, from a slower storage tier.

For Hadoop data, which is typically slow to process due to the Java access layer (SerDe), the benefits of columnar caching of Hadoop splits are even greater. In this case, the columnar cache can be written to the slowest of the tiered storage layer: hard disk. Both because of the typically huge sizes on Hadoop data stores and because the relative performance benefits are still immense.

<FIG> is a block diagram of a database system that includes multiple tiers of storage. Referring to <FIG>, the database system includes two nodes (nodes <NUM> and <NUM>). Node <NUM> has three server-side tiers of storage: DRAM <NUM>, NVRAM <NUM> and NVMe FLASH <NUM>. Similarly, node <NUM> has three server-side tiers of storage: DRAM <NUM>, NVRAM <NUM> and NVMe FLASH <NUM>.

Each of these server-side storage tiers may be used to store IMUs. Specifically, within node <NUM>, DRAM <NUM> is used to store DRAM-LEVEL IMUs <NUM> (IMUs <NUM> and <NUM>), NVRAM <NUM> is used to store NVRAM-level IMUs <NUM> (IMU <NUM>), and NVMe FLASH <NUM> is used to store NVMe-LEVEL IMUs <NUM> (IMU <NUM>). Similarly, within node <NUM>, DRAM <NUM> is used to store DRAM-LEVEL IMUs <NUM> (IMUs <NUM> and <NUM>), NVRAM <NUM> is used to store NVRAM-level IMUs <NUM> (IMU <NUM>), and NVMe FLASH <NUM> is used to store NVMe-LEVEL IMUs <NUM> (IMU <NUM>).

Both of nodes <NUM> and <NUM> have access to storage system <NUM> which includes a disk <NUM> that persistently stores the database that is managed by the database system. In the embodiment illustrated in <FIG>, storage system <NUM> includes four storage-side tiers of storage: DRAM <NUM>, NVRAM <NUM>, NVME FLASH <NUM> and disk <NUM>. Similar to the tiers in nodes <NUM> and <NUM>, the tiers in storage system <NUM> can be used to store IMUs. Specifically, within storage system <NUM>, DRAM <NUM> is used to store DRAM-LEVEL IMUs <NUM> (IMUs <NUM> and <NUM>), NVRAM <NUM> is used to store NVRAM-level IMUs <NUM> (IMU <NUM>), and NVMe FLASH <NUM> is used to store NVMe-LEVEL IMUs <NUM> (IMU <NUM>). Disk <NUM> may also store mirror-format IMUs as shall be described in greater deal hereafter.

Processor <NUM> represents one or more processors that are executing database server instance <NUM> within node <NUM>. Processor <NUM> represents one or more processors that are executing database server instance <NUM> within node <NUM>. Both database server instances <NUM> and <NUM> manage a database stored on disk <NUM> that includes a table T1. Blocks of persistent-format data read from disk <NUM> by database server instance <NUM> may be temporarily stored in buffer cache <NUM>. Similarly, blocks of persistent-format data read from disk <NUM> by database server instance <NUM> may be temporarily stored in buffer cache <NUM>.

Upon evicting an IMU from a storage tier of a device, the IMU may be moved to a lower storage tier in the same device. Thus, evicting an IMU in DRAM <NUM> of node <NUM> may cause the IMU to be moved to another storage tier (NVRAM <NUM> or NVMe FLASH <NUM>) within node <NUM>. Similarly, evicting an IMU in DRAM <NUM> of storage system <NUM> may cause the IMU to be moved to another storage tier (NVRAM <NUM>, NVMe FLASH <NUM> or disk <NUM>) of storage system <NUM>.

For the purpose of explanation, it shall be assumed that the mirror format is column-major, and that data items from table T1 are mirrored in an in-memory unit IMU <NUM> that was initially loaded into DRAM <NUM> of node <NUM>, as illustrated in <FIG>. When IMU <NUM> is to be evicted from DRAM <NUM> to make room for more-heavily-used data, IMU <NUM> may be transferred to a different tier of storage, such as NVRAM <NUM>, which is still faster than magnetic disks. <FIG> is a block diagram illustrating the situation in which IMU <NUM> has been evicted from DRAM <NUM> and copied into NVRAM <NUM> in order to make room for a newly loaded IMU <NUM>.

While IMU <NUM> is stored in NVRAM <NUM>, requests to access the data items from table T1 may cause IMU <NUM> to be copied from NVRAM <NUM> back into DRAM <NUM>. Once back in DRAM <NUM>, IMU <NUM> may be used to process the database request. Copying IMU <NUM> from NVRAM <NUM> into DRAM, and then accessing IMU <NUM> from DRAM <NUM>, may be significantly faster than either reconstructing IMU <NUM> in DRAM <NUM> from scratch, or reading the data items of table T1 in row-major format from magnetic disk (or from a cached copy of the corresponding disk blocks).

According to one embodiment, copying IMU <NUM> from NVRAM <NUM> into DRAM <NUM> does not cause IMU <NUM> to be removed from NVRAM <NUM>. Thus, when IMU <NUM> is again evicted from DRAM <NUM>, IMU <NUM> need not be copied again into NVRAM <NUM>, since a copy is IMU <NUM> still resides in NVRAM <NUM>.

If, at a later point, IMU <NUM> is to be evicted from NVRAM <NUM> to make room in NVRAM <NUM> for a more heavily-used IMU, IMU <NUM> may be moved to a slower storage tier (e.g. NVMe FLASH <NUM>). <FIG> is a block diagram illustrating the situation in which IMU <NUM> has been evicted from NVRAM <NUM> and copied into NVMe FLASH <NUM> in order to make room for an IMU <NUM>.

While IMU <NUM> is stored in NVMe FLASH <NUM>, requests to access the data items from table T1 may cause IMU <NUM> to be copied from NVMe FLASH <NUM> into DRAM <NUM>. Once in DRAM <NUM>, IMU <NUM> may be used to process the database request. Copying IMU <NUM> from NVMe FLASH <NUM> into DRAM, and then accessing IMU <NUM> from DRAM <NUM>, may be significantly faster than either reconstructing IMU <NUM> in DRAM <NUM> from scratch, or reading the data items from table T1 in row-major format from magnetic disk (or from a cached copy of the corresponding disk blocks).

Finally, if IMU <NUM> is to be evicted from NVMe FLASH <NUM> to make room in NVMe FLASH <NUM> for a more heavily-used IMU, IMU <NUM> may simply be deleted/overwritten, forcing future requests to access data items from table T1 to read the data in row-major format from table T1 on magnetic disk <NUM> (or from a cached copy of the corresponding persistent-format disk blocks in buffer cache <NUM>).

Alternatively, upon eviction from NVMe FLASH, IMU <NUM> may be transferred to the same storage tier on which the row-major data (table T1) resides (e.g. magnetic disk <NUM>). <FIG> is a block diagram illustrating the situation in which IMU <NUM> has been evicted from NVMe FLASH <NUM> and copied to disk <NUM> in order to make room for an IMU <NUM>.

When, as shown in <FIG>, both the row-major and mirror-format forms of the same data are on the same tier, no improvement in storage access speed will be realized. However, for certain types of operations, using the mirror-format data may result in more efficient query processing. For example, in cases where a query involves only a single column (or a column group) of a table that contains hundreds of columns, retrieving from disk <NUM> an IMU that contains the column vector for only that column (or column group) may lead to significantly faster query processing, even though that IMU is stored on the same tier of storage as the row-major data for that table. Thus, in response to a query, database system may load the IMU <NUM> from disk <NUM> into DRAM <NUM>, and use the IMU <NUM> to process the query. Because IMU <NUM> need not be reconstructed from scratch, and IMU <NUM> is in mirror-format, many queries may be processed faster than if a copy of IMU <NUM> were not stored to disk <NUM> when evicted from NVMe FLASH <NUM>.

Thus, this new approach takes the cache of IMUs across multiple storage types, with increasing resource availability but successively slower performance:
DRAM -> NVRAM -> NVMe FLASH -> disk.

In the example given above, IMU <NUM> is gradually migrated down the various storage tiers to make room for more heavily accessed data. However, at any point, the access frequency of the data in IMU <NUM> may increase to the point where the database system determines that IMU <NUM> should move up in the storage hierarchy. For example, while stored on the NVME FLASH <NUM>, the access frequency of IMU <NUM> may increase to the point where the database server instance <NUM> determines that IMU <NUM> should move up to NVRAM <NUM>. Moving IMU <NUM> to NVRAM <NUM> may have the consequence of evicting a different IMU (e.g. IMU <NUM>) from NVRAM <NUM>. The evicted IMU <NUM> may then be moved down to NVMe FLASH <NUM>.

In the examples given above, IMUs move up one level at a time in the storage tiers, or down one level at a time in the storage tiers. However, the frequency of access of an IMU may change such that an IMU jumps up or down multiple tiers at a time. For example, data that has previously been accessed so rarely that it is not mirrored at all may start being heavily accessed. Under these circumstances, upon creating the IMU in volatile memory, rather than copy the IMU to the NVMe FLASH tier, the IMU may be copied to the NVRAM tier (thereby leapfrogging the NVMe FLASH tier).

In a similar manner, the access frequency of an IMU in volatile memory may decrease such that, upon eviction from DRAM <NUM>, the IMU is moved directly to the NVMe FLASH tier (thereby leapfrogging the NVRAM tier), to disk, or deleted without being moved anywhere (leaving only the corresponding on-disk row-major data).

In the examples described above, IMU <NUM> is moved down the storage tiers of node <NUM> to make room for more frequently accessed data, and moves up the storage tiers of node <NUM> as its access frequency increases. In a similar manner, IMUs may move between the storage-side tiers of storage system <NUM>. Specifically, IMUs evicted from faster storage may move to slower storage, and IMUs in slower storage may be moved to faster storage in response to increased access frequency.

Because the techniques herein allow an IMU to be present on any one of multiple tiers of storage, a request for a data item may trigger a search for the corresponding IMU. Specifically, in one embodiment, when a table or partition is marked for columnar caching and the table scan driver doesn't find in an IMU for the data in the DRAM cache, it checks the presence of the IMU in the NVRAM cache and, if not there, then in the NVMe FLASH cache. If found in any one of those layers, the IMU is copied from that location into DRAM. This allows the same columnar formats and the same columnar optimizations to be applied to the data without having to constantly maintain the IMU in DRAM, giving a smoother fall off in performance than the all-or-nothing mirror approach. If the data is not found in the columnar cache and it is marked for in-memory caching, then the chunk is read from disk, reformatted into mirror-format and written to a tier of storage.

According to an embodiment, a single load-triggering event may cause creation of the same IMU, containing data in mirror-format, in multiple tiers of storage. Specifically, at the time an IMU is initially built in DRAM in response to a load-triggering event, a copy of the IMU may be created in one or more of the other storage tiers. The other storage tier(s) in which a copy of the IMU is created may be (a) indicated in user-generated metadata, or (b) decided automatically by the database server based on various factors including usage statistics. For example, user-specified metadata associated with a particular database object may indicate that the object is NVRAM-enabled. Under these circumstances, when an IMU is built in DRAM for data from that object, a copy of the IMU is also created in NVRAM. Similarly, marking a database object as NVMe FLASH-enabled may cause IMUs that contain data from the database object to be created in both DRAM and NVMe FLASH.

If a database object is in-memory enabled without having a specified storage tier, the database server may automatically decide which tier is appropriate based on various factors, including access statistics. For example, the database server may cause an IMU containing less-used data to be created in both DRAM and NVMe FLASH, and cause an IMU containing frequently-used data to be crated in both DRAM and NVRAM. Newly created IMUs containing the most-frequently-used data may simply be maintained in DRAM until evicted. Upon eviction from DRAM, such IMUs may be copied to lower tiers of storage, as explained above.

The examples given above involve a system with three tiers of storage (DRAM, NVRAM, NVMe FLASH) on the server-side, and four tiers of storage (DRAM, NVRAM, NVMe FLASH, Magnetic Disk) on the storage-side. However, the techniques described herein may be applied in any system that has at least two tiers of storage. For example, even in a system that has only DRAM and Magnetic Disk storage, IMUs that are evicted from RAM may be moved to Magnetic Disk, rather than simply deleted. Similarly, upon creation, copies of an IMU may be created both in DRAM and on disk.

<FIG> is a block diagram illustrating a system that has only one server-side tier of storage. According to one embodiment, database objects may be marked in-server-memory or in-storage-memory. When an object is marked "in-server-memory", a load-triggering event in node <NUM> will cause an IMU containing data from the object to be created only in DRAM <NUM>. On the other hand, when an object is marked "in-storage-memory", a load-triggering event in node <NUM> will cause an IMU containing data from the object to be created in DRAM <NUM> of storage system <NUM>. In addition to being created in DRAM <NUM>, the load-triggering event may also cause a copy of the IMU to be created one of the other tiers within storage system <NUM>.

For example, the metadata associated with table T1 may indicate "in-storage-memory" and "NVRAM-level". Based on this metadata, a load-triggering event for table T1 may cause an IMU with data from table T1 to be created in both DRAM <NUM> and NVRAM <NUM>. Similarly, if the metadata associated with table T1 indicated "in-storage-memory" and "NVMe-level", a load-triggering event for table T1 would cause an IMU with data from table T1 to be created in both DRAM <NUM> and NVRAM <NUM>.

As is evident with the examples given above, the metadata associated with an object may indicate both (a) which device (server or storage) an IMU is to be created in, and (b) the tier(s) in which the IMU is to be created. The lower-tiered copies of an IMU remain even when a higher-tiered copy of the IMU is evicted. As a result, recreation of the IMU at the DRAM level merely involves copying the IMU from the lower-level storage, rather than recreation of the IMU from scratch.

Using the techniques described herein, IMUs may be made directly accessible to the processing units of storage systems. Consequently, the functionality of creating and using IMUs to perform database operations is also pushed to the storage system. For example, when the task of applying a filter to a set of data is pushed to the storage system, as described in <CIT>, storage system <NUM> may perform that task using mirror-format data stored in an IMU in DRAM <NUM>. If the IMU needed to perform the task is not in DRAM <NUM>, but is at another storage tire within storage system <NUM>, then storage system <NUM> copies the IMU from the storage tier in which it resides to DRAM <NUM>, and then uses the IMU to perform the requested task. Applying a filter is merely one example of a task for which storage system <NUM> make use of an IMU in DRAM <NUM>. However, any task for which a database server instance may make use of mirror-format data may be pushed to storage system <NUM>, which now has direct access to mirror-format data in the IMUs stored in any of its storage tiers.

Further, when a database server asks storage system <NUM> to perform a task that can be accomplished more efficiently with mirror-format data, storage system <NUM> may itself create the needed IMU if the IMU does not already exist in any of the storage-side storage tiers. As explained above, when creating the IMU in DRAM <NUM>, storage system <NUM> may simultaneously create the IMU on one or more of the other storage-side storage tiers. Thus, even though the IMU may be subsequently evicted from DRAM <NUM>, the IMU may be quickly copied back into DRAM <NUM> when needed in the future.

<FIG> is a flowchart illustrating steps performed by a computer system in response to receiving a query, according to an embodiment. Referring to <FIG>, at step <NUM> the system receives a database query. For the purpose of explanation, it shall be assumed that database server instance <NUM> receives a query that requires retrieving the rows of table T1 where column c1 has the value "fred". At step <NUM>, the query optimizer of database server instance <NUM> determines that the filtering required by the query can be handled most efficiently by using an IMU that contains a column vector for column c1.

At step <NUM>, database server instance <NUM> determines whether the IMU is already loaded (either in node <NUM> or in storage system <NUM>). If the IMU already loaded, then control proceeds to step <NUM> to determine which device has the IMU. If the IMU is loaded on the server (i.e. node <NUM>), then control passes to step <NUM>. At step <NUM>, it is determined whether the IMU is at the DRAM tier of the server-side storage. If so, then at step <NUM> the database server instance <NUM> processes the query using the IMU in DRAM <NUM>.

If, at step <NUM>, it is determined that the IMU containing the column vector for column c1 is not currently in DRAM, then at step <NUM> the IMU is copied into DRAM <NUM> from the storage tier in which the IMU resides. Then, at step <NUM>, the database server instance <NUM> processes the query using the IMU in DRAM <NUM>.

If, at step <NUM>, it is determined that the needed IMU is loaded in storage system <NUM> and not in node <NUM>, then control passes to step <NUM>. At step <NUM>, it is determined whether the IMU is at the DRAM tier of the storage-side storage. If so, then at step <NUM> the task of performing the filter operation is pushed to storage system <NUM>, and in step <NUM> the storage system <NUM> performs the filter operation using the IMU in DRAM <NUM>.

If, at step <NUM>, it is determined that the IMU containing the column vector for column c1 is not currently in the storage-side DRAM, then at step <NUM> the IMU is copied into DRAM <NUM> from the storage-side storage tier in which the IMU resides. Then, at step <NUM>, the task of performing the filter operation is pushed to storage system <NUM>, and in step <NUM> the storage system <NUM> performs the filter operation using the IMU in DRAM <NUM>.

If, at step <NUM>, it is determined that the IMU is not loaded in either node <NUM> nor within storage system <NUM>, control passes to step <NUM>. At step <NUM>, it is determined where the IMU should be loaded. This determination may be based on a variety of factors. For example, as mentioned above, metadata associated with table T1 may indicate whether the IMU is to be loaded on the server-side or the storage side. Alternatively, the database server instance <NUM> may determine which device should construct and load the IMU based on factors such as:.

As an example, if node <NUM> frequently accesses column c1, then the database server may decide that the IMU containing the column vector for c1 should be constructed and used within storage system <NUM>. Since both nodes <NUM> and <NUM> access data through storage system <NUM>, both nodes would benefit from having the storage system <NUM> construct and use the IMU. On the other hand, if table T1 is used exclusively by node <NUM>, and node <NUM> has a large pool of DRAM, then the database server may decide to construct the IMU in DRAM <NUM> of node <NUM>.

If it is determined that the IMU is to be loaded on the server side, control passes to step <NUM>. At step <NUM>, the "target tiers" of node <NUM> are determined. The target tiers are the tiers in which to construct the IMU. In some cases, DRAM <NUM> may be the only target tier. In other cases, the target tiers may include DRAM <NUM> and one of the other storage tiers of node <NUM>. For example, if the metadata associated with table T1 indicates that the IMU for column c1 is NVRAM-enabled, then the target tiers are both DRAM <NUM> and NVRAM <NUM>. At step <NUM>, the IMU is constructed in the target tiers. After the IMU has been constructed, in step <NUM> the server processes the query using the IMU.

Steps <NUM> and <NUM> are similar to step <NUM> and <NUM> except that the target tiers are storage-side storage tiers. Once the IMU has been created in the storage-side target tiers, the task is pushed to the storage system (step <NUM>), and the storage system performs the task using the IMU that was created in the storage-side DRAM. After performing the task, the storage system may return results of the task to the server in columnar format. The server can cache those results as an IMU within the server machine, and use that IMU to process subsequent queries.

Claim 1:
A method comprising:
storing, within a storage system (<NUM>) that is operatively coupled to a server machine that is executing a database server instance (<NUM>, <NUM>) that manages a database:
a persistent-format copy of a set of data, in a persistent-format, on persistent storage of the storage system (<NUM>); and
a mirror-format copy of the set of data, in a mirror-format, in volatile memory of the storage system (<NUM>);
wherein the storage system (<NUM>) is separate from the server machine;
wherein the set of data is from a table in the database managed by the database server instance (<NUM>, <NUM>);
wherein the mirror-format is different than the persistent-format;
in response to a query, received by the database server instance (<NUM>, <NUM>), that requires access to the set of data, performing the steps of:
causing the database server instance (<NUM>, <NUM>) to communicate a task to the storage system (<NUM>); and
the storage system (<NUM>) performing the task using the mirror-format copy;
in response to the mirror-format copy being evicted from the volatile memory of the storage system (<NUM>), the storage system (<NUM>) copying the mirror-format copy from the volatile memory to another storage tier of the storage system (<NUM>), wherein the other storage tier has slower access rates than the volatile memory and faster access rates than the persistent storage, thus enabling the mirror-format copy to be accessed with less overhead than would be required to rebuild the mirror-format copy from scratch in the volatile memory based on the persistent-format copy.