Enhancing data retrieval performance in deduplication systems

Various embodiments for processing data in a data deduplication system are provided. For data segments previously deduplicated by the data deduplication system, a supplemental hot-read link is established for those of the data segments determined to be read on at least one of a frequent and recently used basis. Other system and computer program product embodiments are disclosed and provide related advantages.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates in general to computers, and more particularly to a system and computer program product for enhancing data storage and retrieval performance in computing storage environments having integrated data deduplication systems.

Description of the Related Art

Computers and computer systems are found in a variety of settings in today's society. Computing environments and networks may be found at home, at work, at school, in government, and in other settings. Computing environments increasingly store data in one or more storage environments, which in many cases are remote from the local interface presented to a user.

These computing storage environments may use many storage devices such as disk drives, often working in concert, to store, retrieve, and update a large body of data, which may then be provided to a host computer requesting or sending the data. In some cases, a number of data storage subsystems are collectively managed as a single data storage system. These subsystems may be managed by host “sysplex” (system complex) configurations that combine several processing units or clusters of processing units. In this way, multi-tiered/multi-system computing environments, often including a variety of types of storage devices, may be used to organize and process large quantities of data.

SUMMARY OF THE INVENTION

Many multi-tiered/multi-system computing environments implement data deduplication technologies to improve storage performance by reducing the amount of duplicated storage across storage devices. Data deduplication systems are increasingly utilized because they help reduce the total amount of physical storage that is required to store data. This reduction is accomplished by ensuring that duplicate data is not stored multiple times. Instead, for example, if a chunk of data matches with an already stored chunk of data, a pointer to the original data is stored in the virtual storage map instead of allocating new physical storage space for the new chunk of data.

Conventional data deduplication methodologies focus on the data write process (i.e., the process by which data segments are written through the data deduplication system after being analyzed for deduplication with a previously existing file). These systems discretize the data into data chunks, which are deduplicated via a HASH algorithm. If a given data chunk is determined to be a duplicate, as identified by the HASH digest, the duplicate chunk is currently replaced by a link to the primary chunk.

While implementing data deduplication functionality in computing storage environments have resulted in significant gains in data storage efficiency, conventional data deduplication functionality is not yet exploited to its full potential to enhance data storage and retrieval performance in computing environments. Improvements to the mechanisms by which data is stored and retrieved through such data deduplication systems are beneficial and highly desirable.

In view of the foregoing, various embodiments for improving data storage and retrieval performance in a computing environment incorporating a data deduplication system are provided. In one embodiment, by way of example only, a method for processing data in a data deduplication system by a processor is provided. For data segments previously deduplicated by the data deduplication system, a supplemental hot-read link is established for those of the data segments determined to be read on at least one of a frequent and recently used basis.

In addition to the foregoing exemplary embodiment, various system and computer program embodiments are provided and supply related advantages.

DETAILED DESCRIPTION OF THE DRAWINGS

As previously indicated, current deduplication systems tend to focus on the write process of data to minimize the number of writes of duplicate data to storage. In contrast, the illustrated embodiments focus on the reading of previously-deduplicated files, in conjunction with data deduplication functionality, to enhance data retrieval and improve performance in computing storage environments.

To further illustrate, consider the following exemplary embodiment. In such embodiment, a prior determination is made as to whether certain files are Most Frequently Used (MFU). In other words, these files are determined to have a substantially higher frequency of read activity in the storage environment. These MFU files may be stored in cache and accessed via a hot-read link, which supplements, but does not replace, a primary link. As such, if a subsequent read request for one of these MFU files occurs, the supplemental hot-read link may be used to quickly access the data segment in cache, rather than repetitively request the same file from disk or tape storage, for example. Such a mechanism lessens network traffic and improves storage performance.

The illustrated embodiments may implement a table of HASH digests to assist in determining which of the data segments in a given storage are called upon on a frequent or recently used basis, for example. Pursuant to each read operation involving data segments so identified, their corresponding HASH entries are incremented. An associated primary link and designated supplemental hot-read link may also be configured in this HASH table.

In using the mechanisms of the illustrated embodiments, and as one of ordinary skill in the art will appreciate, the same chunk of data can be used in multiple files. When these files are being read, the mechanisms of the illustrated embodiments serve to alleviate the necessity of obtaining the same chunk of data from potentially deep down the storage ladder. If certain HASH digests stand out as, for example, MFU digests during ensuing read processes, then those chunks are stored locally, in cache, rather than in remote storage, such as hard drives or tape. This way, the performance of the recombination of chunks into files during the read process is improved, as the MFU chunks are handily convenient. In an alternative embodiment, the Most Recently Used (MRU) chunks may be tracked. A running window on the tally may be employed, so that stale chunks are not retained in cache memory.

In the following description, reference is made to the accompanying drawings which form a part hereof and which illustrate several embodiments of the present invention. It is understood that other embodiments may be utilized and structural and operational changes may be made without departing from the scope of the present invention.FIG. 1illustrates a computing storage environment in which aspects of the invention may be implemented. A plurality of host systems2a, b . . . ntransmit Input/Output (I/O) requests to one or more storage volumes28,30, and32through a storage controller6which manages access to the storage volumes28,30, and32. In certain implementations, the storage volumes may be physically comprised of a plurality of hard disk drives organized as Just a Bunch of disks (JBOD), a RAID array, Direct Access Storage Devices (DASD), SSD, tape devices, etc.

A number of virtual volumes22,24, and26are presented to the host systems2a, b . . . nin lieu of presenting a number of physical or logical volumes (often which may be physically configured in a complex relationship). The host systems2a, b . . . nmay communicate with the storage controller6over a network8, such as the Internet, a Storage Area Network (SAN), an Intranet, Local Area Network (LAN), Wide Area Network (WAN), etc., using multiple communication protocols such as TCP/IP, Fibre Channel, Ethernet, etc. at different layers in a protocol stack.

The storage controller6includes a processor10executing code12to perform storage controller operations. The storage controller6further includes a cache14and non-volatile storage unit16, such as a battery backed-up memory device. The storage controller6stores in cache14data updates received from the hosts2a, b . . . nto write to the virtual storage volumes22,24, and26(and thereby to volumes28,30, and32) as well as data read from the volumes28,30, and32to return to the hosts2a, b . . . n. When operating in Fast Write mode, data updates received from the hosts2a, b . . . nare copied to both cache14and the NVS16. End status is returned to the host2a, b . . . nsending the data update after the update is copied to both the cache14and NVS16.

Storage controller6also includes a data deduplication engine17in communication with a storage management module18as will be further described. Data deduplication engine17is configured for performing, in conjunction with processor10, data deduplication operations on write data passed through storage controller6to virtual volumes20and volumes28,30, and32.

Cache system14accepts write data from hosts2a, b . . . nor similar devices, that is then placed in cache memory. Data deduplication engine17then tests the write data for duplication in the cache memory14before the write data (should the data be determined necessary to be written) is ultimately written to storage28,30, and32. In the illustrated embodiment, cache14includes Most Frequently Used (MFU) data chunks15, as shown. The MFU chunks15are representative of most frequently read, previously deduplicated files, which are deposited in cache for quick retrieval in lieu of reading from hardened storage. As previously described, these MFU chunks have been determined to be frequently read; as such this data is retained in cache versus a hardened storage location.

FIG. 1, as one of ordinary skill in the art will appreciate, may illustrate a portion of a larger, multi-system/multi-cluster storage environment having a number of interrelated components such as the previously illustrated storage controller6. As previously indicated, while virtual volumes22,24, and26are presented to the user via the host systems2a, b. . . n, the underlying physical configuration may take many possible forms. For example, a number of interrelated storage devices in various classes, such as SSD, HDD, tape, optical disk such as DVD or Blu-Ray, etc. may comprise the storage volumes28,30, and32depending on a particular configuration.

Various components of the storage environment, such as processor10, may be adapted to implement aspects of the present invention and following claimed subject matter. For example, a hash digests table18may operate in conjunction with processor10to perform various functionality to be further described, such as providing information to determine which of the read data chunks are read most frequently and/or read most recently. One of ordinary skill in the art will appreciate that other various data processing and memory components may be implemented to realize these aspects, and may be operational on the storage controller6, or elsewhere. Cache14, along with other components, may further comprise a variety of additional modules as will be further described to implement various portions of functionality. For example, in one embodiment, the cache14may further comprise modules for I/O monitoring, and a data placement module for storing MFU data chunks in the cache14, for example. Such modules are not illustrated for purposes of convenience but would be understood to someone of ordinary skill in the art.

Turning toFIG. 2, an exemplary table100of HASH digests are depicted, in which aspects of the present invention may be implemented as previously described, for example exemplary table100is one embodiment of HASH digests table18inFIG. 1. HASH digests102are listed in succession for each read chunk of previously deduplicated files as shown. When the HASH digests102are used in a subsequent read operation to re-aggregate chunks into a requested file, for example, the tally104is increased for the used HASH digest. The first two entries of tally104in the table100(i.e.,20,19) indicate that those hash digests are used much more frequently than the last two entries (i.e.,3,2). In one embodiment, each HASH digest102used in deduplication has an associated primary link106as depicted. In addition, those HASH digests102identified/determined as MFU chunks have a supplemental hot-read link108(i.e., those digests having tallies20and19, respectively). In an additional embodiment, the table100of HASH digests may include those data segments identified as MFU chunks and MRU (Most Recently Used) chunks, or alternatively, MRU chunks individually. The MRU determination may be used in an alternative embodiment to identify data chunks to deposit in the cache. An alternative embodiment comprises gathering hash digests in column102specific to a desired application, such as financially-related chunks when it is time to process a payroll, image-related chunks when it is time to read medical X-rays, and mpeg-related chunks when it is time to process video-streams.

Here again, the mechanisms of the illustrated embodiments take advantage of the fact that the same chunk of data may be used in multiple files, and when these files are being read, it is much more efficient to obtain the same chunk from a readily accessible location rather than from hardened storage deep in the storage ladder (again, such as tape devices, which have high latency). A running window for tally104, the mechanisms therewith to be further described, may be employed such that stale chunks are not retained in the cache. Such a running window may vary in duration by hour, work-shift, day, week, or by the current application being run. By application being run, it is meant that when a change is made from a spreadsheet processing application to video processing application, the chunks read by the former application are unlikely to be needed by the new application and it is time to flush the chunk read-cache.

InFIG. 3, following, a flow chart of an exemplary read chunk process200, again in which aspects of the illustrated embodiments may be incorporated, is shown. The process200begins by accessing the hash digest table100(FIG. 2, previously) (step204). Then the read-tally104for digest102is incremented (step206). In a following step, the read-chunk process then queries whether a hot-read link108is available (step208). If so, the process flows to step210, where the desired chunk is quickly read via the hot-read link108. Alternatively, if a hot-read link108is not available, as may be indicated, in one embodiment, by an error code such as NA (Not Available), or some other code (e.g., hexadecimal FFFF), then the process200flows to step212, where the desired chunk is read via its primary link106deeper in the storage ladder, such as from a tape library.

It may be noted that any data chunk may always be read from its primary link106; however, in the interest of performance, hot chunks have a supplemental hot-read link108, which enables them to be retrieved faster in order to improve performance and mitigate unnecessary network traffic.

Turning now toFIG. 4, a flow chart of an exemplary cleanup table process300for HASH table100(FIG. 2) is illustrated. Process300begins (step302) by indicating the periodic cleanup access to HASH table100(step304). As a following step, the query is made whether each tally104for HASH digest102exceeds a user or system-defined threshold (step306). A system-defined threshold refers to, in one embodiment, where there is only so much cache, and the threshold is adjusted to prevent overflow of that cache.

If the answer to the query is affirmative in step306, a hot-read link108is created in step308, if one does not already exist. In step310, an associated copy of that chunk is placed in cache without removing the chunk from its primary location, if the chunk is not already there. If the answer to the query is negative (again, step306), the hot-read link108is removed (step312) if it has not already been removed, and replaced by an error “do not use” code such as NA or FFFF as previously indicated. Next, in step314, that copy of the chunk is removed from cache, unless it is already removed, and future access to the chunk is from its primary location in the storage ladder. Both steps310and312flow to optional step316, where tally104is then reset to zero. Step316allows the use of process300on a periodic basis, be it hourly, daily, weekly, monthly, or due to change in application being run, to clean out the cache and prevent previous high-tally chunks, which are now unread and stale, from dominating new and upcoming chunks which are actively being read.

As one of ordinary skill in the art will appreciate, the various steps in processes200and300may be varied to suit a particular application. Here again, various considerations may be undertaken to obtain a specific data chunk, such as the MFU and MRU considerations previously described, and may be combined with other factors related to I/O activity or other storage characteristics that would be apparent to one of ordinary skill in the art.

While one or more embodiments of the present invention have been illustrated in detail, one of ordinary skill in the art will appreciate that modifications and adaptations to those embodiments may be made without departing from the scope of the present invention as set forth in the following claims.