Automation of deduplication storage capacity sizing and trending analysis

A method for automation of deduplication storage capacity sizing and trending analysis is provided. The method includes collecting all file system directories of at least one system for which a deduplication backup storage capacity for files in the all file system directories is to be determined. The method includes determining file counts, file sizes and file types of the files in the all file system directories and obtaining a deduplication ratio of each of the file types. The method includes deriving the deduplication backup storage capacity from the file counts, the file sizes and the file types of the files in the all file system directories, based on the deduplication ratio of each of the file types.

BACKGROUND

Deduplication reduces the amount of storage needed for backup of data in a client system. It is vital that deduplication backup storage be available in sufficient amounts to support ongoing backup regimens, file size and file count growth in a system, unexpected or unusually large files, equipment failure and data retention needs. It is difficult to size a new deduplication backup system, and it is also difficult to estimate when the capacity of an existing deduplication backup system will run out. Deduplication capacity is not linearly proportional to the amount of data being backed up, since relative amounts of data reduction in deduplication may vary considerably. Often, deduplication storage capacity is manually estimated for systems. One known estimating tool, the EMC Avamar™ CATTOOL, applies a modified client and runs an actual or simulated deduplication against some fraction of the total data on a customer system as a sample. The tool produces a log file that can then be used to determine the data commonality or deduplication ratio of this sample. Accurate use of this tool relies on customers identifying representative data, which they may or may not do correctly, and which is time-consuming for the customers. Consequences for inaccurately predicting or allocating deduplication storage capacity, or failing to arrange for a timely upgrade of such capacity, can include system downtime.

SUMMARY

In some embodiments a method for automation of deduplication storage capacity sizing and trending analysis is provided. The method includes collecting all file system directories of at least one system for which a deduplication backup storage capacity for files in the all file system directories is to be determined. The method includes determining file counts, file sizes and file types of the files in the all file system directories and obtaining a deduplication ratio of each of the file types. The method includes deriving the deduplication backup storage capacity from the file counts, the file sizes and the file types of the files in the all file system directories, based on a typical or averaged deduplication ratio of each of the file types in some embodiments, wherein at least one action of the method is performed by a processor. The embodiments may be implemented as a system and code on a computer readable medium.

DETAILED DESCRIPTION

A deduplication storage capacity analysis system analyzes file system directories of a system (or systems), and determines the deduplication backup storage capacity for deduplicated backup of the system(s). Derivation of the deduplication backup storage capacity is based on file counts, file sizes, file types of the files to be backed up, and may be further based on deduplication ratios for each file type in some embodiments. Deduplication ratios can be obtained from lookup tables, or from executing a deduplication algorithm on subsets of files for each of the file types. By applying analysis down to a granularity of file types, the analysis system can more accurately predict deduplication storage capacity sizing and analyze trends than can existing manual and tool-based methods. The analysis system can also project capacity utilization, determine a capacity upgrade date, and calculate disaster recovery time from a deduplicated backup. Various embodiments of the analysis system, and related method, are described below, along with deduplication scenarios.

FIG. 1is a system diagram depicting a file server106sending files112to a deduplication server104for deduplicated backup, with a deduplication storage capacity analysis system102analyzing deduplication backup storage capacity, capacity utilization, and disaster recovery time. In the deduplication scenario depicted inFIG. 1, the file server106consults one or more directories of the file system128, and reads each file112from a storage memory108. The file server106sends each file112to the deduplication server104via a network122coupling the file server106and the deduplication server104. Next, the deduplication server104separates the files112into data portions114, storing some of the data portions114in a storage memory110(usually, dedicated to deduplication backup storage), which is coupled to the deduplication server104by a network124. Some of the data portions116are duplicate data portions116, and are discarded by the deduplication server104, which shown symbolically as discarding the duplicate data portions116into a trashcan126. It should be appreciated that discarding refers to the duplicate data portions116not being written into the deduplication backup storage memory110. Generally, the deduplication process employs data fingerprints, and the deduplication server104performs a fingerprint operation on each of the data portions114,116, discarding duplicate data portions116which match previously stored fingerprints. In this manner, the deduplication server104performs a deduplicated backup of the files in a client system. The above description of deduplication is for purposes of illustration only, and is not meant to limit embodiments of the deduplication storage capacity analysis system102.

As a result of deduplication, less deduplication backup storage memory110is needed than if all of the data portions114,116were stored in a backup storage memory. Various statistics can be applied in deduplication scenarios to determine a deduplication ratio118. For example, a deduplication ratio118of an entire deduplication backup run can be deduced by taking the amount of data that was sourced from the originating system before deduplication, and dividing this amount by the amount of deduplicated data stored, after deduplication. It should be appreciated that this is somewhat analogous to a data compression ratio, which is calculated for data compression. An estimating tool may perform an actual or simulated deduplication on a sample that is a fraction of the total data on a customer system, for which a deduplication ratio118can be determined relative to the sample. However, these tools are time consuming and may be inaccurate.

The deduplication storage capacity analysis system102determines the deduplication backup storage capacity for deduplicated backup of the system as described in more detail below with reference toFIG. 2. The deduplication storage capacity analysis system102is coupled to the file server by the network122, or by a further network, bus connection or other coupling. In various embodiments, the analysis system102could be in a differing server or other computing device as distinct from the file server106and the deduplication server104, or could be combined into the file server106or the deduplication server104. In one embodiment, the deduplication storage capacity analysis system102is included in a converged infrastructure environment, such as supported by a Vblock® or other virtual computing device, and is applied to determining deduplication storage capacity for backing up virtual machines and/or virtual appliances. In order to analyze the deduplication storage capacity sizing and trending, the analysis system102accesses directories of the file system128, e.g., via the network122and the file server106.

FIG. 2is a block diagram of contents of a deduplication storage capacity analysis system102, in one embodiment. Various modules204,206,210,212can be implemented as software executing on a processor202, hardware, or firmware, or various combinations thereof in the analysis system102. The processor202, the various modules204,206,210,212and a data structure208are coupled by a bus218or other communication coupling. The communication module204can be a network communication device, a bus communication device or other mechanism for coupling to further devices and communicating therewith. The data structure208can be in various formats as readily devised in keeping with the teachings herein.

Still referring toFIG. 2, the deduplication analysis module206collects file system directories for every system to be protected by deduplication backup, and for which a deduplication backup storage capacity is to be determined. For example, the file system128of the file server106could be accessed via a network122, and a copy of the file system directory or directories imported into the analysis system102, and stored in the data structure208. This process can be repeated for further systems and further directories, to collect file system directories as a set representing all of the files that will be backed up in a deduplication backup run for a particular client system. Next, the deduplication analysis module206determines file counts, file sizes, and file types for all of the files in however many directories are in the above-described set of directories. For example, the deduplication analysis module206could determine that there are no files of a first file type, a number of files of a second file type, with particular file sizes, and so on. By organizing information from the file system directories in terms of file types, the deduplication analysis module206is able to size the deduplication storage capacity (i.e., estimate the amount of deduplication storage needed) in a manner that takes into account the differences in deduplication between differing file types.

For example, video files, image files and audio files, and other compressed files tend to not have much deduplication. That is, the data in these file types has very low amounts of duplicate data, and the deduplication process does not discard much duplicate data portions116. Video files and audio files thus have low deduplication ratios118(i.e., larger than but close to one). That is, the amount of deduplication backup storage memory110for these types of files is relatively close to the amount of storage memory108consumed at the source for these files. It should be appreciated that any compressed file generally has a low deduplication ratio118, as compression tends to eliminate duplication. Text files tend to have higher deduplication ratios118, due to the duplication of words, phrases or letter sequences in text files. Email systems often have even higher deduplication ratios, as a result of copies of emails and copies of attachments. Database files generally do not deduplicate well, and tend to have low deduplication ratios118. However, variations exist, and some database file types have higher deduplication ratios118than others. Application logs, many of which are text based and have repetitive entries, tend to deduplicate well and have higher deduplication ratios118. Application data can vary as to deduplication tendencies according to the application. For example, MICROSOFT OFFICE™ data and spreadsheet data may have higher deduplication ratios118as compared to some types of applications and associated data, and so on. It may be worthwhile to separate application data by file type according to applications of origin. The above description is not meant to be limiting but rather an example of the different types of deduplication tendencies for different file types.

Continuing with operations performed by the analysis system102ofFIG. 2, the deduplication analysis module206obtains deduplication ratios118for each file type. For example, the deduplication analysis module206obtains a deduplication ratio118for text files, or even a first deduplication ratio118for a first file type of text files and a second deduplication ratio118for a second file type of text files, and a third deduplication ratio118for video files or for a third file type of video files, and so on. In some embodiments, the deduplication analysis module206looks up deduplication ratios for each file type, and stores this information in the data structure208. Such information could come from industry averages of deduplication ratios for various file types, e.g., from an internal or external data source. Alternatively, such information could come from ongoing analysis of client data and previous deduplication backup runs performed by the deduplication server104, e.g., as obtained from communication with the deduplication server104via the communication module204. These embodiments can omit the selector module210and the deduplication algorithm module212.

In further embodiments, the deduplication analysis module206ofFIG. 2obtains the deduplication ratio118for each file type by performing a deduplication algorithm on a subset of data from each file type. In this embodiment, the selector module210auto-selects a subset of files from each file type, by referencing the file system directories earlier collected. For example, the selector module210could select a number of text files, a number of audio files, a number of database files, and a number of application data files (or even a number of application data files from a first application, a further number of application data files from a second application and so on). The selection may be system dependent, and the number of files of each file type to select could be a predetermined number (which could be the same or different for different file types), a predetermined fraction of the files of each type, or could be based on a predetermined amount of data for each file type, etc. Then, the deduplication algorithm module212executes a deduplication algorithm on each subset of files, i.e., for each file type. For example, the deduplication algorithm could perform all of the steps of an actual deduplication except the storage operation, and track the relative amounts of data that would have been stored. The relative amounts of data that would have been stored can then be compared to the amount of source data, and a deduplication ratio118calculated, for each file type based on the subset of data files of that file type. Once a deduplication ratio118is obtained for a particular file type, that deduplication ratio118is then stored in the data structure208as associated with that file type.

Referring toFIGS. 1 and 2, once the deduplication analysis module206obtains the deduplication ratios118for each file type, e.g., from the data structure208, one or more lookup tables, or from results of actions of the selector module210and the deduplication algorithm module212as described above, analysis continues. The deduplication analysis module206adds up the total amount of data for each file type, by referencing the file system directories, e.g., as stored in the data structure208, in some embodiments. This total amount of data for each file type is thus based on the file counts, file sizes, and file types of all of the files in the file system directory or directories. For each file type, the total amount of data of that file type is then divided by the deduplication ratio118for that file type, which results in an estimate of the total amount of deduplication backup storage memory110needed for storing all of the deduplicated backup data of that file type. This deduplication backup storage capacity, for each of the file types, is then summed across all of the file types. That is, the total amount of deduplication backup storage capacity needed for storing the deduplicated backup data of all of the file types (i.e., all of the files in the client system) is equal to the sum of the amount of deduplication backup storage memory110needed for storing all of the deduplicated backup data of each file type.

A margin can be added to the result of the summing, or the result of the summing can be multiplied by a margin ratio in some embodiments. For example, it may be preferable to have twice as much deduplication storage capacity online as compared to the amount of deduplication storage capacity that is consumed in storing the deduplicated data of a client system, so that the client system can expand over time. Users may prefer to not exceed a specified amount of available deduplication storage capacity, for example 75% capacity utilized, with 90% capacity utilized as a danger point at which problems are likely. A two to one margin ratio, or three to one margin, or other margin can be established thusly. The resultant deduplication backup storage capacity, as a total for the client system, with or without margin, can then be presented to a user (e.g., via a user interface) as a result of the above calculations performed by the deduplication analysis module206. With this result, the analysis system102predicts the total amount of deduplication backup storage memory110needed for storing all of the deduplicated backup data of the client system.

Still referring toFIGS. 1 and 2, trends can also be tracked by the analysis system102, and applied to projection and forecasting. In some embodiments, the deduplication storage capacity analysis system102has a capacity projection module214, which projects capacity utilization, e.g., through a trending chart with a projection of consumption. The capacity projection module214communicates with the file system128via the file server106, and tracks changes to the file counts (and, in some embodiments the file sizes) for each of the file types, e.g., on a daily, weekly, monthly or other time basis. This information is applied to calculate a rate of increase by file type, as a discreet calculation or a running average or variation thereof. From the rate increase information, the total amount of data for each file type is then projected out over various time spans, e.g., weeks, months or even a year or more. A capacity utilization projection is then made, in which the deduplication analysis module206cooperates with the capacity projection module214. In this cooperation, the projected data amounts for each file type, as determined by the capacity projection module214, are combined with the deduplication ratios118for each file type, and the deduplication analysis module206calculates the deduplication backup storage capacity as projected forward by a particular time span, to a particular time (e.g., a date). The projected deduplication backup storage capacity is then compared to the installed deduplication backup storage capacity to form a capacity utilization projection. For example, the present capacity utilization could be one quarter of the total amount of installed deduplication backup storage capacity, and this could increase to one half of the total amount of installed deduplication backup storage capacity in a particular time span. It should be appreciated that such projections are useful in planning and execution of deduplication backup storage capacity.

The capacity projection module214ofFIG. 2can be supplied with a target capacity utilization, which the capacity projection module214applies to forecast a capacity upgrade date in some embodiments. From the above-described rate increase information, on a per file type basis, the capacity projection module214calculates capacity utilization as above, trying various time spans until the projected capacity utilization approaches, meets, or exceeds the target capacity utilization. In other words, a time span is calculated to the moment the projected capacity utilization intersects the target capacity utilization. The associated time span (i.e., the time span which results in achieving the target capacity utilization) is then added to the present date to form a capacity upgrade date. Thus, the capacity upgrade date is the date on which the system is projected to reach the target capacity utilization. Such a projection is useful in planning and execution of upgrades to deduplication backup storage capacity as a lead time exists for obtaining additional storage capacity once ordered. It should be appreciated that alerts may be generated as predetermined thresholds of utilization are exceeded in some embodiments.

In some embodiments, the deduplication storage capacity analysis system102has a recovery projection module216, which calculates a disaster recovery time for a full system restore from deduplicated backup data. The recovery projection module216obtains various parameters that are applicable to data recovery from the deduplicated backup. For example, the recovery projection module216could look at network bandwidth or throughput of the network124that couples the deduplication server104and the deduplication backup storage memory110, the network122that couples the deduplication server104and the file server106, and the network120that couples the file server and the storage memory108from which the files originate and are to be restored. The recovery projection module216could also look at network ping time, and various internal performance parameters e.g., in a virtual computing environment such as hosted by a Vblock®, and/or other parameters relating to paths by which data travels before, during and after being reconstituted from the deduplicated backup data. The recovery projection module216can analyze how fast data can travel (e.g., data throughput), and bottlenecks along the way, and factor in network and component delays. These factors are then applied to calculations involving how much data is to be restored, for each file type, and what the deduplication ratios are, for each file type, so that the recovery projection module216can calculate the amount of time the full system restore would require. This information is helpful in determining whether certain requirements from a service level agreement are being met.

As an example scenario, consider restoring all of the files in the directory or directories of the file system128to the storage memory108, from the deduplicated backup data in the deduplication backup storage memory110. The recovery projection module216analyzes the networks120,122,124as to respective throughput rates, and determines for file types having low deduplication ratios118that the throughput on the network124coupling the deduplication server104to the deduplication backup storage memory110is the limiting factor. Based on the throughput of this network124, and the total volume of deduplicated backup data for each of the files of those file types, a total amount of time to retrieve all of the deduplicated backup data from the deduplication backup storage memory110to the deduplication server104can be calculated. Then, the recovery projection module216determines that, for file types having high deduplication ratios118, the throughput on the network122coupling the deduplication server and the file server106is the limiting factor (i.e., dominates the time delay calculations) as a result of the reconstituted files of these file types having a much larger volume of data. Based on throughput of that network122, a total amount of time to send all of the reconstituted files of these file types from the deduplication server104to the file server106can be calculated. Variations on the above examples are readily devised. From the above analysis, the recovery projection module216can then calculate recovery times, on a per file type basis, and add these together to project a total recovery time for all of the files of all of the file types in the client system. Such a disaster recovery time for a full system restore is useful in planning for various disaster scenarios, and in determining whether upgrades to bottlenecks so identified are worthwhile. For example, network bandwidths between various connector points, or other parameters, can be studied as to effect each has on the disaster recovery time, and it can be determined whether the recovery system meets expectations or standards, or requirements of a service level agreement.

FIG. 3is a flow diagram of a method for automation of deduplication storage capacity sizing and trending analysis. The method can be practiced on or by embodiments of the deduplication storage capacity analysis system102, or by a processor, e.g., in a server or a computing device. In an action302, file system directories are collected. The file system directories can be collected from one or more file servers, and should be all of the file system directories for which the files are to be subjected to a deduplication backup, or for which the deduplication backup storage capacity is to be determined. The file counts, file sizes and file types are determined, in an action304. This information is gathered from the file system directories. A deduplication ratio is obtained for each file type of these files, in an action306. The deduplication ratios could be obtained from lookup tables, or by performing a deduplication algorithm on subsets of files of each file type, as described above with reference toFIG. 2.

Still referring toFIG. 3, a deduplication backup storage capacity is derived, in an action308. The derivation is based on file counts, file sizes and file types, as determined in the action304. The derivation is further based on the deduplication ratios of the file types, as obtained in the action306. In order to calculate the deduplication backup storage capacity, the total amount of data of each file type is divided by the deduplication ratio of that file type, and such totals are then summed across all of the file types. Further details of such derivation are discussed above with reference to the deduplication analysis module inFIG. 2. File counts of each file type are tracked, in an action310. Rate of increase is calculated by file type, in an action312. Based on these, a capacity utilization projection is calculated, in an action314. The capacity utilization projection is made by projecting data amounts, for each file type, forward in time and then calculating resultant deduplication backup storage amounts needed for each data type. These amounts are summed to form the projected total deduplication backup storage capacity, which is then followed by forming a ratio of the projected total amount of deduplication backup storage capacity by the installed deduplication backup storage capacity. This is the capacity utilization, as projected. Further details of capacity utilization projection are discussed above with reference to the capacity projection module inFIG. 2.

A capacity upgrade date is determined, in an action316ofFIG. 3. This is based on a target capacity utilization, and the capacity utilization projections achieved in the action314. Parameters applicable to data recovery are obtained, in an action318. These may include network bandwidth or throughput in one or more network paths that data traverses while being restored from deduplicated backup data and transferred as reconstituted data. A disaster recovery time from deduplicated backup is calculated, in an action320. The disaster recovery time is based on file counts, file sizes and file types of the files in the file system directories, and is further based on the deduplication ratio of each of the file types and the one or more parameters applicable to data recovery as obtained in the action318. Further details of the disaster recovery time determination are discussed above with reference to the recovery projection module inFIG. 2.

It should be appreciated that the methods described herein may be performed with a digital processing system, such as a conventional, general-purpose computer system. Special purpose computers, which are designed or programmed to perform only one function may be used in the alternative.FIG. 4is an illustration showing an exemplary computing device which may implement the embodiments described herein. The computing device ofFIG. 4may be used to perform embodiments of the functionality for deduplication storage capacity analysis in accordance with some embodiments. The computing device includes a central processing unit (CPU)401, which is coupled through a bus405to a memory403, and mass storage device407. Mass storage device407represents a persistent data storage device such as a disc drive, which may be local or remote in some embodiments. The mass storage device407could implement a backup storage, in some embodiments. Memory403may include read only memory, random access memory, etc. Applications resident on the computing device may be stored on or accessed via a computer readable medium such as memory403or mass storage device407in some embodiments. Applications may also be in the form of modulated electronic signals modulated accessed via a network modem or other network interface of the computing device. It should be appreciated that CPU401may be embodied in a general-purpose processor, a special purpose processor, or a specially programmed logic device in some embodiments.

Display411is in communication with CPU401, memory403, and mass storage device407, through bus405. Display411is configured to display any visualization tools or reports associated with the system described herein. Input/output device409is coupled to bus405in order to communicate information in command selections to CPU401. It should be appreciated that data to and from external devices may be communicated through the input/output device409. CPU401can be defined to execute the functionality described herein to enable the functionality described with reference toFIGS. 1-3. The code embodying this functionality may be stored within memory403or mass storage device407for execution by a processor such as CPU401in some embodiments. The operating system on the computing device may be MS-WINDOWS™, UNIX™, LINUX™, iOS™, or other known operating systems. It should be appreciated that the embodiments described herein may be integrated with virtualized computing system also.

Detailed illustrative embodiments are disclosed herein. However, specific functional details disclosed herein are merely representative for purposes of describing embodiments. Embodiments may, however, be embodied in many alternate forms and should not be construed as limited to only the embodiments set forth herein.

With the above embodiments in mind, it should be understood that the embodiments might employ various computer-implemented operations involving data stored in computer systems. These operations are those requiring physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. Further, the manipulations performed are often referred to in terms, such as producing, identifying, determining, or comparing. Any of the operations described herein that form part of the embodiments are useful machine operations. The embodiments also relate to a device or an apparatus for performing these operations. The apparatus can be specially constructed for the required purpose, or the apparatus can be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines can be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.

A module, an application, a layer, an agent or other method-operable entity could be implemented as hardware, firmware, or a processor executing software, or combinations thereof. It should be appreciated that, where a software-based embodiment is disclosed herein, the software can be embodied in a physical machine such as a controller. For example, a controller could include a first module and a second module. A controller could be configured to perform various actions, e.g., of a method, an application, a layer or an agent.

The embodiments can also be embodied as computer readable code on a tangible non-transitory computer readable medium. The computer readable medium is any data storage device that can store data, which can be thereafter read by a computer system. Examples of the computer readable medium include hard drives, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. The computer readable medium can also be distributed over a network coupled computer system so that the computer readable code is stored and executed in a distributed fashion. Embodiments described herein may be practiced with various computer system configurations including hand-held devices, tablets, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like. The embodiments can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.