Systems and methods for deduplicating archive objects

A computer-implemented method for deduplicating archive objects may include (1) tagging, using an archiving service, archive objects to indicate data related to the archive objects, (2) assigning an archive object and an additional archive object to be processed by a deduplication engine, (3) determining a degree of overlap between the archive object and the additional archive object based on tags assigned to the archive object and the additional archive object by the archiving service, and (4) adjusting, using the deduplication engine, deduplication between the archive object and the additional archive object using the degree of overlap determined to exist between the archive object and the additional archive object based on the tags assigned by the archiving service. Various other methods, systems, and computer-readable media are also disclosed.

BACKGROUND

Individuals and organizations often process, index, and/or annotate records for archiving. For example, organizations may become involved in requirements for e-discovery of various records and documents. To prepare for e-discovery, these organizations may process their records and documents using an e-discovery service. The e-discovery service may tag strings of text and other data within the records for easy search, recall, and/or other statistics and processing.

Organizations also often deduplicate archived records (e.g., compress them by removing redundancies) to conserve storage space, for example. However, deduplication operations may suffer from several inefficiencies. For example, conventional deduplication systems may attempt to deduplicate records that contain little or no redundant information. Conventional systems may also lack information about which records share redundant information (and obtaining that information may be costly in terms of time and performance). Such systems may also attempt to deduplicate records that have few commonalities before deduplicating records that have more commonalities, despite time limits that prevent the system from deduplicating all records (e.g., misallocated priorities). Moreover, conventional deduplications systems may move records to different deduplication locations without grouping the records to optimize the commonalities between the records at one or more of these locations. The instant disclosure identifies a need, therefore, for improved methods for deduplicating archive objects.

SUMMARY

As will be described in greater detail below, the instant disclosure generally relates to systems and methods for deduplicating archive objects using tags or references assigned by an archiving or e-discovery system. As explained below, these tags or references may enable the disclosed systems and methods to more efficiently prioritize and relocate archive objects for deduplication.

In one example, a computer-implemented method for deduplicating archive objects may include (1) tagging, using an archiving service, archive objects to indicate data related to the archive objects, (2) assigning an archive object and an additional archive object to be processed by a deduplication engine, (3) determining a degree of overlap between the archive object and the additional archive object based on tags assigned to the archive object and the additional archive object using the archiving service, and (4) adjusting, using the deduplication engine, deduplication between the archive object and the additional archive object using the degree of overlap determined to exist between the archive object and the additional archive object based on the tags assigned by the archiving service.

In one example, the archiving service may include an e-discovery service that processes archive objects for legal investigations. In another example, adjusting deduplication may include adjusting a deduplication post-processing stage that follows a deduplication in-line processing stage. In some embodiments, adjusting the deduplication post-processing stage may include excluding backup images that share a predetermined threshold amount of deduplication references from the post-processing stage. Moreover, the excluded backup images may belong to a set of images that include archive objects containing an additional predetermined threshold amount of tags.

In some embodiments, determining the degree of overlap may include determining the degree of overlap based on tags indicating content within the archive object and the additional archive object. In further examples, determining the degree of overlap may include determining the degree of overlap based on tags indicating text content within the archive object and the additional archive object. In some examples, determining the degree of overlap may include determining the degree of overlap based on tags indicating metadata for the archive object and the additional archive object.

In some examples, adjusting deduplication may include comparing the degree of overlap determined to exist between the archive object and the additional archive object with an additional degree of overlap determined to exist between members of a different set of archive objects. The method may also include determining that the degree of overlap is greater than the additional degree of overlap. Moreover, adjusting deduplication may include prioritizing deduplication of the archive object and the additional archive object over deduplication of members of the different set of archive objects based on the determination that the degree of overlap is greater than the additional degree of overlap. Adjusting deduplication may also include prioritizing transmission of the archive object or the additional archive object based on the determination that the degree of overlap is greater than the additional degree of overlap. Additionally, adjusting deduplication may include grouping archive objects at different locations to increase predicted deduplication at one of the different locations.

In one embodiment, a system for implementing the above-described method may include (1) a tagging module, stored in memory, that tags, using an archiving service, archive objects to indicate data related to the archive objects, (2) an assignment module, stored in memory, that assigns an archive object and an additional archive object to be processed by a deduplication engine, (3) a determination module, stored in memory, that determines a degree of overlap between the archive object and the additional archive object based on tags assigned to the archive object and the additional archive object using the archiving service, (4) an adjustment module, stored in memory, that adjusts, using the deduplication engine, deduplication between the archive object and the additional archive object using the degree of overlap determined to exist between the archive object and the additional archive object based on the tags assigned by the archiving service, and (5) at least one physical processor that executes the tagging module, the assignment module, the determination module, and the adjustment module.

In some examples, the above-described method may be encoded as computer-readable instructions on a non-transitory computer-readable medium. For example, a computer-readable medium may include one or more computer-executable instructions that, when executed by at least one processor of a computing device, may cause the computing device to (1) tag, using an archiving service, archive objects to indicate data related to the archive objects, (2) assign an archive object and an additional archive object to be processed by a deduplication engine, (3) determine a degree of overlap between the archive object and the additional archive object based on tags assigned to the archive object and the additional archive object by the archiving service, and (4) adjust, using the deduplication engine, deduplication between the archive object and the additional archive object using the degree of overlap determined to exist between the archive object and the additional archive object based on the tags assigned by the archiving service.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

The present disclosure is generally directed to systems and methods for deduplicating archive objects. As will be explained in greater detail below, the disclosed systems and methods may increase the speed and efficiency of deduplication operations. For example, these systems and methods may more accurately predict a degree to which a deduplication engine may deduplicate two archive objects. The systems and methods may also more efficiently allocate different sets of archive objects to different deduplication engines to optimize deduplication.

The following will provide, with reference toFIGS. 1-2, detailed descriptions of exemplary systems for deduplicating archive objects. Detailed descriptions of corresponding computer-implemented methods will also be provided in connection withFIG. 3. Moreover, detailed descriptions of exemplary archive objects will be provided in connection withFIGS. 4-5. Furthermore, detailed descriptions of exemplary deduplication engines will be provided in connection withFIG. 6. In addition, detailed descriptions of an exemplary computing system and network architecture capable of implementing one or more of the embodiments described herein will be provided in connection withFIGS. 7 and 8, respectively.

FIG. 1is a block diagram of an exemplary system100for deduplicating archive objects. As illustrated in this figure, exemplary system100may include one or more modules102for performing one or more tasks. For example, and as will be explained in greater detail below, exemplary system100may include a tagging module104that tags, using an archiving service, archive objects to indicate data related to the archive objects. Exemplary system100may also include an assignment module106that assigns an archive object and an additional archive object to be processed by a deduplication engine.

In addition, and as will be described in greater detail below, exemplary system100may include a determination module108that determines a degree of overlap between the archive object and the additional archive object based on tags assigned to the archive object and the additional archive object by the archiving service. Furthermore, exemplary system100may include an adjustment module110that adjusts, using the deduplication engine, deduplication between the archive object and the additional archive object using the degree of overlap determined to exist between the archive object and the additional archive object based on the tags assigned by the archiving service. Although illustrated as separate elements, one or more of modules102inFIG. 1may represent portions of a single module or application.

As illustrated inFIG. 1, exemplary system100may also include one or more databases, such as database120. In one example, database120may be configured to store archive objects122. As used herein, the phrase “archive objects” generally refers to electronic objects, such as files, emails, and documents (including scanned documents) within the context of a storage archive. Similarly, database120may be configured to store deduplication references124. As used herein, the phrase “deduplication references” generally refers to metadata identifying and/or containing common underlying data used to deduplicate archive objects, as discussed below. Moreover, database120may be further configured to store tags126. As used herein, the phrase “tags” generally refers to marks or metadata assigned to underlying data or documents to indicate information about the underlying data or documents. An e-discovery service may assign the tags, as discussed below. The systems and methods described herein may leverage assigned tags to predict how well (e.g., how fast, efficiently, and/or to what estimated degree of compression) a deduplication system may deduplicate archive objects.

Database120may represent portions of a single database or computing device or a plurality of databases or computing devices. For example, database120may represent a portion of server206inFIG. 2, computing system710inFIG. 7, and/or portions of exemplary network architecture800inFIG. 8. Alternatively, database120inFIG. 1may represent one or more physically separate devices capable of being accessed by a computing device, such as server206inFIG. 2, computing system710inFIG. 7, and/or portions of exemplary network architecture800inFIG. 8.

Exemplary system100inFIG. 1may be implemented in a variety of ways. For example, all or a portion of exemplary system100may represent portions of exemplary system200inFIG. 2. As shown inFIG. 2, system200may include a computing device202in communication with a server206via a network204. In one example, computing device202may be programmed with one or more of modules102and/or may store all or a portion of the data in database120. Additionally or alternatively, server206may be programmed with one or more of modules102and/or may store all or a portion of the data in database120.

In the example ofFIG. 2, server206may internally or remotely manage, execute, and/or direct archiving processing, including e-discovery services. Accordingly, server206may include an archiving service250, which may further include tagging module104. In contrast, computing device202may manage, execute, and/or direct deduplication processing by itself or in cooperation with one or more deduplication engines, including potentially deduplication engine260and deduplication engine262(although not shown inFIG. 2, these engines may also have connections to network204apart from computing device202). Accordingly, computing device202may include assignment module106, which may assign archive objects for deduplication. Similarly, computing device202may also include determination module108and adjustment module110. Computing device202may cooperate with server206by leveraging and analyzing tags assigned at server206to predict estimated deduplication results and/or more efficiently allocate and/or setup deduplication processing.

Although shown as separate, one or more of computing device202, server206, and modules102may be combined into one or more other combinations or permutations. Furthermore, system200may further include database120(including a further set234of pointers or identifiers of archive objects), which may be stored in a single storage (as inFIG. 2) or across multiple devices, such as computing device202and server206. Computing device202and/or server206may access database120through network204.

In one embodiment, one or more of modules102fromFIG. 1may, when executed by at least one processor of computing device202and/or server206, enable computing device202and/or server206to deduplicate archive objects. For example, and as will be described in greater detail below, one or more of modules102may cause computing device202and/or server206to tag, using archiving service250, archive objects to indicate data related to the archive objects. Moreover, one or more of modules102also may cause computing device202and/or server206to assign an archive object230and an additional archive object232to be processed by a deduplication engine, such as deduplication engine260or deduplication engine262. In some examples, one or more of modules102may subsequently cause computing device202and/or server206to determine a degree of overlap between archive object230and additional archive object232based on tags, such as tag240, tag242, and tag244, assigned to archive object230and additional archive object232by archiving service250. Furthermore, one or more of modules102may additionally cause computing device202and/or server206to adjust, using the deduplication engine, deduplication between archive object230and additional archive object232using the degree of overlap determined to exist between archive object230and additional archive object232based on the tags assigned by archiving service250.

Computing device202generally represents any type or form of computing device capable of reading computer-executable instructions. Examples of computing device202include, without limitation, laptops, tablets, desktops, servers, cellular phones, Personal Digital Assistants (PDAs), multimedia players, embedded systems, combinations of one or more of the same, exemplary computing system710inFIG. 7, or any other suitable computing device.

Server206generally represents any type or form of computing device that is capable of tagging and/or deduplicating archive objects. Examples of server206include, without limitation, application servers and database servers configured to provide various database services and/or run certain software applications.

FIG. 3is a flow diagram of an exemplary computer-implemented method300for deduplicating archive objects. The steps shown inFIG. 3may be performed by any suitable computer-executable code and/or computing system. In some embodiments, the steps shown inFIG. 3may be performed by one or more of the components of system100inFIG. 1, system200inFIG. 2, computing system710inFIG. 7, and/or portions of exemplary network architecture800inFIG. 8.

As illustrated inFIG. 3, at step302one or more of the systems described herein may tag, using an archiving service, archive objects to indicate data related to the archive objects. For example, tagging module104may, as part of server206inFIG. 2, tag, using archiving service250, archive objects to indicate data related to the archive objects. As used herein, the term “archiving service” generally refers to any service or functionality for storing files, data, records, and/or documents for potential subsequent retrieval and/or processing.

Tagging module104may tag, using archiving service250, archive objects to indicate data related to the archive objects in a variety of ways. For example, tagging module104may tag some (e.g., specified) and/or all documents that are incoming to a specified locations, such as an inbox, file folder, disk drive, virtual drive, physical machine, and/or hardware machine. Moreover, tagging module104may tag archive objects by searching those archive objects for specified data, content, metadata, and/or patterns of the same.

In some examples, a user or administrator may specify the data and/or patterns for searching. For example, the user and/or archiving service may specify that particular data and/or patterns indicate relevant documents for e-discovery purposes (as discussed below), and/or are valuable documents for preservation. Upon detecting one or more hits, tagging module104may tag the archive objects to mark the found data or pattern. The tag may indicate the particular data or type of data identified as associated with or located within the document, as well as metadata about the found data, including the timing of discovery and/or the location where the data resides within the document.

Additionally, or alternatively, tagging module104may index some (e.g., specified) part(s) and/or all of the archive objects. For example, tagging module104may parse the archive objects into segments and/or permutations of text, characters, fields, and/or sections, which may have lengths or delimiters specified or formulated by tagging module104and/or a user. These segments may include words, phrases, numbers, sentences, and/or paragraphs, as well as metadata fields including header field or variable name and/or variable value. As used herein, the phrase “metadata” generally refers to data about underlying data and/or files, including data about file content, file/packet payload, and/or displayed content (e.g., metadata may indicate how other data should be displayed, formatted, and/or stored). Examples of such metadata include, without limitation, a file name, file type, last modified date and/or time, encryption/privacy status, file author/creator, edit history, and/or any other metadata stored within the file or otherwise associated with the file by a data structure.

In general, tagging module104may search and/or index documents upon reception, upon a defined schedule, upon user/administrator command, and/or upon detecting that the documents are candidates for e-discovery, for example. In addition, archiving service250may enable users and/or programs to search, process, display, and/or reference archive objects using queries in terms of assigned tags.

In some examples, tagging module104may use an archiving service that includes or constitutes an e-discovery service. In these examples, the e-discovery service may process archive objects for legal investigations. As used herein, the phrase “legal investigations” generally refers to civil and/or criminal litigation, arbitration, and/or government or quasi-government investigations and/or discovery requirements. Accordingly, the e-discovery service may include functionality for identifying relevant documents (e.g., relevant to a legal investigation), preserving relevant documents, collecting and transferring relevant documents to legal counsel, converting documents into formats more convenient for batch review and/or e-discovery, and/or extracting text and/or metadata.

As further shown inFIG. 3, at step304one or more of the systems described herein may assign an archive object and an additional archive object to be processed by a deduplication engine. For example, assignment module106may, as part of computing device202inFIG. 2, assign archive object230and additional archive object232to be processed by a deduplication engine distinct from (or included within) archiving service250.

Notably, the systems and methods herein may facilitate or improve deduplication by analyzing previously-assigned tags (e.g., by another system, such as an e-discovery service) prior to the deduplication engine searching for, identifying, and/or marking its own deduplication references to begin or complete deduplication. In other words, the e-discovery tags and the deduplication references may be substantially or fundamentally different, such that the e-discovery tags enable prediction of how well the deduplication engine will perform deduplication, and the deduplication engine later creates and/or processes its own deduplication references to actually (as opposed to predictably) implement and/or complete deduplication.

Assignment module106may assign archive object230and additional archive object232to be processed by a deduplication engine in a variety of ways. For example, assignment module106may autonomously assign the archive objects for deduplication based on a deduplication policy and/or in response to user/administrator commands. This deduplication policy may specify that some (e.g., designated) or all documents satisfying established conditions should be assigned for deduplication, including conditions related to size, type, timing, location, relocation, arrival, and/or relevance.

In some examples, assignment module106may initially assign archive object230and additional archive object232for initial, in-line, lazy, and/or more efficient deduplication. As used herein, the phrase “in-line deduplication” generally refers to deduplication at a storage location as the data is first received, retrieved, and/or stored at the storage location (e.g., in real time). In general, in-line deduplication contrasts with post-processing deduplication, which may occur after an in-line stage of deduplication and which may involve greater amounts of time (e.g., slower than real time processing), storage, and/or computing power. Assignment module106may also assign archive object230and additional archive object232to be processed as candidates for a post-processing stage of deduplication. As used herein, the phrase “assign [ . . . ] to be processed by a deduplication engine” generally refers to direct or immediate deduplication, identifying candidates for deduplication, and/or identifying archive objects to be sorted, evaluated, prioritized, and/or relocated for deduplication. Notably, assignment module106may also, or additionally, assign archive object230and additional archive object232for deduplication prior to, and/or based on, results or output from determination module108and/or adjustment module110, as discussed below.

As further shown inFIG. 3, at step306one or more of the systems described herein may determine a degree of overlap between the archive object and the additional archive object based on tags assigned to the archive object and the additional archive object using the archiving service. For example, determination module108may, as part of computing device202, determine a degree of overlap between archive object230and additional archive object232based on tags assigned to archive object230and additional archive object232by archiving service250.

Determination module108may determine a degree of overlap between archive object230and additional archive object232in a variety of ways. In some examples, determination module108may calculate a proportion and/or number of tags assigned to archive object230that are (1) also assigned to additional archive object232, and/or (2) vice versa (throughout this description, archive object230and additional archive object232may be largely interchangeable). In these examples, determination module108may calculate a degree of estimated overlap using a function based on values (1) and/or (2), such as a weighted or non-weighted average, median, sum, and/or product.

Moreover, determination module108may weight different tags differently, based on the type, nature, and/or content of the tag. For example, determination module108may weigh tags of larger text or content more heavily than tags of smaller text or content, and/or weigh tags of underlying data more heavily than tags of metadata (which tends to be smaller than underlying data).

In further examples, determination module108may perform the calculation over some (e.g., designated) or all tags assigned to an archive object. For example, determination module108may limit its calculation of overlap to tags satisfying one or more conditions, including conditions related to tag type (e.g., text, metadata, images), tag keyword (e.g., e-discovery keyword), and/or tag creation or modification date. Similarly, determination module108may determine the degree of overlap based on tags indicating content, such as text, and/or metadata, within and/or associated with the archive object and the additional archive object.

FIG. 4shows a block diagram illustrating how different tags may apply to different archive objects. In the example ofFIG. 4, archive object230and additional archive object232may correspond to archived emails. Tagging module104may have assigned tag240to metadata440in archive object230, which may indicate that archive object230was sent on “11/28/10.” Similarly, tagging module104may have assigned tag244to additional metadata450in archive object232, which may indicate that archive object232was sent on “09/13/12.” Furthermore, tagging module104may have assigned tag242to both of the archive objects, because both of these archive objects contain the name “Bill” within their content (e.g., as a signature). Accordingly, determination module108may determine that archive object230has a 33% (⅓) degree of overlap with the additional archive object (and vice versa), because archive object230shares one third of its own tags and one third of the tags assigned additional archive object232(in some or most cases, the proportions may be different for two archive objects under comparison). Moreover, determination module108may determine a granular or absolute/binary degree of overlap, such as by initially calculating a granular degree of overlap, and then comparing that degree to a threshold level to reach a binary conclusion (i.e., overlap or no overlap) or category of overlap (e.g., low, medium, high). AlthoughFIG. 4is limited to three tags each for both archive objects, these particular tags are merely examples and far more tags, including tags of different types and/or lengths, may be used.

As further shown inFIG. 3, at step308one or more of the systems described herein may adjust, using the deduplication engine, deduplication between the archive object and the additional archive object using the degree of overlap determined to exist between the archive object and the additional archive object based on the tags assigned by the archiving service. For example, adjustment module110may, as part of computing device202, adjust, using the deduplication engine, deduplication between archive object230and additional archive object232using the degree of overlap determined to exist between archive object230and additional archive object232based on the tags assigned by archiving service250.

Adjustment module110may adjust deduplication in a variety of ways. In one example, adjustment module110may begin adjustment by limiting deduplication to (1) archive objects having at least a defined amount or proportion of tags (indicating higher potential deduplication) and/or (2) backup images (where archive objects may be stored) that contain less than a defined amount or proportion of deduplication references (otherwise indicating that these images may already been sufficiently deduplicated). For example, adjustment module110may adjust a deduplication post-processing stage that follows a deduplication in-line processing stage, such as by excluding archive objects and/or backup images from the post-processing stage based on factors (1) and/or (2) above. In other words, adjustment module110may adjust the deduplication post-processing stage at least in part by excluding backup images that share a predetermined threshold amount of deduplication references (e.g., references generated during the in-line stage) from the post-processing stage. Moreover, the excluded backup images may belong to a set of images that each include archive objects containing an additional predetermined threshold amount of tags assigned by archiving service250. Notably, the predetermined threshold and the additional predetermined threshold may have the same, similar, or different values.

Tags assigned by tagging module104and deduplication references assigned or marked by a deduplication engine may be identical, substantially identical, different, and/or mapped to each other. In addition, the deduplication engine may create deduplication references based on previously-assigned tags, such as by copying tags (e.g., while formatting and/or modifying them) as deduplication references, while nevertheless creating its own deduplication references (e.g., the copies) to actualize deduplication. Notably, archiving service250may assign tags to archive objects even when the tags do not identify underlying content (e.g., potentially redundant and removable content), and so the phrase “tag” may be broader than “deduplication reference” in that respect.

In one illustrative example, tagging module104may tag a file as created on a particular date, even if the file contains no data indicating its creation date and therefore no data about the date that can be removed or compressed. In this example, the deduplication engine may be unable, for that reason, to copy the tag as a deduplication reference. Nevertheless, the deduplication engine may predict that other files created on the same date are more likely to have common content (e.g., increased deduplication potential) than files not created on the same date, and therefore not sharing that metadata tag.

In other examples, adjustment module110may adjust deduplication at least in part by comparing the degree of overlap determined to exist between the archive object and the additional archive object with an additional degree of overlap determined to exist between members of a different set of archive objects. The following discussion outlines how these examples may function with reference toFIG. 5. As shown in the figure, set234may include a different set of archive objects than the set including archive object230and additional archive object232, as discussed above. Specifically, set234may also include archive object230(reproduced fromFIG. 4), but lack additional archive object232, and instead include a different archive object532, which may further contain metadata550and content552. Notably, although set234shares an archive object with the previously discussed set of archive object230and additional archive object232, these two sets may instead share no members (and may contain an arbitrary number of archive objects).

As shown inFIG. 5, determination module108may determine that archive object230and different archive object532have no or substantially no overlap. In contrast to the archive objects shown inFIG. 4, those shown inFIG. 5contain both different dates in their metadata, and completely different names in their underlying content. Accordingly, adjustment module110may determine that the degree of overlap between the archive objects shown inFIG. 4is greater than the additional degree of overlap between the archive objects shown inFIG. 5.

Adjustment module110may adjust deduplication in various ways based on the determination that the overlap for one set of archive objects is greater than for another set.FIG. 6is a block diagram showing exemplary deduplication engines that may illustrate various adjustments that adjustment module110may make. As shown inFIG. 6, archive objects122may include archive object230and additional archive object232(as discussed above), as well as archive object634, archive object636, archive object638, and archive object640(the last two of which may have been relocated to deduplication engine260and deduplication engine262, respectively).

In one example, adjustment module110may adjust deduplication at least in part by prioritizing deduplication of the archive object and the additional archive object over deduplication of members of the different set of archive objects based on the determination that the degree of overlap is greater than the additional degree of overlap. More specifically, adjustment module110may order and/or reorder a queue to prioritize, in time, sets of archive objects that have higher overlap than other sets having lower overlap.

Similarly, adjustment module110may adjust deduplication at least in part by prioritizing transmission of the archive object and/or the additional archive object based on the determination that the degree of overlap is greater than the additional degree of overlap. For example, adjustment module110may have prioritized transmitting archive object638and/or archive object640ahead of the remaining archive objects. Similarly, based on the determination of greater overlap (as discussed above), adjustment module110may prioritize transmitting one or both of archive object230and/or232. In particular, adjustment module110may prioritize sending archive objects that have greater overlap (e.g., with each other and/or with archive objects already at the destination, such as archive object638and/or archive object640) over other sets of archive objects that have lower overlap.

More generally, and without necessarily prioritizing archive objects in time, adjustment module110may perform calculations to group or sort archive objects to different destinations (e.g., deduplication engines) to maximize and/or optimize deduplication between archive objects at one or more destination, as discussed further below. According to one algorithm, adjustment module110may identify constraints (if any) on locations, movement, and/or permutations of destinations and archive objects. For example, certain locations may have constraints on the number of archive objects (e.g., zero), archive object type, and/or computation, storage, and/or bandwidth. Similarly, archive objects may have constraints on movement (e.g., cannot move, or can move to certain destinations but not others). Given those constraints, adjustment module110may generate permutations of destinations and archive objects at those destinations. Adjustment module110may then calculate, according to one or more formulas or heuristics (as discussed above regarding determination module108), such as a weighted or non-weighted sum or product, a degree of deduplication or overlap, over some (e.g., more than one) or all destinations and/or archive objects. In other words, adjustment module110may load balance (in an optimized or more efficient manner) deduplication across multiple deduplication engines based on estimated degrees of overlap between archive objects at one or more same deduplication engines, among other factors.

In the example ofFIG. 6, adjustment module110may determine that any permutation of the six archive objects results in optimal or improved overlap or deduplication, such as moving all archive objects to one deduplication engine, dividing them evenly between the two deduplication engines, or any other permutation (within any existing constraints such as constraints on bandwidth, workload, software protocols, user preferences, etc.). Accordingly, adjustment module110may adjust deduplication at least in part by grouping archive objects at different locations to increase predicted deduplication at one of the different locations. Moreover, all comments about analyzing, comparing, prioritizing, transmitting, grouping, and/or sorting archive objects and sets of archive objects, as discussed above, may similarly apply to backup images, as a whole, that contain sets of archive objects (e.g., but cannot be conveniently divided). Similarly, although the above discussion has focused on comparing overlap between two archive objects, archive object230and additional archive object232, the disclosed systems and methods may generally compare larger sets or backup images to determine degrees of overlap between those larger sets or backup images. Lastly, adjustment module110may, in some cases, adjust deduplication by excluding one or more archive objects or backup images from the post-processing stage of deduplication.

As explained above, the disclosed systems and methods may increase the speed and efficiency of deduplication operations. For example, these systems and methods may more accurately predict a degree to which a deduplication engine may deduplicate two archive objects. The systems and methods may also more efficiently allocate different sets of archive objects to different deduplication engines to optimize deduplication.

FIG. 7is a block diagram of an exemplary computing system710capable of implementing one or more of the embodiments described and/or illustrated herein. For example, all or a portion of computing system710may perform and/or be a means for performing, either alone or in combination with other elements, one or more of the steps described herein (such as one or more of the steps illustrated inFIG. 3). All or a portion of computing system710may also perform and/or be a means for performing any other steps, methods, or processes described and/or illustrated herein.

Computing system710broadly represents any single or multi-processor computing device or system capable of executing computer-readable instructions. Examples of computing system710include, without limitation, workstations, laptops, client-side terminals, servers, distributed computing systems, handheld devices, or any other computing system or device. In its most basic configuration, computing system710may include at least one processor714and a system memory716.

Processor714generally represents any type or form of physical processing unit (e.g., a hardware-implemented central processing unit) capable of processing data or interpreting and executing instructions. In certain embodiments, processor714may receive instructions from a software application or module. These instructions may cause processor714to perform the functions of one or more of the exemplary embodiments described and/or illustrated herein.

System memory716generally represents any type or form of volatile or non-volatile storage device or medium capable of storing data and/or other computer-readable instructions. Examples of system memory716include, without limitation, Random Access Memory (RAM), Read Only Memory (ROM), flash memory, or any other suitable memory device. Although not required, in certain embodiments computing system710may include both a volatile memory unit (such as, for example, system memory716) and a non-volatile storage device (such as, for example, primary storage device732, as described in detail below). In one example, one or more of modules102fromFIG. 1may be loaded into system memory716.

In certain embodiments, exemplary computing system710may also include one or more components or elements in addition to processor714and system memory716. For example, as illustrated inFIG. 7, computing system710may include a memory controller718, an Input/Output (I/O) controller720, and a communication interface722, each of which may be interconnected via a communication infrastructure712. Communication infrastructure712generally represents any type or form of infrastructure capable of facilitating communication between one or more components of a computing device. Examples of communication infrastructure712include, without limitation, a communication bus (such as an Industry Standard Architecture (ISA), Peripheral Component Interconnect (PCI), PCI Express (PCIe), or similar bus) and a network.

Memory controller718generally represents any type or form of device capable of handling memory or data or controlling communication between one or more components of computing system710. For example, in certain embodiments memory controller718may control communication between processor714, system memory716, and I/O controller720via communication infrastructure712.

I/O controller720generally represents any type or form of module capable of coordinating and/or controlling the input and output functions of a computing device. For example, in certain embodiments I/O controller720may control or facilitate transfer of data between one or more elements of computing system710, such as processor714, system memory716, communication interface722, display adapter726, input interface730, and storage interface734.

Communication interface722broadly represents any type or form of communication device or adapter capable of facilitating communication between exemplary computing system710and one or more additional devices. For example, in certain embodiments communication interface722may facilitate communication between computing system710and a private or public network including additional computing systems. Examples of communication interface722include, without limitation, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, and any other suitable interface. In at least one embodiment, communication interface722may provide a direct connection to a remote server via a direct link to a network, such as the Internet. Communication interface722may also indirectly provide such a connection through, for example, a local area network (such as an Ethernet network), a personal area network, a telephone or cable network, a cellular telephone connection, a satellite data connection, or any other suitable connection.

In certain embodiments, communication interface722may also represent a host adapter configured to facilitate communication between computing system710and one or more additional network or storage devices via an external bus or communications channel. Examples of host adapters include, without limitation, Small Computer System Interface (SCSI) host adapters, Universal Serial Bus (USB) host adapters, Institute of Electrical and Electronics Engineers (IEEE) 1394 host adapters, Advanced Technology Attachment (ATA), Parallel ATA (PATA), Serial ATA (SATA), and External SATA (eSATA) host adapters, Fibre Channel interface adapters, Ethernet adapters, or the like. Communication interface722may also allow computing system710to engage in distributed or remote computing. For example, communication interface722may receive instructions from a remote device or send instructions to a remote device for execution.

As illustrated inFIG. 7, computing system710may also include at least one display device724coupled to communication infrastructure712via a display adapter726. Display device724generally represents any type or form of device capable of visually displaying information forwarded by display adapter726. Similarly, display adapter726generally represents any type or form of device configured to forward graphics, text, and other data from communication infrastructure712(or from a frame buffer, as known in the art) for display on display device724.

As illustrated inFIG. 7, exemplary computing system710may also include at least one input device728coupled to communication infrastructure712via an input interface730. Input device728generally represents any type or form of input device capable of providing input, either computer or human generated, to exemplary computing system710. Examples of input device728include, without limitation, a keyboard, a pointing device, a speech recognition device, or any other input device.

As illustrated inFIG. 7, exemplary computing system710may also include a primary storage device732and a backup storage device733coupled to communication infrastructure712via a storage interface734. Storage devices732and733generally represent any type or form of storage device or medium capable of storing data and/or other computer-readable instructions. For example, storage devices732and733may be a magnetic disk drive (e.g., a so-called hard drive), a solid state drive, a floppy disk drive, a magnetic tape drive, an optical disk drive, a flash drive, or the like. Storage interface734generally represents any type or form of interface or device for transferring data between storage devices732and733and other components of computing system710. In one example, database120fromFIG. 1may be stored in primary storage device732.

In certain embodiments, storage devices732and733may be configured to read from and/or write to a removable storage unit configured to store computer software, data, or other computer-readable information. Examples of suitable removable storage units include, without limitation, a floppy disk, a magnetic tape, an optical disk, a flash memory device, or the like. Storage devices732and733may also include other similar structures or devices for allowing computer software, data, or other computer-readable instructions to be loaded into computing system710. For example, storage devices732and733may be configured to read and write software, data, or other computer-readable information. Storage devices732and733may also be a part of computing system710or may be a separate device accessed through other interface systems.

The computer-readable medium containing the computer program may be loaded into computing system710. All or a portion of the computer program stored on the computer-readable medium may then be stored in system memory716and/or various portions of storage devices732and733. When executed by processor714, a computer program loaded into computing system710may cause processor714to perform and/or be a means for performing the functions of one or more of the exemplary embodiments described and/or illustrated herein. Additionally or alternatively, one or more of the exemplary embodiments described and/or illustrated herein may be implemented in firmware and/or hardware. For example, computing system710may be configured as an Application Specific Integrated Circuit (ASIC) adapted to implement one or more of the exemplary embodiments disclosed herein.

FIG. 8is a block diagram of an exemplary network architecture800in which client systems810,820, and830and servers840and845may be coupled to a network850. As detailed above, all or a portion of network architecture800may perform and/or be a means for performing, either alone or in combination with other elements, one or more of the steps disclosed herein (such as one or more of the steps illustrated inFIG. 3). All or a portion of network architecture800may also be used to perform and/or be a means for performing other steps and features set forth in the instant disclosure.

Client systems810,820, and830generally represent any type or form of computing device or system, such as exemplary computing system710inFIG. 7. Similarly, servers840and845generally represent computing devices or systems, such as application servers or database servers, configured to provide various database services and/or run certain software applications. Network850generally represents any telecommunication or computer network including, for example, an intranet, a WAN, a LAN, a PAN, or the Internet. In one example, client systems810,820, and/or830and/or servers840and/or845may include all or a portion of system100fromFIG. 1.

As illustrated inFIG. 8, one or more storage devices860(1)-(N) may be directly attached to server840. Similarly, one or more storage devices870(1)-(N) may be directly attached to server845. Storage devices860(1)-(N) and storage devices870(1)-(N) generally represent any type or form of storage device or medium capable of storing data and/or other computer-readable instructions. In certain embodiments, storage devices860(1)-(N) and storage devices870(1)-(N) may represent Network-Attached Storage (NAS) devices configured to communicate with servers840and845using various protocols, such as Network File System (NFS), Server Message Block (SMB), or Common Internet File System (CIFS).

Servers840and845may also be connected to a Storage Area Network (SAN) fabric880. SAN fabric880generally represents any type or form of computer network or architecture capable of facilitating communication between a plurality of storage devices. SAN fabric880may facilitate communication between servers840and845and a plurality of storage devices890(1)-(N) and/or an intelligent storage array895. SAN fabric880may also facilitate, via network850and servers840and845, communication between client systems810,820, and830and storage devices890(1)-(N) and/or intelligent storage array895in such a manner that devices890(1)-(N) and array895appear as locally attached devices to client systems810,820, and830. As with storage devices860(1)-(N) and storage devices870(1)-(N), storage devices890(1)-(N) and intelligent storage array895generally represent any type or form of storage device or medium capable of storing data and/or other computer-readable instructions.

In certain embodiments, and with reference to exemplary computing system710ofFIG. 7, a communication interface, such as communication interface722inFIG. 7, may be used to provide connectivity between each client system810,820, and830and network850. Client systems810,820, and830may be able to access information on server840or845using, for example, a web browser or other client software. Such software may allow client systems810,820, and830to access data hosted by server840, server845, storage devices860(1)-(N), storage devices870(1)-(N), storage devices890(1)-(N), or intelligent storage array895. AlthoughFIG. 8depicts the use of a network (such as the Internet) for exchanging data, the embodiments described and/or illustrated herein are not limited to the Internet or any particular network-based environment.

In at least one embodiment, all or a portion of one or more of the exemplary embodiments disclosed herein may be encoded as a computer program and loaded onto and executed by server840, server845, storage devices860(1)-(N), storage devices870(1)-(N), storage devices890(1)-(N), intelligent storage array895, or any combination thereof. All or a portion of one or more of the exemplary embodiments disclosed herein may also be encoded as a computer program, stored in server840, run by server845, and distributed to client systems810,820, and830over network850.

As detailed above, computing system710and/or one or more components of network architecture800may perform and/or be a means for performing, either alone or in combination with other elements, one or more steps of an exemplary method for deduplicating archive objects.

In addition, one or more of the modules described herein may transform data, physical devices, and/or representations of physical devices from one form to another. For example, one or more of the modules recited herein may receive archive data to be transformed, transform the data by deduplicating, reordering, and/or re-prioritizing the data, output a result of the transformation to memory or an output device such as a display, use the result of the transformation to conserve storage space, and store the result of the transformation to disk or memory. Additionally or alternatively, one or more of the modules recited herein may transform a processor, volatile memory, non-volatile memory, and/or any other portion of a physical computing device from one form to another by executing on the computing device, storing data on the computing device, and/or otherwise interacting with the computing device.