Parallel processing of input data to locate landmarks for chunks

Input data is divided into a plurality of segments, which are processed, in parallel, by respective first processing elements to locate landmarks in the segments. At least one other processing element is used to produce chunks from the input data based on positions of the landmarks provided by the first processing elements.

BACKGROUND

As the amount of information to be stored and transmitted by computer systems or other electronic devices has dramatically increased, techniques have been developed to allow for more efficient data storage and processing. In some cases, chunking algorithms have been used to achieve improved efficiency and speed. Chunking algorithms partition one or more data objects into non-overlapping chunks. By dividing one or more data objects into chunks, a system is able to identify chunks that are shared by more than one data object or occur multiple times in the same data object, such that these shared chunks are stored just once to avoid or reduce the likelihood of storing duplicate data.

One type of chunking algorithm is a landmark chunking algorithm, which performs partitioning of one or more data objects by first locating landmarks present in the one or more data objects. The landmarks are short predefined patterns of data whose locations are used in determining chunk boundaries. By convention, each landmark is considered to occur at a single position, often the position immediately following that landmark's data.

The landmark chunking algorithm then determines chunk boundaries from the landmark locations. The simplest landmark chunking algorithm places a chunk boundary at each landmark. More complicated landmark chunking algorithms take into account the distance between landmarks in order to, for example, avoid too small or too large chunks. Note that for such algorithms, not all landmarks will be designated chunk boundaries and not all chunk boundaries are located at landmark positions. In one example, a landmark may be considered to be located at any position in a data stream immediately following a new line character (the pattern). Landmark chunking a text file using the new line character as the landmark definition would partition the text file into a sequence of chunks, where lines of the text file may be separate chunks. Landmark definitions that are actually used in practice tend to be more complicated to enable proper handling of file types other than text files. For example, a position in a data stream can be defined as a landmark location if the immediately preceding 48 bytes of data has a particular calculated value, such as a Rabin fingerprint equal to −1 mod a predefined number related to the average desired chunk size.

A benefit of landmark chunking algorithms is that local changes are likely to disturb only a small number of chunks. For example, in a text file, adding a word to one line in the middle of the document only disturbs that line (chunk). In contrast, if a text file were to be simply divided into fixed-size 80-character records, an added word in the text file would cause every record after the added word to be changed, which leads to increased computer processing.

Conventional landmark chunking algorithms that are applied to large input data can be very computing-intensive. For example, in the data backup or archiving context, relatively large amounts of data are processed during the backup or archiving operation. If the landmark chunking algorithm is not performed efficiently, then the backup or archiving operation may take a long time to complete.

DETAILED DESCRIPTION

In accordance with some embodiments, to improve the efficiency and processing speed of a chunking algorithm, input data is divided into multiple segments, and the multiple segments are processed, in parallel, by respective processing elements to locate landmarks in the segments. A landmark refers to a short sequence of bytes found within the input data that satisfies predetermined criteria. An example of the predetermined criteria includes applying a function to a portion of the input data—the short sequence of bytes—and determining whether the output of the function has some relation with respect to a threshold. Another example of the predetermined criteria includes determining whether a portion of the input data has certain characteristics (e.g., the input data portion has a certain character, is a new line, is a section break, etc.). The predetermined criteria may deal only with the intrinsic qualities of the byte sequence itself rather than extrinsic qualities like the location of the byte sequence or the data around the byte sequence. The locations of landmarks are used to derive chunk boundaries that are the endpoints of chunks of the input data. The task of identifying landmarks in the input data is typically the most computing-intensive part of any landmark chunking algorithm. By processing multiple segments in parallel to identify landmarks using multiple processing elements, the processing speed of a landmark chunking algorithm can be enhanced.

A “chunk” refers to an element of a partition of input data, where the input data can be in the form of a file or other data object. As examples, the input data can be a document (such as a document produced or edited by a software application), an image file, a video file, an audio file, a tape image, or any other collection or sequence of data. A “boundary” or “chunk boundary” refers to a position within the input data that defines an endpoint of a chunk. Two successive boundaries define the two endpoints of a corresponding chunk.

As noted above, a landmark chunking algorithm is used, in which the landmarks identified by the landmark chunking algorithm are used to derive the boundaries that define corresponding chunks. Note that not all landmarks necessarily define boundaries, and not all boundaries are landmarks. Landmarks are defined based on local content of the input data. For example, one technique of locating landmarks is to use a sliding window algorithm where, for each position within the input data, a fingerprint is computed for the sequence of data within the respective sliding window. The sliding window contains bytes within the input data that precedes the position of the input data being considered. If the computed fingerprint satisfies a particular criterion, the position is designated as a landmark. In one specific example, a position in the input file is a landmark if the immediately preceding 48 bytes (sliding window) have a Rabin fingerprint equal to −1 mod a predefined number related to the average desired chunk size. In other implementations, other fingerprints or other values computed from other functions can be computed based on the content of the input data. As yet another implementation, the landmarks can be predefined characters or other types of objects within the input data, such as a new line character, a paragraph break, a page break, and so forth.

Based on the identified landmarks, chunk boundaries can be determined using a landmark chunking algorithm. In some algorithms, to improve performance of chunking, minimum and maximum chunk sizes can be set. Thus, any landmark that is closer than a minimum chunk size following the previous chunk boundary is disregarded, since using such landmark as a chunk boundary would result in a chunk that would be considered too small.

Moreover, chunk sizes may not be allowed to exceed a maximum chunk size. If the chunking algorithm detects that no landmark has been detected in a region of the input data that starts at the previous chunk boundary and ends at a position that corresponds to the maximum chunk size, then the position corresponding to the maximum chunk size from the previous chunk boundary can be used as the next chunk boundary, in one implementation.

In an alternative implementation, the concept of primary and secondary landmarks can be used. Primary and secondary landmarks are computed based on the same input data value (such as the value of input data within a sliding window) but using different predetermined criteria—the concept here is that a secondary landmark is easier to detect than a primary landmark such that a secondary landmark would be more likely to be present within an input data region than a primary landmark. If a primary landmark is found in a region of input data that is between [previous chunk boundary+minimum chunk size] and [previous chunk boundary+maximum chunk size], then the first such primary landmark would be used as the next chunk boundary. The position [previous chunk boundary+minimum chunk size] refers to a position that is equal to the position of the previous chunk boundary plus the minimum chunk size. The position [previous chunk boundary+maximum chunk size] refers to a position that is equal to the position of the previous chunk boundary plus the maximum chunk size.

If a primary landmark is not found in the region between [previous chunk boundary+minimum chunk size] and [previous chunk boundary+maximum chunk size], then the chunking algorithm determines if a secondary landmark is present in the region—if so, then the last such secondary landmark is used as the chunk boundary. If no secondary landmark is found in this region, then the position [previous chunk boundary+maximum chunk size] is used as the next chunk boundary.

Details regarding the above chunking algorithm in which minimum and maximum chunk sizes are set and in which primary and secondary landmarks are generated are described further in U.S. Pat. No. 7,269,689. In other embodiments, other chunking algorithms for determining chunk boundaries can be employed.

The multiple processing elements to process respective segments of the input data in parallel to find landmarks (primary and/or secondary landmarks) can be distinct processors that are provided in one or more computers or other types of electronic devices (e.g., storage devices, communications devices, etc.). Alternatively, the multiple processing elements can be separate cores of a multi-core processor. The multiple cores of a processor are independent cores that are packaged into a single integrated circuit, such as a single die or multiple dies that are packaged together. Each core of the processor is able to independently perform execution of a respective set of instructions, such as instructions of distinct threads or processes.

As noted above, the most computing-intensive aspect of a landmark chunking algorithm can be the computations involved in finding landmarks (primary and/or secondary landmarks). By locating landmarks using multiple processing elements in parallel, the landmarks of a relatively large input data can be more quickly located, such that the landmark chunking algorithm can proceed at a more rapid pace.

FIG. 1illustrates an exemplary arrangement that includes computers100connected over a network102to a storage system104. In one example, the storage system can be a backup storage system or an archiving storage system to back up or archive data106contained in respective computers100. InFIG. 1, components of one of the computers100are depicted, with the other computers containing the same or similar components.

Although multiple computers100are shown, it is noted that a single computer100can be used in another implementation.

Each computer100includes a storage108(e.g., disk-based storage, integrated circuit storage, etc.) to store the data106. A central processing unit (or multiple central processing units)110is (are) connected to the storage108, and software112(application software, operating system software, etc.) is executable on the CPU(s)110.

A network interface114is provided in each computer100to communicate the data106over the network102to the storage system104. The storage system104accepts the data106from the multiple computers100as input data that is subjected to a landmark chunking algorithm, in accordance with some embodiments. Although the chunking algorithm according to some embodiments is performed at the storage system104in one specific example, it is noted that the chunking algorithm can alternatively be performed elsewhere in the network arrangement depicted inFIG. 1, such as in any one of the computers100. The landmark chunking algorithm according to some embodiments is advantageously performed in the storage system104since the storage system104is used to store data associated with multiple computers100, such that the amount data that is processed at the storage system104can be relatively large.

Input data from the computers100communicated over the network102is received by a network interface116in the storage system104. The network interface116is in turn connected to a processor118, which has multiple cores120_1,120_2,120_3, and120_4. The number of cores in the processor118can differ in different implementations.

Although the storage system104is depicted as including a multi-core processor118, it is noted that in an alternative embodiment, multiple distinct processors118can be provided in the storage system104, where each of such distinct processors118can be a single-core or multi-core processor.

In accordance with some embodiments, multiple landmark locating software instances122_1,122_2, and122_3are executable on corresponding cores120_1,120_2, and120_3of the processor118to locate landmarks in corresponding segments of the input data. The landmark locating software instances122_1to122_3can be multiple threads or processes that are spawned from a landmark locating software module124stored in a storage128in the storage system104. The multiple landmark locating software instances122_1,122_2, and122_3spawned from the landmark locating software module124are executed in parallel on the cores120_1to120_3of the processor118to process respective segments of the input data.

In addition to the landmark locating software instances122_1to122_3, a chunking software instance130is spawned from a chunking software module132in the storage128. The chunking software instance130is executable on the core120_4of the processor118to perform chunking of the input data to produce chunks based on the landmarks identified by the landmark locating software instances122_1to122_3. The unique chunks are stored as chunks134in a database136or other storage structure of the storage128. A benefit of chunking input data into chunks is that deduplication of data can be applied. Data objects may share common data portions, By partitioning data objects into chunks, the chunks corresponding to the common data portions can be stored just once to avoid storing duplicate data, which is wasteful of storage resources.

The output produced by each landmark locating software instance122_1to122_3is a landmark list that lists landmarks identified by the respective landmark locating software instance in the corresponding segment of the input data. InFIG. 1, three landmark lists138_1to138_3are created by the three respective landmark locating software instances122_1to122_3. Alternatively, the landmarks identified by the multiple landmark locating software instances can be provided in one landmark list. The landmark list(s) is (are) accessed by the chunking software instance130to produce the chunks134.

FIG. 2is a flow diagram of a chunking procedure in accordance with an embodiment. Input data is received (at202), such as input data received by the storage system104from multiple computers100. The received input data is divided (at204) into multiple segments. Assuming there are n processing elements (such as the n cores in the processor118ofFIG. 1), the input data is divided into n−1 segments, in one embodiment. In alternative embodiments, the input data can be divided into n−2 segments, or less.

The multiple segments are processed, in parallel, by corresponding processing elements to locate landmarks in the respective segments. Thus, in the example ofFIG. 1, the landmark locating software instances122_1to122_3when executed on respective cores120_1to120_3process corresponding multiple segments of the input data to locate landmarks in such segments. The located landmarks are then listed (at208) in corresponding landmark lists, such as the landmark lists138_1to138_3inFIG. 1. The listed landmarks can include primary landmarks and/or secondary landmarks. In other embodiments, there is just one type of landmark. If primary and secondary landmarks are used, then each landmark list can include indicators of whether an identified landmark is a primary landmark or secondary landmark.

Next, the chunking software instance130creates (at210) chunks using the landmark lists created by the landmark locating software instances122_1to122_3. The landmarks contained in the landmark lists are used to derive boundaries of chunks. In one example, minimum and maximum chunk size thresholds can be set, as described above, where any landmarks located in a region between a previous chunk boundary and a position corresponding to a minimum chunk size after the previous chunk boundary are disregarded. Once landmarks are identified (which can involve computing-intensive operations such as calculations of fingerprints or other functions), then the chunk software instance130can quickly produce chunks based on the identified landmarks.

In one example, assuming that the input data is to be divided into three segments, then the input data can be divided as follows: ABCABCABCABCABC . . . . In this example, instead of dividing the input data into three long discrete segments, the three segments are interleaved with each other. In other words, each segment A, B, or C is divided into pieces that are interleaved. In this example, segment A is processed by a first core (core A), segment B is processed by a second core (core B), and segment C is processed by a third core (core C). Core A searches each piece labeled A for landmarks, and produces a list of landmarks for the data of these pieces. In one embodiment, core A starts at the left and works towards the right within each piece of segment A. Core B performs similar operations on pieces labeled B, and core C performs a similar procedure on pieces labeled C.

The core executing the chunking software instance130uses the landmark lists produced by the other cores as input, so that the core executing the chunking software instance130does not have to scan the input data to locate landmarks. Note that the chunking software instance130can actually be executed in parallel with the landmark locating software instances122_1to122_3, such that the chunking software instance130is able to access the landmark lists while the landmark lists are being updated by the landmark locating software instances.

As the chunking software instance130proceeds from piece to piece, it may switch back and forth between the various landmark lists. For example, it may use the landmark list created by Core A when dealing with a piece A then use the landmark list created by Core B to deal with a following piece B.

In some embodiments, concurrency control is performed so that the core executing the chunking software instance130does not attempt to access a part of a landmark list before that part of that landmark list is ready. Thus, initially, the core executing the chunking software instance130may have to wait until core A has finished its first piece before the core executing the chunking software instance130can begin the chunking procedure. If finer-grained concurrency control is used, the core executing the chunking software instance may only have to wait until core A has produced some landmark information.

Instructions of software described above (including the landmark locating software instances122_1to122_3and chunking software instance130ofFIG. 1) are loaded for execution on respective processing elements (such as processor cores120_1to120_4inFIG. 1). A processing element can include a microprocessor, a core of a microprocessor, a microcontroller, a processor module or subsystem (including one or more microprocessors or microcontrollers), or other control or computing devices.