Source: http://www.google.com/patents/US7603391?dq=5,867,764
Timestamp: 2016-12-04 23:42:20
Document Index: 635472218

Matched Legal Cases: ['art 1102', 'art 1104', 'art 1102', 'art 1102', 'art 1104', 'art 1', 'art 1', 'art 1104', 'art 1', 'art 1', 'art 1104', 'art 1', 'art 1', 'art 1104', 'art 1102', 'art 1104', 'art 1']

Patent US7603391 - System and method for determining changes in two snapshots and for ... - Google PatentsSearch Images Maps Play YouTube News Gmail Drive More »Sign inPatentsA system and method for remote asynchronous replication or mirroring of changes in a source file system snapshot in a destination replica file system using a scan (via a scanner) of the blocks that make up two versions of a snapshot of the source file system, which identifies changed blocks in the respective...http://www.google.com/patents/US7603391?utm_source=gb-gplus-sharePatent US7603391 - System and method for determining changes in two snapshots and for transmitting changes to a destination snapshotAdvanced Patent SearchTry the new Google Patents, with machine-classified Google Scholar results, and Japanese and South Korean patents.Publication numberUS7603391 B1Publication typeGrantApplication numberUS 11/336,021Publication dateOct 13, 2009Filing dateJan 20, 2006Priority dateMar 19, 2002Fee statusPaidAlso published asUS6993539, US7818299, US20030182313Publication number11336021, 336021, US 7603391 B1, US 7603391B1, US-B1-7603391, US7603391 B1, US7603391B1InventorsMichael L. Federwisch, Shane S. Owara, Stephen L. Manley, Steven R. KleimanOriginal AssigneeNetapp, Inc.Export CitationBiBTeX, EndNote, RefManPatent Citations (37), Non-Patent Citations (25), Referenced by (57), Classifications (17), Legal Events (1) External Links: USPTO, USPTO Assignment, EspacenetSystem and method for determining changes in two snapshots and for transmitting changes to a destination snapshot
A system and method for remote asynchronous replication or mirroring of changes in a source file system snapshot in a destination replica file system using a scan (via a scanner) of the blocks that make up two versions of a snapshot of the source file system, which identifies changed blocks in the respective snapshot files based upon differences in volume block numbers identified in a scan of the logical file block index of each snapshot. Trees of blocks associated with the files are traversed, bypassing unchanged pointers between versions and walking down to identify the changes in the hierarchy of the tree. These changes are transmitted to the destination mirror or replicated snapshot. This technique allows regular files, directories, inodes and any other hierarchical structure to be efficiently scanned to determine differences between versions thereof. The changes in the files and directories are transmitted over the network for update of the replicated destination snapshot in an asynchronous (lazy write) manner. The changes are described in an extensible, system-independent data stream format layered under a network transport protocol. At the destination, source changes are used to update the destination snapshot. Any deleted or modified inodes already on the destination are moved to a temporary or “purgatory” directory and, if reused, are relinked to the rebuilt replicated snapshot directory. The source file system snapshots can be representative of a volume sub-organization, such as a qtree.
Another type of file system is a write-anywhere file system that does not over-write data on disks. If a data block on disk is retrieved (read) from disk into memory and “dirtied” with new data, the data block is stored (written) to a new location on disk to thereby optimize write performance. A write-anywhere file system may initially assume an optimal layout such that the data is substantially contiguously arranged on disks. The optimal disk layout results in efficient access operations, particularly for sequential read operations, directed to the disks. A particular example of a write-anywhere file system that is configured to operate on a filer is the Write Anywhere File Layout (WAFL™) file system available from Network Appliance, Inc. of Sunnyvale, Calif. The WAFL file system is implemented within a microkernel as part of the overall protocol stack of the filer and associated disk storage. This microkernel is supplied as part of Network Appliance's Data ONTAP™ software, residing on the filer, that processes file-service requests from network-attached clients.
In order to improve reliability and facilitate disaster recovery in the event of a failure of a filer, its associated disks' or some portion of the storage infrastructure, it is common to “mirror” or replicate some or all of the underlying data and/or the file system that organizes the data. In one example, a mirror is established and stored at a remote site, making it more likely that recovery is possible in the event of a true disaster that may physically damage the main storage location or it's infrastructure (e.g. a flood, power outage, act of war, etc.). The mirror is updated at regular intervals, typically set by an administrator, in an effort to catch the most recent changes to the file system. One common form of update involves the use of a “snapshot” process in which the active file system at the storage site, consisting of inodes and blocks, is captured and the “snapshot” is transmitted as a whole, over a network (such as the well-known Internet) to the remote storage site. Generally, a snapshot is an image (typically read-only) of a file system at a point in time, which is stored on the same primary storage device as is the active file system and is accessible by users of the active file system. By “active file system” it is meant the file system to which current input/output operations are being directed. The primary storage device, e.g., a set of disks, stores the active file system, while a secondary storage, e.g. a tape drive, may be utilized to store backups of the active file system. Once snapshotted, the active file system is reestablished, leaving the snapshotted version in place for possible disaster recovery. Each time a snapshot occurs, the old active file system becomes the new snapshot, and the new active file system carries on, recording any new changes. A set number of snapshots may be retained depending upon various time-based and other criteria. The snapshotting process is described in further detail in United States Patent Publication No. US2002/0083037, entitled INSTANT SNAPSHOT by Blake Lewis et al., which is hereby incorporated by reference as though fully set forth herein. In addition, the native Snapshot™ capabilities of the WAFL file system are further described in TR3002 File System Design for an NFS File Server Appliance by David Hitz et al., published by Network Appliance, Inc., and in commonly owned U.S. Pat. No. 5,819,292 entitled METHOD FOR MAINTAINING CONSISTENT STATES OF A FILE SYSTEM AND FOR CREATING USER-ACCESSIBLE READ-ONLY COPIES OF A FILE SYSTEM by David Hitz et al., which are hereby incorporated by reference.
The complete recopying of the entire file system to a remote (destination) site over a network may be quite inconvenient where the size of the file system is measured in tens or hundreds of gigabytes (even terabytes). This full-backup approach to remote data replication may severely tax the bandwidth of the network and also the processing capabilities of both the destination and source filer. One solution has been to limit the snapshot to only portions of a file system volume that have experienced changes. Hence, FIG. 1 shows a prior art volume-based mirroring where a source file system 100 is connected to a destination storage site 102 (consisting of a server and attached storage—not shown) via a network link 104. The destination 102 receives periodic snapshot updates at some regular interval set by an administrator. These intervals are chosen based upon a variety of criteria including available bandwidth, importance of the data, frequency of changes and overall volume size.
In brief summary, the source creates a pair of time-separated snapshots of the volume. These can be created as part of the commit process in which data is committed to non-volatile memory in the filer or by another mechanism. The “new” snapshot 110 is a recent snapshot of the volume's active file system. The “old” snapshot 112 is an older snapshot of the volume, which should match the image of the file system replicated on the destination mirror. Note, that the file server is free to continue work on new file service requests once the new snapshot 112 is made. The new snapshot acts as a checkpoint of activity up to that time rather than an absolute representation of the then-current volume state. A differencer 120 scans the blocks 122 in the old and new snapshots. In particular, the differencer works in a block-by-block fashion, examining the list of blocks in each snapshot to compare which blocks have been allocated. In the case of a write-anywhere system, the block is not reused as long as a snapshot references it, thus a change in data is written to a new block. Where a change is identified (denoted by a presence or absence of an ‘X’ designating data), a decision process 200, shown in FIG. 2, in the differencer 120 decides whether to transmit the data to the destination 102. The process 200 compares the old and new blocks as follows: (a) Where data is in neither an old nor new block (case 202) as in old/new block pair 130, no data is available to transfer. (b) Where data is in the old block, but not the new (case 204) as in old/new block pair 132, such data has already been transferred, (and any new destination snapshot pointers will ignore it), so the new block state is not transmitted. (c) Where data is pre-sent in the both the old block and the new block (case 206) as in the old/new block pair 134, no change has occurred and the block data has already been transferred in a previous snapshot. (d) Finally, where the data is not in the old block, but is in the new block (case 208) as in old/new block pair 136, then a changed data block is transferred over the network to become part of the changed volume snapshot set 140 at the destination as a changed block 142. In the exemplary write-anywhere arrangement, the changed blocks are written to new, unused locations in the storage array. Once all changed blocks are written, a base file system information block, that is the root pointer of the new snapshot, is then committed to the destination. The transmitted file system information block is committed, and updates the overall destination file system by pointing to the changed block structure in the destination, and replacing the previous file system information block. The changes are at this point committed as the latest incremental update of the destination volume snapshot. This file system accurately represents the “new” snapshot on the source. In time a new “new” snapshot is created from further incremental changes.
According to an illustrative embodiment, the source scans, with the scanner, along the index of logical file blocks for each snapshot looking for changed volume block numbers between the two source snapshots. Since disk blocks are always rewritten to new locations on the disk, a difference indicates changes in the underlying inodes of the respective blocks. Using the scanner, unchanged blocks are efficiently overlooked, as their inodes are unchanged. Using an inode picker process, that receives changed blocks from the scanner the source picks out inodes from changed blocks specifically associated with the selected qtree (or other sub-organization of the volume). The picker process looks for versions of inodes that have changed between the two snapshots and picks out the changed version. If inodes are the same, but files have changed (based upon different generation numbers in the inodes) the two versions of the respective inodes are both picked out. The changed versions of the inodes (between the two snapshots) are queued and transferred to a set of inode handlers/workers or handlers that resolve the changes in underlying blocks by continuing to scan (with the scanner, again) file offsets down “trees” of the inodes until differences in underlying blocks are identified via their block pointers, as changed inodes in one version will point to different data blocks than those in the other version. Only the changes in the trees are transmitted over the network for update of the destination file system in an asynchronous (lazy write) manner. The destination file system is exported read-only the user. This ensures that only the replicator can alter the state of the replica file system.
In an illustrative embodiment, a file system-independent format is used to transmit a data stream of change data over the network. This format consists of a set of standalone headers with unique identifiers. Some headers refer to follow-on data and others carry relevant data within their stream. For example, the information relating to any source snapshot deleted files are carried within “deleted files” headers. All directory activity is transmitted first, followed by file data. File data is sent in chunks of varying size, separated by regular headers until an ending header (footer) is provided. At the destination, the format is unpacked and inodes contained therein are transmitted over the network are mapped to a new directory structure. Received file data blocks are written according to their offset in the corresponding destination file. An inode map stores entries which map the source's inodes (files) to the destination's inodes (files). The inode map also contains generation numbers. The tuple of (inode number, generation number) allows the system to create a file handle for fast access to a file. It also allows the system to track changes in which a file is deleted and its inode number is reassigned to a newly created file. To facilitate construction of a new directory tree on the destination, an initial directory stage of the destination mirror process receives source directory information via the format and moves any deleted or moved files to a temporary or “purgatory” directory. The purgatory files which have been moved are hard linked from the purgatory directory to the directories where they have been moved to. Newly created source files are entered into map and built into the directory tree. After the directory tree is built, the transfer of file data begins. Changes to file data from the source are written to the corresponding replica files (as identified by the inode map). When the data stream transfer is complete, the purgatory directory is removed and any unlinked files (including various deleted files) are permanently deleted. In one embodiment, a plurality of discrete source qtrees or other sub-organizations derived from different source volumes can be replicated/mirrored on a single destination volume.
By way of further background, FIG. 3 is a schematic block diagram of a storage system environment 300 that includes a pair of interconnected file servers including a source file server 310 and a destination file server 312 that may each be advantageously used with the present invention. For the purposes of this description, the source file server is a networked computer that manages storage one or more source volumes 314, each having an array of storage disks 360 (described further below). Likewise, the destination filer 312 manages one or more destination volumes 316, also comprising arrays of disks 360. The source and destination file servers or “filers” are linked via a network 318 that can comprise a local or wide area network, such as the well-known Internet. An appropriate network adapter 330 residing in each filer 310, 312 facilitates communication over the network 318. Also for the purposes of this description, like components in each of the source and destination filer, 310 and 312 respectively, are described with like reference numerals. As used herein, the term “source” can be broadly defined as a location from which the subject data of this invention travels and the term “destination” can be defined as the location to which the data travels. While a source filer and a destination filer, connected by a network, is a particular example of a source and destination used herein, a source and destination could be computers/filers linked via a direct link, or via loopback (a “networking” arrangement internal to a single computer for transmitting a data stream between local source and local destination), in which case the source and the destination are the same filer. As will be described further below, the source and destination are broadly considered to be a source sub-organization of a volume and a destination sub-organization of a volume. Indeed, in at least one special case the source and destination sub-organizations can be the same at different points in time.
It will be understood to those skilled in the art that the inventive technique described herein may apply to any type of special-purpose computer (e.g., file serving appliance) or general-purpose computer, including a standalone computer, embodied as a storage system. To that end, the filers 310 and 312 can each be broadly, and alternatively, referred to as storage systems. Moreover, the teachings of this invention can be adapted to a variety of storage system architectures including, but not limited to, a network-attached storage environment, a storage area network and disk assembly directly-attached to a client/host computer. The term “storage system” should, therefore, be taken broadly to include such arrangements.
In the illustrative embodiment, the memory 325 comprises storage locations that are addressable by the processor and adapters for storing software program code. The memory comprises a form of random access memory (RAM) that is generally cleared by a power cycle or other reboot operation (i.e., it is “volatile” memory). The processor and adapters may, in turn, comprise processing elements and/or logic circuitry configured to execute the software code and manipulate the data structures. The operating system 400, portions of which are typically resident in memory and executed by the processing elements, functionally organizes the filer by, inter alia, invoking storage operations in support of a file service implemented by the filer. It will be apparent to those skilled in the art that other processing and memory means, including various computer readable media, may be used for storing and executing program instructions pertaining to the inventive technique described herein.
To facilitate generalized access to the disks 360, the storage operating system 400 (FIG. 4) implements a write-anywhere file system that logically organizes the information as a hierarchical structure of directories and files on the disks. Each “on-disk” file may be implemented as a set of disk blocks configured to store information, such as data, whereas the directory may be implemented as a specially formatted file in which references to other files and directories are stored. As noted and defined above, in the illustrative embodiment described herein, the storage operating system is the NetApp® Data ONTAP™ operating system available from Network Appliance, Inc., of Sunnyvale, Calif. that implements the Write Anywhere File Layout (WAFL™) file system. It is expressly contemplated that any appropriate file system can be used, and as such, where the term “WAFL” is employed, it should be taken broadly to refer to any file system that is otherwise adaptable to the teachings of this invention.
Bridging the disk software layers with the network and file system protocol layers is a file system layer 450 of the storage operating system 400. Generally, the layer 450 implements a file system having an on-disk format representation that is block-based using, e.g., 4-kilobyte (KB) data blocks and using inodes to describe the files. In response to transaction requests, the file system generates operations to load (retrieve) the requested data from volumes if it is not resident “in-core”, i.e., in the filer's memory 325. If the information is not in memory, the file system layer 450 indexes into the inode file using the inode number to access an appropriate entry and retrieve a volume block number. The file system layer 450 then passes the volume block number to the disk storage (RAID) layer 440, which maps that volume block number to a disk block number and sends the latter to an appropriate driver (for example, an encapsulation of SCSI implemented on a fibre channel disk interconnection) of the disk driver layer 445. The disk driver accesses the disk block number from volumes and loads the requested data in memory 325 for processing by the filer 310, 312. Upon completion of the request, the filer (and storage operating system) returns a reply, e.g., a conventional acknowledgement packet 374 defined by the CIFS specification, to the client 370 over the respective network connection 372.
It should be noted that the software “path” 470 through the storage operating system layers described above needed to perform data storage access for the client request received at the filer may alternatively be implemented in hardware or a combination of hardware and software. That is, in an alternate embodiment of the invention, the storage access request data path 470 may be implemented as logic circuitry embodied within a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). This type of hardware implementation increases the performance of the file service provided by filer 310, 312 in response to a file system request packet 374 issued by the client 370.
The inherent Snapshot™ capabilities of the exemplary WAFL file system are further described in TR3002 File System Design for an NFS File Server Appliance by David Hitz et al., published by Network Appliance, Inc., which is hereby incorporated by reference. Note, “Snapshot” is a trademark of Network Appliance, Inc. It is used for purposes of this patent to designate a persistent consistency point (CP) image. A persistent consistency point image (PCPI) is a point-in-time representation of the storage system, and more particularly, of the active file system, stored on a storage device (e.g., on disk) or in other persistent memory and having a name or other unique identifiers that distinguishes it from other PCPIs taken at other points in time. A PCPI can also include other information (metadata) about the active file system at the particular point in time for which the image is taken. The terms “PCPI” and “snapshot” shall be used interchangeably through out this patent without derogation of Network Appliance's trademark rights.
An exemplary file system inode structure 500 according to an illustrative embodiment is shown in FIG. 5. The inode for the inode file or more generally, the “root” inode 505 contains information describing the inode file 508 associated with a given file system. In this exemplary file system inode structure root inode 505 contains a pointer to the inode file indirect block 510. The inode file indirect block 510 points to one or more inode file direct blocks 512, each containing a set of pointers to inodes 515 that make up the inode file 508. The depicted subject inode file 508 is organized into volume blocks (not separately shown) made up of inodes 515 which, in turn, contain pointers to file data (or “disk”) blocks 520A, 520B and 520C. In the diagram, this is simplified to show just the inode itself containing pointers to the file data blocks. Each of the file data blocks 520(A-C) is adapted to store, in the illustrative embodiment, 4 kilobytes (KB) of data. Note, however, where more than a predetermined number of file data blocks are referenced by an inode (515) one or more indirect blocks 525 (shown in phantom) are used. These indirect blocks point to associated file data blocks (not shown). If an inode (515) points to an indirect block, it cannot also point to a file data block, and vice versa.
In accordance with an illustrative embodiment of this invention the source utilizes two snapshots, a “base” snapshot, which represents the image of the replica file system on the destination, and an “incremental” snapshot, which is the image that the source system intends to replicate to the destination, to perform needed updates of the remote snapshot mirror to the destination. In one example, from the standpoint of the source, the incremental snapshot can comprise a most-recent snapshot and the base can comprise a less-recent snapshot, enabling an up-to-date set of changes to be presented to the destination. This procedure shall now be described in greater detail.
Having described the general procedure for deriving a snapshot, the mirroring of snapshot information from the source filer 310 (FIG. 3) to a remote destination filer 312 is described in further detail. As discussed generally above, the transmission of incremental changes in snapshot data based upon a comparison of changed blocks in the whole volume is advantageous in that it transfers only incremental changes in data rather than a complete file system snapshot, thereby allowing updates to be smaller and faster. However, a more efficient and/or versatile procedure for incremental remote update of a destination mirror snapshot is contemplated according to an illustrative embodiment of this invention. Note, as used herein the term “replica snapshot,” “replicated snapshot” or “mirror snapshot” shall be taken to also refer generally to the file system on the destination volume that contains the snapshot where appropriate (for example where a snapshot of a snapshot is implied.
As indicated above, it is contemplated that this procedure can take advantage of a sub-organization of a volume known as a qtree. A qtree acts similarly to limits enforced on collections of data by the size of a partition in a traditional Unix® or Windows® file system, but with the flexibility to subsequently change the limit, since qtrees have no connection to a specific range of blocks on a disk. Unlike volumes, which are mapped to particular collections of disks (e.g. RAID groups of n disks) and act more like traditional partitions, a qtree is implemented at a higher level than volumes and can, thus, offer more flexibility. Qtrees are basically an abstraction in the software of the storage operating system. Each volume may, in fact, contain multiple qtrees. The granularity of a qtree can be a sized to just as a few kilobytes of storage. Qtree structures can be defined by an appropriate file system administrator or user with proper permission to set such limits.
Before describing further the process of deriving changes in two source snapshots, from which data is transferred to a destination for replication of the source at the destination, general reference is made again to the file block structures shown in FIGS. 5-7. Every data block in a file is mapped to disk block (or volume block). Every disk/volume block is enumerated uniquely with a discrete volume block number (VBN). Each file is represented by a single inode, which contains pointers to these data blocks. These pointers are VBNs—each pointer field in an inode having a VBN in it, whereby a file's data is accessed by loading up the appropriate disk/volume block with a request to the file system (or disk control) layer. When a file's data is altered, a new disk block is allocated to store the changed data. The VBN of this disk block is placed in the pointer field of the inode. A snapshot captures the inode at a point in time, and all the VBN fields in it.
In order to scale beyond the maximum number of VBN “pointers” in an inode, “indirect blocks” are used. In essence, a disk block is allocated and filled with the VBNs of the data blocks, the inode pointers then point to the indirect block. There can exist several levels of indirect blocks, which can create a large tree structure. Indirect blocks are modified in the same manner as regular data blocks are—every time a VBN in an indirect block changes, a new disk/volume block is allocated for the altered data of the indirect block.
In the example of a write-anywhere file layout, storage blocks are not immediately overwritten or reused. Thus changes in a file comprised of a series of volume blocks will always result in the presence of a new volume block number (newly written-to) that can be detected at the appropriate logical file block offset relative to an old block. The existence of a changed volume block number at a given offset in the index between the base snapshot inode file and incremental snapshot inode file generally indicates that one or more of the underlying inodes and files to which the inodes point have been changed. Note, however, that the system may rely on other indicators of changes in the inodes or pointers—this may be desirable where a write-in-place file system is implemented.
Block pairs (e.g. blocks 7 and 8) that have been identified as changed are forwarded (as they are detected by the scan/scanner 820) to the rest of the source process, which includes an inode picker process 830. The inode picker identifies specific inodes (based upon qtree ID) from the forwarded blocks that are part of the selected qtree being mirrored. In this example the qtree ID Q2 is selected, and inodes containing this value in their file metadata are “picked” for further processing. Other inodes not part of the selected qtree(s) (e.g. inodes with qtree IDs Q1 and Q3) are discarded or otherwise ignored by the picker process 830. Note that a multiplicity of qtree IDs can be selected, causing the picker to draw out a group of inodes—each having one of the selected qtree associations.
The appropriately “picked” inodes from changed blocks are then formed into a running list or queue 840 of changed inodes 842. These inodes are denoted by a discrete inode number as shown. Each inode in the queue 840 is handed off to an inode handler or worker 850, 852 and 854 as a worker becomes available. FIG. 8A is a table 835 detailing the basic set of rules the inode picker process 830 uses to determine whether to send a given inode to the queue for the workers to process.
The function of the worker is to determine changes between each snapshot's versions of the files and directories. As described above, the source snapshot mirror application is adapted to analyze two versions of inodes in the two snapshots and compares the pointers in the inodes. If the two versions of the pointers point to the same block, we know that that block hasn't changed. By extension, if the pointer to an indirect block has not changed, then that indirect block has no changed data, so none of its pointers can have changed, and, thus, none of the data blocks underneath it in the tree have changed. This means that, in a very large file, which is mostly unchanged between two snapshots, the process can skip over/overlook VBN “pointers” to each data block in the tree to query whether the VBNs of the data blocks have changed.
The operation of a worker 850 is shown by way of example in FIG. 9. Once a changed inode pair are received by the worker 850, each inode (base and incremental, respectively) 910 and 912 is scanned to determine whether the file offset between respective blocks is a match. In this example, blocks 6 and 7 do not match. The scan then continues down the “tree” of blocks 6 and 7, respectively, arriving at underlying indirect blocks 8/9 (920) and 8/10 (922). Again the file offset comparison indicates that blocks 8 both arrive at a common block 930 (and thus have not changed). Conversely, blocks 9 and 10 do not match due to offset differences and point to changed blocks 940 and 942. The changed block 942 and the metadata above can be singled out for transmission to the replicated snapshot on the destination (described below; see also FIG. 8). The tree, in an illustrative embodiment extends four levels in depth, but this procedure may be applied to any number of levels. In addition, the tree may in fact contain several changed branches, requiring the worker (in fact, the above-described scanner 820 process) to traverse each of the branches in a recursive manner until all changes are identified. Each inode worker, thus provides the changes to the network for transmission in a manner also described below. In particular, new blocks and information about old, deleted blocks are sent to the destination. Likewise, information about modified blocks is sent.
With further reference to FIG. 10, the transmission of changes from the source snapshot to the replicated destination snapshot is described in an overview 1000. As already described, the old and new snapshots present the inode picker 830 with changed inodes corresponding to the qtree or other selected sub-organization of the subject volume. The changed inodes are placed in the queue 840, and then their respective trees are walked for changes by a set of inode workers 850, 852 and 854. The inode workers each send messages 1002, 1004 and 1006 containing the change information to a source pipeline 1010. Note that this pipeline is only an example of a way to implement a mechanism for packaging file system data into a data stream and sending that stream to a network layer. The messages are routed first to a receiver 1012 that collects the messages and sends them on to an assembler 1014 as a group comprising the snapshot change information to be transmitted over the network 318. Again, the “network” as described herein should be taken broadly to include anything that facilitates transmission of volume sub-organization (e.g. qtree) change data from a source sub-organization to a destination sub-organization, even where source and destination are on the same file server, volume or, indeed (in the case of rollback as described in the above-incorporated U.S. patent application entitled SYSTEM AND METHOD FOR REMOTE ASYNCHRONOUS MIRRORING USING SNAPSHOTS) are the same sub-organization at different points in time. An example of a “network” used as a path back to the same volume is a loopback. The assembler 1014 generates a specialized format 1020 for transmitting the data stream of information over the network 318 that is predictable and understood by the destination. The networker 1016 takes the assembled data stream and forwards it to a networking layer. This format is typically encapsulated within a reliable networking protocol such as TCP/IP. Encapsulation can be performed by the networking layer, which constructs, for example, TCP/IP packets of the formatted replication data stream
The format 1020 is described further below. In general, its use is predicated upon having a structure that supports multiple protocol attributes (e.g. Unix permissions, NT access control lists (ACLs), multiple file names, NT streams, file type, file-create/modify time, etc.). The format should also identity the data in the stream (i.e. the offset location in a file of specific data or whether files have “holes” in the file offset that should remain free). The names of files should also be relayed by the format. More generally, the format should also be independent of the underlying network protocol or device (in the case of a tape or local disk/non-volatile storage) protocol and file system—that is, the information is system “agnostic,” and not bound to a particular operating system software, thereby allowing source and destination systems of different vendors to share the information. The format should, thus, be completely self-describing requiring no information outside the data stream. In this manner a source file directory of a first type can be readily translated into destination file directory of a different type. It should also allow extensibility, in that newer improvements to the source or destination operating system should not affect the compatibility of older versions. In particular, a data set (e.g. a new header) that is not recognized by the operating system should be ignored or dealt with in a predictable manner without triggering a system crash or other unwanted system failure (i.e. the stream is backwards compatible). This format should also enable transmission of a description of the whole file system, or a description of only changed blocks/information within any file or directory. In addition, the format should generally minimize network and processor overhead.
The destination pipeline 1030 forwards data and directory information to the main destination snapshot mirror process 1040, which is described in detail below. The destination snapshot mirror process 1040 consists of a directory stage 1042, which builds the new replicated file system directory hierarchy on the destination side based upon the received snapshot changes. To briefly summarize, the directory stage creates, removes and moves files based upon the received formatted information. A map of inodes from the destination to the source is generated and updated. In this manner, inode numbers on the source file system are associated with corresponding (but typically different) inode numbers on the destination file system. Notably, a temporary or “purgatory” directory 1050 (described in further detail below) is established to retain any modified or deleted directory entries 1052 until these entries are reused by or removed from the replicated snapshot at the appropriate directory rebuilding stage within the directory stage. In addition, a file stage 1044 of the destination mirror process populates the established files in the directory stage with data based upon information stripped from associated format headers.
The format into which source snapshot changes are organized is shown schematically in FIGS. 11 and 12. In the illustrative embodiment, the format is organized around 4 KB blocks. The header size and arrangement can be widely varied in alternate embodiments, however. There are 4 KB headers (1100 in FIG. 11) that are identified by certain “header types.” Basic data stream headers (“data”) are provided for at most every 2 megabytes (2 MB) of changed data. With reference to FIG. 11, the 4 KB standalone header includes three parts, a 1 KB generic part 1102, a 2 KB non-generic part 1104, and an 1 KB expansion part. The expansion part is not used, but is available for later versions.
The generic part 1102 contains an identifier of header type 1110. Standalone header types (i.e. headers not followed by associated data) can indicate a start of the data stream; an end of part one of the data stream; an end of the data stream; a list of deleted files encapsulated in the header; or the relationship of any NT streamdirs. Later versions of Windows NT allow for multiple NT “streams” related to particular filenames. A discussion of streams is found in U.S. Pat. No. 6,446,653, entitled SYSTEM AND METHOD FOR REPRESENTING NAMED DATA STREAMS WITHIN AN ON-DISK STRUCTURE OF A FILE SYSTEM, by Kayuri Patel, et al, the teachings of which are expressly incorporated herein by reference. Also in the generic part 1102 is a checksum 1112 that ensures the header is not corrupted. In addition other data such as a “checkpoint” 1114 used by the source and destination to track the progress of replication is provided. By providing a list of header types, the destination can more easily operate in a backwards-compatible mode—that is, a header type that is not recognized by the destination (provided from a newer version of the source) can be more easily ignored, while recognized headers within the limits of the destination version are processed as usual.
FIG. 12 describes the format 1020 of the illustrative replication data stream in further detail. The format of the replicated data stream is headed by a standalone data stream header 1202 of the type “start of data stream.” This header contains data in the non-generic part 1104 generated by the source describing the attributes of the data stream.
Next a series of headers and follow-on data in the format 1020 define various “part 1” information (1204). Significantly, each directory data set being transmitted is preceded by a basic header with no non-generic data. Only directories that have been modified are transmitted, and they need not arrive in a particular order. Note also that the data from any particular directory need not be contiguous. Each directory entry is loaded into a 4 KB block. Any overflow is loaded into a new 4 KB block. Each directory entry is a header followed by one or more names. The entry describes an inode and the directory names to follow. NT stream directories are also transmitted.
The part 1 format information 1204 also provides ACL information for every file that has an associated ACL. By transmitting the ACLs before their associated file data, the destination can set ACLs before file data is written. ACLs are transmitted in a “regular” file format. Deleted file information (described above) is sent with such information included in the non-generic part 1104 of one or more standalone headers (if any). By sending this information in advance, the directory tree builder can differentiate between moves and deletes.
Once various part 1 information 1204 is transmitted, the format calls for an “end of part 1 of the data stream” header 1206. This is a basic header having no data in the non-generic part 1104. This header tells the destination that part 1 is complete and to now expect file data.
After the part 1 information, the format presents the file and stream data 1208. A basic header 1210 for every 2 MB or less of changed data in a file is provided, followed by the file data 1212 itself. The files comprising the data need not be written in a particular order, nor must the data be contiguous. In addition, referring to the header in FIG. 11, the basic header includes a block numbers data structure 1130, associated with the non-generic part 1104 works in conjunction with the “holes array” 1132 within (in this example) the generic part 1102. The holes array denotes empty space. This structure, in essence, provides the mapping from the holes array to corresponding blocks in the file. This structure instructs the destination where to write data blocks or holes.
Finally, the end of the replicated data stream format 1020 is marked by a footer 1220 consisting of standalone header of the type “end of data stream.” This header has no specific data in its non-generic part 1104 (FIG. 11).
Next the directory stage undertakes a tree cleaning process (1312). This step removes all directory entries form the replicated snapshot directory 1330 that have been changed on the source snapshot. The data stream format (1020) indicates whether a directory entry has been added or removed. In fact, directory entries from the base version of the directory and directory entries from the incremental version of the directory are both present in the format. The destination snapshot mirror application converts the formatted data stream into a destination directory format in which each entry that includes an inode number, a list of relative names (e.g. various multi-protocol names) and a “create” or “delete” value. In general each file also has associated therewith a generation number. The inode number and the generation number together form a tuple used to directly access a file within the file system (on both the source and the destination). The source sends this tuple information to the destination within the format and the appropriate tuple is stored on the destination system. Generation numbers that are out of date with respect to existing destination files indicate that the file has been deleted on the source. The use of generation numbers is described further below.
The destination processes base directory entries as removals and incremental directory entries as additions. A file which has been moved or renamed is processed as a delete (from the old directory or from the old name), then as an add (to the new directory or with a new name). Any directory entries 1052 that are deleted, or otherwise modified, are moved temporarily to the temporary or “purgatory” directory, and are not accessible in this location by users. The purgatory directory allows modified entries to be, in essence, “moved to the side” rather than completely removed as the active file system's directory tree is worked on. The purgatory directory entries, themselves point to data, and thus prevent the data from becoming deleted or losing a link to a directory altogether.
For efficiency, the source side depends upon inode numbers and directory blocks rather than pathnames. In general, a file in the replicated directory tree (a qtree in this example) on the destination cannot expect to receive the same inode number as the corresponding file has used on the source (although it is possible). As such, an inode map is established in the destination. This map 1400, shown generally in FIG. 14, enables the source to relate each file on the source to the destination. The mapping is based generally upon file offsets. For example a received source block having “offset 20 KB in inode 877” maps to the block at offset 20 KB in replicated destination inode 9912. The block can then be written to the appropriate offset in the destination file.
By maintaining the source generation number, the destination can determine if a file has been modified or deleted on the source (and its source associated inode reallocated), as the source generation number is incremented upwardly with respect to the stored destination. When the source notifies the destination that an inode has been modified, it sends the tuple to the destination. This tuple uniquely identifies the inode on the source system. Each time the source indicates that an entirely new file or directory has to be created (e.g. “create”) the destination file system creates that file. When the file is created, the destination registers data as a new entry in its inode map 1400. Each time the source indicates that an existing file or directory needs to be deleted, the destination obliterates that file, and then clears the entry in the inode map. Notably, when a file is modified, the source only sends the tuple and the data to be applied. The destination loads the source inode's entry from the inode map. If the source generation number matches, then it knows that the file already exists on the destination and needs to be modified. The destination uses the tuple recorded in the inode map to load the destination inode. Finally, it can apply the file modifications by using the inode.
As part of the tree building process reused entries are “moved” back from the purgatory directory to the replicated snapshot directory 1330. Traditionally, a move of a file requires knowledge of the name of the moved file and the name of the file it is being moved to. The original name of the moved file may not be easily available in the purgatory directory. In addition, a full move would require two directories (purgatory and replicated snapshot) to be modified implicating additional overhead.
The new directory tree may contain files with no data or old data. When the “end of part 1” format header is read, the destination mirror process 1040 enters the file stage 1044 in which snapshot data files 1340 referenced by the directory tree are populated with data (e.g. change data). FIG. 15 shows a simplified procedure 1500 for writing file data 1502 received from the source. In general, each (up to) 2 MB of data in 4 KB blocks arrives with corresponding source inode numbers. The inode map 1400 is consulted for corresponding entries 1402. Appropriate offsets 1504 are derived for the data, and it is written into predetermined empty destination snapshot data files 1340.
At the end of both the directory stage 1042 and data stage 1044, when all directory and file data have been processed, and the data stream transfer from the source is complete, the new replicated snapshot is exposed atomically to the user. At this time the contents of the purgatory directory 1050 (which includes any entries that have not be “moved” back into the rebuilt tree) is deleted.
It should be noted that the initial creation (the “level zero” transfer) of the replicated snapshot on the destination follows the general procedures discussed above. The difference between a level zero transfer and a regular update is that there is no base snapshot; so the comparisons always process information in the incremental snapshot as additions and creates rather than modifications. The destination mirror application starts tree building by processing any directories already known to it. The initial directory established in the destination is simply the root directory of the replicated snapshot (the qtree root). A destination root exists on the inode map. The source eventually transmits a root (other files received may be buffered until the root arrives), and the root is mapped to the existing destination root. Files referenced in the root are then mapped in turn in a “create” process as they are received and read by the destination. Eventually, the entire directory is created, and then the data files are populated. After this, a replica file system is complete.
As described above, a source and destination can be the same qtree, typically at different points in time. In this case, it is contemplated that an incremental change to a snapshot can be undone by applying a “rollback” procedure. In essence, the base and incremental snapshot update process described above with reference to FIG. 8 is performed in reverse so as to recover from a disaster, and return the active file system to the state of a given snapshot.
Reference is made to FIG. 16, which describes a generalized rollback procedure 1600 according to an illustrative embodiment. As a matter of ongoing operation, in step 1605, a “first” snapshot is created. This first snapshot may be an exported snapshot of the replicated snapshot on the destination. In the interim, the subject destination active file system (replicated snapshot) is modified by an incremental update from the source (step 1610).
After the halt, a “second” exported snapshot of the modified active file system in its most current state is now created (step 1625).
Next, in step 1630, the incremental changes are computed between the second and the first snapshots. This occurs in accordance with the procedure described above with reference to FIGS. 8 and 9, but using the second snapshot as the base and the first snapshot as the incremental. The computed incremental changes are then applied to the active file system (now frozen in its present state) in step 1635. The changes are applied so that the active file system is eventually “rolled back” to the state contained in the first snapshot (step 1640). This is the active file system state existing before the exigency that necessitated the rollback.
One noted advantage to the rollback according to this embodiment is that it enables the undoing of set of changes to a replicated data set without the need to maintain separate logs or consuming significant system resources. Further the direction of rollback—past-to-present or present-to-past—is largely irrelevant. Furthermore, use of the purgatory directory, and not deleting files, enables the rollback to not affect existing NFS clients. Each NFS client accesses files by means of file handles, containing the inode number and generation of the file. If a system deletes and recreates a file, the file will have a different inode/generation tuple. As such, the NFS client will not be able to access the file without reloading it (it will see a message about a stale file handle). The purgatory directory, however, allows a delay in unlinking files until the end of the transfer. As such, a rollback as described above can resurrect files that have just been moved into purgatory, without the NFS clients taking notice.
One way to provide a source-centric inode map is to perform a “flip” of map entries. FIG. 17 details a procedure 1700 for performing the flip. The flip operation is initiated (step 1705) as part of a rollback initiated as part of a disaster recovery procedure of for other reasons (automatically or under user direction). Next, the destination and source negotiate to transfer the inode map file to the source from the destination. The negotiation can be accomplished using known data transfer methodologies and include appropriate error correction and acknowledgements (step 1710). The inode is thereby transferred to the source from the destination and is stored.
In step 1725, the new destination looks up the Nth inode from the entries associated with the old destination in the stored inode map file (i.e. the map from the old destination/new source). Next, the new destination determines if such an entry exists (decision step 1730). If no entry exists, then a zero entry is created in the new inode map file, representing that the Nth inode of the new source (old destination) is not allocated. However, if there exists an Nth inode of the new source/old destination, then the decision step 1730 branches to step 1740, and creates a new entry in the new inode map file (created in step 1715). The new entry maps the new source (old destination) Nth inode to the proper new destination (old source) inode. Note, in an alternate embodiment, the new inode map is provided with a full field of zero entries before the mapping begins, and the creation of a “zero entry,” in this case should be taken broadly to include leaving a preexisting zero entry in place in the inode map.
In the illustrated example, the Destination Snapshot A (2002) is now prepared to transfer changes so as to establish a mirror in Destination Snapshot B (2004). However, the reverse is also contemplated, i.e. Destination Snapshot B establishing a Mirror in Destination Snapshot A. Thus, Destination Snapshot A (2002) becomes the new “source” in the transfer with Destination Snapshot B (2004) acting as the desired destination system for replication data from Destination Snapshot A. As in the above-described flip embodiment, the new source 2002 transfers its inode map A 2012 to the destination system 2004. The destination system 2004 then determines the relationship between the two system's inodes. In this case, both the new source and the new destination system have their own inode maps A and B (2012 and 2014), indexed off the old source 2001, and referencing the inodes in their respective trees. Given the existence of two respective inode maps, an “associative” process 2016 walks the inode maps concurrently, inode-by-inode. For each inode from the original source 2001, the process extracts the “destination inode/generation number” from each of the inode maps A and B. It then treats the new source as the appropriate map index for the new associated inode map 2018. In the associated map, it stores the new source generation number for the new source index inode number, with each index entry also associated with/mapped to the new destination inode/generation number extracted from the inode map B (2014). The new map is used by the new destination 2004 in accordance with the principles described above to build trees in the directory based upon changes in the new source with respect to various points in time.
Patent CitationsCited PatentFiling datePublication dateApplicantTitleUS4570217Mar 28, 1983Feb 11, 1986Allen Bruce SMan machine interfaceUS5163131Sep 8, 1989Nov 10, 1992Auspex Systems, Inc.Parallel i/o network file server architectureUS5202979Jun 26, 1992Apr 13, 1993Thinking Machines CorporationStorage system using multiple independently mechanically-driven storage unitsUS5278979Dec 20, 1990Jan 11, 1994International Business Machines Corp.Version management system using pointers shared by a plurality of versions for indicating active lines of a versionUS5355453Oct 13, 1992Oct 11, 1994Auspex Systems, Inc.Parallel I/O network file server architectureUS5485579Apr 8, 1994Jan 16, 1996Auspex Systems, Inc.Multiple facility operating system architectureUS5802366Oct 11, 1994Sep 1, 1998Auspex Systems, Inc.Parallel I/O network file server architectureUS5819292 *May 31, 1995Oct 6, 1998Network Appliance, Inc.Method for maintaining consistent states of a file system and for creating user-accessible read-only copies of a file systemUS5835953Nov 8, 1996Nov 10, 1998Vinca CorporationBackup system that takes a snapshot of the locations in a mass storage device that has been identified for updating prior to updatingUS5931918Jul 30, 1997Aug 3, 1999Auspex Systems, Inc.Parallel I/O network file server architectureUS5941972Dec 31, 1997Aug 24, 1999Crossroads Systems, Inc.Storage router and method for providing virtual local storageUS6065037Jun 7, 1995May 16, 2000Auspex Systems, Inc.Multiple software-facility component operating system for co-operative processor control within a multiprocessor computer systemUS6425035Sep 27, 2001Jul 23, 2002Crossroads Systems, Inc.Storage router and method for providing virtual local storageUS6446653 *Jun 26, 2001Sep 10, 2002Bs&B Safety Systems, Inc.Rupture disk assemblyUS6574591 *Oct 25, 1999Jun 3, 2003Network Appliance, Inc.File systems image transfer between dissimilar file systemsUS6604118 *Jul 31, 1998Aug 5, 2003Network Appliance, Inc.File system image transferUS6993539Mar 19, 2002Jan 31, 2006Network Appliance, Inc.System and method for determining changes in two snapshots and for transmitting changes to destination snapshotUS7007046 *Mar 19, 2002Feb 28, 2006Network Appliance, Inc.Format for transmission file system information between a source and a destinationUS7010553 *Mar 19, 2002Mar 7, 2006Network Appliance, Inc.System and method for redirecting access to a remote mirrored snapshotUS7039663 *Apr 19, 2002May 2, 2006Network Appliance, Inc.System and method for checkpointing and restarting an asynchronous transfer of data between a source and destination snapshotUS7043485 *Mar 19, 2002May 9, 2006Network Appliance, Inc.System and method for storage of snapshot metadata in a remote fileUS7225204 *Mar 19, 2002May 29, 2007Network Appliance, Inc.System and method for asynchronous mirroring of snapshots at a destination using a purgatory directory and inode mappingUS7284101 *Aug 4, 2004Oct 16, 2007Datalight, Inc.Reliable file system and method of providing the sameUS7296125 *Nov 29, 2001Nov 13, 2007Emc CorporationPreserving a snapshot of selected data of a mass storage systemUS7529782 *Nov 15, 2004May 5, 2009Commvault Systems, Inc.System and method for performing a snapshot and for restoring dataUS20020083037 *Aug 17, 2001Jun 27, 2002Network Appliance, Inc.Instant snapshotUS20020178146May 24, 2001Nov 28, 2002International Business Machines CorporationSystem and method for selective object history retentionUS20030041211Mar 7, 2002Feb 27, 2003Merkey Jeffrey VernonDual axis RAID systems for enhanced bandwidth and reliabilityUS20030070043Mar 7, 2002Apr 10, 2003Jeffrey Vernon MerkeyHigh speed fault tolerant storage systemsUS20030158861Feb 15, 2002Aug 21, 2003International Business Machines CorporationProviding a snapshot of a subset of a file systemUS20030158863Feb 15, 2002Aug 21, 2003International Business Machines CorporationFile system snapshot with ditto address featureUS20030159007Feb 15, 2002Aug 21, 2003International Business Machines CorporationDeferred copy-on-write of a snapshotUS20030182325 *Mar 19, 2002Sep 25, 2003Manley Stephen L.System and method for asynchronous mirroring of snapshots at a destination using a purgatory directory and inode mappingUS20090030983 *May 30, 2008Jan 29, 2009Prasanna Kumar MalaiyandiSystem and method for non-disruptive check of a mirrorEP1003103A2Jun 2, 1994May 24, 2000Network Appliance, Inc.Write anywhere file-system layoutWO2000007104A1Jul 28, 1999Feb 10, 2000Network Appliance, Inc.File system image transferWO2001031446A1Oct 25, 2000May 3, 2001Network Appliance, Inc.File system image transfer between dissimilar file systems* Cited by examinerNon-Patent CitationsReference1"Storage Management Mechanism for Managing Snapshot Pages" IBM Technical Disclosure Bulletin, IBM Corp. New York, US, vol. 35, No. 4B, Sep. 1, 1992, pp. 26-29.2Akyurek, Sedat, Placing Replicated Data to Reduce Seek Delays, Department of Computer Science, University of Maryland, UMIACS-TR-91-121, CS-TR-2746, Aug. 1991.3Bitton, Dina, Disk Shadowing, Proceedings of the 14th VLDB Conference, LA, CA 1988.4Chaudhuri, Surajit, et al., Self-Tuning Technology in Microsoft SQL Server, Data Engineering Journal 22, 2 1999, pp. 20-27.5Chutani, Sailesh, et al., The Episode File System, In Proceedings of the USENIX Winter 1992.6Coyne, Robert A., et al., Storage Systems for National Information Assets, Proc. Supercomputing 92, Minneapolis, Nov. 1992, pp. 626-633.7European Search Report, Munich, Leuridan, K., Jan. 4, 2005.8Finlayson, Ross S., et al., Log Files: An Extended File Service Exploiting Write-Once Storage Department of Computer Science, Stanford University, Report No. STAN-CS-87-1177, Sep. 1987.9Gray, Jim, et al., The Recovery Manager of the System R Database Manager, ACM Computing Surveys, (13)2:223-242 1981.10Hecht, Matthew S., et al. Shadowed Management of Free Disk Pages with a Linked List, ACM Transactions on Database Systems, 8/4, Dec. 1983, pp. 503-514.11Hitz, Dave et al., File System Design for an NFS File Server Appliance (Technical Report 3002), presented Jan. 19, 1994, UDENIX Winter 1994 - San Francisco CA, 23 pages.12Howard, John H., An Overview of the Andrew File System, Carnegie Mellon University, CMU-ITC-88-062 1988.13Howard, John, H. et al., Scale and Performance in a Distributed File System, ACM Trans. Computer System, 6(1), Feb. 1988 pp. 51-81.14Howard, John, H. et al., Scale and Performance in a Distributed File System, Carnegie Mellon University, CMU-ITC-87-068, Aug. 1987.15Kazar, Michael L., et al., DEcorum File System Architectural Overview, USENIX Summer Conference, Anaheim, California, 1990.16Kazar, Michael Leon, Synchronization and Caching Issues in the Andrew File System, Carnegie Mellon University, CMU-ITC-88-063, date : 1988.17Kemper, Alfons, et al., Performance Tuning for SAP R/3, Data Engineering Journal 22, 2 1999 pp. 33-40.18Kent, Jack et al., Optimizing Shadow Recovery Algorithms, IEEE Transactions on Software Engineering, 14( 2): 155-168, Feb. 1988.19Kistler, et al., Disconnected Operation in the Coda File System, ACM Transactions on Computer Systems, vol. 10, No. 1, Feb. 1992, pp. 3-25.20Lorie, Raymond, A. Physical Integrity in a Large Segmented Database, ACM Trans. Database Syst., vol. 2, Mar. 1977, pp. 91-104.21Ousterhout, John et al., Beating the I/O Bottleneck: A Case for Log-Structured File Systems, Technical Report, Computer Science Division, Electrical Engineering and Computer Sciences, University of California at Berkeley, Oct. 30, 1988.22Partial European Search Report, Munich, Leuridan, K., Jan. 4, 2005.23Patterson, D., et al., A Case for Redundant Arrays of Inexpensive Disks (RAID),-SIGMOD International Conference on Management of Data, Chicago, IL, USA, Jun. 1-3, 1988, Sigmod Record (17)3:109-16 Sep. 1988.24Patterson, D., et al., A Case for Redundant Arrays of Inexpensive Disks (RAID),-Technical Report, CSD-87-391, Computer Science Division, Electrical Engineering and Computer Sciences, University of California at Berkeley 1987.25Peterson, Zachary Nathaniel Joseph, Data Placement for Copy-on-Write Using Virtual Contiguity, University of CA, Santa Cruz, Master's Thesis for the Department of Science in Computer Science, Sep. 2002.Referenced byCiting PatentFiling datePublication dateApplicantTitleUS7685388Jan 25, 2008Mar 23, 2010Netapp, Inc.Method and system for single pass volume scanning for multiple destination mirroringUS7702869Dec 8, 2008Apr 20, 2010Netapp, Inc.System and method for verifying the consistency of mirrored data setsUS7716445 *Apr 19, 2006May 11, 2010Oracle America, Inc.Method and system for storing a sparse file using fill countsUS7756844 *May 23, 2008Jul 13, 2010Pillar Data Systems, Inc.Methods of determining and searching for modified blocks in a file systemUS7836029 *Sep 25, 2009Nov 16, 2010Pillar Data Systems, Inc.Systems and methods of searching for and determining modified blocks in a file systemUS7925749 *Apr 24, 2007Apr 12, 2011Netapp, Inc.System and method for transparent data replication over migrating virtual serversUS7949843Dec 7, 2009May 24, 2011Netapp, Inc.Method and system for single pass volume scanning for multiple destination mirroringUS7987156 *Mar 5, 2008Jul 26, 2011American Megatrends, Inc.Method, system, apparatus, and computer-readable medium for taking and managing snapshots of a storage volumeUS8046547 *Jan 11, 2008Oct 25, 2011American Megatrends, Inc.Storage system snapshots for continuous file protectionUS8082407Apr 16, 2008Dec 20, 2011American Megatrends, Inc.Writable snapshots for boot consolidationUS8117158Mar 5, 2008Feb 14, 2012American Megatrends, Inc.Method, system, apparatus, and computer-readable medium for taking and managing snapshots of a storage volumeUS8245226Jan 13, 2009Aug 14, 2012Microsoft CorporationOffline migration from prior operating system installationUS8260744Mar 5, 2008Sep 4, 2012American Megatrends, Inc.Method, system, apparatus, and computer-readable medium for taking and managing snapshots of a storage volumeUS8301791May 30, 2008Oct 30, 2012Netapp, Inc.System and method for non-disruptive check of a mirrorUS8392423Jul 2, 2010Mar 5, 2013International Business Machines CorporationData set index record preservationUS8489549Nov 24, 2010Jul 16, 2013Beijing Innovation Works Technology Company LimitedMethod and system for resolving conflicts between revisions to a distributed virtual file systemUS8489654 *Aug 27, 2010Jul 16, 2013Beijing Innovation Works Technology Company LimitedMethod and system for forming a virtual file system at a computing deviceUS8548957Nov 24, 2010Oct 1, 2013Beijing Innovation Works Technology Company LimitedMethod and system for recovering missing information at a computing device using a distributed virtual file systemUS8560716 *Dec 19, 2008Oct 15, 2013Emc CorporationTime and bandwidth efficient recoveries of space reduced dataUS8655840 *Dec 3, 2008Feb 18, 2014Nokia CorporationMethod, apparatus and computer program product for sub-file level synchronizationUS8688711Mar 31, 2009Apr 1, 2014Emc CorporationCustomizable relevancy criteriaUS8694564Nov 24, 2010Apr 8, 2014Beijing Innovation Works Technology Company LimitedMethod and system for providing on-demand services through a virtual file system at a computing deviceUS8713270Sep 21, 2011Apr 29, 2014Microsoft CorporationEfficient file system metadata scanning using scoped snapshotsUS8725690Dec 19, 2008May 13, 2014Emc CorporationTime and bandwidth efficient backups of space reduced dataUS8726080Aug 15, 2011May 13, 2014International Business Machines CorporationMerging multiple contexts to manage consistency snapshot errorsUS8751523 *Jun 5, 2009Jun 10, 2014Apple Inc.Snapshot based searchUS8819361Sep 12, 2011Aug 26, 2014Microsoft CorporationRetaining verifiability of extracted data from signed archivesUS8839446Sep 12, 2011Sep 16, 2014Microsoft CorporationProtecting archive structure with directory verifiersUS8918400Feb 19, 2013Dec 23, 2014International Business Machines CorporationData set index record preservationUS8930499 *Jul 9, 2010Jan 6, 2015International Business Machines CorporationData replication between software versionsUS8935209 *Sep 14, 2012Jan 13, 2015Novell, Inc.Techniques for establishing a file system that exposes a virtual file system for backup operationsUS8949182Jun 17, 2011Feb 3, 2015International Business Machines CorporationContinuous and asynchronous replication of a consistent datasetUS8949183Aug 13, 2012Feb 3, 2015International Business Machines CorporationContinuous and asynchronous replication of a consistent datasetUS8972967Sep 12, 2011Mar 3, 2015Microsoft CorporationApplication packages using block mapsUS9146816Jun 14, 2012Sep 29, 2015International Business Machines CorporationManaging system image backupUS9251002Dec 16, 2013Feb 2, 2016Stratus Technologies Bermuda Ltd.System and method for writing checkpointing dataUS9348827 *Mar 27, 2014May 24, 2016Emc CorporationFile-based snapshots for block-based backupsUS9411821 *Mar 27, 2014Aug 9, 2016Emc CorporationBlock-based backups for sub-file modificationsUS20070106863 *Apr 19, 2006May 10, 2007Sun Microsystems, Inc.Method and system for storing a sparse file using fill countsUS20070174484 *Jan 23, 2006Jul 26, 2007Stratus Technologies Bermuda Ltd.Apparatus and method for high performance checkpointing and rollback of network operationsUS20080133543 *Nov 30, 2007Jun 5, 2008General Instrument CorporationSystem and Method for Dynamic and On-Demand Data Transfer and Synchronization Between Isolated NetworksUS20090006496 *May 23, 2008Jan 1, 2009Kurt Alan ShoensMethods of determining and searching for modified blocks in a file systemUS20090030983 *May 30, 2008Jan 29, 2009Prasanna Kumar MalaiyandiSystem and method for non-disruptive check of a mirrorUS20090229626 *Mar 11, 2008Sep 17, 2009Wei-Cheng HsuFirm structured hair clipUS20090307276 *Jun 6, 2008Dec 10, 2009Microsoft CorporationMigration using file system linksUS20100138387 *Dec 3, 2008Jun 3, 2010Kim Kristian SimeliusMethod, apparatus and computer program product for sub-file level synchronizationUS20100179959 *Sep 25, 2009Jul 15, 2010Kurt Alan ShoensSystems and methods of searching for and determining modified blocks in a file systemUS20100180281 *Jan 13, 2009Jul 15, 2010Microsoft CorporationOffline migration from prior operating system installationUS20100312783 *Jun 5, 2009Dec 9, 2010Donald James BradySnapshot based searchUS20110066668 *Nov 24, 2010Mar 17, 2011Guarraci Brian JMethod and System for Providing On-Demand Services Through a Virtual File System at a Computing DeviceUS20110072062 *Nov 24, 2010Mar 24, 2011Guarraci Brian JMethod and System for Resolving Conflicts Between Revisions to a Distributed Virtual File SystemUS20120011495 *Jul 9, 2010Jan 12, 2012International Business Machines CorporationData replication between software versionsUS20120054253 *Aug 27, 2010Mar 1, 2012Beijing Innovation Works Technology Company LimitedMethod and System for Forming a Virtual File System at a Computing DeviceUS20120117027 *Jun 29, 2011May 10, 2012Teradata Us, Inc.Methods and systems for hardware acceleration of database operations and queries for a versioned database based on multiple hardware acceleratorsUS20130013565 *Sep 14, 2012Jan 10, 2013Dhairesh OzaTechniques for establishing a file system that exposes a virtual file system for backup operationsUS20150120667 *Jan 8, 2015Apr 30, 2015Novell, Inc.Techniques for establishing a file system that exposes a virtual file system for backup operationsCN102306168A *Aug 23, 2011Jan 4, 2012成都市华为赛门铁克科技有限公司Log operation method and device and file system* Cited by examinerClassifications U.S. Classification1/1, 707/999.203, 707/999.202, 707/999.01International ClassificationG06F17/30, G06F11/14, G06F11/20Cooperative ClassificationY10S707/99954, Y10S707/99953, Y10S707/99952, G06F11/1451, G06F11/2074, G06F11/2066, G06F17/30067European ClassificationG06F11/20S2L, G06F17/30F, G06F11/20S2P2Legal EventsDateCodeEventDescriptionApr 15, 2013FPAYFee paymentYear of fee payment: 4RotateOriginal ImageGoogle Home - Sitemap - USPTO Bulk Downloads - Privacy Policy - Terms of Service - About Google Patents - Send FeedbackData provided by IFI CLAIMS Patent Services