Patent Description:
The instant disclosure relates to the archival, search, retrieval, and restoration of data In particular, the instant disclosure relates to methods and systems that permit real time data search and retrieval in a manner that preserves chain of custody and other important data characteristics.

<CIT> relates to a system for archiving electronic correspondence to facilitate accessing such correspondence from remote locations, wherein the correspondence is stored by being indexed.

<CIT> relates to databases and to improving query performance in large databases by employing a progressive indexing approach that builds partial indexes over subsets of the data as the data is scanned, i.e., traversed, during execution of ad-hoc queries.

Disclosed herein is a method of archiving data, as claimed in claim <NUM>.

The plurality of data item properties for the data stream can include a data item date property and a data item custodian property.

The method according to the invention also includes: receiving a search query including a plurality of search criteria, the plurality of search criteria including a data item date criterion and a data item custodian criterion; indexing the stored data stream after receiving the search query; applying the search query to the indexed stored data stream; returning search results after applying the search query to the indexed stored data stream. According to aspects of the disclosure, the step of indexing the stored data stream after receiving the search query can include indexing only data items within the stored data stream matching the data item date criterion and the data item custodian criterion. According to other aspects of the disclosure, the step of indexing the stored data stream after receiving the search query can include indexing only data item properties of the stored data stream corresponding to the plurality of search criteria.

According to the invention, the step of receiving data streams to be archived includes: receiving a plurality of data items to be archived; separating the plurality of data items to be archived into a plurality of data streams according to a plurality of preset criteria. The receiving, defining, creating, and storing steps are repeated for each data stream of the plurality of data streams. In addition, the method includes: receiving a search query including a plurality of search criteria that correspond to data item properties common to two or more stored data streams of the plurality of stored data streams; indexing the two or more stored data streams after receiving the search query; applying the search query to the two or more indexed stored data streams; returning search results after applying the search query to the two or more indexed stored data streams. According to aspects of the disclosure, the step of indexing the two or more stored data streams after receiving the search query can include indexing only data item properties of the two or more stored data streams corresponding to the plurality of search criteria. Alternatively or additionally, the plurality of search criteria can include a data item date criterion and a data item custodian criterion, and the step of indexing the two or more stored data streams after receiving the search query can include indexing only data items within the two or more stored data streams matching the data item date criterion and the data item custodian criterion.

It is contemplated that the record manifest for the data stream can be created by extracting the metadata for the one or more data items in the data stream from the one or more data items in the data stream.

Also disclosed herein is a method of restoring a data item from an archive, including the steps of receiving identification of a stub of a data item to be restored, the stub including a pointer to a record in the archive; retrieving the record from the archive; copying from the record into the stub of the data item only data contained in the record that is not present in the stub of the data item, thereby creating a restored data item. The restored data item can then be transferred from a first data storage location to a second data storage location, and the restored data item can be deleted from the first data storage location after transferring the restored data item to the second data storage location.

The foregoing and other aspects, features, details, utilities, and advantages of the present invention will be apparent from reading the following description and claims, and from reviewing the accompanying drawings.

The appendix to the specification contains additional illustrations of the methods and systems disclosed herein.

The present disclosure provides methods and systems (e.g., computer systems and software) for archiving and retrieving data. It should be understood that the teachings herein can be software- and/or hardware-implemented, and that they may be executed on a single CPU, which may have one or more threads, or distributed across multiple CPUs, each of which may have one or more threads, in a parallel processing environment.

For purposes of illustration, several exemplary embodiments will be described in detail herein in the context of archiving and retrieving various corporate records. It is contemplated, however, that the methods and systems described herein can be utilized in other contexts.

<FIG> is a flowchart <NUM> of representative steps that can be carried out to archive data according to aspects of the instant disclosure. In block <NUM>, a plurality of data items are received. This is illustrated in the schematic representation <NUM> of the archiving of data (e.g., corporate records <NUM>) in <FIG>.

In block <NUM>, the data items (e.g., corporate records <NUM>) are separated into a plurality of data streams <NUM> based upon a plurality of corresponding criteria. In embodiments of the disclosure, there can be a one-to-one correspondence between data streams <NUM> and criteria (that is, each data stream has a single corresponding defining criterion).

The criteria can correspond to a characteristic or category of the data items received. For example, corporate records <NUM> can be separated into a first data stream that contains employee personnel records, a second data stream that contains customer invoices, a third data stream that contains employee emails, a fourth data stream that contains corporate contracts, and so forth.

Each of the plurality of data streams <NUM> will therefore contain data items of a common type (e.g., all data items in a given data stream will be employee emails). Further processing of only a single data stream (e.g., stream N) containing employee emails will be described herein for the sake of illustration, though it should be understood that analogous steps can be carried out with respect to the remaining data streams (e.g., streams <NUM>, <NUM>, <NUM>,. , N-<NUM>).

In block <NUM>, a plurality of data item properties are defined for the data stream. The data item properties correspond to and describe aspects of a particular data item within the data stream, such as corporate record <NUM> shown in <FIG>.

It is desirable for the data item properties to include at least a data item date property and a data item custodian property. Additional data item properties can be user defined. For example, for the illustrative data stream of employee emails, then the data item properties can include, without limitation: CUSTODIAN; DATE; SENDER; RECIPIENTS; SUBJECT; and the like.

In block <NUM>, a record manifest <NUM> is created for the data stream. The record manifest includes metadata for the data items within the data stream corresponding to the data item properties for the data stream. This information can either be manually entered or extracted directly from the data item.

For example, consider an email collected from John Smith's mailbox that was sent from Jane Doe to John Smith on January <NUM>, <NUM>, with the subject "Today's Meeting Agenda. " For this data item, the custodian (e.g., John Smith), the date (e.g., January <NUM>, <NUM>), the sender (e.g., Jane Doe), the recipient(s) (e.g., John Smith), and the subject ("Today's Meeting Agenda") can be extracted from the email and added to record manifest <NUM>. Those of ordinary skill in the art will appreciate that, after repeating this process for all the data items within the data stream, the record manifest will, in effect, become a database for the data stream, with each entry in the database corresponding to an individual data item within the data stream.

In block <NUM>, the data items <NUM> from the data stream and the record manifest <NUM> are stored in storage <NUM>. Storage <NUM> can be any suitable storage medium, including, without limitation, a storage area network (SAN), a network attached storage (NAS) device, cloud storage (e.g., Microsoft Azure, Amazon S3), a private cloud storage, a hybrid cloud storage, or the like.

Advantageously, the data within storage <NUM> is merely at rest, and is not indexed except on demand, and then only to the extent necessary to satisfy a user request, as will now be described with reference to the flowchart <NUM> of exemplary steps shown in <FIG>.

In block <NUM>, a search query including a plurality of search criteria is received. The search criteria desirably include both a data item date criterion and a data item custodian criterion. The search criteria can also include an identification of a particular data stream to search.

For example, if a user wishes to search only employee emails prior to January <NUM>, <NUM> collected from John Smith's mailbox, the search criteria can be structured to specify the employee email data stream, a data item custodian criterion of "CUSTODIAN = John Smith," and a data item date criterion of "DATE < January <NUM>, <NUM>. " (Those of ordinary skill in the art will appreciate that the precise syntax of the search query may differ from the exemplary syntax shown here.

In block <NUM>, the stored data stream, which is otherwise at rest, is indexed. By waiting until the search query is received to index the data stream, computing resources, and thus financial resources, are conserved.

Further computing and financial resources can be conserved by limiting the extent to which the data stream is indexed in block <NUM>. According to aspects of the disclosure, only data items having data item properties that match the data item custodian and data item date criteria are indexed, leaving the remaining data at rest. In the example above, therefore, only John Smith's emails from before January <NUM>, <NUM> would be indexed; emails from other custodians or from other date ranges would be ignored.

In additional aspects of the disclosure, data items are only indexed with respect to data item properties corresponding to the search criteria. For example, if the user is interested only in emails collected from John Smith's mailbox that were sent from Jane Doe, there would be no need to index the "SUBJECT" data item properties; only the "SENDER" data item property would be relevant to the search.

Further computing resource and financial savings can be realized by indexing from record manifest <NUM> created in block <NUM>, rather than from the raw data items <NUM> themselves.

In block <NUM>, the search criteria are applied to the indexed data stream. Search results are returned in block <NUM>.

The description above illustrates a search of a single data stream. It is contemplated, however, that searches can be executed on multiple data streams in parallel (rather than in series, as in extant search methodologies). In embodiments, therefore, the search criteria received as part of the search query in block <NUM> can correspond to data item properties that are common to multiple data streams stored in storage <NUM>. Blocks <NUM>, <NUM>, and <NUM> can then be applied in parallel to the multiple data streams.

In another embodiment, the instant disclosure provides a method of restoring a data item, such as an email, a contact, a calendar entry, or the like, from an archive. It is known to archive such data items by creating copies thereof in the archival storage location and then removing the majority of the contents of the data item (e.g., the body of an email message and any attachments) from the primary storage location in order to reduce storage consumption. The portion of the data item that remains in primary storage is known as a "stub," and contains certain information regarding the data item (e.g., SENDER, RECIPIENT, SUBJECT, and the like), as well as a pointer to the copy of the original message in archival storage.

It is possible to modify the stub. For example, a user may wish to assign or remove a flag or assign, change, or remove a category of the data item represented by the stub.

Yet, because these changes to the stub are not reflected in the corresponding data item in archival storage, extant methods of restoring data items from an archive can lose these changes. In particular, extant methods of restoring data items from an archive typically delete the stub when importing the corresponding data item from the archive, thereby creating a data item that appears to be identical to the data item when it was archived, and that does not exhibit any post-archive changes made in the stub. This data loss can be disadvantageous.

<FIG> depicts a flowchart <NUM> of representative steps that allow data items to be restored from an archive without loss of data present in a stub by treating the stub and the corresponding archived data item as complementary parts of a whole. The stub of the data item to be restored is identified in block <NUM>. In block <NUM>, the corresponding data item is retrieved from the archive.

In block <NUM>, the stub and the retrieved data item are combined in a manner that preserves any changes to the stub. In particular, only data contained in the retrieved data item that are not already present in the stub (e.g., the body of an email message and any attachments) are copied into the stub.

In optional block <NUM>, the restored data item can be transferred to a new live (rather than archival) storage location. The data item can then be deleted in full from the original live (rather than archival) storage location.

All directional references (e.g., upper, lower, upward, downward, left, right, leftward, rightward, top, bottom, above, below, vertical, horizontal, clockwise, and counterclockwise) are only used for identification purposes to aid the reader's understanding of the present invention, and do not create limitations, particularly as to the position, orientation, or use of the invention. Joinder references (e.g., attached, coupled, connected, and the like) are to be construed broadly and may include intermediate members between a connection of elements and relative movement between elements. As such, joinder references do not necessarily infer that two elements are directly connected and in fixed relation to each other.

Claim 1:
A method of archiving data, comprising:
receiving (<NUM>) a data stream to be archived, the data stream including a plurality of data items;
defining (<NUM>) a plurality of data item properties for the data stream;
creating (<NUM>) a record manifest for the data stream, wherein the record manifest comprises metadata for the plurality of data items in the data stream, wherein the metadata corresponds to the plurality of data item properties for the data stream;
storing (<NUM>) the data stream and the record manifest for the data stream prior to indexing the data stream;
receiving (<NUM>) a search query including a plurality of search criteria;
characterized by
indexing (<NUM>) the stored data stream after receiving the search query and according to the plurality of search criteria; and
applying (<NUM>) the search query to the stored indexed data stream,
wherein receiving (<NUM>) a data stream to be archived further comprises separating the plurality of data items to be archived into a plurality of data streams according to a plurality of preset criteria; and
repeating the receiving (<NUM>), defining (<NUM>), creating (<NUM>), and storing (<NUM>) steps for each data stream of the plurality of data streams, further comprising
receiving (<NUM>) a search query including a plurality of search criteria that correspond to data item properties common to two or more stored data streams of the plurality of stored data streams;
indexing (<NUM>) the two or more stored data streams after receiving the search query;
applying (<NUM>) the search query to the two or more indexed stored data streams;
returning (<NUM>) search results after applying the search query to the two or more indexed stored data streams.