Patent ID: 12242446

DETAILED DESCRIPTION OF SOME EXAMPLE EMBODIMENTS

Embodiments of the present invention generally relate to generation of datastreams having various attributes. More particularly, at least some embodiments of the invention relate to systems, hardware, software, computer-readable media, and methods for generating a collection of files based on datastreams that have desired characteristics.

In more detail, example embodiments of the invention may be employed in connection with various types of datastreams including, for example, datastreams that have particular characteristics. Such characteristics include, but are not limited to, dedupability (that is, the extent, if any, to which the datastream can be deduplicated), compressibility, commonality, data change rate, and clustering, for example. It is noted that dedupability may be present within a single datastream and/or across multiple datastreams. Systems and methods for generating such datastreams are disclosed in the Related Applications.

Example embodiments of the present invention are implemented in connection with a file system that has a user-defined file structure. More particularly, a number of parameters such as total size of the collection of files, collection growth, average size of each entity in the collection, variance in the size of the entities in the collection, distribution of collection entities at each folder and sub-folder, and the minimum, maximum, variance at each folder level, are collected and used to create a simulated file structure of a filesystem. Then, each entity that is to be included in the filesystem is sequentially created, and the entities are populated with data from a datastream. Where multiple datastreams are employed, the entities need not be created sequentially.

In this way, various characteristics of the datastream, or datastreams, are imparted to the files in the filesystem. Considered from another perspective, embodiments of the invention generate and morph a collection of files in a filesystem having a specified folder/sub-folders file structure configuration, so that the resulting grouping of files collectively possesses characteristics, such as dedupability, compression, clustering and commonality, for example, desired by a customer. The following example is illustrative.

The generation of a base collection of files may involve taking an LO (which can also be referred to as day 0 in some contexts) version of a datastream having particular characteristics, and then using data from that datastream to populate the simulated files to generate a base file collection with those characteristics. Embodiments of the invention are not limited solely to generation of base file collections however.

For example, other embodiments of the invention provide for morphing a collection of files, either in forward or in reverse, to another version of that collection of files. This will be briefly illustrated below with an example in which a collection of files is forward morphed from generation #N−1 to generation #N.

This particular morphing example involves generating Ln (day n) and Ln−1 (day n−1) versions of datastreams in memory, and then comparing these with each other in smaller chunks, such as 1 KB for example. The chunk size can be selected as needed. By way of illustration, in an embodiment that simulates the operation of a tracker, such as VM (VMware) CBT (Changed Block Tracking) for example, that tracks changes at a 64 KB level, the chunk size can be 64 KB. When a difference is identified, that is, when a generation N version of a block in a file is different from the generation N−1 version of the block, the generation N−1 version of the block is overwritten with the generation N version of the block. Since the data generation speed may be extremely high, both the datastream comparison process and the file collection morphing process can be performed at high speed. Moreover, since all existing entities are neither read nor written again, only the impacted files and only the impacted areas are altered. As disclosed in more detail elsewhere herein, and in the Related Applications, file collection morphing can be performed without ad hoc datastream generation, through the use of masks.

Advantageously then, embodiments of the invention may provide various benefits and improvements relative to the configuration and operation of conventional hardware, software, systems and methods. For example, an embodiment of the invention provides for high speed generation of file collections that have desired characteristics imparted by the data upon which the file collection was generated. An embodiment of the invention is particularly well suited for testing and evaluation of a backup application and associated components and processes, since some backup applications are not configured to handle datastreams, but can handle data in the form of file collections. Further, an embodiment of the invention can operate to morph a file collection at the same speed, or nearly so, as the speed at which a datastream is generated. An embodiment of the invention eliminates the need to copy entire files and/or collections and instead writes only the data that has changed.

It should be noted that the foregoing advantageous aspects of various embodiments are presented only by way of example, and various other advantageous aspects of example embodiments of the invention will be apparent from this disclosure. It is further noted that it is not necessary that any embodiment implement or enable any of such advantageous aspects disclosed herein.

A. Aspects of an Example Operating Environment

The following is a discussion of aspects of example operating environments for various embodiments of the invention. This discussion is not intended to limit the scope of the invention, or the applicability of the embodiments, in any way.

In general, embodiments of the invention may be implemented in connection with the testing, analysis, diagnosis, and evaluation, of systems, software, and components, that individually and/or collectively implement, and/or cause the implementation of, data generation and data management operations. Such data management operations may include, but are not limited to, data read/write/delete operations, data deduplication operations, data backup operations, data restore operations, data cloning operations, data archiving operations, and disaster recovery operations. Thus, while the discussion herein may, in some aspects, be directed to a discussion of data protection environments and operations, the scope of the invention is not so limited. More generally then, the scope of the invention embraces any operating environment in which the disclosed concepts may be useful. In some instances, embodiments of the invention generate datastreams for use in testing systems and applications in various environments, one example of which is a data protection environment.

A data protection environment, for example, may take the form of a public or private cloud storage environment, an on-premises storage environment, and hybrid storage environments that include public and private elements, although the scope of the invention extends to any other type of data protection environment as well. Any of these example storage environments, may be partly, or completely, virtualized. The storage environment may comprise, or consist of, a datacenter which is operable to service read and write operations initiated by one or more clients.

In addition to the storage environment, the operating environment may also include one or more host devices, such as clients for example, that each host one or more applications. As such, a particular client may employ, or otherwise be associated with, one or more instances of each of one or more applications that generate data that is desired to be protected. In general, the applications employed by the clients are not limited to any particular functionality or type of functionality. Some example applications and data include email applications such as MS Exchange, filesystems, as well as databases such as Oracle databases, and SQL Server databases, for example. The applications on the clients may generate new and/or modified data that is desired to be protected.

Any of the devices, including the clients, servers and hosts, in the operating environment can take the form of software, physical machines, or virtual machines (VM), or any combination of these, though no particular device implementation or configuration is required for any embodiment. Similarly, data protection system components such as databases, storage servers, storage volumes (LUNs), storage disks, replication services, backup servers, restore servers, backup clients, and restore clients, for example, can likewise take the form of software, physical machines or virtual machines (VM), though no particular component implementation is required for any embodiment. Where VMs are employed, a hypervisor or other virtual machine monitor (VMM) can be employed to create and control the VMs.

As used herein, the term ‘data’ is intended to be broad in scope. Thus, that term embraces, by way of example and not limitation, data segments such as may be produced by datastream segmentation processes, data chunks, data blocks, atomic data, emails, objects of any type, files, contacts, directories, sub-directories, volumes, and any group of one or more of the foregoing.

Example embodiments of the invention are applicable to any system capable of storing and handling various types of objects, in analog, digital, or other form. Although terms such as document, file, block, or object may be used by way of example, the principles of the disclosure are not limited to any particular form of representing and storing data or other information. Rather, such principles are equally applicable to any object capable of representing information.

With particular attention now toFIG.1, one example of an operating environment is denoted generally at100. In some embodiments, the operating environment may comprise, consist of, or be a part of, a data protection environment, although none of these arrangements is required. The operating environment can include an enterprise datacenter, or a cloud datacenter, or both. The data protection environment may support various data protection processes, including data replication, data deduplication, cloning, data backup, and data restoration, for example. As used herein, the term backups is intended to be construed broadly and includes, but is not limited to, partial backups, incremental backups, full backups, clones, snapshots, continuous replication, and any other type of copies of data, and any combination of the foregoing. Any of the foregoing may, or may not, be deduplicated.

In the illustrated example, the operating environment100generally includes a plurality of clients200that communicate with one or more backup servers300. The backup server300communicates with a filesystem synthesis server400and a datacenter500, which may be a cloud datacenter in some embodiments, though that is not required. Finally, a datastream generator600communicates with the datacenter500, simulation server400, and backup server300. Further details concerning each of these components is provided in the following discussion.

The clients200may comprise any number of clients from 1 . . . n. In the example ofFIG.1, clients202,204and206are indicated. Each of the clients200includes a respective instance of backup agent202a,204a, and206a, respectively. The backup agents202a,204a, and206acooperate with a backup application302on the backup server300to create backups of client200data for storage at the datacenter500.

The backup server300can include, in addition to the backup application302, a filesystem synthesis module304, and a deduplication engine306. In general, and as discussed in more detail elsewhere herein, the filesystem synthesis module304operates to generate a simulated filesystem with a file structure that includes one or more directories and sub-directories. As well, the filesystem synthesis module304operates to populate the filesystem with data generated by the datastream generator600. Further details concerning an example filesystem synthesis module304are set forth in the discussion ofFIG.2below. The deduplication engine306deduplicates backups, including files, generated by the backup application302and then transmits the deduplicated data to the datacenter500for storage in the datacenter storage501. In some embodiments, the filesystem synthesis module304and/or the deduplication engine306may be hosted at respective stand-alone servers, rather than being integrated in a backup server300or other component.

Thus, and with continued reference toFIG.1, some implementations include a stand-alone filesystem server400that communicates with the backup server300and transmits a collection of simulated files to the backup server300for use by the backup application302. In still other embodiments, a filesystem synthesis module502is included as an element of the datacenter500, or a filesystem synthesis module602can be included as an element of the datastream generator600. As well, in some implementations, the datastream generator600may be omitted, and the datastream functionality implemented by a datastream generation module504of the datacenter.

As will be apparent from the configuration and discussion ofFIG.1, the functional allocations disclosed herein are presented only by way of example. In general, the disclosed functions can be allocated amongst elements of the operating environment100in any other suitable manner, and the scope of the invention is not limited to the disclosed configurations and arrangements.

With continued reference toFIG.1, a brief introductory discussion is provided concerning operational aspects of some of the disclosed elements. In general, the datastream generator600operates to generate one or more high speed datastreams with various desired characteristics. These datastreams are provided, or otherwise made accessible to, simulation logic such as the filesystem synthesis module304. The filesystem synthesis module304, creates, based on the datastream and its associated characteristics, a simulated filesystem having a file structure that includes files, directories, and sub-directories. The simulated filesystem is then populated with data from the datastream to create a collection of files that collectively possess the datastream characteristics and which can be used to test the operation and performance of the backup application302and/or deduplication engine306, among others. The collection of files created by the filesystem synthesis module304can be morphed forward or backward one or more generations by the filesystem synthesis module304, using different generations of the datastreams and/or through the use of masks, examples of which are disclosed in the Related Applications.

With reference now toFIG.2, details are provided concerning an example implementation of a simulation server700which may comprise, or consist of, simulation logic that may or may not be in the form of a filesystem synthesis module. In the illustrated example, the simulation server700may include a file structure and files engine702that, in general, operates to create a simulated filesystem that includes a particular file structure and simulated files. The particular characteristics of the file structure and files can be specified in a characteristics file or module704that, in some embodiments at least, is configured to receive input from a user. The population engine706receives data from one or more datastreams created by a datastream generator and populates the files with that data. Finally, a morph engine708operates to morph a collection of files, cooperatively created by the file structure and files engine702and the population engine706, forward or backward one or more generations. Morphing can be performed using data from one or more datastreams and/or through the use of one or more masks that indicate changes that have occurred to data over one or more generations of a datastream.

B. Example Host and Server Configurations

With reference briefly now toFIG.3, any one or more of the clients200, backup agents202a,204a,206a, the backup server300, backup application302, filesystem synthesis module304, deduplication engine306, filesystem synthesis server400, datacenter500, storage501, filesystem synthesis module502, datastream generator504, datastream generator600, filesystem synthesis module602, file structure and files engine702, characteristics data704, population engine706, and morph engine708can take the form of, or include, or be implemented on, or hosted by, a physical computing device, one example of which is denoted at800. As well, where any of the aforementioned elements comprise or consist of a virtual machine (VM), that VM may constitute a virtualization of any combination of the physical components disclosed inFIG.3.

In the example ofFIG.3, the physical computing device800includes a memory802which can include one, some, or all, of random access memory (RAM), non-volatile random access memory (NVRAM)804, read-only memory (ROM), and persistent memory, one or more hardware processors806, non-transitory storage media808,1/O device810, and data storage812. One or more of the memory components802of the physical computing device800can take the form of solid state device (SSD) storage. As well, one or more applications814are provided that comprise executable instructions. Such executable instructions can take various forms including, for example, instructions executable to perform any method or portion thereof disclosed herein, and/or executable by/at any of a storage site, whether on-premises at an enterprise, or a cloud storage site, client, datacenter, backup server, or network node, to perform functions disclosed herein. As well, such instructions may be executable to perform part, or all of, any of the other operations and processes disclosed herein including, but not limited to, datastream generation, file generation, file structure generation, morphing, populating, deduplication, data storage and retrieval, as well as read, write, backup, clone, and restore, operations and/or any other data protection operation, auditing operations, cloud service operations.

C. Creation of Simulated Filesystem

Directing attention now toFIG.4, details are provided concerning systems and processes for generating simulated filesystems and files that can be used for example, to test and evaluate the performance of applications, such as a backup application for example. Such filesystems and files may be referred to herein as ‘simulated’ because they are not ‘real world’ files that have been generated by applications operating in a production environment, but are instead configured to closely mimic real world files and data. As noted, data used to populate the simulated files can be generated by a datastream generator. Until the population process is performed, there are no actual files in the simulated filesystem. Instead only simulated files that are not yet populated with data are initially present in the simulated filesystem. The files and directories in the filesystem can be created in various ways. For example, empty files and directories can all be created before any data population is performed. Alternatively, the files and directories can be created, and populated, on an individual basis. Finally, the simulated filesystem can be of any type, and the scope of the invention is not limited to any particular type of filesystem(s). Some example simulated filesystems include Windows NTFS, ReFS, ExFAT, Linux ext4, btrfs, zfs, and Mac.

As indicated inFIG.4, an example simulated filesystem900is disclosed that has a file structure902. The filesystem900and associated file structure902can be specified by a user. In general, the file structure can be of any size and configuration, and include any number and size of directories and files. As well, each entity, or file, in the filesystem900has a simulated size as specified by various input parameters, as discussed in more detail below. Thus, the file structure902shown inFIG.4is presented only for the purposes of illustration and is not intended to limit the scope of the invention in any way.

Embodiments of the invention are flexible in terms of the filesystem900parameters that can be specified, such as by a user, or by a computing entity, for example. Those parameters include, but are not limited to: configuration of the file structure902including the structural relationships and arrangements between/among files, directories, and/or subdirectories; size of files; number of files; types of files; number of directories; number of sub-directories; size/number/type(s) of files in a directory/sub-directory; particular distribution of files throughout the file structure; total size of the file system; and, the size and content of directories and sub-directories. The values for any of these parameters can be specified and stored for later use, or can be provided to the system in real time and inline as part of a test and evaluation procedure involving, for example, a backup application and/or deduplication engine.

As noted, the collection of simulated files in the filesystem900can be populated with data from a datastream generated by a datastream generator. Thus, not only can parameters of the filesystem900be specified, but parameters particularly concerning the simulated files can be specified as well. These may be referred to herein as file collection parameters as they concern the particular collection of files in the filesystem900. Such file collection parameters include, but are not limited to: total size of the file collection; collection growth; average size of each entity (file) in the collection; variance in the size of the entities in the collection; distribution of collection entities at s each folder and sub-folder; and, minimum/maximum/variance at each folder level. Along with the filesystem parameters, the file collection parameters are collected and used to build the simulated filesystem900.

In some embodiments, the simulated filesystem900is created ad hoc, based on user input. In other embodiments, the simulated filesystem900can be created automatically by drawing one or more parameters from another simulated filesystem and/or from an actual real world file system.

Finally, the file structure902may, but need not necessarily, mimic an actual or anticipated real world file structure that is to be backed up with a backup application. Because backup applications are typically configured to operate in connection with files and filesystems, the simulated filesystem900provides a good mechanism to test the performance and operation of a backup application and/or a deduplication application/engine. Thus, testing and performance results obtained with a simulated filesystem, as provided by embodiments of the invention, can provide useful insights into the backup application and deduplication application/engine performance and functionality. For example, one or more embodiments of the invention can be used to determine the best and most effective way of backing up a file system. Particularly, by walking the file structure in various ways, and knowing the good/bad points of the dedupe engine, collectively a better backup mechanism can be developed.

D. Creation of Simulated File Collection

With reference now toFIG.5, details are provided concerning an example structure and process for generating a collection of files using data from a datastream created by a datastream generator. As shown inFIG.5, a datastream1000is generated that embodies various specified characteristics1002that are desired to be imparted to the collection of files that will be created using data from the datastream1000. Detailed information concerning both the characteristics, and generation, of the datastream, are set forth in the Related Applications and, as such, an in-depth discussion of those elements is not provided here. In general however, some example characteristics1002of the datastream1000that can be specified in connection with the creation of the datastream1000, and which can be reflected in a file collection created using data from that datastream1000, may include, but are not limited to, dedupability, compression, clustering, and commonality.

When the datastream1000has been created with the desired characteristics1002, the datastream1000is sliced up into portions1004whose respective sizes correspond to the respective sizes of the simulated files in the filesystem900. The collective characteristics of the portions1004correspond to the desired characteristics of the file collection that is to be included in the filesystem900.

The slicing process can be performed, for example, by the filesystem synthesis module and/or by a datastream generator, and may take place inline as the datastream is being generated, or after the datastream has been generated and stored. The slicing process may be based on the file characteristics that were specified in connection with the configuration of the filesystem and include, for example, total size of the file collection, collection growth, average size of each entity (file) in the collection, and variance in the size of the entities in the collection. To illustrate, the file characteristics may specify that 13 files (File A to File M) are to be created, whose average size is 20 MB. Thus, the slicing process will slice 13 pieces from the datastream, while also ensuring that the average size of the collection of 13 pieces is 20 MB.

The particular portion(s) of the datastream1000that are targeted by the slicing process can vary as necessary to ensure that the resulting group of 13 files, in this example, collectively possess the desired characteristics1002. Thus, the slicing process can take the characteristics of the datastream1000into account when making decisions about which data to cut, and when/where, from the datastream1000. The slicing of the datastream1000may, or may not, be random. In some embodiments, the slicing can target, or avoid, one or more particular portions of the datastream1000, such as the first 1 GB for example, or the middle 10 GB, although neither approach is necessarily required. In still other embodiments, the slicing process, which may be a virtual process, may be performed in a uniform manner over the entire datastream1000.

With continued reference toFIG.4, and as noted earlier, the data slices taken from the datastream1000are then used to populate the filesystem900. Each slice corresponds to, and may constitute, a particular file. In some embodiments, the files in the filesystem900are sequentially created, using the data taken from the datastream. This file creation may be performed in tandem with the slicing process such that upon slicing, the slice data is immediately used to populate the corresponding file. In other cases, one or more of the slices can be made and stored for later use in populating one or more files of a collection of files. Once the simulated files have been populated with data from the datastream1000, the collection of files can then be used in the testing and evaluation, for example, of a backup application and/or deduplication engine.

With continued attention toFIG.5, and referring now toFIG.6as well, details are provided concerning methods for creating a simulated filesystem, and then generating a collection of files from a datastream, where one example of such a method is denoted generally at1100. Part, or all, of the method1100can be performed by a filesystem synthesis module, for example.

In general, the method disclosed inFIG.6may be employed for creation of a collection of files that does not yet exist. As such, the method disclosed inFIG.6and discussed herein may be referred to as a method of generating a base collection of files. As discussed in connection withFIG.7for example, and by way of contrast withFIG.6, an existing collection of files can be morphed forward or backward to another generation of that existing collection of files.

The example method1100may begin when one or more filesystem parameters are received1102that specify various attributes targeted for inclusion in a simulated filesystem that is to be created. The parameters can then be used to create1104a simulated filesystem.

Before, during, or after, creation1104of the simulated filesystem, target characteristics for a file collection, to be stored in the simulated filesystem, are received1106. As noted herein, and in the Related Applications, such target characteristics may include, for example, compressibility, commonality, data change rate, and clustering. Various other target characteristics for a file collection can additionally, or alternatively, be received1106. It is noted that processes1102and1106can be performed in parallel, or sequentially as shown inFIG.6.

Next, a datastream is sliced1108into data portions that correspond to the files to be included in the file collection. In general, the files are carved out of the datastream1108one after another from a single large stream of data. The datastream may be generated prior to, or concurrently with, the slicing process1108, and the datastream that is sliced1108possesses the target characteristics that are desired to be reflected in the file collection. In at least some embodiments, the target characteristics are used as a basis for generation of the datastream, and/or the target characteristics are used to drive the datastream slicing process1108, so as to ensure that the file collection ultimately created collectively reflects the target characteristics. It is noted that processes1104and1108can be performed on a delta basis, that is, at each file level.

After, or during, the datastream slicing process1108, the resulting data portions of the datastream are used to populate1110the simulated files that were created1104. In at least some embodiments, the simulated files are populated1110in a sequence that corresponds to the order in which data slices are taken1108from the datastream, although that sequence is not necessarily required. It is noted that the files occupy data so the sizes of the files are simulated in the process1100. Moreover, the collective size of this simulated file system is same as the size of the data taken from the datastream.

After the simulated files are populated1110with data from the datastream, the collection of files can be stored and/or output1112to an end user. The file collection may be stored1112, for example, at a datacenter, client, backup server, and/or other entities in an operating environment.

Additionally, or alternatively, the file collection can be transmitted1112to an end user, such as a backup server that includes a backup application, for testing and evaluation1114purposes. For example, a backup application can create a backup using the file collection, and then transmit the backup to a deduplication engine for deduplication, prior to storage of the backup at a datacenter or other target. In this way, the operation of the backup application and/or deduplication engine can be tested with the use of files that mimic real world files such as may be created by clients at a production site and transmitted to a backup server for creation of a backup and deduplication.

When the backup application runs and takes a backup of the file collection for the first time, the backup applications scans all the files, reads the data in the files, and sends the data to the dedupe engine. Even when the carve-out sequence and backups sequence are not the same as each other, the dedupe engine sees the backup data as being very close to having the same characteristics as if the original stream were written as a single large file. Thus, the desired dedupe/compression/commonality can still be achieve when the data is processed and sent by the backup application. Accordingly, even if the backup application is not able to directly process the long datastreams, it can still operate effectively with the simulated carved-out file system.

E. Morphing a Simulated File Collection

Turning next toFIG.7, details are provided concerning processes and an arrangement for morphing a file collection forward from one generation to another. In the illustrated example, a file collection is morphed from generation N−1 to generation N. However, a file collection can be morphed forward multiple generations, and morphed backward one or more generations. Thus, the example ofFIG.7is presented for purposes of illustration and does not limit the scope of the invention.

It was noted in one or more of the Related Applications that a data generation algorithm can regenerate streams that depict a known change rate. Thus, it is possible to re-apply the same carving logic utilizing the method ofFIG.6and recreate all files and process these with a backup application. In this way, reprocessing of the file collection by the dedupe engine will generate the desired dedupabilty along with the compression/commonality, and any other characteristics of interest. However, this approach does not reflect the typical real life case, since only a small fraction of files actually change between generations. Moreover, this approach would require all files to be written again, and so is not particularly efficient in terms of time and processing resources utilized.

Thus, the example method ofFIG.7generally involves generating a datastream corresponding to the generation N−1, and generating another stream corresponding to the generation N. Generation of these two datastreams can be performed as disclosed in the Related Applications. The two datastreams are sequentially compared to identify the differences between the two streams and, at the same time, the method also moves forward in the files that are being compared. Whenever compared data is the same, no overwrite of the data in the file is performed, but when the data is different, an overwrite is performed and, in that way, certain files that map to the changed areas are modified. Since the streams for generation N−1 and generation N can be generated at very high rate, as disclosed in the Related Applications, and the compare operation is likewise performed relatively quickly, changes to the simulated filesystem can be effective relatively quickly. Thus, all files that did not map to changed areas remain unmodified. It is noted that comparing datastreams is one way to identify changes that need to be made in order to morph a file collection, while another approach to identify changes is the use of an interface as discussed below in connection withFIG.7.

Now, when the backup application runs and performs a backup of the simulated filesystem, the backup application can very quickly skip all files that have not changed and read only the files that have changed. When the contents of changed files are sent to dedupe engine, the modified data has almost the same dedupability, commonality, and compressibility that were configured for the data generation algorithm that generated the datastream from which the file data was taken. Thus, by having two incoming datastreams, and by comparing their contents, it is possible to very quickly morph a filesystem forward from one generation to another. As well, the same morphing logic can be applied to perform a reverse morphing process where data that is written on the mapped files is taken from generation N−1 rather from the generation N.

It is to be noted that in some embodiments, and as disclosed in the Related Applications, it is possible to make this work with just the gen 0 datastream. For example, in a forward morphing process where X is the value of the generation to which a file collection is to be forward morphed from generation N, it is enough to have a generation X mask, and the gen-0 datastream. The forward morphing process can read the X mask and then alter any block of the gen-N file collection whose generation value is greater than N in the X mask utilizing gen-0 datastream. Finally, it is to be noted that it is possible to make this work with just the generation mask X as that mask signifies the changes that are needed beyond generation N and the impacted areas can be read, modified from generation N to generation X, and then written back. It is to be noted that the last scheme should not be utilized if the gap between N to X is quite high as that will require excessive amount of read/modify/write and thus slowing the process.

For a reverse morphing process in which a file collection of generation X is to be reverse morphed to a prior generation N of that file collection, only generation N mask and generation X mask are needed, along with the gen-0 datastream. In general, the generation N mask value and generation X mask value are compared to each other and the gen-0 datastream transformed to gen-N state, or not, based on the outcome of that comparison. That is, when the generation X value of a block is the same as the generation N value of that block, the block is not reverted, but when the generation X value of the block is different from the generation N value, the gen-0 block is transformed to generation N and the block is then written to the file to which it belongs. One example of a reverse morphing process is discussed later in connection withFIG.9.

With particular reference now to the arrangement and method disclosed inFIG.7, one example implementation1200involves a first set of inputs1210which constitute characteristics to be implemented in a corresponding datastream1215. The inputs1210may include, and as also noted in the Related Applications, a generation designation for the datastream, seed(s) used to generate the datastream, and desired compression, commonality, and other, characteristics for the datastream1215that is to be generated. In this example, the datastream1215is a generation N−1 datastream. Each of the subdivisions1212in the datastream1215denotes a file boundary in the datastream. Similarly, another set of inputs1220, which may or may not be the same as the inputs1210, constitute characteristics to be implemented in a corresponding generation N datastream1225. Each of the subdivisions1222in the datastream1215denotes a file boundary in the datastream. As indicated inFIG.7, the two datastreams1215and1225map to the same set of files.

With continued reference to the example ofFIG.7, it can be seen that a comparison of the generation N−1 datastream1215with the generation N datastream1225reveals that only small portions of two different files, File E and File I, have changed between the generation N−1 datastream1215and the generation N datastream1225. That is, File E portion1214of the generation N−1 datastream1215has changed to File E portion1217in the generation N datastream1225, and File I portion1216of generation N−1 datastream1215has changed to File I portion1219in the generation N datastream1225. Thus, the generation N file collection1250N is created by overwriting, in the file collection1250, the portion1214with portion1217, and overwriting portion1216with portion1219, and no other portions of Files E and I, or any other files, are modified, or need to be.

In more detail, an example interface for a situation where an N−1 generation datastream and N generation datastream are employed could take the following form:

Reset-offsets(N−1, N, other parameters)

Get-me-Next-change(offset-where-change-is, buffer)

Get-me-Next-change(offset-where-change-is, buffer)

Get-me-Next-change(offset-where-change-is, buffer)

. . . Done.

With this example interface, on the first call, the logic returns the portion1217in the supplied buffer and returns an offset that corresponds to somewhere in file E. On the next call, the logic returns portion1219in the buffer and an offset that corresponds to somewhere in file I. In either case, the logic knows exactly where the next change will be, whether by a datastream comparison, or the use of a gen-0 datastream along with N−1 and N masks.

As the example ofFIG.7illustrates, embodiments of the invention can be economical in terms of the time and processing resources needed since only those portions of files that have been changed are overwritten, and there is no need to rewrite entire files or groups of files. Moreover, because the modified data takes the form of the file collection1250, the modified data can be readily handled and processed by a backup application and deduplication engine.

It can also be seen, with continued reference to the example ofFIG.7, that comparison of the generation N−1 datastream1215with the generation N datastream1225enables the file collection1250to be reverse morphed from generation N to generation N−1. Particularly, a comparison of the datastreams reveals that File E portion1217of the generation N datastream1225corresponds to File E portion1214in the generation N−1 datastream1215, and File I portion1219of the generation N datastream1215corresponds to File I portion1216in the generation N−1 datastream1215. Thus, the file collection1250can be reverse morphed from generation N to generation N−1 by overwriting portion1217with portion1214, and overwriting portion1219with portion1216.

F. Aspects of Some Example Methods

With reference now toFIG.8, details are provided concerning aspects of example methods for morphing a simulated file collection from one generation to another, wherein one example of a file collection morphing method is denoted generally at1300. In the example ofFIG.8, a file collection of generation N is morphed forward to generation X although, as noted herein, a file collection may be reverse morphed as well.

The example method1300can begin at1302wherein a gen-N datastream and a gen-X datastream are received. Next, the gen-N datastream and the gen-X datastream are compared with each other1304to determine1306whether any changes have been made to the gen-N datastream that are reflected in the gen-X datastream. This comparison1304can be performed on a block basis, file basis, or any other suitable basis. Thus, while the illustrative example ofFIG.8employs a block basis comparison, the scope of the invention is not limited to that example.

With continued reference toFIG.8, the determination1306can comprise a determination as to whether or not the gen-X datastream block is the same as the gen-N datastream block. If the gen-X datastream block is determined1306to be the same as the gen-N datastream block, the method advances to1308and no overwrite of the gen-N datastream block is performed. On the other hand, if it is determined1306that the gen-X datastream block is different from the gen-N datastream block, then the gen-N datastream block is overwritten1307with the gen-X datastream block.

Whether a block is overwritten1307, or not1308, the method1300then advances to1310where a determination is made as to whether or not the datastream comparison1304has been completed. If not, the method1300returns to1304and, if so, the method advances to1312, completing the morphing of the file collection from generation N to generation X. It is noted with respect to the method ofFIG.8that X=N+a, where a is any integer ≥1. Thus, a file collection can be morphed forward any number of generations.

With continued reference toFIG.8, it can be seen that the method1300can be readily adapted to perform a reverse morphing of a gen-X file collection to a gen-file collection, where N=X−a, and a is any integer ≥1. In particular, if it is determined at1306that the generation value of the block in the gen-X datastream is not the same as the generation value of that same block in the gen-N datastream, then, in a reverse morphing scenario, the gen-X version of the block is overwritten with the gen-N version of that block, thus reverting the block from gen-X to gen-N. Except for this change at1307, the method1300is otherwise applicable to reverse morphing of a file collection from gen-X to gen-N.

Directing attention next toFIG.9, details are provided concerning methods and processes for reverse morphing a gen-X file collection to a gen-N file collection, where one example method is denoted generally at1400. It is noted that reverse morphing can be performed over a single generation, or multiple generations. Thus, the example method ofFIG.9is like that ofFIG.8, in that both methods can be used to morph a file collection from one generation to another. However, aside from the fact thatFIG.8is concerned with forward morphing andFIG.9is concerned with reverse morphing, the method ofFIG.8involves the comparison of blocks of data of two different datastreams, while the method ofFIG.9involves the comparison of mask values. Thus, both methods are effective in morphing a file collection from one generation to another, but employ different respective mechanisms to do so. Moreover, the method ofFIG.9can be performed with only a single datastream. In general, the gen-0 datastream is used as a basis for the generation of modified blocks, if those are needed for morphing.

The example method1400can begin when a gen-0 datastream, gen-X mask, and gen-N mask are received1402. The respective block generation values in the gen-X mask and gen-N mask are then compared1404. If those block generation values are determined1406to be the same, then no change to the block is needed1408. That is, the fact that the block generation values are the same indicates that the gen-X and gen-N versions of the block are the same. Note that as used herein, “generation value” refers to the generation of a particular block of a particular datastream. Thus, for example, a gen-3 mask entry having a generation value of 2 means that the block associated with that entry was last modified in the second generation datastream.

On the other hand, if it is determined1406that the respective block generation values in the gen-X mask and gen-N mask are different, then the corresponding block in the gen-0 datastream is transformed to the gen-N value of that block1407and the transformed block is then written to the file to which it belongs. Whether the gen-0 block is transformed and written to the file1407, or not1408, the method1400then advances to1410where a check is performed to determine if more data is needed or not.

When no more data is needed, the mask comparison, and any corresponding overwrites, are determined1410to be complete, and the process1400advances to1412and the process of morphing the file collection from gen-X to gen-N is completed. On the other hand, If additional data is needed, the mask comparison, and any corresponding overwrites, are determined1410not to be complete, then the process1400returns to1404.

G. Alternative Embodiments—Simulated Databases/Database Collections

G.1 Introduction

In general, the disclosure herein concerning filesystems applies as well to databases. Thus, for example, a datastream generator may be used to generate one or more simulated databases that may be used to provide predictable changes/compression. The datastream may be used, for example, in dedupe analysis and benchmarking operations. Thus, at least some example embodiments may extend and compliment the disclosure herein concerning simulated filesystems, at least insofar as such embodiments may create slices in a simulated DB (database) environment rather than in a simulated file system as disclosed above.

By way of illustration, an example method according to some embodiments may comprise receiving a set of database parameters, creating one or more simulated databases based on the database parameters, receiving a set of target characteristics for the database, based on the target characteristics, slicing a datastream into a grouping of data slices, populating the simulated database(s) with the data slices to create the database collection and forward, and/or reverse, morphing the database from one generation to another without rewriting the entire database collection. Except as may be noted hereafter, the present disclosure applies equally as well to databases/database collections as it does to filesystems/file collections.

G.2 Compare/Contrast—Use of Concepts for Databases/Database Collections

As noted, the disclosure preceding Section G hereof is concerned largely with filesystems/file collections, but also finds application as well to databases and database collections. Some differences between the two applications are noted hereafter. It is noted that the embodiments ofFIGS.10-16, discussed below, may be employed in connection with various types of relational databases. For example, such embodiments may be implemented in connection with relational databases such as MySQL, MariaDB, Oracle, PostgreSQL, MSSQL. Thus, the scope of the invention should not be construed to be limited to any particular type of relational database.

For example, there is little or no practical limit to the possible depth of a filesystem. That is, a filesystem may have any size and depth of directories and sub-directories. In contrast however, a database may effectively have only three levels, namely, (1) tables, (2) records, or rows, included in the tables, and (3) fields within each record. As such, some embodiments may generate simulated databases having a depth of three, or fewer, levels. Further, some embodiments may generate structures with greater than three levels, and those structures may then be modified so that they have no more than three levels.

In the simulated filesystem approaches disclosed herein, the data generator may have direct access to the filesystem and may simply create two streams, such as rev N and rev N+1, and from the differences between the streams and their known mappings to the existing files, only the files that have the modification may be altered. This simulation logic may be extended to the context of databases and database collections to achieve the same, or similar, results.

G.3 Aspects of the Database Use Case

In general, a database may include one or more tables, where each table may have one or more records, and each record may have one or more fields. A database engine may operate to manage, within a database, these tables and records, which may be distributed across many nodes. Note that while reference is made, in the discussion ofFIGS.10-16, and elsewhere, to various operations performed in connection with a database, it should be understood that those operations may be performed with respect to a single database, and/or with respect to a database collection that includes multiple databases.

Some embodiments of the invention may provide for the intelligent mapping of variable slices from a datastream, such as may be created by a data generator. Particularly, these slices may be mapped to row-fields, that is, to particular rows and/or fields of one or more rows, within a database, so as to achieve, for the case of databases, the similar kind of dedupe characteristic that may be obtained from the simulated file systems disclosed herein. This approach may also enable implementation of the similar kind of incremental change property as disclosed herein with respect to files.

In general,FIGS.10-16, discussed in further detail below, disclose structures and operations relating to the simulation of a DB collection, rather than a FS (filesystem) collection, from the same datagen stream. Among other things, these Figures disclose approaches for mapping slices of a datastream to row-fields of a database.

Note that in the case of FS simulation, the data/content of the files may map directly on the storage media, such as disk for example, and there may be a relatively large number of files. By way of contrast, in the case of databases, the data/content may collective map to just few very large files on the storage media. The files in turn may comprise tables, rows, and fields, and the DB engine of the database that stores the files may manage the mapping of data, that is, the rows and fields, by itself to the database storage that has been allocated to the files. Thus, some embodiments may create slices, of a data stream, that in the file/filesystem use case would be mapped to files, but that in the database use case may be mapped instead to records/fields of a table of a DB.

Note that the order/mapping of these slices to records/fields in DB storage may be managed by the DB engine. Moreover, Microsoft SQL (MSSQL) may perform this order/mapping differently than Oracle, for example, and the record/field size needed to obtain the required amount of dedupe from the DB engine, after the simulation, may require the knowledge of the DB engine so the stream generation may be modified as needed.

G.4 Detailed Discussion of Database Use Case

With attention now toFIGS.10-16, further details are provided concerning the creating/morphing of a collection of databases. In this regard, it is noted that the embodiments ofFIGS.10-16may be employed in connection with various types of databases. For example, such embodiments may be implemented in connection with relational databases such as SQL databases. Some example SQL databases include MySQL, MariaDB, Oracle, PostgreSQL, and MSSQL. Some example NoSQL databases include MongoDB, CouchDB, CouchBase, Cassandra, HBase, Redis, Riak, Neo4J.

Turning first toFIG.10, there is disclosed an example operating environment100′ for some embodiments of the invention. Except as noted hereafter, the example operations and components disclosed inFIGS.1-9may be similar, or identical, to the operations and components disclosed inFIGS.10-16.

Particularly, inFIG.10, the example operating environment100′ may comprise a backup server300′ with a database synthesis module304′. As well, a database server400′ may be provided that is able to transmit a collection of simulated databases to the backup server300′ for use by the backup application302. A datacenter500′ may be provided that includes a database synthesis module502′, and the datacenter500′ may be operable to communicate with a data generator600′ that includes a database synthesis module602′.

With continued reference toFIG.10, a brief introductory discussion is provided concerning operational aspects of some of the disclosed elements. In general, the datastream generator600′ operates to generate one or more high speed datastreams with various desired characteristics. These datastreams are provided, or otherwise made accessible to, simulation logic such as the database synthesis module304′. The database synthesis module304′, creates, based on the datastream and its associated characteristics, one or more simulated databases each having a structure that includes tables, rows, and fields. The simulated database(s) may then be populated with data slices taken from the datastream to create part, or all of, a database collection including multiple simulated databases that collectively possess the datastream characteristics and which can be used to test the operation and performance of the backup application302and/or deduplication engine306, among others. The collection of databases created by the database synthesis module304′ can be morphed forward or backward one or more generations by the database synthesis module304′, using different generations of the datastreams and/or through the use of masks.

With reference now toFIG.11, details are provided concerning an example implementation of a simulation server700′ which may comprise, or consist of, simulation logic that may or may not be in the form of a database synthesis module. In the illustrated example, the simulation server700′ may include a database engine702′ that, in general, operates to create one simulated databases, each having a structure that includes particular tables, rows, and fields. The particular characteristics of the database structure can be specified in a characteristics file or module704that, in some embodiments at least, is configured to receive input from a user. The population engine706receives data, such as data slices, from one or more datastreams created by a datastream generator and populates the simulated database with that data. Finally, a morph engine708operates to morph a database collection, cooperatively created by the database engine702′ and the population engine706, forward or backward one or more generations. Morphing can be performed using data from one or more datastreams.

G.4.1 Creation of Simulated Databases and Mapping Schemes

Directing attention now toFIGS.12aand12b, details are provided concerning systems and processes for generating simulated relational databases and database collections that can be used for example, to test and evaluate the performance of applications, such as a backup application for example. Such database collections, databases, and associated data, may be referred to herein as ‘simulated’ because they are not ‘real world’ databases that have been generated by applications operating in a production environment, but are instead configured to closely mimic real world databases. As noted, data used to populate the simulated databases can be generated by a datastream generator. Until the population process is performed, there is no actual data in the simulated database. Instead only simulated databases that not yet populated with data are initially present in the simulated database collection. The databases in the database collection can be created in various ways. For example, databases can all be created before any data population is performed. Alternatively, the databases can be created, and populated, on an individual basis. Finally, the simulated database can be of any type, and the scope of the invention is not limited to any particular type of database(s).

With reference first toFIG.12a, an embodiment may operate to create, or synthesize, a simulated database950that comprises one or more tables952, and each of the tables952may comprise one or more slices954that may be created by slicing a datastream. In an embodiment, each slice954may comprise, or consist of, a row. That is, a datastream may be sliced, so as to create the slices954, and the slices954may then be used to create a new database, that is, the database950. In the particular illustrative example ofFIG.12a, it can be seen that thirteen slices954‘A’ . . . ‘M’ may mapped to a single simulated database950that comprises three tables952. As shown in the map956, each of the slices954may be mapped to the database950, denoted as ‘DB1’ in the map956. Within the database950DB1, each slice954may be mapped to a particular table ‘T1’ . . . ‘T3’952, and a particular row ‘R1’ . . . ‘R5’957of the table952.

It is noted that the configuration disclosed inFIG.12ais presented only by way of illustration and is not intended to limit the scope of the invention in any way. More generally, the database950can be of any size and configuration, and include any number and size of tables, rows, and fields. As well, each slice954in the simulated database950may have a simulated size as specified by various input parameters, which may be provided by a user.

In more detail, embodiments of the invention are flexible in terms of the simulated database950parameters that can be specified, such as by a user, or by a computing entity, for example. Those parameters include, but are not limited to: configuration of the database950including the structural relationships and arrangements between/among tables, rows or records, and fields; number of records and fields; types of records and fields; particular distribution of records throughout the database structure; and the total size of the database950. The values for any of these parameters may be specified and stored for later use, or may be provided to the system in real time and inline as part of a test and evaluation procedure involving, for example, a backup application and/or deduplication engine.

As noted, a simulated database, such as the simulated database950, may be populated with data, such as the slices954, from a datastream generated by a datastream generator. Thus, not only may parameters of the simulated database950be specified, but parameters particularly concerning the simulated data held in the simulated database950may be specified as well. These may be referred to herein as database parameters as they concern the data in the simulated database950. Such database parameters may include, but are not limited to: total amount of the data; data growth; and the number of tables, rows, and fields in a database, and in the simulated database950collectively.

In some embodiments, the simulated database950may be created ad hoc, based on user input. In other embodiments, the simulated database950may be created automatically by drawing one or more parameters from another simulated databases and/or from an actual real world databases. Further, a database collection may be created that includes one or more simulated databases, such as the simulated database950.

Finally, the simulated database950may, but need not necessarily, mimic an actual or anticipated real world database that is to be backed up with a backup application. The simulated database950may provide a good mechanism to test the performance and operation of a backup application and/or a deduplication application/engine. Thus, testing and performance results obtained with a simulated database, as provided by embodiments of the invention, can provide useful insights into the backup application and deduplication application/engine performance and functionality.

Turning next toFIG.12b, an embodiment may operate to create a database collection that comprises two, or more, simulated databases, such as the databases970and980for example. In general, the databases970and980may be created in the same way as the simulated database950discussed earlier in connection withFIG.12a. As well, the various considerations, elements, and features, relating to the creation, structure, and use, of the database950apply with equal force to databases970and980and, accordingly, are not repeated here.

In the illustrative example ofFIG.12b, the simulated database970may be constructed that comprises one or more tables972, and each of the tables972may comprise one or more slices974that may be created by slicing a datastream. In an embodiment, each slice974may comprise, or consist of, a row. That is, a datastream may be sliced, so as to create the slices974, and the slices974may then be used to create part, or all, of a new database, that is, the database970. In the particular illustrative example ofFIG.12b, it can be seen that a group of slices ‘A’ . . . ‘M’ may include eight slices974‘A,’ ‘B,’ ‘C,’ ‘D,’ ‘E,’ ‘F,’ ‘L,’ and ‘M’ which may be mapped to a single simulated database970that comprises two tables972. As shown in the map990, each of the slices974may be mapped to the database970, denoted at ‘DB1’ in the map990. Within the database970DB1, each slice974may be mapped to a particular table972‘T1’ or ‘T2,’ and a particular row in the group of rows ‘R1’ . . . ‘R5’976of the table972.

With continued reference toFIG.12b, an embodiment may operate to create multiple simulated databases by defining a mapping of the slices in a group of slices to respective simulated databases, where each of the databases in the group comprises one or more tables, and each of the tables comprises one or more rows.

In the illustrative example ofFIG.12b, the simulated database980may be constructed that comprises one or more tables982, and each of the tables982may comprise one or more slices984that may be created by slicing a datastream. In an embodiment, each slice984may comprise, or consist of, a row. That is, a datastream may be sliced, so as to create the slices984, and the slices984may then be used to create part, or all, of a new database, that is, the database980. In the particular illustrative example ofFIG.12b, it can be seen that the tables982each comprise one or more slices984taken from the group of slices ‘A’ . . . ‘M.’ Particularly, the slices984‘G,’ ‘H, (‘table1’)’ ‘I,’ ‘J,’ and ‘K’ (‘table2’) are included in the tables982. That is, the aforementioned slices984may be mapped to a single simulated database980that comprises the two tables982. As shown in the map990, each of the slices984may be mapped to the database980, denoted as ‘DB2’ in the map990. Within the database980DB2, each slice984may be mapped to a particular table982‘T1’ or ‘T2’.

As collectively illustrated by the examples ofFIGS.12aand12b, an embodiment may operate to create one or more simulated databases using a group of slices, such as slices of a datastream for example. Each of the simulated databases may have a respective group of one or more tables, and each of the tables may comprise one or more rows. In this way, an embodiment may provide great flexibility in terms of the number, and structure, of simulated databases that may be constructed.

G.4.2 Creation of Simulated Databases, and Database Collections

With the examples ofFIGS.12aand12bin view, and with reference now toFIG.12c, details are provided concerning an example process flow, and associated entities, for creating one or more simulated databases. In general, a data generator1050is disclosed that may communicate with a slicer engine1054. The data generator1050may comprise any of the disclosed data generators, or the data generators disclosed in the Related Applications. The slicer engine1054, in turn, may operate to slice a datastream received from the data generator1050and transmit database commands, slices, and slice information, to a scripting module1056that may use the database commands, slices, and slice information, to populate the final consumers of the data slices, namely, one or more database engines1058and1060, for example. Further details concerning these operations and entities is provided below.

In more detail, the data generator1050may receive various inputs1051that the data generator1050may use in the generation of one or more datastreams. For example, one set of inputs1051may comprise information and parameters regarding seed, compression, commonality, clustering, change, and revision. Another set of inputs1051may comprise inputs regarding a desire DB (database) structure, such as the number of tables, rows, and record sizes, that should be included in a particular structure of a simulated database created by an embodiment of the invention. Still another set of inputs1051may comprise the DB engine provider, such as MSSQL or Oracle for example, to which a simulated database will be provided, and the inputs1051may further comprise the layout of files, such as disks and/or spindles for example, used by a particular DB engine provider.

After the datastream has been generated by the database generator1050, based on the inputs1051, the datastream may be transmitted by the database generator1050to the slicer engine1054. In an embodiment, the output datastream may be represented, in a Perl program for example, using the handle ‘stdout.’ The slicer engine1054may then slice the datastream, according to any of the slicing processes disclosed herein, and then transmit the slices and database commands to the scripting module1056. The scripting module1056may comprise a Python or Perl scripting module, but neither is necessarily required by any embodiment, and any other scripting or compiled language may alternatively be used.

The scripting module1056may, after receipt of the database configuration commands from the slicer engine1054, cause the creation of one or more simulated databases via an appropriate database engine, such as the database engines1058and1060. In the example ofFIG.12c, the database engine1058may comprise a Microsoft SQL DB engine, and the database engine1060may comprise an Oracle DB engine, although neither of these example database engines is necessarily required for any particular embodiment. The scripting module1056may, after receipt of the database population commands and the slices of data from the slicer engine1054, cause the population of one or more simulated databases via an appropriate database engine, such as the database engines1058and1060. In the example ofFIG.12c, the database engine1058may comprise a Microsoft SQL DB engine, and the database engine1060may comprise an Oracle DB engine, although neither of these example database engines is necessarily required for any particular embodiment. The simulated databases may then be used, for example, to test computing systems and equipment.

With brief reference toFIG.12d, some sample commands1070are disclosed that may be used by the slicer engine1054to instruct the scripting module1056to build the simulated databases. The commands1070are presented only by way of example, and are not intended to limit the scope of the invention in any way.

With the examples ofFIGS.12a-12din view, and with reference now toFIG.13, details are provided concerning some further example structures and processes for generating a database, or collection of databases, using data from a datastream created by a datastream generator. As shown inFIG.13, a datastream1000′ is generated that embodies various specified characteristics1002′ that are desired to be imparted to the database(s) that will be created using data from the datastream1000′. Detailed information concerning both the characteristics, and generation, of the datastream, are set forth in the Related Applications (and applications referred to in the Related Applications) and, as such, an in-depth discussion of those elements is not provided here. In general however, some example characteristics1002′ of the datastream1000′ that can be specified in connection with the creation of the datastream1000′, and which can be reflected in one or more databases created using data from that datastream1000′, may include, but are not limited to, dedupability, compression, clustering, and commonality.

When the datastream1000′ has been created with the desired characteristics1002′, the datastream1000′ is sliced up into portions1004′ whose respective sizes may correspond to respective portions, such as records for example, of a simulated database such as950,970, or980(seeFIGS.12a-12c). The collective characteristics of the portions1004′ correspond to the desired characteristics of the data that is to be included in the simulated database. Note that in the discussion hereafter ofFIG.13, the simulated database referred to may comprise, by way of example, any of the simulated databases950,970, or980.

The slicing process can be performed, for example, by the database synthesis module and/or by a datastream generator, and may take place inline as the datastream is being generated, or after the datastream has been generated and stored. The slicing process may be based on the database record characteristics that were specified in connection with the configuration of the database and include, for example, total size of the data in the database, data growth, the number of tables, and the number of records in the database. To illustrate, the characteristics may specify that 13 records (Record A to Record M) are to be created, whose average size is 8 KB. Thus, the slicing process will slice 13 pieces from the datastream, while also ensuring that the average size of the collection of 13 records is 8 KB. Note that in this example, each slice can map to one of the records. Additionally, or alternatively, one or more of the slices can be further sliced into two or more sub-slices, and the sub-slices each mapped to a respective field of a record.

The particular portion(s) of the datastream1000′ that are targeted by the slicing process can vary as necessary to ensure that the resulting group of 13 records, in this example, collectively possess the desired characteristics1002′. Thus, the slicing process can take the characteristics of the datastream1000′ into account when making decisions about which data to cut, and when/where, from the datastream1000′. The slicing of the datastream1000′ may, or may not, be random. In some embodiments, the slicing can target, or avoid, one or more particular portions of the datastream1000′, such as the first 1 GB for example, or the middle 10 GB, although neither approach is necessarily required. In still other embodiments, the slicing process, which may be a virtual process, may be performed in a uniform manner over the entire datastream1000.

With continued reference toFIG.13, and as noted earlier, the data slices taken from the datastream1000′ are then used to populate the simulated database. Each slice may correspond, and may constitute, a respective record, or field of a record. In some embodiments, the records in the simulated database are sequentially created, using the data taken from the datastream. This database population may be performed in tandem with the slicing process such that upon slicing, the slice data is immediately used to populate the corresponding record. In other cases, one or more of the slices can be made and stored for later use in populating one or more databases of a collection of databases. Once the simulated database has been populated with data from the datastream1000′, the collection of records, for example, can then be used in the testing and evaluation, for example, of a backup application and/or deduplication engine.

With continued attention toFIG.13, and referring now toFIG.14as well, details are provided concerning methods for creating a simulated database, and then generating a collection of records, or other data groupings, from a datastream, where one example of such a method is denoted generally at1100′. Part, or all, of the method1100′ can be performed by a database synthesis module, for example.

In general, the method disclosed inFIG.14may be employed for creation of a database, or group of databases, that include data that does not yet exist. As such, the method disclosed inFIG.14and discussed herein may be referred to as a method of generating a base collection of one or more databases. As discussed below in connection withFIG.15for example, and by way of contrast withFIG.14, an existing collection of databases can be morphed forward or backward to another generation of that existing collection of databases.

The example method1100′ may begin when one or more database parameters are received1102′ that specify various attributes targeted for inclusion in a simulated database, or simulated databases, that is/are to be created. The parameters can then be used to create1104′ one or more simulated databases. Where multiple simulated databases are created, those simulated databases may collectively form a database collection, or collection of databases.

Before, during, or after, creation1104′ of the simulated filesystem, target characteristics for a record, or other data collection, to be stored in the simulated database, are received1106′. As noted herein, and in the Related Applications, such target characteristics may include, for example, compressibility, commonality, data change rate, and clustering. Various other target characteristics for a database can additionally, or alternatively, be received1106′. It is noted that processes1102′ and1106′ can be performed in parallel, or sequentially as shown inFIG.14.

Next, a datastream is sliced1108′ into data portions that may correspond to the records, or other data collections such as the sub-fields of rows/records, to be included in the database. In general, the records are carved out of the datastream1108′ one after another from a single large stream of data. The datastream may be generated prior to, or concurrently with, the slicing process1108′, and the datastream that is sliced1108′ possesses the target characteristics that are desired to be reflected in the database, that is, data in the database. In at least some embodiments, the target characteristics are used as a basis for generation of the datastream, and/or the target characteristics are used to drive the datastream slicing process1108′, so as to ensure that the database ultimately created collectively reflects the target characteristics. It is noted that processes1104′ and1108′ can be performed on a delta basis, that is, at each table, record, or field, level.

After, or during, the datastream slicing process1108′, the resulting data portions of the datastream are used to populate1110′ the simulated database that was created1104′. In at least some embodiments, the simulated database(s) is/are populated1110′ in a sequence that corresponds to the order in which data slices are taken1108′ from the datastream, although that sequence is not necessarily required. Moreover, the collective size of the simulated data in the simulated database is the same as the size of the data taken from the datastream.

After the simulated database is populated1110′ with data from the datastream, the simulated database, or collection of simulated databases, as applicable, can be made available1112′ to an end user. The end user may use the simulated databases for various purposes.

For example, the database collection can be transmitted to an end user, such as a backup server that includes a backup application, for testing and evaluation1114′ purposes. For example, a backup application can create a backup of the database collection, and then transmit the backup to a deduplication engine for deduplication, prior to storage of the backup at a datacenter or other target. In this way, the operation of the backup application and/or deduplication engine can be tested with the use of databases, and associated data, that mimic real world databases and data such as may be created by clients at a production site and transmitted to a backup server for creation of a backup and deduplication.

When the backup application runs and takes a backup of the database, or database collection, for the first time, the backup applications gets all the data in the databases, and sends the data to the dedupe engine. Even though the sliced data sequence and the backup data sequence are not the same as each other, the dedupe engine sees, in relevant part, the backup data as having the similar characteristics as if the original stream were written as a single large BLOB of data. Thus, the desired dedupe/compression/commonality is generally achieved when the data is processed and sent by the backup application. Note that a backup application may have two ways available to backup the database. One is to ask the DB engine to provide a backup copy as a stream and the DB engine does the required work and sends a stream and the backup app simply stores it on a normal media or sends it to a dedupe engine for a cost-effective storage. Another way for the backup app to backup the database is to request the DB engine to commit all data from running operations and then take a snapshot, or point-in-time copy, of the files that had the data of the database and simply backup these files. Either of these methods may be utilized. For the case where the dedupe/compression/commonality numbers are not close to the desired numbers, the datagen streamer engine can be further made to learn by feeding the observed results.

G.5 Morphing a Simulated Database Collection

It was noted in one or more of the Related Applications that a data generation algorithm can regenerate streams that depict a known change rate. Thus, it is possible to re-apply the same carving logic utilizing the method ofFIGS.13-14and recreate all databases and process these with a backup application. In this way, reprocessing of the database, or database collection, by the dedupe engine will generate the desired dedupabilty along with the compression/commonality, and any other characteristics of interest. However, this approach does not reflect the typical real life case, since only a small fraction of records, or other data collections in a database, actually change between generations. Moreover, this approach would require all records to be written again, and so is not particularly efficient in terms of time and processing resources utilized.

Thus, some embodiments are directed to an approach that does not require recreation, and processing, of all the databases, or entire databases. Rather, such embodiments are directed to morphing a database, or collection of databases, using only the changed portion(s) of the databases.

Turning next toFIG.15, details are provided concerning processes and an arrangement for morphing a database, or collection of databases, forward from one generation to another. In the illustrated example, a database collection is morphed from generation N−1 to generation N. However, a database collection can be morphed forward multiple generations, and morphed backward one or more generations. Thus, the example ofFIG.15is presented for purposes of illustration and does not limit the scope of the invention.

Thus, the example method ofFIG.15generally involves generating a datastream corresponding to the generation N−1, and generating another stream corresponding to the generation N. Generation of these two datastreams can be performed as disclosed in the Related Applications (or applications referred to in the Related Applications). The two datastreams are sequentially compared to identify the differences between the two streams and, at the same time, the method also moves forward in the records that are being compared. Whenever compared data is the same, no overwrite of the data in the record, or overwrite of the record, is performed, but when the data is different, an overwrite is performed and, in that way, certain records that map to the changed areas are modified. Since the streams for generation N−1 and generation N can be generated at very high rate, as disclosed in the Related Applications, and the compare operation is likewise performed relatively quickly, changes to the simulated database can be implemented relatively quickly. Thus, all records that did not map to changed areas remain unmodified.

Now, when the backup application runs and performs a backup of the simulated database (now at gen-N), the backup application can request “changes since gen N−1” (which may be referred to as log backup) from the DB engine and receives just the data corresponding to the updated records (that is, changes in gen N). When this content (that is, the changes in gen N) is sent to the dedupe engine, the modified data has the similar dedupability, commonality, and compressibility that were configured for the data generation algorithm that generated the datastream from which the record data was taken.

Similarly, the backup application can request a “cumulative backup” (which may be referred to as a differential backup) from the DB engine and receives all of the data corresponding to all changes since a specified generation (gen N−3, for example). When this differential content (that is, changes in gen N−2+changes in gen N−1+changes in gen N) is sent to the dedupe engine, the most of this data (that is, changes in gen N−2+changes in gen N−1) gets deduped and remaining data (changes in gen N) has the similar dedupability, commonality, and compressibility that were configured for the data generation algorithm that generated the datastream from which the record data was taken.

Alternatively, the backup application can request a “complete backup” (which may be referred to as a full backup) from the DB engine and receives all of the data corresponding to all active records at generation N. When this content is sent to the dedupe engine, most of this data may be deduped and remaining data (that is, changes in Gen-N) has the similar dedupability, commonality, and compressibility that were configured for the data generation algorithm that generated the datastream from which the record data was taken.

Additionally, the backup application can request a “commit” (which may be referred to as a sync point) from the DB engine and take a snapshot of the system/DB files and perform a file level backup of all DB files. When these DB files are sent to the dedupe engine, most of this data may be deduped and the remaining data may have similar dedupability, commonality, and compressibility that were configured for the data generation algorithm that generated the datastream from which the record data was taken.

It is to be noted that the actual results may be off by a factor (e.g. 1.3 or 30%) from what would have happened if the original stream was deduped as a single large object. This is to be expected and can easily be accounted and adjusted by feeding the observed results back to the data streamer so it learns and adjusts. Thus, by having two incoming datastreams, and by comparing their contents, it is possible to very quickly morph a database, or collection of databases, forward from one generation to another. As well, the same morphing logic can be applied to perform a reverse morphing process where data that is written on the mapped records is taken from generation N−1 rather from the generation N. It is to be noted that in some embodiments, and as disclosed in the Related Applications, it is also possible to make this morphing work with just the gen 0 datastream and masks for the respective generations.

With particular reference now to the arrangement and method disclosed inFIG.15, one example implementation1200′ involves a first set of inputs1210which constitute characteristics to be implemented in a corresponding datastream1215′. The inputs1210may include, and as also noted in the Related Applications, a generation designation for the datastream, seed(s) used to generate the datastream, and desired compression, commonality, and other, characteristics for the datastream1215′ that is to be generated. In this example, the datastream1215′ is a generation N−1 datastream. Each of the subdivisions1212′ in the datastream1215′ denotes a record boundary in the datastream. Similarly, another set of inputs1220, which may or may not be the same as the inputs1210, constitute characteristics to be implemented in a corresponding generation N datastream1225′. Each of the subdivisions1222′ in the datastream1225′ denotes a record boundary in the datastream. As indicated inFIG.15, the two datastreams1215′ and1225′ map to the same set of records.

With continued reference to the example ofFIG.15, it can be seen that a comparison of the generation N−1 datastream1215′ with the generation N datastream1225′ reveals that two different records, Record E and Record I, have changed between the generation N−1 datastream1215′ and the generation N datastream1225′. Note that, in contrast with the example ofFIG.7which discloses operations that involve modification of only small portions of files,FIG.15is concerned with the replacement of whole records of a database. That is,FIG.15discloses that Record E1214′ of the generation N−1 datastream1215′ has changed to Record E1217′ in the generation N datastream1225′, and Record11216′ of generation N−1 datastream1215′ has changed to Record I1219′ in the generation N datastream1225′. Thus, the generation N record collection1250′ is created by overwriting, in the record collection1250′, the Record E1214′ with Record E1217′, and overwriting Record I1216′ with Record I1219′, and no other records, are modified, or need to be.

As the example ofFIG.15illustrates, embodiments of the invention can be economical in terms of the time and processing resources needed since only those records, or other data collections, that have been changed are overwritten, and there is no need to rewrite entire databases. Moreover, because the modified data takes the form of the record collection1250′, the modified data can be readily handled and processed by a backup application and deduplication engine.

It can also be seen, with continued reference to the example ofFIG.15, that comparison of the generation N−1 datastream1215′ with the generation N datastream1225′ enables the database collection1250′ to be reverse morphed from generation N to generation N−1. Particularly, a comparison of the datastreams reveals that Record E1217′ of the generation N datastream1225′ corresponds to Record E1214′ in the generation N−1 datastream1215′, and Record I1219′ of the generation N datastream1225′ corresponds to Record11216′ in the generation N−1 datastream1215′. Thus, the database collection1250′ can be reverse morphed from generation N to generation N−1 by overwriting1217′ with1214′, and overwriting1219′ with1216′.

G.6 Aspects of Some Example Methods

With reference now toFIG.16, details are provided concerning aspects of example methods for morphing the data in a simulated database, from one generation to another, wherein one example of a database morphing method is denoted generally at1300′. This example method may be referred to simply as morphing the simulated database. In the example ofFIG.16, a database collection of generation N is morphed forward to generation X although, as noted herein, a database collection may be reverse morphed as well.

The example method1300′ can begin at1302′ wherein a gen-N datastream and a gen-X datastream are received. Next, the gen-N datastream and the gen-X datastream are compared with each other1304′ to determine1306′ whether any changes have been made to the gen-N datastream that are reflected in the gen-X datastream. This comparison1304′ can be performed on a slice level, table basis, record basis, field basis, or any other suitable basis. Thus, while the illustrative example ofFIG.16employs a record basis comparison, the scope of the invention is not limited to that example.

With continued reference toFIG.16, the determination1306′ can comprise a determination as to whether or not the gen-X datastream record is the same as the gen-N datastream record. If the gen-X datastream block is determined1306′ to be the same as the gen-N datastream record, the method1300′ advances to1308′ and no overwrite of the gen-N datastream record is performed. On the other hand, if it is determined1306′ that the gen-X datastream record is different from the gen-N datastream record, then the gen-N datastream record is overwritten1307′ with the gen-X datastream record.

Whether a record is overwritten1307′, or not1308′, the method1300′ then advances to1310′ where a determination is made as to whether or not the datastream comparison1304′ has been completed. If not, the method1300′ returns to1304′ and, if so, the method advances to1312′, completing the morphing of the database collection from generation N to generation X. It is noted with respect to the method ofFIG.16that X=N+a, where a is any integer ≥1. Thus, a database or database collection can be morphed forward any number of generations.

With continued reference toFIG.16, it can be seen that the method1300′ can be readily adapted to perform a reverse morphing of a gen-X database collection to a gen-N database collection, where N=X−a, and a is any integer ≥1. In particular, if it is determined at1306′ that the generation value of the record in the gen-X datastream is not the same as the generation value of that same record in the gen-N datastream, then, in a reverse morphing scenario, the gen-X version of the record is overwritten with the gen-N version of that record, thus reverting the record from gen-X to gen-N. Except for this change at1307′, the method1300′ is otherwise applicable to reverse morphing of a database collection from gen-X to gen-N.

H. Example Computing Devices and Associated Media

The embodiments disclosed herein may include the use of a special purpose or general-purpose computer including various computer hardware or software modules, as discussed in greater detail below. A computer may include a processor and computer storage media carrying instructions that, when executed by the processor and/or caused to be executed by the processor, perform any one or more of the methods disclosed herein.

As indicated above, embodiments within the scope of the present invention also include computer storage media, which are physical media for carrying or having computer-executable instructions or data structures stored thereon. Such computer storage media can be any available physical media that can be accessed by a general purpose or special purpose computer.

By way of example, and not limitation, such computer storage media can comprise hardware storage such as solid state disk/device (SSD), RAM, ROM, EEPROM, CD-ROM, flash memory, phase-change memory (“PCM”), or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other hardware storage devices which can be used to store program code in the form of computer-executable instructions or data structures, which can be accessed and executed by a general-purpose or special-purpose computer system to implement the disclosed functionality of the invention. Combinations of the above should also be included within the scope of computer storage media. Such media are also examples of non-transitory storage media, and non-transitory storage media also embraces cloud-based storage systems and structures, although the scope of the invention is not limited to these examples of non-transitory storage media.

Computer-executable instructions comprise, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts disclosed herein are disclosed as example forms of implementing the claims.

As used herein, the term ‘module’ or ‘component’ can refer to software objects or routines that execute on the computing system. The different components, modules, engines, and services described herein may be implemented as objects or processes that execute on the computing system, for example, as separate threads. While the system and methods described herein can be implemented in software, implementations in hardware or a combination of software and hardware are also possible and contemplated. In the present disclosure, a ‘computing entity’ may be any computing system as previously defined herein, or any module or combination of modules running on a computing system.

In at least some instances, a hardware processor is provided that is operable to carry out executable instructions for performing a method or process, such as the methods and processes disclosed herein. The hardware processor may or may not comprise an element of other hardware, such as the computing devices and systems disclosed herein.

In terms of computing environments, embodiments of the invention can be performed in client-server environments, whether network or local environments, or in any other suitable environment. Suitable operating environments for at least some embodiments of the invention include cloud computing environments where one or more of a client, server, or other machine may reside and operate in a cloud environment.

The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.