Patent Description:
When computer storage systems are being developed, there is a need to conduct testing on their performance. There are many purpose-built workload generator tools in the prior art to meet specific input/output (IO) requirements for this purpose. Many of these workload generator tools serve to benchmark performance. Examples include fio - Flexible I/O tester, developed by Jens Axboe and described at https://fio. readthedocs. io/en/latest/, IOzone Filesystem Benchmark, developed by William Norcott et al. and described at http://www. org/, and SPEC SFS® <NUM>, provided by Standard Performance Evaluation Corporation of Gainesville, Virginia and described at https://www. org/sfs2014/. Such tools generate specified workload (input/output operations per second (IOPS) centric, bandwidth centric, etc.) by generating IO within a predefined storage capacity by continually over-writing the data within the given capacity. Many tend to create a single large file and assign a range within the file to each IO thread.

Computer storage systems are often optimized or tuned for workloads specific to an industry or customer. When they are tested before being deployed, a hypothetical workload is typically used. Such a workload may test the storage system under specific and narrow use cases or where the system under test is known to perform well. However, storage systems may not perform as well under real-world conditions after being deployed.

Due to the lack of versatile tooling, the exposure to real-world IO patterns continues to be part of the product deployment phase rather than development phase and is typically owned by the application owner/customers. This makes the overall product stability process very long and challenging, requiring product to be deployed in a variety of environments to uncover systemic issues. Additionally, the deployment phase tends to have other constraints and often has a short timeframe. Finding and fixing code bugs typically are not objectives of this phase.

Accordingly, there is greater need for tooling to simulate real-world IO to ensure breadth and depth of system testing during the product development cycle and for performance evaluation under multitudes and varieties of workloads that the storage system is subjected to when deployed as an enterprise-wide storage system.

Document <CIT> discloses a method for testing integrity of data transmitted to and from a target device through a data connection.

Document <CIT> discloses a method for evaluating the performance of an application when migrated from a first environment in which the application is currently executing to a second.

Document <CIT> discloses a parallel storage system testing.

Document <CIT> discloses a computer-implemented method for simulating file system instances which may include identifying a file system to host a simulated file system instance.

The description and drawings also present additional non-claimed embodiments, exemplary embodiments, examples, aspects and implementations for the better understanding of the claimed embodiments defined in the appended claims.

According to aspects of the present disclosure, a system is configured for simulating real-world input/output (IO) workload for testing in a parallel and distributed storage system. Such a workload simulator uses real metadata that is derived from an already deployed production environment. This can be accomplished by using a UNIX Find command to dump the directory tree of the real data. Alternatively, other similar tooling may be used for this purpose, such as a policy engine of IBM Spectrum Scale, formerly known as IBM General Parallel File System (GPFS). This directory/file tree may then be used to simulate the IO patterns (e.g. metadata intensive or data intensive, read/write ratio, etc.) that are prevalent in the existing workload. Additionally, the tool can be easily configured to generate any IO patterns to closely mimic the desired workload for system level and performance testing in a more realistic fashion.

The simulation tools disclosed herein may also verify data that have already been written, thus ensuring the underlying file system/storage system is able to safely manage data over the long term as bits comprising the file contents and meta data get shuffled around different storage media, and maintain desired data availability and redundancies across data centers or within a data center.

Before describing the inventive simulation tools in more detail, the drawbacks of prior art methods will be further described. In addition to the performance tools described in the Background section of this disclosure, there are other "poor man's tools" based on copying publicly available data archives (e.g. from NASA and other web data) into the storage using parallel copy tools such as rsync (see https://en. org/wiki/Rsync for further information) or similar customized tooling. Such tooling does not provide much control to manage IO patterns, and these tools rely on an entire source data set. A copy of this data set (which may be very large), or access to the data set, needs to be maintained for an extended period of time if extensive testing is to be done on the storage system. Problems can emerge if these data sets become corrupted between the start and conclusion of the testing.

There is another class of tools that typically writes data and verifies it immediately after writing or before the data set is overwritten in a next iteration. However, these tools cannot ensure data correctness days or months after the data were ingested. Additionally, the same dataset repeatedly gets overwritten to constraint capacity utilization. Such tooling poses several limitations in being able to test a complex storage system in a realistic manner. A checksum routine is sometimes used to verify data in these prior art tools, but such routines are very processor-intensive. The simulation systems disclosed herein overcome these drawbacks and provide additional advantages, as will become apparent from the following detailed description.

Referring to <FIG>, an exemplary system <NUM> is illustrated. System <NUM> is configured for simulating real-world IO workload for testing in a parallel and distributed storage system, in accordance with one or more implementations of the present disclosure. In some implementations, system <NUM> may include one or more computing platforms <NUM>. Computing platform(s) <NUM> may be configured to communicate with one or more remote platforms <NUM> according to a client/server architecture, a peer-to-peer architecture, and/or other architectures. Remote platform(s) <NUM> may be configured to communicate with other remote platforms via computing platform(s) <NUM> and/or according to a client/server architecture, a peer-to-peer architecture, and/or other architectures. Users may access system <NUM> via remote platform(s) <NUM>.

Computing platform(s) <NUM> may be configured by machine-readable instructions <NUM>. Machine-readable instructions <NUM> may include one or more instruction modules. The instruction modules may include computer program modules. The instruction modules may include one or more of data identifying module <NUM>, workload simulation module <NUM>, directory tree structure writing module <NUM>, data reading module <NUM>, integrity verification module <NUM>, file modification module <NUM>, percentage selection module <NUM>, file size selection module <NUM>, and/or other instruction modules.

Data identifying module <NUM> is configured to identify real-world data from a deployed production environment. In some implementations, module <NUM> may be fully autonomous, semi-autonomous, or replaced by a human user performing the identifying function. Module <NUM> and/or the user may serve to identify a set or sets of real-world data that will be as close as possible to the types of data that the storage system will interact with once deployed. This identified real-world data will serve as the basis for a simulated workload for performance, data integrity and or operational testing of the storage system.

The real-world data includes a directory tree structure and files. The directory tree may have a single level but will typically have many levels. Each of the files located in the level or various levels of the directory tree includes original metadata and original file contents.

Workload simulation module <NUM> is configured to simulate a workload by using the original directory tree structure identified by module <NUM> and or the user of the system. The original metadata from the files is also used by simulation module <NUM>. However, the original contents of the files are replaced with dummy content to create dummy files. The dummy content may be any arbitrary content, as will be later described in more detail. The dummy files may each have a file size that approximates a file size of a corresponding file of the real-world data. As such, the dummy files may closely mimic the original files (e.g. they may have the same names, sizes, metadata, locations in the directory tree, etc.) but have different content from the original real-world data files.

Workload simulation module <NUM> may be configured to simulate a plurality of workloads to simultaneously/concurrently write, read or write and read the directory tree structure and the dummy files of the plurality of simulated workloads to the system of storage devices. In other words, system <NUM> may be configured to run a single workload at any one time, run two workloads, three workloads, or a multitude of workloads at the same time. Simulating a workload may include allowing a user to select a read/write ratio which specifies relative amounts of time to be spent in the reading and the writing steps. In some implementations, simulating a workload may include selecting a read/write ratio which specifies relative amounts of data to read and write. The simulating a workload step may be implemented by simulation module <NUM> without obtaining a copy of the original file contents of the real-world data from the deployed production environment.

Writing module <NUM> is configured to write the directory tree structure and dummy files to a system of storage devices. Data reading module <NUM> may be configured to read data from the directory tree structure and dummy files on the system of storage devices. Integrity verification module <NUM> may be configured to verify the integrity of the dummy files over the course of a plurality of data management processes and a plurality of data availability processes employed by the storage system. For example, such operations can include a data rebuild from parity/RAID code, a data rebalance across available storage media as media is damaged, or as data and/or media is added over a period of time. In this manner, the simulated workload may be used for performance, data integrity and or operational testing of the storage system.

Replacing the original contents of the files in the workload simulation module includes creating a dummy data block and replicating the dummy block multiple times within a file to obtain a desired file size for each of a plurality of the dummy files. The dummy block may have a size that is user-definable. The dummy block has a size of at least <NUM> KB. Each file may have a size of at least <NUM> GB and the directory tree may include at least <NUM> files.

The contents of the dummy block may be stored in a buffer having the same size as the contents. The contents of the dummy block may be randomly generated. The dummy block may include binary content. The dummy block may include text content. Each of the dummy files may further include a unique data block that is different from the dummy block. Each of the dummy files may be created from a single dummy block that is used to create the dummy content for all of the dummy files.

In some implementations, the content of each of the dummy files is not created from a single dummy block. Rather, each of the dummy files may be created from a unique dummy block (i.e. each file having a different dummy block) in order to "fingerprint" the dummy files to aid in identifying the nature of any data corruption or integrity issues. For example, the unique dummy block may include a full file name and directory path. The unique dummy block may include a file name, timestamp and or inode.

Verifying the integrity of the dummy files with module <NUM> includes comparing a first block of one of the dummy files to a second block of the same dummy file. By way of non-limiting example, verifying the integrity of the dummy files may further include comparing the first block of the one dummy file to a third block of the same dummy file, and if the first, second and third blocks match perfectly, defining one of the three blocks as a "pristine block. " The pristine block may be defined after it has been compared to and perfectly matches n other blocks, where n may be an integer that can be customized by a user.

The pristine block is compared to all other dummy blocks in the same dummy file. If a dummy block does not match the pristine block, a determination may be made as to whether neighboring blocks also do not match the pristine block but match each other in order to help diagnose the nature of corruption or integrity issues. The process of defining a pristine block and comparing it to all other blocks in the same dummy file may be performed on each of the dummy files in the directory tree structure.

A checksum procedure may be used instead of using a pristine block to verify the integrity of all files in the directory tree structure that are no greater than a predetermined size. The predetermined size may be <NUM> KB. Alternatively, a checksum procedure may be used instead of a pristine block for all files in the directory tree structure that have less than a predetermined number of dummy blocks. In some implementations, the predetermined number of dummy blocks is two.

In some implementations, a file modification module <NUM> may be provided to modify the dummy files on the system of storage devices after the dummy files have been written by writing module <NUM>. In these implementations, the dummy files may be written, modified and read many times before their integrity is verified by module <NUM>.

Percentage selection module <NUM> and file selection module <NUM> may be configured to provide a first percentage of the files to be created with the dummy content that are to have the first file size. In some implementations, a user is allowed to select both the first percentage of files and the first file size. Modules <NUM> and <NUM> may also be configured to provide a second percentage of the files to be created with the dummy content that are to have the second file size. In some implementations, a user is allowed to select both the second percentage of files and the second file size. File size selection module <NUM> may also be configured to provide a default file size such that any remaining percentage of the files to be created with the dummy content that fall outside of the first and the second percentages (and any other similarly selected percentages) are to have the default file size. In some implementations, a user is allowed to select the default file size. For example, a user may select a first percentage of <NUM>%, a first file size of <NUM> kilobytes (K), a second percentage of <NUM>%, a second file size of <NUM> gigabytes (G), and a default file size of <NUM> kilobyte (K). In this example, when system <NUM> is creating dummy files, <NUM>% of the files will have a size of <NUM>, <NUM>% will have a size of <NUM>, and <NUM>% (the remainder of the files) will have the default file size of <NUM>. In some implementations, a third percentage and a third file size may be selected in a similar manner. In some implementations, additional percentages and file sizes may be selected. If the total of the percentages add up to be less than <NUM>%, the remaining percentage of files will have the default file size.

As previously mentioned, system <NUM> may be configured to simulate a plurality of workloads to simultaneously/concurrently write, read or write and read the directory tree structure and the dummy files of the plurality of simulated workloads to the system of storage devices. In such implementations, percentage selection module <NUM> and file size selection module <NUM> may be configured to allow a user to select different (unique) percentages and different (unique) files sizes for each of the simulated workloads. More specifically, percentage selection module <NUM> may be configured to select a first unique percentage of the files to be created with the dummy content that are to have a first file size for each of the plurality of simulated workloads. Percentage selection module <NUM> may also be configured to select a second unique percentage of the files to be created with the dummy content that are to have a second file size for each of the plurality of simulated workloads. File size selection module <NUM> may be configured to select a first unique file size for each of the plurality of simulated workloads, and a second unique file size for each of the plurality of simulated workloads. File size selection module <NUM> may also be configured to select a default unique file size for each of the plurality of simulated workloads, such that any remaining percentage of the files to be created with the dummy content that fall outside of the first and the second unique percentages and any other similarly selected percentages are to have the default unique file size.

In some implementations, by way of non-limiting example, the metadata may include at least one item from the group consisting of a file's owner, a group that the owner may belong to, file permissions, file size, tags and labels. Metadata operations that the workload can generate include changing access permissions, file over, and other tags that can be added to the file for real-world simulated testing.

The dummy files may be self-verified during the verifying step without accessing the real-world data from the deployed production environment.

In some implementations, computing platform(s) <NUM>, remote platform(s) <NUM>, and/or external resources <NUM> may be operatively linked via one or more electronic communication links. For example, such electronic communication links may be established, at least in part, via a network such as the Internet and/or other networks. It will be appreciated that this is not intended to be limiting, and that the scope of this disclosure includes implementations in which computing platform(s) <NUM>, remote platform(s) <NUM>, and/or external resources <NUM> may be operatively linked via some other communication media.

A given remote platform <NUM> may include one or more processors configured to execute computer program modules. The computer program modules may be configured to enable an expert or user associated with the given remote platform <NUM> to interface with system <NUM> and/or external resources <NUM>, and/or provide other functionality attributed herein to remote platform(s) <NUM>. By way of non-limiting example, a given remote platform <NUM> and/or a given computing platform <NUM> may include one or more of a server, a desktop computer, a laptop computer, a handheld computer, a tablet computing platform, a NetBook, a Smartphone, and/or other computing platforms.

External resources <NUM> may include sources of information outside of system <NUM>, external entities participating with system <NUM>, and/or other resources. In some implementations, some or all of the functionality attributed herein to external resources <NUM> may be provided by resources included in system <NUM>.

Computing platform(s) <NUM> may include electronic storage <NUM>, one or more processors <NUM>, and/or other components. Computing platform(s) <NUM> may include communication lines, or ports to enable the exchange of information with a network and/or other computing platforms. Illustration of computing platform(s) <NUM> in <FIG> is not intended to be limiting. Computing platform(s) <NUM> may include a plurality of hardware, software, and/or firmware components operating together to provide the functionality attributed herein to computing platform(s) <NUM>. For example, computing platform(s) <NUM> may be implemented by a cloud of computing platforms operating together as computing platform(s) <NUM>.

Electronic storage <NUM> may comprise non-transitory storage media that electronically stores information. The electronic storage media of electronic storage <NUM> may include one or both of system storage that is provided integrally (i.e., substantially non-removable) with computing platform(s) <NUM> and/or removable storage that is removably connectable to computing platform(s) <NUM> via, for example, a port (e.g., a USB port, a firewire port, etc.) or a drive (e.g., a disk drive, etc.). Electronic storage <NUM> may include one or more of optically readable storage media (e.g., optical disks, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard drive, floppy drive, etc.), electrical charge-based storage media (e.g., EEPROM, RAM, etc.), solid-state storage media (e.g., flash drive, etc.), and/or other electronically readable storage media. Electronic storage <NUM> may include one or more virtual storage resources (e.g., cloud storage, a virtual private network, and/or other virtual storage resources). Electronic storage <NUM> may store software algorithms, information determined by processor(s) <NUM>, information received from computing platform(s) <NUM>, information received from remote platform(s) <NUM>, and/or other information that enables computing platform(s) <NUM> to function as described herein.

Processor(s) <NUM> may be configured to provide information processing capabilities in computing platform(s) <NUM>. As such, processor(s) <NUM> may include one or more of a digital processor, an analog processor, a digital circuit designed to process information, an analog circuit designed to process information, a state machine, and/or other mechanisms for electronically processing information. Although processor(s) <NUM> is shown in <FIG> as a single entity, this is for illustrative purposes only. In some implementations, processor(s) <NUM> may include a plurality of processing units. These processing units may be physically located within the same device, or processor(s) <NUM> may represent processing functionality of a plurality of devices operating in coordination. Processor(s) <NUM> may be configured to execute modules <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and/or <NUM>, and/or other modules. Processor(s) <NUM> may be configured to execute modules <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and/or <NUM>, and/or other modules by software; hardware; firmware; some combination of software, hardware, and/or firmware; and/or other mechanisms for configuring processing capabilities on processor(s) <NUM>. As used herein, the term "module" may refer to any component or set of components that perform the functionality attributed to the module. This may include one or more physical processors during execution of processor readable instructions, the processor readable instructions, circuitry, hardware, storage media, or any other components.

It should be appreciated that although modules <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and/or <NUM> are illustrated in <FIG> as being implemented within a single processing unit, in implementations in which processor(s) <NUM> includes multiple processing units, one or more of modules <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and/or <NUM> may be implemented remotely from the other modules. The description of the functionality provided by the different modules <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and/or <NUM> described below is for illustrative purposes, and is not intended to be limiting, as any of modules <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and/or <NUM> may provide more or less functionality than is described. For example, one or more of modules <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and/or <NUM> may be eliminated, and some or all of its functionality may be provided by other ones of modules <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and/or <NUM>. As another example, processor(s) <NUM> may be configured to execute one or more additional modules that may perform some or all of the functionality attributed below to one of modules <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and/or <NUM>.

<FIG>, <FIG>, <FIG>, and/or 2D illustrates a method <NUM> for simulating real-world IO workload for testing in a parallel and distributed storage system, in accordance with one or more implementations. The operations of method <NUM> presented below are intended to be illustrative. In some implementations, method <NUM> may be accomplished with one or more additional operations not described. Additionally, the order in which the operations of method <NUM> are illustrated in <FIG>, <FIG>, <FIG>, and/or 2D and described below is not intended to be limiting.

In some implementations, method <NUM> may be implemented in one or more processing devices (e.g., a digital processor, an analog processor, a digital circuit designed to process information, an analog circuit designed to process information, a state machine, and/or other mechanisms for electronically processing information). The one or more processing devices may include one or more devices executing some or all of the operations of method <NUM> in response to instructions stored electronically on an electronic storage medium. The one or more processing devices may include one or more devices configured through hardware, firmware, and/or software to be specifically designed for execution of one or more of the operations of method <NUM>.

<FIG> illustrates method <NUM>, in accordance with one or more implementations.

An operation <NUM> includes identifying real-world data from a deployed production environment. The data may include a directory tree structure and files. The files include original metadata and original file contents. Operation <NUM> may be performed by one or more hardware processors configured by machine-readable instructions including a module that is the same as or similar to data identifying module <NUM>, in accordance with one or more implementations.

An operation <NUM> includes simulating a workload by using the original directory tree structure and the original metadata from the files and replacing the original contents of the files with dummy content to create dummy files. Operation <NUM> may be performed by one or more hardware processors configured by machine-readable instructions including a module that is the same as or similar to workload simulation module <NUM>, in accordance with one or more implementations.

An operation <NUM> includes writing the directory tree structure and dummy files to a system of storage devices. Operation <NUM> may be performed by one or more hardware processors configured by machine-readable instructions including a module that is the same as or similar to directory tree structure writing module <NUM>, in accordance with one or more implementations.

An operation <NUM> includes reading data from the directory tree structure and dummy files on the system of storage devices. Operation <NUM> may be performed by one or more hardware processors configured by machine-readable instructions including a module that is the same as or similar to data reading module <NUM>, in accordance with one or more implementations.

An operation <NUM> includes verifying the integrity of the dummy files over the course of a plurality of data management processes and a plurality of data availability processes employed by the storage system. Operation <NUM> may be performed by one or more hardware processors configured by machine-readable instructions including a module that is the same as or similar to integrity verification module <NUM>, in accordance with one or more implementations.

An operation <NUM> may include the step of modifying the dummy files on the system of storage devices. Operation <NUM> may be performed by one or more hardware processors configured by machine-readable instructions including a module that is the same as or similar to file modification module <NUM>, in accordance with one or more implementations. In some implementations, some or all of operations <NUM>, <NUM> and <NUM> may be performed many times on some or all of the dummy files.

An operation <NUM> may include selecting a first percentage of the files to be created with the dummy content that are to have a first file size. Operation <NUM> may be performed by one or more hardware processors configured by machine-readable instructions including a module that is the same as or similar to percentage selection module <NUM>, in accordance with one or more implementations.

An operation <NUM> may include selecting a second percentage of the files to be created with the dummy content that are to have a second file size. Operation <NUM> may be performed by one or more hardware processors configured by machine-readable instructions including a module that is the same as or similar to percentage selection module <NUM>, in accordance with one or more implementations.

An operation <NUM> may include simulating a plurality of workloads and simultaneously/concurrently writing, reading or writing and reading the directory tree structure and the dummy files of the plurality of simulated workloads to the system of storage devices. Operation <NUM> may be performed by one or more hardware processors configured by machine-readable instructions including a module that is the same as or similar to workload simulation module <NUM>, in accordance with one or more implementations.

An operation <NUM> may include selecting a first unique file size for each of the plurality of simulated workloads. Operation <NUM> may be performed by one or more hardware processors configured by machine-readable instructions including a module that is the same as or similar to file size selection module <NUM>, in accordance with one or more implementations.

An operation <NUM> may include selecting a first unique percentage of the files to be created with the dummy content that are to have the first file size for each of the plurality of simulated workloads. Operation <NUM> may be performed by one or more hardware processors configured by machine-readable instructions including a module that is the same as or similar to percentage selection module <NUM>, in accordance with one or more implementations.

An operation <NUM> may include selecting a second unique file size for each of the plurality of simulated workloads. Operation <NUM> may be performed by one or more hardware processors configured by machine-readable instructions including a module that is the same as or similar to file size selection module <NUM>, in accordance with one or more implementations.

An operation <NUM> may include selecting a second unique percentage of the files to be created with the dummy content that are to have the second file size for each of the plurality of simulated workloads. Operation <NUM> may be performed by one or more hardware processors configured by machine-readable instructions including a module that is the same as or similar to percentage selection module <NUM>, in accordance with one or more implementations.

An operation <NUM> may include selecting a default unique file size for each of the plurality of simulated workloads, such that any remaining percentage of the files to be created with the dummy content that fall outside of the first and the second unique percentages and any other similarly selected percentages are to have the default unique file size. Operation <NUM> may be performed by one or more hardware processors configured by machine-readable instructions including a module that is the same as or similar to file size selection module <NUM>, in accordance with one or more implementations.

The exemplary configuration file illustrated in <FIG> allows a user to select a first unique file size for each of a plurality of simulated workloads. In this example, <NUM> is selected for a first workload, <NUM> is selected for a second workload, and <NUM> is selected for a third workload. The configuration file also allows a user to select a first unique percentage of files to have the first file size for each of the plurality of simulated workloads. In this example, <NUM>% is selected for the first workload, <NUM>% is selected for the second workload, and <NUM>% is selected for the third workload. The configuration file also allows a user to select a second unique file size for each of the plurality of simulated workloads. In this example, <NUM> is selected for the first workload, <NUM> is selected for the second workload, and <NUM> is selected for the third workload. The configuration file also allows a user to select a second unique percentage of files to have the second file size for each of the plurality of simulated workloads. In this example, <NUM>% is selected for the first workload, <NUM>% is selected for the second workload, and <NUM>% is selected for the third workload. The configuration file also allows a user to select a default unique file size for each of the plurality of simulated workloads. In this example, <NUM> is selected for the first workload, <NUM> is selected for the second workload, and <NUM> is selected for the third workload. The configuration file may also allow the user to provide the number of workloads or threads to simulate (three in this example), their directory locations, the locations of dummy content, and/or other configuration parameters. In this example, "file /gpfs/gcp-<NUM>-me1/WORKLOAD-A /corp-file-aa. gz" may be the first of three data sets provided by a customer as input for the simulation tool.

Claim 1:
A system (<NUM>) configured for simulating real-world IO workload for testing a parallel and distributed storage system, the system (<NUM>) comprising:
one or more hardware processors (<NUM>) configured by machine-readable instructions (<NUM>) to:
identify real-world data from a deployed production environment (<NUM>), the data comprising a directory tree structure and files, wherein the files comprise original metadata and original file contents;
simulate a workload by using the original directory tree structure and the original
metadata from the files and replacing the original contents of the files with dummy content to create dummy files (<NUM>), wherein the step of replacing the original contents of the files comprises creating a dummy data block and replicating the dummy block multiple times within a file to obtain a desired file size for each of a plurality of the dummy files, wherein the dummy data block has a size of at least <NUM> KB;
write the directory tree structure and dummy files to a system of storage devices (<NUM>);
read data from the directory tree structure and dummy files on the system of storage devices (<NUM>); and
verify the integrity of the dummy files over the course of a plurality of data management processes and a plurality of data availability processes employed by the storage system by comparing different dummy blocks of the dummy files with each other (<NUM>).