Generating parameter values for snapshot schedules utilizing a reinforcement learning framework

An apparatus comprises a processing device configured to detect a request for an updated snapshot schedule for an information technology asset, and to determine a current state of the information technology asset comprising a set of snapshot parameters of a current snapshot schedule and one or more performance metric values. The processing device is also configured to generate, utilizing a reinforcement learning framework, an updated parameter value for at least one of the snapshot parameters based at least in part on the current state. The processing device is further configured to monitor performance of the information technology asset utilizing the updated snapshot schedule comprising the updated parameter value for the at least one snapshot parameter, and to update the reinforcement learning framework based at least in part on a subsequent state of the information technology asset determined while monitoring performance of the information technology asset utilizing the updated snapshot schedule.

RELATED APPLICATION

The present application claims priority to Chinese Patent Application No. 202310265727.1, filed on Mar. 13, 2023 and entitled “Generating Parameter Values for Snapshot Schedules Utilizing a Reinforcement Learning Framework,” which is incorporated by reference herein in its entirety.

FIELD

The field relates generally to information processing, and more particularly to management of information processing systems.

BACKGROUND

Information processing systems increasingly utilize reconfigurable virtual resources to meet changing user needs in an efficient, flexible and cost-effective manner. For example, cloud computing and storage systems implemented using virtual resources such as virtual machines have been widely adopted. Other virtual resources now coming into widespread use in information processing systems include Linux containers. Such containers may be used to provide at least a portion of the virtualization infrastructure of a given cloud-based information processing system. However, significant challenges can arise in the management of services in cloud-based information processing systems.

SUMMARY

Illustrative embodiments of the present disclosure provide techniques for generating parameter values for snapshot schedules utilizing a reinforcement learning framework.

In one embodiment, an apparatus comprises at least one processing device comprising a processor coupled to a memory. The at least one processing device is configured to detect a request for an updated snapshot schedule for an information technology asset in an information technology infrastructure, and to determine a current state of the information technology asset, the current state of the information technology asset comprising a set of snapshot parameters of a current snapshot schedule for the information technology asset and one or more performance metric values for the information technology asset. The at least one processing device is also configured to generate, utilizing a reinforcement learning framework, at least one updated parameter value for at least one snapshot parameter of the set of snapshot parameters to be utilized in the updated snapshot schedule for the information technology asset based at least in part on the current state of the information technology asset. The at least one processing device is further configured to monitor performance of the information technology asset utilizing the updated snapshot schedule comprising the at least one updated parameter value for the at least one snapshot parameter of the set of snapshot parameters, and to update the reinforcement learning framework based at least in part on a subsequent state of the information technology asset determined while monitoring the performance of the information technology asset utilizing the updated snapshot schedule.

DETAILED DESCRIPTION

FIG.1shows an information processing system100configured in accordance with an illustrative embodiment. The information processing system100is assumed to be built on at least one processing platform and provides functionality for generating parameter values for snapshot schedules utilizing a reinforcement learning framework. The information processing system100includes a set of client devices102-1,102-2, . . .102-M (collectively, client devices102) which are coupled to a network104. Also coupled to the network104is an information technology (IT) infrastructure105comprising one or more IT assets106, a snapshot database108, and a snapshot scheduling management system110. The IT assets106may comprise physical and/or virtual computing resources in the IT infrastructure105. Physical computing resources may include physical hardware such as servers, storage systems, networking equipment, Internet of Things (IoT) devices, other types of processing and computing devices including desktops, laptops, tablets, smartphones, etc. Virtual computing resources may include virtual machines (VMs), containers, etc.

The IT assets106of the IT infrastructure105may host applications that are utilized by respective ones of the client devices102, such as in accordance with a client-server computer program architecture. In some embodiments, the applications comprise web applications designed for delivery from assets in the IT infrastructure105to users (e.g., of client devices102) over the network104. Various other examples are possible, such as where one or more applications are used internal to the IT infrastructure105and not exposed to the client devices102. It is assumed that the client devices102and/or IT assets106of the IT infrastructure105utilize one or more machine learning algorithms as part of such applications. As described in further detail below, the snapshot scheduling management system110can advantageously be used to determine an optimal or improved snapshot schedule for the client devices102and/or IT assets106which balances various factors, including but not limited to performance and data security factors.

In some embodiments, the snapshot scheduling management system110is used for an enterprise system. For example, an enterprise may subscribe to or otherwise utilize the snapshot scheduling management system110for controlling snapshot policies for its assets (e.g., IT assets106in the IT infrastructure105). As used herein, the term “enterprise system” is intended to be construed broadly to include any group of systems or other computing devices. For example, the IT assets106of the IT infrastructure105may provide a portion of one or more enterprise systems. A given enterprise system may also or alternatively include one or more of the client devices102. In some embodiments, an enterprise system includes one or more data centers, cloud infrastructure comprising one or more clouds, etc. A given enterprise system, such as cloud infrastructure, may host assets that are associated with multiple enterprises (e.g., two or more different business, organizations or other entities).

The client devices102may comprise, for example, physical computing devices such as IoT devices, mobile telephones, laptop computers, tablet computers, desktop computers or other types of devices utilized by members of an enterprise, in any combination. Such devices are examples of what are more generally referred to herein as “processing devices.” Some of these processing devices are also generally referred to herein as “computers.” The client devices102may also or alternately comprise virtualized computing resources, such as VMs, containers, etc.

The client devices102in some embodiments comprise respective computers associated with a particular company, organization or other enterprise. Thus, the client devices102may be considered examples of assets of an enterprise system. In addition, at least portions of the information processing system100may also be referred to herein as collectively comprising one or more “enterprises.” Numerous other operating scenarios involving a wide variety of different types and arrangements of processing nodes are possible, as will be appreciated by those skilled in the art.

The snapshot database108is configured to store and record snapshots and various information that is used by the snapshot scheduling management system110for setting and updating snapshot policies for different ones of the IT assets106. Such information may include, for example, performance information characterizing performance of different types of IT assets106which are running different workloads, information utilized in a reinforcement learning algorithm used to control updates to snapshot policies (e.g., state information, an action space, reward information, etc.), etc. In some embodiments, one or more of the storage systems utilized to implement the snapshot database108comprise a scale-out all-flash content addressable storage array or other type of storage array.

Other particular types of storage products that can be used in implementing storage systems in illustrative embodiments include all-flash and hybrid flash storage arrays, software-defined storage products, cloud storage products, object-based storage products, and scale-out NAS clusters. Combinations of multiple ones of these and other storage products can also be used in implementing a given storage system in an illustrative embodiment.

Although not explicitly shown inFIG.1, one or more input-output devices such as keyboards, displays or other types of input-output devices may be used to support one or more user interfaces to the snapshot scheduling management system110, as well as to support communication between the snapshot scheduling management system110and other related systems and devices not explicitly shown.

The client devices102are configured to access or otherwise utilize the IT infrastructure105. In some embodiments, the client devices102are assumed to be associated with system administrators, IT managers or other authorized personnel responsible for managing the IT assets106of the IT infrastructure105(e.g., where such management includes setting or otherwise controlling snapshot scheduling policies for the IT assets106). For example, a given one of the client devices102may be operated by a user to access a graphical user interface (GUI) provided by the snapshot scheduling management system110to manage a snapshot schedule for one or more of the IT assets106of the IT infrastructure105. The snapshot scheduling management system110may be provided as a cloud service that is accessible by the given client device102to allow the user thereof to manage snapshot schedules for one or more of the IT assets106of the IT infrastructure105. In some embodiments, the IT assets106of the IT infrastructure105are owned or operated by the same enterprise that operates the snapshot scheduling management system110(e.g., where an enterprise such as a business provides support for the assets it operates). In other embodiments, the IT assets106of the IT infrastructure105may be owned or operated by one or more enterprises different than the enterprise which operates the snapshot scheduling management system110(e.g., a first enterprise provides support for assets that are owned by multiple different customers, business, etc.). Various other examples are possible.

In some embodiments, the client devices102and/or the IT assets106of the IT infrastructure105may implement host agents that are configured for automated transmission of information regarding snapshot schedules or policies. Such host agents may also or alternatively be configured to automatically receive from the snapshot scheduling management system110commands or instructions to update or modify snapshot schedules or policies.

It should be noted that a “host agent” as this term is generally used herein may comprise an automated entity, such as a software entity running on a processing device. Accordingly, a host agent need not be a human entity.

The snapshot scheduling management system110in theFIG.1embodiment is assumed to be implemented using at least one processing device. Each such processing device generally comprises at least one processor and an associated memory, and implements one or more functional modules or logic for controlling certain features of the snapshot scheduling management system110. In theFIG.1embodiment, the snapshot scheduling management system110implements IT asset state detection logic112, reinforcement learning logic114, and snapshot scheduling logic116. The snapshot scheduling management system110is configured to detect requests for updated snapshot schedules for the IT assets106in the IT infrastructure105. Responsive to detecting a request for an updated snapshot schedule for a given one of the IT assets106, the IT asset state detection logic112is configured to determine a current state of the given IT asset (e.g., a set of snapshot parameters of a current snapshot schedule for the given IT asset and one or more performance metric values for the IT asset). The reinforcement learning logic114is configured to generate at least one updated parameter value for at least one snapshot parameter of the set of snapshot parameters to be utilized in the updated snapshot schedule for the given IT asset based at least in part on the current state of the IT asset. The snapshot scheduling logic116is configured to attach the updated snapshot schedule to the given IT asset. The snapshot scheduling management system110then monitors performance of the given IT asset utilizing the updated snapshot schedule. The reinforcement learning logic114is further configured to update its reinforcement learning framework based at least in part on a subsequent state of the given IT asset determined by the IT asset state detection logic112while the snapshot scheduling management system110is monitoring the performance of the given IT asset utilizing the updated snapshot schedule.

It is to be appreciated that the particular arrangement of the client devices102, the IT infrastructure105and the snapshot scheduling management system110illustrated in theFIG.1embodiment is presented by way of example only, and alternative arrangements can be used in other embodiments. As discussed above, for example, the snapshot scheduling management system110(or portions of components thereof, such as one or more of the IT asset state detection logic112, the reinforcement learning logic114, and the snapshot scheduling logic116) may in some embodiments be implemented internal to one or more of the client devices102and/or the IT infrastructure105.

At least portions of the IT asset state detection logic112, the reinforcement learning logic114and the snapshot scheduling logic116may be implemented at least in part in the form of software that is stored in memory and executed by a processor.

The snapshot scheduling management system110and other portions of the information processing system100, as will be described in further detail below, may be part of cloud infrastructure.

The snapshot scheduling management system110and other components of the information processing system100in theFIG.1embodiment are assumed to be implemented using at least one processing platform comprising one or more processing devices each having a processor coupled to a memory. Such processing devices can illustratively include particular arrangements of compute, storage and network resources.

The client devices102, IT infrastructure105, the snapshot database108and the snapshot scheduling management system110or components thereof (e.g., the IT asset state detection logic112, the reinforcement learning logic114and the snapshot scheduling logic116) may be implemented on respective distinct processing platforms, although numerous other arrangements are possible. For example, in some embodiments at least portions of the snapshot scheduling management system110and one or more of the client devices102, the IT infrastructure105and/or the snapshot database108are implemented on the same processing platform. A given client device (e.g.,102-1) can therefore be implemented at least in part within at least one processing platform that implements at least a portion of the snapshot scheduling management system110.

Additional examples of processing platforms utilized to implement the snapshot scheduling management system110and other components of the information processing system100in illustrative embodiments will be described in more detail below in conjunction withFIGS.11and12.

It is to be understood that the particular set of elements shown inFIG.1for generating parameter values for snapshot schedules utilizing a reinforcement learning framework is presented by way of illustrative example only, and in other embodiments additional or alternative elements may be used. Thus, another embodiment may include additional or alternative systems, devices and other network entities, as well as different arrangements of modules and other components.

In this embodiment, the process includes steps200through208. These steps are assumed to be performed by the snapshot scheduling management system110utilizing the IT asset state detection logic112, the reinforcement learning logic114and the snapshot scheduling logic116. The process begins with step200, detecting a request for an updated snapshot schedule for an IT asset (e.g., one of the IT assets106) in an IT infrastructure (e.g., IT infrastructure105). The IT asset may comprise a VM.

In step202, a current state of the IT asset is determined. The current state of the IT asset may comprise a set of snapshot parameters of a current snapshot schedule for the IT asset and one or more performance metric values for the IT asset. The set of snapshot parameters may comprise a frequency at which snapshots are taken and a retention time for the snapshots. The one or more performance metric values for the IT asset may comprise at least one of information characterizing input-output operations per second (IOPS), throughput, processor resource utilization, and latency. The current state of the IT asset may further comprise configuration information of the IT asset, the configuration information comprising at least one of an operating system (OS) running on the IT asset, processing resources of the IT asset, memory resources of the IT asset, and storage resources of the IT asset. The current state of the IT asset may further comprise information characterizing application types of one or more applications running on the IT asset. The current state of the IT asset may further comprise information characterizing input-output (IO) patterns of one or more applications running on the IT asset, the information characterizing IO patterns comprising information characterizing at least one of IO size of IO operations, a read-write ratio of the IO operations, and a ratio of sequential to random IO operations.

TheFIG.2process continues in step204with generating, utilizing a reinforcement learning framework, at least one updated parameter value for at least one snapshot parameter of the set of snapshot parameters to be utilized in the updated snapshot schedule for the IT asset based at least in part on the current state of the IT asset determined in step202. In step206, performance of the IT asset is monitored while utilizing the updated snapshot schedule comprising the at least one updated parameter value for the at least one snapshot parameter of the set of snapshot parameters. The reinforcement learning framework is updated in step208based at least in part on a subsequent state of the IT asset determined while monitoring the performance of the IT asset utilizing the updated snapshot schedule.

In some embodiments, generating the at least one updated parameter value for the at least one snapshot parameter of the set of snapshot parameters in step204is further based at least in part on learned experience of the reinforcement learning framework, the learned experience comprising characterizations of whether different sets of one or more actions that modify parameter values for the set of snapshot parameters, taken from the current state of the IT asset, meet one or more designated goals for performance and data protection of the IT asset. The one or more designated goals may comprise meeting at least a threshold acceptable performance level while also meeting at least a threshold data protection level. The reinforcement learning framework may utilize a reward function which assigns a reward to the generated at least one updated parameter value for the at least one snapshot parameter of the set of snapshot parameters based at least in part on whether the subsequent state of the IT asset advances the one or more designated goals for performance and data protection of the IT asset. The request for the updated snapshot schedule for the IT asset may be detected in step200responsive to determining that a previous iteration of monitoring the performance of the IT asset did not meet the one or more designated goals for performance and data protection of the IT asset.

Step202may comprise determining whether the current state of the IT asset matches any of a plurality of state-action records of learned experience maintained by the reinforcement learning framework, each of the plurality of state-action records specifying a given value characterizing an extent to which taking a given set of one or more actions for modifying the at least one updated parameter value for the at least one snapshot parameter of the set of snapshot parameters meets one or more designated goals for performance and data protection of the IT asset. Responsive to determining that the current state of the IT asset does not match any of the plurality of state-action records, step204may include selecting a set of one or more actions for modifying the at least one updated parameter value for the at least one snapshot parameter of the set of snapshot parameters randomly from an action space, the action space defining permissible modifications to respective ones of the snapshot parameters in the set of snapshot parameters. Responsive to determining that the current state of the IT asset matches a given one of the plurality of state-action records, step204may include: selecting, with a first probability, a first set of one or more actions specified in the given one of the plurality of state-action records matching the current state of the IT asset; and selecting, with a second probability, a second set of one or more actions for modifying the at least one updated parameter value for the at least one snapshot parameter of the set of snapshot parameters randomly from an action space, the action space defining permissible modifications to respective ones of the snapshot parameters in the set of snapshot parameters.

Illustrative embodiments provide technical solutions for autonomous snapshot scheduling management (e.g., for VMs or other types of IT assets), based on a reinforcement learning framework that takes into account system performance impacts from applications, IO patterns and snapshot protection policies. In some embodiments, an end-to-end autonomous solution uses a machine learning approach which simulates the human brain to “learn” in a trial-and-error manner to find an optimal or improved snapshot schedule or policy for different IT assets (e.g., VMs). The machine learning approach in some embodiments utilizes a reinforcement learning framework that takes into account multiple applications or workloads which run on an IT asset (e.g., a VM), and determines a snapshot schedule or policy that provides an optimal or improved balance of different snapshot performance metrics (e.g., such as providing optimal or improved data protection while minimizing or reducing snapshot performance overhead). In this way, the technical solutions described herein can improve the overall performance of multiple applications while also improving data protection.

Various embodiments will be described below with respect to snapshot policies for VMs. It should be appreciated, however, that the technical solutions described herein may be applied for other types of IT assets and are not limited solely to use with managing snapshot scheduling or policies for VMs. VM snapshotting may be used to enforce service level agreements (SLAs) in VM environments such as VMware vSphere®. A snapshot preserves the state and data of a VM at a specific point in time. VM performance may be impacted by a snapshot schedule, application types (e.g., of one or more applications running on a VM), and IO patterns (e.g., of one or more applications running on the VM).

FIGS.3A-3Dshow respective plots300,305,310and315showing the impact of VM snapshotting on the performance of guest applications running inside a VM using different benchmark workloads, where the performance of such workloads is evaluated for three different datastores-Virtual Machine File System (VMFS), Virtual Storage Area Network (vSAN) and Virtual Volume (vVOL). For each of the workload scenarios, VM performance without any snapshotting is used as a baseline. Testing is performed while varying the number of snapshots (e.g., from 1 to 12). With the addition of each new VM snapshot, the benchmarks are re-run to capture new performance numbers.FIG.3Ashows a plot300of input-output operations per second (IOPS) as a function of a number of snapshots with random IO operations. Here, a higher number of IOPS corresponds to better performance.FIG.3Bshows a plot305of IOPS as a function of a number of snapshots with sequential IO operations, where again a higher IOPS corresponds to better performance. Flexible Input-Output (FIO) benchmark testing may be used for obtaining the plots300and305ofFIGS.3A and3B.FIG.3Cshows a plot310of processed requests per second as a function of a number of snapshots, which may be obtained using a Standard Performance Evaluation Corporation (SPEC) SPECjbb benchmark test, where the number of processed requests corresponds to jOPS (e.g., a measure of request injection rate), where a higher number of processed requests corresponds to better performance.FIG.3Dshows a plot315of database transactions per second as a function of a number of snapshots, where a higher number of transactions per second corresponds to better performance. In some embodiments, HammerDB benchmarking is utilized.

FIG.4shows a table400which summarizes guest application performance loss in the presence of one snapshot with a variety of workloads on the VMFS, vSAN and vVOL datastores. For each of the datastore scenarios, the performance loss is shown with one snapshot relative to the baseline performance without any snapshots on the respective datastore.

As can be seen from the plots300,305,310and315ofFIGS.3A-3Dand the table400ofFIG.4, guest application performance suffers greatly with IO-based applications (e.g., FIO, HammerDB) in the presence of snapshots, with as much as 85% reduction in guest IOPS or database transactions per second when using a VMFS datastore. CPU and memory-heavy workloads (e.g., SPECjbb) with no disk IO components remain unaffected in the presence of snapshots. There is a minimal impact on guest performance when snapshots are created on a vVOL datastore. Although sequential IO-based workloads have minimal impact, random IO-based workloads suffer greatly in a vSAN environment in the presence of snapshots. Thus, it is concluded that a combination of factors may impact VM performance, including: snapshot schedule (e.g., where the snapshot schedule includes snapshot taken frequency and snapshot retention parameters, which may impact the snapshot number and further impact performance over time); application type (e.g., such as VMFS, vSAN and vVOL); and IO pattern (e.g., sequential IO, random IO, IO patterns with specific read/write ratios).

Conventional approaches for VM snapshot scheduling thus suffers from various technical problems. An approach which statically sets a snapshot schedule (e.g., the frequency at which snapshots are taken, the length of time that snapshots are retained) leads to inefficiencies, as such an approach does not leverage VM performance impact of the snapshot schedule, IO patterns and application types. If the snapshot schedule is not gracefully configured, a static snapshot schedule may impact performance or SLAs. For example, setting a more aggressive snapshot schedule may lead to more snapshots while some applications with specific IO patterns may have significant performance loss. Setting a less aggressive snapshot schedule, on the other hand, could lead to not protecting data well.

In one approach, customers or end-users are guided to set the frequency at which snapshots are taken along with the retention period for snapshots for different protection groups (e.g., a group of VMs or other IT assets). This may be done at different intervals, such as standard frequency snapshots (e.g., every hour, every 4 hours, every 6 hours, every 8 hours, every 12 hours, daily, weekly, monthly, etc.) or high frequency snapshots (e.g., every 30 minutes, every hour, every 2 hours, every 4 hours, every 8 hours, every 12 hours, daily, weekly, monthly, etc.). Similarly, snapshot retention duration may be set in terms of hours, days, weeks, months, years, etc.

Another technical problem is that setting an optimal or improved snapshot schedule for different applications and IO patterns to achieve a best or improved combination of protection and performance is heavily dependent on experience and manual effort.

Illustrative embodiments provide technical solutions which simulate the human brain to “learn” an optimal or improved snapshot schedule using a trial-and-error approach with multiple iterations to improve the performance of multiple applications (e.g., with different IO patterns and other workload characteristics) while best protecting data. The technical solutions described herein leverage the system performance impacts of different snapshot schedule, IO pattern and application type combinations. This provides improved performance relative to approaches which statically set a snapshot schedule. Advantageously, the technical solutions described herein do not rely on human experience and manual effort. The technical solutions instead provide an end-to-end autonomous solution for setting and updating snapshot schedules, which may continuously learn an optimal or improved snapshot schedule (e.g., using a reinforcement machine learning approach that simulates the human brain to “learn” in a trial-and-error fashion).

An example implementation of the end-to-end autonomous solution for determining snapshot schedules will now be described. Suppose that an application A1is running on virtual machine VM1, and the goal is to find an optimal snapshot schedule S1for VM1that maximizes the following value:

α*Performance_Score+β*Data_Protection⁢_Score
where α and β are used to weight performance and data protection parameters for snapshot schedules in order to best protect data while minimizing the impact of VM snapshot operations on application performance.

For different applications (e.g., having different IO patterns) on different VMs, conventional approaches force customers or end-users to rely on manual effort and experience to try several steps to determine an optimal snapshot schedule. This may be viewed as similar to playing video games, such as a virtual golf game which is a famous use case of reinforcement learning where the goal is to hit a golf ball from any starting position into the hole with as few swings as possible. Here, the environment is a golf course with complex terrain types (e.g., organized from least to most difficult as the green, fairway, rough, sand trap and water hazard), with actions (e.g., aiming a swing in any of the cardinal directions north, east, south or west or halfway between the cardinal directions northeast, southeast, northwest or southwest) and a goal (e.g., hitting the golf ball from the starting position into the hole with as few swings as possible, where the golf ball moves some designated amount per swing).

The technical solutions described herein implement a reinforcement learning framework which helps customers or end-users to determine an optimal or improved snapshot schedule for multiple applications with fewer trials. Reinforcement learning is a class of learning problems framed in the context of planning on a Markov Decision Process (MDP), in which agents train a model by interacting with the environment (e.g., a VM snapshot schedule) and where the agents receive rewards from the actions performed correctly (e.g., which meet one or more designated performance goals for snapshot scheduling) and penalties from the actions performed incorrectly (e.g., which do not meet or further the one or more designated performance goals for snapshot scheduling). After multiple trial-and-error training rounds, the autonomous snapshot scheduling management solution will know how to reach the target (e.g., the one or more designated performance goals for snapshot scheduling) without a person explicitly telling the autonomous snapshot scheduling management solution how to do so.

FIG.5illustrates a reinforcement learning framework500, which includes a reinforcement learning agent501and an environment503(e.g., a VM or other IT asset to which a snapshot schedule is applied). As shown, the reinforcement learning agent501receives or observes a state St at a time t. The reinforcement learning agent501selects an action Atbased on its action selection policy, and transitions to a next state St+1at a time t+1. The reinforcement learning agent501receives a reward Rt+1at a time t+1. The reinforcement learning agent501leverages a reinforcement learning algorithm, which may include but is not limited to a Q-learning algorithm, a Deep Q Networks (DQN) algorithm, a Double DQN (DDQN) algorithm, etc., to update an action-value function Q(Si, Ai). The action-value function defines a long-term value of taking an action Aiin a state Si, as will be described in further detail below. Over time, the reinforcement learning agent501learns to pursue actions that lead to the greatest cumulative reward at any state.

Techniques for defining states, actions and rewards will now be described. A state space S includes a set of possible state values. A state St∈S is a vector of values from S={S1, S2, . . . , Sn} at time step t. Strepresents the schedule and runtime system status on a specific application at time step t:
St={VMinfo,Snapshot_Scheduleinfo,Applicationinfo,IO_patterninfo,runtime_infot}

VMinfois a static value representing information of the VM which may include, but is not limited to, guest operating system (OS) type, central processing unit (CPU) number, memory size, hard disk size, etc. Snapshot_Scheduleinfoincludes the VM snapshot schedule information, such as the frequency at which snapshots are taken and the retention period for snapshots. Applicationinfodetermines the format of the application, such as VMFS, vSAN, vVol, etc. IO_patterninforepresents the average IO pattern information during time step t, which includes but is not limited to IO size, read/write ratio, IO type (e.g., random, sequential, etc.). runtime_infotrepresents an average runtime status (e.g., such as performance status) during time step t, such as the rounding value of average total throughput, average CPU utilization, average latency during the execution of a snapshot schedule on the VM, etc.FIG.6shows an example 600 of St.

The action space will now be described. The reinforcement learning agent501, as noted above, observes the current state Stat each time step t and takes an action At. In some embodiments, the action Atinvolves modifying a single property of the snapshot schedule based on some specified snapshot scheduling performance parameter tuning policy. In some embodiments, the snapshot schedule includes two properties: the frequency at which snapshots are taken, denoted Snapshot_Taken_Frequency; and duration that snapshots are retained, denoted Snapshot_Retention. A snapshot schedule, denoted Snapshot_Schedule can thus be represented as:

{Snapshot_Taken⁢_Frequency=1⁢hour,Snapshot_Retention=7⁢days}
The acceptable values for Snapshot_Taken_Frequency may be 30 minutes, 1 hour, 2 hours, 4 hours and 8 hours. The acceptable values for Snapshot_Retention may be 5 days, 6 days, 7 days, 8 days, 9 days and 10 days. The snapshot schedule in the above example means that a snapshot of the VM is taken once an hour, and that snapshots are kept for 7 days. The customer or end-user could accept changes to this schedule, ranging from snapshots being taken 30 minutes, 2 hours, 4 hours or 8 hours, and could accept snapshots being retained for 5 days, 6 days, 8 days, 9 days or 10 days. The shorter the Snapshot_Taken_Frequency and the longer the Snapshot_Retention means the VM will get more protection. The acceptable values for Snapshot_Taken_Frequency and Snapshot_Retention may be set by customers or end-users.

The action space may include actions such as: changing the Snapshot_Taken_Frequency within its acceptable value list, such as moving from a current value to the next smaller or the next bigger value; and changing the Snapshot_Retention within its acceptable value list, such as moving from a current value to the next smaller or the next bigger value.FIG.7shows a table700, illustrating the action space with actions to change between current and new snapshot schedules for each iteration of a reinforcement learning framework.

The reward space will now be described. A reward function R is defined to guide the reinforcement learning agent501towards good solutions for a given objective (e.g., one or more designated performance goals for a snapshot schedule). The given objective, in some embodiments, is to find combinations of Snapshot_Taken_Frequency and Snapshot_Retention which have the most effective impact to snapshot schedule performance (e.g., best protecting data while minimizing the impact of VM snapshots on performance). The reward Rt+1may thus be defined as:

Rt+1=α*Performance_Score+β*Data_Protection⁢_Score
The Performance_Score may be defined as:

Performance_Score=W⁢1*(-Latencyaverage+LatencyinitialLatencyinitial)+W⁢2*(throughputaverage-throughputinitialthroughputinitial)
The Data_Protection_Score may be defined as:

Data_Protection⁢_Score=w⁢1*maximum⁢acceptable⁢SnaphotTakenFrequencyCurrent⁢SnaphotTakenFrequency+w⁢2*Current⁢Snapshot_Retentionmaximum⁢acceptable⁢Snaphot_Retention
Suppose that the initial performance of the VM is with latency as Latencyinitialand throughput as throughputinitial. The reinforcement learning agent501changes the VM snapshot schedule and during time step t, Latencyaverageis the average latency and throughputaverageis the average throughput. For the Performance_Score, the reward generated at time step t will be greater with less latency and more throughput being observed. For the Data_Protection_Score, the reward generated at time step t will be greater with shorter Snapshot_Taken_Frequency and longer the Snapshot_Retention. The value of the weights W1and W2will depend on the customer or end-user's focus on latency versus throughput for measuring performance, and Σi=1NWi=1, where Wi denotes the weight of factor i. Similarly, the value of the weights w1and w2will depend on the customer or end-user's focus on the frequency at which snapshots are taken versus the snapshot retention period, and Σi=1Nwi=1, where wi denotes the weight of the factor i. The values of α and β will depend on the customer or end-user's focus on performance versus data protection, where α+β=1. It should be noted that various other key performance indicators (KPIs) may be used to define the reward function in addition to or in place of throughput and latency, and that embodiments are not limited to the specific examples of throughput and latency.

FIG.8shows a system800for a VM802having a snapshot schedule which obtains an updated (e.g., improved, optimized) snapshot schedule from a snapshot scheduling agent804. The VM802may run on one of the IT assets106of the IT infrastructure105, on one of the client devices102in the system100ofFIG.1. The snapshot scheduling agent804implements a number of functional modules which are utilized in implementing a learning-based autonomous snapshot scheduling agent that generates updated (e.g., improved, optimized) snapshot schedules for the VM802. Such functional modules include a state detection module806, an action selection module808, a reward computation module810and an experience module812. The snapshot scheduling agent804aims to find out an optimal or improved snapshot schedule for the VM802, where the optimal or improved snapshot schedule provides the best data protection service for the applications running on the VM802while minimizing the snapshot performance overhead.

The state detection module806is configured to get the state Stof the VM802, which may include static and runtime information including but not limited to a runtime performance matrix (e.g., IO latency, IOPS, CPU utilization, memory utilization, disk utilization, bandwidth utilization, etc.), runtime or static IO load pattern information (e.g., IO size, read/write ratio, load type), a current snapshot schedule (e.g., the frequency at which snapshots are taken, the snapshot retention time, etc.), etc. The action selection module808is configured to observe the current state Stof the VM802and determine a snapshot schedule changing action At. The reward computation module810is configured to calculate the reward of action Atin state Stbased on the goal or objective (e.g., optimizing the snapshot schedule for VM802which provides a desired balance between data protection for applications running on the VM802while minimizing snapshot performance overhead). The experience module812is configured to utilize a reinforcement learning algorithm to update the experience Q(S, A) according to the current state St, action At, reward Rtand next state St+1. The experience Q(S, A) is a mapping between the environment states and actions that maximizes a long term reward.

FIG.9shows a process flow900which may be performed by the snapshot scheduling agent804. The process flow900starts in step901, and a training policy for the reinforcement learning framework is customized in step903. Customizing the training policy may include, for example, defining or updating the acceptable action space (e.g., acceptable values for Snapshot_Taken_Frequency and Snapshot_Retention), the maximum training attempts or iterations, an acceptable performance range (e.g., latency less than some designated threshold such as 5 milliseconds (ms)), etc. Customizing the training policy may also include defining or updating various weight values, such as α, β, W1, W2, w1and w2. In step905, the snapshot scheduling agent804receives a snapshot schedule change request from the VM802, and then the state detection module806gets the current state Stof the VM802. The snapshot schedule change request may be triggered manually or automatically (e.g., responsive to determining that the VM802is having performance issues, responsive to expiration of some designated period of time since the snapshot schedule for the VM802was last updated, etc.). The current state Stmay be defined with parameters such as VMinfo, Snapshot_Scheduleinfo, Applicationinfo, IO_patterninfoand runtime_infotas described above.

In step907, a determination is made as to whether the current state Stexists in the experience network Q. If the result of the step907determination is no, the process flow900proceeds to step909where an exploration and exploitation tradeoff parameter ε for time step t, denoted ε(t), is set to 1 to randomly select and take an action to explore the unknown state. ε(t) is the possibility of taking a random action for exploration at time step t. If the result of the step907determination is yes, the process flow900proceeds to step911where ε(t) is set to a value between 0 and 1, with the value of ε being gradually decreased at the end of each attempt or iteration. In step913, the action selection module808of the snapshot scheduling agent804selects with probability ε(t) a random action, otherwise (with probability 1−ε (t)) the action selection module808takes the best action (e.g., the action with the highest Q(St, At) that has been observed thus far). When the result of the step907determination is yes, the action selection module808enters an “exploration and exploitation tradeoff” mode where the current state Stis a known state and the value of ε(t) is set between (0,1) and decreases over successive training attempts or iterations. As experience is gained through successive training attempts or iterations, the snapshot scheduling agent804thus tends to leverage the learned experience (e.g., exploitation). Before having enough experience, the snapshot scheduling agent804tends to take random actions (e.g., exploration). When the result of the step907determination is no, the action selection module808enters an “exploration” mode where the current state Stis an unknown state. The snapshot scheduling agent804adds the current state Stto the experience network Q, and sets ε(t)=1 which means that the action selection module808will 100% explore (e.g., take random action) when the current state Stis a new state.

In step913, the action selection module808selects an action with probability ε, where ε is set in either step909or step911responsive to the step907determination. The snapshot scheduling agent804uses the selected action in step915to modify the snapshot schedule for the VM802. Such action may include, for example, modifying the Snapshot_Taken_Frequency or the Snapshot_Retention parameter to generate the updated snapshot schedule. In step917, the updated snapshot schedule is attached to the VM802, and the performance of the VM802is monitored while the reward computation module810gets the reward Rt+1and the state detection module806gets the next state St+1.

In step919, the experience module812uses the reinforcement learning algorithm and records of (St, At, Rt+1, St+1) to update Q(S, A) in order to approximate the optimal snapshot schedule policy. Various reinforcement learning algorithms may be used to extrapolate the optimal snapshot schedule policy, including but not limited to Q-learning, Deep Q networks (DQN), and double DQN (DDQN). The experience Q(S, A) is an action-value mapping which represents the long-term value of action A at any state. Q(S, A) represents the possibility of hitting the goal of the snapshot scheduling agent804in the future (e.g., even if the snapshot scheduling agent804does not hit the goal immediately after taking the current action).

Long-term value is illustrated inFIG.10, which shows various examples of actions that may be taken from a state S11001. At state S11001, after taking a first action A1a state S21002is reached. From state S21002, there is no possibility of hitting the one or more designated goals such as making the VM802performance within the acceptable value (from the experience learned thus far). Thus, Q(S1, A1)=0, which means the first action A1does not have long-term value. At state S11001, after taking a second action A2a state S31003is reached. In state S31003the one or more designated goals are not achieved, but upcoming actions starting from the state S31003do eventually lead to achieving the one or more designated goals. Thus, the second action A2has value for the long term instead of the short term, and Q(S1, A2)=2. At state S11001, after taking a third action A3the state S41004is reached where the one or more designated testing goals are achieved immediately, and thus Q(S1, A3)=10. The experience Q(Si, Ai) will get more and more accurate with every training iteration or episode. If enough training is performed, it will converge and represent the true Q-value.

Following step919, a determination is made in step921as to whether the one or more designated goals are achieved. If the one or more designated goals are not achieved, the process flow900proceeds to step923where a determination is made as to whether a maximum number of iterations is reached (e.g., where the maximum number of iterations is set in step903when the training policy is customized). If the result of the step923determination is yes (e.g., that the maximum number of iterations is not yet reached), then the process flow proceeds to step925where the state St+1is set as the current state Stand the process flow900returns to step907. Steps907through925are then repeated as necessary, with the value of ε being gradually decreased as the end of each iteration and the experience Q being updated over time. The process flow900ends in step927when step921determines that the one or more designated goals are achieved, or when step923determines that the maximum number of iterations is reached. It should be noted that even in cases where the one or more designated goals (e.g., achieving VM802performance within a predefined acceptable value, such as a threshold latency and/or throughput metric), the learned experience Q of trying different actions will benefit decision-making by the snapshot scheduling agent804for the future.

Illustrative embodiments of processing platforms utilized to implement functionality for generating parameter values for snapshot schedules utilizing a reinforcement learning framework will now be described in greater detail with reference toFIGS.11and12. Although described in the context of system100, these platforms may also be used to implement at least portions of other information processing systems in other embodiments.

FIG.11shows an example processing platform comprising cloud infrastructure1100. The cloud infrastructure1100comprises a combination of physical and virtual processing resources that may be utilized to implement at least a portion of the information processing system100inFIG.1. The cloud infrastructure1100comprises multiple virtual machines (VMs) and/or container sets1102-1,1102-2, . . .1102-L implemented using virtualization infrastructure1104. The virtualization infrastructure1104runs on physical infrastructure1105, and illustratively comprises one or more hypervisors and/or operating system level virtualization infrastructure. The operating system level virtualization infrastructure illustratively comprises kernel control groups of a Linux operating system or other type of operating system.

The cloud infrastructure1100further comprises sets of applications1110-1,1110-2,1110-L running on respective ones of the VMs/container sets1102-1,1102-2, . . .1102-L under the control of the virtualization infrastructure1104. The VMs/container sets1102may comprise respective VMs, respective sets of one or more containers, or respective sets of one or more containers running in VMs.

In some implementations of theFIG.11embodiment, the VMs/container sets1102comprise respective VMs implemented using virtualization infrastructure1104that comprises at least one hypervisor. A hypervisor platform may be used to implement a hypervisor within the virtualization infrastructure1104, where the hypervisor platform has an associated virtual infrastructure management system. The underlying physical machines may comprise one or more distributed processing platforms that include one or more storage systems.

In other implementations of theFIG.11embodiment, the VMs/container sets1102comprise respective containers implemented using virtualization infrastructure1104that provides operating system level virtualization functionality, such as support for Docker containers running on bare metal hosts, or Docker containers running on VMs. The containers are illustratively implemented using respective kernel control groups of the operating system.

The processing platform1200in this embodiment comprises a portion of system100and includes a plurality of processing devices, denoted1202-1,1202-2,1202-3, . . .1202-K, which communicate with one another over a network1204.

The processing device1202-1in the processing platform1200comprises a processor1210coupled to a memory1212.

The memory1212may comprise random access memory (RAM), read-only memory (ROM), flash memory or other types of memory, in any combination. The memory1212and other memories disclosed herein should be viewed as illustrative examples of what are more generally referred to as “processor-readable storage media” storing executable program code of one or more software programs.

Also included in the processing device1202-1is network interface circuitry1214, which is used to interface the processing device with the network1204and other system components, and may comprise conventional transceivers.

The other processing devices1202of the processing platform1200are assumed to be configured in a manner similar to that shown for processing device1202-1in the figure.

It should therefore be understood that in other embodiments different arrangements of additional or alternative elements may be used. Atleast a subset of these elements may be collectively implemented on a common processing platform, or each such element may be implemented on a separate processing platform.