Allocating cores to threads running on one or more processors of a storage system

Allocating CPU cores to a thread running in a system that supports multiple concurrent threads includes training a first model to optimize core allocations to threads using training data that includes performance data, initially allocating cores to threads based on the first model, and adjusting core allocations to threads based on a second model that uses run time data and run time performance measurements. The system may be a storage system. The training data may include I/O workload data obtained at customer sites. The I/O workload data may include data about I/O rates, thread execution times, system response times, and Logical Block Addresses. The training data may include data from a site that is expected to run the second model. The first model may categorize storage system workloads and determine core allocations for different categories of workloads. Initially allocating cores to threads may include using information from the first model.

TECHNICAL FIELD

This application relates to the field of computer systems and storage systems therefor and, more particularly, to the field of dynamically allocating processing cores to code threads running in a storage system.

BACKGROUND OF THE INVENTION

Host processor systems may store and retrieve data using a storage system containing a plurality of host interface units (I/O modules), physical storage units (e.g., disk drives, solid state drives), and disk interface units (disk adapters). The host systems access the storage system through a plurality of channels provided therewith. Host systems provide data and access control information through the channels to the storage system and the storage system provides data to the host systems also through the channels. The host systems do not address the physical storage units of the storage system directly, but rather, access what appears to the host systems as a plurality of logical volumes. The logical volumes may or may not correspond to the physical storage units. Allowing multiple host systems to access the single storage system allows the host systems to share data stored therein.

A storage system may include multiple code threads that facilitate storage system operations and tasks, including exchanging data with hosts, fetching data from disks, etc. When the storage system is initially configured, each of the threads may be statically allocated a certain number of processing cores for performing processing on the storage system. Since the number of allocated cores is static, there is a tendency to allocate what is expected to be the maximum number of cores needed throughout the life of each thread. However, in some cases, the workloads of threads may be dynamic and vary significantly. Allocating a number of cores corresponding to a maximum expected workload is inefficient in instances where a thread becomes much less busy or even idle for some periods of time. Moreover, since core allocation is a zero sum game, allocating more cores to one thread results in allocating less cores to another thread. A possible solution would be to have an operator manually adjust core allocation while the storage system is running, but this is impractical because it would require that a user to monitor and manage core allocation over a period of time and intuitively know when to make proper resource allocation changes and adjust settings.

Accordingly, it is desirable to provide a mechanism for automatically and dynamically adjusting processor core allocation in in response to actual workloads of code threads running in a storage system.

SUMMARY OF THE INVENTION

According to the system described herein, allocating CPU cores to a thread running in a system that supports multiple concurrent threads includes training a first model to optimize core allocations to threads using training data that includes performance data, initially allocating cores to threads based on the first model, and adjusting core allocations to threads based on a second model that uses run time data and run time performance measurements. The system may be a storage system. The training data may include I/O workload data obtained at customer sites. The I/O workload data may include data about I/O rates, thread execution times, system response times, and Logical Block Addresses. The training data may include data from a site that is expected to run the second model. The first model may categorize storage system workloads and determine core allocations for different categories of workloads. Initially allocating cores to threads may include using information provided by the first model.

According further to the system described herein, a non-transitory computer readable medium contains software that allocates CPU cores to a thread running in a system that supports multiple concurrent threads. The software includes executable code that trains a first model to optimize core allocations to threads using training data that includes performance data, executable code that initially allocates cores to threads based on the first model, and executable code that adjusts core allocations to threads based on a second model that uses run time data and run time performance measurements. The system may be a storage system. The training data may include I/O workload data obtained at customer sites. The I/O workload data may include data about I/O rates, thread execution times, system response times, and Logical Block Addresses. The training data may include data from a site that is expected to run the second model. The first model may categorize storage system workloads and determine core allocations for different categories of workloads. Executable code that initially allocates cores to threads may use information provided by the first model.

According further to the system described herein, a storage system includes a plurality of interconnected director boards, each having a CPU and providing functionality for a host adaptor that exchanges data with one of more hosts coupled to the storage system, a disk adaptor that exchanges data with one or more physical storage units of the storage system, and/or a remote adaptor that exchanges data with one or more remote storage systems. The storage system also includes a memory coupled to the director boards and a non-transitory computer-readable medium containing software that is executed by at least one of the CPUs on at least one of the director boards. The software includes executable code that trains a first model to optimize core allocations to threads using training data that includes performance data, executable code that initially allocates cores to threads based on the first model, and executable code that adjusts core allocations to threads based on a second model that uses run time data and run time performance measurements. The training data may include I/O workload data obtained at customer sites. The I/O workload data may include data about I/O rates, thread execution times, system response times, and Logical Block Addresses. The training data may include data from a site that is expected to run the second model. The first model may categorize storage system workloads and determine core allocations for different categories of workloads. Executable code that initially allocates cores to threads may use information provided by the first model.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

The system described herein provides an automated mechanism for dynamically allocating processing cores to code threads in a storage system by using predictive modeling, workload data, and performance data to initially allocate cores for threads and to adjust the allocated cores during run time.

FIG.1is a schematic illustration of a storage system20showing a relationship between a host22and a storage system24that may be used in connection with an embodiment of the system described herein. In an embodiment, the storage system24may be a Symmetrix or VMAX storage system produced by Dell EMC of Hopkinton, Mass.; however, the system described herein may operate with other appropriate types of storage systems. Also illustrated is another (remote) storage system26that may be similar to, or different from, the storage system24and may, in various embodiments, be coupled to the storage system24, using, for example, a network. The host22reads and writes data from and to the storage system24via an HA28(host adapter), which facilitates an interface between the host22and the storage system24. Although the diagram20shows the host22and the HA28, it will be appreciated by one of ordinary skill in the art that multiple host adaptors (possibly of different configurations) may be used and that one or more HAs may have one or more hosts coupled thereto.

In an embodiment of the system described herein, in various operations and scenarios, data from the storage system24may be copied to the remote storage system26via a link29. For example, the transfer of data may be part of a data mirroring or replication process that causes data on the remote storage system26to be identical to the data on the storage system24. Although only the one link29is shown, it is possible to have additional links between the storage systems24,26and to have links between one or both of the storage systems24,26and other storage systems (not shown). The storage system24may include a first plurality of remote adapter units (RA's)30a,30b,30c. The RA's30a-30cmay be coupled to the link29and be similar to the HA28, but are used to transfer data between the storage systems24,26.

The storage system24may include one or more physical storage units (including disks, solid state storage devices, etc.), each containing a different portion of data stored on the storage system24.FIG.1shows the storage system24having a plurality of physical storage units33a-33c. The storage system24(and/or remote storage system26) may be provided as a stand-alone device coupled to the host22as shown inFIG.1or, alternatively, the storage system24(and/or remote storage system26) may be part of a storage area network (SAN) that includes a plurality of other storage systems as well as routers, network connections, etc. (not shown inFIG.1). The storage systems may be coupled to a SAN fabric and/or be part of a SAN fabric. The system described herein may be implemented using software, hardware, and/or a combination of software and hardware where software may be stored in a computer readable medium and executed by one or more processors.

Each of the physical storage units33a-33cmay be coupled to a corresponding disk adapter unit (DA)35a-35cthat provides data to a corresponding one of the physical storage units33a-33cand receives data from a corresponding one of the physical storage units33a-33c. An internal data path exists between the DA's35a-35c, the HA28and the RA's30a-30cof the storage system24. Note that, in other embodiments, it is possible for more than one physical storage unit to be serviced by a DA and that it is possible for more than one DA to service a physical storage unit. The storage system24may also include a global memory37that may be used to facilitate data transferred between the DA's35a-35c, the HA28and the RA's30a-30cas well as facilitate other operations. The memory37may contain task indicators that indicate tasks to be performed by one or more of the DA's35a-35c, the HA28and/or the RA's30a-30c, and may contain a cache for data fetched from one or more of the physical storage units33a-33c.

The storage space in the storage system24that corresponds to the physical storage units33a-33cmay be subdivided into a plurality of volumes or logical devices. The logical devices may or may not correspond to the storage space of the physical storage units33a-33c. Thus, for example, the physical storage unit33amay contain a plurality of logical devices or, alternatively, a single logical device could span both of the physical storage units33a,33b. Similarly, the storage space for the remote storage system26may be subdivided into a plurality of volumes or logical devices, where each of the logical devices may or may not correspond to one or more physical storage units of the remote storage system26.

In some embodiments, an other host22′ may be provided. The other host22′ is coupled to the remote storage system26and may be used for disaster recovery so that, upon failure at a site containing the host22and the storage system24, operation may resume at a remote site containing the remote storage system26and the other host22′. In some cases, the host22may be directly coupled to the remote storage system26, thus protecting from failure of the storage system24without necessarily protecting from failure of the host22.

FIG.2is a schematic diagram40illustrating an embodiment of the storage system24where each of a plurality of directors42a-42nare coupled to the memory37. Each of the directors42a-42nrepresents at least one of the HA28, RAs30a-30c, or DAs35a-35c. The diagram40also shows an optional communication module (CM)44that provides an alternative communication path between the directors42a-42n. Each of the directors42a-42nmay be coupled to the CM44so that any one of the directors42a-42nmay send a message and/or data to any other one of the directors42a-42nwithout needing to go through the memory37. The CM44may be implemented using conventional MUX/router technology where one of the directors42a-42nthat is sending data provides an appropriate address to cause a message and/or data to be received by an intended one of the directors42a-42nthat is receiving the data. Some or all of the functionality of the CM44may be implemented using one or more of the directors42a-42nso that, for example, the directors42a-42nmay be interconnected directly with the interconnection functionality being provided on each of the directors42a-42n. In addition, one or more of the directors42a-42nmay be able to broadcast a message to all of the other directors42a-42nat the same time.

In some embodiments, one or more of the directors42a-42nmay have multiple processor systems thereon and thus may be able to perform functions for multiple discrete directors. In some embodiments, at least one of the directors42a-42nhaving multiple processor systems thereon may simultaneously perform the functions of at least two different types of directors (e.g., an HA and a DA). Furthermore, in some embodiments, at least one of the directors42a-42nhaving multiple processor systems thereon may simultaneously perform the functions of at least one type of director and perform other processing with the other processing system. In addition, all or at least part of the global memory37may be provided on one or more of the directors42a-42nand shared with other ones of the directors42a-42n. In an embodiment, the features discussed in connection with the storage system24may be provided as one or more director boards having CPUs, memory (e.g., DRAM, etc.) and interfaces with Input/Output (I/O) modules.

Note that, although specific storage system configurations are disclosed in connection withFIGS.1and2, it should be understood that the system described herein may be implemented on any appropriate platform. Thus, the system described herein may be implemented using a platform like that described in connection withFIGS.1and2or may be implemented using a platform that is somewhat or even completely different from any particular platform described herein.

A storage area network (SAN) may be used to couple one or more host devices with one or more storage systems in a manner that allows reconfiguring connections without having to physically disconnect and reconnect cables from and to ports of the devices. A storage area network may be implemented using one or more switches to which the storage systems and the host devices are coupled. The switches may be programmed to allow connections between specific ports of devices coupled to the switches. A port that can initiate a data-path connection may be called an “initiator” port while the other port may be deemed a “target” port.

Referring toFIG.3, a schematic illustration showing hardware and software layers for a storage system100that is similar to (or possibly identical to) the storage systems24,26discussed elsewhere herein. The storage system100has independent hypervisors as threads, which are described in more detail elsewhere herein. A hardware layer110includes hardware components for the storage system100, such as memory and processors (CPUs) and/or other components like that discussed in connection withFIGS.1and2. A storage system operating system (OS) layer120is shown as the operating system for the storage system100. In an embodiment, the OS layer120may be a Symmetrix storage system OS, such as Enginuity, with a Symm/K kernel that provides OS services and scheduling. Other operating systems may be used, such as the Linux operating system.

An instance is a single binary image of an OS that performs a specific set of operations. In an embodiment, there may be up to eight instances configured on a director board at any given time. A thread is a separately schedulable set of code or process of an instance. Threads may be co-operative and/or preemptive, and may be scheduled by the OS. An instance may run on more than one core, that is, an instance may provide a symmetric multiprocessing (SMP) environment to threads running within the instance.

According to the system described herein, a thread may be provided that runs as a hypervisor within the storage system OS environment. As previously discussed, a hypervisor is a software implementation providing a software virtualization environment in which other software may run with the appearance of having full access to the underlying system hardware, but in which such access is actually under the complete control of the hypervisor. The hypervisor running as the OS thread is a container hypervisor. The container hypervisor may manage a virtual hardware environment for a guest operating system (Guest OS), and, in an embodiment, the container hypervisor may run multiple OS threads (e.g.,1to N threads) within a single instance. The Guest OS is an operating system that may be loaded by a thread of the container hypervisor, and runs in the virtual environment provided by the container hypervisor. The Guest OS may also access real hardware devices attached to a director board using a virtual device provided by the container hypervisor or via a peripheral component interconnect (PCI) pass-through device/driver. There may be multiple container hypervisors running within a single instance at the same time. There may also be multiple container hypervisors running within different instances on the same director board at the same time.

InFIG.3, a hypervisor layer130is shown as including hypervisor-A131and hypervisor-B132that may be examples of container hypervisors in accordance with the system described herein. Each of the container hypervisors131,132may run as threads embedded within the storage system OS operating environment (the storage system OS120). The container hypervisor131is shown running as a thread to and may be running independently of the container hypervisor132. The container hypervisor132is shown running two threads t1and t2. These threads may run independently of each other as well as the thread to of the container hypervisor131. In each case, the threads t0, t1and t2of the container hypervisors131,132may run as threads of one or more instances of the storage system OS120. For example, in an embodiment, the container hypervisors131,132may be threads running as part of an Enginuity instance or a Linux instance. The container hypervisors131,132may be scheduled like any other thread and may be preempted and interrupted as well as started and stopped. Advantageously, since the container hypervisors131,132runs as threads within the storage system OS environment, physical resource sharing of the underlying hardware is already provided for according to the storage system OS scheduling.

According to an embodiment of the system described herein, a Guest OS140is loaded using the thread to of the container hypervisor-A131and, for example, runs an application in the virtual environment provided thereby. As shown, a Guest OS151may be loaded using independent threads t1, t2of the container hypervisor132. As further discussed elsewhere herein, threads t0, t1and t2may all be run independently of each other. The ability to run a container hypervisor as a storage system OS thread provides that the storage system100may run with no performance penalty until the container hypervisor thread is enabled. Even when the hypervisor thread is enabled and running an application in a Guest OS, the performance impact may be controlled. Additionally, developments in physical hardware may be accommodated through a software development process that is decoupled from modifications to the hypervisor code. Accordingly, releases of new storage system code, hypervisor code and Guest OS, and applications code may all be realized in an independent manner.

In various embodiments, the container hypervisors131,132may each provide for one or more of the following features: boot a Guest OS; run the Guest OS as a storage system OS thread (e.g., Symm/K); be scheduled, preemptable, etc.; reset the Guest OS without restarting the instance; allow the Guest OS to access storage systems (e.g., Symmetrix) using a Cut-through Device (CTD); and allow the Guest OS to access the I/O Modules using a PCI pass-through device.

Referring toFIG.4, a schematic illustration shows a storage system200with nested hypervisors. The storage system200is similar to the storage system100discussed elsewhere herein which may include a hardware layer210and a storage system OS layer220. A Guest OS240may be loaded using the thread to of a container hypervisor (hypervisor-A)231. As shown in connection with a container hypervisor (hypervisor-B)232, the container hypervisor232may host one or more other hypervisors (hypervisor-C250). In various embodiments, the hypervisor-C250may be another container hypervisor and/or may be another type of hypervisor, such as an ESXi hypervisor from VMware, Inc. of Palo Alto, Calif. The ability to host another hypervisor (hypervisor-C250), as a nested hypervisor, provides the capability of the system200to host any guest operating system, such as Guest OS's251,252(e.g., Linux) that may be hosted by the hypervisor250(e.g., ESXi) itself without needing to modify the code of the container hypervisor232. It is noted that additional layers of hypervisors may further be nested in accordance with the system described herein. By embedding hypervisors within hypervisors in a storage system environment in the manner according to the system described herein, physical resource sharing may be provided using the storage system OS scheduling and, thereby, resource trampling that may occur with the addition of another hypervisor, without such system OS scheduling, is avoided.

Referring toFIG.5, a flow diagram300shows processing for operating a hypervisor and a Guest OS according to various embodiments of the system described herein. At a step302, a container hypervisor is run as a thread of an underlying OS, for example, a storage system OS, such as Enginuity with Symm/K operating a Symmetrix storage system or the Linux operating system. After the step302, processing proceeds to a step304where a Guest OS is loaded using the container hypervisor based on the thread of the storage system OS. The container hypervisor may be run independently of the Guest OS and independently of other hypervisors running as other threads of the storage system OS. After the step304, processing proceeds to a step306where the hypervisor accesses resources according to a scheduler of the storage system OS and in connection with resource requirements of the Guest OS (and/or an application of the Guest OS). As further discussed elsewhere herein, the hypervisor may share resources with the other hypervisors according to the scheduling of the storage system OS. In an embodiment, the container hypervisor may be embedded with the storage system OS. As further discussed elsewhere herein, code of container hypervisor may be modified independently of code of the Guest OS and/or code of other hypervisors running as threads of the storage system OS. After the step306, processing is complete. One or more of the above-noted processing steps may be implemented via executable code stored on a non-transitory computer readable medium and executable by at least one processor according to an embodiment of the system described herein.

Referring toFIG.6, a flow diagram350shows processing for operating nested hypervisors according to an embodiment of the system described herein. At a step352, a container hypervisor (e.g., a first hypervisor) is run as a thread of an underlying OS, for example, a storage system OS, such as Enginuity with Symm/K operating a Symmetrix storage system. After the step352, processing proceeds to a step354, where a second hypervisor is run nested, and/or embedded within, the first hypervisor (the container hypervisor). In various embodiments, the second hypervisor may be a known hypervisor (e.g., ESXi) and/or may be another container hypervisor. Other hypervisors may be further nested in accordance with the system described herein. After the step354, processing proceeds to a step356, where a Guest OS is loaded using the first (container) hypervisor and the second hypervisor. After the step356, processing is complete. One or more of the above-noted processing steps may be implemented via executable code stored on a non-transitory computer readable medium and executable by at least one processor according to an embodiment of the system described herein.

According to another embodiment, by using a thread of a container hypervisor in the storage system OS environment (e.g., Enginuity running Symm/K), it is possible for a Guest OS to operate in several modes. The container hypervisor thread may inherit the same number of CPU cores as that of the OS instance and may run as a single thread on those cores when active. However, since the container hypervisor is running as a thread, rather than being scheduled as an OS instance, as described elsewhere herein, other OS threads may also continue to run on other cores in the same SMP environment. The use of the OS scheduling algorithms (e.g., Symm/K) for scheduling the threads of the container hypervisors thus provide the ability to schedule fractions of CPU time on multiple cores for the Guest OS's. Furthermore, it is possible for the container hypervisor to allocate fewer virtual cores than physical cores available to the instance, and allow the Guest OS to operate SMP on those cores while still allowing other OS threads to operate with full CPU core resources, and to adjust the CPU allocation between Guest OS's and other threads. In an embodiment, in a VMAX system from EMC Corporation of Hopkinton, Mass., the granularity of the CPU time scheduling according to the system described herein may be on the order of 500 microseconds or less.

Referring toFIG.7, a schematic illustration shows a storage system500that is similar to the storage systems100,200discussed elsewhere herein which may include a hardware layer510and a storage system OS layer520. The storage system500has fractional SMP capabilities extended to one or more Guest OS's540,551,552according to an embodiment of the system described herein. The storage system500includes a container hypervisor layer530, with a container hypervisor531(hypervisor-A) and a container hypervisor532(hypervisor-B) illustrated by way of example. The Guest OS's540,551and552may be loaded using the container hypervisors531,532. The container hypervisors531,532map virtual CPU cores to the physical CPU cores511,512of the hardware layer510. That is, in accordance with the system described herein, the one or more Guest OS's may only have access to a different number of available CPU cores (virtual CPU cores) than are available as physical CPU cores on the hardware510. Through the use of the container hypervisors531,532running as storage system OS threads t0, t1, and t2(rather than being scheduled as storage system OS instances), the system described herein provides for the ability to schedule fractions of CPU time on multiple cores for one or more of the Guest OS's540,551,552according to the scheduling algorithms of the storage system OS components (e.g., Symm/K).

The scheduling of fractional CPU time on the physical CPU cores511,512is shown schematically as fractions511a-cand512a-cof each of the CPU cores511,512. Each of the threads t0, t1, and t2of the container hypervisors531,532may operate in an SMP regime on multiple ones of the cores511,512while allowing others of the threads to also operate with full CPU core resources. The system described herein provides for flexible control of physical CPU allocation between Guest OS's540,551,552without causing one or more of the Guest OS's540,551,552to become inactive due to resource overlaps. In this way, the Guest OS's540,551,552may run based on the threads of the container hypervisors531,532using varying amounts of CPU time per CPU core in an SMP regime.

Each of the hypervisors and the base OS of a storage system may schedule and run one or more threads that compete for resources of the storage system, including use of the CPU cores of the system. It is desirable to be able to dynamically allocate CPU cores to each of the threads according to the workloads of each of the threads at different times. Thus, for example, a particular thread may be allocated more cores during a time when the thread has a relatively heavy workload than when the thread has a relatively light workload. Ideally, the dynamic allocation would occur prior to the change in workload so that, for example, if it is expected that a thread will experience a significant workload increase at a particular time, it would be desirable to allocate additional cores to the thread just prior to the particular time in anticipation of the increased workload. Note also that, in some cases, allocating cores is a zero sum game where allocating resources to one thread deallocates the same resources from one or more other threads. Thus, it is useful to be able to predict activity for all of the threads of a system to be able to allocate cores to threads that are expected to be more active and to deallocate cores from threads that are expected to be less active. The system described herein uses a core allocation model with machine learning and artificial intelligence to provide a mechanism that allocates and deallocates cores to threads in anticipation of workloads of the threads.

Referring toFIG.8A, an offline training model800includes canned customer data802, which is data from I/O workloads that was previously obtained at customer sites and includes information such as I/O rates, thread execution times, system response times, Logical block address(es) (LBA), etc. The canned customer data802may be field-support data from storage systems of a number of customers. Using field support data rather than specific data at each customer site may reduce the model initial training period of a newly installed system. In some embodiments, the canned customer data802may be a combination of generic field support data and some percentage (e.g., 10%) of data from a specific customer for whom the core allocation model is being provided.

The canned customer data802is provided to a workload analyzer804, which may be implemented using a Deep Neural Network (DNN) that categorizes different workloads according to patterns in the canned customer data802detected by the DNN. The workload analyzer804derives a set of categories that represent workloads having closely related behavior profiles. The set of categories allow aggregation of complex system operations into a much smaller set of features that can be used as labels for subsequent processing stages. The canned customer data802also provides performance data806that corresponds to performance-related parameters, such as I/O latencies (detected at a host) and IOPS, along with a number of cores allocated in each situation.

The workload categories from the workload analyzer804and the performance data806are used to label time-sequenced performance data and provide the result thereof to a recurrent core allocation model808, which is a Recurrent Neural Network (RNN) that iteratively determines an optimal number of cores to allocate to threads corresponding to different workloads. Examples of data output from the workload analyzer804(DNN) that is processed by the recurrent core allocation model808(RNN) may include elements such as read probability, pattern signature(s), LBA range, block rate(s), etc. The output of the recurrent core allocation model808includes core counts that may be applied to each situation (workload category) that maximizes system performance for a next sample period. Maximizing system performance includes reducing read and write latency times and increasing data throughput rates. A core allocations module812represents different permutations of core allocations that are provided as input to the recurrent core allocation model808. The core allocations module812also receives the results from the recurrent core allocation model808for each iteration.

The offline training model800iteratively processes the canned customer data802to produce a table that contains workload characterizations (categories) and corresponding core allocations. As explained in more detail elsewhere herein, the table may be used during runtime to set initial core allocations for threads, but the number of cores for a thread may be dynamically adjusted. In some cases, however, the dynamic adjustments are constrained to inhibit significant changes while a thread is running.

Referring toFIG.8B, an online production model850is similar to the offline training model800. The online production model850uses actual customer workload data852provided in real time to obtain, using a DNN, performance data854and workload categorization data856which is similar to the performance data804and the workload categorization data806of the offline training model800. The workload categorization data854is used with the table (described above) generated by the offline training model800to set an initial number of cores to be allocated for threads.

The output categories from the workload analyzer854and the performance data856are used to provide labeled time-sequenced performance data to a recurrent core allocation model858, which is a Recurrent Neural Network (RNN) that iteratively determines an optimal number of cores to allocate to threads corresponding to workloads that are currently running. The recurrent core allocation model858is similar to the recurrent core allocation model808of the offline training model800. The output of the recurrent core allocation model858includes core counts that may be applied to a current workload that will maximize performance for a next period. A core allocations module862represents different permutations of core allocations that are provided as input to the recurrent core allocation model858. The core allocations module862also receives the results from the recurrent core allocation model858for each iteration.

In an embodiment herein, the output of the recurrent core allocation model858is used to adjust a number of cores for threads currently running in a storage system. However, as discussed in more detail elsewhere herein, the amount of adjustment for each iteration is restricted. For example, the number of cores for iteration i may be given by:
NCi=NCi−1+(OMC−NCi+i)/DF
where NCiis a number of cores for a current iteration, NCi−1is a number of cores for a previous iteration, OMC is a number of cores determined by the online production model (described above), and DF is a damping factor that ranges from one to infinity. The value for DF may be set according to empirical observations of a system and other functional factors. Note that, as DF increases, the value for NCichanges more slowly between iterations. In an embodiment herein, DF is ten, but of course other values may be used.

Referring toFIG.9, a flow diagram900illustrates processing performed by the offline training model800, described above, in connection with determining numbers of cores to initially assigned to different categories of workloads. Processing begins at a first step902where the canned data802is loaded into the offline training model800. As discussed elsewhere herein, the canned data802includes field-support data for a number of customers and may also include some percentage (e.g., 10%) of data from a specific customer for whom the core allocation model is being provided. Following the step902is a step904where the different workloads provided in the canned data802is categorized by the workload categorizer804using a deep neural network (DNN) or some other appropriate technology capable of analyzing the canned data802. Following the step904is a step906where the workload categories and the performance data806from the canned data802are provided to the recurrent core allocation model808, discussed above. Following the step906is a step908where the recurrent core allocation model808makes an initial determination as to the number of cores to assign to different categories of workloads.

Following the step908is a step912where the recurrent core allocation model808improves the selection of the number of cores for each of the workloads. The recurrent core allocation model808essentially runs iteratively (using the core allocations module812) to attempt to continuously improve (optimize) the number of cores assigned to each workload category using the workload categories and the performance data from the canned data802. Note that since there are only a finite number of cores that can be allocated at any one time, allocating an additional core (or fraction of a core) to one thread deallocates the core (or fraction of a core) for another thread. Following the step912is a test step914where it is determined if the attempt at optimization at the step912improved overall system performance. If not, then processing is complete because there is apparently no more optimization. Otherwise, control transfers back to the step912, described above, for another iteration of optimization.

Referring toFIG.10, a flow diagram950indicates processing performed by the online production model850in connection with adjusting core allocations for threads of a system using real time data. Processing begins at a first step952where workloads of threads are categorized using, for example, the workload categorizer856, described above. Following the step952is a step954where initial core values are assigned to threads according to the table, described above, generated by the offline training model800. Following the step954is a step956where real time data, including workload categories and performance data, is provided to the recurrent core allocation model858, described above. Following the step956is a step958where the recurrent core allocation model858determines optimal core allocations based on current operational real time data, as described elsewhere herein. Following the step958is a step962where numbers of cores allocated to the various threads are adjusted, as described elsewhere herein. Following the step962, control transfers back to the step956, described above, for another iteration.

The system described herein may be implemented using any system that supports multiple concurrent threads. Of course, in systems that are not storage systems, the training data needs to be adjusted to whatever data is appropriate for the system being used. Essentially, the training data used for the training model800should be the same type of data that is expected to be processed for the online production model.

Various embodiments discussed herein may be combined with each other in appropriate combinations in connection with the system described herein. Additionally, in some instances, the order of steps in the flow diagrams, flowcharts and/or described flow processing may be modified, where appropriate. Further, various aspects of the system described herein may be implemented using software, hardware, a combination of software and hardware and/or other computer-implemented modules or devices having the described features and performing the described functions. The system may further include a display and/or other computer components for providing a suitable interface with a user and/or with other computers.

Software implementations of the system described herein may include executable code that is stored in a non-transitory computer-readable medium and executed by one or more processors. The computer-readable medium may include volatile memory and/or non-volatile memory, and may include, for example, a computer hard drive, ROM, RAM, flash memory, portable computer storage media such as a CD-ROM, a DVD-ROM, an SD card, a flash drive or other drive with, for example, a universal serial bus (USB) interface, and/or any other appropriate tangible or non-transitory computer-readable medium or computer memory on which executable code may be stored and executed by a processor. The system described herein may be used in connection with any appropriate operating system.