TECHNOLOGIES FOR MOVING WORKLOADS BETWEEN HARDWARE QUEUE MANAGERS

Technologies for moving workloads between hardware queue managers include a compute device. The compute device includes a set of hardware queue managers. Each hardware queue manager is to manage one or more queues of queue elements and each queue element is indicative of a data set to be operated on by a thread. The compute device also includes circuitry to execute a workload with a first hardware queue manager of the set of hardware queue managers, determine whether a workload migration condition is present, determine whether a second hardware queue manager of the set of hardware queue managers has sufficient capacity to manage a set of queues associated with the workload, move, in response to a determination that the second hardware queue manager does have sufficient capacity, the workload to the second hardware queue manager, and reduce, after the move of the workload to the second hardware queue manager, a power usage of the first hardware queue manager.

BACKGROUND

Some compute devices include multiple cores (e.g., processing units that each read and execute instructions, such as in separate threads) which operate on data using queues and a credit scheme. The credit scheme operates as a mechanism for determining whether a queue has room for additional data to be operated on (e.g., by a thread). In the credit scheme, some threads may produce queue elements, representing sets of data (e.g., packets) to be operated on by other threads. In adding a queue element to a queue to be processed by another thread (e.g., a worker thread or a consumer thread), a producer thread subtracts a credit from a credit pool. Conversely, a thread that removes the queue element from the queue and operates on the data adds a credit back to the credit pool. The management of the queues and the credits may be performed in software or, in some compute devices, in specialized circuitry (e.g., hardware queue managers) that enables more efficient management of the queues and credits. In systems that do utilize hardware queue managers (e.g., to provide queue and credit management operations for a relatively large number of cores and workloads), inefficiencies may arise, as each hardware queue manager operates at full power (e.g., not in a low power state) regardless of whether the hardware queue manager is managing a relatively low load or a relatively high load.

DETAILED DESCRIPTION OF THE DRAWINGS

Referring now toFIG. 1, a compute device110for moving a workload (e.g., an application, a virtual machine, a process, etc.) between hardware queue managers (HQMs)130is in communication with a client device150through a network160. The compute device110, in operation, may execute multiple workloads (e.g., on behalf of the client device150), using separate hardware queue managers130(e.g., one for each workload) and selectively move workloads off of one of the hardware queue managers130and onto another one of the hardware queue managers130to enable the original hardware queue manager130to be placed in a low power mode (e.g., deactivated). In doing so, and as explained in more detail herein, the compute device110continually determines whether conditions are present that would enable a workload to be moved from one hardware queue manager130to another hardware queue manager130, including determining whether the present level of activity of the workload satisfies a threshold (e.g., is relatively low) and whether another hardware queue manager130present in the compute device110has sufficient capacity to manage the workload. By moving workloads off of a hardware queue manager130, the compute device110may consolidate the workloads to fewer than the total amount of hardware queue managers130present in the compute device110and deactivate those that are not presently managing any workloads, thereby improving the power efficiency of the compute device110over typical compute devices.

The compute device110may be embodied as any type of device capable of performing the functions described herein, including executing a workload with one hardware queue manager130of a set of hardware queue managers130, determining whether a workload migration condition is present, determining whether another hardware queue manager130in the set of hardware queue managers130has sufficient capacity to manage a set of queues associated with the workload, move, in response to a determination that the other hardware queue manager130does have sufficient capacity, the workload to the other hardware queue manager130, and reduce, after moving the workload to the other hardware queue manager130, a power usage of the hardware queue manager130that the workload was moved from.

As shown inFIG. 1, the illustrative compute device110includes a compute engine112, an input/output (I/O) subsystem118, communication circuitry120, and one or more data storage devices124. Of course, in other embodiments, the compute device110may include other or additional components, such as those commonly found in a computer (e.g., a display, peripheral devices, etc.). Additionally, in some embodiments, one or more of the illustrative components may be incorporated in, or otherwise form a portion of, another component. The compute engine112may be embodied as any type of device or collection of devices capable of performing various compute functions described below. In some embodiments, the compute engine112may be embodied as a single device such as an integrated circuit, an embedded system, a field-programmable gate array (FPGA), a system-on-a-chip (SOC), or other integrated system or device. In the illustrative embodiment, the compute engine112includes or is embodied as a processor114and a memory116. The processor114may be embodied as any type of processor capable of performing the functions described herein. For example, the processor114may be embodied as a multi-core processor(s), a microcontroller, or other processor or processing/controlling circuit. In some embodiments, the processor114may be embodied as, include, or be coupled to an FPGA, an application specific integrated circuit (ASIC), reconfigurable hardware or hardware circuitry, or other specialized hardware to facilitate performance of the functions described herein. In the illustrative embodiment, the processor114includes a set of hardware queue managers132,134,136, and138and a corresponding set of cores142,144,146, and148(collectively, the cores140). The hardware queue managers130may each be embodied as any device or circuitry capable of managing the enqueueing of queue elements from producer threads and assigning the queue elements to worker threads and consumer threads of a workload for operation on the data associated with each queue element. Each of the cores140may be embodied as any device or circuitry capable of receiving instructions and performing calculations or actions based on those instructions and executing the threads of a workload to produce queue elements and to operate on the queue elements (e.g., with worker and/or consumer threads). While four hardware queue managers130and four cores140are shown in the processor114, it should be understood that in other embodiments, the number of hardware queue elements130and cores140may be different.

In some embodiments, 3D crosspoint memory (e.g., Intel 3D XPoint™ memory) may comprise a transistor-less stackable cross point architecture in which memory cells sit at the intersection of word lines and bit lines and are individually addressable and in which bit storage is based on a change in bulk resistance. In some embodiments, all or a portion of the main memory116may be integrated into the processor114. In operation, the main memory116may store various software and data used during operation such as workload data, hardware queue manager data, migration condition data, applications, programs, libraries, and drivers.

The compute engine112is communicatively coupled to other components of the compute device110via the I/O subsystem118, which may be embodied as circuitry and/or components to facilitate input/output operations with the compute engine112(e.g., with the processor114and/or the main memory116) and other components of the compute device110. For example, the I/O subsystem118may be embodied as, or otherwise include, memory controller hubs, input/output control hubs, integrated sensor hubs, firmware devices, communication links (e.g., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.), and/or other components and subsystems to facilitate the input/output operations. In some embodiments, the I/O subsystem118may form a portion of a system-on-a-chip (SoC) and be incorporated, along with one or more of the processor114, the main memory116, and other components of the compute device110, into the compute engine112.

The communication circuitry120may be embodied as any communication circuit, device, or collection thereof, capable of enabling communications over the network160between the compute device110and another compute device (e.g., the client device150, etc.). The communication circuitry120may be configured to use any one or more communication technology (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, Bluetooth®, Wi-Fi®, WiMAX, etc.) to effect such communication.

The illustrative communication circuitry120includes a network interface controller (NIC)122, which may also be referred to as a host fabric interface (HFI). The NIC122may be embodied as one or more add-in-boards, daughter cards, network interface cards, controller chips, chipsets, or other devices that may be used by the compute device110to connect with another compute device (e.g., the client device150, etc.). In some embodiments, the NIC122may be embodied as part of a system-on-a-chip (SoC) that includes one or more processors, or included on a multichip package that also contains one or more processors. In some embodiments, the NIC122may include a local processor (not shown) and/or a local memory (not shown) that are both local to the NIC122. In such embodiments, the local processor of the NIC122may be capable of performing one or more of the functions of the compute engine112described herein. Additionally or alternatively, in such embodiments, the local memory of the NIC122may be integrated into one or more components of the compute device110at the board level, socket level, chip level, and/or other levels.

The one or more illustrative data storage devices124may be embodied as any type of devices configured for short-term or long-term storage of data such as, for example, memory devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices. Each data storage device124may include a system partition that stores data and firmware code for the data storage device124. Each data storage device124may also include one or more operating system partitions that store data files and executables for operating systems.

The client device150may have components similar to those described inFIG. 1with reference to the compute device110. The description of those components of the compute device110is equally applicable to the description of components of the client device and is not repeated herein for clarity of the description. Further, it should be appreciated that any of the compute device110and the client device150may include other components, sub-components, and devices commonly found in a computing device, which are not discussed above in reference to the compute device110and not discussed herein for clarity of the description.

As described above, the compute device110and the client device150are illustratively in communication via the network160, which may be embodied as any type of wired or wireless communication network, including global networks (e.g., the Internet), local area networks (LANs) or wide area networks (WANs), cellular networks (e.g., Global System for Mobile Communications (GSM), 3G, Long Term Evolution (LTE), Worldwide Interoperability for Microwave Access (WiMAX), etc.), digital subscriber line (DSL) networks, cable networks (e.g., coaxial networks, fiber networks, etc.), or any combination thereof.

Referring now toFIG. 2, the compute device110may establish an environment200during operation. The illustrative environment200includes a network communicator210and a workload manager220. Each of the components of the environment200may be embodied as hardware, firmware, software, or a combination thereof. As such, in some embodiments, one or more of the components of the environment200may be embodied as circuitry or a collection of electrical devices (e.g., network communicator circuitry210, workload manager circuitry220, etc.). It should be appreciated that, in such embodiments, one or more of the network communicator circuitry210or workload manager circuitry220may form a portion of one or more of the compute engine112, the processor114, the memory116, the communication circuitry120, the I/O subsystem118and/or other components of the compute device110. In the illustrative embodiment, the environment200includes workload data202, which may be embodied as any data indicative of workloads and the threads associated with each workload, input data to be operated on by each workload (e.g., data received from the client device150) and output data produced by each workload (e.g., data to be sent to the client device150). The illustrative environment200also includes hardware queue manager data204, which may be embodied as any data indicative of identifiers of the hardware queue managers130, the present resources available on each hardware queue manager130(e.g., ports, queue identifiers, etc.), assignments of workloads to the hardware queue managers130, memory addresses used by each hardware queue manager130, the status (e.g., number of queue entities in each queue) of each queue managed by each hardware queue manager130, and the number of credits in a credit pool (e.g., a global variable shared by the threads of a given workload) for each workload associated with the corresponding hardware queue manager130. Additionally, the illustrative environment200includes migration condition data206, which may be embodied as any data indicative of conditions under which a workload should be migrated from one hardware queue manager130to another hardware queue manager130(e.g., a predefined level of activity such as a number of queue elements processed by the threads of the workload over a predefined time period, a time period typically associated with a relatively low level of activity or a relatively high level of activity, etc.) to either consolidate workloads onto a fewer number of hardware queue managers130(e.g., during periods of low activity) and deactivate the other hardware queue managers130, or to distribute the workloads across more of the hardware queue managers130(e.g., during periods of higher activity).

In the illustrative environment200, the network communicator210, which may be embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof as discussed above, is configured to facilitate inbound and outbound network communications (e.g., network traffic, network packets, network flows, etc.) to and from the compute device110, respectively. To do so, the network communicator210is configured to receive and process data packets from one system or computing device (e.g., the client device150, etc.) and to prepare and send data packets to a computing device or system (e.g., the client device150, etc.). Accordingly, in some embodiments, at least a portion of the functionality of the network communicator210may be performed by the communication circuitry120, and, in the illustrative embodiment, by the NIC122.

The workload manager220, which may be embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof, is configured to execute workloads and selectively consolidate workloads onto a relatively lower number of hardware queue managers130(e.g., during periods of low activity) and deactivate unused hardware queue managers130, or distribute the workloads across relatively more hardware queue managers130(e.g., during periods of higher activity). To do so, in the illustrative embodiment, the workload manager220includes a workload executor222, a migration condition determiner224, and a migration coordinator226. The workload executor222, in the illustrative embodiment, is configured to execute workloads using the cores140of the processor114. In doing so, the workload executor222may receive packets from the communication circuitry120using a dedicated core142(e.g., an Rx core) to produce queue element(s) representative of the data in the received packets. Further the workload executor222may operate on the data in the packets associated with the queue element(s) using worker threads corresponding to other cores, such as the cores144,146, and may send outgoing packets resulting from the operations of the worker threads using another core, such as the core148(e.g., a Tx core).

The migration condition determiner224, in the illustrative embodiment, is configured to continually determine whether a condition has occurred under which one or more workloads should moved between hardware queue managers130, either to consolidate the workloads onto fewer hardware queue managers130or to distribute the workloads across more hardware queue managers130. In the illustrative embodiment, the migration condition determiner224may compare a present level of activity associated with each workload (e.g., a number of packets being processed by the threads of the workload during a predefined period of time, such as a second or a minute) and determine whether the level of activity is low enough to satisfy a predefined threshold indicative of a low level of activity under which the workload should be moved to another hardware queue manager130to enable the source hardware queue manager130(e.g., the hardware queue manager130from which the workload is moved) to be deactivated. Conversely, the migration condition determiner224may determine whether the level of activity satisfies a higher predefined threshold, in which case the workload should be moved to a less heavily loaded hardware queue manager130. In some embodiments, the migration condition determiner224may be configured determine whether the present time is within a time period known to be associated with a low level of activity for a workload, and if so, determine that the workload should be consolidated with other workloads onto another hardware queue manager130or conversely that the workload should be moved to a less heavily loaded hardware queue manager130to accommodate an expected higher level of activity. The migration coordinator226, in the illustrative embodiment, is configured to determine which hardware queue manager130has sufficient capacity (e.g., a threshold number of ports, queue identifiers, etc.) to manage the queues for a workload to be moved. The migration coordinator is further to provide signals to the threads of the workload that the workload is to be moved to another hardware queue manager130and move the workload to the hardware queue manager130that has been determined to have sufficient capacity, including remapping memory addresses used by the workload, to enable the threads of the workload to communicate with the target hardware queue manager130(e.g., the hardware queue manager130to which the workload will be moved) rather than the source hardware queue manager130(e.g., the hardware queue manager130from which the workload will be moved).

Referring now toFIG. 3, the compute device110, in operation, may execute a method300for moving a workload between hardware queue managers130. The method300begins with block302, in which the compute device110executes a workload. In doing so, and as indicated in block304, the compute device110manages queues of the workload with a source hardware queue manager130associated with the workload (e.g., the hardware queue manager130to which the workload is presently assigned). In managing the queues, the compute device110, in the illustrative embodiment, tracks the status of the credit pool associated with the workload, as indicated in block306. Further, in the illustrative embodiment, the compute device110manages the enqueueing of queue elements (e.g., by one or more producer threads of the workload) and the dequeueing of queue elements (e.g., by worker threads and other consumer threads of the workload), as indicated in block308. As indicated in block310, the compute device110also determines whether a workload migration condition is present. In doing so, the compute device110may determine whether an activity level of the workload satisfies a predefined threshold, as indicated in block312. As described above, the activity level may be embodied as the number of packets processed by the threads of the workload over a predefined period of time, or another measure of throughput of the workload. As indicated in block314, the compute device110may determine whether the number of inflight packets (e.g., queue elements that have not been completely processed by the consumer thread(s)) satisfies a predefined threshold. In some embodiments, if the number of inflight packets is equal to or greater than a predefined number, the compute device110may determine that the risk of dropping the packets during a migration is too great and that a migration condition is not present. Additionally or alternatively, as indicated in block316, the compute device110may determine whether the present time is within a predefined time window (e.g., a time window associated with a particular level of activity that warrants moving the workload to another hardware queue manager130).

In block318, the compute device110determines the subsequent course of action as a function of whether a migration condition was determined to be present in block310. If a migration condition is not present, the method300loops back to block302, in which the compute device110continues execution of the workload. Otherwise, if a migration condition is present, the method300advances to block320in which the compute device110selects a hardware queue manager130from the set of hardware queue managers130as a candidate for receiving the workload. In block322, the compute device110determines whether the candidate hardware queue manager130has sufficient capacity to manage the queues of the workload. In doing so, the compute device110determines whether the candidate hardware queue manager130has sufficient available ports for the workload (e.g., the number of the ports that the thread(s) of the workload presently utilize on the source hardware queue manager130), as indicated in block324. Additionally or alternatively, the compute device110may determine whether the candidate hardware queue manager130has sufficient queue ids (e.g., available indexes to assign to queues utilized by the threads of the workload), as indicated in block326. Additionally, and as indicated in block328, the compute device110may subtract credit (e.g., in an atomic operation) from another workload utilizing the candidate hardware queue manager130to provide additional capacity for the workload that is to be moved. Subsequently, the method300advances to block330ofFIG. 4, in which the compute device110determines whether the candidate hardware queue manager130has sufficient capacity to manage the queues of the workload.

Referring now toFIG. 4, if the compute device110has determined that the candidate hardware queue manager130does not have sufficient capacity, the method300advances to block332, in which the compute device110determines whether other hardware queue managers130are present in the compute device110that have not been tested for their capacity. If so, the method300loops back to block320ofFIG. 3, in which the compute device110selects one of the other hardware queue managers130and determines whether that hardware queue manager130has sufficient capacity for the workload. Otherwise, the method300loops back to block302, in which the compute device110continues execution of the workload. Referring back to block330, if the compute device110instead determines that the candidate hardware queue manager130does have sufficient capacity, the method300advances to block334, in which the compute device110moves the workload to the candidate hardware queue manager130, which is referred to in the subsequent blocks as the target hardware queue manager130.

In moving the workload to the target hardware queue manager130, the compute device110may check, with one or more producer threads (e.g., with one or more of the cores assigned to provide packets to a hardware queue manager130for insertion into a queue as queue element(s)) whether a move flag (e.g., a designated bit) in the credit pool (e.g., a global variable indicative of the number of credits available for use by threads of the workload) has been set (e.g., to one), as indicated in block336. In response to detecting that the move flag has been set, the compute device110may donate any outstanding credits to the credit pool, as indicated in block338. Further, the producer thread(s) of the workload may send, in response to a detection that the move flag has been set, a move request to a driver for the hardware queue managers130(e.g., through an application programming interface (API) call), as indicated in block340. Further, the producer thread(s) may direct incoming packets (e.g. from the communication circuitry120) to the target hardware queue manger130, as indicated in block342. In some embodiments, the API call to the driver causes the redirection of incoming packets to the target hardware queue manager130(e.g., the driver may remap the page tables of the workload such that the target hardware queue manager130is mapped to the memory location that the source hardware queue manager130was previously mapped to).

As indicated in block344, the compute device110may check, with one or more consumer threads (e.g., threads that dequeue queue elements and operate on the underlying data), whether a move bit has been set in any of the queue elements. Further, in response to detection that the move bit has been set, the consumer thread(s) may discard the queue element(s) as dummy (e.g., fake) queue element(s) and send a move request to a driver for the hardware queue managers130(e.g., through an API call), as indicated in block346. While blocks336through342are performed by producer thread(s) and blocks344through346are performed by consumer thread(s), in the illustrative embodiment, blocks348through362are performed by a kernel executed by the compute device110to complete the move. In block348, the compute device110remaps logical addresses used by the workload from physical addresses used by the source hardware queue manager130to physical addresses used by the target hardware queue manager130. As indicated in block350, the compute device110may remap the credit pool (e.g., a global variable) for the workload. Further, as indicated in block the compute device110may remap ports used by the workload to those of the target hardware queue manager130(e.g., map logical memory addresses used by the thread(s) of the workload to physical memory addresses for ports of the target hardware queue manager130, rather than to physical memory addresses for ports of the source hardware queue manager130). As indicated in block354, the compute device110may set, with the kernel, a predefined move flag to alert producer thread(s) of the workload that they are to be moved to the target hardware queue manager130(e.g., the flag referenced in block336above). As indicated in block356, the compute device110may wait for queue elements to drain from the source hardware queue manager130(e.g., be processed by the worker and consumer threads of the workload and removed from the queues). The compute device110may continually poll the internal state of the hardware queue manager130to determine when the queue elements have completely drained from the source hardware queue manager130. As indicated in block358, after the queue elements have drained from the source hardware queue manager130, the compute device110may write dummy queue element(s) with a move bit set into the queues of the consumer threads (e.g., the queue elements referenced in blocks344and346). Additionally, the compute device110, in the illustrative embodiment, maps consumer queue pointers to correspond queue elements in the target hardware queue manager (e.g., queue elements resulting from the producer thread(s) redirecting incoming packets to the target hardware queue manager130in block342), as indicated in block360. Further, the compute device110, through the kernel, may reset resources of the source hardware queue manager (e.g., wiping any variables or other data maintained by the source hardware queue manager), as indicated in block362. Subsequently, the method300advances to block364ofFIG. 5, in which the compute device110may reduce a power consumption of the source hardware queue manager (e.g., if the source hardware queue manager is no longer assigned to any workloads). In doing so, in the illustrative embodiment, the compute device110deactivates (e.g., fully power gates) the source hardware queue manager130, as indicated in block366. Subsequently, the method300loops back to block302, in which the compute device110continues execution of the workload.

EXAMPLES

Example 1 includes a compute device comprising a plurality of hardware queue managers, wherein each hardware queue manager is to manage one or more queues of queue elements and wherein each queue element is indicative of a data set to be operated on by a thread; and circuitry to (i) execute a workload with a first hardware queue manager of the plurality of hardware queue managers, (ii) determine whether a workload migration condition is present, (iii) determine whether a second hardware queue manager of the plurality of hardware queue managers has sufficient capacity to manage a set of queues associated with the workload, (iv) move, in response to a determination that the second hardware queue manager does have sufficient capacity, the workload to the second hardware queue manager, and (v) reduce, after the move of the workload to the second hardware queue manager, a power usage of the first hardware queue manager.

Example 2 includes the subject matter of Example 1, and wherein to reduce the power usage of the first hardware queue manager comprises to deactivate the first hardware queue manager.

Example 3 includes the subject matter of any of Examples 1 and 2, and wherein to determine whether a workload migration condition is present comprises to determine whether an activity level of the workload satisfies a predefined threshold.

Example 4 includes the subject matter of any of Examples 1-3, and wherein to determine whether a workload migration condition is present comprises to determine whether the present time is within a predefined time window.

Example 5 includes the subject matter of any of Examples 1-4, and wherein to determine whether a workload migration condition is present comprises to determine whether a number of inflight packs associated with the workload satisfies a predefined threshold.

Example 6 includes the subject matter of any of Examples 1-5, and wherein to determine whether the second hardware queue manager has sufficient capacity comprises to determine whether the second hardware queue manager has a predefined number of available ports.

Example 7 includes the subject matter of any of Examples 1-6, and wherein the circuitry is further to subtract, prior to moving the workload to the second hardware queue manager, one or more credits from a credit pool associated with a second workload managed by the second hardware queue manager.

Example 8 includes the subject matter of any of Examples 1-7, and wherein to move the workload to the second hardware queue manager comprises to remap a logical address used by the workload from a first physical address used by the first hardware queue manager to a second physical address used by the second hardware queue manager.

Example 9 includes the subject matter of any of Examples 1-8, and wherein to move the workload to the second hardware queue manager comprises to direct packets from one or more producer threads of the workload to the second hardware queue manager.

Example 10 includes the subject matter of any of Examples 1-9, and wherein to move the workload to the second hardware queue manager comprises to set a predefined move flag in a credit pool used by one or more producer threads of the workload.

Example 11 includes the subject matter of any of Examples 1-10, and wherein to move the workload to the second hardware queue manager comprises to set a move bit in a queue element and enqueue the queue element into a queue used by a consumer thread of the workload.

Example 12 includes the subject matter of any of Examples 1-11, and wherein to move the workload to the second hardware queue manager comprises to send, in response to detection of a move flag in a credit pool or in a queue element, a move request from a thread of the workload to a hardware queue manager driver.

Example 13 includes the subject matter of any of Examples 1-12, and further including a plurality of processor cores, wherein each core corresponds to a thread of the workload.

Example 14 includes one or more machine-readable storage media comprising a plurality of instructions stored thereon that, in response to being executed, cause a compute device to execute a workload with a first hardware queue manager of a plurality of hardware queue managers, wherein each hardware queue manager is to manage one or more queues of queue elements and wherein each queue element is indicative of a data set to be operated on by a thread; determine whether a workload migration condition is present; determine whether a second hardware queue manager of the plurality of hardware queue managers has sufficient capacity to manage a set of queues associated with the workload; move, in response to a determination that the second hardware queue manager does have sufficient capacity, the workload to the second hardware queue manager; and reduce, after the move of the workload to the second hardware queue manager, a power usage of the first hardware queue manager.

Example 15 includes the subject matter of Example 14, and wherein to reduce the power usage of the first hardware queue manager comprises to deactivate the first hardware queue manager.

Example 16 includes the subject matter of any of Examples 14 and 15, and wherein to determine whether a workload migration condition is present comprises to determine whether an activity level of the workload satisfies a predefined threshold.

Example 17 includes the subject matter of any of Examples 14-16, and wherein to determine whether a workload migration condition is present comprises to determine whether the present time is within a predefined time window.

Example 18 includes the subject matter of any of Examples 14-17, and wherein to determine whether a workload migration condition is present comprises to determine whether a number of inflight packs associated with the workload satisfies a predefined threshold.

Example 19 includes the subject matter of any of Examples 14-18, and wherein to determine whether the second hardware queue manager has sufficient capacity comprises to determine whether the second hardware queue manager has a predefined number of available ports.

Example 20 includes the subject matter of any of Examples 14-19, and wherein the circuitry is further to subtract, prior to moving the workload to the second hardware queue manager, one or more credits from a credit pool associated with a second workload managed by the second hardware queue manager.

Example 21 includes the subject matter of any of Examples 14-20, and wherein to move the workload to the second hardware queue manager comprises to remap a logical address used by the workload from a first physical address used by the first hardware queue manager to a second physical address used by the second hardware queue manager.

Example 22 includes the subject matter of any of Examples 14-21, and wherein to move the workload to the second hardware queue manager comprises to direct packets from one or more producer threads of the workload to the second hardware queue manager.

Example 23 includes the subject matter of any of Examples 14-22, and wherein to move the workload to the second hardware queue manager comprises to set a predefined move flag in a credit pool used by one or more producer threads of the workload.

Example 24 includes the subject matter of any of Examples 14-23, and wherein to move the workload to the second hardware queue manager comprises to set a move bit in a queue element and enqueue the queue element into a queue used by a consumer thread of the workload.

Example 25 includes a compute device comprising circuitry for executing a workload with a first hardware queue manager of a plurality of hardware queue managers, wherein each hardware queue manager is to manage one or more queues of queue elements and wherein each queue element is indicative of a data set to be operated on by a thread; means for determining whether a workload migration condition is present; means for determining whether a second hardware queue manager of the plurality of hardware queue managers has sufficient capacity to manage a set of queues associated with the workload; means for moving, in response to a determination that the second hardware queue manager does have sufficient capacity, the workload to the second hardware queue manager; and circuitry for reducing, after the move of the workload to the second hardware queue manager, a power usage of the first hardware queue manager.

Example 26 includes a method comprising executing, by a compute device, a workload with a first hardware queue manager of a plurality of hardware queue managers, wherein each hardware queue manager is to manage one or more queues of queue elements and wherein each queue element is indicative of a data set to be operated on by a thread; determining, by the compute device, whether a workload migration condition is present; determining, by the compute device, whether a second hardware queue manager of the plurality of hardware queue managers has sufficient capacity to manage a set of queues associated with the workload; moving, by the compute device and in response to a determination that the second hardware queue manager does have sufficient capacity, the workload to the second hardware queue manager; and reducing, by the compute device and after the move of the workload to the second hardware queue manager, a power usage of the first hardware queue manager.

Example 27 includes the subject matter of Example 26, and wherein reducing the power usage of the first hardware queue manager comprises deactivating the first hardware queue manager.

Example 28 includes the subject matter of any of Examples 26 and 27, and wherein determining whether a workload migration condition is present comprises determining whether an activity level of the workload satisfies a predefined threshold.