Patent Description:
Packet processing applications typically provision a number of "worker" processing threads running on processor cores (sometimes called "worker cores") to perform the processing work of the applications. Worker cores consume packets from dedicated queues which in some scenarios is fed by one or more network interface controllers (NICs) or by input/output (I/O) threads. The number of worker cores provisioned is usually a function of the maximum predicted throughput. However, packet traffic rates vary widely both in short durations (e.g., seconds) and over longer periods of time (for example, many networks experience significantly less traffic at night or on a weekend).

In some packet processing applications, queues are polled continuously by the worker cores. This results in inefficient core resource utilization because the worker cores continue to poll the queues at a constant rate regardless of the current packet traffic rate. Some existing power saving schemes permit scaling up or down processor frequency depending on traffic rates. However, processor frequency changes impact usage of the entire set of queues accessed by a processor. If a processor core is polling more than one queue, changing the processor's frequency impacts the polling rate of all queues accessed by the processor. This approach does not take into account the individual traffic rate of each queue. Since some queues have a higher rate of traffic than others, this results in suboptimal processor utilization. <CIT> relates to power management of a multicore processor. One or more cores may be placed in a low power state, e.g. with reduced clock frequency. A core can be transitioned from a high power use state to a low power use state associated with deactivation of a corresponding queue. A measure of queue depth against a threshold is used to decide whether a deactivated queue is to be activated or to remain deactivated. <CIT> relates to power management in network packet processing applications. When a power management component detects network traffic becoming lower, processor frequencies may be scaled down to idle or sleep state. Depending on a queue state, a packet polling frequency from this queue is adapted. <CIT> relates to a power management controller to monitor ingress and egress processor queue levels. The controller may monitor a number of messages in a queue and may be notified of traffic threshold crossing. In response to a number of queued messages exceeding a first threshold, a processor may be turned on to a low-clock frequency state. In response to the queued messages exceeding a second threshold, the processor may be enabled to a higher operating frequency. <CIT> relates to a dynamic service management for multicore processors. A service with an increasing load may be reassigned from a slower processor to a faster processor and a service with a decreasing load may be reassigned from a faster processor to a slower processor. A service monitor may generate statistical parameters based on monitored data flow, e.g. a rate of packets processed.

The present invention provides a method of operating a system comprising a plurality of interface queues and a plurality of cores, a corresponding system and at least one machine readable medium, as defined in the claims.

Embodiments of the present invention provide an approach to dynamically assigning data packet traffic interface queues to cores running at different frequencies using power savings mechanisms without impacting packet processing performance. Polling rates of the interface queues may be adjusted based on the data packet transfer rate of interface queues by dynamically assigning high volume interface queues to high frequency cores and low volume interface queues to low frequency cores.

In one embodiment, cores may be divided into at least two classes: high frequency or low frequency. High frequency cores have their processor clock frequency set to a first predetermined speed. Low frequency cores have their processor clock frequency set to a second predetermined speed, lower than the first predetermined speed. In other embodiments of the present invention, other numbers of classes may be used, such as three, four, five, and so on, each class having a different processor clock frequency. Having more classes results in the capability for a finer granularity in controlling dynamic assignment of interface queues. Two classes will be described herein, but it is understood that any number of classes and associated processor frequencies, and various throughput thresholds to be used to match traffic to each class, may be used in various embodiments.

Embodiments of the present invention may be used with a software switch (also called a virtual switch). In one embodiment, the virtual switch is an Open vSwitch (OVS) application program integrated with Data Plane Development Kit (DPDK) (i.e., an OVS-DPDK) running on a multicore computing system for high performance virtual machine (VM) to virtual machine (or container to container), or physical machine (PHY) to virtual machine to physical machine (or PHY to container to PHY), packet transfers. In other embodiments, other virtual switch implementations may be used. To support many VMs or containers (such as <NUM> or more, for example), the virtual switch requires a multicore processing environment to scale up processing throughput as traffic rates increase.

<FIG> illustrates an example system architecture <NUM> of embodiments of the present invention. System <NUM> includes a collection <NUM> of I/O interfaces and interface queues. Collection <NUM> includes a plurality of physical or virtual interfaces, such as physical/virtual interface <NUM><NUM>, physical/virtual interface <NUM><NUM>,. physical/virtual interface N, where N is a natural number. Each physical/virtual interface includes one or more of a physical NIC, a virtual NIC, a VM, or any other physical or virtual component to provide data packets to system <NUM>. In some systems the number of physical/virtual interfaces may number in the tens, hundreds, or even thousands. Collection <NUM> includes a plurality of interface queues to store data packets, such as Q1 <NUM>, Q2 <NUM>, Q3 <NUM>, Q4 <NUM>, Q5 <NUM>,. QK-<NUM><NUM>, QK-<NUM><NUM>, QK-<NUM><NUM>, and QK <NUM>, where K is a natural number. In some systems the number of interface queues may number in the tens, hundreds, thousands, or even tens of thousands. The memory size allotted for each interface queue is implementation dependent. Any set of zero or more interface queues may be assigned to a physical/virtual interface, whereby the physical/virtual interface stores data packets into assigned interface queues. For example, physical/virtual interface <NUM><NUM> may be assigned interface queues Q1 <NUM>, Q2 <NUM>, and Q3 <NUM>; physical/virtual interface <NUM><NUM> may be assigned interface queues Q4 <NUM> and Q5 <NUM>; and physical/virtual interface N <NUM> may be assigned interface queues QK-<NUM><NUM>, QK-<NUM><NUM>, QK-<NUM><NUM>, and QK <NUM>.

System <NUM> includes a plurality of cores, such as core <NUM><NUM>, core <NUM><NUM>,. core M <NUM>, where M is a natural number. A core is running at a processor frequency. In an embodiment, a core is set to run at either a high frequency (HF) or a low frequency (LF). For example, core <NUM><NUM> and core <NUM><NUM> are in a high frequency class of cores <NUM> running at a high frequency, and core M is in a low frequency class of cores <NUM> running at a low frequency. Generally, a high frequency core is processing a higher traffic rate of packets than a low frequency core, and a high frequency core uses more power than a low frequency core. Any set of cores may be in the high frequency class of cores, and any set of cores may be in the low frequency class of cores, with the caveat that the two sets are mutually exclusive (a core can be moved between the classes but cannot be a high frequency core and a low frequency core at the same time).

A core can execute a packet processing functionality. For example, core <NUM><NUM> executes packet processing (PP) function <NUM><NUM>, core <NUM><NUM> executes PP <NUM><NUM>, and core M executes PP M <NUM>. Collectively, in one embodiment PP <NUM><NUM>, PP1 <NUM>,. PP M <NUM> are implemented in software as a virtual switch. A PP application includes zero or more queue polling logic elements to read data packets from one or more interface queues assigned to a core. In one embodiment, a queue polling logic element is implemented in software instructions being executed by the core (e.g., within the PP application). In another embodiment, a queue polling logic element is implemented in circuitry in the core. For example, high frequency core <NUM><NUM> includes queue polling <NUM><NUM> to read data packets from interface queues Q1 <NUM>, Q2 <NUM>, and Q3 <NUM> (assigned to physical/virtual interface <NUM><NUM> in this example). High frequency core <NUM><NUM> includes queue polling <NUM><NUM> to read data packets from interface queues Q4 <NUM> and Q5 <NUM> (assigned to physical /virtual interface <NUM><NUM> in this example). Low frequency core M <NUM> includes queue polling M <NUM> to read data packets from interface queues QK-<NUM><NUM>, QK-<NUM><NUM>, QK-<NUM><NUM>, and QK <NUM> (assigned to physical/virtual interface N <NUM> in this example).

System <NUM> includes shared memory <NUM> that is accessible by any of the cores over interconnect <NUM>. In an embodiment, access to shared memory <NUM> is controlled by a known mutual exclusion locking mechanism. In an embodiment, shared memory <NUM> stores one queue bitmask for each core in the system <NUM>. For example, shared memory <NUM> includes core <NUM> queue bitmask <NUM>, core <NUM> queue bitmask <NUM>,. core M queue bitmask <NUM>. A queue bitmask includes K bits, with one bit representing each interface queue. In an embodiment, a core (such as core <NUM><NUM> executing PP <NUM><NUM>) reads the core's associated queue bitmask <NUM> and when a bit of core <NUM> queue bitmask <NUM> is set to <NUM>, core <NUM><NUM> polls the interface queue referenced by the bit using the core's queue polling logic element (e.g., queue polling <NUM><NUM> of PP <NUM><NUM>).

System <NUM> includes core <NUM> to include monitor <NUM>. In an embodiment, core <NUM> is a standalone core not used for processing packets from interface queues (e.g., core <NUM> does not execute a packet processing application). In another embodiment, core <NUM> is one of the high frequency cores <NUM>. In an embodiment, monitor <NUM> is implemented in software executing on core <NUM>. In another embodiment, monitor <NUM> is implemented in circuitry in core <NUM>. Monitor <NUM> initializes queue bitmasks <NUM>, <NUM>,. <NUM> for cores <NUM>, <NUM>,. <NUM>, respectively. Monitor <NUM> monitors data packet transfer rates at the interface queues (either input to or output from interface queues) and adjusts queue bitmasks <NUM>, <NUM>,. <NUM> based at least in part on the monitored data packet transfer rates of the interface queues. Generally, when a data packet transfer rate of an interface queue indicates lower traffic, then the interface queue is dynamically reassigned from a HF core to a LF core, and when a data packet transfer rate of an interface queue indicates higher traffic, then the interface queue is dynamically reassigned from a LF core to a HF core. Thus, system <NUM> dynamically adjusts during runtime which cores are handling which interface queues based on data packet transfer rates of the interface queues. In an embodiment, the granularity of control over system power consumption is therefore at the interface queue level. When LF cores handle data packet transfers over more interface queues, system power consumption is lower (since LF cores use less power than HF cores). When HF cores handle data transfers over more interface queues, system power consumption is higher (since HF cores use more power than LF cores).

In an embodiment, core <NUM> of system <NUM> includes a capability to monitor the data packet transfer rates at core <NUM><NUM>, core <NUM><NUM>,. core M <NUM>. When the data packet transfer rate at a core is less than or equal to a predetermined threshold, the frequency of the core is reduced (e.g., the core is moved from a high frequency to a low frequency). When the data packet transfer rate at a core is more than the predetermined threshold, the frequency of the core is increased (e.g., the core is moved from a low frequency to a high frequency). Thus, system <NUM> dynamically adjusts the frequencies of cores based at least in part on data packet transfer rates at the cores. In an embodiment, when a core is moved from a high frequency to a low frequency, or when a core is moved from a low frequency to a high frequency, monitor <NUM> reassesses the mapping of interface queues to cores and amends queue bitmasks <NUM>, <NUM>,. <NUM> accordingly.

In one embodiment, system <NUM> changes the frequency of a processor by using a DPDK Power Management library feature. In one embodiment, using a Linux® kernel, a cpufreq module for processor frequency scaling for each core may be used.

<FIG> illustrates an example flow diagram of dynamic assignment of interface queues. In one embodiment the steps of <FIG> are performed by monitor <NUM> on core <NUM>. As part of initialization of one or more of packet processing applications PP0 <NUM>, PP <NUM><NUM>,. PP M <NUM> executing on core <NUM><NUM>, core <NUM><NUM>,. core M <NUM>, respectively, HF and LF cores in system <NUM> are detected at block <NUM>. In an embodiment, monitor <NUM> stores the initial frequency settings and associated core classes and updates this information during runtime as needed (e.g., when a core moves from a high frequency setting to a low frequency setting and vice versa). In an embodiment, all cores are HF cores at the time of system initialization (i.e., high frequency is the default setting for each core). At block <NUM>, processing threads of packet processing applications are assigned to cores and cores are assigned to interface queues. In an embodiment, one thread per core is assigned. At block <NUM>, queue bitmasks <NUM>, <NUM>,. <NUM> for all cores are initialized to reflect the detected HF and LF cores and initial interface queue assignments. During runtime of packet processing applications, monitor <NUM> monitors data packet transfer rates at interface queues at block <NUM>. How often the monitoring occurs is implementation dependent.

At block <NUM>, in one embodiment, when comparing a data transfer rate to a predetermined threshold, if the data packet transfer rate at an interface queue is less than or equal to the predetermined threshold, then processing continues with block <NUM>. At block <NUM>, the bit in the queue bitmask of the HF core associated with the interface queue is disabled (i.e., cleared), and the bit in the queue bitmask of a LF core to now be associated with the interface queue is enabled (i.e., set). This reassigns the interface queue from a HF core to a LF core for subsequent processing of data packets. If the interface queue was already assigned to a LF core (e.g., the value of the bit in the queue bitmask is already the value the bit would be set to in block <NUM>, then the bit setting operation of block <NUM> is omitted.

At block <NUM>, in one embodiment, when comparing a data transfer rate to a predetermined threshold, if a data packet transfer rate at an interface queue is greater than a predetermined threshold, then processing continues with block <NUM>. At block <NUM>, the bit in the queue bitmask of the LF core associated with the interface queue is disabled (i.e., cleared), and the bit in queue bitmask of a HF core associated with the interface queue is enabled (i.e., set). This moves the interface queue from a LF core to a HF core for subsequent processing of data packets. If the interface queue was already assigned to a HF core (e.g., the value of the bit in the queue bitmask is already the value the bit would be set to in block <NUM>, then the bit setting operation of block <NUM> is omitted. This scenario occurs when no change to the assignment of the interface queue to a core is needed.

Processing in either block <NUM> or block <NUM> continues with further monitoring at block <NUM>. In an embodiment, blocks <NUM>-<NUM> are performed for all interface queues.

Thus, queue bitmasks <NUM>, <NUM>,. <NUM> are changed depending on the monitored data packet transfer rates at the interface queues. When the queue bitmasks are changed, this results in corresponding changes in assignments of interfaces queues to cores. Changing the interface queues assignments to cores results in different power consumption amounts of the system because different cores are now being used to process data packets from interface queues. This enables efficient processor core utilization according to traffic rates by dynamically assigning interface queues to cores based at least in part on the core frequencies. This approach is scalable for multicore virtual switches running with any number of high bandwidth physical/virtual interfaces. This approach can be used in conjunction with processor power management schemes by assigning different frequencies to different cores for dynamic interface queues assignments.

In an embodiment, whenever the frequency of a core is changed (either from a high frequency to a low frequency, or from a low frequency to a high frequency), processing may be returned to block <NUM> detect the changed frequency characteristics of the cores in system <NUM>.

Let's assume that C is a set of m processor cores that the application is running on.

CH is a subset of C where all the cores in CH are running at high frequency. CL is a subset of C where all the cores in CL are running in low frequency. CH and CL are mutually exclusive. For simplicity let us assume that CH and CL are not empty. There are separate sets of processes PH and PL which run on each of CH and CL respectively.

Let's assume that Q is a set of k interface queues on which the process PH and PL of packet processing applications are polling for data.

QH is a subset of Q where all the interface queues are associated with high frequency cores and are polled by PH. QL is a subset of Q where all the interface queues are associated with low frequency cores and are polled by PL. QH and QL are mutually exclusive.

Let's assume that B is a bitmask of k bits which represents k interface queues as below.

The bit bi=<NUM> represents an interface queue as enabled in that bitmask where i = [<NUM>, k - <NUM>]. There is a queue bitmask for each core to be used by processes PH and PL. The queue bitmasks contain one bit for each interface queue that the processes poll for data packet traffic.

A monitoring process (e.g., monitor <NUM>) running on a core (preferably on a high frequency core) is responsible to get the statistic counters for each of the interface queues to indicate the data packet transfer rate. Let's assume that Si is the statistic counter for each interface queue where i = [<NUM>, k - <NUM>]. Let's say there is a traffic threshold Ti which is used to determine interface queue i managed by a particular process.

<FIG> illustrates an example time series representation of dynamic assignment of interface queues based on traffic rate. When a data packet transfer rate Si for an interface queue goes below an interface queue threshold T at point <NUM>, then block <NUM> of <FIG> is performed to disable a bit for this interface queue in the queue bitmask of the HF core assigned to this interface queue and enable a bit for this interface queue in the queue bitmask of a LF core. This causes the system to adjust to the lower data packet transfer rate by reassigning the interface queue from a HF core to a LF core. When a data packet transfer rate Si for an interface queue goes above an interface queue threshold T at point <NUM>, then block <NUM> of <FIG> is performed to disable a bit for this interface queue in the queue bitmask of the LF core assigned to this interface queue and enable a bit for this interface queue in the queue bitmask of a HF core. This causes the system to adjust to the higher data packet transfer rate by reassigning the interface queue from a LF core to a HF core.

<FIG> illustrates an example diagram of a virtual switch. In an embodiment, virtual switch <NUM> is an application that when executed on one or more cores of a computing system processes data packets. Switch <NUM> is coupled to one or more physical NIC devices <NUM>, and a plurality of virtual machine (VMs) such as VM <NUM><NUM>, VM <NUM><NUM>,. VM J <NUM>, where J is a natural number, to receive data packets. In an embodiment, VMs are replaced with containers. As described above, the data packets are stored in interface queues. Switch <NUM> includes a queue bitmask for each core in the computing system that is executing the switch. In this example, switch <NUM> includes core <NUM> queue bitmask <NUM>, core <NUM> queue bitmask <NUM>,. core M queue bitmask <NUM> (assuming M cores in the computing system). Switch <NUM> includes monitor <NUM> monitor data packet transfer rates at the interface queues and to initialize and update the queue bitmasks. Switch <NUM> also includes packet processing threads for high frequency cores (PH) <NUM> and packet processing threads for low frequency cores (PL).

Below is an example of pseudocode for initialization of the queue bitmasks by monitor <NUM>.

In an embodiment, the data packet transfer rate of an interface queue is used, but in other embodiments, there could be other metrics used to determine the processor core load. In one embodiment, a flow lookup rate by the OVS application for the traffic received from a particular interface queue may be used.

Below is an example of pseudocode for using the queue bitmasks when processing packets by a packet processing thread (e.g., PH and PL) running on a core (such as core <NUM><NUM>, core <NUM><NUM>,. core M <NUM>).

<FIG> illustrates an example computing system <NUM>. As shown in <FIG>, computing system <NUM> includes a computing platform <NUM> coupled to a network <NUM> (which may be the Internet, for example). In some examples, as shown in <FIG>, computing platform <NUM> is coupled to network <NUM> via network communication channel <NUM> and through at least one network I/O device <NUM> (e.g., a network interface controller (NIC) such as NIC <NUM>) having one or more ports connected or coupled to network communication channel <NUM>. In an embodiment, network communication channel <NUM> includes a PHY device (not shown). In an embodiment, network I/O device <NUM> is an Ethernet NIC. Network I/O device <NUM> transmits data packets from computing platform <NUM> over network <NUM> to other destinations and receives data packets from other destinations for forwarding to computing platform <NUM>.

According to some examples, computing platform <NUM>, as shown in <FIG>, includes circuitry <NUM>, primary memory <NUM>, network (NW) I/O device driver <NUM>, operating system (OS) <NUM>, at least one application <NUM>, and one or more storage devices <NUM>. In one embodiment, OS <NUM> is Linux®. In another embodiment, OS <NUM> is Windows® Server. Network I/O device driver <NUM> operates to initialize and manage I/O requests performed by network I/O device <NUM>. In an embodiment, data packets and/or packet metadata transmitted to network I/O device <NUM> and/or received from network I/O device <NUM> are stored in one or more of primary memory <NUM> and/or storage devices <NUM>. In an embodiment, interface queues are stored in one or more of primary memory <NUM> and/or storage devices <NUM>. In at least one embodiment, application <NUM> is a packet processing (PP) application such as virtual switch <NUM> of <FIG>. Application <NUM> includes queue polling components and runs on one or more cores. Application <NUM> may also be run in or in conjunction with one or more VMs. In at least one embodiment, storage devices <NUM> may be one or more of hard disk drives (HDDs) and/or solid-state drives (SSDs). In an embodiment, storage devices <NUM> may be non-volatile memories (NVMs). In some examples, as shown in <FIG>, circuitry <NUM> may communicatively couple to network I/O device <NUM> via communications link <NUM>. In one embodiment, communications link <NUM> is a peripheral component interface express (PCIe) bus conforming to version <NUM> or other versions of the PCIe standard published by the PCI Special Interest Group (PCI-SIG). In some examples, operating system <NUM>, NW I/O device driver <NUM>, and application <NUM> are implemented, at least in part, via cooperation between one or more memory devices included in primary memory <NUM> (e.g., volatile or non-volatile memory devices), storage devices <NUM>, and elements of circuitry <NUM> such as processing cores <NUM>-<NUM> to <NUM>-m, where "m" is any positive whole integer greater than <NUM>. In an embodiment, OS <NUM>, NW I/O device driver <NUM>, and application <NUM> are executed by one or more processing cores <NUM>-<NUM> to <NUM>-m. In an embodiment, shared memory <NUM> of <FIG> is implemented by one or more of primary memory <NUM> and/or storage devices <NUM>.

In some examples, computing platform <NUM>, includes but is not limited to a server, a server array or server farm, a web server, a network server, an Internet server, a work station, a mini-computer, a main frame computer, a supercomputer, a network appliance, a web appliance, a distributed computing system, multiprocessor systems, processor-based systems, a laptop computer, a tablet computer, a smartphone, or a combination thereof. In one example, computing platform <NUM> is a disaggregated server. A disaggregated server is a server that breaks up components and resources into subsystems. Disaggregated servers can be adapted to changing storage or compute loads as needed without replacing or disrupting an entire server for an extended period of time. A server could, for example, be broken into modular compute, I/O, power and storage modules that can be shared among other nearby servers.

Circuitry <NUM> having processing cores <NUM>-<NUM> to <NUM>-m may include various commercially available processors, including without limitation Intel® Atom®, Celeron®, Core (<NUM>) Duo®, Core i3, Core i5, Core i7, Itanium®, Pentium®, Xeon® or Xeon Phi® processors, ARM processors, and similar processors. Circuitry <NUM> may include at least one cache <NUM> to store data.

According to some examples, primary memory <NUM> may be composed of one or more memory devices or dies which may include various types of volatile and/or non-volatile memory. Volatile types of memory may include, but are not limited to, dynamic random-access memory (DRAM), static random-access memory (SRAM), thyristor RAM (TRAM) or zero-capacitor RAM (ZRAM). Non-volatile types of memory may include byte or block addressable types of non-volatile memory having a <NUM>-dimensional (<NUM>-D) cross-point memory structure that includes chalcogenide phase change material (e.g., chalcogenide glass) hereinafter referred to as "<NUM>-D cross-point memory". Non-volatile types of memory may also include other types of byte or block addressable non-volatile memory such as, but not limited to, multi-threshold level NAND flash memory, NOR flash memory, single or multi-level phase change memory (PCM), resistive memory, nanowire memory, ferroelectric transistor random access memory (FeTRAM), magneto-resistive random-access memory (MRAM) that incorporates memristor technology, spin transfer torque MRAM (STT-MRAM), or a combination of any of the above. In another embodiment, primary memory <NUM> may include one or more hard disk drives within and/or accessible by computing platform <NUM>.

In an embodiment, data packets are transmitted to and/or received from network I/O device <NUM>, and packets transferred between cores.

<FIG> illustrates an example of a storage medium <NUM>. Storage medium <NUM> may comprise an article of manufacture. In some examples, storage medium <NUM> may include any non-transitory computer readable medium or machine readable medium, such as an optical, magnetic or semiconductor storage. Storage medium <NUM> may store various types of computer executable instructions, such as instructions <NUM> to implement logic flow <NUM> of <FIG> (e.g., monitor <NUM>), example pseudocodes, packet processing applications <NUM>, <NUM>,. <NUM> of <FIG>, and virtual switch <NUM>. Examples of a computer readable or machine-readable storage medium may include any tangible media capable of storing electronic data, including volatile memory or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writeable or re-writeable memory, and so forth. Examples of computer executable instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, and the like. The examples are not limited in this context.

<FIG> illustrates an example computing platform <NUM>. In some examples, as shown in <FIG>, computing platform <NUM> may include a processing component <NUM>, other platform components <NUM> and/or a communications interface <NUM>.

According to some examples, processing component <NUM> may execute processing operations or logic for instructions stored on storage medium <NUM>. Processing component <NUM> may include various hardware elements, software elements, or a combination of both. Examples of hardware elements may include devices, logic devices, components, processors, microprocessors, circuits, processor circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, and so forth), integrated circuits, application specific integrated circuits (ASIC), programmable logic devices (PLD), digital signal processors (DSP), field programmable gate array (FPGA), memory units, logic gates, registers, semiconductor device, chips, microchips, chip sets, and so forth. Examples of software elements may include software components, programs, applications, computer programs, application programs, device drivers, system programs, software development programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces (API), instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. Determining whether an example is implemented using hardware elements and/or software elements may vary in accordance with any number of factors, such as desired computational rate, power levels, heat tolerances, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds and other design or performance constraints, as desired for a given example.

In some examples, other platform components <NUM> may include common computing elements, such as one or more processors, multi-core processors, co-processors, memory units, chipsets, controllers, peripherals, interfaces, oscillators, timing devices, video cards, audio cards, multimedia input/output (I/O) components (e.g., digital displays), power supplies, and so forth. Examples of memory units may include without limitation various types of computer readable and machine readable storage media in the form of one or more higher speed memory units, such as read-only memory (ROM), random-access memory (RAM), dynamic RAM (DRAM), Double-Data-Rate DRAM (DDRAM), synchronous DRAM (SDRAM), static RAM (SRAM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), types of non-volatile memory such as <NUM>-D cross-point memory that may be byte or block addressable. Non-volatile types of memory may also include other types of byte or block addressable non-volatile memory such as, but not limited to, multi-threshold level NAND flash memory, NOR flash memory, single or multi-level PCM, resistive memory, nanowire memory, FeTRAM, MRAM that incorporates memristor technology, STT-MRAM, or a combination of any of the above. Other types of computer readable and machine-readable storage media may also include magnetic or optical cards, an array of devices such as Redundant Array of Independent Disks (RAID) drives, solid state memory devices (e.g., USB memory), solid state drives (SSD) and any other type of storage media suitable for storing information.

In some examples, communications interface <NUM> may include logic and/or features to support a communication interface. For these examples, communications interface <NUM> may include one or more communication interfaces that operate according to various communication protocols or standards to communicate over direct or network communication links or channels. Direct communications may occur via use of communication protocols or standards described in one or more industry standards (including progenies and variants) such as those associated with the PCIe specification. Network communications may occur via use of communication protocols or standards such those described in one or more Ethernet standards promulgated by IEEE. For example, one such Ethernet standard may include IEEE <NUM>. Network communication may also occur according to one or more OpenFlow specifications such as the OpenFlow Switch Specification.

The components and features of computing platform <NUM>, including logic represented by the instructions stored on storage medium <NUM> may be implemented using any combination of discrete circuitry, ASICs, logic gates and/or single chip architectures. Further, the features of computing platform <NUM> may be implemented using microcontrollers, programmable logic arrays and/or microprocessors or any combination of the foregoing where suitably appropriate. It is noted that hardware, firmware and/or software elements may be collectively or individually referred to herein as "logic" or "circuit.

It should be appreciated that the exemplary computing platform <NUM> shown in the block diagram of <FIG> may represent one functionally descriptive example of many potential implementations. Accordingly, division, omission or inclusion of block functions depicted in the accompanying figures does not infer that the hardware components, circuits, software and/or elements for implementing these functions would necessarily be divided, omitted, or included in embodiments.

Various examples may be implemented using hardware elements, software elements, or a combination of both. In some examples, hardware elements may include devices, components, processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, and so forth), integrated circuits, ASIC, programmable logic devices (PLD), digital signal processors (DSP), FPGA, memory units, logic gates, registers, semiconductor device, chips, microchips, chip sets, and so forth. In some examples, software elements may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces (API), instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. Determining whether an example is implemented using hardware elements and/or software elements may vary in accordance with any number of factors, such as desired computational rate, power levels, heat tolerances, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds and other design or performance constraints, as desired for a given implementation.

Some examples may include an article of manufacture or at least one computer-readable medium. A computer-readable medium may include a non-transitory storage medium to store logic. In some examples, the non-transitory storage medium may include one or more types of computer-readable storage media capable of storing electronic data, including volatile memory or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writeable or re-writeable memory, and so forth. In some examples, the logic may include various software elements, such as software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, API, instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof.

Some examples may be described using the expression "in one example" or "an example" along with their derivatives. These terms mean that a particular feature, structure, or characteristic described in connection with the example is included in at least one example. The appearances of the phrase "in one example" in various places in the specification are not necessarily all referring to the same example.

Included herein are logic flows or schemes representative of example methodologies for performing novel aspects of the disclosed architecture. While, for purposes of simplicity of explanation, the one or more methodologies shown herein are shown and described as a series of acts, those skilled in the art will understand and appreciate that the methodologies are not limited by the order of acts. Some acts may, in accordance therewith, occur in a different order and/or concurrently with other acts from that shown and described herein. For example, those skilled in the art will understand and appreciate that a methodology could alternatively be represented as a series of interrelated states or events, such as in a state diagram. Moreover, not all acts illustrated in a methodology may be required for a novel implementation.

A logic flow or scheme may be implemented in software, firmware, and/or hardware. In software and firmware embodiments, a logic flow or scheme may be implemented by computer executable instructions stored on at least one non-transitory computer readable medium or machine readable medium, such as an optical, magnetic or semiconductor storage.

Some examples are described using the expression "coupled" and "connected" along with their derivatives. For example, descriptions using the terms "connected" and/or "coupled" may indicate that two or more elements are in direct physical or electrical contact with each other.

It is emphasized that the Abstract of the Disclosure is provided to comply with <NUM> C. Section <NUM>(b), requiring an abstract that will allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in a single example for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed examples require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed example. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate example. In the appended claims, the terms "including" and "in which" are used as the plain-English equivalents of the respective terms "comprising" and "wherein," respectively. Moreover, the terms "first," "second," "third," and so forth, are used merely as labels, and are not intended to impose numerical requirements on their objects.

Claim 1:
A method of operating a system (<NUM>) comprising a plurality of interface queues (<NUM>, <NUM>, ... <NUM>) and a plurality of cores (<NUM>, <NUM>, <NUM>), wherein the cores are divided into two classes of cores of a high frequency and cores of a low frequency, said method comprising:
assigning (<NUM>) a packet processing thread to each core (<NUM>, <NUM>, <NUM>) and each core (<NUM>, <NUM>, <NUM>) to one or more interface queues (<NUM>, <NUM>, ... <NUM>);
initializing (<NUM>) a bitmask (<NUM>, <NUM>, <NUM>) for each core (<NUM>, <NUM>, <NUM>), each bitmask comprising a number of bits equal to a number of the interface queues (<NUM>, <NUM>, ... <NUM>), a bit in the bitmask being associated with an interface queue;
for each of the interface queues (<NUM>, <NUM>, ... <NUM>), monitoring (<NUM>) either an input or an output data packet transfer rate at the interface queue (<NUM>, <NUM>, ... <NUM>);
for each of the interface queues (<NUM>, <NUM>, ... <NUM>), based at least in part on a comparison of the data packet transfer rate at the interface queue (<NUM>, <NUM>, ... <NUM>) to a threshold, assigning (<NUM>) the interface queue (<NUM>, <NUM>, ... <NUM>) from a core (<NUM>, <NUM>) of high frequency to a core (<NUM>) of low frequency, or assigning (<NUM>) the interface queue (<NUM>, <NUM>, ... <NUM>) from a core (<NUM>) of low frequency to a core of high frequency,
wherein assigning the interface queue (<NUM>, <NUM>, ... <NUM>) from the core (<NUM>, <NUM>) of high frequency to the core (<NUM>) of low frequency comprises disabling the bit associated with the interface queue (<NUM>, <NUM>, ... <NUM>) in the bitmask of the core (<NUM>, <NUM>) of high frequency and enabling the bit associated with the interface queue (<NUM>, <NUM>, ... <NUM>) in the bitmask of the core (<NUM>) of low frequency, or assigning the interface queue (<NUM>, <NUM>, ... <NUM>) from the core (<NUM>) of low frequency to the core (<NUM>, <NUM>) of high frequency comprises disabling the bit associated with the interface queue (<NUM>, <NUM>, ... <NUM>) in the bitmask of the core (<NUM>) of low frequency and enabling the bit associated with the interface queue (<NUM>, <NUM>, ... <NUM>) in the bitmask of the core (<NUM>, <NUM>) of high frequency.