Patent Description:
High performance computing and other demanding scale out applications in the datacenter continue to require higher port counts, larger bandwidth, and reduced latency in packet processing devices such as network switches and routers. Power consumption improvements from process node geometry reductions are fast approaching hard physical limitations. Thus, sole reliance on process node improvements may be insufficient to keep up with increasing performance demands for packet processing devices.

"Collaborative fuzzy-based partially-throttling dynamic thermal management scheme for three-dimensional networks-on-chip" (XP006095399) relates to methods for dynamic thermal management of 3D networks-on-chip devices.

<CIT> relates to energy efficiency and energy conservation methods for use in integrated circuits according to maximum performance metric and power budget.

According to the present invention, a system according to claim <NUM> is provided.

Advantageously, the target clock rate is determined based on a surplus power budget that is credited while the target clock rate exceeds the adjusted power gated clock rate.

Advantageously, the surplus power budget is reset to zero at a start of the thermal average period.

Advantageously, the surplus power budget is set to an initial budget value at a start of the thermal average period.

Advantageously, the energy usage estimate corresponds to the packet processing block processing a determined number of packet cells from the ingress packets.

Advantageously, each of the plurality of measurement periods comprise a substantially equal time period.

Advantageously, the adjusting comprises stepwise adjustments of the power gated clock rate using a plurality of adjustment periods after each of the plurality of measurement periods.

Advantageously, the plurality of adjustment periods comprises a combined time period substantially equal to one of the plurality of measurement periods.

Advantageously, each of the plurality of adjustment periods is a time period that is approximately <NUM> to <NUM> percent of one of the plurality of measurement periods.

Advantageously, the thermal average period is approximately <NUM> milliseconds to <NUM> seconds.

Advantageously, each of the plurality of measurement periods is a time period that is approximately <NUM> to <NUM> percent of the thermal average period.

Advantageously, the target clock rate is bounded by a predetermined range.

According to a second aspect of the present invention, a method according to claim <NUM> is provided.

According to an aspect not covered by the claims, a non-transitory storage medium comprises instructions that, when read by one or more processors, cause a method comprising:.

Various objects, features, and advantages of the present disclosure can be more fully appreciated with reference to the following detailed description when considered in connection with the following drawings, in which like reference numerals identify like elements. The following drawings are for the purpose of illustration only and are not intended to be limiting of this disclosure, the scope of which is set forth in the claims that follow.

While aspects of the subject technology are described herein with reference to illustrative examples for particular applications, it should be understood that the subject technology is not limited to those particular applications. Those skilled in the art with access to the teachings provided herein will recognize additional modifications, applications, and aspects within the scope thereof and additional fields in which the subject technology would be of significant utility.

To meet the increasing performance demands of modern datacenter and high performance computing applications, device architectural improvements are needed to maintain performance momentum in view of slowing process node improvements. In particular, packet processing devices such as switches may require improved power management to meet performance requirements within a target thermal design power (TDP), which may be bounded according to device form factor and available cooling capacity. Existing power management approaches may perform a fixed down-clocking such as a decoupled packet rate, which may negatively affect device performance. Further, these approaches may require devices to be designed with lower TDP bounds to accommodate a sustained peak packet rate that corresponds to approximately double the anticipated average packet rate, resulting in overdesigning of devices and inefficient use of device capacity.

Systems and methods are provided for power throttle for network switches, wherein the throttling is managed over thermal time periods based on meeting, for each thermal time period, a target energy consumption by periodically measuring estimated power consumption and adjusting throttle targets accordingly. Clock gating may be utilized to modularly reduce power consumption while avoiding the performance penalty of down clocking, thereby maintaining packet processing performance and enabling devices to be designed with TDP bounds more closely aligned to anticipated workloads. Further, by using a relatively long thermal time period, such as from hundreds of milliseconds to several seconds, burst traffic can be absorbed for longer time periods before throttling occurs. Various policies for conservative or aggressive ramping of clock gating may be chosen based on use case requirements and application preferences. The automatic throttle control may also be bypassed by manually overriding the throttle targets.

Power throttle for network switches may be especially relevant for high performance computing (HPC) applications, which may generate regular spikes of high burst network traffic, such as large numbers of small sized control packets. As discussed above, the use of relatively long thermal time periods for throttle control allows burst traffic to be absorbed for a longer period of time before throttling or ramping down of clock gating rate is required. By tuning the thermal time period and throttling policies, optimized power management may be provided for packet switching devices while minimizing performance penalties.

<FIG> depicts an example network environment <NUM> in which power throttle for network switches may be implemented, according to various aspects of the subject technology. Not all of the depicted components may be used in all implementations, however, and one or more implementations may include additional or different components than those shown in the figure. Variations in the arrangement and type of the components may be made without departing from the scope of the claims as set forth herein. Additional components, different components, or fewer components may be provided.

The network environment <NUM> includes one or more electronic devices 102A-C connected via a network switch <NUM>. The electronic devices 102A-C may be connected to the network switch <NUM>, such that the electronic devices 102A-C may be able to communicate with each other via the network switch <NUM>. The electronic devices 102A-C may be connected to the network switch <NUM> via wire (e.g., Ethernet cable) or wirelessly. The network switch <NUM>, may be, and/or may include all or part of, the network switch discussed below with respect to the ingress packet processing <NUM> of <FIG> and/or the electronic system discussed below with respect to <FIG>. The electronic devices 102A-C are presented as examples, and in other implementations, other devices may be substituted for one or more of the electronic devices 102A-C.

For example, the electronic devices 102A-C may be computing devices such as laptop computers, desktop computers, servers, peripheral devices (e.g., printers, digital cameras), mobile devices (e.g., mobile phone, tablet), stationary devices (e.g. set-top-boxes), or other appropriate devices capable of communication via a network. In <FIG>, by way of example, the electronic devices 102A-C are depicted as network servers. The electronic devices 102A-C may also be network devices, such as other network switches, and the like.

The network switch <NUM> may implement power throttle as described herein. For example, <FIG> depicts a logical block diagram of power throttle <NUM> within an example network switch <NUM>, according to various aspects of the subject technology. As shown in <FIG>, power throttle <NUM> asserts an enable signal which is sent to packet processing <NUM>, which processes incoming ingress packets. For example, the enable signal may be asserted M times for every N edges of the clock signal, wherein power throttle <NUM> may adjust M upwards for higher performance or downwards for throttling. The value of N may be kept to a constant value according to required clock resolution, and the enable signal may be asserted to provide a smooth distribution over time. For example, if M is set to <NUM> and N is <NUM>, then the enable signal (M) may be asserted with every other edge of the clock signal (N). The enable signal allows the clock signal to be power gated at packet processing <NUM> to reduce power consumption while maintaining performance. Further, the power gated clock is also provided back to power throttle <NUM> for power estimation purposes, which allows power throttle <NUM> to adjust and throttle the enable signal (M) in a feedback loop.

<FIG> depicts a logical block diagram of a packet processing block <NUM> within an example network switch, according to various aspects of the subject technology. Power gating block <NUM> receives a clock signal and an enable signal to output a power gated clock, as described above in <FIG>. Since power gating block <NUM> may have a low leakage power, there is a potential for large power savings compared to a fixed down-clocking approach. Ingress scheduler (IS) <NUM> receives ingress packets, which are then scheduled for processing through ingress processing (IP) <NUM>. For example, the ingress packets may correspond to network packets. Packets may be divided into fixed sized packet cells and queued into IP <NUM> and/or cell buffer <NUM>. For example, if an ingress packet is <NUM> bytes and the cells are <NUM> bytes, then the ingress packet may be divided into <NUM> cells. IP <NUM> may be configured to process up to a defined bandwidth of packet cells, such as <NUM> billion packets worth of cells per second, and cell buffer <NUM> may hold cells to be processed that exceed the available bandwidth of IP <NUM>. After processing through IP <NUM>, the ingress packets may proceed to memory management unit (MMU) <NUM> for further processing, for example by proceeding through an egress scheduler for outbound arbitration.

<FIG> depicts a logical block diagram of a power throttle block, or power throttle <NUM>, within an example network switch, according to various aspects of the subject technology. Throttle control <NUM> is able to count the number or rate of incoming ingress packets, and thus cells, for throttling. Further, throttle control <NUM> may receive ingress scheduler (IS) metadata, for example buffer fill rates. Based on an energy usage estimate from energy estimator <NUM>, throttle control <NUM> can determine a target clock rate according to configurable throttle policies to meet a target average energy value, which may correspond to a target TDP that is configured for the example network switch. The target clock rate is provided to step control <NUM>, which gradually adjusts the clock rate stepwise towards the target clock rate in adjustment periods after each measurement period. Step control <NUM> may send the adjustments to clock regulator <NUM>, which in turn asserts the enable signal (M) at an adjusted rate for a defined number of clock edges (N) at power gating block <NUM>. The resulting power gated clock is then fed back to energy estimator <NUM>, which uses defined energy consumption estimates to calculate the estimated energy usage that is provided to throttle control <NUM>. For example, the "energy per clock - with packet" may correspond to the estimated energy used when the ingress packets or cells are being processed (and thus enable is asserted), and the "energy per clock - without packet" may correspond to the estimated energy used when the ingress packets or cells are not being processed (and thus enable is not asserted). In some implementations, these values may be calibrated at manufacture time, and in other implementations, these values may be calibrated at run time by actual power consumption measurements, for example by measuring power consumption at various corners of the package die that implements IP <NUM>.

As discussed above, throttle control <NUM> can determine a target clock rate to meet a target average energy value, which may correspond to a target TDP that is configured for the example network switch. This target average energy value may be averaged for a thermal average period, which may be calibrated to be a relatively long period, such as <NUM> milliseconds to <NUM> seconds. This may provide a buffer for burst traffic to be absorbed during each thermal average period to reduce throttling. When the thermal average period elapses, throttle control <NUM> may assert a reset signal to energy estimator <NUM>, which in turn may perform a reset or initialization of surplus power budgets and other values according to configurable throttle policies for each thermal average period.

In some implementations, the power throttle <NUM> may be bypassed by manual override of the target clock rate. For example, a host or a controller may manually provide the target clock rate to step control <NUM>, thereby overriding throttle control <NUM>.

<FIG> depicts a logical block diagram of a power throttle block, or power throttle <NUM>, and a packet processing block <NUM> within an example network switch, according to various aspects of the subject technology. As shown, <FIG> corresponds to a combination of <FIG> and <FIG>, which is also depicted in simplified form in <FIG>.

<FIG> depicts an example graph of power throttling when using a zero budget policy with a constant high load, according to various aspects of the subject technology. The example graph may depict a single thermal average period. As shown, the power gated clock rate or "ThrottleFactor" may be adjustable within a defined range, or a ceiling correspond to "ConfiguredHiClock" and a floor corresponding to "ConfiguredLoClock". As discussed above, throttle control <NUM> may be configurable with various throttle policies, some which are described in <FIG>. In the example shown in <FIG>, the throttle policy corresponds to a "Zero Budget Policy". This means that when energy estimator <NUM> receives a reset signal from throttle control <NUM> to indicate the start of a new thermal average period, a surplus power budget is reset to zero. Throttle control <NUM> may be configured not to adjust the "ThrottleFactor", or target clock rate, beyond the clock rate needed to achieve the target TDP for the thermal average period, or the "Target Energy", when there is no credit available from the surplus power budget. Since the packet processing workload or "OfferedLoad" is a constant high load, there is no opportunity to fill the surplus power budget, and therefore the clock rate is kept constant without exceeding the "Target Energy" value.

<FIG> depicts an example graph of power throttling when using a zero budget policy with a variable load, according to various aspects of the subject technology. As shown in <FIG>, the "OfferedLoad" now varies over time, which allows the surplus power budget to be credited when the target clock rate or "Throttle Factor" exceeds the actual adjusted power gated clock rate, which can be measured from the energy usage estimate received from energy estimator <NUM>. For example, as discussed above, the step control <NUM> may gradually adjust the clock rate towards the target clock rate, in which case the power gated clock rate may be less than it could be at that time. The difference between the measured and target clock rate is therefore credited to the surplus power budget. When a spike in the "OfferedLoad" workload is later encountered, the surplus power budget may be expended to raise the "Throttle Factor", as indicated by the upwards triangle portion of the "Throttle Factor". After the surplus power budget is expended, throttling may resume to meet the target energy average for the thermal average period, as indicated by the downwards triangle portion of the "Throttle Factor". Note that the "Throttle Factor" is bounded by the "ConfiguredLoClock" floor and does not throttle any further. Thus, the conservative zero budget policy helps to conserve power budget to address future high workloads.

<FIG> depicts an example graph of power throttling when using an aggressive ramping policy with a constant high load, according to various aspects of the subject technology. When the throttle policy is set to "Aggressive Ramp Policy", energy estimator <NUM> may set the surplus power budget to an initial non-zero value when a reset signal is received from throttle control <NUM> to indicate the start of a new thermal average period. Thus, as shown in <FIG>, the "Throttle Factor" may be immediately set to the ceiling, or "ConfiguredHiClock", and remain there until the surplus power budget is exhausted, at which point throttling is incurred and the "Throttle Factor" ramps down to "ConfiguredLoClock" for the remainder of the thermal average period.

<FIG> depicts an example graph of power throttling when using an aggressive ramping policy with a variable load, according to various aspects of the subject technology. As shown in <FIG>, since the aggressive ramping policy prioritizes the immediate usage of any surplus power budget, the budget may be quickly exhausted and the "ThrottleFactor" may be forced to throttle towards the end of the thermal average period, as shown. Thus, the aggressive ramping policy prioritizes burst performance but may quickly exhaust power budget for future workloads. Whether conservative or aggressive ramping is preferred may depend on application latency and jitter requirements.

In the present invention, a policy corresponds to a buffer fill rate policy. For example, if the buffer fill rate of cell buffer <NUM> exceeds a upwards threshold, as indicated by the IS metadata, then the "ThrottleFactor" may be set to the maximum possible, or "ConfiguredHiClock". Similarly, if the buffer fill rate of cell buffer <NUM> falls below a downwards threshold, then the "ThrottleFactor" may be set to the minimum possible, or "ConfiguredLoClock". In some implementations, multiple threshold levels may be provided so that clock rate targets can be chosen that are distributed between the "ConfiguredHiClock" and "ConfiguredLoClock". This buffer fill rate policy allows the power gated clock rate to quickly ramp according to incoming workload.

<FIG> depicts an example process <NUM> for providing power throttle for network switches, according to various aspects of the subject technology. For explanatory purposes, the various blocks of example process <NUM> are described herein with reference to <FIG>, and the components and/or processes described herein. The one or more of the blocks of process <NUM> may be implemented, for example, by a computing device, including a processor and other components utilized by the device. In some implementations, one or more of the blocks may be implemented apart from other blocks, and by one or more different processors or devices. Further for explanatory purposes, the blocks of example process <NUM> are described as occurring in serial, or linearly. However, multiple blocks of example process <NUM> may occur in parallel. In addition, the blocks of example process <NUM> need not be performed in the order shown and/or one or more of the blocks of example process <NUM> need not be performed.

In the depicted example flow diagram, for each of a plurality of measurement periods within a thermal average period, an energy usage estimate is determined for a packet processing block configured to process ingress packets at a power gated clock rate (<NUM>). Referring to <FIG>, this may correspond to energy estimator <NUM> determining an energy usage estimate for a plurality of measurement periods within a thermal average period, wherein the energy usage estimate is for IP <NUM> operating at a power gated clock rate provided by power gating block <NUM>. The measurement periods may each correspond to, for example, approximately <NUM> to <NUM> percent of the thermal average period. The measurement periods may be substantially equal time periods.

Process <NUM> may continue with determining, for each of the plurality of measurement periods, a target clock rate for the packet processing block based on the determined energy usage estimates to meet a target energy value that is averaged for the thermal average period (<NUM>). Referring to <FIG>, this may correspond to throttle control <NUM> determining, for each of the plurality of measurement periods, a target clock rate for IP <NUM> based on the energy usage estimates from <NUM> to meet a target TDP that is averaged for the thermal average period. For example, referring to <FIG>, the target TDP that is averaged for the thermal average period may correspond to the depicted "Target Energy" line, wherein <FIG> each depict a single thermal average period.

Process <NUM> may continue with adjusting, for each of the plurality of measurement periods, the power gated clock rate towards the target clock rate, wherein the adjusting causes the packet processing block to process the ingress packets at the adjusted power gated clock rate (<NUM>). Referring to <FIG>, this may correspond to step control <NUM> adjusting, for each of the plurality of measurement periods, the power gated clock rate towards the target clock rate from <NUM>. In some implementations, this may be done stepwise within a plurality of adjustment periods after each measurement period. An adjustment period may correspond to a time length that is <NUM> to <NUM> percent of a measurement period. The plurality of adjustment periods may correspond to a combined time length of a single measurement period. As discussed above, the clock regulator <NUM> may implement the power gated clock rate by asserting M enable signals out of every N clock edges. As shown in <FIG>, this causes the power gated clock to be output to the input of IP <NUM>, which in turn causes IP <NUM> to process the ingress packets at the adjusted power gated clock rate. The steps of process <NUM> may be repeated for successive thermal average periods.

Many aspects of the above-described example process <NUM>, and related features and applications, may also be implemented as software processes that are specified as a set of instructions recorded on a computer readable storage medium (also referred to as computer readable medium), and may be executed automatically (e.g., without user intervention). When these instructions are executed by one or more processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. The computer readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections.

The term "software" is meant to include, where appropriate, firmware residing in read-only memory or applications stored in magnetic storage, which can be read into memory for processing by a processor. Also, in some implementations, multiple software aspects of the subject disclosure can be implemented as sub-parts of a larger program while remaining distinct software aspects of the subject disclosure. In some implementations, multiple software aspects can also be implemented as separate programs. Finally, any combination of separate programs that together implement a software aspect described here is within the scope of the subject disclosure. In some implementations, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs.

<FIG> illustrates an electronic system <NUM> with which one or more implementations of the subject technology may be implemented. The electronic system <NUM> can be, and/or can be a part of, the network switch <NUM> shown in <FIG>. The electronic system <NUM> may include various types of computer readable media and interfaces for various other types of computer readable media. The electronic system <NUM> includes a bus <NUM>, one or more processing unit(s) <NUM>, a system memory <NUM> (and/or buffer), a ROM <NUM>, a permanent storage device <NUM>, an input device interface <NUM>, an output device interface <NUM>, and one or more network interfaces <NUM>, or subsets and variations thereof.

Finally, as shown in <FIG>, the bus <NUM> also couples the electronic system <NUM> to one or more networks and/or to one or more network nodes, through the one or more network interface(s) <NUM>. In this manner, the electronic system <NUM> can be a part of a network of computers (such as a LAN, a wide area network ("WAN"), or an Intranet, or a network of networks, such as the Internet. Any or all components of the electronic system <NUM> can be used in conjunction with the subject disclosure.

It is understood that any specific order or hierarchy of blocks in the processes disclosed is an illustration of example approaches. Based upon design preferences, it is understood that the specific order or hierarchy of blocks in the processes may be rearranged, or that all illustrated blocks be performed. Any of the blocks may be performed simultaneously. In one or more implementations, multitasking and parallel processing may be advantageous.

The word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment described herein as "exemplary" or as an "example" is not necessarily to be construed as preferred or advantageous over other embodiments. Furthermore, to the extent that the term "include", "have", or the like is used in the description or the claims, such term is intended to be inclusive in a manner similar to the term "comprise" as "comprise" is interpreted when employed as a transitional word in a claim.

Claim 1:
A system configured to:
determine (<NUM>) an energy usage estimate for a packet processing block (<NUM>) for each of a plurality of measurement periods within a thermal average period,
determine (<NUM>), for each of the plurality of measurement periods, a target clock rate for a packet processing block (<NUM>) based on the determined energy usage estimate to meet a target energy value that is averaged for the thermal average period; and
adjust (<NUM>), for each of the plurality of measurement periods, a clock rate towards the target clock rate to cause the packet processing block (<NUM>) to process ingress packets at the adjusted clock rate;
characterized in that
the system is configured to determine the target clock rate based on whether a buffer fill rate for the ingress packets exceeds a threshold fill rate.