Patent Description:
Demands by individuals, researchers, and enterprises for increased compute performance and storage capacity of computing devices have resulted in various computing technologies developed to address those demands. For example, compute intensive applications, such as enterprise cloud-based applications (e.g., software as a service (SaaS) applications), data mining applications, data-driven modeling applications, scientific computation problem solving applications, etc., typically rely on complex, large-scale computing environments (e.g., high-performance computing (HPC) environments, cloud computing environments, etc.) to execute the compute intensive applications, as well as store voluminous amounts of data. Such large-scale computing environments can include tens of hundreds (e.g., enterprise systems) to tens of thousands (e.g., HPC systems) of multi-processor/multi-core network nodes connected via high-speed interconnects (e.g., fabric interconnects in a unified fabric).

To carry out such processor intensive computations, various computing technologies have been implemented to distribute workloads across different network computing devices, such as parallel computing, distributed computing, etc. In support of such distributed workload operations, multiprocessor hardware architecture (e.g., multiple multi-core processors that share memory) has been developed to facilitate multiprocessing (i.e., coordinated, simultaneous processing by more than one processor) across local and remote shared memory systems using various parallel computer memory design architectures, such as non-uniform memory access (NUMA), and other distributed memory architectures.

Accordingly, memory requests from multiple interconnected network nodes can occupy the same shared buffer (e.g., super queues, table of requests, etc.) as local memory requests of a particular network node. However, such shared buffers are limited in size (e.g., containing tens of entries), which can result in other memory requests being queued until data returns from the memory subsystems for those memory requests presently in the shared buffer. As such, entries of the shared buffers tend to be occupied by those memory requests targeting memory that provides high latency access (e.g., memory requests received from remote network nodes) or that is being over-utilized. As a result, other requests (e.g., local memory requests) targeting faster or non-congested memory (i.e., memory requests that would be served faster) can become starved in the core due to no available shared buffer entries available to execute said memory requests.

<CIT> relates to a system that using adaptive pre-fetching of memory data using a dynamic table to determine the maximum number of pre-fetched cache lines permissible per stream.

The invention is defined in the claims. In the following description, any embodiment referred to and not falling within the scope of the claims is merely an example useful to the understanding of the invention.

Additionally, it should be appreciated that items included in a list in the form of "at least one of A, B, and C" can mean (A); (B); (C): (A and B); (A and C); (B and C); or (A, B, and C).

The disclosed embodiments may be implemented, in some cases, in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on one or more transitory or non-transitory machine-readable (e.g., computer-readable) storage media (e.g., memory, data storage, etc.), which may be read and executed by one or more processors. A machine-readable storage medium may be embodied as any storage device, mechanism, or other physical structure for storing or transmitting information in a form readable by a machine (e.g., a volatile or non-volatile memory, a media disc, or other media device).

Referring now to <FIG>, in an illustrative embodiment, a system <NUM> for quality of service based throttling in a fabric architecture includes multiple interconnected network nodes <NUM> communicatively coupled via an interconnect fabric <NUM>. The illustrative system <NUM> includes various types of network nodes <NUM> including multiple compute nodes <NUM> and storage nodes <NUM>. The illustrative compute nodes <NUM> include a first compute node, which is designated as compute node (<NUM>) <NUM>, a second compute node, which is designated as compute nodes (<NUM>) <NUM>, and a third compute node, which is designated as compute nodes (N) <NUM> (i.e., the "Nth" compute node of the compute nodes <NUM>, wherein "N" is a positive integer and designates one or more additional compute nodes <NUM>). It should be appreciated that, in other embodiments, there may be any number of compute nodes <NUM> and/or storage nodes <NUM>. Illustratively, the interconnect fabric <NUM> includes a network switch <NUM> and a number of fabric interconnects <NUM> for communicatively coupling the network nodes <NUM>. It should be appreciated, however, that while only a single network switch <NUM> is shown, there may be any number of network switches <NUM> in other interconnect fabric embodiments.

In use, the network nodes <NUM> monitor quality of service levels associated with local resources (e.g., physical and/or virtual components) to detect throttling conditions (e.g., congestion, saturation, over-utilization, workload distribution unfairness, etc.) associated with such resources and transmit throttling messages to other network nodes <NUM> of the fabric architecture requesting a throttling action to be performed by the receiving network nodes <NUM> upon detection of such throttling conditions. The throttling messages may include various types throttling requests directed toward throttling particular resources of a network node <NUM>. For example, the throttling messages may include a memory throttle request, an I/O throttle request, an accelerator throttle processing request, an HFI saturation throttle request, etc. It should be appreciated that the throttling messages are transmitted periodically over the period of time in which the throttling condition is detected. In other words, the network node <NUM> continues to transmit throttling messages until the corresponding throttling condition subsides.

To do so, unlike present technologies in which the network nodes <NUM> do not externally transmit the throttling messages, thereby leaving throttling restricted to being able to only throttle those resources local to the network nodes <NUM>, components of the network nodes <NUM> and the associated interconnect fabric <NUM> are extended to transmit throttling information (e.g., generate new throttling messages, propagate existing throttling signals, etc.) to other network nodes <NUM> that are presently requesting access to a shared structure (e.g., a shared buffer) of the respective one of the network nodes <NUM> having detected the throttling condition.

In an illustrative example, certain coherency protocols include agent entities, such as the caching agents and home agents of Intel® coherency protocols, are configured to initiate transactions into coherent memory (e.g., via the caching agents) and service the coherent transactions (e.g., via the home agents). Such agent entities are presently configured to detect certain conditions local to a respective one of the network nodes <NUM> and issue local processor core throttling signals to throttle one or more cores of the processor. However, contention in fabric architectures can occur not only at the shared paths within each of the network nodes <NUM>, but also in shared paths of the interconnect fabric <NUM>, such as shared buffers (e.g., super queues in the processor core, table of requests in the caching/home agents, etc.).

In an illustrative example, the compute node (<NUM>) <NUM> may be accessing memory of the compute node (<NUM>) <NUM>, which may be configured to monitor memory access requests (e.g., memory accesses received locally, memory accesses received from another of the compute nodes <NUM>, etc.) and memory utilization level(s). Under certain conditions, the compute node (<NUM>) <NUM> may experience high and unequal contention due to memory request queue entries being occupied by requests to slower memory (e.g., non-cache memory) of the compute node (<NUM>) <NUM> that have been received from the compute node (<NUM>) <NUM>. Accordingly, under such conditions, the compute node (<NUM>) <NUM> is configured to transmit a throttling message to the compute node (<NUM>) <NUM> indicating that memory of the compute node (<NUM>) <NUM> is presently saturated, which the compute node (<NUM>) can use to reduce an injection rate of memory requests directed to the compute node (<NUM>) <NUM>.

In some embodiments, the network nodes <NUM> are configured to expose present node throttling techniques between different network nodes <NUM> of the system <NUM> using the transport layer (i.e., Layer <NUM> (L4)) of the Open Systems Interconnection (OSI) model. Accordingly, new and/or existing throttling signals originating from one of the network nodes <NUM> (e.g., from caching agents, home agents, input/output operations, schedulers, etc.) may be propagated over the fabric interconnects <NUM> to other network nodes <NUM>, such as those requesting access to shared structures of the one of the network nodes <NUM> from which the throttling signals originated.

The network nodes <NUM> may be embodied as any type of network traffic (e.g., network packets, messages, data, etc.) computing and/or storage computing device that is capable of performing the functions described herein, such as, without limitation, a server (e.g., stand-alone, rack-mounted, blade, etc.), a network appliance (e.g., physical or virtual), a switch (e.g., rack-mounted, standalone, fully managed, partially managed, full-duplex, and/or half-duplex communication mode enabled, etc.), a router, a web appliance, a distributed computing system, and/or a multiprocessor-based system. As described previously, the illustrative network nodes <NUM> include compute nodes <NUM> and storage nodes <NUM>; however, it should be appreciated that the network nodes <NUM> may include additional and/or alternative network nodes, such as controller nodes, network nodes, utility nodes, etc., which are not shown to preserve clarity of the description.

As shown in <FIG>, an illustrative network node <NUM> includes a first processor, designated as processor (<NUM>) <NUM>, a second processor, designated as processor (<NUM>) <NUM>, an input/output (I/O) subsystem <NUM>, a main memory <NUM>, a data storage device <NUM>, and communication circuitry <NUM>. It should be appreciated that the compute nodes <NUM> and/or storage nodes <NUM> of <FIG> may include the components described in <FIG> of the illustrative network node <NUM>.

Of course, the network node <NUM> may include other or additional components, such as those commonly found in a computing device, in other embodiments. Additionally, in some embodiments, one or more of the illustrative components may be incorporated in, or otherwise form a portion of, another component. For example, the cache memory <NUM>, or portions thereof, may be incorporated in one or both of the processors <NUM>, <NUM> in some embodiments. Further, in some embodiments, one or more of the illustrative components may be omitted from the network node <NUM>. For example, although the illustrative network node <NUM> includes two processors <NUM>, <NUM>, the network node <NUM> may include a greater number of processors, in other embodiments.

Each of the processors <NUM>, <NUM> (i.e., physical processor packages) may be embodied as any type of multi-core processor capable of performing the functions described herein, such as, but not limited to, a single physical multi-processor core chip, or package. The illustrative processor (<NUM>) <NUM> includes a number of processor cores <NUM>, while the illustrative processor (<NUM>) <NUM> similarly includes a number of processor cores <NUM>. As described previously, each of the processors <NUM>, <NUM> includes more than one processor cores (e.g., <NUM> processors cores, <NUM> processors cores, <NUM> processors cores, <NUM> processors cores, etc.).

Each of processor cores <NUM>, <NUM> is embodied as an independent logical execution unit capable of executing programmed instructions. In some embodiments, the processor cores <NUM>, <NUM> may include a portion of cache memory (e.g., an L1 cache) and functional units usable to independently execute programs or threads. It should be appreciated that in some embodiments of the network node <NUM>, such as supercomputers, the network node <NUM> may include thousands of processor cores. Each of the processors <NUM>, <NUM> may be connected to a physical connector, or socket, on a motherboard (not shown) of the network node <NUM> configured to accept a single physical processor package (i.e., a multi-core physical integrated circuit).

The illustrative processor (<NUM>) <NUM> additionally includes a cache memory <NUM>. Similarly, the illustrative processor (<NUM>) <NUM> also includes a cache memory <NUM>. Each cache memory <NUM>, <NUM> may be embodied as any type of cache that the respective processor <NUM>, <NUM> can access more quickly than the main memory <NUM>, such as an on-die cache, or on-processor cache. In other embodiments, the cache memory <NUM>, <NUM> may be an off-die cache, but reside on the same system-on-a-chip (SoC) as the respective processor <NUM>, <NUM>. It should be appreciated that, in some embodiments, the cache memory <NUM>, <NUM> may have a multi-level architecture. In other words, in such multi-level architecture embodiments, the cache memory <NUM>, <NUM> may be embodied as an L1, L2, or L3 cache, for example.

The main memory <NUM> may be embodied as any type of volatile or non-volatile memory or data storage device capable of performing the functions described herein. In operation, the main memory <NUM> may store various data and software used during operation of the network node <NUM>, such as operating systems, applications, programs, libraries, and drivers. The main memory <NUM> is communicatively coupled to the processors <NUM>, <NUM> via the I/O subsystem <NUM>, which may be embodied as circuitry and/or components to facilitate input/output operations with the processors <NUM>, <NUM>, the main memory <NUM>, and other components of the network node <NUM>. For example, the I/O subsystem <NUM> may be embodied as, or otherwise include, memory controller hubs, input/output control hubs, firmware devices, communication links (i.e., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.) and/or other components and subsystems to facilitate the input/output operations. In some embodiments, the I/O subsystem <NUM> may form a portion of a SoC and be incorporated, along with one or both of the processors <NUM>, <NUM>, the main memory <NUM>, and/or other components of the network node <NUM>, on a single integrated circuit chip.

The data storage device <NUM> may be embodied as any type of device or devices configured for short-term or long-term storage of data such as, for example, memory devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices. It should be appreciated that the data storage device <NUM> and/or the main memory <NUM> (e.g., the computer-readable storage media) may store various data as described herein, including operating systems, applications, programs, libraries, drivers, instructions, etc., capable of being executed by a processor (e.g., the processor <NUM>, the processor <NUM>, etc.) of the network node <NUM>.

The communication circuitry <NUM> may be embodied as any communication circuit, device, or collection thereof, capable of enabling communications between the network node <NUM> and other computing devices (e.g., a compute node <NUM>, a storage node <NUM>, etc.) over a network. The communication circuitry <NUM> may be configured to use any one or more communication technologies (e.g., wireless or wired communication technologies) and associated protocols (e.g., Internet Protocol (IP), Ethernet, Bluetooth®, Wi-Fi®, WiMAX, LTE, <NUM>, etc.) to effect such communication.

The illustrative communication circuitry <NUM> includes a host fabric interface (HFI) <NUM>. The HFI <NUM> may be embodied as one or more add-in-boards, daughtercards, network interface cards, controller chips, chipsets, or other devices that may be used by the network node <NUM>. For example, in some embodiments, the HFI <NUM> may be integrated with one or both of the processors <NUM>, <NUM> (e.g., on a coherent fabric within one or both of the processors <NUM>, <NUM>), embodied as an expansion card coupled to the I/O subsystem <NUM> over an expansion bus (e.g., PCI Express (PCIe)), part of a SoC that includes one or more processors, or included on a multichip package that also contains one or more processors. Additionally or alternatively, in some embodiments, functionality of the HFI <NUM> may be integrated into one or more components of the network node <NUM> at the board level, socket level, chip level, and/or other levels. The HFI <NUM> is configured to facilitate the transfer to data/messages to enable tasks executing on the processors <NUM>, <NUM> to access shared structures (e.g., shared physical memory) of the other network nodes <NUM>, such as may be necessary during parallel or distributed computing operations.

It should be appreciated that those network nodes <NUM> implemented as storage nodes <NUM> may generally include more data storage device <NUM> capacity than those network nodes <NUM> implemented as compute nodes <NUM>. Similarly, it should also be appreciated that those network nodes <NUM> implemented as compute nodes <NUM> may generally include more processor capability that those network nodes <NUM> implemented as storage nodes <NUM>. In other words, the storage nodes <NUM> may be embodied as physical servers including numerous hard-disk drives (HDDs) or solid-state drives (SDDs) relative to the number of storage devices of the compute nudes <NUM>, whereas the compute nodes <NUM> may be embodied as physical servers including numerous processors having multiple cores relative to the number of processors of the storage nodes <NUM>. However, it should be further appreciated that any of the network nodes <NUM> may be implemented as a compute node <NUM> and/or a storage node <NUM>, regardless of the component configuration relative to the other network nodes <NUM>.

Referring again to <FIG>, the interconnect fabric <NUM>, illustratively the combination of the network switch <NUM> and the fabric interconnects <NUM>, may be embodied as one or more buses, switches, and/or networks configured to support transmission of network traffic as a function of various interconnect protocols and/or network protocols. In use, the interconnect fabric <NUM> is utilized by the network nodes <NUM> (e.g., via respective HFIs <NUM>) to communicate with the other network nodes <NUM> (i.e., across the interconnect fabric <NUM>). Accordingly, the network switch <NUM> may be embodied as any type of switching device (e.g., a crossbar switch) capable of network traffic forwarding via the fabric interconnects <NUM> in a switched, or switching, fabric architecture.

Referring now to <FIG>, in an illustrative embodiment, the network node <NUM> of <FIG> includes one or more non-uniform memory access (NUMA) domains <NUM> communicatively coupled to the HFI <NUM>. The illustrative NUMA domains <NUM> include a first NUMA domain, designated as NUMA domain (<NUM>) <NUM>, and a second NUMA domain, designated as NUMA domain (<NUM>) <NUM>. Each of the NUMA domains <NUM> includes a number of allocated processor cores of a physical processor package, referred to herein as a processor. As shown in the illustrative embodiment, the NUMA domain (<NUM>) <NUM> includes the processor cores <NUM> of processor (<NUM>) <NUM> and the NUMA domain (<NUM>) <NUM> includes the processor cores <NUM> of processor (<NUM>) <NUM>. However, it should be appreciated that, in some embodiments, the processor cores <NUM> of the processor <NUM> and/or the processor cores <NUM> of the processor <NUM> may be divided and each set of divided processor cores may be allocated to a different NUMA domain <NUM>. It should be appreciated that each set of allocated processor cores assigned to a respective one of the NUMA domains <NUM> may be referred to as socket cores. In other words, the number of allocated cores of a physical processor package may be referred to as a socket.

Additionally, each of the NUMA domains <NUM> corresponds to a particular memory type (e.g., double data rate (DDR) memory, disk, etc.) and includes a portion of that memory type of local memory (e.g., the main memory <NUM>), which has been allocated to the processor cores of the respective NUMA domain <NUM>. Further, the local memory is directly linked to the physical processor package on which the processor cores reside. In the illustrative embodiment, the NUMA domain (<NUM>) <NUM> includes a local memory (<NUM>) <NUM> and the NUMA domain (<NUM>) <NUM> includes a local memory (<NUM>) <NUM>. In some embodiments, data may be transmitted between the NUMA domains <NUM> via an interconnect <NUM> (e.g., an Intel® UltraPath Interconnect (UPI), an Intel® QuickPath Interconnect (QPI), an AMD® Unified Media Interface (UMI) interconnect, or the like). The local memory of one of the NUMA domains <NUM> is considered to be remote, or foreign, relative to the other NUMA domains <NUM>. Accordingly, it should be appreciated that network traffic transmitted across the interconnect <NUM> may introduce load/contention, increase overall bandwidth usage, and reduce latency associated with accesses to remote memory, as compared to data being processed using the local memory.

Each of the illustrative processors <NUM>, <NUM> additionally includes an on-die interconnect (e.g., the on-die interconnect <NUM> of the processor <NUM> and the on-die interconnect <NUM> of the processor <NUM>) configured to interface with the HFI <NUM> via point-to-point interfaces <NUM> capable of facilitating the transfer of data between the HFI <NUM> and the processors <NUM>, <NUM>. In some embodiments, the NUMA domains <NUM> may be defined internally in the HFI <NUM>. In an illustrative example, one of the NUMA domains <NUM> (e.g., the NUMA domain (<NUM>) <NUM>) of one of the network nodes <NUM> (e.g., the compute node (<NUM>) <NUM>) may correspond to transactions processed by the HFI <NUM> from another of the network nodes <NUM> (e.g., the compute node (<NUM>) <NUM>). Accordingly, the HFI <NUM> of the compute node (<NUM>) <NUM> can issue throttling messages to the compute node (<NUM>) <NUM> upon a determination by the compute node (<NUM>) <NUM> that the compute node (<NUM>) <NUM> is issuing too many requests to the compute node (<NUM>) <NUM>. In some embodiments, such throttling messages may include information propagated from caching agents of the processor <NUM> received by the HFI <NUM> via the point-to-point interfaces <NUM>.

Referring now to <FIG>, in an illustrative embodiment, one of the network nodes <NUM> establishes an environment <NUM> during operation. The illustrative environment <NUM> includes a communication management module <NUM>, a quality of service (QoS) monitoring module <NUM>, a throttling message transmission module <NUM>, a throttling message reception module <NUM>, and a throttling response execution module <NUM>. The various modules of the environment <NUM> may be embodied as hardware, firmware, software, or a combination thereof. As such, in some embodiments, one or more of the modules of the environment <NUM> may be embodied as circuitry or collection of electrical devices (e.g., a communication management circuit <NUM>, a QoS monitoring circuit <NUM>, a throttling message transmission circuit <NUM>, a throttling message reception circuit <NUM>, a throttling response execution circuit <NUM>, etc.).

It should be appreciated that, in such embodiments, one or more of the communication management circuit <NUM>, the QoS monitoring circuit <NUM>, the throttling message transmission circuit <NUM>, the throttling message reception circuit <NUM>, and the throttling response execution circuit <NUM> may form a portion of one or more processors (e.g., processor (<NUM>) <NUM> and processor (<NUM>) <NUM> of <FIG>), the I/O subsystem <NUM>, the communication circuitry <NUM>, and/or other components of the network nodes <NUM>. Additionally, in some embodiments, one or more of the illustrative modules may form a portion of another module and/or one or more of the illustrative modules may be independent of one another. Further, in some embodiments, one or more of the modules of the environment <NUM> may be embodied as virtualized hardware components or emulated architecture, which may be established and maintained by the one or more processors and/or other components of the network nodes <NUM>.

In the illustrative environment <NUM>, the network node <NUM> further includes network node data <NUM>, monitoring result data <NUM>, request monitoring data <NUM>, and NUMA identification data <NUM>, each of which may be stored in the main memory <NUM> and/or the data storage device <NUM> of the network node <NUM>. Further, each of the network node data <NUM>, the monitoring result data <NUM>, the request monitoring data <NUM>, and the NUMA identification data <NUM> may be accessed by the various modules and/or sub-modules of the network node <NUM>. Additionally, it should be appreciated that in some embodiments the data stored in, or otherwise represented by, each of the network node data <NUM>, the monitoring result data <NUM>, the request monitoring data <NUM>, and the NUMA identification data <NUM> may not be mutually exclusive relative to each other.

For example, in some implementations, data stored in the network node data <NUM> may also be stored as a portion of the monitoring result data <NUM>, and/or vice versa. As such, although the various data utilized by the network node <NUM> is described herein as particular discrete data, such data may be combined, aggregated, and/or otherwise form portions of a single or multiple data sets, including duplicative copies, in other embodiments. It should be further appreciated that the network node <NUM> may include additional and/or alternative components, sub-components, modules, sub-modules, and/or devices commonly found in a computing device, which are not illustrated in <FIG> for clarity of the description.

The communication management module <NUM>, which may be embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof as discussed above, is configured to facilitate inbound and outbound wired and/or wireless network communications (e.g., network traffic, network packets, network flows, etc.) to and from the network node <NUM>. To do so, the communication management module <NUM> is configured to receive and process network packets from other network nodes <NUM> via the interconnect fabric. Additionally, the communication management module <NUM> is configured to prepare and transmit network packets to other network nodes <NUM> via the interconnect fabric. Accordingly, in some embodiments, at least a portion of the functionality of the communication management module <NUM> may be performed by the communication circuitry <NUM> of the network node <NUM>, or more specifically by the HFI <NUM> of the communication circuitry <NUM>. In some embodiments, data usable to communicate with the other network nodes <NUM> of the fabric architecture, such as IP address information, flow information, etc., may be stored in the network node data.

The QoS monitoring module <NUM>, which may be embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof as discussed above, is configured to monitor various characteristics of the network node <NUM>. To do so, the illustrative QoS monitoring module <NUM> includes a resource utilization monitoring module <NUM>, a load balancing monitoring module <NUM>, and an HFI saturation monitoring module <NUM>. It should be appreciated that each of the resource utilization monitoring module <NUM>, the load balancing monitoring module <NUM>, and the HFI saturation monitoring module <NUM> of the QoS monitoring module <NUM> may be separately embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof. For example, the resource utilization monitoring module <NUM> may be embodied as a hardware component, while the load balancing monitoring module <NUM> and/or the HFI saturation monitoring module <NUM> may be embodied as a virtualized hardware component or as some other combination of hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof.

The resource utilization monitoring module <NUM> is configured to monitor utilization levels of the resources (i.e., physical and/or virtual components) of the network node <NUM>. In an illustrative example, the resource utilization monitoring module <NUM> may be configured to monitor memory utilization levels. To do so, in some embodiments, the resource utilization monitoring module <NUM> may be configured to receive throttling signals presently generated by one or more local caching agents of a processor of the network node <NUM> that is usable to slow down or otherwise reduce an injection rate to a given memory type indicated by the throttling signals. Additionally or alternatively, the resource utilization monitoring module <NUM> may be configured to identify present usage levels of the resources to determine a saturation level of the monitored resources.

The load balancing monitoring module <NUM> is configured to monitor the distribution of workloads across the resources (i.e., physical and/or virtual components) of the network node <NUM>. The HFI utilization monitoring module <NUM> is configured to monitor utilization of the HFI <NUM>. Accordingly, the HFI utilization monitoring module <NUM> can detect a saturation of the HFI <NUM> even if the resources attached thereto have not become saturated. In an illustrative example, one of the compute nodes <NUM> may saturate an HFI <NUM> of one of the storage nodes <NUM> when accessing storage devices of the storage node <NUM>. Under such conditions, the HFI <NUM> of the storage node <NUM> may become saturated, while the storage devices of the storage node <NUM> may not be fully utilized (i.e., saturated). In some embodiments, the monitoring results (e.g., present/historical utilization values, present/historical load balancing information, etc.) may be stored in the monitoring result data <NUM>.

The throttling message transmission module <NUM>, which may be embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof as discussed above, is configured to generate and transmit throttling messages to the other network nodes <NUM>. As described previously, certain conditions (i.e., throttling conditions) may exist on the network node <NUM> such that resource access requests generated by the network node <NUM> requesting access to local resources of the network node <NUM> may become starved due to the other network nodes <NUM> maintaining an unencumbered injection rate to resources that are locally throttled. Accordingly, unlike present technologies that only provide local throttling, the network node <NUM> is configured to detect such throttling conditions and generate a throttling message for transmission to the other network nodes <NUM> responsible for, or otherwise contributing to, the throttling conditions.

To generate and transmit throttling messages to the other network nodes <NUM>, the illustrative throttling message transmission module <NUM> includes a throttling condition detection module <NUM> and a transmission mode determination module <NUM>. It should be appreciated that each of the throttling condition detection module <NUM> and the transmission mode determination module <NUM> of the QoS monitoring module <NUM> may be separately embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof. For example, the throttling condition detection module <NUM> may be embodied as a hardware component, while the transmission mode determination module <NUM> is embodied as a virtualized hardware component or as some other combination of hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof.

The throttling condition detection module <NUM> is configured to detect whether a throttling condition exists. To do so, the throttling condition detection module <NUM> may be configured to compare present quality of service conditions (e.g., as may be determined by the QoS monitoring module <NUM>) to corresponding thresholds. For example, the throttling condition detection module <NUM> may be configured to compare a present memory utilization level against a memory saturation threshold. Accordingly, the throttling condition detection module <NUM> may detect a throttling condition upon a determination that the present memory utilization level exceeds the memory saturation threshold. Additionally or alternatively, the throttling condition detection module <NUM> may be configured to process throttling signals generated inside the network node <NUM>. As described previously, an agent entity (e.g., a caching agent, a home agent, etc.) may generate local throttle requests to particular on-die clusters, such as memory or I/O. Accordingly, the throttling condition detection module <NUM> is configured to interpret such local throttle requests to determine whether they indicate a throttling condition whereby one or more of the other network nodes <NUM> should be notified to take an appropriate throttling action.

The transmission mode determination module <NUM> is configured to determine which transmission mode to use to transmit the throttling message generated in response to a detected throttling condition, as may be detected by the throttling condition detection module <NUM>. To do so, the transmission mode determination module <NUM> is configured to detect which one or more of the network nodes <NUM> to transmit the throttling message based on the identified other network node(s) <NUM> responsible for, or otherwise contributing to, the throttling condition. For example, the transmission mode determination module <NUM> may determine a single network node <NUM> is issuing too many memory access requests, in which case the transmission mode determination module <NUM> may determine to transmit the generated throttling message using a unicast mode. Otherwise, if the transmission mode determination module <NUM> determines more than one of the network nodes <NUM> is responsible for, or otherwise contributing to, the present throttling condition, the transmission mode determination module <NUM> may determine to transmit the generated throttling message using a multicast mode.

As described previously, the throttling message transmission module <NUM> is configured to transmit throttling messages requesting another network node <NUM> take an action (e.g., throttle processor cores of a particular NUMA domain) in response to receipt of the throttling messages. Each of the NUMA domains <NUM> of each of the network nodes <NUM> has a corresponding NUMA domain identifier usable by the throttling message transmission module <NUM> to determine which NUMA domain <NUM> is to be throttled. Accordingly, the network node <NUM> includes NUMA domain identifiers of the NUMA domains <NUM> local to the network node <NUM> as well as NUMA domain identifiers of the NUMA domains of the other network nodes <NUM>. However, in some embodiments, the NUMA domain identifiers may not be known, such as in distributed tag directory schemes. In such embodiments, the throttling message transmission module <NUM> can predict which NUMA domain <NUM> the receiving network nodes <NUM> will perform the responsive action.

To do so, the throttling message transmission module <NUM> may be further configured to predict which NUMA domain <NUM> a receiving network node <NUM> will take action upon based on the principle that applications accessing NUMA domains will operate within a certain range of memory addresses in that NUMA domain. An on-die interconnect interface (e.g., one of the point-to-point interfaces <NUM> of <FIG>) of the network node <NUM> is configured to generate requests to an agent entity, such as the caching agents. Accordingly, the on-die interconnect interface may be extended to use a domain prediction table to determine which NUMA domain corresponds to the throttling message, as well as whether the NUMA domain (e.g., a processor core of the NUMA domain) is presently distressed (i.e., has been throttled for that NUMA domain). If a component of the NUMA domain is presently distressed, the throttling message may not be issued (i.e., injected) to the agent entity until the distress is no longer present and acknowledged by the agent entity. Accordingly, use of the domain prediction table may allow the network node <NUM> to speculate the affected NUMA domain <NUM> of the other network node <NUM>. In some embodiments, data of the domain prediction table may be stored in the NUMA identification data <NUM>.

The domain prediction table may include an identifier of each agent entity, a NUMA level that is known by each of the network nodes <NUM>, a last address range (e.g., formatted as a bit mask) accessed for each of the NUMA domains <NUM> and/or agent entities, and a granularity which may be configurable per NUMA domain <NUM>. In an illustrative embodiment, the granularity may be 4GB for a particular NUMA domain <NUM> and a last address (e.g., 0x78C9657FA) sent to a particular agent entity targeting a particular NUMA level belongs to the address range 0x700000000 - 0x700000000+4GB. As described previously, applications accessing NUMA domains will operate within a certain range of memory addresses in that NUMA domain. As such, by appropriately specifying a granularity, it may yield a more accurate prediction, resulting in a high hit rate, as well as return a result within a few cycles. Accordingly, in some embodiments, to predict the NUMA domain of a throttling message targeting a particular address and a particular agent entity, the throttling message transmission module <NUM> may be configured to access the domain prediction table to retrieve the predicted NUMA domain as a content-addressable memory (CAM) structure. For example, if a use case for an application is to allocate a 10GB memory block of a storage node <NUM> via memory exposed as a NUMA domain, the domain prediction requests would most likely hit on the prediction table if the granularity chosen is GM.

In some embodiments, the flow for a throttling message to a particular address to a particular agent entity may include determining a modulus of the last memory address and the granularity to predict the NUMA domain to which the last memory address belongs. If the predicted NUMA domain request returns NULL (i.e., none of the NUMA domains matched) then it may be assumed the closest NUMA domain is NUMA level <NUM>. As described previously, the processor core does not send transactions to the agent entity when a distress signal for the NUMA domain is active and only issues transactions after the distress signal is deactivated and acknowledged by the agent entity. Accordingly, if the distress signal is active for the predicted NUMA domain result, the processor core does not send transactions to the agent entity until the distress signal is deactivated and acknowledged by the agent entity. Additionally, in some embodiments, the agent entity may perform the system address decoding, update appropriate counters (e.g., throttling request counters), and generate the distress signal as necessary. Further, the prediction table is updated according to feedback received upon the agent entity having returned an acknowledgement and a NUMA domain identifier for that specific domain prediction request.

It should be appreciated that the targeted fiber architecture for the QoS-based throttling scheme described herein is directed toward enterprise systems with a scale of hundreds of network nodes <NUM>. Accordingly, in such embodiments with a greater scale, such as a scale of thousands of nodes of high performance computing (HPC) embodiments, the multicast mode may not be ideal for implementation due to the voluminous amount of messages that may be transmitted therein. However, sub-domains of network nodes <NUM> (e.g., consisting of only those network nodes <NUM> connected to a specific network switch <NUM>) of the fabric architecture may be defined, such as by using specific multicast topologies, in order to propagate the throttling message to only a subset of the network nodes <NUM>. It should be further appreciated that the multicast mode may be a non-reliable multicast. As described previously, the throttling messages are transmitted periodically over the duration of time in which the throttling condition exists, thereby negating the need to acknowledge the receipt of the throttling messages. The reliability may be improved, such as by adding receipt acknowledgements into the flow; however, such reliability improvement is likely to add more pressure into the fabric.

In some embodiments, the throttling message transmission module <NUM> (e.g., the throttling condition detection module <NUM> and/or the transmission mode determination module <NUM>) may utilize a request monitoring table to determine when a throttling message is to be generated and/or to which network nodes <NUM> the generated throttling message is to be sent. In an illustrative example, the throttling message transmission module <NUM> may be configured to account for external transactions targeting the local NUMA domain <NUM> of a particular network node <NUM>. As described previously, each of the NUMA domains <NUM> has a corresponding NUMA domain identifier usable by the throttling message transmission module <NUM> to determine which NUMA domain <NUM> a received throttling message corresponds, as well as a request counter that is incremented with each access.

In some embodiments, the NUMA domain identifier, a value of the request counter, as well as other values (e.g., enumerated values of throttling message request types) may be stored in model-specific registers (MSRs). Accordingly, the throttling message transmission module <NUM> may be configured to read a value of the request counter to determine whether the request counter exceeds a threshold value. It should be appreciated that the MSR values can be configured during operation or boot time (e.g., using ring zero functions), and may be exposed to the operating system of the network node <NUM>.

In some embodiments, the request counter may be stored in the request monitoring table that includes an identifier of the network node <NUM> from which the throttling message was received, a present value of the request counter, the NUMA domain identifier, and the threshold value. In some embodiments, the data of the request monitoring table may be stored in the request monitoring data <NUM>. If the request counter exceeds a threshold value, the throttling message transmission module <NUM> may be configured to generate a throttling message for transmission in unicast mode (i.e., to just the network node <NUM> responsible for the present state of the request counter). Additionally, the throttling message transmission module <NUM> may be configured to generate a throttling message for transmission in multicast mode (i.e., to all the other network nodes <NUM> issuing transactions to a particular NUMA domain) upon receiving a throttling signal internally, such as from a caching agent. As described previously, the throttling message transmission module <NUM> is configured to generate the throttling message periodically while the throttling condition is detected.

It should be appreciated that the system configurations, such as the NUMA domain identifiers, MSRs in the different network nodes <NUM>, etc., should be done holistically to ensure coherency. Accordingly, the system configurations should be enforced at system boot time (e.g., when the routing system address decoding scheme is performed) to ensure that information conveyed in the throttling messages is consistent across the different network nodes <NUM>. For example, in an embodiment wherein NUMA domain identifiers for compute node (<NUM>) <NUM> are being propagated to compute node (<NUM>) <NUM>, compute node (<NUM>) <NUM> should already be aware of which NUMA domain identifier corresponds to the particular NUMA domain <NUM> of the compute node (<NUM>) <NUM>. In some embodiments, the NUMA domain identifiers of the other network nodes <NUM> may be stored in the NUMA identification data <NUM>.

The throttling message reception module <NUM>, which may be embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof as discussed above, is configured to receive and process throttling messages from the other network nodes <NUM>. To do so, the illustrative throttling message reception module <NUM> includes a throttling type identification module <NUM> and a NUMA target identification module <NUM>. It should be appreciated that each of the throttling type identification module <NUM> and the NUMA target identification module <NUM> of the throttling message reception module <NUM> may be separately embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof. For example, the throttling type identification module <NUM> may be embodied as a hardware component, while the NUMA target identification module <NUM> is embodied as a virtualized hardware component or as some other combination of hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof.

The throttling type identification module <NUM> is configured to identify a type associated with the received throttling message. As described previously, the throttling message request types associated with the throttling messages may include a memory throttle request, an I/O throttle request, an accelerator throttle processing request, an HFI saturation throttle request, etc. In some embodiments, the throttling message request types may be enumerated such that they can be mapped to a particular action. Additionally, some embodiments, the enumerated values of the throttling message request types may be stored in a throttle action table that maps the enumerated values to the corresponding action. The NUMA target identification module <NUM> is configured to identify a NUMA domain target, or component thereof, associated with the received throttling message.

The throttling response execution module <NUM>, which may be embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof as discussed above, is configured to take an action in response to having received a throttling message from another network node <NUM>. To do so, the illustrative throttling response execution module <NUM> includes a processor core throttling execution module <NUM>, a software interrupt execution module <NUM>, and an HFI throttling execution module <NUM>. It should be appreciated that each of the processor core throttling execution module <NUM>, the software interrupt execution module <NUM>, and the HFI throttling execution module <NUM> of the throttling response execution module <NUM> may be separately embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof. For example, the processor core throttling execution module <NUM> may be embodied as a hardware component, while the software interrupt execution module <NUM> and/or the HFI throttling execution module <NUM> may be embodied as a virtualized hardware component or as some other combination of hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof.

The processor core throttling execution module <NUM> is configured to throttle processor cores in response to receiving a propagated throttling message. To do so, the processor core throttling execution module <NUM> is configured to translate a received throttling message to corresponding on-die interconnect throttling signals supported by the network node <NUM> architecture to reduce an injection rate of externally transmitted access requests. The software interrupt execution module <NUM> is configured to perform a software interrupt in response to having received a software interrupt request throttling message. To do so, the throttling message is propagated to the software stack via a software interrupt in such embodiments wherein the software stack supports load balancing and injection control mechanisms.

The HFI throttling execution module <NUM> is configured to throttle injections at the HFI <NUM> based on the type of throttling message received. In other words, the HFI <NUM> is responsible for reducing the injection rate or stopping the injection altogether. Accordingly, such a response may be a suitable solution for throttle message types not supported by the fabric architecture of the network nodes <NUM>. It should be appreciated that the processor cores and other injectors of the network node <NUM> are not being throttled.

Referring now to <FIG>, in use, a network node <NUM> (e.g., one of the network nodes <NUM> of <FIG>) may execute a method <NUM> for processing a local memory request from a remote network node (i.e., another one of the network nodes <NUM> of the fabric architecture). The method <NUM> begins in block <NUM>, in which the network node <NUM> determines whether a memory access request has been received from a remote network node. If not, the method <NUM> loops back to block <NUM> to determine whether a memory access request has been received from a remote network node; otherwise, the method <NUM> advances to block <NUM>. In block <NUM>, the network node <NUM> inserts the received remote memory access request into a shared buffer of the network node <NUM>. It should be appreciated that, under certain conditions, the shared buffer may be full for a period of time before the network node <NUM> can insert the received remote memory access request into the shared buffer.

In block <NUM>, the network node <NUM> determines whether to process the received request (e.g., pop the corresponding entry from the shared buffer and process the request). If so, the method <NUM> advances to block <NUM>, in which the network node <NUM> performs an action in response to the received remote memory access request. For example, in block <NUM>, the network node <NUM> may transmit request data in response to a remote memory access request having requested data stored in memory (e.g., the main memory <NUM>) of the network node <NUM>. Alternatively, in block <NUM>, the network node <NUM> may store data received with the remote memory access request. In some embodiments, in block <NUM>, the network node <NUM> may transmit an acknowledgement in response to having received/processed the remote memory access request.

Referring now to <FIG>, in use, a network node <NUM> (e.g., one of the network nodes <NUM> of <FIG>) may execute a method <NUM> for accessing memory of a remote network node (i.e., another one of the network nodes <NUM> of the fabric architecture). The method <NUM> begins in block <NUM>, in which the network node <NUM> determines whether to access memory located in another network node <NUM>. For example, the network node <NUM> may be retrieving data replicated in remote memory (i.e., memory of the remote network node), executing an application utilizing distributed data structures on one or more remote network nodes, employing log shipping (i.e., relying on a log or micro-log stored on the remote network node for failure recovery), or performing some other operation that requires accessing memory of a remote network node.

If not, the method <NUM> loops back to block <NUM> to again determine whether to access memory located in another network node <NUM>; otherwise, the method <NUM> advances to block <NUM>. In block <NUM>, the network node <NUM> generates a remote memory access request that includes memory address information usable to retrieve or store data of the remote memory access request. Additionally, in block <NUM>, the network node <NUM> includes source identifying information of the network node <NUM>. In block <NUM>, the network node <NUM> inserts the memory access request into a message transmission queue.

In block <NUM>, the network node <NUM> determines whether an injection rate corresponding to the component(s) from which the remote memory access request is requesting access has been throttled as a result of throttling messages received from the remote network node (see, e.g., the method <NUM> of <FIG> directed toward generating throttling messages for external transmission to one or more remote network nodes). If not, the method <NUM> branches to block <NUM>, in which the network node <NUM> transmits the remote memory access request at a non-throttled injection rate; otherwise, the method <NUM> branches to block <NUM>, in which the network node <NUM> transmits the remote memory access request at a throttled rate.

Referring now to <FIG>, in use, a network node <NUM> (e.g., one of the network nodes <NUM> of <FIG>) may execute a method <NUM> for generating throttling messages for external transmission to one or more remote network nodes (i.e., one or more of the other network nodes <NUM>). The method <NUM> begins in block <NUM>, in which the network node <NUM> monitors quality of service levels of the network node <NUM>. For example, in block <NUM>, the network node <NUM> monitors utilization levels of the resources (e.g., memory, processors, components of a NUMA domain, etc.) of the network node <NUM>, in some embodiments. Additionally or alternatively, in block <NUM>, the network node <NUM> monitors the distribution of the workloads distributed across the components of the network node <NUM>, in some embodiments. In block <NUM>, the network node <NUM> additionally or alternatively monitors saturation levels of the HFI <NUM> of the network node <NUM>, in some embodiments. As described previously, in some embodiments, the network node <NUM> may rely on a request monitoring table to determine when the throttling messages are to be generated for a particular NUMA domain <NUM>.

As also described previously, certain conditions (i.e., throttling conditions) may exist on the network node <NUM> such that resource access requests generated by the network node <NUM> requesting access to local resources of the network node <NUM> may become starved due to remote network nodes <NUM> maintaining an unencumbered injection rate to resources that are locally throttled. Accordingly, in block <NUM>, the network node <NUM> determines whether a throttling condition (e.g., congestion, saturation, over-utilization, workload distribution unfairness, etc., of a component of the network node <NUM>) has been detected (i.e., presently exists) as a result of the quality of service monitoring performed in block <NUM>.

If the network node <NUM> determines that a throttling condition does not exist, the method <NUM> loops back to block <NUM> to continue monitoring the quality of service levels of the network node <NUM>; otherwise, the method <NUM> advances to block <NUM>, in which the network node <NUM> generates a throttling message. In block <NUM>, the network node <NUM> includes a throttling message request type indicator with the throttling message. As described previously, the throttling message request types associated with the throttling messages may include a memory throttle request, an I/O throttle request, an accelerator throttle processing request, an HFI saturation throttle request, etc. Additionally, in block <NUM>, the network node <NUM> includes a throttling message source indicator. The throttling message source indicator may include an identifier of the component (e.g., a NUMA domain identifier, an HFI identifier) for which the throttling condition has been detected and/or an identifier of the network node <NUM>.

In block <NUM>, the network node <NUM> identifies one or more target network nodes (i.e., one or more of the other network nodes <NUM> of the fabric architecture) that are to receive the throttling message generated in block <NUM>. As described previously, in some embodiments, the network node <NUM> may rely on a request monitoring table to determine the one or more target network nodes. In block <NUM>, the network node <NUM> transmits the generated throttling message to the one or more target network nodes identified in block <NUM>. To do so, in block <NUM>, the network node <NUM> transmits the generated throttling message based on a cycle rate corresponding to each of the target network nodes.

Depending on the number of target network nodes identified in block <NUM>, the network node <NUM> may transmit the generated throttling message via a multicast transmission (i.e., more than one target network node) in block <NUM>, or via a unicast transmission (i.e., a single target network node) in block <NUM>. Additionally, in some embodiments, in block <NUM>, the network node <NUM> may transmit the generated throttling message via the transport layer of the OSI model. To do so, in some embodiments, the fabric may be extended with a new type of virtual channel that facilitates the transfer of the throttling messages in order to segregate the throttling messages from the existing channels of the fabric. Such embodiments may be implemented via a new type of physical wire that takes fastest paths inside the fabric in order to deliver the throttling messages as fast as possible.

As described previously, the throttling messages are transmitted periodically over the course of the detected throttling condition. As such, the method <NUM> monitors quality of service levels relative to that specific throttling condition and iterate the method <NUM> as a result of that specific quality of service level monitoring.

Referring now to <FIG>, in use, a network node <NUM> (e.g., one of the network nodes <NUM> of <FIG>) may execute a method <NUM> for processing throttling messages received from a remote network node (i.e., one of the other network nodes <NUM>). The method <NUM> begins in block <NUM>, in which the network node <NUM> determines whether a throttling message has been received from a remote network node. If not, the method <NUM> loops back to block <NUM> to again determine whether a throttling message has been received from a remote network node; otherwise, the method <NUM> advances to block <NUM>, in which the network node <NUM> identifies information associated with the throttling message received in block <NUM>.

For example, in block <NUM>, the network node <NUM> identifies a type of the throttling message. As described previously, the throttling message request types associated with the throttling messages may include a memory throttle request, an I/O throttle request, an accelerator throttle processing request, an HFI saturation throttle request, etc. Additionally, in block <NUM>, the network node <NUM> identifies a source of the throttling message. The source of the throttling message may include information that identifies the target network node from which the throttling message was received. Additionally, the source of the throttling message may include a component identifier (e.g., a NUMA identifier) identifying a component of the remote network node from which the throttling quest was received. In some embodiments, the throttling message may additionally include component information of the receiving network node <NUM> usable to identify which network node <NUM> resources, from which remote memory accesses are being requested, are to be throttled.

In block <NUM>, the network node <NUM> performs an action based on the received throttling message, such as may be based on the type of the throttling message identified in block <NUM>. For example, in block <NUM>, the network node <NUM> may reduce an injection rate for shared resource access requests being transmitted to (i.e., targeting) the remote network node by self-throttling requests by the HFI <NUM> of the network node <NUM>. In another example, in block <NUM>, the network node <NUM> may throttle processor cores of the network node <NUM> by using the existing throttling schemes. To do so, the network node <NUM> may propagate the received throttling message to an agent entity (e.g., a caching agent) via a corresponding on-die interconnect (e.g., the on-die interconnect <NUM> of the processor <NUM>, the on-die interconnect <NUM> of the processor <NUM>, etc.) to throttle processor cores of the network node <NUM> by using the existing throttling schemes. In still another example, in block <NUM>, in such embodiments wherein the software stack supports load balancing and injection control mechanisms, the network node <NUM> may propagate the received throttling message to a software stack via a software interrupt.

Claim 1:
A network node comprising means for quality of service based throttling in a fabric architecture in which the network node is one of a plurality of interconnected network nodes of the fabric architecture, the network node comprising:
a processor (<NUM>, <NUM>, <NUM>, <NUM>);
a host fabric interface, HFI, (<NUM>) configured to facilitate the transmission of data between the plurality of interconnected network nodes over an interconnect fabric of the fabric architecture; and
one or more data storage devices (<NUM>) having stored therein a plurality of instructions that, when executed by the processor, cause the network node to:
monitor quality of service levels of the network node; and
detect a throttling condition based on a result of the monitored quality of service levels,
characterized in that the plurality of instructions cause the network node to:
generate, in response to having detected the throttling condition, a throttling message based on a request type associated with the throttling condition detected; and
transmit the generated throttling message to one or more of the plurality of interconnected network nodes communicatively coupled to the network node via the interconnect fabric.