Patent Description:
In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, one having ordinary skill in the art should recognize that the various implementations may be practiced without these specific details. In some instances, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the approaches described herein. For example, the dimensions of some of the elements may be exaggerated relative to other elements.

Various systems, apparatuses, and methods for routing interrupts on a coherency probe network are disclosed herein. In one implementation, a computing system includes at least a plurality of processing nodes, a coherency probe network, and one or more control units. The coherency probe network carries coherency probe messages between coherent agents. Interrupts that are detected by a control unit are converted into messages that are compatible with coherency probe messages and then routed to a target destination via the coherency probe network. Interrupts are generated with a first encoding while coherency probe messages have a second encoding. Cache subsystems determine whether a message received via the coherency probe network is an interrupt message or a coherency probe message based on an encoding embedded in the received message. Interrupt messages are routed to interrupt controller(s) while coherency probe messages are processed in accordance with a coherence probe action field embedded in the message.

Referring now to <FIG>, a block diagram of one implementation of a computing system <NUM> is shown. In one implementation, computing system <NUM> includes at least core complexes 105A-N, input/output (I/O) interfaces <NUM>, bus <NUM>, memory controller(s) <NUM>, and network interface <NUM>. In other implementations, computing system <NUM> includes other components and/or computing system <NUM> is arranged differently. In one implementation, each core complex 105A-N includes one or more general purpose processors, such as central processing units (CPUs). It is noted that a "core complex" is also referred to as a "processing node" or a "CPU" herein. In some implementations, one or more core complexes 105A-N include a data parallel processor with a highly parallel architecture. Examples of data parallel processors include graphics processing units (GPUs), digital signal processors (DSPs), and so forth. In various implementations, each processor core within core complex 105A-N includes an interrupt controller and a cache subsystem with one or more levels of caches. In one implementation, each core complex 105A-N includes a cache (e.g., level three (L3) cache) which is shared between multiple processor cores.

Memory controller(s) <NUM> are representative of any number and type of memory controllers accessible by core complexes 105A-N. Memory controller(s) <NUM> are coupled to any number and type of memory devices (not shown). For example, the type of memory in memory device(s) coupled to memory controller(s) <NUM> can include Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), NAND Flash memory, NOR flash memory, Ferroelectric Random Access Memory (FeRAM), or others. I/O interfaces <NUM> are representative of any number and type of I/O interfaces (e.g., peripheral component interconnect (PCI) bus, PCI-Extended (PCI-X), PCIE (PCI Express) bus, gigabit Ethernet (GBE) bus, universal serial bus (USB)). Various types of peripheral devices can be coupled to I/O interfaces <NUM>. Such peripheral devices may include displays, keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, and so forth.

In various implementations, computing system <NUM> is a computer, laptop, mobile device, game console, server, streaming device, wearable device, or any of various other types of computing systems or devices. It is noted that the number of components of computing system <NUM> varies from implementation to implementation. For example, in other implementations, there are more or fewer of each component than the number shown in <FIG>. It is also noted that in other implementations, computing system <NUM> includes other components not shown in <FIG>. Additionally, in other implementations, computing system <NUM> is structured in other ways than shown in <FIG>.

Turning now to <FIG>, a block diagram of another implementation of a computing system <NUM> is shown. In one implementation, system <NUM> includes control unit <NUM>, coherency probe network <NUM>, interrupt controller <NUM>, devices 225A-N, and nodes 230A-D. In one implementation, control unit <NUM> is located within a coherence unit. In other implementations, control unit <NUM> is part of any of various other types of components. Alternatively, in a further implementation, control unit <NUM> is a standalone component. Devices 225A-N are representative of any number and type of peripheral or input/output (I/O) devices connected to control unit <NUM> via interrupt controller <NUM>.

In one implementation, system <NUM> is a system on chip (SoC). In other implementations, system <NUM> is any of various other types of computing systems. Nodes 230A-D are representative of any number and type of processing nodes. Each node 230A-D includes any number of processor cores 245A-N, 250A-N, 255A-N, and 260A-N, respectively. Although four nodes 230A-D are shown in system <NUM> in <FIG>, this is shown merely for illustrative purposes. It should be understood that the number of nodes included in system <NUM> varies from implementation to implementation. In other implementations, system <NUM> includes other components and/or is organized in other suitable manners.

In one implementation, system <NUM> enforces a memory coherency protocol to ensure that a processor core or device does not concurrently access data that is being modified by another core or device. To comply with the memory coherency protocol, the cores and devices of system <NUM> transmit coherency messages (e.g., coherency probe message and probe responses) over coherency probe network <NUM>. Accordingly, coherency probe network <NUM> is designed to carry coherency probe message and probe responses between coherent agents of system <NUM>. A coherency probe message is a message that seeks the coherency state of data associated with a particular memory location. A probe response is typically sent back to the coherent agent that generated the coherency probe message. A probe response indicates the coherency state of the referenced data, transfers data in response to a probe, or provides other information in response to a probe. Typically, a coherency probe network <NUM> only carries coherency probe messages and probe responses. However, in system <NUM>, coherency probe network <NUM> also carries interrupts targeting one or more of cores 230A-D. This allows the interrupts to benefit from using a dedicated, low-latency network that spans multiple components within system <NUM> and is scalable to an arbitrary number of threads.

In various implementations, each device 225A-N is able to generate an interrupt by asserting an interrupt signal which is detected by interrupt controller <NUM>. In response to detecting the interrupt signal, interrupt controller <NUM> generates an interrupt message with information such as destination identifier, delivery mode, interrupt vector, or other suitable information. Interrupt controller <NUM> then conveys the interrupt message to control unit <NUM>. In one implementation, control unit <NUM> converts the interrupt message into a coherency probe message with a special encoding, and then control unit <NUM> conveys the specially encoded coherency probe message on coherency probe network <NUM> to one or more targets.

To facilitate the transfer of interrupts on coherency probe network <NUM>, control unit <NUM> includes logic for generating, receiving, processing, and forwarding interrupts. This logic also handles the normal processing of coherency probe messages. In one implementation, when control unit <NUM> detects or receives an interrupt, control unit <NUM> generates an interrupt message that is compatible with the format of a coherency probe message. Generating the interrupt message in a compatible format allows coherency probe network <NUM> to carry the interrupt message in a similar fashion to a coherency probe message. While the interrupt message is compatible with a coherency probe message, the interrupt message includes embedded encodings which allow other components to distinguish the interrupt message from a coherency probe message. After generating an interrupt message in a coherency-compatible format, control unit <NUM> conveys the interrupt message on coherency probe network <NUM> to one or more nodes 230A-D targeted by the interrupt. In one implementation, control unit <NUM> broadcasts the interrupt message on coherency probe network <NUM> to all nodes 230A-D. In another implementation, control unit <NUM> sends the interrupt message on coherency probe network <NUM> only to the node(s) targeted by the interrupt message.

In one implementation, coherency probe network <NUM> is connected to a cache subsystem 240A-D in each node 230A-D, respectively. Each cache subsystem 240A-D includes any number of cache levels. For example, in one implementation, each cache subsystem 240A-D includes a level three (L3) cache and a level two (L2) cache. In this implementation, each core includes a local level one (L1) cache. In other implementations, each cache subsystem 240A-D includes other cache levels. When a given cache subsystem 240A-D receives a message via coherency probe network <NUM>, the given cache subsystem 240A-D determines whether the message is an interrupt message or a coherency probe message. If the message is an interrupt message, the given cache subsystem 240A-D sends the interrupt message to the interrupt controller(s) within the corresponding core(s). As shown in system <NUM>, nodes 230A-D include interrupt controllers 247A-N, 252A-N, 257A-N, and/or 262A-N within cores 245A-N, 250A-N, 255A-N, and/or 260A-N, respectively. In one implementation, in response to receiving an interrupt message, a given cache subsystem 240A-D broadcasts the interrupt message to all of the cores in the corresponding node. In another implementation, in response to receiving an interrupt message, a given cache subsystem 240A-D sends the interrupt message only to those cores targeted by the interrupt message. The interrupt controller(s) in the core(s) will examine the interrupt message and generate interrupts to send to the targeted core(s).

Referring now to <FIG>, a block diagram of one implementation of a core complex <NUM> is shown. In one implementation, core complex <NUM> includes four processor cores 310A-D. In other implementations, core complex <NUM> includes other numbers of processor cores. It is noted that a "core complex" can also be referred to as a "processing node", "node", or "CPU" herein. In one implementation, the components of core complex <NUM> are included within core complexes 105A-N (of <FIG>).

Each processor core 310A-D includes a cache subsystem for storing data and instructions retrieved from the memory subsystem (not shown). For example, in one implementation, each core 310A-D includes a corresponding level one (L1) cache 315A-D. Each processor core 310A-D also includes or is coupled to a corresponding level two (L2) cache 320A-D. Additionally, in one implementation, core complex <NUM> includes a level three (L3) cache <NUM> which is shared by the processor cores 310A-D. It is noted that in other implementations, core complex <NUM> can include other types of cache subsystems with other numbers of caches and/or with other configurations of the different cache levels.

L3 cache <NUM> is coupled to a bus/fabric via coherency probe network <NUM>. L3 cache <NUM> receives both coherency probes and interrupt messages via coherency probe network <NUM>. L3 cache <NUM> forwards coherency probes and interrupt messages to L2 caches 320A-D. In one implementation, L3 cache <NUM> broadcasts received coherency probes and interrupt messages to all L2 caches 320A-D. In another implementation, L3 cache <NUM> forwards a received coherency probe or interrupt message to only those L2 caches 320A-D targeted by the probe or interrupt message. In this implementation, L3 cache <NUM> includes logic to examine coherency probes and interrupt messages to determine their targets. Upon receiving messages from L3 cache <NUM>, L2 caches 320A-D examine the messages to determine whether the messages are interrupts or coherency probes. The L2 caches 320A-D forward interrupt messages for processing to interrupt controllers 317A-D, respectively. The L2 caches 320A-D process coherency probes according to their embedded coherency probe action fields.

Turning now to <FIG>, examples of encoding coherency probe messages and interrupt messages in a hybrid message format are shown. Table <NUM> illustrates examples of the types of messages that can be sent using a hybrid message format. The leftmost column of table <NUM> indicates the message type <NUM>, with two different types of messages shown in table <NUM>: coherency probe message 410A and interrupt message 410B. In other implementations, other numbers of different types of messages are encoded in the hybrid message format. Using a hybrid message format allows interrupt message 410B to be formatted in a similar manner to coherency probe message 410A. Accordingly, the fields, or in some cases combinations of fields, of interrupt message 410B are aligned to match the fields of coherency probe message 410A. The hybrid message format includes any number of fields, with the number of fields varying from implementation to implementation. As shown in table <NUM>, the hybrid message format includes a coherency probe action field <NUM>, address field <NUM>, response field <NUM>, and any number of other fields.

The first entry of table <NUM> shows an example of a coherency probe message 410A. For coherency probe message 410A, field <NUM> is encoded with a coherency probe action indicator 415A. The coherency probe action indicator 415A can be set equal to any of various different values depending on the probe action type. For interrupt message 410B, field <NUM> is encoded with interrupt delivery indicator 415B to indicate that the message is an interrupt. In one implementation, control logic in a cache subsystem (e.g., cache subsystem 240A of <FIG>) looks at field <NUM> to determine if a received message is a coherency probe message or an interrupt message.

Field <NUM> specifies the address of a corresponding memory location being targeted by coherency probe message 415A. For interrupt message 410B, field <NUM> stores interrupt type indicator 420B in a first subset of bits and field <NUM> stores target indicator 420C in a second subset of bits. In other words, address field <NUM> is repurposed to hold both the interrupt type indicator 420B and the target indicator 420C of interrupt message 410B. This is possible since the combination of interrupt type indicator 420B and target indicator 420C is the same size as address field 420A. Interrupt type indicator 420B stores the type of interrupt that is being conveyed by interrupt message 410B and target field 420C specifies the target of interrupt message 410B.

Field <NUM> specifies the type of response that should be generated after processing the message. For coherency probe message 410A, field <NUM> is encoded with any of various response indicator 425A values specifying the type of response to send back to the source. For interrupt message 410B, response field <NUM> is encoded with a no response indicator 425B to indicate that no response needs to be sent back to the source. In other implementations, the hybrid message format includes other fields. For example, in another implementation, the hybrid message format includes an interrupt vector field to store the memory location of an interrupt handler. Other types of fields are possible and are contemplated for the hybrid message format.

Referring now to <FIG>, one implementation of a method <NUM> for generating messages to send over a coherency probe network is shown. For purposes of discussion, the steps in this implementation and those of <FIG> are shown in sequential order. However, it is noted that in various implementations of the described methods, one or more of the elements described are performed concurrently, in a different order than shown, or are omitted entirely. Other additional elements are also performed as desired. Any of the various systems or apparatuses described herein are configured to implement method <NUM>.

Control logic in a fabric interconnect receives a message in a hybrid message format (block <NUM>). In response to receiving the message in the hybrid message format, the control logic determines whether the message is a coherency probe message or an interrupt message (block <NUM>). One example of how to determine whether a message is a coherency probe message or an interrupt message is described in the discussion regarding method <NUM> of <FIG>. If the received message is an interrupt message (conditional block <NUM>, "yes" leg), then the control logic retrieves a target field from the interrupt message, wherein the target field is a subset of an address field of the hybrid message format (block <NUM>). In other words, if the address field is Y bits long, then the target field is X bits long, wherein X is less than Y, and wherein X and Y are both positive integers. An example of a target field being a subset of an address field is shown in table <NUM> of <FIG>. Next, the control unit routes, via a coherency probe network, the interrupt message to the device(s) specified in the target field (block <NUM>). If the received message is a coherency probe message (conditional block <NUM>, "no" leg), then the control logic retrieves an address field from the coherency probe message (block <NUM>). Next, the control logic forwards, via a coherency probe network, the coherency probe message to the device(s) corresponding to an address specified in the address field (block <NUM>). After blocks <NUM> and <NUM>, method <NUM> ends.

Turning now to <FIG>, one implementation of a method for determining whether a message is a coherency probe message or an interrupt message is shown. Control logic receives a message via a coherency probe network (block <NUM>). In response to receiving the message, the control logic retrieves a coherency probe action field from the received message (block <NUM>). If the coherency probe action field is encoded with an interrupt delivery indicator (conditional block <NUM>, "yes" leg), then the control logic treats the received message as an interrupt message (block <NUM>). If the coherency probe action field is encoded with a coherency probe action indicator (conditional block <NUM>, "no" leg), then the control logic treats the received message as a coherency probe message (block <NUM>). In other words, if the coherency probe action field of the message is encoded with any value other than the interrupt delivery indicator, then the control logic treats the received message as a coherency probe message. After blocks <NUM> and <NUM>, method <NUM> ends.

Referring now to <FIG>, one implementation of a method <NUM> for generating an interrupt message is shown. Control logic receives an interrupt (block <NUM>). Depending on the implementation, the control logic is located in a cache subsystem, coherence point, or other location within a computing system. In response to receiving the interrupt, the control logic generates an interrupt message that is compatible with a coherency probe message, wherein fields of the generated interrupt message are aligned with fields of the coherency probe message (block <NUM>). Then, the control logic forwards the interrupt message to a targeted destination via a coherency probe network (block <NUM>). After block <NUM>, method <NUM> ends.

Turning now to <FIG>, one implementation of a method <NUM> for processing a received message at a cache subsystem is shown. Control logic in a cache subsystem receives a message via a coherency probe network (block <NUM>). In one implementation, the control logic is part of a L2 cache. In other implementations, the control logic is located at other levels of the cache subsystem. In response to receiving the message, the control logic determines whether the message is a coherency probe message or an interrupt message (block <NUM>). One example of how to determine whether the message is a coherency probe message or an interrupt message is described in method <NUM> of <FIG>.

If the message is an interrupt message (conditional block <NUM>, "yes" leg), then the control logic retrieves a target field from the message (block <NUM>). Then the control logic routes the interrupt message to the interrupt controller(s) of the processor core(s) targeted by the interrupt (block <NUM>). Alternatively, in another implementation, the control logic broadcasts the interrupt message to the interrupt controllers of all processor cores in the node. If the message is a coherency probe message (conditional block <NUM>, "no" leg), then the control logic retrieves a coherency probe action field and an address field from the message (block <NUM>). Next, the control logic processes the coherency probe message in accordance with the probe action specified in the coherency probe action field (block <NUM>). After blocks <NUM> and <NUM>, method <NUM> ends.

In various implementations, program instructions of a software application are used to implement the methods and/or mechanisms described herein. For example, program instructions executable by a general or special purpose processor are contemplated. In various implementations, such program instructions are represented by a high level programming language. In other implementations, the program instructions are compiled from a high level programming language to a binary, intermediate, or other form. Alternatively, program instructions are written that describe the behavior or design of hardware. Such program instructions are represented by a high-level programming language, such as C. Alternatively, a hardware design language (HDL) such as Verilog is used. In various implementations, the program instructions are stored on any of a variety of non-transitory computer readable storage mediums. The storage medium is accessible by a computing system during use to provide the program instructions to the computing system for program execution. Generally speaking, such a computing system includes at least one or more memories and one or more processors configured to execute program instructions.

Claim 1:
A method comprising:
receiving (<NUM>), by a control unit (<NUM>), coherency probe messages and interrupts, the coherency probe messages being used to comply with a memory coherency protocol;
generating (<NUM>), by the control unit (<NUM>), an interrupt message in a hybrid message format that is compatible with a format of coherency probe messages used by the coherency probe network (<NUM>), responsive to detecting the interrupt, wherein an embedded encoding (<NUM>) of the hybrid message format distinguishes interrupt messages from coherency probe messages;
sending (<NUM>), by the control unit (<NUM>) via the coherency probe network (<NUM>), the interrupt message in the hybrid message format ; and
receiving (<NUM>), by a cache subsystem (<NUM>) via the coherency probe network (<NUM>), the interrupt message in the hybrid message format;
determining (<NUM>), by the cache subsystem (<NUM>), whether the received interrupt message in the hybrid message format is a coherency probe message or an interrupt message based on the embedded encoding; and
conveying (<NUM>) the interrupt message to a processor core (<NUM>), responsive to the interrupt message in the hybrid message format being determined to be an interrupt message.