Patent Description:
Accordingly, it may be understood that these statements are to be read in this light, and not as admissions of prior art.

Programmable logic devices are a class of integrated circuits that can be programmed to perform a wide variety of operations. A programmable logic device may include programmable logic elements programmed by a form of memory known as configuration random access memory (CRAM). Thus, to program a circuit design into a programmable logic device, the circuit design may be compiled into a bitstream and programmed into CRAM cells. The values programmed into the CRAM cells define the operation of programmable logic elements of the programmable logic device.

The highly flexible nature of programmable logic devices makes them an excellent fit for accelerating many computing tasks. Thus, programmable logic devices are increasingly used as accelerators for machine learning, video processing, voice recognition, image recognition, and many other highly specialized tasks, particularly those that would be too slow or inefficient in software running on a processor. Network on chip circuitry may be used by the programmable logic devices to communicate data throughout the programmable logic devices.

<CIT> discloses a stacked die network-on-chip (NoC) for FPGA. A programmable device system according to this document includes one or more NoC die layers vertically connected to one or more programmable chip dice layers. The NoC die layer includes interconnects, a bus or non-blocking switches, and optionally memory blocks and direct memory access engines. The NoC die layer improves on-chip communications by providing fast and direct interconnection circuitry between various parts of the programmable chip die.

<CIT> discloses a hybrid programmable logic device which includes a programmable field programmable gate array logic fabric and a many-core distributed processing subsystem. The device integrates both a fabric of programmable logic elements and processors in the same device, i.e., the same chip. The programmable logic elements may be sized and arranged such that place and route tools can address the processors and logic elements as a homogenous routing fabric. The programmable logic elements may provide hardware acceleration functions to the processors that can be defined after the device is fabricated. The device may include scheduling circuitry that can schedule the transmission of data on horizontal and vertical connectors in the logic fabric to transmit data between the programmable logic elements and processor in an asynchronous manner.

<CIT> discloses a method and system for hybrid and/or distributed implementation of generation and/or execution of power profile management instructions. An embodiment of this document provides a hardware element of a SoC/NoC that can be configured to generate and/or execute power profile management instructions using a hybrid combination of software and hardware, wherein the hardware element can be run in parallel with other hardware elements of the SoC/NoC to generate and execute power profile management instructions for different segments or regions of the SoC/NoC for efficient and safe working thereof.

<CIT> discloses methods and apparatus for managing application-specific power gating on multichip packages. A multichip package is provided that includes multiple integrated circuit (IC) dies mounted on a shared interposer. The IC dies may communicate with one another via corresponding input-output (IO) elements on the dies. The interposer may include a system-level power management block that is configured to coordinate low-power entry and exit for the IO elements based on customer application needs. Performing application-specific power gating, which may include a combination of coarse-grained and finegrained power gating control of the IO elements while the IO interface is sitting idle, can help maximize power savings in memory and a variety of other user applications.

<CIT> relates to computer architecture using rapidly reconfigurable circuits and high-bandwidth memory interfaces. A programmable device comprises one or more programming regions, each comprising a plurality of configurable logic blocks, where each of the plurality of configurable logic blocks is selectively connectable to any other configurable logic block via a programmable interconnect fabric. The programmable device further comprises configuration logic configured to, in response to an instruction in an instruction stream, reconfigure hardware in one or more of the configurable logic blocks in a programming region independently from any of the other programming regions.

The present invention as defined by the independent claims. Advantageous embodiments are described by the dependent claims.

It may be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another. Moreover, it may be appreciated that such a development effort might be complex and time consuming, but would nevertheless be a routine undertaking of design, fabrication, and manufacture for those of ordinary skill having the benefit of this disclosure.

Furthermore, the phrase A "based on" B is intended to mean that A is at least partially based on B. Moreover, unless expressly stated otherwise, the term "or" is intended to be inclusive (e.g., logical OR) and not exclusive (e.g., logical XOR). In other words, the phrase A "or" B is intended to mean A, B, or both A and B.

The highly flexible nature of programmable logic devices makes them an excellent fit for accelerating many computing tasks. Thus, programmable logic devices are increasingly used as accelerators for machine learning, video processing, voice recognition, image recognition, and many other highly specialized tasks, particularly those that would be too slow or inefficient in software running on a processor. In certain embodiments, as different sectors, portions, or regions of a programmable logic device are used to perform different operations, it may be useful to transfer data between the two sectors (e.g., regions). However, due to the limited space available on a fabric die that incorporates the programmable logic device, the performance impacts on the fabric die, the increased connectivity to the fabric die, and the like, it may be difficult to include a network-on-chip (NOC) circuit that facilitates the communication of data between various sectors, across different fabric die, between different programmable logic devices, and the like. For instance, placing the NOC in the fabric die interrupts the fabric die and reduces the performace capabilities of the fabric die.

With this in mind, in certain embodiments, the programmable logic device may be composed of at least two separate die. The programmable logic device includes a first die that contains primarily programmable logic fabric, and a second die that contains fabric support circuitry to support the operation of the programmable logic fabric. For example, the second die may contain at least some fabric support circuitry that may operate the programmable logic fabric (e.g., the fabric support circuitry of the second die may be essential to the operation of the programmable logic fabric of the first die).

In certain embodiments, a network on chip (NOC) is embedded on second die that includes the fabric support circuitry to facilitate data communication between sectors (e.g., regions, portions) on the second die, between sectors on the first die, between sectors on the first die and the second die, and the like. The NOC of the fabric support circuitry may thus be used to provide an additional avenue for data transfers across the fabric via another die (e.g., base die) that may be communicatively coupled to the fabric. By incorporating the NOC into the fabric support circuitry, the NOC may resolve periphery shoreline bandwidth issues of the fabric, while increasing the bandwidth of the fabric. In addition, the communication routes available via the NOC embedded in the fabric support circuitry may enable the fabric to implement design relocations or reconfigurations, provide alternate pathways around powered-down sectors of the fabric, provide security isolation features, and increase the speed in which the fabric may be configured (e.g., less than <NUM>). Each of these features may not be available without using the NOC in the fabric support circuitry because the above-referenced limitations of the fabric would prevent the fabric from including the infrastructure or network-on-chip to communicate data across the fabric in this manner.

In addition to the NOC, the fabric support circuitry, in certain embodiments, includes, among other things, a device controller (sometimes referred to as a secure device manager (SDM)), a sector controller (sometimes referred to as a local sector manager (LSM), region controller), a configuration network on chip (CNOC), data routing circuitry, local (e.g., sectorized, sector-aligned, region-aligned) memory used to store and/or cache configuration programs (bitstreams) or data, memory controllers used to program the programmable logic fabric, input/output (I/O) interfaces or modules for the programmable logic fabric, external memory interfaces (e.g., for a high bandwidth memory (HBM) device), an embedded processor (e.g., an embedded Intel® Xeon® processor by Intel Corporation of Santa Clara, California) or an interface to connect to a processor (e.g., an interface to an Intel® Xeon® processor by Intel Corporation of Santa Clara, California), voltage control circuitry, thermal monitoring circuitry, decoupling capacitors, power clamps, or electrostatic discharge circuitry, to name just a few circuit elements that may be present on the second die. With this in mind, by including the NOC in the fabric support circuitry, the first die may entirely or almost entirely contain programmable logic fabric, and the second die may contain all or almost all of the fabric support circuitry that controls the programmable logic fabric because the NOC may enable data to be accessible across the first die and the second die.

By way of introduction, <FIG> illustrates a block diagram of a system <NUM> that employs a programmable logic device <NUM> that communicates via a network-on-chip disposed on a separate die that does not include programmable logic fabric, in accordance with embodiments presented herein. Using the system <NUM>, a designer implements a circuit design functionality on an integrated circuit, such as a reconfigurable programmable logic device <NUM>, such as a field programmable gate array (FPGA). The designer may implement a circuit design to be programmed onto the programmable logic device <NUM> using design software <NUM>, such as a version of Intel® Quartus® by Intel Corporation of Santa Clara, California. The design software <NUM> may use a compiler <NUM> to generate a low-level circuit-design defined by bitstream <NUM>, sometimes known as a program object file and/or configuration program, that programs the programmable logic device <NUM>. Thus, the compiler <NUM> may provide machine-readable instructions representative of the circuit design to the programmable logic device <NUM>. For example, the programmable logic device <NUM> may receive one or more configuration programs (bitstreams) <NUM> that describe the hardware implementations that should be stored in the programmable logic device <NUM>. A configuration program (e.g., bitstream) <NUM> may be programmed into the programmable logic device <NUM> as a configuration program <NUM>. The configuration program <NUM> may, in some cases, represent an accelerator function to perform for machine learning, video processing, voice recognition, image recognition, or other highly specialized task.

To carry out the systems and methods of this disclosure, the programmable logic device <NUM> may take any suitable form that includes a network-on-chip (NOC) providing the ability to communicate data across the sectors of the programmable logic device <NUM>. As such, in one embodiment, the programmable logic device <NUM> has two separate integrated circuit die where at least some of the programmable logic fabric is separated from at least some of the fabric support circuitry that operates the programmable logic fabric, which may include the NOC.

One example of the programmable logic device <NUM> is shown in <FIG>, but any suitable programmable logic device may be used. In the example of <FIG>, the programmable logic device <NUM> includes a fabric die <NUM> and a base die <NUM> that are connected to one another via microbumps <NUM>. Although the fabric die <NUM> and base die <NUM> appear in a one-to-one relationship in <FIG>, other relationships may be used. For example, a single base die <NUM> may attach to several fabric die <NUM>, or several base die <NUM> may attach to a single fabric die <NUM>, or several base die <NUM> may attach to several fabric die <NUM> (e.g., in an interleaved pattern along the x- and/or y-direction). Peripheral circuitry <NUM> may be attached to, embedded within, and/or disposed on top of the base die <NUM>, and heat spreaders <NUM> may be used to reduce an accumulation of heat on the programmable logic device <NUM>. The heat spreaders <NUM> may appear above, as pictured, and/or below the package (e.g., as a double-sided heat sink). The base die <NUM> may attach to a package substrate <NUM> via C4 bumps <NUM>. In the example of <FIG>, two pairs of fabric die <NUM> and base die <NUM> are shown communicatively connected to one another via a silicon bridge <NUM> (e.g., an embedded multi-die interconnect bridge (EMIB)) and microbumps <NUM> at a silicon bridge interface <NUM>.

Although the microbumps <NUM> and the microbumps <NUM> are described as being employed between the fabric die <NUM> and the base die <NUM> or between the edge devices, such as the silicon bridge <NUM> and the silicon bridge interface <NUM>, it should be noted that microbumps may be employed at any suitable position between the components of the programmable logic device <NUM>. For example, the microbumps may be incorporated in any suitable position (e.g., middle, edge, diagonal) between the fabric die <NUM> and the base die <NUM>. In the same manner, the microbumps may be incorporated in any suitable pattern or amorphous shape to facilitate interconnectivity between various components (e.g., NOC) described herein.

In combination, the fabric die <NUM> and base die <NUM> operate as a programmable logic device such as a field programmable gate array (FPGA). For example, the fabric die <NUM> and the base die <NUM> may operate in combination as an FPGA <NUM>, shown in <FIG>. It should be understood that the FPGA <NUM> shown in <FIG> is meant to represent the type of circuitry and/or a logical arrangement of a programmable logic device when the both the fabric die <NUM> and the base die <NUM> operate in combination. In other words, some of the circuitry of the FPGA <NUM> shown in <FIG> may be found in the fabric die <NUM> and some of the circuitry of the FPGA <NUM> shown in <FIG> may be found in the base die <NUM>. Moreover, for the purposes of this example, the FPGA <NUM> is referred to as an FPGA, though it should be understood that the device may be any suitable type of programmable logic device (e.g., an application-specific integrated circuit and/or application-specific standard product).

In the example of <FIG>, the FPGA <NUM> may include transceiver circuitry (HSSI) <NUM> for driving signals off of the FPGA <NUM> and for receiving signals from other devices. The transceiver circuitry (HSSI) may be part of the fabric die <NUM>, the base die <NUM>, or a separate die altogether. Interconnection resources <NUM> may be used to route signals, such as clock or data signals, through the FPGA <NUM>. The FPGA <NUM> of <FIG> is shown to be sectorized, meaning that programmable logic resources may be distributed through a number of discrete programmable logic sectors <NUM> (e.g., region, portion). Each programmable logic sector <NUM> may include a number of programmable logic elements <NUM> (also referred herein as FPGA fabric <NUM>) having operations defined by configuration memory <NUM> (e.g., configuration random access memory (CRAM)). The programmable logic elements <NUM> may include combinational or sequential logic circuitry. For example, the programmable logic elements <NUM> may include look-up tables, registers, multiplexers, routing wires, and so forth. A designer may program the programmable logic elements <NUM> to perform a variety of desired functions. A power supply <NUM> may provide a source of voltage and current to a power distribution network (PDN) <NUM> that distributes electrical power to the various components of the FPGA <NUM>. Operating the circuitry of the FPGA <NUM> causes power to be drawn from the power distribution network <NUM>.

There may be any suitable number of programmable logic sectors <NUM> on the FPGA <NUM>. Indeed, while <NUM> programmable logic sectors <NUM> are shown here, it should be appreciated that more or fewer may appear in an actual implementation (e.g., in some cases, on the order of <NUM>, <NUM>, or <NUM> sectors or more). Each programmable logic sector <NUM> may include a sector controller (SC) <NUM> that controls the operation of the programmable logic sector <NUM>. Each sector controller <NUM> may be in communication with a device controller (DC) <NUM>. Each sector controller <NUM> may accept commands and data from the device controller <NUM> and may read data from and write data into its configuration memory <NUM> based on control signals from the device controller <NUM>. In addition to these operations, the sector controller <NUM> and/or device controller <NUM> may be augmented with numerous additional capabilities. Such capabilities may include coordinating memory transactions between local in-fabric memory (e.g., local fabric memory or CRAM being used for data storage) via the NOC, transactions between sector-aligned memory associated with that particular programmable logic sector <NUM> via the NOC, decrypting configuration data (bitstreams) <NUM>, and locally sequencing reads and writes to implement error detection and correction on the configuration memory <NUM>, and sequencing test control signals to effect various test modes.

The sector controllers <NUM> and the device controller <NUM> may be implemented as state machines and/or processors. For example, each operation of the sector controllers <NUM> or the device controller <NUM> may be implemented as a separate routine in a memory containing a control program. This control program memory may be fixed in a read-only memory (ROM) or stored in a writable memory, such as random-access memory (RAM). The ROM may have a size larger than would be used to store only one copy of each routine. This may allow each routine to have multiple variants depending on "modes" the local controller may be placed into. When the control program memory is implemented as random access memory (RAM), the RAM may be written with new routines to implement new operations and functionality into the programmable logic sectors <NUM>. This may provide usable extensibility in an efficient and easily understood way. This may be useful because new commands could bring about large amounts of local activity within the sector at the expense of only a small amount of communication between the device controller <NUM> and the sector controllers <NUM>.

Each sector controller <NUM> thus may communicate with the device controller <NUM>, which may coordinate the operations of the sector controllers <NUM> and convey commands initiated from outside the FPGA device <NUM>. To support this communication, the interconnection resources <NUM> may act as a network between the device controller <NUM> and each sector controller <NUM>. The interconnection resources may support a wide variety of signals between the device controller <NUM> and each sector controller <NUM>. In one example, these signals may be transmitted as communication packets.

The FPGA <NUM> may be electrically programmed. With electrical programming arrangements, the programmable elements <NUM> may include one or more logic elements (wires, gates, registers, etc.). For example, during programming, configuration data is loaded into the configuration memory <NUM> using pins <NUM> and input/output circuitry <NUM>. In one example, the configuration memory <NUM> may be implemented as configuration random-access-memory (CRAM) cells. The use of configuration memory <NUM> based on RAM technology is described herein is intended to be only one example. Moreover, configuration memory <NUM> may be distributed (e.g., as RAM cells) throughout the various programmable logic sectors <NUM> the FPGA <NUM>. The configuration memory <NUM> may provide a corresponding static control output signal that controls the state of an associated programmable logic element <NUM> or programmable component of the interconnection resources <NUM>. The output signals of the configuration memory <NUM> may configure the may be applied to the gates of metal-oxide-semiconductor (MOS) transistors that control the states of the programmable logic elements <NUM> or programmable components of the interconnection resources <NUM>.

As stated above, the logical arrangement of the FPGA <NUM> shown in <FIG> results from a combination of the fabric die <NUM> and base die <NUM>. The circuitry of the fabric die <NUM> and base die <NUM> may be divided in any suitable manner. In one example, shown in block diagram form in <FIG>, the fabric die <NUM> contains primarily programmable logic fabric resources, such as the programmable logic elements <NUM> and configuration memory <NUM>. In some cases, this may also entail certain fabric control circuitry such as the sector controller (SC) <NUM> or device controller (DC) <NUM>. The base die <NUM> may include supporting circuitry to operate the programmable logic elements <NUM> and configuration memory <NUM>. Shown here, the base die <NUM> includes sector <NUM> support circuitry 70A and sector <NUM> support circuitry 70B to support two corresponding sectors of the programmable logic elements <NUM> and configuration memory <NUM> of the fabric die <NUM>. The base die <NUM> may also include support circuitry for other sectors of the fabric die <NUM>.

Thus, while the fabric die <NUM> may include primarily programmable logic fabric resources, such as the programmable logic elements <NUM> and configuration memory <NUM>, the base die <NUM> includes, among other things, a device controller (DC) <NUM>, a sector controller (SC) <NUM>, a network-on-chip (NOC), a configuration network on chip (CNOC), data routing circuitry, sector-aligned memory used to store and/or cache configuration programs (bitstreams) or data, memory controllers used to program the programmable logic fabric, input/output (I/O) interfaces or modules for the programmable logic fabric, external memory interfaces (e.g., for a high bandwidth memory (HBM) device), an embedded processor (e.g., an embedded Intel® Xeon® processor by Intel Corporation of Santa Clara, California) or an interface to connect to a processor (e.g., an interface to an Intel® Xeon® processor by Intel Corporation of Santa Clara, California), voltage control circuitry, thermal monitoring circuitry, decoupling capacitors, power clamps, and/or electrostatic discharge (ESD) circuitry, to name just a few elements that may be present on the base die <NUM>. It should be understood that some of these elements that may be part of the fabric support circuitry of the base die <NUM> may additionally or alternatively be a part of the fabric die <NUM>. For example, the device controller (DC) <NUM> and/or the sector controllers (SC) <NUM> may be part of the fabric die <NUM>.

While <FIG> represents an example where the fabric die <NUM> contains primarily programmable logic fabric, with most other components located in the base die <NUM>, the fabric die <NUM> may contain some of the other components to support the programmable logic fabric. Thus, in some embodiments, the fabric die <NUM> or the base die <NUM> includes one or more of a device controller (DC) <NUM>, a sector controller (SC) <NUM>, a network-on-chip (NOC), a configuration network on chip (CNOC), data routing circuitry, sector-aligned memory used to store and/or cache configuration programs (bitstreams) or data, memory controllers used to program the programmable logic fabric, input/output (I/O) interfaces or modules for the programmable logic fabric, external memory interfaces (e.g., for a high bandwidth memory (HBM) device), an embedded processor (e.g., an embedded Intel® Xeon® processor by Intel Corporation of Santa Clara, California) or an interface to connect to a processor (e.g., an interface to an Intel® Xeon® processor by Intel Corporation of Santa Clara, California), voltage control circuitry, thermal monitoring circuitry, decoupling capacitors, power clamps, and/or electrostatic discharge (ESD) circuitry, and other elements.

One example physical arrangement of the fabric die <NUM> and the base die <NUM> is shown by <FIG> and <FIG>. In <FIG>, the fabric die <NUM> is shown to contain an array of fabric sectors <NUM> that include fabric resources <NUM> (e.g., programmable elements programmed by CRAM and/or certain fabric control circuitry such as the sector controller (SC) <NUM> or device controller (DC) <NUM>) and interface circuitry <NUM>. The interface circuitry <NUM> may include data routing and/or clocking resources or may include an interface to data routing and/or clocking resources on the base die <NUM>. Thus, the interface circuitry <NUM> may connect with a micro-bump (ubump) interface to connect to the base die <NUM>.

<FIG> provides an example complementary arrangement of the base die <NUM>. The base die <NUM> may represent an active interposer with several sectors <NUM> surrounded by peripheral circuitry <NUM> and the silicon bridge interface <NUM>. Although not shown in <FIG>, each sector <NUM> may include a variety of fabric support circuitry, which may described in greater detail below. In any case, the base die <NUM>, in some embodiments, may include data and/or configuration routers <NUM>, and/or data or configuration pathways <NUM>. In some embodiments, portions of the data or configuration pathways <NUM> may communicate data in one direction, while other portions may communicate data in the opposite direction. In other embodiments, the data or configuration pathways <NUM> may communicate data bi-directionally.

With the foregoing in mind, the data and/or configuration pathways <NUM> may make up a network on chip (NOC) system <NUM>. In the embodiment depicted in <FIG>, the NOC system <NUM> may be integrated between each sector <NUM> of the base die <NUM>. As such, the NOC system <NUM> may enable each of the sectors <NUM> disposed on the base die <NUM> to be accessible to each other. Indeed, the NOC system <NUM> may provide communication paths between each sector <NUM> via routers <NUM> or the like. In certain embodiments, the routers <NUM> may route user data between sectors <NUM> of the base die <NUM>, to sectors <NUM> of the fabric die <NUM>, and the like. Since the base die <NUM> is separate from the fabric die <NUM>, the NOC system <NUM> may be continuously powered on, even when various sectors <NUM> of the fabric die <NUM> are powered down. In this way, the NOC system <NUM> of the base die <NUM> may provide an available route to different sectors <NUM> of the fabric die <NUM> regardless of the positions of powered down sectors <NUM>.

In some embodiments, the NOC system <NUM> may include features such as Quality of Service management, Security Management, Debug and Performance measurement and Address virtualization services, and the like. In addition, the NOC system <NUM> may support caching features and interconnect protocols allowing the memory components of the programmable logic device <NUM> to be part of a coherent memory system supported by a caching agent.

By vertically aligning the fabric die <NUM> and the base die <NUM>, the NOC <NUM> disposed on the base die <NUM> may physically span across the same surface area of the fabric die <NUM>. In certain embodiments, microbumps may be positioned at various locations between the base die <NUM> and the fabric die <NUM> to enable the NOC <NUM> to communicate data between sectors <NUM> of the base die and sectors <NUM> of the fabric die <NUM>. In the example embodiment of the NOC <NUM> depicted in <FIG>, the NOC <NUM> may be positioned around each sector <NUM>, which may be aligned with a corresponding sector <NUM> of the fabric die <NUM>. As such, the NOC <NUM> may provide additional horizontal and vertical routing wires or pathways to facilitate to communication between sectors <NUM> of the fabric die <NUM>, between sectors <NUM> of the base die <NUM>, or between sectors <NUM> of the fabric die <NUM> and sectors <NUM> of the base die <NUM>. The additional horizontal and vertical lines provided by the NOC <NUM> may reduce the amount of quantization performed by the programmable logic device <NUM>.

Although the data or configuration pathways <NUM> of the NOC <NUM> is illustrated in <FIG> as being routed around the sectors <NUM> of the base die <NUM>, it should be noted that data or configuration pathways <NUM> of the NOC <NUM> may be routed across the base die <NUM> in any suitable manner. By way of example, <FIG> illustrate the NOC <NUM> implemented with data or configuration pathways <NUM> disposed across the center of the sector <NUM>. As such, the router <NUM> may also be positioned in the center of the sector <NUM>.

In addition, in some embodiments, the data or configuration pathways <NUM> of the NOC <NUM> may be wider in certain sectors <NUM> as compared to other sectors <NUM>. In any case, it should be understood that the data or configuration pathways <NUM> may be incorporated into the base die <NUM>, such that serve desired functions or operations that may be specific to the operations performed by the programmable logic device <NUM>. That is, if the programmable logic device <NUM> includes functions that involve regularly transferring data across the fabric die <NUM>, it may be beneficial to use more the base die <NUM> space to include data or configuration pathways <NUM> that take up more space on the base die <NUM> to provide increased bandwidth.

With the foregoing in mind, the NOC system <NUM> may include the data or configuration pathways <NUM> that allow for efficient multi-dimensional (e.g., three-dimensional, two-dimensional) integration of the NOC system <NUM> on the programmable logic device <NUM>. Indeed, the NOC system <NUM> may enable the fabric die <NUM> to connect to the peripheral circuitry <NUM> and other parts of the fabric without communicating via the fabric die <NUM> or the programmable logic sectors <NUM> of the fabric die <NUM>. That is, by employing the NOC system <NUM> in the base die <NUM>, the programmable logic device <NUM> may increase the available bandwidth for communication across the programmable logic sectors <NUM> because the NOC system <NUM> provides additional pathways to different parts of the programmable logic device <NUM>.

In addition, the NOC <NUM> resolves shoreline issues, supports fast reconfiguration, and enables relocate-ability of functions in the programmable logic sectors <NUM> based on the increased bandwidth and accessibility to different portions of the fabric die <NUM> via the base die <NUM>. That is, as shown in <FIG>, the NOC system <NUM> may be integrated between the sector-aligned memory <NUM> of the base die <NUM>, such that it spans across the entire base die <NUM>. As such, the NOC system <NUM> may access different fabric sectors <NUM> via various routes in the base die <NUM>. In addition, the additional routes enable the NOC system <NUM> to serve as functional bypass around powered down sectors <NUM> without affecting the performance of the programmable logic device <NUM> by avoiding blockages across the programmable logic sectors <NUM> of the programmable logic device <NUM>. That is, in some situations, certain sectors <NUM> of the fabric die <NUM> are powered down, thereby preventing communication across the powered down sectors <NUM>. In this case, the NOC system <NUM> may provide alternate communication pathways around the powered down sectors <NUM> to maintain communicative connectedness across the sectors <NUM> of the fabric die <NUM> regardless of whether certain sectors <NUM> are powered down.

The design relocate-ability of the programmable logic sectors <NUM> is also enhanced by the NOC system <NUM> because the NOC system <NUM> may access different sectors <NUM> of the fabric die <NUM>. That is, the increased communication flexibility provided by the NOC system <NUM> being disposed in the base die <NUM> enables the programmable logic sectors <NUM> to be repositioned in various sectors <NUM> of the fabric die <NUM>, while maintaining communication capabilities between the relocated programmable logic sectors <NUM>.

In certain embodiments, the NOC system <NUM> may also enable the programmable logic device <NUM> to provide security isolation for one or more of the programmable logic sectors <NUM>. That is, the NOC system <NUM> may be employed to communicate certain sensitive or secure data to a portion of programmable logic sectors <NUM> that may be designated as a security portion of the programmable logic device <NUM>. Third party programmers may be unable to access the security portion of the programmable logic device <NUM> without access to the NOC system <NUM>. Instead, the NOC system <NUM> may be limited to communication by certain programmers with a level of security credentials.

Although <FIG> and <FIG> illustrate three examples in which the NOC <NUM> may be configured, it should be noted that the base die <NUM> may be configured to include a data or configuration pathways <NUM> in a variety of shapes, forms, positions, and the like. For example, the data or configuration pathways <NUM> of different sectors <NUM> may overlap each other, the entire sector <NUM> may incorporate the data or configuration pathway <NUM>, or the like. In addition, microbumps may be used to facilitate communication between the NOC <NUM> and various sectors <NUM> of the fabric die <NUM> and the like.

In addition to facilitating communication of data between sectors <NUM>, sectors <NUM>, and the like, the presence of the NOC <NUM> in the base die <NUM> may also provide the programmable logic device <NUM> to incorporate additional circuit features by leveraging the NOC <NUM> of the base die <NUM> to improve the communication across the fabric die <NUM>. <FIG> provides another example of an arrangement of the base die <NUM>. Similar to the base die <NUM> described above, the base die <NUM> of <FIG> may represent an active interposer with several sectors <NUM> surrounded by peripheral circuitry <NUM> and the silicon bridge interface <NUM>. As shown in <FIG>, each sector <NUM> may include a variety of fabric support circuitry, such as sector-aligned memory <NUM>, memory control circuitry <NUM>, non-user input control circuitry <NUM>, non-user output control circuitry <NUM>, a voltage regulator such as a fully integrated voltage regulator (FIVR) <NUM>, one or more thermal sensors <NUM>, data and configuration routers <NUM>, and/or data or configuration pathways <NUM>.

Although the following description of the additional circuit features enabled by the NOC <NUM> embedded in the base die <NUM> focuses on the ability to transfer data to and from the sector-aligned memory <NUM>, it should be noted that the additional circuit features are not limited to technologies involving the sector-aligned memory <NUM>. Indeed, the NOC <NUM> embedded in the base die <NUM> may enable a variety of circuit operations to be performed more efficiently and effectively via the programmable logic device <NUM>. However, for the purposes of discussion, certain operations that involve the sector-aligned memory <NUM> and the NOC <NUM> will be described to illustrate some of the functions enabled by incorporating the NOC <NUM> into the base die <NUM>.

Referring back to <FIG>, the memory control circuitry <NUM> may be used to program the sector-aligned memory <NUM>, the CRAM of the fabric die <NUM>, or both. The non-user input control circuitry <NUM> and non-user output control circuitry <NUM> may allow the circuitry of the sectors <NUM> to exchange data and/or control signals (e.g., via configurable data routing network - on-chip (NOC) or a configuration network on chip (CNOC)). In one example, the non-user input control circuitry <NUM> and non-user output control circuitry <NUM> may operate as the sector controller (SC) <NUM> for a corresponding fabric sector <NUM> (as shown in <FIG>).

The FIVR <NUM> and the one or more thermal sensors <NUM> may be used to provide a desired voltage to the corresponding fabric sector <NUM> (as shown in <FIG>), enabling the voltage to be selectively scaled up or down, or removed, depending on power and thermal specifications (e.g., based at least in part on temperature as measured by a thermal sensor <NUM> and/or in accordance with a dynamic voltage and frequency scaling (DVFS) scheme). Even though the thermal sensors <NUM> are in a separate die from that of the programmable logic fabric elements, when the base die <NUM> is directly adjacent to the fabric die <NUM> as in this example, the temperature measured by the thermal sensor <NUM> in the base die <NUM> may correspond well enough to the fabric die <NUM> to allow for temperature-based operations (e.g., turn off power to the corresponding fabric sector <NUM> to prevent a permanent-denial-of-service (PDOS) condition).

In certain embodiments, the data or configuration pathways <NUM> that make up the NOC system <NUM> may provide communication paths between each sector <NUM> via routers <NUM> mentioned above. As shown in <FIG>, by vertically aligning the fabric die <NUM> and the base die <NUM> and incorporating the NOC <NUM> in the base die <NUM>, memory located in the base die <NUM> may be accessible in parallel to fabric sectors <NUM> of the fabric die <NUM>. <FIG> shows an example in which sector-aligned memory <NUM> is contained in the base die <NUM>. The sector-aligned memory <NUM> is directly accessible to respective fabric sectors <NUM> of the fabric die <NUM> and may contain user data (generated by or accessible by a circuit design programmed into the programmable logic fabric of the base die <NUM>) or configuration data that may be used to program the programmable logic fabric of the base die <NUM>. In this disclosure, "directly accessible" refers to a connection between a region of the sector-aligned memory <NUM> that is associated with a particular fabric sector <NUM> and that particular fabric sector <NUM>. In some embodiments, each respective region of the sector-aligned memory <NUM> associated with a particular fabric sector <NUM> may be directly accessible to that particular fabric sector <NUM>, thereby providing each fabric sector <NUM> with direct access to that region of the sector-aligned memory <NUM>. For example, there may be N regions of sector-aligned memory <NUM> that can be accessible by N corresponding fabric sectors <NUM> at the same time (e.g., in parallel). In some cases, the sector-aligned memory <NUM> may be accessible to more than one fabric sector <NUM> or multiple sectors of sector-aligned memory <NUM> may be accessible to a single fabric sector <NUM>. Thus, in some cases, the same region of sector-aligned memory <NUM> may be directly accessible to multiple fabric sectors <NUM>, while in other cases, a region of sector-aligned memory <NUM> may be directly accessible only to a single fabric sector <NUM>. In the example of <FIG>, the fabric sectors <NUM> may access specific regions of sector-aligned memory <NUM>. The sector-aligned memory <NUM> is shown in <FIG> as vertically stacked memory. This may allow a large amount of memory to be located within the base die <NUM>. However, the sector-aligned memory <NUM> may occupy a single plane of the base die <NUM> in some embodiments.

As mentioned above, to facilitate the use of the sector-aligned memory <NUM>, the embedded NOC <NUM> may enable configuration data and user data to be communicated between memory components of the sector-aligned memory <NUM>, between the sectors <NUM> or other components (e.g., CRAM) of the fabric die <NUM> and a component in the sector-aligned memory <NUM>, and the like. Additional details with regard to how the NOC <NUM> may communicate with the CRAM of the fabric die <NUM>, memory components of the fabric die <NUM>, sectors <NUM> of the fabric <NUM>, and the like will be provided below with reference to <FIG>.

Referring again to <FIG>, to facilitate communication between the fabric die <NUM> and the base die <NUM>, the NOC system <NUM> may employ the memory control circuitry <NUM> to control ingress and egress of data between the NOC system <NUM> to sector-aligned memory <NUM> on the base <NUM>. That is, the memory control circuitry <NUM> may enables DMA and other memory operations, such as data operations onto and off of the NOC system <NUM>, as well as providing access to sector-aligned memory <NUM> via the microbumps <NUM>.

Ingress and egress operations from the NOC system <NUM> to and from the microbumps <NUM> is designed to support mapping connections via the NOC system <NUM> through the microbumps <NUM> to a variety of destinations. For example, direct parallel or serial connection to the sector input data register of each sector <NUM> (e.g., part of non-user input control circuitry <NUM>) may be accessible to the NOC system <NUM> via the microbumps <NUM>. In addition, H/V wires of the programmable logic device <NUM> with a hardware programmable protocol adapter in a ubump interface in the base die <NUM> that may map wires coming through the microbumps <NUM> to a user selectable level of protocol including, but not limited to, AXI-<NUM> and Avalon MM. Further, direct parallel or serial connections to the sector output data register (e.g., part of non-user input control circuitry <NUM>) may be employed for trace, debug, and test data.

In some embodiments, the NOC system <NUM> may be split into different portions that facilitate user data and other portions that facilitate non-user data. The user data may be routed to the peripheral circuitry <NUM> and other programmable logic sectors <NUM> of the fabric die <NUM> at approximately <NUM> Gbps. The non-user data communicated via the other portion of the NOC system <NUM> may be used for fast configuration operations from high bandwidth/medium (HBM), double data rate (DDR), and Xeon components at approximately <NUM> Gbps.

The NOC system <NUM> may connect to the peripheral circuitry <NUM> via adapters designed to support connections to the peripheral circuitry <NUM>. For example, a serial periphery circuit, such as, but not limited to, PCIe and Ethernet, may connects into the NOC system <NUM> via destination decoders, which examine a packet and determine which stop on the NOC system <NUM> is the data to be sent to. Other types of the peripheral circuitry <NUM> may include memory or other protocol interfaces through stops designed to adapt the NOC transactions to a periphery protocol.

In some embodiments, the programmable logic device <NUM> may include circuit designs that define an accelerator function may benefit from access to a large amount of data stored in memory. Since accessing an external memory device may be a relatively slow process, and the capacity of memory in the fabric die <NUM> may be limited, the NOC <NUM> may enable the fabric die <NUM> to access sector-aligned memory <NUM> that is neither directly within the programmable fabric nor external to the programmable logic device. In other words, the embedded NOC <NUM> of the base die <NUM> provides the ability for data to be accessible to the sector-aligned memory <NUM> and to different sectors <NUM> of the fabric die <NUM>.

In some embodiments, since the sector-aligned memory <NUM> is located on a separate die from the fabric <NUM>, the sector-aligned memory <NUM> may have a much larger capacity than a capacity of local in-fabric memory. Indeed, in some cases, the sector-aligned memory <NUM> may have a capacity of one-thousand times or greater than the capacity of local in-fabric memory.

The sector-aligned memory <NUM> may not only have a higher capacity than local in-fabric memory, but the sector-aligned memory <NUM> may also have a higher bandwidth than an external memory device. The high bandwidth may be made possible by physically locating the sector-aligned memory <NUM> near to the fabric die <NUM> (e.g., in the base die v24 vertically aligned with the fabric die <NUM>) and/or by physically or logically dividing the sector-aligned memory <NUM> into separate sectors <NUM> that may transfer data in parallel to corresponding different sectors of the programmable logic fabric. This may also allow the sector-aligned memory <NUM> to be secured from access by other sectors <NUM> of the fabric die <NUM>. Furthermore, depending on the physical arrangement of the fabric die <NUM> that contains the programmable logic sectors <NUM> and the base die <NUM> that contains the sector-aligned memory <NUM>, the sector-aligned memory <NUM> may be pipelined into the programmable logic sectors <NUM>, allowing for even faster data utilization.

With this in mind, <FIG> illustrates one embodiment in which the sector-aligned memory <NUM> is made accessible to the fabric die <NUM> via the NOC <NUM>. Referring now to <FIG>, the sector-aligned memory <NUM> of the base die <NUM> may be accessible by the programmable logic fabric (e.g., programmable logic elements <NUM> and associated configuration memory <NUM>) of the fabric die <NUM> via a memory interface (I/F) <NUM>. There may be one or more memory interfaces (I/F) <NUM> for each fabric sector, allowing different fabric sectors to access their respective sectors of the sector-aligned memory <NUM> in parallel. The memory interface (I/F) <NUM> may occupy a row of a fabric sector <NUM> and may be made of hardened logic or soft logic, or both. In the example of <FIG>, the memory interface (I/F) <NUM> may occupy an outermost row of a fabric sector <NUM>. This may allow the memory interface (I/F) <NUM> to facilitate communication not just with rows of programmable logic elements <NUM> and associated configuration memory <NUM> in the fabric sector <NUM> where the memory interface (I/F) <NUM> is located, but also with rows of programmable logic elements <NUM> and associated configuration memory <NUM> in an adjacent fabric sector <NUM>.

The memory interface (I/F) <NUM> may receive or transmit data via a data path <NUM> to a memory data interface (I/F) <NUM> and may communicate control signals via a control signal path <NUM> to and from a memory control interface (I/F) <NUM>. The memory interface (I/F) <NUM> may receive control and/or data signals and route them through the rows of programmable logic elements <NUM> and associated configuration memory <NUM> to a particular memory address or logic element via routing circuitry <NUM>. The control signal path <NUM> and the data path <NUM> may represent a first physical connection between a first sector of programmable logic fabric and a first sector of the sector-aligned memory <NUM>. It should be understood that similar pathways may represent a second physical connection between a second sector of programmable logic fabric and a second sector of the sector-aligned memory <NUM>.

Regardless of its exact placement, the sector-aligned memory <NUM> may be located near enough to a particular area of the programmable logic fabric of the programmable logic device <NUM> to be able to provide very rapid data transfers. This may enable the sector-aligned memory <NUM> to be used for caching of data and/or configuration programs that may be programmed into the programmable logic fabric. One example of circuitry <NUM> that may use the sector-aligned memory <NUM> appears in <FIG>. The various components shown in <FIG> may be located in a single die or may be distributed through several die (e.g., distributed through the fabric die <NUM> or the base die <NUM>). Indeed, when programmable logic device <NUM> includes the fabric die <NUM> and the base die <NUM>, each element of circuitry <NUM> represented by the block diagram of <FIG> may be found in at least one of the fabric die <NUM> and the base die <NUM>, as desired. In many situations, however, the sector-aligned memory <NUM> may have a sufficiently high capacity that it may not fit in the fabric die <NUM>, and thus may be located in the base die <NUM>.

The circuitry <NUM> shown in <FIG> includes the device controller (DC) <NUM> that may receive, among other things, a bitstream <NUM>. The bitstream <NUM> may represent data that may be received by the device controller (DC) <NUM>, such as configuration data that may program the configuration memory (CRAM) <NUM> of a particular sector of programmable logic elements (FPGA fabric) <NUM> and/or data that may be processed by the programmable logic fabric (e.g., in a request to accelerate a compute task). The device controller <NUM> may receive the bitstream <NUM> from an external data source, such as an external data storage device or external memory device and may direct the bitstream <NUM> to the sector controller (SC) <NUM> of the particular sector via a configuration network on chip (CNOC) <NUM> or any other suitable pathway.

When the circuitry <NUM> of <FIG> is used for configuring the programmable logic device elements <NUM> by programming the configuration memory (CRAM) <NUM>, routing circuitry <NUM> (e.g., a multiplexer) may provide the bitstream <NUM> to the sector controller (SC) <NUM> via a main signal path <NUM>. When directed by the bitstream <NUM>, when determined by a routine running on the sector controller (SC) <NUM>, and/or when directed by the circuit design programmed into the programmable logic elements <NUM>, the sector controller (SC) <NUM> may issue a selection signal over a selection pathway <NUM> to direct the routing circuitry <NUM> to receive the bitstream <NUM> from the CNOC <NUM> or to receive data from the sector-aligned memory <NUM>, and/or whether to cache the bitstream <NUM> into the sector-aligned memory <NUM>. Based on the selection signal on the selection pathway <NUM>, the routing circuitry <NUM> may provide either data on the CNOC <NUM> or on a data pathway <NUM> from the sector-aligned memory <NUM> to the sector controller (SC) <NUM>. A control pathway <NUM> may enable control communication between the sector controller (SC) <NUM> and the sector-aligned memory <NUM>. The sector controller (SC) <NUM> may use the control pathway <NUM> to cause the sector-aligned memory <NUM> to retrieve data from or store data into the sector-aligned memory <NUM>.

A configuration program <NUM> implemented in the programmable logic fabric, as defined by the configuration of programmable logic elements <NUM> programmed by the configuration memory (CRAM) <NUM>, may utilize the sector-aligned memory <NUM>. The configuration program <NUM> programmed into the programmable logic elements <NUM> may do so in several ways. In one example, the configuration program <NUM> may directly access (e.g., read from or write to) the sector-aligned memory via the control pathway <NUM> coupled to the control interface (CTRL I/F) <NUM> and the data pathway <NUM> coupled to the data interface (DATA I/F) <NUM> for direct data transfers between the programmable logic fabric and the sector-aligned memory <NUM>. The configuration program <NUM> may include a memory controller for the sector-aligned memory <NUM> implemented in the programmable logic elements <NUM>, which may be referred to as a memory controller implemented in soft logic, or a hardened memory controller may be accessible to the control interface (CTRL I/F) <NUM> and the data interface (DATA I/F) <NUM>. In another example, the configuration program <NUM> may communicate control signals to the sector controller (SC) <NUM> via the control pathway <NUM> instructing the sector controller (SC) <NUM> to coordinate a data transfer to or from the sector-aligned memory <NUM>. The configuration program <NUM> thus may include a soft processor or soft state machine to communicate with the sector controller (SC) <NUM> in this way, or hardened control circuitry may be disposed among the programmable logic elements <NUM> to communicate with the sector controller (SC) <NUM>.

A memory address register / data register (AR/DR) <NUM> may program the configuration memory (CRAM) <NUM> and/or in-fabric memory <NUM> based on instructions from the sector controller (SC) <NUM> on a control pathway <NUM> and using data received on a data pathway <NUM>. In this way, the AR/DR <NUM> may rapidly program the CRAM <NUM> and/or in-fabric memory <NUM> with data, such as data from the bitstream <NUM> received on the CNOC <NUM> or directly from sector-aligned memory <NUM> when so instructed. This may take place much more quickly than the time involved in receiving the entire bitstream <NUM> via the CNOC <NUM>, which may face latencies due to accessing a memory device external to the programmable logic device <NUM>. In some cases, this may be <NUM>% faster, twice as fast, 5x as fast, 10x as fast, 20x as fast, 50x as fast, 100x as fast, 200x as fast, 500x as fast, 1000x as fast, or faster, to program the CRAM <NUM> and/or in-fabric memory <NUM> with data directly from the sector-aligned memory <NUM> than to program the CRAM <NUM> and/or in-fabric memory <NUM> with the bitstream <NUM> from the CNOC <NUM>. Here, it may also be noted that the amount of memory available in the in-fabric memory <NUM> may be much smaller than the amount of memory available in the sector-aligned memory <NUM>. In fact, the sector-aligned memory <NUM> may have a capacity many times that of the in-fabric memory <NUM> (e.g., 10x, 100x, 1000x, or more).

For even faster programming, the programming of the CRAM <NUM> and/or in-fabric memory <NUM> may be pipelined, as shown in <FIG>. A memory manager <NUM> may coordinate control of the AR/DR <NUM> via control pathways <NUM> and <NUM>. The memory manager <NUM> may be located in the fabric die <NUM> and/or in the base die <NUM>. The memory manager <NUM> may be implemented as a state machine and/or as a processor running software or firmware and may control the data transfers to and/or from the sector-aligned memory <NUM> and the AR/DR <NUM> over a data pathway <NUM>. The data pathway <NUM> may communicate data more rapidly than may be provided over the CNOC <NUM>. The data pathway <NUM> may have a faster frequency and/or may carry data more widely, in parallel, than the CNOC <NUM>.

The sector controller (SC) <NUM> may coordinate with the AR/DR <NUM> and the memory manager <NUM> to receive the bitstream <NUM> via a data pathway <NUM> from the CNOC <NUM> or from the sector-aligned memory <NUM>. As mentioned above, the sector controller (SC) <NUM> may control whether to receive data of the bitstream <NUM> from the CNOC <NUM> or to get it from the sector-aligned memory <NUM>, and/or whether to cache or pre-cache (e.g., in a cache prefetch) the bitstream <NUM> into the sector-aligned memory <NUM>.

As with the circuitry <NUM> of <FIG>, in the circuitry <NUM> of <FIG>, a configuration program <NUM> implemented in the programmable logic fabric, as defined by the configuration of programmable logic elements <NUM> programmed by the configuration memory (CRAM) <NUM>, may utilize the sector-aligned memory <NUM>. The configuration program <NUM> programmed into the programmable logic elements <NUM> may do so in several ways. In one example, the configuration program <NUM> may directly access (e.g., read from or write to) the sector-aligned memory via the control pathway <NUM> coupled to the control interface (CTRL I/F) <NUM> and the data pathway <NUM> coupled to the data interface (DATA I/F) <NUM> for direct data transfers between the programmable logic fabric and the sector-aligned memory <NUM>. The configuration program <NUM> may include a memory controller for the sector-aligned memory <NUM> implemented in the programmable logic elements <NUM>, which may be referred to as a memory controller implemented in soft logic, or a hardened memory controller may be accessible to the control interface (CTRL I/F) <NUM> and the data interface (DATA I/F) <NUM>. In another example, the configuration program <NUM> may communicate control signals to the sector controller (SC) <NUM> and/or the memory manager <NUM> via the control pathway <NUM>. The configuration program <NUM> may instruct the sector controller (SC) <NUM> and/or the memory manager <NUM> to coordinate a data transfer to or from the sector-aligned memory <NUM>. The configuration program <NUM> thus may include a soft processor or soft state machine to communicate with the sector controller (SC) <NUM> in this way, or hardened control circuitry may be disposed among the programmable logic elements <NUM> to communicate with the sector controller (SC) <NUM> and/or the memory manager <NUM>.

Data from the CNOC <NUM> or the sector-aligned memory <NUM> may be loaded into the AR/DR <NUM> and pipelined into the CRAM <NUM> and/or in-fabric memory <NUM> via pipelining circuitry <NUM>. The pipelining circuitry <NUM> may allow multiple cells of the configuration memory (CRAM) <NUM> to be programmed at once by pipelining multiple bits of data into registers of the AR/DR <NUM> before the AR/DR <NUM> programs multiple cells of the configuration memory (CRAM) <NUM> at once (e.g., instead of programming the configuration memory (CRAM) <NUM> one cell at a time). This may allow large quantities of data from the sector-aligned memory <NUM> to rapidly enter the CRAM <NUM> cells or the in-fabric memory <NUM>. As noted above, this may take place much more quickly than the time involved in receiving the entire bitstream <NUM> via the CNOC <NUM>. In some cases, it may be <NUM>% faster, twice as fast, 5x as fast, 10x as fast, 20x as fast, 50x as fast, 100x as fast, 200x as fast, 500x as fast, 1000x as fast, or faster, to program the CRAM <NUM> and/or in-fabric memory <NUM> with bitstream <NUM> directly from sector-aligned memory <NUM> than to program the CRAM <NUM> and/or in-fabric memory <NUM> with the bitstream <NUM> from the CNOC <NUM>.

In addition to transferring data to or from the sector-aligned memory <NUM>, the circuitry <NUM> and the circuitry <NUM> depicted in <FIG> and <FIG> may also be employed to all the FPGA fabric <NUM> to communicate directly with the NOC <NUM>, which may be embedded in the sector-aligned memory <NUM>. That is, the same interface used to communicate configuration and user data in the circuitry <NUM> of <FIG> may be used to enable direct communication between the FPGA fabric <NUM> and the NOC <NUM>. In another embodiment, the sector controller (SC) <NUM> of the circuitry <NUM> in <FIG> may coordinate the communication of data between the FPGA fabric <NUM> and the NOC <NUM>. In the same manner, the circuitry <NUM> of <FIG> may be employed to allow the FPGA fabric <NUM> to communicate directly with the NOC <NUM> over the same interface used for transferring configuration data, as described above, using a direct FPGA fabric <NUM> to sector-aligned memory <NUM>/NOC <NUM> connection or using a coordinated effort between the sector controller (SC) <NUM> and or the memory manager <NUM>.

With the foregoing in mind, <FIG> illustrates a communication scheme <NUM> that may be employed by the circuitry <NUM> of <FIG> to communicate data from the CRAM <NUM> to the NOC <NUM>, in accordance with embodiments presented herein. In addition, <FIG> illustrates a flow chart of a method <NUM> for transferring data from the CRAM <NUM> to the NOC <NUM> using the communication scheme <NUM> of <FIG>.

Keeping both of these figures in mind, in one embodiment, at block <NUM>, the control interface (CTRL I/F) <NUM> may issue a control signal on the control pathway <NUM> to the sector controller (SC) <NUM>. The control signal may include a request for a data transfer from the CRAM <NUM> to the NOC <NUM>. In one embodiment, the request may include an address range specified by the FPGA fabric <NUM>. After receiving the request for the data transfer from the CRAM <NUM> to the NOC <NUM>, at block <NUM>, the sector controller (SC) <NUM> may cause desired contents of the CRAM <NUM> at the specified address range to be transferred to the NOC <NUM> via the data register <NUM> and the data pathway <NUM>.

In addition to directing the data transfer from the CRAM <NUM> to the NOC <NUM>, the circuit <NUM> may be employed to transfer data from the NOC <NUM> to the CRAM <NUM> in accordance with the communication scheme <NUM> depicted in <FIG>. By way of example, <FIG> illustrates a flow chart of a method <NUM> for transferring data from the NOC <NUM> to the CRAM <NUM> using the communication scheme <NUM>.

Referring now to <FIG>, at block <NUM>, the control interface (CTRL I/F) <NUM> may issue a control signal on the control pathway <NUM> to the sector controller (SC) <NUM> for a data transfer from the NOC <NUM> to the CRAM <NUM>. After receiving the request for the data transfer from the CRAM <NUM> to the NOC <NUM>, at block <NUM>, the sector controller (SC) <NUM> may cause desired contents of the NOC <NUM> to be transferred to the CRAM <NUM> via the data pathway <NUM>.

As discussed above, the FPGA fabric <NUM> may also communicate directly with the NOC <NUM> using the circuitry <NUM> of <FIG>. As such, in some embodiments, the sector controller (SC) <NUM> may coordinate the transfer of data between the FPGA fabric <NUM> and the NOC <NUM> by delegating the transfer to the memory manager <NUM>. With this in mind, <FIG> illustrates a communication scheme <NUM> that may be employed by the circuitry <NUM> of <FIG> to communicate data from the CRAM <NUM> to the NOC <NUM>, in accordance with embodiments presented herein. In addition, <FIG> illustrates a flow chart of a method <NUM> for transferring data from the CRAM <NUM> to the NOC <NUM> using the communication scheme <NUM> of <FIG>.

Keeping both of these figures in mind, in one embodiment, at block <NUM>, the control interface (CTRL I/F) <NUM> may issue a control signal on the control pathway <NUM> to the sector controller (SC) <NUM>. The control signal may include a request for a data transfer from the CRAM <NUM> to the NOC <NUM>. In one embodiment, the request may include an address range specified by the FPGA fabric <NUM>. After receiving the request for the data transfer from the CRAM <NUM> to the NOC <NUM>, at block <NUM>, the sector controller (SC) <NUM> may delegate the transfer request to the memory manager <NUM>. That is, the sector controller (SC) <NUM> may relay the request to the memory manager <NUM>. At block <NUM>, the memory manager <NUM> may cause desired contents of the CRAM <NUM> at the specified address range to be transferred to the NOC <NUM> via the data register <NUM> and the data pathway <NUM>, as shown in the communication scheme <NUM> of <FIG>.

Referring now to <FIG>, at block <NUM>, the control interface (CTRL I/F) <NUM> may issue a control signal on the control pathway <NUM> to the sector controller (SC) <NUM>. The control signal may include a request for a data transfer from the NOC <NUM> to the CRAM <NUM>. After receiving the request for the data transfer from the NOC <NUM> to the CRAM <NUM>, at block <NUM>, the sector controller (SC) <NUM> may delegate the transfer request to the memory manager <NUM>. That is, the sector controller (SC) <NUM> may relay the request to the memory manager <NUM>. At block <NUM>, the memory manager <NUM> may cause desired contents of the NOC <NUM> to be transferred to the CRAM <NUM> via the data register <NUM> and the data pathway <NUM>, as shown in the communication scheme <NUM> of <FIG>.

In addition to transferring data between the NOC <NUM> and the CRAM <NUM>, the circuitry <NUM> and the circuitry <NUM> depicted in <FIG> and <FIG> may also be employed to facilitate communication between the in-fabric memory <NUM> and the NOC <NUM>. For example, <FIG> illustrates a communication scheme <NUM> that may be employed by the circuitry <NUM> of <FIG> to communicate data from the in-fabric memory <NUM> to the NOC <NUM>, in accordance with embodiments presented herein. In addition, <FIG> illustrates a flow chart of a method <NUM> for transferring data from the in-fabric memory <NUM> to the NOC <NUM> using the communication scheme <NUM> of <FIG>.

Referring to <FIG>, in one embodiment, at block <NUM>, the control interface (CTRL I/F) <NUM> may issue a control signal on the control pathway <NUM> to the sector controller (SC) <NUM>. The control signal may include a request for a data transfer from the in-fabric memory <NUM> to the NOC <NUM>. In one embodiment, the request may include an address range specified by the FPGA fabric <NUM>. After receiving the request for the data transfer from the in-fabric memory <NUM> to the NOC <NUM>, at block <NUM>, the sector controller (SC) <NUM> may cause desired contents of the in-fabric memory <NUM> at the specified address range to be transferred to the NOC <NUM> via the data register <NUM> and the data pathway <NUM>.

In addition to directing the data transfer from the in-fabric memory <NUM> to the NOC <NUM>, the circuit <NUM> may be employed to transfer data from the NOC <NUM> to the in-fabric memory <NUM> in accordance with the communication scheme <NUM> depicted in <FIG>. By way of example, <FIG> illustrates a flow chart of a method <NUM> for transferring data from the NOC <NUM> to the in-fabric memory <NUM> using the communication scheme <NUM>.

Referring now to <FIG>, at block <NUM>, the control interface (CTRL I/F) <NUM> may issue a control signal on the control pathway <NUM> to the sector controller (SC) <NUM> for a data transfer from the NOC <NUM> to the in-fabric memory <NUM>. After receiving the request for the data transfer from the in-fabric memory <NUM> to the NOC <NUM>, at block <NUM>, the sector controller (SC) <NUM> may cause desired contents of the NOC <NUM> to be transferred to the in-fabric memory <NUM> via the data register <NUM> and the data pathway <NUM>.

In certain embodiments, the sector controller (SC) <NUM> may coordinate the transfer of data between the in-fabric memory <NUM> and the NOC <NUM> by delegating the transfer to the memory manager <NUM> using the circuitry <NUM> of <FIG>. With this in mind, <FIG> illustrates a communication scheme <NUM> that may be employed by the circuitry <NUM> of <FIG> to communicate data from the in-fabric memory <NUM> to the NOC <NUM>, in accordance with embodiments presented herein. In addition, <FIG> illustrates a flow chart of a method <NUM> for transferring data from the in-fabric memory <NUM> to the NOC <NUM> using the communication scheme <NUM> of <FIG>.

Referring now to <FIG>, in one embodiment, at block <NUM>, the control interface (CTRL I/F) <NUM> may issue a control signal on the control pathway <NUM> to the sector controller (SC) <NUM>. The control signal may include a request for a data transfer from the in-fabric memory <NUM> to the NOC <NUM>. In one embodiment, the request may include an address range specified by the FPGA fabric <NUM>. After receiving the request for the data transfer from the in-fabric memory <NUM> to the NOC <NUM>, at block <NUM>, the sector controller (SC) <NUM> may delegate the transfer request to the memory manager <NUM>. That is, the sector controller (SC) <NUM> may relay the request to the memory manager <NUM>. At block <NUM>, the memory manager <NUM> may cause desired contents of the in-fabric memory <NUM> at the specified address range to be transferred to the NOC <NUM> via the data register <NUM> and the data pathway <NUM>, as shown in the communication scheme <NUM> of <FIG>.

Referring now to <FIG>, at block <NUM>, the control interface (CTRL I/F) <NUM> may issue a control signal on the control pathway <NUM> to the sector controller (SC) <NUM>. The control signal may include a request for a data transfer from the NOC <NUM> to the in-fabric memory <NUM>. After receiving the request for the data transfer from the NOC <NUM> to the in-fabric memory <NUM>, at block <NUM>, the sector controller (SC) <NUM> may delegate the transfer request to the memory manager <NUM>. That is, the sector controller (SC) <NUM> may relay the request to the memory manager <NUM>. At block <NUM>, the memory manager <NUM> may cause desired contents of the NOC <NUM> to be transferred to the in-fabric memory <NUM> via the data register <NUM> and the data pathway <NUM>, as shown in the communication scheme <NUM> of <FIG>.

In certain embodiments, the FPGA fabric <NUM> may forgo using the sector controller (SC) <NUM> with regard to data transfers directly between the FPGA fabric <NUM> and the NOC <NUM>. For example, <FIG> represents another example of a data transfer from the FPGA fabric <NUM> and/or registers of the programmable logic elements <NUM> to the NOC <NUM> using the circuitry <NUM> described above with reference to <FIG>. Additionally, <FIG> represents an example of a data transfer from the NOC <NUM> to the FPGA fabric <NUM> and/or registers of the programmable logic elements <NUM> using the circuitry <NUM>. As such, additional description of elements with the same numbering as those in <FIG> may be found in the text above. Here, the configuration program <NUM>, control circuitry such as a soft or hardened controller implemented in the programmable logic elements <NUM>, and/or the control interface (CTRL I/F) <NUM> and the data interface (DATA I/F) <NUM> may coordinate data transfer from the FPGA fabric <NUM> and/or registers of the programmable logic elements <NUM> to the NOC <NUM>, as depicted in <FIG>, or from the NOC <NUM> to the FPGA fabric <NUM> and/or registers of the programmable logic elements <NUM>, as depicted in <FIG>. The data from the FPGA fabric <NUM> and/or registers of the programmable logic elements <NUM> may be transferred between the data interface (DATA I/F) <NUM> to the NOC <NUM> using the data pathway <NUM> under instruction from control signals on the control pathway <NUM> by the control interface (CTRL I/F) <NUM>.

<FIG> represents another example of a data transfer from the FPGA fabric <NUM> and/or registers of the programmable logic elements <NUM> to the NOC <NUM> using the circuitry <NUM> described above with reference to <FIG>. As such, additional description of elements with the same numbering as those in <FIG> may be found in the text above. Here, the configuration program <NUM>, control circuitry such as a soft or hardened controller implemented in the programmable logic elements <NUM>, and/or the control interface (CTRL I/F) <NUM> and the data interface (DATA I/F) <NUM> may coordinate data transfer from FPGA fabric <NUM> and/or registers of the programmable logic elements <NUM> to the NOC <NUM>. The data from the NOC <NUM> may be transferred to the data interface (DATA I/F) <NUM> to the NOC <NUM> using the data pathway <NUM> under instruction from control signals on the control pathway <NUM> by the control interface (CTRL I/F) <NUM>, and subsequently programmed into the FPGA fabric <NUM> and/or registers of the programmable logic elements <NUM>.

In the same manner, <FIG> represents another example of a data transfer from the NOC <NUM> to the FPGA fabric <NUM> and/or registers of the programmable logic elements <NUM> using the circuitry <NUM>. As shown in <FIG>, the configuration program <NUM>, control circuitry such as a soft or hardened controller implemented in the programmable logic elements <NUM>, and/or the control interface (CTRL I/F) <NUM> and the data interface (DATA I/F) <NUM> may coordinate data transfer from the NOC <NUM> to the FPGA fabric <NUM> and/or registers of the programmable logic elements <NUM>. The data from the NOC <NUM> to the FPGA fabric <NUM> and/or registers of the programmable logic elements <NUM> may be transferred between the data interface (DATA I/F) <NUM> using the data pathway <NUM> under instruction from control signals on the control pathway <NUM> by the control interface (CTRL I/F) <NUM>.

With the foregoing in mind, it should be noted that the fabric sectors <NUM> may initiate a transfer of data directly between memory locations of the sector-aligned memory <NUM> of the base die <NUM> using the NOC <NUM>, between different fabric sectors <NUM> of the fabric die <NUM>, between fabric sectors <NUM> and memory locations of the sector-aligned memory <NUM>, and the like. In certain embodiments, the sector controller (SC) <NUM> may initiate the transfer of data between sectors <NUM> of the fabric die <NUM>, between memory locations of the sector-aligned memory <NUM>, between sectors <NUM> of the fabric die <NUM> and memory locations of the sector-aligned memory <NUM>, and the like. That is, the sector controller (SC) <NUM> may coordinate the operation of the NOC <NUM> to facilitate the transfer of the data between the source and destination targets, as specified by the section controller (SC) <NUM>. In some embodiments, the section controller (SC) <NUM> may act as a master to initiate the respective transfer and the transfer would then be performed using the NOC <NUM> after the section controller (SC) <NUM> initiates the data transfer process.

By way of example, <FIG> includes a block diagram of illustrating the transfer of data between sectors <NUM> of the programmable logic fabric via the NOC circuitry <NUM> of the base die <NUM>. Referring to <FIG>, in one embodiment, the sector controller (SC) <NUM> may initiate a transfer of data from sector "A" to sector "B" of the fabric die <NUM> using the NOC <NUM> of the base die <NUM>. That is, the sector controller (SC) <NUM> may transfer data to a region of the sector-aligned memory <NUM> aligned with sector "A" of the fabric die <NUM>, use the NOC <NUM> to transfer the data to a second region of the sector-aligned memory <NUM> aligned with sector "B" of the fabric die <NUM>, and transfer the data from the second region of the sector-aligned memory <NUM> to sector "B" of the fabric die <NUM>. Although the route of the data transfer illustrated in <FIG> corresponds to straight paths, it should be noted that the data transferred to different sectors <NUM> of the fabric die <NUM> or regions of the sector-aligned memory <NUM> may use a variety of directions and routes.

In another example, <FIG> includes a block diagram of illustrating the transfer of data between a sector <NUM> of the programmable logic fabric to a region of the sector-aligned memory <NUM> via the NOC circuitry <NUM> of the base die <NUM>. Referring to <FIG>, in one embodiment, the sector controller (SC) <NUM> may initiate a transfer of data from sector "A" of the fabric die <NUM> to region "C" of the sector-aligned memory <NUM> using the NOC <NUM> of the base die <NUM>. That is, the sector controller (SC) <NUM> may transfer data to a first region of the sector-aligned memory <NUM> aligned with sector "A" of the fabric die <NUM> and use the NOC <NUM> to transfer the data to region "C" of the sector-aligned memory <NUM> via different regions of the sector-aligned memory <NUM> or the like. Like <FIG>, although the route of the data transfer illustrated in <FIG> corresponds to straight paths, it should be noted that the data transferred to different regions of the sector-aligned memory <NUM> may use a variety of directions and routes. In addition, in certain embodiments, the transfer of data described herein may employ the circuitry <NUM> and the circuitry <NUM> described above with reference to <FIG> and <FIG>. However, it should be noted that other suitable circuits may also be employed to transfer the data in accordance with the embodiments presented herein.

As shown in <FIG> and <FIG>, the sector controller (SC) <NUM> may initiate a transfer of data directly between memory locations within the base die <NUM> using the NOC system <NUM>. In this case, the sector controller (SC) <NUM> may act as the master to initiate the transfer, but then the transfers would be performed directly in the sector-aligned memory <NUM> and the NOC system <NUM> of the base die <NUM> after the sector controller (SC) <NUM> initiates the transfer.

It should also be mentioned that, in some embodiments, that the sector controller (SC) <NUM> and similar components of the fabric die <NUM> may also initiate the components (e.g., memory control circuitry <NUM>, non-user input control circuitry <NUM>, non-user output control circuitry <NUM>) of the base die <NUM> to perform transfers between the sector-aligned memory <NUM>, the peripheral circuitry <NUM>, and other components attached to the base die. As a result, data transfers may occur in the base die <NUM> without involvement of components in the fabric die <NUM>.

The programmable logic device <NUM> may be, or may be a component of, a data processing system. For example, the programmable logic device <NUM> may be a component of a data processing system <NUM>, shown in <FIG>. The data processing system <NUM> includes a host processor <NUM>, memory and/or storage circuitry <NUM>, and a network interface <NUM>. The data processing system <NUM> may include more or fewer components (e.g., electronic display, user interface structures, application specific integrated circuits (ASICs)). The host processor <NUM> may include any suitable processor, such as an Intel® Xeon® processor or a reduced-instruction processor (e.g., a reduced instruction set computer (RISC), an Advanced RISC Machine (ARM) processor) that may manage a data processing request for the data processing system <NUM> (e.g., to perform machine learning, video processing, voice recognition, image recognition, data compression, database search ranking, bioinformatics, network security pattern identification, spatial navigation, or the like). The memory and/or storage circuitry <NUM> may include random access memory (RAM), read-only memory (ROM), one or more hard drives, flash memory, or the like. The memory and/or storage circuitry <NUM> may be considered external memory to the programmable logic device <NUM> and may hold data to be processed by the data processing system <NUM>. In some cases, the memory and/or storage circuitry <NUM> may also store configuration programs (bitstreams) for programming the programmable logic device <NUM>. The network interface <NUM> may allow the data processing system <NUM> to communicate with other electronic devices. The data processing system <NUM> may include several different packages or may be contained within a single package on a single package substrate.

In one example, the data processing system <NUM> may be part of a data center that processes a variety of different requests. For instance, the data processing system <NUM> may receive a data processing request via the network interface <NUM> to perform machine learning, video processing, voice recognition, image recognition, data compression, database search ranking, bioinformatics, network security pattern identification, spatial navigation, or some other specialized task. The host processor <NUM> may cause the programmable logic fabric of the programmable logic device <NUM> to be programmed with a particular accelerator related to requested task. For instance, the host processor <NUM> may instruct that configuration data (bitstream) stored on the memory / storage <NUM> or cached in sector-aligned memory of the programmable logic device <NUM> to be programmed into the programmable logic fabric of the programmable logic device <NUM>. The configuration data (bitstream) may represent a circuit design for a particular accelerator function relevant to the requested task. Due to the high density of the programmable logic fabric, the proximity of the substantial amount of sector-aligned memory to the programmable logic fabric, or other features of the programmable logic device <NUM> that are described here, the programmable logic device <NUM> may rapidly assist the data processing system <NUM> in performing the requested task. Indeed, in one example, an accelerator may assist with a voice recognition task less than a few milliseconds (e.g., on the order of microseconds) by rapidly accessing and processing large amounts of data in the accelerator using sector-aligned memory.

The methods and devices of this disclosure may be incorporated into any suitable circuit. For example, the methods and devices may be incorporated into numerous types of devices such as microprocessors or other integrated circuits. Exemplary integrated circuits include programmable array logic (PAL), programmable logic arrays (PLAs), field programmable logic arrays (FPLAs), electrically programmable logic devices (EPLDs), electrically erasable programmable logic devices (EEPLDs), logic cell arrays (LCAs), field programmable gate arrays (FPGAs), application specific standard products (ASSPs), application specific integrated circuits (ASICs), and microprocessors, just to name a few.

Moreover, while the method operations have been described in a specific order, it should be understood that other operations may be performed in between described operations, described operations may be adjusted so that they occur at slightly different times or described operations may be distributed in a system which allows the occurrence of the processing operations at various intervals associated with the processing, as long as the processing of overlying operations is performed as desired.

Claim 1:
An integrated circuit device comprising:
programmable logic fabric disposed on a first integrated circuit die (<NUM>), wherein the programmable logic fabric comprises a first region of programmable logic fabric, wherein the first region of programmable logic fabric is configured to be programmed with a circuit design that operates on a first set of data;
network on chip, NOC, circuitry (<NUM>) disposed on a second integrated circuit die (<NUM>), wherein the NOC circuitry (<NUM>) is configured to communicate data between the first integrated circuit die (<NUM>) and the second integrated circuit die (<NUM>), and the NOC circuitry (<NUM>) is configured to communicate a second set of data from the first region of programmable logic fabric to a second region of programmable logic fabric, wherein a plurality of powered down sectors (<NUM>, <NUM>) is disposed between the first region of programmable logic fabric and the second region of programmable logic fabric on the first integrated circuit die (<NUM>), thereby preventing communication across the plurality of powered down sectors (<NUM>, <NUM>); and
a sector-aligned memory (<NUM>) disposed on the second integrated circuit die (<NUM>), wherein the sector-aligned memory (<NUM>) comprises a first region of sector-aligned memory (<NUM>) directly accessible by the first region of programmable logic fabric and a second region of sector-aligned memory (<NUM>) directly accessible by a second region of programmable logic fabric.