Patent Description:
Electronic devices, such as tablets, computers, copiers, digital cameras, smart phones, control systems and automated teller machines, among others, often employ electronic components such as dies that are connected by various interconnect components. The dies may include memory, logic or other integrated circuit (IC) device.

ICs may be implemented to perform specified functions. Example ICs include mask-programmable ICs, such as general purpose ICs, application specific integrated circuits (ASICs), and the like, and field programmable ICs, such as field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and the like.

ICs have become more "dense" over time, i.e., more logic features have been implemented in an IC. More recently, Stacked-Silicon Interconnect Technology ("SSIT") allows for more than one semiconductor die to be placed in a single package. SSIT ICs may be used to address increased demand for having various ICs within a single package. Conventionally, SSIT products are implemented using an interposer that includes an interposer substrate layer with through-silicon-vias (TSVs) and additional metallization layers built on top of the interposer substrate layer. The interposer provides connectivity between the IC dies and the package substrate.

Chip-to-chip interfaces (also called interconnects) provide a bridge between host devices, such as between ICs, system-on-chip (SoCs), FPGAs, ASICs, central processing units (CPUs), graphic processing units (GPUs), etc..

As the data rates which can be processed by systems increases, providing interfaces that can keep up with the processing speed of the chip becomes increasingly difficult. Power-efficient, robust, and low-cost chip-to-chip interfaces are desirable to meet the needs of high-performance systems.

High speed chip-to-chip interfaces sometime involve tradeoffs between pin count, input/output (I/O) die area, power, etc. Some examples of chip-to-chip interfaces include low voltage complementary metal oxide semiconductor (LVCMOS) I/O, low voltage differential signaling (LVDS) I/O, high speed serializer/deserializer (SERDES) I/O.

High bandwidth memory (HBM) is a high-performance random access memory (RAM) interface for 3D-stacked dynamic RAM (DRAM) and has been adopted by the Joint Electron Device Engineering Council (JEDEC) standards body. The HBM standard defines a new type of physical interface for communication between an HBM DRAM device and a host device such as an ASIC, CPU, GPU, or FPGA. The HBM physical interface can improve tradeoff point as far as I/O die area and power as compared to certain other interfaces. HBM can achieve high bandwidth using less power in a small form factor.

For some systems, a high speed interface is desirable to efficiently integrate other host devices on a single interposer. Thus, techniques for a high bandwidth chip-to-chip interface would be useful.

<CIT>) describes methods and apparatus for adding one or more features (e.g., HBM) to a qualified SSI technology programmable IC region by providing an interface (e.g., an HBM buffer region with a switch network) between the added feature device and the programmable IC region. One example IC package generally includes a package substrate; at least one interposer disposed above the package substrate; a programmable IC region disposed above the interposer; at least one fixed feature die disposed above the interposer; and an interface region disposed above the interposer and configured to couple the programmable IC region to the fixed feature die via a first set of interconnection lines routed through the interposer between a first plurality of ports of the interface region and the fixed feature die and a second set of interconnection lines routed between a second plurality of ports of the interface region and the programmable IC region.

<CIT> describes a memory system in which at least one DRAM chip and a memory controller chip are mounted in a side-by-side relationship on an interposer. The DRAM chip is connected to the interposer via a Wide I/O interface to enable the DRAM chip and the memory controller chip to communicate with each other via the Wide I/O interface. The memory controller chip has a SerDes interface for communicating with a SerDes interface of an integrated circuit (IC) chip of the memory system.

<CIT>) describes a memory system divided into memory subsystems. Each subsystem includes a slave controller. Each slave controller is coupled to a serial link. A master controller is coupled to the slave controllers via the serial links, and the master controller is capable of initiating a memory access to a memory subsystem by communicating with the slave controller via the serial link. Each memory subsystem includes memory arrays coupled to the slave controller. Each memory array includes memory channels. Memory accesses to a memory array on a memory subsystem are interleaved by the slave controller between the memory channels, and memory accesses to a memory subsystem are striped by the slave controller between the memory arrays on the memory subsystem. Memory accesses are striped between memory subsystems by the master controller. The master controller and slave controllers communicate by sending link packets and protocol packets over the serial links.

"ARM AMBA <NUM> CHI Architecture Specification, ARM IHI 0050B, ID080717", XP093129844, describes the AMBA <NUM> CHI architecture. The Coherent Hub Interface (CHI) architecture provides a comprehensive layered specification to build small, medium, and large systems comprising of multiple components using a scalable coherent hub interface and on chip interconnect. The CHI architecture permits flexibility on the topology of the component connections, and this can be driven from the system performance, power, and area requirements. The components of CHI based systems can comprise of standalone processors, processor clusters, graphic processors, memory controllers, I/O bridges, PCIe subsystems and the interconnect itself.

Techniques related to a high bandwidth chip-to-chip interface using the high bandwidth memory (HBM) physical interface are described.

According to a first aspect of the disclosure, there is provided a computing system according to the appended claims.

In some embodiments, the first and second host devices may each be configurable as a master device, a slave device, or both.

In some embodiments, the HBI may provide a plurality of independent directional channels.

In some embodiments, the first host device may include a 3D programmable integrated circuit (IC) and the second host device may include an application-specific IC (ASIC).

In some embodiments, the physical layer protocol may be configured in a continuous data flow per input/output (I/O) word mode and a <NUM>:<NUM> serializer/deserializer (SERDES) mode.

In some embodiments, the transport layer protocol includes an output channel configured to issue credits used by an input channel of the transport layer protocol.

In some embodiments, the protocol layer comprises a high-level streaming or memory-mapped advanced extensible interface (AXI) protocol.

In some embodiments, the memory-mapped AXI protocol may include a <NUM>-bit write strobe signal. A first <NUM> bits may indicate a number of zeros in a first region of the write strobe in which all bits are zero and a second <NUM> bits may indicate a number of ones in a second region of the write strobe in which all bits are one. The write strobe may further include a third region in which all bits are zero.

In some embodiments, the memory-mapped AXI protocol may support a mixed command channel packet for a simultaneous read command and write command.

In some embodiments, the memory-mapped AXI protocol may support a mixed response channel packet for a simultaneous read response and write response.

In some embodiments, the first die and the die may be connected via a plurality of wires. Bumps on the first die may be connected to bumps in a corresponding location on the second die. The transport layer may be further configured to reorder bits and the first die and the second die may have a same orientation or a different orientation with respect to each other on the interposer.

According to another aspect of the disclosure, there is provided a method for communication between devices on an interposer, according to the appended claims.

In some embodiments, the first signal may be sent via a first independent directional HBI channel and the second signal may be received via a second directional HBI channel.

In some embodiments, the first device may include a 3D programmable integrated circuit (IC) and the second device may include an application-specific IC (ASIC).

In some embodiments, the protocol layer has a high-level streaming or memory-mapped advanced extensible interface (AXI) protocol.

In some embodiments, the first or second signal may include a <NUM>-bit write strobe signal. A first <NUM> bits may indicate a number of zeros in a first region of the write strobe in which all bits are zero and a second <NUM> bits may indicate a number of ones in a second region of the write strobe in which all bits are one. The write strobe may further include a third region in which all bits are zero.

In some embodiments, the first or second signal may include a mixed command channel packet for a simultaneous read command and write command.

In some embodiments, the first or second signal may include a mixed response channel packet for a simultaneous read response and write response.

These and other aspects may be understood with reference to the following detailed description.

So that the manner in which the above recited features can be understood in detail, a more particular description, briefly summarized above, may be had by reference to example implementations, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical example implementations and are therefore not to be considered limiting of its scope.

It is contemplated that elements of one example may be beneficially incorporated in other examples.

Various features are described hereinafter with reference to the figures. It should be noted that the figures may or may not be drawn to scale and that the elements of similar structures or functions are represented by like reference numerals throughout the figures. It should be noted that the figures are only intended to facilitate the description of the features.

Examples of the disclosure relate to techniques and apparatus for high bandwidth interface (HBI), such as a high speed chip-to-chip interface, at least partially using the high bandwidth memory (HBM) physical interface to efficiently integrate host devices on a single interposer. In some examples, the HBI interface uses HBM at the physical layer (PHY) and uses different protocols or adjustments of the HBM for other layers.

Before describing exemplary implementations illustratively depicted in the several figures, a general introduction is provided to further understanding.

Silicon stacked interconnect technology (SSIT) involves packaging multiple integrated circuit (IC) dies into a single package that includes an interposer and a package substrate. Utilizing SSIT expands IC products, such as and including FPGA products and other types of products, into higher density, lower power, greater functionality, and application specific platform solutions with low cost and fast-to-market advantages.

<FIG> is a cross-sectional schematic diagram illustrating an exemplary SSIT product (electronic device <NUM>) according to an example implementation. The electronic device <NUM> includes an integrated chip package <NUM> disposed in a housing <NUM>. The electronic device <NUM> may be used in a computer, tablet, cell phone, smart phone, consumer appliance, control system, automated teller machine, programmable logic controller, printer, copier, digital camera, television, monitor, stereo, radio, radar, or other device.

The integrated chip package <NUM> includes a plurality of IC dies <NUM> (e.g., IC dies <NUM>(<NUM>) and <NUM>(<NUM>) are shown by example) connected optionally by a silicon-through-via (TSV) interposer <NUM> (also referred to as "interposer <NUM>") to a package substrate <NUM>. The chip package <NUM> may also have an overmold covering the IC dies <NUM> (not shown). The interposer <NUM> includes circuitry (not shown) for electrically connecting the IC dies <NUM> to circuitry (not shown) of the package substrate <NUM>. The circuitry of the interposer <NUM> may optionally include transistors. Package bumps <NUM>, also known as "C4 bumps," are utilized to provide an electrical connection between the circuitry of the interposer <NUM> and the circuitry of the package substrate <NUM>. The package substrate <NUM> may be mounted and connected to a printed circuit board (PCB) <NUM>, utilizing solder balls <NUM>, wire bonding or other suitable technique. The PCB <NUM> can be mounted in the interior of a housing <NUM> of the electronic device <NUM>.

The IC dies <NUM> are mounted to one or more surfaces of the interposer <NUM>, or alternatively, to the package substrate <NUM>. The IC dies <NUM> may be programmable logic devices, such as FPGAs, memory devices, optical devices, processors or other IC logic structures. In the example depicted in <FIG>, the IC dies <NUM> are mounted to a top surface of the interposer <NUM> by a plurality of micro-bumps <NUM>. The micro-bumps <NUM> electrically connect the circuitry of each IC die <NUM> to circuitry of the interposer <NUM>. The circuitry of the interposer <NUM> connects the micro-bumps <NUM> to package bumps <NUM>, and hence, connects selective circuitry of each IC dies <NUM> to the package substrate <NUM>, to enable communication of the IC dies <NUM> with the PCB after the chip package <NUM> is mounted within the electronic device <NUM>. When the interposer <NUM> is not present, the micro-bumps <NUM> connect selective circuitry of each IC die <NUM> to the package substrate <NUM> to enable communication of the IC dies <NUM> with the PCB after the chip package <NUM> is mounted within the electronic device <NUM>. Although not shown, it is contemplated that one or more additional IC dies may be stacked on one or both of the IC dies <NUM>.

The electrical components of integrated chip package <NUM>, such as the IC dies <NUM>, communicate via traces formed on electrical interconnect components. The interconnect components having the traces can include one or more of the PCB <NUM>, the package substrate <NUM> and interposer <NUM>, among others components.

As mentioned, currently, The HBM standard defines a new type of physical interface for communication between an HBM DRAM device and a host device such as an ASIC, CPU, GPU, or FPGA. In one example, the printed circuit board <NUM> is a graphics card and the IC <NUM>(<NUM>) is a GPU. In this case, the IC <NUM>(<NUM>) may include a 3D engine, a display controller, and an HBM controller; and the IC <NUM>(<NUM>) may include stacked DRAM dies and an optional base HBM controller die interconnected by through-silicon vias (TSVs) and microbumps. The interface is divided into independent channels, each operating as a data bus.

In some examples, HBM devices have up to <NUM> independent DRAM channels. Each DRAM channel includes two <NUM>-bit data channels known as a pseudochannel (PC), and one command/address channel shared by the two PCs. Each PC can operate at a maximum data rate of <NUM> MT/sec, double data rate, using a <NUM> clock. HBM features include: typically a die stack with <NUM>-<NUM> channels per die; <NUM> x 128b independent channels; <NUM> or <NUM> banks with optional bank grouping; <NUM> Kb page size; <NUM>-<NUM> Gbit of storage per channel; <NUM> Gbps (<NUM>) operation (<NUM> GByte/sec per <NUM> bit channel); burst length (BL) of <NUM>, thus, minimum access unit per PC is <NUM> bytes; <NUM> V (+/- <NUM> %) I/O and core voltage (independent); <NUM> V (+/- <NUM> %) pump voltage (VPP); unterminated I/O with nominal drive current <NUM>-<NUM> mA; write data mask (DM) support; error correcting code (ECC) support by using the DM signals, <NUM> bits per <NUM> b data (partial write not supported when ECC is used); data bus inversion (DBI) support; separate read and write data strobe (DQS) signals (differential); separate row and column command channels; command/address parity support; data parity support (in both directions); and address/data parity error indication. <FIG> is a table <NUM> summarizing the interface signals of a single <NUM>-bit channel in HBM according to an example.

In some cases, however, it may be desirable to have a high-speed interface to interconnect multiple host devices, for example, on the interposer <NUM> (i.e., rather than a DRAM and a host device). Thus, aspects of the present disclosure relate implementing portions of the HBM interface as a high-speed interconnect between host devices, which may be on a same interposer.

<FIG> is an example of host devices connected with a high bandwidth interface (HBI) interface according to an example. As shown in <FIG>, an interposer <NUM> has a host device1 <NUM> in communication with host device(s) <NUM>. 204n via HBI interface(s) <NUM>. The Host devices <NUM>, <NUM>. 204n may be any type of host device, such as an ASIC, CPU, GPU, or FPGA. In some examples, host device1 <NUM> is an all programmable device and the host device(s)<NUM>. 204n are ASICs. In some examples, the host device1 <NUM> may be a 3D IC. In some examples, the host device1 <NUM> may be also be in communication with a dummy device, an HBM DRAM, or other device, which may also be on the interposer <NUM>. In some examples, the host device1 <NUM> may have multiple interconnects (e.g., multiple HBI interfaces) with the same host device. In some examples, the host device(s) <NUM>. 204n may be client customer/user devices.

The HBI(s) <NUM>. 206n is a high performance chip-to-chip interface. The HBI(s) <NUM>. 206n may be at least partially based on the JEDEC HBM specifications. In some examples, the HBI interface uses the physical layer (PHY) and I/O as defined by the HBM specification, but uses different protocols or adjustments of the HBM for other layers. For example, since the HBI is an interconnect for host devices, the HBI may dispense with DRAM specific protocols of HBM.

The HBI interface (e.g., HBI(s) <NUM>. 206n) may be compatible with HBM PHY and I/O at a date rate up to <NUM> MT/s. In some examples, the HBI uses the PHY in a "bit-slice mode".

The HBI interface (e.g., HBI(s) <NUM>. 206n) may support a user-side interface at one-fourth the HBM data rate (e.g. <NUM>). While the nominal HBM clock rate described herein may be <NUM> and the HBI user-side rate may be <NUM>, other rates may be used as actual device rates may vary depending on implementation, speed grades, etc..

Portions of the HBM interface may not be symmetrical between a master (e.g., controller) and a slave (e.g., DRAM). For example, the command/address channel is unidirectional.

To ensure symmetry and interoperability with either master or slave HBM PHY (or both simultaneously), the HBI interface (e.g., HBI(s) <NUM>. 206n) may use only a subset of the HBM standard interface which is symmetrical, i.e., which makes it possible for either side to transmit or receive data. For example, the HBM signals "Command/Address", "DERR", and "AERR" signals may not be used by the HBI interface.

The HBI interface (e.g., HBI(s) <NUM>. 206n) may support multiple independent communication channels.

The HBI interface (e.g., HBI(s) <NUM>. 206n) may be configured and calibrated once at start time. In some examples, the HBI may require little or no maintenance after the initial configuration and calibration.

As mentioned above, the HBI interface (e.g., HBI(s) <NUM>. 206n) may provide multiple channels. Each channel may operate in one direction (e.g., output or input).

As mentioned above, the HBI interface (e.g., HBI(s) <NUM>. 206n) may provide multiple channels. The number of HBI channels may vary depending on the application. In some examples, an HBI may use an HBM PHY consisting of <NUM><NUM>-bit data channels, however, a different number of channels may be used. An HBM PHY with <NUM> X <NUM>-bit channels may be referred to as an "HBM PHY unit".

The HBI interface (e.g., HBI(s) <NUM>. 206n) may support three protocol layers as shown in <FIG>. The layered protocol may allow seamless chip-to-chip communication using a high level protocol and may provide flexibility to implement other high level protocols, for example, by only replacing the layer-<NUM> protocol. The protocol layers includes the layer-<NUM><NUM>, a PHY layer (e.g., exposed HBM PHY); the layer-<NUM><NUM>, a transport layer that may provide basic data transport, parity, framing, and flow control; and the layer-<NUM><NUM>, a protocol layer. In some examples, the layer-<NUM><NUM> is an HBM PHY. In some examples, the layer-<NUM> is a mapping of the AXI4-MM or AXI4-S onto the layer-<NUM>.

The HBI interface (e.g., HBI(s) <NUM>. 206n) may use the physical interface as a general purpose communication channel between the chips (e.g., the host devices <NUM>, <NUM>. 204n on the interposer <NUM>. The PHY layer may be the first and lowest layer (also referred to as layer <NUM>) and may refer to the circuitry that implements physical layer functions in communications. The PHY may define the electrical and physical specifications of the data connection.

The HBI layer-<NUM><NUM> is the direct access to the HBM PHY. A "controller bypass" mode may be used in which PHY signals are directly exposed to the PL, and data flows continuously. The HBM standard defines eight <NUM>-bit legacy channels, or sixteen <NUM>-bit pseudo channels. The basic data unit in layer-<NUM><NUM> is a <NUM>-bit data word. Therefore, thirty-two layer-<NUM> channels are available per HBM PHY unit. The HBM PHY may operate in a <NUM>:<NUM> SERDES (serializer/deserializer) mode, meaning that it provides bus-width conversion and corresponding clock speed conversion at a <NUM>:<NUM> ratio from the user's point of view.

In some examples, on the I/O side, each L0 (i.e., PHY) channel is <NUM>-bit wide, operating at <NUM> DDR (<NUM> MT/s), while on the user side the L0 channel is seen as a single-data-rate <NUM>-bit channel operating at <NUM>. The subset of HBM signals available for L0 is summarized in the table <NUM> shown in <FIG>, while on the user side, each L0 channel is seen as a <NUM>-bit unidirectional data pipe (e.g., because data (DQ), data bit inversion (DBI), data mask (DM), and parity (PAR) signals are all treated the same way by the HBM PHY).

For the HBI PHY Internet Protocol (IP), only an HBM PHY may be needed. As discussed above, the HBM PHY may be directly accessible, the HBM PHY may be in a "bit slice mode" that allows continuous data flow per <NUM>-bit I/O word, the I/O direction may selectable per <NUM>-bit I/O word, and the HBM PHY may be in a <NUM>:<NUM> SERDES mode.

<FIG> illustrates the HBI transport layer protocol <NUM> (i.e., layer-<NUM> or L1). As shown in <FIG>, the HBI layer-<NUM> defines a transport protocol <NUM> on top of the PHY <NUM>. The HBI L1 may have sixteen <NUM>-bit unidirectional user channels (per HBM PHY unit). As shown in <FIG>, the HBI L1 provides parity protection, DBI support, flow control, and framing/alignment. As shown in <FIG>, each HBI L1 channel may use two L0 channels <NUM>, <NUM>, thus, a total of sixteen L1 (data) channels are available per HBM PHY unit. Each HBI L1 channel may be configurable as input or output. Each HBI L1 channel may provide a <NUM>-bit data bus in one direction (i.e., channel bus width), for example, by using the two L0 channels <NUM>, <NUM>.

The HBI L1 provides the DBI functionality as defined in the HBM standard. The purpose of the DBI is to reduce I/O power by minimizing the number of transitions on the data bus.

The HBI L1 may provide the parity protection as defined in the HBM standard. For example, each <NUM>-bit word is protected with one parity bit. The HBI L1 provides parity generation on the transmit side, and parity checking, error logging, and error reporting on the receive side. Error recovery can be implemented external to the HBI.

As shown in <FIG>, the HBI L1 provides credit-based flow control mechanism. Since each channel is unidirectional, an output L1 channel can be used to issue credits used by another input L1 channel.

As shown in <FIG>, the HBI L1 provides framing and alignment. The HBI L1 provides intra-channel framing and alignment within an L0 channel <NUM>, <NUM> as shown in <FIG> and inter-channel framing and alignment between the two L0 channels <NUM>, <NUM> used by the L1 channel as shown in <FIG>. In some examples, framing is achieved using framing signals sent along with the data to provide alignment and identification of the first word in the serialized sequence created by the PHY SEDES function.

The HBI L1 receive logic may be responsible for achieving and maintaining alignment, detecting alignment errors, and recovering from such errors. Alignment errors may be reported via a status/ interrupt register.

The HBI L1 may reorder the bits from the L0, for example, prior to use. The reordering may depend on the die and PHY orientation.

The HBI L1 user-side interface may be defined as shown in the Table <NUM> in the <FIG>. The HBI L1 provides a <NUM>-bit interface at <NUM>. All signals flow in the same direction (in or out) depending on how the L1 channel is configured. Table <NUM> in <FIG> shows how the L1 signals are mapped to the available L0 I/O signals. Each L0 I/O signal may transport four L2 bits.

For the HBI L1 IP, the L1 function may be implemented as soft logic IP in the PL on the side of the host device1 <NUM>.

Ignoring the user side channel and other overhead signals, each HBI L1 channel may sustain a throughput of around <NUM> GBytes/sec. Total HBI L1 throughput per HBI (for <NUM> HBM PHY unit) is therefore <NUM> GBytes/sec, or <NUM> Tbits/sec.

As shown in <FIG>, the HBI protocol layer <NUM> (i.e., the layer-<NUM> or L2) is on top of the L1. The HBI L2 is used to encapsulate high level protocols over L1. Given the flexible layered approach, multiple L2 implementations are possible. Different L2 may be implemented depending on the host device(s) <NUM>. 204n being used (e.g., for different customers). Two examples of the L2 includes a memory-mapped protocol (e.g., an AXI4 (L2m) L2 protocol) and a streaming protocol (e.g., an AXI4 (L2s) protocol).

The memory-mapped HBI L2 protocol is illustrated in <FIG>. The memory mapped HBI L2 protocol may use a <NUM>-bit AXI MM interface at <NUM>. As shown in <FIG>, the memory mapped HBI L2 protocol may map to two L1 channel-one inbound and one outbound. Thus, one HBM PHY unit can support <NUM> AXI-MM interfaces, which may be configurable as a master AXI or slave AXI. The AXI commands/responses (e.g., read/write) may be packetized. In <FIG>, the AXI-MM interface is configured as a node master unit (NMU), and local AXI masters can access remote AXI slaves on the other die. In <FIG>, the AXI-MM interface is configured as a node slave unit (NSU) and remote AXI masters on the other die can access local AXI slaves. Each L2 channel, whether NMU or NSU, uses two L1 channels, for example, as defined above.

The outbound master (inbound slave) channel is used for read and write commands, and the inbound master (outbound slave) channel is used for read and write responses. Each HBI L2 channel may support two virtual channels (VCs). The VCs may ensure independent forward progress of the read and write transactions. There may be separate flow control credit management per VC.

The HBI L2 may not employ read tags or reorder buffers. The HBI L2 may not support ECC.

Features of an HBI protocol layer AXI4 interface are summarized in the Table <NUM> in <FIG>.

In some systems, <NUM> bits of write strobe (WSTRB) are used per data beat for a <NUM>-bit AXI bus <NUM> to allow any combination of write strobes. However, such flexibility, though allowed, is rarely required. The strobes are often used in single beat partial writes or unaligned burst writes-in both cases, the WSTRB pattern can be encoded with far fewer than <NUM> bits. In some examples, the memory-mapped HBI L2 may support only single-beat partial writes for WSTRB containing zeros. The WSTRB word has only one contiguous region of nonzero WSTRB bits. In the data beat there are only three contiguous strobe regions: region <NUM> in which all WSTRB bits are <NUM>; region <NUM> in which all WSTRB bits are <NUM>; and region <NUM> in which all WSTRB bits are <NUM>. Such case can be fully described using two values-a value N1 describing the number of <NUM>'s in region <NUM> (<NUM>-<NUM>) and a value N2 describing the number of <NUM>'s in region <NUM> (<NUM>-<NUM>). <NUM> bits may be used for the encoding. In some examples, for multi-beat transactions, no partial writes are allowed, i.e. all WSTRB bits must be set. Multi-beat unaligned writes are chopped prior to entering the memory-mapped HBI L2. The memory-mapped HBI L2 hardware may include a detector for violations and debugging of WSTRB restrictions. Allowed WSTRB values are shown in the Table <NUM> in the <FIG>.

Transmissions in the memory-mapped HBI L2 may be packetized. The AXI4 protocol has five channels: write address, write data, write response, read address, read response. The memory-mapped HBI L2 may combines the write address and write data channels, and packetize the transactions into four VCs of packets: Write command packet (includes both address and data); Read command packet; Write response packet; and Read response packet. The command packets are outbound (from master to slave), while the response packets are inbound (from slave to master).

Each VC has separate flow control credit management and can make forward progress independent of other VCs. For example, the outbound channel can issue two credits per cycle, one each for the two inbound VCs, and the inbound channel can issue two credits per cycle, one each for the two outbound VCs. In some examples, the credits are per word, not per packet. Write commands and read commands share the same outbound memory-mapped HBI L2 channel, while read and write responses share the same inbound memory-mapped HBI L2 channel. Packetization improves throughput per wire and is widely used in network-on-chip (NoC) solutions. <FIG> show the packet header formats 1400A, 1400B, 1400C, for the Write Command packet, Read Command packet, and No Operation packet, respectively. A value of the HTYPE field (header type) may indicate the type of the packet header-for example, <NUM> for NoP, <NUM> for Read, <NUM> for Write. The packet headers may be <NUM>-bit words. In some examples, two packet headers may be sent simultaneously on the same <NUM>-bit word, for example, subject to flow control credit availability.

<FIG> show the memory-mapped HBI L2 command channel packet formats 1500A, 1500B, 1500C, for a Write Command packet, a mixed Write and Read Command packet (consumes both a read and a write credit), and a Read Command Packet, respectively. The command channel packet may not use the L1 <NUM>-bit user side channel. Multi-word command packets (i.e., write commands) are not interleaved. However, read command packets may be interleaved in between the words of a write command packet. The L1 CFLAG signal is used to distinguish between header words and data words.

The response channel carries read data, read response, and write response packets. <FIG> show the memory-mapped HBI L2 response channel packet formats 1600A, 1600B, 1600C, according to an example. Response packets have no header, and each word can be independently identified and routed. To achieve full read throughput in the presence of write traffic, simultaneous read and write responses may be provided in the response packets, as shown in <FIG>. As shown in <FIG>, each read response word may use <NUM> bits for data; <NUM> bits for AXI Read ID (RID); <NUM> bits for the response type (RRESP); <NUM> bit for last word indication (RLAST); and <NUM> bit to indicate read valid response (RV). As shown in <FIG>, each write response word uses <NUM> bits for the AXI Write ID (WID); <NUM> bits for the response type (WRESP); and <NUM> bit to indicate write valid response (WV).

For maximum read throughput, the memory-mapped HBI L2 response channel may sustain a read response every cycle. The read response is allocated <NUM> bits of data and <NUM> bits of the L1 user side-channel. For maximum write throughput, a write response is may be performed at most once per two cycles, since the shortest write packet has one header word and one data word, and takes two cycles to transmit. Therefore, the write response can be transmitted over two cycles without loss of throughput. The write response channel may be allocated <NUM> bits of the L1 user side channel; <NUM> bit to mark the response start; and <NUM> bits for the first or second half of the <NUM>-bit write response. The read response words (e.g., with different AXI ID) may be interleaved.

As discussed above, another example of the HBI L2 is a streaming protocol (e.g., such as an AXI4 (L2s) protocol). The streaming HBI L2 protocol may use a <NUM>-bit AXI-S interface at <NUM> mapped to one L1 channel. Thus, one HBM PHY unit can support <NUM> such AXI-S interfaces. The interface may be configurable as a master (outbound) or a slave (inbound). The streaming HBI L2 protocol may support credit-based flow control, full throughput (e.g., no packetization overhead), two mode of operation (e.g., a "Normal" mode and a "Simple" mode). The streaming HBI L2 protocol creates a <NUM>-bit data stream. The AXI valid-ready handshake is replaced by credit-based flow control, and all other AXI-S signals are carried over the available <NUM> bits of the L1 user side channel. The Table <NUM> in <FIG> shows the streaming HBI L2 protocol signal mapping.

The streaming HBI L2 protocol may not support the TSTRB signal. The TKEEP signal may be supported. In some examples, the TKEEP signal allows a streaming packet to start and end on an unaligned boundary, but otherwise the packet must contain a contiguous stream of valid bytes. In the first word of an AXI-S packet (TLAST=<NUM>), TKEEP indicates the location of the first valid byte; in the last word of an AXI-S packet (TLAST=<NUM>), TKEEP indicates the location of the first invalid byte; in other packet words TKEEP should not be used. <FIG> is a table <NUM> showing the encoding of the TKEEP signal for the streaming HBI L2 protocol. The TKEEP encoding is done by the streaming HBI L2 protocol, but the user may ensure compliance with the restrictions.

TID may be the source ID. The TID may be useful if multiple streams are interleaved onto a single physical channel. The TDEST is the destination ID. The TDEST may be used to route streaming packets to their final destination. Depending on the application, either TID or TDEST may or may not be required. A total of <NUM> bits are allocated for both TID and TDEST. The user may choose one of the static configurations shown in the Table <NUM> in <FIG>, depending on the application.

The streaming HBI L2 protocol "simple" mode, may be a subset of AXI-S in which only flow control is provided. In some examples, the simple mode may be a point to point, single-source to single-destination stream, and provide continuous flow of whole words. In the simple mode, TID/TDEST, TSTRB/TKEEP, and TLAST signals may be omitted. Instead, the user may be given the full available <NUM> bits of the side channel as TUSER bit, to be used for any purpose.

For the HBI L2 IP, the L2 function may be implemented as soft logic IP in the PL on the side of the host device1 <NUM>.

The dies connected by the HBI, such as the host device <NUM><NUM> and the host device(s) <NUM>. 204n, may be reset and initialized independently. For HBI initialization, calibration, and data flow initiation it is assumed that there is one or more controller entities (e.g. a CPU) responsible for sequencing the process. The controller entities can be on-chip or off-chip, and the communication between the <NUM> controller entities is done out-of-band (i.e., not via the HBI). For example, there may be a simple micro-controller on each die, and some message passing interface between the dies (e.g., such asl<NUM>C, SPI, or Ethernet).

The HBI activation steps may include initialization, configuration, link training, FIFO training, and link activation. For the initialization step, the HBI logic (including the PHY) is powered up, reset, provided with a stable clock, and taken out of reset and into the idle, inactive state. For the configuration step, runtime programmable features of the HBI may be initialized with desired values. For example, this may include channel direction, parity, DBI, PHY initialization, self-calibration, and redundant wire assignment, etc. For the link training step, each L0 channel configured as output transmits a special training pattern that allows the receiving L0 channel on the other die to center the DQS edge relative to the DQ (data) eye. For the FIFO training step, each L0 channel configured as output transmits a special incrementing pattern that allows the receiving L0 channel on the other die to adjust the receive FIFO such that the FIFO operates near the half full point, providing the most tolerance to jitter. In applications where low latency is desired, the FIFO level may be trained to a different point to reduce latency. For the link activation step, when all previous steps are successfully completed, the data flow may begin. The L1 function may start issuing idle data words and
the DQS will toggle continuously. Then user-side traffic can be enabled and real data may start flowing across the HBI.

The HBI-based system may operate as a mesochronous network. For example, the HBM-related clocks on both dies (interconnected by the HBI) may run at the same frequency, but with unknown phase relationships. This may be achieved by both die sharing the same reference clock used by the PLL in the HBM PHY (or equivalent).

The transmitted data may be source-synchronous. For example, the clock, or DQS is sent along with the data from transmitter to receiver. In addition, phase and jitter variations may be absorbed in the receive FIFO which is part of the PHY. The HBM channel clock and clock enable signals (CK_t, CK_c, and CKE) may not be used. Long-term jitter variations between the dies may be controlled such that they do not exceed a level which could overflow or underflow the PHY receive FIFO. For example, the long-term jitter of the <NUM> clock may be maintained such that is does not exceed <NUM> UI (<NUM> ps).

Coarse grain power management for the HBI may be achieved by the external controller entities terminating activity on both die and then powering down the HBI link.

The HBM micro-bump and ballout arrangement has been selected for ease of routing between the master device and the HBM stack. In HBI systems, when both devices (i.e., the host devices interconnected by the HBI) may have the same orientation of the PHY ballout when placed on the interposer, the die-to-die wiring may be simple. For example, the die-to-die wiring may follow signal routing as in the HBM protocol between master device and HBM stack device. When one die is rotated, wiring becomes more complex. HBI may support both same-orientation, and rotated die cases. <FIG> illustrates an example ball layout for HBI following the HBM protocol, with data flowing horizontally from chip to chip. <FIG> illustrates an example ball layout for HBI following the HBM protocol for single HBM DWORD using <NUM> wires, with data flowing vertically from chip to chip. The difference between a master and a slave PHY is only the direction of the unidirectional signals as shown in the Table <NUM> in <FIG>.

When both die have the same orientation, the connections may be <NUM>-to-<NUM> (e.g. DQS is connected to DQS, etc.), with the exception of the WDQS_t/c of one chip may be connected to the RDQS_t/c of the other chip, and vice versa. For example, the read and write DQS may be crossed. The HBI may not use a DERR.

When one die is rotated, maintaining the same wiring may lead to long wires and complex interposer routing. In some examples, the HBI may use a <NUM>-to-<NUM> wiring on the interposer as shown in <FIG> and the Table 2300B in the <FIG>, with bit reordering done at the L1 module to undo the swapping of bits done on the interposer.

The HBI may handle redundant data wires according to the HBM standard. The HBM standard defines <NUM> redundant data wires per <NUM> data bits, or <NUM> redundant bits per DWORD. Two lane remapping modes are defined, as detailed below. In HBI, the redundant data wires can be used for lane repair only when both die have the same orientation.

In Mode <NUM> it is allowed to remap one lane per byte. No redundant pin is allocated in this mode, and DBI functionality is lost for that byte only; however, other bytes continue to support DBI function as long as the Mode Register setting for DBI function is enabled. If the Data Parity function is enabled in the Mode Register and a lane is remapped, both DRAM and host may assume DBI input as "<NUM>" for parity calculation for Read and Write operation in this mode. In Mode <NUM> each byte is treated independently.

In Mode <NUM>, one lane per double byte may be remapped. One redundant pin per double byte is allocated in this mode, and DBI functionality is preserved as long as the Mode Register setting for DBI function is enabled. Two adjacent bytes (e.g. DQ [<NUM>:<NUM>])) may be treated as a pair (double byte), but each double byte is treated independently.

Certain signals, such as the WDQS_c, WDQS_t, RDQS_c, RDQS_t, PAR, and DERR signals may not be remapped. In mode <NUM>, the DBI signal is lost; so DBI pins cannot be interchangeable with other pins. Therefore for the rotated die case, where DBI is wired to DM, mode <NUM> may not be used. In mode <NUM>, no functionality is lost but PAR cannot be remapped, so mode <NUM> may not be used for rotated die.

<FIG> is a flow diagram illustrating example operations <NUM> for communication between devices on an interposer. Operations <NUM> include, at <NUM>, sending at least a first signal from a first device on the interposer to a second device on the interposer via a HBI. Sending the first signal via the HBI includes sending the first signal using a layered protocol. The layered protocol includes a physical layer protocol that is configured according to a HBM physical layer protocol. Operations <NUM> include, at <NUM>, receiving at least a second signal from the second device on the interposer via the HBI.

Claim 1:
A computing system (<NUM>), comprising:
a first host device (<NUM>) comprising a first die on an interposer (<NUM>); and
at least one second host device (<NUM>) comprising a second die on the interposer (<NUM>), wherein:
the first host device (<NUM>) and the at least one second host device (<NUM>) are interconnected via at least one high bandwidth interface, HBI, (<NUM>) that configured to implement a layered protocol for chip-to-chip communication between the first host device (<NUM>) and the at least one second host device (<NUM>), the layered protocol includes a protocol layer (<NUM>), a transport layer protocol (<NUM>), and a physical layer protocol, wherein the physical layer protocol is configured according to a high bandwidth memory, HBM, physical layer protocol (<NUM>), wherein the first host device and the at least one second host device are configured to communicate via the at least one HBI (<NUM>), wherein the transport protocol layer (<NUM>) is configured to provide parity protection and data bit inversion, DBI, functionality and credit-based flow control, and intra-channel framing and alignment.