Deterministic operation of an input/output interface

In one embodiment, the present invention includes a method for receiving data from a second device in a first device, forwarding the data from an input/output (I/O) clock domain to a system clock domain of the first device, and providing the data to a functional unit of the first device at a deterministic time. In such manner, the two devices may operate in lockstep fashion. Other embodiments are described and claimed.

BACKGROUND

Embodiments of the present invention relate to data communication and more particularly to deterministic data transfer between connected devices.

Most computer systems are formed of components coupled together using one or more buses, which are used to transmit information between the various system components. Present bus standards such as the Peripheral Component Interconnect (PCI) Specification, Rev. 2.1 (published Jun. 1, 1995) provide a multi-drop bus in which multiple devices are coupled to the same bus. Accordingly, it is easy to read or write to devices on the same bus. However, as bus interface speeds increase, bus architectures are moving away from multi-drop architectures towards point-to-point architectures. In point-to-point architectures, peer-to-peer communication becomes more difficult as synchronization, ordering, and coherency of such communications becomes more difficult. One example of a point-to-point architecture is a PCI Express™ architecture in accordance with the PCI Express Base Specification, Rev. 1.0 (published Jul. 22, 2002).

Communication between serially connected devices typically involves buffering data to be sent in a transmitting device and then sending the data, for example, in packetized form to a receiving device. The two communicating devices, which may be different integrated circuits of a system, are typically connected via off-chip links to communicate data between the devices. Such links can be used for inter-processor communication or communication from an agent to memory, as two examples. Often data from a clock-forwarded off-chip interface can arrive at an agent such as a processor (and more particularly into a processor's core) non-deterministically, that is, at an arbitrary execution cycle. Non-deterministic operation can cause various problems. For example, debug of an integrated circuit on a high-speed tester becomes difficult if the chip does not respond identically each time test code is run. Likewise, debug on a system platform becomes difficult if the chip contains intrinsic execution uncertainties. Also, scan techniques such as restart/replay can become confused. Furthermore, it is impossible to create a lockstep system, where two processors operate side-by-side and constantly compare results. A need thus exists for improved system operation, including deterministic transmission of data.

DETAILED DESCRIPTION

In various embodiments, deterministic operation of a clock-forwarded (i.e., source-synchronous) input/output (I/O) interface may be provided. As used herein, deterministic operation refers to the arrival of data at a functional unit of a processor, microcontroller, or other agent at a known cycle relative to some fixed event, for example, a de-assertion of a reset signal that initiates execution of code in the agent.

Using embodiments of the present invention, a device such as a processor including one or more cores, memory, an I/O system and other functionality (collectively a processor), is not sensitive to non-determinism at an input, such as at the processor's pins. While a processor may contain multiple non-phase-aligned clock domains with on-chip clock-forwarded (source-synchronous) interfaces between them, transfer of data between these domains may occur deterministically. Thus data transfers may be insensitive to both an absolute delay between domains and to dynamic variations in the phase difference between the domains.

To achieve determinism regardless of a static cycle-based uncertainty with regard to receive data at an agent's input, properties of one or more initialization protocols may be used to embed information that allows logic in an I/O portion of the agent to examine the incoming data and adjust the latency through the I/O portion based on the incoming data.

Across on-chip clock-forwarded interfaces between clock domains, first-in first-out (FIFO) buffers may be used to achieve deterministic data transfer. On a receive side, initialization of a FIFO (i.e., release of a FIFO load pointer) may occur with a looped-back version of a selected signal, such as a reset signal. Release of a FIFO unload pointer, on the other hand, may be controlled from the domain that generates the original version of the selected (e.g., reset) signal, which guarantees determinism across this clock domain crossing.

Referring now toFIG. 1, shown is a block diagram of a portion of a device in accordance with one embodiment of the present invention. More specifically,FIG. 1shows a block diagram of on-chip components of an I/O portion of a device such as a bus agent coupled to an off-chip link. While a bus agent may take various forms, in some embodiments the bus agent may be a processor, controller, I/O hub, bridging agent or the like. As shown inFIG. 1, an I/O system10of an agent includes various components. Groups of these components may form different clock domains of the I/O system. While described herein as different components, it is to be understood that one or more such components may be handled by the same or similar hardware for performing the desired functionality.

As shown inFIG. 1, a system interface20(on the left side ofFIG. 1) represents the data interface into a networking link and higher layers of the agent. For example, inter-processor data may interface with a networking router (not shown inFIG. 1), while memory data may interface with a memory controller (not shown inFIG. 1). These two entities pass data to other parts of the device, including one or more execution cores or other core logic. System interface20may operate at a first clock domain which corresponds to a system interface domain. As one example, system interface20may operate using a fractional clock data transfer (FCDT) domain.

Still referring toFIG. 1, a routing path40of I/O system10, which may include a transmit path44and a receive path57, may be an on-chip clock-forwarded interface to couple different clock domains together. Thus routing path40may provide routing between system interface20and an I/O physical layer60. Because these two entities are physically distant from one another, this on-chip clock-forwarded interface provides for data transfer between them. The transfer mechanism of routing path40may dramatically reduce power compared to a full-chip single clock domain solution. However, in other embodiments, such an on-chip clock-forwarded interface may not be present, and data may pass directly from a system interface to an I/O physical layer, in some embodiments. In addition to data, it is to be understood that routing path40may additionally provide a path for routing of additional signals, including clock signals, control signals, a reset signal and the like.

I/O physical layer60operates at a different clock domain than system interface20. Specifically, I/O physical layer60may operate at a clock domain based upon a received clock signal, which may be received from another device, such as another processor or other agent. I/O physical layer60includes I/O circuitry and logic to transmit to and receive data from an off-chip interconnect. The transmit path is shown along the top ofFIG. 1, while the receive path is shown along the bottom ofFIG. 1.

In certain components of I/O system10, non-determinism may be caused by analog delays that vary with process, voltage and temperature (PVT). For example, routing path40may have delays that vary with PVT. Similarly, off-chip links to which I/O system10are coupled also may have non-deterministic analog delay that varies with PVT. Still further, a clock of I/O physical layer60(e.g., PCLK74) may also display analog delay non-determinism, in some embodiments. Other parts of I/O system10may have discrete cycle non-determinism. That is, these portions may have propagation delays that may vary for each reset event. However, using embodiments of the present invention, upon initialization the non-determinism may snap to one of several possible cycles. As examples, a synchronizer70of a transmit FIFO65and a synchronizer83of a receive FIFO80may have such determinism.

As shown inFIG. 1, data to be transmitted from an agent is provided to system interface10via a latch or flip-flop21. Flip-flop21is controlled by a system interface clock (i.e., SysInt_clk) to pass the data through routing path40. Specifically, data sent from flip-flop21is provided through a delay line formed of a plurality of inverters42a-42dto I/O physical layer60. Similarly, the system interface clock is also provided through routing path40to I/O physical layer60through a delay line formed of a plurality of inverters46a-46d.

System interface10also receives a reset signal, which is passed though a flip-flop22and is provided to routing path40. Through routing path40, the reset signal is passed through a delay line formed of a plurality of inverters48a-48dand provided to I/O physical layer60. As further shown inFIG. 1, the reset signal is also coupled to a counter24, which in turn is coupled to a receive FIFO25(also referred to herein as an FCDT FIFO).

Still referring to the transmit path of I/O system10, from routing path40data to be transmitted is passed through a flip-flop62and into a transmit buffer65that includes a plurality of storage registers (two of which are shown as a first register66and a second register67inFIG. 1). Transmit buffer65is controlled through two independent pointers, one for loading of data and one for unloading of the data. As shown inFIG. 1, a load pointer68is used to load data into a selected one of the registers. Load pointer68is controlled via the delayed reset signal, received at load pointer68from a flip-flop64synchronized by a forwarded clock (FWD_CLK) obtained via an inverter63. Data is unloaded from a selected register of buffer65according to an independently controlled unload pointer69. Unload pointer69is controlled by a signal, ResetP, obtained from a flip-flop72coupled to a synchronizer70, which in turn is coupled to receive the forwarded reset signal via flip-flop64. Data output from transmit FIFO65is coupled through a transmitter90and is provided to an output pin (i.e., tx_data).

In a clock-forwarded interface, I/O inputs include data pins and a clock pin. The clock pin continuously toggles, providing edges indicating when data is valid. On-chip circuits receive, distribute, and phase align the incoming forwarded clock to data receivers, where it is used to capture the incoming data. Because of PVT delay variations in a transmitter, the phase of the incoming clock and data at the receiver, and therefore the phase of the receive clock, may vary with respect to the phase of the I/O logic clock (PCLK74inFIG. 1) and core logic clock (sysint_clk ofFIG. 1) in the receiver. Where two clocks have the same average frequency but are not aligned with a known phase relationship, a mesochronous system exists. Likewise, the term mesochronous can also be used to describe any system where clocks are aligned with a maximum phase deviation, including those with variable frequency.

Still referring toFIG. 1, incoming data from an off-chip link is provided to I/O system10via data pads, one of which is shown inFIG. 1as coupled to receive data (rx_data). I/O system10further is coupled to receive a forwarded clock from the off-chip link. As shown inFIG. 1, this forwarded clock (i.e., rx_clk) is provided through a pin and a buffer92. The incoming data is coupled through a flip-flop94controlled by the incoming clock and is provided to a receive FIFO80. Receive FIFO80includes a plurality of storage registers to store data received via off-chip links. As shown inFIG. 1, receive FIFO80includes a first register87and a second register86(of course additional registers may also be present). As further shown inFIG. 1, receive FIFO80is controlled through two independent pointers, one used for loading of data and one for unloading of data. A load pointer89is synchronous with the receive clock domain (i.e., rx_clk), advancing when new data is ready to be loaded. In contrast, an unload pointer88is synchronous with the device's core clock domain, advancing when it is ready to see new data. Thus as shown inFIG. 1, unload pointer88is controlled by the synchronized reset signal (ResetP), discussed above. In various embodiments, receive FIFO80further includes a synchronizer83to transfer the synchronized reset signal to unload pointer88and initialize it. De-assertion of the reset signal starts the pointers advancing. Because the phase relationship between rx_clk and PCLK74is arbitrary, synchronizer83may thus resolve potential metastability issues in the rx_clk domain upon a transition of reset.

Provided unload pointer88follows load pointer89with adequate margin, the two domains can successfully transfer data. For example, FIFO80may load data into register6while unloading data from register2. Load pointer89can advance to load data into register7whenever new data is ready based on the incoming forwarded clock independently from unload pointer88advancing to read data from register3, based on its timing. The deviations of the phases of the two clocks should be limited to prevent the pointers from advancing past each other. In the case of a fixed clock frequency, deviations from the average frequency are to be limited. If one clock's frequency is greater than the other clock's frequency for long enough, it could still cause the load and unload pointers to cross, but this situation can be analyzed and solved through good clock frequency control. In a system with variable frequency, the phases of the two clocks should track with limited phase deviations between the two clocks.

Still referring toFIG. 1, received data is passed from FIFO80through a deskew shift register71and a buffer73. From there, serial data is coupled through a flip-flop53, which in turn is coupled through routing path40via a delay line formed of a plurality of inverters56a-56d. The received data is then passed through a flip-flop33and is provided to FIFO25, which includes a plurality of storage registers, two of which are shown inFIG. 1as a first register26and a second register27. Furthermore, FCDT FIFO25includes an independently controlled unload pointer28and a load pointer29. The data output from FIFO25is passed through a flip-flop23and is provided to the other circuitry of the agent.

Load pointer29is controlled by a forwarded (i.e., looped back) version of the synchronized reset signal, ResetP, generated in synchronizer70. More specifically, the synchronized reset signal, ResetP, is provided via a flip-flop51and a reset delay path of routing path40via a plurality of inverters54a-54dand through a flip-flop32(as reset_af_h) to control load pointer29. Unload pointer28, in contrast, is controlled by counter24which receives the direct version of the reset signal. Flip-flop32is in turn controlled by a looped-back version of the I/O logic clock signal (PCLK74) via a delay line formed of a plurality of inverters52a-52d.

As will be discussed further below, along the receive data path, incoming data is provided to a pattern detect circuit78as deskew_data. Pattern detect circuit78is also coupled to receive a clock signal from PCLK74and a count value from a counter76. In turn, pattern detect circuit78controls a deskew pointer79, which in turn controls the unloading of received data from buffer73. In such manner, an appropriate delay may be added to the data path to resolve static cycle-based uncertainty.

In a given system, lack of determinism may arise from four main causes, in part from the clocking architecture used for a large device and in part from the intrinsic characteristics of a clock-forwarded I/O interface. As illustrated inFIG. 1, data arriving from off-chip into I/O system10crosses between a receiving clock domain and the I/O logic clock domain (via receive FIFO80). Second, this data passes through another clock domain crossing between I/O physical layer60and core logic (i.e., system interface20). Third, transmit data must cross from the core clock domain into the I/O logic clock domain (i.e., transmit FIFO65ofFIG. 1). Fourth, delays in the on-chip receive and transmit data paths will vary with process, voltage, and temperature conditions. Embodiments of the present invention may overcome these causes to yield deterministic, lockstep operation.

In various embodiments, an I/O system may provide deterministic operation whereby instructions and data appear at a core or other logic at a known execution cycle that is reproducible every time the device is powered on. To provide for deterministic operation, it may first be noted that an I/O system is not sensitive to non-determinism at the agent's pins. That is, no part of the system examines the “state” of the pins. Although in a functioning system the data at the pins is captured by a receiving chip, the I/O interface does not examine this data. Instead, it merely passes the data to a system interface, which does examine it. Therefore, transmit uncertainties do not matter as long as they can be compensated for in the logic receiving the uncertain timed data. Even a high-speed functional tester, which captures and examines a device's outputs, will not be sensitive to some amount of uncertainty in the data transmitted to it. These testers capture data from the device, but do not respond to it. They simply provide a pre-programmed sequence of bits to the device's input pins while receiving data from the device's outputs, which is then compared with an expected sequence. More sophisticated testers can align data received from the device to the expected sequence, adjusting the alignment based on an expected pattern that occurs during initialization of the I/O link. Appearance of output data to the tester at a pre-determined tester cycle is therefore not necessary.

Accordingly, some amount of non-determinism may be permitted in the transmit data path. Thus, along with data provided from system interface20to I/O physical layer60, the delay-matched reset wire formed of inverters48a-48dwithin path40may be used to release load pointer68of transmit FIFO65, as discussed above. This synchronously propagated reset signal releases load pointer68and is also provided to a synchronizer70and another flip-flop72, where it is passed into the PCLK domain, and releases unload pointer69. The cycle uncertainty through synchronizer70can cause the output data to appear at a different absolute cycle with each power up of the device, however nothing is sensitive to absolute time determinism at the output pins. Upon release of unload pointer69, output data from transmit FIFO65is coupled through a transmitter90and is provided to the output pins (one of which is shown in the embodiment ofFIG. 1).

In the transmit direction, the delay variation with PVT of the on-chip forwarded clock path from system interface20to I/O physical layer60through routing path40may be handled in one of two ways depending on the clock configuration in I/O physical layer60. In one implementation, PCLK74is derived from an independent off-chip source that is mesochronous with the on-chip forwarded clock. In this case, the on-chip data path delay will result in a phase variation of up to approximately one cycle between the two clocks. As long as unload pointer69trails load pointer68by at least this amount, the phase variation will be absorbed in transmit FIFO65. In the other implementation, PCLK74is derived from FWD_CLK through a phase-locked loop (PLL). In this case, there will only be a small amount of variation, much less than one cycle, between FWD_CLK and PCLK74. The delay variation in the on-chip forwarded clock path will appear as variation in the timing of the output data. This variation will not be common between the transmitted forward clock, appearing at the inputs of a receiving device and that device's I/O clock. Therefore, the transmit data's PVT variation may be absorbed in the receiving device's receive FIFO. Because FIFOs (i.e., transmit FIFO65and a receiving device's receive FIFO) absorb the dynamic timing variation, data transmitted from I/O system10will arrive at an I/O system of the receiving device with only static cycle-based uncertainty that occurs through the initialization of transmit and receive FIFOs.

Referring now toFIG. 2, shown is a flow diagram of a method of controlling a transmit data path in accordance with an embodiment of the present invention. As shown inFIG. 2, method100may begin by receiving a reset signal in an I/O system of a device (block110). While described in the embodiment ofFIG. 2as being a reset signal, it is to be understood that in various embodiments any one of a number of selected control or status signals may be used for effective transmit path determinism.

Still referring toFIG. 2, next a load pointer of a transmit FIFO may be released according to a synchronously propagated reset signal (block120). For example, the reset signal received in the I/O system may be synchronously propagated to an I/O physical layer in which the transmit buffer resides. Next, an unload pointer of the transmit FIFO may be released according to a synchronized version of the synchronously propagated reset signal (block130). That is, the propagated reset signal may pass through a synchronizer chain of the I/O physical layer before it releases the unload pointer, to remove the potential of metastability. In doing so, the delay of a rising or falling edge though the synchronizer may vary by one or more cycles. In various embodiments, independently controlling the unload pointer to trail the load pointer by at least an amount equal to a phase variation between the clock domain of the device's core logic and the I/O physical layer's clock domain may provide for deterministic behavior. Accordingly, the transmit buffer is ready to load and unload data to be transferred.

Thus still referring toFIG. 2, the transmit FIFO may receive data to be transmitted from the device (block140). For example, such data may be sent from a core of the device (e.g., a processor core) and through a system interface portion of the I/O system (and in some embodiments a routing path) to the transmit buffer of the I/O physical layer. This data may be stored in the transmit FIFO in a register pointed to by the load pointer (block150). Finally, the data may be output from the transmit FIFO under control of the unload pointer (block160).

Referring back toFIG. 1, to create determinism in a receive data path, FCDT FIFO25may be appropriately controlled. The initialization of FIFO25(i.e, release of load pointer29) occurs via a looped-back version of the reset signal (reset_af_h) received from routing path40from I/O physical layer60. This causes the FIFO load to be totally deterministic with respect to reset de-assertion in I/O physical layer60. For example, receive data sent to system interface20eight cycles after reset de-assertion in I/O physical layer60will always be placed in entry eight of FIFO25. More generally, I/O input data from cycle N will always be placed in FIFO entry N modulo the depth of FIFO25.

Release of unload pointer28, on the other hand, may be controlled from system interface20, which guarantees determinism across this clock domain crossing. One implementation is for reset de-assertion in the system interface20to start a counter running in system interface20. When counter24, which receives the reset signal, reaches a predetermined or trigger value, it releases unload pointer28. In various embodiments, the trigger value may be slightly larger than the longest round trip delay for the reset signal to travel from system interface20to I/O physical layer60and back to release load pointer29. Thus, load pointer29releases prior to unload pointer28, for proper FIFO operation. In these embodiments, the total depth of the FIFO25may have a depth sufficient to cover dynamic variation (drift) plus initialization uncertainties in the path traveled by the reset signal.

FIFO25may thus be completely deterministic for I/O physical layer60to system interface20data transfers, regardless of PVT variations in the on-chip clock-forwarded path delays. For example, assume counter24is set to release unload pointer28twelve cycles after reset de-asserts in system interface20. Suppose further that it takes eight cycles for reset to make its round trip and release load pointer29. I/O data placed in FIFO entry zero will be stored eight cycles after reset de-asserts in system interface20, but will be unloaded twelve cycles after reset de-asserts. Now suppose on another device it takes ten cycles for reset to make this round trip. Data will thus be placed in FIFO entry zero ten cycles after reset de-asserts in system interface20, but still unloaded in cycle twelve. As demonstrated, the distance between the load and unload pointers is irrelevant and data transfer from I/O physical layer60to system interface20is deterministic with respect to reset de-assertion in both regions.

Referring now toFIG. 3, shown is a flow diagram of a method of controlling data flow in a receive data path of a device in accordance with an embodiment of the present invention. As shown inFIG. 3, method200may begin by receiving a reset signal in an I/O system (block210). While described in the embodiment ofFIG. 3as being a reset signal, it is to be understood that in various embodiments any one of a number of selected control or status signals may be used for effective receive path determinism. Next a counter in the system interface portion of the I/O system may be initiated upon receipt of the reset signal (block220). As will be described further below, this counter may be used to control release of an unload pointer of a receive FIFO.

Still referring toFIG. 3, next a load pointer of the receive FIFO may be released according to a looped-back version of the reset signal (block230). This looped-back reset signal is a propagated version of the reset signal having passed from its initial receipt at the system interface through a routing path (if present) and into the I/O physical layer of the I/O system, and then routed in a feedback fashion back to the system interface, where it is used to release the load pointer.

Next, it may be determined whether the counter has reached a predetermined value (diamond240). In various embodiments, the predetermined value may be a trigger count at which a trigger signal is sent from the counter. In various embodiments, the predetermined value may be at least equal to the round-trip delay for the reset signal to loop back (as discussed above) to the system interface portion. If not, a loop occurs until the counter reaches its trigger value. Next, an unload pointer of the receive FIFO may be released under control of a trigger signal from the counter (block250). Accordingly, the load and unload pointers of the receive FIFO are independently controlled. Accordingly, data transfer between the I/O physical layer and the system interface of the I/O system is deterministic with respect to de-assertion of the reset signal.

As a result, incoming data to a device may be received and stored in a receive FIFO of the system interface deterministically. Specifically, data received from another device is provided to the system interface receive FIFO (block260). As discussed above, this data may be received from another agent, such as another processor of a multiprocessor system and be passed through an I/O physical layer and through a routing path (if present) into the system interface receive FIFO. This received data may then be stored in the system interface receive FIFO at a location pointed to by its load pointer (block270). Finally, the data may be output from the system interface receive FIFO to core logic of the agent according to the unload pointer (block280).

To resolve static cycle-based uncertainty of receive data, discussed above, properties of an initialization protocol may be used to embed information that allows logic in an I/O system to examine incoming data and adjust the latency through the I/O system. At a high-level, as part of link initialization a certain value or sequence of values may be expected on an input data bus Q cycles after reset de-asserts. Logic in the I/O system may be used to examine the incoming data around this time. If the logic detects the expected data value(s) too early, it can add cycles to the data path through the I/O system until the actual delay matches the expected delay of Q cycles. If instead, the expected data values are detected too late, the I/O system may be reset and initialization may be performed again with a longer target latency Q.

Referring now toFIG. 4, shown is a flow diagram of a method of resolving cycle-based non-determinism in accordance with one embodiment of the present invention. As shown inFIG. 4, method300may be performed during initialization of a device, and more specifically during link initialization. During link initialization, training data and certain expected data patterns may be sent from a transmitting device to a receiving device. Before the receipt of such data, the receive device may be programmed with a target latency (i.e., an expected delay) (block310). For example, the receive device may be programmed according to a target link latency stored in a command and status register (CSR) of the device. Then, incoming data may be examined for one or more predetermined data values (block320).

Next, it may be determined whether the data values were received earlier than expected (diamond330). That is, it may be determined whether the data values were received earlier than the programmed target latency of the device. If not, the incoming data may be passed to receive chain circuitry. For example, the incoming data may be passed to a system interface portion of an I/O system, and more particularly to a receive FIFO (block340).

If instead at diamond330it is determined that the expected data values are received earlier than expected, control may pass to block350. There, one or more cycles may be inserted into the data path until the expected delay (i.e., the target latency) is reached (block350). Then control passes to block340.

While discussed with these particular operations in the flow diagrams ofFIG. 2-FIG.4, it is to be understood that in other embodiments, different manners of effecting deterministic behavior may be performed.

In one implementation, which may be used in a system implementation employing a point-to-point protocol interconnect protocol, a clock boundary indicator (CBI) counter cycles every 200 megahertz (MHz), representing the cycle of alignment with respect to the input reference clock. Although described with specific values for this particular embodiment for purposes of illustration, the scope of the present invention is not so limited. The value at which the counter wraps back to zero (i.e., the maximum value stored in the counter), Nmax-1, depends on the frequency of the I/O system. For example, the value may be 9 for 4.0 gigabits per second (Gb/s) operation, 11 for 4.8 Gb/s, and 15 for 6.4 Gb/s. Part of this initialization protocol allows the CBI on the transmit device and the target link latency, stored in a CSR, to be sent to the receiving device. The receiving device may then add the transmit device's CBI and the target link latency and take that result modulo Nmax. This result may be compared to the CBI on the receiving device. If the data comes too early, I/O logic may add pipe latch stages to the data until the correct CBI is reached. In such manner, this mechanism effectively links together the CBI values on the transmit and receive devices.

For example, consider the case of two components (A and B) where the target link latency is 24 cycles and Nmax is 16. Assume component A communicates a CBI value of 4 during link initialization, and it is received in component B with its count value equal to nine. To match the target link latency of 24 cycles, component B would need to receive the CBI value when its counter had a value of 4+24 MOD 16, or at a value of twelve. Since it was actually received at a count value of nine in this example, three extra cycles (i.e., 12−9) of latency may be injected by B's receiver to match the target latency.

In the embodiment shown inFIG. 1, incoming data may be provided as deskew_data to pattern detect circuit78. Pattern detect circuit78may look for the target link latency value and the CBI value received from a sending device. Furthermore, pattern detect circuit78may receive a CBI value of the device and another value from counter76. Based on these values, pattern detect circuit78may determine whether additional latency cycles should be inserted into the received data path. Accordingly, pattern detect circuit78may provide control signals to deskew pointer79, which in turn may be used to control the output of received data from buffer73accordingly.

In an implementation employing a fully buffered dual in-line memory module (DIMM) (FBD) architecture, since the memory channel always responds in a fixed number of cycles from the packets transmitted by the processor (fixed at initialization, but may vary from system to system, and from reset to reset), deterministic memory latency may be achieved by counting the number of cycles from command to response during initialization. This count is compared to the target latency, and additional delay may be added as described above.

Referring toFIG. 5, shown is a block diagram of a system in accordance with one embodiment of the present invention. As shown inFIG. 5, system410may be a multiprocessor system including a coherent interface in accordance with an embodiment of the present invention. That is, system410may represent any one of a desired desktop, mobile, server or other such platform, in different embodiments. In certain embodiments, interconnections between different components ofFIG. 5may be point-to-point interconnects that provide for coherent shared memory within system410, and in one such embodiment the interconnects and protocols used to communicate therebetween may form a coherent system. In such manner, multiple processors, memories, and other components of system410may coherently interface with each other.

Referring toFIG. 5, system410may include a first processor450(CPU A) and a second processor455(CPU B). In various embodiments, each processor may include memory controller functionality such that the processors may directly interface with an associated shared memory via a point-to-point interconnect. For example, as shown inFIG. 5, processor450may be coupled to a memory460(memory A) via a point-to-point interconnect and processor455may be coupled to a memory465(memory B) via a similar point-to-point interconnect. More so, processors450and455may be coupled to each other via a point-to-point interconnect. Using embodiments of the present invention, deterministic arrival of data between processors450and455may occur. Accordingly, these processors may operate in a lockstep manner in which the processors operate concurrently and continually compare results of certain operations. Similarly, each of processors450and455may be coupled via point-to-point interconnects to each of a first input/output (I/O) hub (IOH A)420and a second IOH430(IOH B).

In the embodiment ofFIG. 5, all components within box415may collectively form a coherent system (i.e., coherent system415). Such a coherent system may accommodate coherent transactions without any ordering between channels through which transactions flow. While discussed herein as a coherent system, it is to be understood that both coherent and non-coherent transactions may be passed through and acted upon by components within the system. For example, a region of one or both of memories460and465may be reserved for non-coherent transactions. While the embodiment ofFIG. 5shows a platform topology having two processors and two I/O hubs, it is to be understood that other embodiments may include more or fewer such components. For example, a single processor system may be implemented having a single processor, a single I/O hub and associated I/O devices coupled thereto. Alternately, a multiprocessor system having 4, 8, 16, 32 or another number of processors may be implemented, and an appropriate number of I/O hubs and other components may be coupled thereto. Any such platform topologies may take advantage of point-to-point interconnections to provide for coherency within a coherent portion of the system, and also permit non-coherent peer-to-peer transactions between I/O devices coupled thereto. Such point-to-point interconnects may thus provide multiple paths between components.

As shown inFIG. 5, I/O hubs420and430may each include a plurality of ports (e.g., ports421-424in IOH420and ports431-434in IOH430) to interface with I/O devices coupled thereto. For example, in certain embodiments, such I/O devices may be devices in accordance with one or more bus schemes. In one embodiment, such I/O devices may be PCI Express™ devices. For simplicity,FIG. 5shows a single I/O device coupled to each I/O hub, namely I/O device (I/O device A)440coupled via port421to IOH420and I/O device (I/O device B)445coupled via port431to IOH430. It is to be understood that the number of ports in an I/O hub in accordance with an embodiment of the present invention may vary, and the number of ports and devices coupled thereto shown inFIG. 5are for illustrative purposes only.

Also shown inFIG. 5is a legacy I/O controller hub (ICH)470coupled to IOH430. In one embodiment, ICH470may be used to couple legacy devices such as a keyboard, mouse, and Universal Serial Bus (USB) devices (e.g., devices in accordance with the USB Specification Rev. 2.0 (published December 2000)) to coherent system415.

While the I/O hubs shown inFIG. 5include a plurality of ports, it is to be understood that the hubs may realize various functions using a combination of hardware, firmware and software. Such hardware, firmware, and software may be used so that the I/O hub may act as an interface between coherent system415(e.g., shared memories460and465, processors450and455, and IOHs420and430), and devices coupled thereto such as I/O devices440and445. In addition, the I/O hubs ofFIG. 5may be used to support various bus or other communication protocols of devices coupled thereto. IOH420and IOH430may act as agents to provide a central connection between two or more communication links. In particular, IOH420and IOH430may be high-speed link agents that provide a connection between different I/O devices coupled to coherent system415. In various embodiments, other components within coherent system415may also act as such agents.

In various embodiments, each port of I/O hubs420and430may include a plurality of channels, often referred to herein as “virtual channels” that together may form one or more virtual networks and associated buffers to communicate data, control and status information between various devices. In one particular embodiment, each port may include up to at least six such channels.

Further, while discussed herein as being used within a coherent system, it is to be understood that other embodiments may be implemented in a non-coherent system to provide for deadlock-free routing of transactions. In some embodiments, the channels may keep traffic separated through various layers of the system, including, for example, physical, link, and routing layers, such that there are no dependencies.

Embodiments may be implemented in a computer program. As such, these embodiments may be stored on a medium having stored thereon instructions which can be used to program a system to perform the embodiments. The storage medium may include, but is not limited to, any type of disk including floppy disks, optical disks, compact disk read-only memories (CD-ROMs), compact disk rewritables (CD-RWs), and magneto-optical disks, semiconductor devices such as read only memories (ROMs), random access memories (RAMs) such as dynamic RAMs (DRAMs) and static RAMs (SRAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), flash memories, magnetic or optical cards, or any type of media suitable for storing electronic instructions. Similarly, embodiments may be implemented as software modules executed by a programmable control device, such as a general-purpose processor or a custom designed state machine.