Implementing serial link training patterns separated by random data for training a serial link in an interconnect system

A method and circuit for implementing serial link training sequences, and a design structure on which the subject circuit resides are provided. A transmitter device transmits a training sequence (TS) pattern; then the transmitter device transmits random data for a predefined time duration. The steps of transmitting the TS-pattern, then transmitting the random data for the fixed time duration are repeated. A receiver device detecting a plurality of the TS-patterns separated by the predefined time interval of random data, performs receiver initialization steps. The receiver device performs a plurality of receiver initialization steps including, for example, acquiring byte lock, and a link width determination.

FIELD OF THE INVENTION

The present invention relates generally to the data processing field, and more particularly, relates to a method and circuits for implementing serial link training patterns separated by random data, and a design structure on which the subject circuit resides.

DESCRIPTION OF THE RELATED ART

High speed serial (HSS) links are used for communications in various known computer chips and computer systems. High speed serial links typically are initialized or trained by the transmission and reception of a special Link Training Sequence (LTS).

In conventional HSS link arrangements, the Link Training Sequence (LTS) typically includes repeatedly sending a fixed training sequence (TS)-pattern. A particular TS-pattern, and its possible variants, are repeatedly sent by the transmitter to convey information to the receiver and to allow the receiver to perform various initialization steps. These receiver initialization steps include, for example, acquiring byte lock, a link width determination, and lane deskew.

In such prior art arrangements, repeatedly sending a fixed pattern on a serial link can introduce problems, which it is desirable to avoid. Some problems resulting from the use of a fixed repeating pattern include: (a) lack of perfect DC balancing within the bit stream; (b) electromagnetic interference (EMI) radiation is concentrated on specific frequencies; and (c) difficulty of the receiver (Rx) in acquiring bit lock while receiving a repeating fixed pattern, as compared to an optimal random pattern.

A need exists for an effective method and circuit to implement enhanced serial link training sequences. Such method and circuit are needed so that many problems of conventional serial link training arrangements using fixed repeating training sequence patterns are substantially eliminated.

SUMMARY OF THE INVENTION

Principal aspects of the present invention are to provide a method and circuits for implementing serial link training patterns separated by random data, and a design structure on which the subject circuit resides. Other important aspects of the present invention are to provide such method, circuitry, and design structure substantially without negative effect and that overcome many of the disadvantages of prior art arrangements.

In brief, a method and circuits for implementing serial link training sequences, and a design structure on which the subject circuit resides are provided. A transmit device coupled to the serial link transmits a training sequence (TS) pattern, then the transmit device transmits random data for a predefined time duration. The steps of transmitting the training sequence (TS) pattern, then transmitting the random data for the fixed time duration are repeated.

In accordance with features of the invention, a multiple-path local rack interconnect system includes a plurality of interconnect chips, and a plurality of serial links connected between each of the plurality of interconnect chips. Each of the interconnect chips includes a link interface for implementing serial link training patterns separated by random data. The link interface includes a transmit device coupled to the serial link for transmitting data and a receiver device coupled to the serial link for receiving data. The transmit device transmits a training sequence (TS) pattern; then the transmit device transmits random data for a predefined time duration. The steps of transmitting the training sequence (TS) pattern, then transmitting the random data for the fixed time duration are repeated.

In accordance with features of the invention, the predefined time duration of the transmitted random data is about ten times longer than the time interval of the transmitted random data training sequence (TS) pattern. The periodicity of the random data generation is much larger than the overall time spent performing serial link training sequences.

In accordance with features of the invention, the receiver device detecting a plurality of the TS-patterns separated by the predefined time interval of random data, performs receiver initialization steps. The receiver device performs a plurality of receiver initialization steps including, for example, acquiring byte lock, a link width determination, and lane deskew.

In accordance with features of the invention, the transmitted training sequence (TS) patterns with the random data inserted between each TS-pattern enable acquiring bit lock by the receiver device in an optimized amount of time. The transmitted TS-patterns with the random data inserted between each TS-pattern provide enhanced DC balancing of the bit stream as compared to repeatedly sending TS-patterns.

In accordance with features of the invention, the transmitted TS-patterns with the random data inserted between each TS-pattern provide electromagnetic interference (EMI) radiation effectively distributed among various frequencies. The TS-patterns with the random data inserted between each TS-pattern enable a longer link training sequence as compared to a time-limit for repeatedly sending TS-patterns of prior art arrangements.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In accordance with features of the invention, circuits and methods are provided for implementing an enhanced Link Training Sequence (LTS) that substantially eliminates all risks introduced by sending a repeated fixed TS-pattern during the Link Training Sequence. The circuits and methods substantially eliminate any time-limit constraints for sending the Link Training Sequence.

Having reference now to the drawings, inFIG. 1A, there is shown an example multiple-path local rack interconnect system generally designated by the reference character100used for implementing serial link training patterns separated by random data in accordance with the preferred embodiment. The multiple-path local rack interconnect system100supports computer system communications between multiple servers, and enables an Input/Output (IO) adapter to be shared across multiple servers. The multiple-path local rack interconnect system100supports network, storage, clustering and Peripheral Component Interconnect Express (PCIe) data traffic.

The multiple-path local rack interconnect system100includes a plurality of interconnect chips102in accordance with the preferred embodiment arranged in groups or super nodes104. Each super node104includes a predefined number of interconnect chips102, such as 16 interconnect chips, arranged as a chassis pair including a first and a second chassis group105, each including 8 interconnect chips102. The multiple-path local rack interconnect system100includes, for example, a predefined maximum number of nine super nodes104. As shown, a pair of super nodes104are provided within four racks or racks 0-3, and a ninth super node104is provided within the fifth rack or rack 4.

InFIG. 1A, the multiple-path local rack interconnect system100is shown in simplified form sufficient for understanding the invention, with one of a plurality of local links (L-links)106shown between a pair of the interconnect chips102within one super node104. The multiple-path local rack interconnect system100includes a plurality of L-links106connecting together all of the interconnect chips102of each super node104. A plurality of distance links (D-links)108, or as shown eight D-links108connect together the example nine super nodes104together in the same position in each of the other chassis pairs. Each of the L-links106and D-links108comprises a bi-directional (×2) high-speed serial (HSS) link.

Referring also toFIGS. 1B and 1C, multiple interconnect chips102defining a super node104are shown connected together inFIG. 1B. A first or top of stack interconnect chip102, labeled1,1,1is shown twice inFIG. 1B, once off to the side and once on the top of the stack. Connections are shown to the illustrated interconnect chip102, labeled1,1,1positioned on the side of the super node104including a plurality of L-links106and a connection to a device110, such as a central processor unit (CPU)/memory110. A plurality of D links108or eight D-links108as shown inFIG. 1A, (not shown inFIG. 1B) are connected to the interconnect chips102, such as interconnect chip102, labeled1,1,1inFIG. 1B.

As shown inFIG. 1B, each of a plurality of input/output (I/O) blocks112, is connected to respective interconnect chips102, and respective ones of the I/O112are connected together. A source interconnect chip102, such as interconnect chip102, labeled1,1,1transmits or sprays all data traffic across all L-links106. A local I/O112may also use a particular L-link106of destination I/O112. For a destination inside a super node104, or chassis pair of first and second chassis group105, a source interconnect chip or an intermediate interconnect chip102forwards packets directly to a destination interconnect chip102over an L-link106. For a destination outside a super node104, a source interconnect chip or an intermediate interconnect chip102forwards packets to an interconnect chip102in the same position on the destination super node104over a D-link108. The interconnect chip102in the same position on the destination super node104forwards packets directly to a destination interconnect chip102over an L-link106.

In the multiple-path local rack interconnect system100, the possible routing paths with the source and destination interconnect chips102within the same super node104include a single L-link106; or a pair of L-links106. The possible routing paths with the source and destination interconnect chips102within different super nodes104include a single D-link108(D); or a single D-link108, and a single L-link106(D-L); or a single L-link106, and single D-link108(L-D); or a single L-link106, a single D-link108, and a single L-link106(L-D-L). With an unpopulated interconnect chip102or a failing path, either the L-link106or D-link108at the beginning of the path is removed from a spray list at the source interconnect102.

As shown inFIGS. 1B and 1C, a direct path is provided from the central processor unit (CPU)/memory110to the interconnect chips102, such as chip102, labeled1,1,1inFIG. 1B, and from any other CPU/memory connected to another respective interconnect chip102within the super node104.

Referring now toFIG. 1C, a chassis view generally designated by the reference character118is shown with a first of a pair of interconnect chips102connected a central processor unit (CPU)/memory110and the other interconnect chip102connected to input/output (I/O)112connected by local rack fabric L-links106, and D-links108. Example connections shown between each of an illustrated pair of servers within the CPU/memory110and the first interconnect chip102include a Peripheral Component Interconnect Express (PCIe) G3×8, and a pair of 100 GbE or 2-40 GbE to a respective Network Interface Card (NIC). Example connections of the other interconnect chip102include up to 7-40/10 GbE Uplinks, and example connections shown to the I/O112include a pair of PCIe G3×16 to an external MRIOV switch chip, with four×16 to PCI-E I/O Slots with two Ethernet slots indicated 10 GbE, and two storage slots indicated as SAS (serial attached SCSI) and FC (fibre channel), a PCIe×4 to a IOMC and 10 GbE to CNIC (FCF).

Referring now toFIGS. 1D and 1E, there are shown block diagram representations illustrating an example interconnect chip102. The interconnect chip102includes an interface switch120connecting a plurality of transport layers (TL)122, such as 7 TLs, and interface links (iLink) layer124or 26 iLinks. An interface physical layer protocol, or iPhy126is coupled between the interface links layer iLink124and high speed serial (HSS) interface128, such as 7 HSS128. As shown inFIG. 1E, the 7 HSS128are respectively connected to the illustrated 18 L-links106, and 8 D-links108. In the example implementation of interconnect chip102, 26 connections including the illustrated 18 L-links106, and 8 D-links108to the 7 HSS128are used, while the 7 HSS128would support28connections.

The TLs122provide reliable transport of packets, including recovering from broken chips102and broken links106,108in the path between source and destination. For example, the interface switch120connects the 7 TLs122and the 26 iLinks124in a crossbar switch, providing receive buffering for iLink packets and minimal buffering for the local rack interconnect packets from the TLO122. The packets from the TL122are sprayed onto multiple links by interface switch120to achieve higher bandwidth. The iLink layer protocol124handles link level flow control, error checking CRC generating and checking, and link level retransmission in the event of CRC errors. The iPhy layer protocol126handles training sequences, lane alignment, and scrambling and descrambling. The HSS128, for example, are 7×8 full duplex cores providing the illustrated 26×2 lanes.

InFIG. 1E, a more detailed block diagram representation illustrating the example interconnect chip102is shown. Each of the 7 transport layers (TLs)122includes a transport layer out (TLO) partition and transport layer in (TLI) partition. The TLO/TLI122respectively receives and sends local rack interconnect packets from and to the illustrated Ethernet (Enet), and the Peripheral Component Interconnect Express (PCI-E), PCI-E×4, PCI-3 Gen3 Link respectively via network adapter or fabric adapter, as illustrated by blocks labeled high speed serial (HSS), media access control/physical coding sub-layer (MAC/PCS), distributed virtual Ethernet bridge (DVEB); and the PCIE_G3×4, and PCIE_G3 2×8, PCIE_G3 2×8, a Peripheral Component Interconnect Express (PCIe) Physical Coding Sub-layer (PCS) Transaction Layer/Data/Link Protocol (TLDLP) Upper Transaction Layer (UTL), PCIe Application Layer (PAL MR) TAGGING to and from the interconnect switch120. A network manager (NMan)130coupled to interface switch120uses End-to-End (ETE) small control packets for network management and control functions in multiple-path local rack interconnect system100. The interconnect chip102includes JTAG, Interrupt Handler (INT), and Register partition (REGS) functions.

In accordance with features of the invention, a protocol method and circuit are provided for implementing an enhanced Link Training Sequence (LTS). A plurality of LTS operations includes separating the training sequence (TS) pattern by a predefined time interval and sending random data during that separation. The duration of each random data transmission is approximately ten times (10×) the amount of time spent sending each TS-pattern. The periodicity of the random data generation is much larger than the overall time spent performing link training.

Referring now toFIG. 2, there is shown a single high speed serial link circuit generally designated by the reference character200for implementing serial link training patterns separated by random data in accordance with the preferred embodiment. The single high speed serial link circuit200includes a respective HSS interface circuit designated by the reference character202included in each interconnect chip102, A and B connected by L-link106or D-link108. The HSS interface circuit202includes a respective transmit device203and a transmit Linear Feedback Shift Register (LFSR)204coupled to the transmit side of an L link106or a D link108of the high speed serial link circuit200. The HSS interface circuit202includes a respective receiver206coupled to a receive side of the L link106or the D link108of the high speed serial link circuit200. The high speed serial (HSS) link circuit200is implemented in the HSS interface128of the interconnect chip102shown inFIGS. 1D and 1E.

FIG. 3Aillustrates an example portion of a Link Training Sequence (LTS) generally designated by the reference character300performed by the high speed serial link circuit200ofFIG. 2. The LTS300includes a plurality of the training sequence (TS) patterns302separated by the predefined time interval with inserted random data304in accordance with the preferred embodiment. Each TS-pattern302includes a predefined link packet format containing header information and data with the header information identifying different TS-patterns from an initial or first TS-pattern302through a final TS-pattern transmitted in the Link Training Sequence (LTS).

FIG. 3Billustrates a prior art link training sequence including fixed repeating training sequence patterns. In the method of the invention by forcing a large proportion of the Link Training Sequence300to be random data304separating the TS-patterns302, advantages over prior art arrangements are gained.

In accordance with features of the invention, implementing an enhanced Link Training Sequence (LTS) of the invention substantially eliminates all risks introduced by sending repeated fixed TS-patterns during the Link Training Sequence (LTS). The circuits and methods of the invention provide improved DC balancing of the bit stream, effective EMI radiation distribution among various frequencies, and optimized time for obtaining Rx bit lock, while substantially eliminating any time-limit constraints for sending the Link Training Sequence (LTS) with the LFSRs204transmitting random data304separating the TS-patterns302.

For example, the duration of each random data transmission has a predefined fixed value, and should be at least approximately ten times the amount of time spent sending each TS-pattern in a preferred embodiment of the invention. However, it should be understood that the present invention is not limited to this example time duration, for example, a smaller or longer duration of each random data transmission relative to the amount of time spent sending each TS-pattern can be used.

Referring now toFIG. 4, there are shown exemplary operations performed by the high speed serial link circuit200for implementing the Link Training Sequence (LTS) with serial link training patterns separated by random data in accordance with the preferred embodiment. The LTS starts as indicated at a block400. A transmit (Tx) device203of chip102, A, transmits an initial TS-pattern302to a receive Rx device206of chip102, B, as indicated at a block402. The transmit LFSR204of chip102, A, transmits random data to the receive Rx device206of chip102, B, as indicated at a block404. During the LTS operations, the receive Rx device206of chip102, B looks for header information of TS-patterns separated by the predefined distance.

As indicated at a block406, the transmit device203and LFSR204of chip102, A, repeat the transmitting steps of blocks402and404, with transmitting the TS-pattern, then transmitting random data to the receive Rx device206of chip102, B, and the Rx device detects header information of TS-patterns separated by the predefined distance for a set number N consecutive TS-patterns separated by the predefined distance, and the Rx device performs set receiver initialization steps. The Rx device performs set receiver initialization steps including, for example, acquiring byte lock, a link width determination, and lane deskew. Checking whether the transmit device or LFSR204should advance to a next TS-pattern and the Rx device detects the current TS-pattern is performed as indicated at a decision block408. For example, advancing to a next TS-pattern is determined with the Rx device at the transmit device203receiving back the current TS-pattern and the current operations of the LTS step are done at decision block408. If the transmit device203should not advance to a next TS-pattern, then the LTS operations continue returning to block406and the transmit device203and LFSR204respectively repeat transmitting the current TS-pattern, and transmitting random data to the receive Rx device206.

When determined that the transmit device203should advance to a next TS-pattern and the Rx device detects the current TS-pattern, then checking whether the final TS-pattern is being transmitted as indicated by the header information of the current TS-pattern as indicated at a decision block410. If the final TS-pattern is not being transmitted, then the transmit device transmits the next TS-pattern continuing at block402. Otherwise the LTS operations continue as indicated at a block412.

FIG. 5shows a block diagram of an example design flow500that may be used for high speed serial link circuit and the interconnect chip described herein. Design flow500may vary depending on the type of IC being designed. For example, a design flow500for building an application specific IC (ASIC) may differ from a design flow500for designing a standard component. Design structure502is preferably an input to a design process504and may come from an IP provider, a core developer, or other design company or may be generated by the operator of the design flow, or from other sources. Design structure502comprises circuits102,200in the form of schematics or HDL, a hardware-description language, for example, Verilog, VHDL, C, and the like. Design structure502may be contained on one or more machine readable medium. For example, design structure502may be a text file or a graphical representation of circuits102,200. Design process504preferably synthesizes, or translates, circuits102,200into a netlist506, where netlist506is, for example, a list of wires, transistors, logic gates, control circuits, I/O, models, etc. that describes the connections to other elements and circuits in an integrated circuit design and recorded on at least one of machine readable medium. This may be an iterative process in which netlist506is resynthesized one or more times depending on design specifications and parameters for the circuits.

Design process504may include using a variety of inputs; for example, inputs from library elements508which may house a set of commonly used elements, circuits, and devices, including models, layouts, and symbolic representations, for a given manufacturing technology, such as different technology nodes, 32 nm, 45 nm, 90 nm, and the like, design specifications510, characterization data512, verification data514, design rules516, and test data files518, which may include test patterns and other testing information. Design process504may further include, for example, standard circuit design processes such as timing analysis, verification, design rule checking, place and route operations, and the like. One of ordinary skill in the art of integrated circuit design can appreciate the extent of possible electronic design automation tools and applications used in design process504without deviating from the scope and spirit of the invention. The design structure of the invention is not limited to any specific design flow.

Design process504preferably translates an embodiment of the invention as shown inFIGS. 1A-1E,2,3A, and4along with any additional integrated circuit design or data (if applicable), into a second design structure520. Design structure520resides on a storage medium in a data format used for the exchange of layout data of integrated circuits, for example, information stored in a GDSII (GDS2), GL1, OASIS, or any other suitable format for storing such design structures. Design structure520may comprise information such as, for example, test data files, design content files, manufacturing data, layout parameters, wires, levels of metal, vias, shapes, data for routing through the manufacturing line, and any other data required by a semiconductor manufacturer to produce an embodiment of the invention as shown inFIGS. 1A-1E,2,3A, and4. Design structure520may then proceed to a stage522where, for example, design structure520proceeds to tape-out, is released to manufacturing, is released to a mask house, is sent to another design house, is sent back to the customer, and the like.