Patent Description:
<CIT> relates to a packet-based scalable protocol that enables a number of memory and processing combinations, provides bit-efficient data transfer operations, and is concordant with a variety of bus types (e.g., electrical, optical). For example, a memory device includes a memory component that store data and a processor. The processor generates one or more data packets associated with the memory component. Each data packet includes a transaction type field that includes data indicative of a first size of a payload of the respective data packet and a second size of an error control code in the respective data packet. Each packet also has a payload field that includes the payload and an error control code field that includes the error control code. The processor transmits the data packets to a requesting component, such that the requesting component identifies the payload field and the error control field of each data packet based on the data of the transaction type field in each data packet.

The present disclosure provides technologies that allow for increased packet transmission efficiencies between electronic components connected by PCIe links. The increased efficiencies are possible through assembly of transaction layer packet (TLP) information and data link layer packet (DLLP) information into a "flit". Flits are sent across one or more lanes of a link, and each flit is protected by a flit-level cyclic redundancy code (CRC) scheme. Flits can be further protected by either per-lane or flit-level forwarded error correction (FEC) schemes. In per-lane FEC schemes, flit information sent across each lane of a link is protected with FEC codes sent across the same lane. In flit-level FEC schemes, one set of FEC codes protect the entire flit. The flit-level CRC schemes described herein can provide TLP transmission efficiencies greater than those in current PCIe protocol versions wherein a link CRC (LCRC) is transmitted with TLPs sent across a link. For example, a <NUM>-symbol flit may comprise eight flit-level CRC symbols to achieve a desired error detection probability against a target bit error rate, which is significantly less than the <NUM> LCRC symbols that would accompany the <NUM> TLPs transmitted across a link according to current PCIe protocol requirements. Further efficiencies arise from TLP information not having to be wrapped by physical layer encoding. With flit-based packetization, it is not necessary to wrap each TLP in a flit as TLP locations within a flit are defined by a flit format definition. Although the technologies described herein are discussed in certain embodiments with regards to the PCIe protocol, flit-based packetization can be utilized in other communication protocols.

In the following description, specific details are set forth, but embodiments of the technologies described herein may be practiced without these specific details. Well-known circuits, structures, and techniques have not been shown in detail to avoid obscuring an understanding of this description. "An embodiment," "various embodiments," "some embodiments," and the like may include features, structures, or characteristics, but not every embodiment necessarily includes the particular features, structures, or characteristics.

Some embodiments may have some, all, or none of the features described for other embodiments. "First," "second," "third," and the like describe a common object and indicate different instances of like objects are being referred to. Such adjectives do not imply objects so described must be in a given sequence, either temporally or spatially, or in ranking, or in any other manner, unless expressly stated. "Connected" may indicate elements are in direct physical or electrical contact with each other and "coupled" may indicate elements cooperate or interact with each other, but they may or may not be in direct physical or electrical contact.

The description may use the phrases "in an embodiment," "in embodiments," "in some embodiments," and/or "in various embodiments," each of which may refer to one or more of the same or different embodiments.

Reference is now made to the drawings, wherein similar or same numbers may be used to designate same or similar parts in different figures. The use of similar or same numbers in different figures does not mean all figures including similar or same numbers constitute a single or same embodiment.

<FIG> illustrates an exemplary computing system comprising a set of interconnected components. System <NUM> includes processor <NUM> and memory <NUM> connected to controller hub <NUM>. Processor <NUM>, as well as any processor described herein, comprises any processing element, such as a microprocessor, a host processor, an embedded processor, a co-processor, processor core, or other processor. Processor <NUM> is connected to controller hub <NUM> by connection <NUM>. In one embodiment, connection <NUM> is a serial point-to-point interconnect. In another embodiment, connection <NUM> includes a serial differential interconnect architecture that is compliant with different interconnect standards.

Controller hub <NUM> is connected to switch <NUM> by connection <NUM>. Switch <NUM> is connected to devices <NUM>-<NUM> by connections <NUM>-<NUM>. In some embodiments, connections <NUM>, <NUM>-<NUM> are serial point-to-point interconnections. In some embodiments, connections <NUM>, <NUM>-<NUM> are links that conform to the PCI Express (Peripheral Component Interconnect Express) standard. A PCI Express (PCIe) link is a serial point-to-point communication channel that allows ports at the ends of the link (e.g., ports <NUM> and <NUM> connected to link <NUM>) to send and receive information. At the physical level, a PCIe link (hereinafter "link") is comprised of one or more lanes. A lane comprises two differential wire pairs, one receiving pair and one transmitting pair. Thus, one lane comprises four wires. A "x4" link has four lanes (<NUM> wires), a "x16" link has <NUM> lanes (<NUM> wires), a "x32" link has <NUM> lanes (<NUM> wires), etc. Any connection between devices disclosed herein can also be referred to as a bus.

Memory <NUM> includes any memory device, such as random-access memory (RAM), non-volatile memory (including chalcogenide-based phase-change non-volatile memories), or other memory accessible by devices in system <NUM>. Memory <NUM> is coupled to controller hub <NUM> through memory interface <NUM>. Memory interface <NUM> can be a double-data rate (DDR) memory interface, a quad data rate (QDR) memory interface, dual-channel DDR memory interface, and a dynamic RAM (DRAM) memory interface or any other memory interface.

In one embodiment, controller hub <NUM> is a PCIe root complex. In other embodiments, the controller hub <NUM> can comprise a chipset, a platform controller hub (PCH), memory controller hub (MCH), a northbridge, an interconnect controller hub (ICH), a southbridge, or a root controller/hub. The term chipset can refer to two or more physically separate controller hubs, such as a memory controller hub (MCH) coupled to an interconnect controller hub (ICH) in a two-controller-hub chipset configuration. In some embodiments, the controller hub <NUM> can be part of the processor <NUM>. In other embodiments, a portion or all the controller hub <NUM> functionality can be integrated into processor <NUM>.

Input/output ports that provide communication between components shown in <FIG> can implement a protocol stack to provide communication between components. For example, ports <NUM> and <NUM> allow communication between controller hub <NUM> and switch <NUM> via connection <NUM>. Switch <NUM> routes messages from devices <NUM>-<NUM> upstream, i.e., up the interconnection hierarchy towards controller hub <NUM>, and downstream, i.e., down the hierarchy away from the controller hub <NUM> to devices <NUM>-<NUM>. As such, ports <NUM>, <NUM>, <NUM>, and <NUM> and can be referred to as upstream ports and ports <NUM>, <NUM>, <NUM>, and <NUM> can be referred to as downstream ports. As shown in <FIG>, controller hub <NUM> does not need to communicate through a switch to communicate to downstream devices (e.g., devices <NUM>-<NUM>). Controller hub <NUM> can directly connect to downstream devices, as shown by its connection to device <NUM> by connection <NUM>, enabled by ports <NUM> and <NUM>.

Devices <NUM>-<NUM> can comprise any internal or external device or component included in or coupled to a computing system, such as an I/O device, a Network Interface Controller (NIC), a graphics card, any other type of add-in card, an audio processor, a network processor, a hard-drive, a solid-state drive (SSD), a flash memory device, other storage device, a CD/DVD ROM, a monitor, a printer, a mouse, a keyboard, a router, a portable storage device, a Firewire device, a Universal Serial Bus (USB) device, a scanner, and other input/output devices.

In embodiments where connections <NUM>-<NUM> are PCIe links, devices <NUM>-<NUM> are referred to as PCIe endpoints. PCIe endpoint devices are often classified as legacy, PCIe, or root complex integrated endpoints. Further, a PCIe link can include one or more extension devices (e.g., <NUM>), such as retimes or repeaters.

The term "device" can refer to any of the components in system <NUM> as well as any other device. Thus, a device can refer to a processor, controller hub, root complex, switch, endpoint, peripheral card, I/O device, etc..

<FIG> illustrates an exemplary computing system comprising a set of components interconnected via PCIe links. System <NUM> comprises a processor <NUM> that includes PCIe root complex functionality. Processor <NUM> is connected to PCI endpoints <NUM> and <NUM> via links <NUM> and <NUM>, respectively. PCI endpoint <NUM> is an NVMe (Non-Volatile Memory Express) SSD and PCIe endpoint <NUM> is a graphic processing unit (GPU) or a graphics card. Processor <NUM> is also connected to PCIe switch <NUM>. Switch <NUM> is connected to PCIe endpoints <NUM>-<NUM> via links <NUM>-<NUM>. Switch <NUM> is further connected to PCI/PCI-X bridge <NUM> (Peripheral Connection Interface/Peripheral Connection Extended Interface) via link <NUM> bridge to support PCI devices <NUM>. Upstream and downstream PCIe ports are shown in <FIG> as filled and unfilled squares, respectively.

<FIG> illustrates an exemplary pair of protocol stacks implemented in a transmitting-receiving device pair. Protocol stacks <NUM> and <NUM> can be any kind of protocol or network stack, such as a PCIe stack. Although references are made herein to a PCIe stack, the same concepts may be applied to other protocol or network stacks. In one embodiment, protocol stack <NUM> is a PCIe protocol stack comprising transaction layer <NUM>, data link layer <NUM>, and physical layer <NUM>. A communication port, such as ports <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> in <FIG>, may be referred to as a module, circuitry, interface, or port implementing or including protocol stack <NUM> or any other protocol stack. Thus, in one embodiment, a protocol stack can comprise a transaction layer module, a data link layer module, and a physical layer module. Such a protocol stack can also be considered as comprising transaction layer circuitry, data link layer circuitry, and physical layer circuitry. When referencing the protocol stack as a whole, a protocol stack can be referred to as protocol stack circuitry or a protocol stack module.

PCI Express uses packets to communicate information between a root complex, switches, and endpoints. For example, information to be sent from transmitting device <NUM> to receiving device <NUM> is sent from processor <NUM> (or any other processing device of transmitting device <NUM>) to protocol stack <NUM>. Transaction layer <NUM> packets carry the information to be delivered from processor <NUM> of transmitting device <NUM> to processor <NUM> of receiving device <NUM>. As the packets that are formed at transaction layer <NUM> move down the protocol stack to data link layer <NUM> and then to physical layer <NUM>, they are extended with information to handle packets at those layers. Physical layer <NUM> of transmitting device <NUM> transmits packets over link <NUM> to physical layer <NUM> of receiving device <NUM>. There, the reverse process occurs, and the information added to the packets as they moved down protocol stack <NUM> is stripped as the packets move up protocol stack <NUM>. Transaction layer <NUM> of protocol stack <NUM> delivers the payload of the packets sent from transmitting device <NUM> to processor <NUM> of receiving device <NUM>, thus resulting a message being sent from transmitting device <NUM> to receiving device <NUM>.

In one embodiment, transaction layer <NUM> provides an interface for processor <NUM> to PCIe protocol stock <NUM>. Transaction layer <NUM> is responsible for the assembly and disassembly of transaction layer packets (TLPs) that deliver information between devices. Upon receipt of information from processor <NUM>, transaction layer <NUM> assembles one or more TLP packets that will deliver the information to receiving device <NUM>. TLP packet <NUM> is one embodiment of a TLP packet. TLP packet <NUM> comprises TLP header <NUM> and payload <NUM>. Additional information can be included in a TLP packet, such as an end-to-end cyclic redundancy code (ECRC), an error-detection code that a PCIe transaction layer can calculate for a TLP packet.

Data link layer <NUM> acts as an intermediate stage between transaction layer <NUM> and physical layer <NUM>. In one embodiment, one responsibility of data link layer <NUM> is to provide a reliable mechanism for exchanging Transaction Layer Packets (TLPs) between devices across a link. In one embodiment, the reliability mechanism is implemented in part by determining a packet sequence number for a TLP packet and calculating a link CRC (LCRC) value based on the contents of the TLP packet and the packet's sequence number. Data link layer <NUM> appends information <NUM> to the head of TLP packet <NUM> that includes the packet sequence number and appends LCRC value <NUM> to the tail of TLP packet <NUM>. Data link layer <NUM> submits TLP <NUM> with appended information to physical layer <NUM> for transmission across link <NUM> to receiving device <NUM>.

In some embodiments, data link layer <NUM> comprises a replay buffer that stores copies of transmitted flits until transmitting device <NUM> receives an acknowledgment, implicitly or explicitly, that the transmitted flits have been transmitted successfully. Transmitting device <NUM> can receive an ACK (acknowledgment) signal from receiving device <NUM> that a transmitted flit has been successfully received and can receive a NAK (no acknowledgment) signal from receiving device <NUM> if a transmitted flit has not been successfully received. Receiving device <NUM> can determine that a flit has not been successively received by, for example, not receiving an expected flit within a pre-determined or other specified time or by determining that a received flit is corrupt. Receiving device <NUM> can determine that a received flit is corrupted by, for example, the flit failing a flit-level CRC check, a per-lane FEC check, or a flit-level FEC check.

In response to determining that a flit has not successfully received, receiving device <NUM> can send a retry request to transmitting device <NUM>. The retry request can include information identifying which flits are to be retransmitted. Such flits may be referred to herein as "retry flits". The retry request can identify one or more individual flits to be retransmitted, that all flits starting from an identified flit are to be retransmitted or identify the flits to be retransmitted in any other matter. In response to receiving a retry request, transmitting device <NUM> retransmits the retry flits identified by the retry request. In some embodiments, transmitting device <NUM> retrieves the retry flits from the replay buffer. In other embodiments, transmitting device <NUM> can determine that a retry flit was a null flit (a flit that does not contain any transaction layer packet information, as discussed in further detail below) and reconstruct the null flit if the null flit was not stored in the replay buffer prior to transmission.

In some embodiments, transmitting device <NUM> can determine that a transmitted flit is a null flit and not store the null flit in the replay buffer prior to transmission. In some embodiments, transmitting device <NUM> can keep track of whether transmitted flits are null flits. In other embodiments, transmitting device <NUM> can determine that a retry flit is a null flit by determining the presence of a non-contiguous flit sequence in the replay buffer that corresponds to the retry flits. A non-contiguous flit sequence can be determined from flit sequence numbers or other flit identifiers. For example, if transmitting device <NUM> transmits five flits with sequence numbers <NUM> through <NUM>, and flit <NUM> is a null flit, transmitting device <NUM> may only store flits <NUM>, <NUM>, <NUM>, and <NUM> in the replay buffer. If transmitting device <NUM> then receives a retry request requesting retransmission of flits <NUM> through <NUM>, during retrieval of the retry flits from the replay buffer, transmitting device <NUM> can determine the presence of non-contiguous flits stored in the replay buffer (i.e., transmitting device <NUM> detects that the flit sequence numbers for flits stored in the replay buffer jump from <NUM> to <NUM>), retrieve flits <NUM>, <NUM>, and <NUM> from the replay buffer, reconstruct flit <NUM> as a null flit, and retransmit flits <NUM> through <NUM> (including reconstructed flit <NUM>) to receiving device <NUM>. Thus, transmitting device <NUM> can reconstruct flits corresponding to gaps in the non-contiguous flit sequence as null flits and send the reconstructed null flits as part of the retry flits to receiving device <NUM>. As discussed in greater detail below, a receiving device may not send a request retry for null flits. For debug and test purposes, a receiver can have a mode that allows for the retry of null flits.

Moving down the protocol stack <NUM> of transmitting device <NUM>, in one embodiment, physical layer <NUM> includes logical sub-layer <NUM> and electrical sub-layer <NUM> to physically transmit a packet to an external device. Here, logical sub-layer <NUM> is responsible for the "digital" functions of physical layer <NUM>. In this regard, the logical sub-layer includes a transmit section to prepare outgoing information for transmission by physical sub-layer <NUM>, and a receive section to identify and prepare received information before passing it to data link layer <NUM>. Logical sub-layer <NUM> frames the appended TLP packet with start transaction packet (STP) field <NUM> to generate an as-transmitted TLP <NUM> that is transmitted across link <NUM>. The STP field <NUM> comprises sequence number <NUM>, frame CRC bits, the length of the TLP, and other information.

As used herein, the term "transaction layer packet" refers to TLPs generated by transaction layer <NUM> and comprises TLP header and payload information. The terms "transaction layer packet information" and "transaction layer packet data" can refer to either the TLP header, the TLP payload, or both. As used herein, the term "transaction layer packet" does not include information added to a TLP, such as sequence number, LCRC, or framing data added by the data link and physical layers, unless expressly stated. Similarly, the term "data link layer packet" refers to data link layer packets that do not contain framing data added by a physical layer, unless expressly stated.

Electrical sub-layer <NUM> includes transmitter <NUM> and receiver <NUM>. Transmitter <NUM> is supplied symbols by logical sub-layer <NUM>, which transmitter <NUM> serializes and transmits to receiving device <NUM> across link <NUM>. Receiver <NUM> is supplied with received serialized symbols sent across link <NUM> by receiving device <NUM> and transforms the received signals into a bitstream. The bitstream is deserialized and supplied to logical sub-layer <NUM>. In one embodiment, packets are transmitted across link <NUM> using an 8b/10b data encoding scheme, wherein eight bits of data are transmitted as ten-bit symbols.

Although transaction layer <NUM>, data link layer <NUM>, and physical layer <NUM> are discussed in reference to an embodiment of a PCIe protocol stack, a protocol stack is not so limited. In other embodiments, protocol stack <NUM> can have different layers and/or fewer or more layers than those shown. Further, in other embodiments, the separate layers shown in <FIG> can be combined into a single layer, and a single layer as shown in <FIG> can be split into multiple layers. Any of the layers in the protocol stack shown in <FIG> can be implemented as part of the operating system of a computing device, one or more software applications independent of the operating system, or operate at another software layer. The layers shown in <FIG> can be implemented in software, hardware, firmware or combinations thereof and can be alternately referred to as modules or circuitry (e.g., "data link layer module," "physical layer circuitry") or a combination thereof. A computer device referred to as being programmed to perform a method can be programmed to perform the method via software, hardware, firmware or combinations thereof.

<FIG> illustrates an exemplary format of a PCIe TLP as transmitted across a link. TLP <NUM> comprises TLP header <NUM> and TLP payload <NUM>, LCRC <NUM>, and STP <NUM>. In one embodiment, TLP header <NUM> can be <NUM> or <NUM> bytes (<NUM>/20B), TLP payload <NUM> can be <NUM>-<NUM>,<NUM> bytes (<NUM>-4KB), STP <NUM> is four bytes (4B), and LCRC <NUM> is four bytes (4B).

For the TLP packet format shown in <FIG>, the overhead per TLP comprises 8B, or <NUM> double words (2DW), where one double word is four bytes (1DW = 4B). Because the TLP packet format allows for payloads of varying widths, the TLP transmission efficiency is also variable. For TLPs carrying a large payload, the overhead is relatively small, and the efficiency is high, but for TLP packets carrying little or no payload, the overhead can be expensive. For example, a TLP with a 4DW header and carrying no payload is 6DW in length, yielding a transmission efficiency of <NUM>% (or, data link layer and physical layer encoding overhead of <NUM>%).

The data link layer packet (DLLP) is a second PCI Express packet type. In current versions of the PCIe protocol, a DLLP can be <NUM> bytes (8B), as transmitted along a link. A DLLP packet can be, for example, an ACK packet, which acknowledges that a transmitted TLP has been successfully received, a NAK packet, which indicates that a TLP arrived corrupted at the receiver (or did not arrive at all), and flow control DLLPs that are used as part of the data link layer's credit-based flow control mechanism. A third packet type is the idle packet (IDL), which is one DW in length and is transmitted on an active link when no message is to be sent. In addition to transmission inefficiencies due to link CRC and physical layer encoding overhead, other transmission inefficiencies can exist. For example, periodic DLLP updates can consume link bandwidth. In some PCI Express embodiments, periodic DLLP updates can consume <NUM>-<NUM>% of link bandwidth. Further details of PCIe packet formats in various PCIe protocol version can be found at the PCI-SIG website.

In some embodiments, additional error correction beyond LCRCs and ECRCs may be employed to achieve a desired bit error rate (BER). For example, forward error correction (FEC) may be applied on a per-lane basis to achieve a desired BER in protocol stacks comprising various techniques for achieving high-speed digital links, such as PAM-<NUM> (pulse amplitude modulation) multi-level signaling or other techniques for achieving high-speed digital links. PAM-<NUM> signaling with FEC may be used to achieve an acceptable range of BER while running at <NUM> GT/s, the planned speed for version <NUM> of the PCIe protocol. In a per-lane FEC scheme, n bits of data transmitted on a lane has k bits of information and (n-k) error correction code bits. Per-lane FEC schemes help with lane-to-lane correlation of errors as errors on each lane are corrected independently. Accordingly, n bits are to be received and checked before a TLP can be passed to the data link layer and verified using the TLP's LCRC codes. The inclusion of FEC at the lane level consumes additional link band width due to the inclusion of the k FEC check bits. Additional latency arises from the store-and-forward latency resulting from the implementation of FEC at the receiver.

<FIG> illustrates an exemplary flit definition with a per-lane FEC scheme for a x4 PCIe link. A flit comprises one or more TLPs and DLLPs protected by a flit-level CRC scheme and a per-lane FEC scheme. In some embodiments, a flit may be protected by a flit-level FEC scheme, as will discussed in greater detail below. Flit <NUM> comprises transaction layer packet symbols <NUM>, data link layer packet symbols <NUM>, flit-level CRC code symbols <NUM>, and FEC symbols <NUM>. Flit <NUM> comprises <NUM> total symbols. In some embodiments, a symbol can be 1DW in length, but a symbol can be different lengths in other embodiments. The <NUM> symbols in flit <NUM> comprise <NUM> symbols carrying transaction layer packet information (TL0-TL287), <NUM> symbols carrying data link layer packet information (DLL0-<NUM>), <NUM> flit-level CRC symbols (CRC0-<NUM>), and <NUM> FEC symbols (<NUM> check symbols (C) and <NUM> parity symbols (P)). Eight CRC symbols are used to ensure that multiple TL/DPP symbol errors are covered adequately. Information from a single TLP can span multiple TL symbols in a flit. For example, if a symbol is 1DW long, flit <NUM> would need <NUM> TL symbols to store a TLP that is 8DWs in length.

Flit <NUM> is transmitted across lanes <NUM> (L0-L3). That is, a different portion of flit <NUM> is sent across each of the four lanes. <FIG> shows successive TL, DLL, and CRC symbols being transmitted along adjacent lanes, but in other embodiments, symbols could be apportioned among lanes in differing fashions.

Flit <NUM> can have a flit sequence number, which may be specified explicitly, implicitly, or opportunistically. If specified explicitly, the flit sequence number can be coded, for example, in reserved bits in the flit. If specified implicitly, the flit sequence number may not be sent across the link and a receiver can determine the flit packet number in another fashion, such as, for example, by counting flits as they are received. If specified opportunistically, the flit sequence number can be sent, for example, as payload in a DLLP packet that is sent in full or in part as part of a flit. In one example of flit sequence numbers being sent implicitly, the CRC can include the flit sequence number as, for example, consecutive bits in the CRC code during CRC computation. Accordingly, any dropped or replayed flits could be detected at a receiving end.

In flit <NUM>, four one-byte symbols (DLL0-<NUM>) are allocated for one DLLP. In other embodiments, information from multiple DLLPs can be carried in a single flit. In still other embodiments, a DLLP can be constructed over consecutive flits. For example, in a flit format that allocates five DLLP bytes, four DLLP bytes can carry one DLLP and the fifth DLLP byte from four consecutive flits can contribute to an additional DLLP.

The flit-level CRC code protects the flit contents minus the FEC check and parity symbols. For example, the flit-level CRC codes in symbols CRC0-<NUM> in flit <NUM> protect the TL, DLL and CRC symbols in flit <NUM>. The per-lane FEC scheme protects all symbols within a flit that are transmitted across one lane. For example, the six FEC check and parity symbols to be transmitted along one of the lanes L0-L3 protect the <NUM> symbols (<NUM> flit symbols / <NUM> lanes) transmitted along that lane. The FEC scheme further employs interleaving. The FEC scheme employed in flit <NUM> uses three-way interleaving, as represented by the shading of each symbol. For example, check and parity symbols <NUM>, shown with no shading, protect the symbols transmitted along lane L3 that are also shown with no shading (TL3, TL7, TL11. TL287, CRC3, CRC7, check/parity symbols <NUM>). Thus, in the FEC scheme illustrated in <FIG>, two FEC symbols - one check symbol and one parity symbol - are used to protect <NUM> flit symbols (<NUM> symbols transmitted along lane L3 / <NUM>-way interleaving). In other embodiments, alternative interleaving schemes (<NUM>-way, <NUM>-way, etc.), as FEC schemes with more of fewer check and parity bits per lane can be used to achieve a desirable bit error rate.

Different flit definitions can be used for different link widths. <FIG> illustrates exemplary flit definitions with per-lane FEC schemes for x8 and x2 PCIe links. Flit <NUM> defines a <NUM>-symbol flit format for a PCIe x8 link. Flit <NUM> comprises <NUM> symbols containing transaction layer packet data (TL0-TL599), <NUM> symbols containing data link layer packet data (DLL0-<NUM>), <NUM> flit-level CRC symbols (CRC0-<NUM>), and <NUM> FEC symbols (<NUM> check symbols and <NUM> parity symbols). The <NUM> symbols are transmitted across eight lanes (L0-L7). As in flit <NUM>, the six FEC check and parity symbols transmitted in a lane protect <NUM> symbols (<NUM> symbols / <NUM> lanes) using <NUM>-way interleaving, with each pair of FEC check and parity symbols protecting <NUM> symbols (<NUM> symbols / <NUM>-way interleaving).

Flit <NUM> is a <NUM>-symbol flit definition for a PCIe x2 link. Flit <NUM> comprises <NUM> symbols carrying transaction layer packet information (TL0-TL299), <NUM> symbols carrying data link layer packet information (DLL0-<NUM>), eight CRC symbols (CRC0-<NUM>), and <NUM> FEC symbols (<NUM> check symbols and <NUM> parity symbols). The <NUM> symbols are transmitted across two lanes (L0-L1). The six FEC check and parity symbols transmitted in each lane protect <NUM> symbols (<NUM> symbols / <NUM> lanes) using three-way interleaving, with each pair of FEC check and parity symbols protecting <NUM> symbols (<NUM> symbols / <NUM>-way interleaving).

Flits <NUM>, <NUM> and <NUM> show exemplary flit definitions for particular PCIe link widths. Other flit definitions are possible for a particular PCIe link. The number of TL and/or DLL symbols can vary from one flit definition to another, as can the number flit-level CRC symbols used to protect a flit, the number of FEC symbols used to protect a lane, and the degree of interleaving used for the FEC scheme. Increasing the number of flit-level CRC symbols and/or the FEC symbols used can reduce BER, but at the cost of increased overhead, and thus, efficiency.

In some embodiments, the flit definition for a particular link can be defined, determined, or supplied by an operating system or BIOS during system start-up. The flit definition for a link can be stored in software or firmware local to the system, be encoded in hardware, or stored remotely and accessed during system start-up. The flit definition for a particular link need not be static and can vary during system operation. For example, the physical layer, protocol stack, or operating system can receive an updated flit definition for a particular link and the link can begin using the updated flit definition at some point after receiving the updated flit definition. An updated flit definition can be provided in response to various events. For example, a physical layer can monitor link performance and signal to the protocol stack or operating system that the BER of the link is too high, and the system can switch to a flit definition that has a flit-level CRC scheme with more CRC bytes, a per-lane FEC scheme that has more check and parity bits per lane, a flit-level FEC scheme that has more check and parity bits per flit, or a combination thereof. In another example, the system can determine that greater information throughput is possible while keeping the BER at an acceptable level and a flit definition providing greater transmission efficiency can be used. Greater efficiency can be obtained in an updated flit definition by increasing the number of TLP/DLLP packet symbols in the flit, or by reducing the number of flit-level CRC symbols and/or the number of FEC check/parity symbols.

Referring to <FIG>, in some embodiments, a protocol stack implementing flit-based packetization can have the same configuration as protocol stack <NUM>. That is, a protocol stack implementing flit-based packetization can comprise a transaction layer, a data link layer, and a physical layer. The transaction layer can generate TLPs based on information received from a processing core and pass the TLPs to a data link layer that places TLP and DLLP information into a flit and calculates flit-level CRC codes for placement in the flit. As discussed below, TLP and DLLP information may be placed in a flit based on placement or packing rules. A filled flit does not mean that every symbol reserved for TLP or DLLP information has TLP/DLLP header or payload information in it. A filled flit can contain NULL TLPs or IDL symbols if fields reserved for transaction layer packet information. A physical layer can calculate FEC check and parity symbols for flit symbols to be transmitted across a lane, add the FEC symbols to the flit, and send the completed flit across the link.

In other embodiments, flit assembly and disassembly tasks can be apportioned differently to the data link layer and physical layers. For example, the physical layer can calculate the flit-level CRCs along with the per-lane FECs. In another embodiment, the data link layer and physical layers can be combined, and flit assembly and disassembly tasks are performed by a single layer. Other arrangements are possible.

<FIG> shows a table of flit characteristics for exemplary flit definitions with per-lane FEC schemes for various PCIe link widths. Table <NUM> shows flit characteristics for flits <NUM>, <NUM>, and <NUM> for x4, x8 and x2 links, respectively, and flit characteristics for possible flit definitions for x1 and x16 links. The FEC latencies shown are approximate latency times at transfer speeds proposed for PCIe Gen <NUM>. The narrower links (x1, x2) have higher FEC latency to derive improved efficiency by amortizing the fixed overhead (flit-level CRC symbols and data link layer packet information) as shown. Flit definitions having characteristics different than those shown in Table <NUM> can be used for varying link widths. Different flit sizes for a given link width can be selected to trade-off latency for efficiency.

Flit-based packetization can provide at least the following advantages. First, flit-based packetization removes the need for an LCRC to be attached to each TLP since the flit is protected by a CRC scheme at the flit level. Flit <NUM> in <FIG> comprises TLP data from <NUM> TLPs and only eight flit-level CRC symbols. If sent across a link using current PCIe protocols, those <NUM> TLPs would be accompanied by <NUM> LCRCs. In other words, a flit does not contain a CRC sent along with each TLP included in the flit. With flit-based packetization, link layer retry happens at the flit level. In one embodiment, a failed flit can be retried with a receiver storing successfully transmitted flits received at the receiver after the corrupted or missing flit. In some embodiments, a transmitter can be informed that a transmitted flit was not successfully received through receipt of a NAK DLLP containing the sequence number of the flit to be resent. In some embodiments, a retried flit can indicate that it is being resent via a "retry flit" encoding in the corresponding DLLP along with the sequence number of the retried flit.

Second, flit-based packetization allows for the adoption of guaranteed DLLP frequency policies. In one embodiment, upon transmission of a flit, the transmitter is to receive a DLLP acknowledgment packet indicating that the flit was received no later than n flits after being sent. In another embodiment, a receiver is to send a DLLP packet providing credit updates to the transmitter no later than m flits after accruing x number of credits. Such policies can reduce storage overhead at the receiver relative to current receiver storage needs, which account for current PCIe DLLP scheduling policies and having to handle such situations as sending an ACK or NAK DLLP behind a maximum payload size TLP.

Third, since flit definitions provide guaranteed positions for DLLPs in a flit, the <NUM>-<NUM>% variable bandwidth loss due to DLLP scheduling is exchanged for a fixed bandwidth reduction for a given flit definition. For example, in the x4 flit definition of flit <NUM>, the bandwidth reduction due to DLLP overhead is only <NUM>% (<NUM>/<NUM>).

Fourth, the bandwidth loss due to the addition of framing information added by the physical layer (i.e., start frame <NUM> in <FIG>) to each TLP is eliminated since transaction layer packet information is in fixed locations in the flit and flit boundaries are known. That is, physical layer framing information is not added to a TLP for every TLP transmitted as part of a flit.

Fifth, because a particular flit format has a defined length, the synchronization header that is periodically sent in current PCIe protocols can be replaced by a periodic ordered set (OS), such as an SKP OS (Skip Ordered Set), EIOS (Electrical Idle Ordered Set), or EIEOS (Electrical Idle Exit Ordered Set). In one embodiment, a periodic OS can be sent every <NUM> flits for flits that are <NUM>. 25ns long where there is a <NUM>-ppm clock spread. Replacing the synchronization header with a periodic OS can reclaim at least a portion of the bandwidth consumed through use of a synchronization header. In current PCIe protocol formats, synchronization headers consume approximately <NUM>% of bandwidth.

<FIG> illustrate exemplary flit packet sequences according to various placement rules. A flit definition defines which slots or fields in the flit may be occupied by TLP or DLLP information, but, in some embodiments, placement or packing rules are used to determine where TLP or DLLP information for individual TLPs/DLLPs to be sent in a flit are placed. <FIG> show a sequence of <NUM> flit packets transmitted across a link. The packets are sent in order from left to right in each row, with the first row being sent first and the third row being sent last. Each column is 1DW wide and each set of four columns is 4DW wide.

<FIG> illustrates TLPs placed in a flit according to the rule that only TLP header information or TLP payload (data) information can reside in each aligned 4DW set. Starting with the upper left-most packet, the first two 4DW sets contain header information <NUM> (h0-h3) and payload information <NUM> (d0-d3) of a first TLP. The next two 4DW sets contain NULL TLPs <NUM> and <NUM>. The first 4DW set in the second row contains header information <NUM> (h0-h2) for a second header-only TLP. Because the header of this second TLP is only 3DWs wide, the fourth DW set is filled with an IDL packet. Idle packets can be used to fill in 4DW sets where the header or payload information for a packet is less than four DWs in length. Continuing along the second row, the second and third 4DW sets contain header information <NUM> (h0-h2) and payload information <NUM> (d0) for a third TLP. Because the header and payload information for this third TLP is less than 4DWs, the second and third 4DW sets are filled with IDL packets. The fourth 4DW set in the second row is filled with header information <NUM> (h0-h2) for a fourth TLP, and an idle packet. The payload <NUM> (d0-d1) for the fourth TLP is placed in the first two DWs of the third row, followed by two idle packets. The remaining three 4DW sets in the third row comprise header information <NUM> (h0-h2) for a fifth TLP, and header information <NUM> (h0-h3) and payload information <NUM> (d0-d3) for a sixth TLP. In sum, six TLPs are packed into the <NUM> DWs illustrated in <FIG>.

<FIG> illustrates TLPs and DLLPs placed in a flit according to the rule that each aligned 4DW set can contain header and data information from one TLP, with DLLPs placed opportunistically. Starting with the upper left-most packet of <FIG>, the first two 4DW sets contain header information <NUM> (h0-h3) and payload information <NUM> (d0-d3) of a first TLP. The third 4DW set contains NULL TLP <NUM> and the fourth 4DW set contains a first DLLP <NUM> followed by three IDL packets. Moving to the second row, the first 4DW set contains header information <NUM> (h0-h2) for a second TLP, followed by an IDL packet. The second DW contains header information <NUM> (h0-h2) and payload information <NUM> (d0) for a third TLP. The first three DWs in the third 4DW set in the second row is filled with header information <NUM> (h0-h2) for a fourth TLP and an opportunistically placed second DLLP <NUM>. The fourth DW set in the second row contains payload information <NUM> (d0-d1) for the fourth TLP, with the last two DWs containing idle packets. Continuing to the last row, the first 4DW set contains header information <NUM> (h0-h2) for a fifth TLP and an idle packet. The second and third 4DW sets contain header information <NUM> (h0-h3) and payload information <NUM> (d0-d3) for a sixth TLP. The last 4DW set comprises header information <NUM> (h0-h2) and payload information <NUM> (d0) for a seventh TLP. The first six TLPs placed in <FIG> are the same TLPs placed in <FIG>. The looser packing rules associated with <FIG> allow for a seventh TLP and two DLLPs to be additionally placed in the same <NUM> DWs.

<FIG> illustrates TLPs and DLLPs placed in a flit according to the rule that TLP header and payload information can be placed at any DW boundary, with opportunistic DLLP placement. Placing the same <NUM> TLPs and <NUM> DLLPs that were placed in <FIG>, the first TLP and the first DLLP are placed in the first row of <FIG> in the same manner as they were in <FIG>. Moving to the second row, the first 4DW set contains information from the second and third TLPs: header <NUM> (h0-h2) from the second TLP and the first DW (h0) of header information <NUM> (h0-h2) of the third TLP. The remaining header information (h1-h2) of the third TLP is placed in the first two DWs of the second 4DW set. Header information <NUM> (h0-h2) for a fourth DLP is contained in the last DW of the second 4DW set and the first two DWs of the third 4DW set. The payload information <NUM> (d0-d1) of the fourth TLP is placed in the second half of the third 4DW set. The last 4DW set of the second row contains header information <NUM> (h0-h2) of the fifth TLP. The header information <NUM> (h0-h3) of the sixth DLP fills the last DW of the second row and continues into the first three DWs of the first 4DW set of the third row. The payload information <NUM> (d0-d3) of the sixth TLP fills the last DW of the first 4DW set in the third row and the first three DWs of the second 4DW set. The remainder of the last row is filled with the header information <NUM> (h0-h2) and payload information <NUM> (d0) of the seventh TLP, a second DLLP <NUM> and header information <NUM> (h0-h2) of an eighth TLP. Allowing TLP header and payload data to be placed in any DW allows for the seven TLPs and <NUM> DLLPs placed in <FIG> to be placed more compactly - they fit in five fewer DWs and allow for placement of an eighth TLP.

<FIG> shows a table of PCIe <NUM> TLP efficiencies. Table <NUM> shows how TLP efficiency under the most current version of the PCIe protocol (version <NUM>, released May <NUM>, <NUM>) varies with transaction size, yielding efficiencies above <NUM>% for large payloads but dropping down to <NUM>% for header-only transactions. These efficiencies are the same regardless of link width. Table <NUM> in <FIG> shows flit-based packetization achieving TLP efficiencies ranging from <NUM>% - <NUM>% across link widths, a relatively uniform efficiency as compared to the range of TLP efficiencies in PCIe <NUM> across transaction sizes.

<FIG> illustrates an exemplary method of transmitting a flit. The method <NUM> can be performed by, for example, a PCIe root complex connected to a graphics card via a x4 PCIe link. At <NUM>, information is received from a processor. In the example, the root complex PCIe protocol stack receives a message to be sent to the graphics card from a root complex processor. At <NUM>, transaction layer packets are generated based on the received information. In the example, the root complex PCIe protocol stack generates TLPs based on the message to be sent to the graphics card. At <NUM>, flits comprising the TLPs are generated. The flits comprise the transaction layer packets and are each protected by a flit-level cyclic redundancy check (CRC) scheme. In the example, the root complex PCIe protocol stack generates one or more <NUM>-symbol flits containing the TLPs to be sent to the graphics card. The protocol stack protects each flit with eight flit-level CRC codes based on the contents of TLP. At <NUM>, the flits are transmitted across the lanes of a link to a receiver. In the example, the root complex transmits the flit across the four lanes of the x4 PCIe link to the graphics card. The <NUM>-symbol flit is broken into four streams of <NUM> symbols and each stream is transmitted along one of the lanes of the x4 PCIe link.

Method <NUM> can optionally include additional actions. For example, the flit-level CRC codes can be transmitted across the one or lanes. In the example, the root complex transmits the eight flit-level CRC codes across the four lanes, two per lane. In another example, each flit is protected with a forward error correction (FEC) scheme applied on a per-lane basis. In the example, the root complex PCIe protocol stack determines which symbols of the <NUM>-symbol flits will be transmitted across each lane, calculates FEC check and parity codes based on the symbols to be transmitted along each lane, and transmits the generated FEC codes across the appropriate lane.

<FIG> illustrates an exemplary method of receiving a flit. The method <NUM> can be performed by, for example, a network interface card (NIC). At <NUM> flits are received over the lanes of a link. The flits comprise transaction layer packets and flit-level CRC codes. In the example, the NIC receives a <NUM>-symbol flit comprising <NUM> TLPs over a x8 PCIe link from a root complex. At <NUM>, it is determined whether the flit has been successively received based on the TLPs and the flit-level CRC codes in the flit. In the example, the NIC determines that the flit has been successively received based on the <NUM> TLPs and the eight flit-level CRC symbols in the received flit. At <NUM>, if the flit has been successively received, the information contained in the TLPs is sent to a processor. In the example, the payload contained in the TLPs of the successively received flit is sent to the NIC processor.

Method <NUM> can optionally include additional actions. For example, the received flit can comprise forward error check (FEC) information received at each of the one or more lanes. In the example, the flit received at the NIC comprises FEC information or codes, three check symbols and three parity symbols, received at each of the eight lanes of the x8 link. In another example, a portion of the flit is received at each lane and determining whether the flit has been successfully received further comprises performing, for each lane, an FEC of the portion of the flit and the FEC codes received at the lane. In the example, <NUM> flit symbols and <NUM> FEC symbols are received at each of the eight lanes. The NIC performs eight FEC checks, one per lane, on the <NUM> flit symbols received at each lane using the <NUM> FEC symbols received at each lane. The NIC determines that the flit has not been received successfully if any of the eight lane-based FEC checks fail.

In another example, an acknowledgment is sent to the transmitting device before the receiver device receives a pre-determined number of flits after receipt of a flit. In the example, the NIC is programmed to send a flit acknowledgment message to the root complex that the <NUM>-symbol flit has been received before the next, for example, <NUM> flits are received. In another example, credits are accrued upon determining that a received flit and successive flits have been successfully transmitted, determining that a threshold number of credits have been accrued, and sending credit updates to the transmitting device before a pre-determined number of flits have been received after a pre-determined number of credits have been accrued. In the example, the NIC accrues credits for the received flit and successfully received flit. After the NIC accrues a set of number of credits, the NIC sends credit updates within the receipt of, for example, <NUM> flits after the set of number of credits have been accrued at the NIC.

In some embodiments, a flit can be protected with a flit-level FEC or parallel-FEC scheme instead of a per-lane FEC scheme. For wider links, flit-level FEC schemes can provide lower latencies and increased efficiencies over per-lane FEC schemes as flit-level FEC schemes may utilize fewer FEC symbols per flit.

<FIG> illustrates an exemplary flit definition for a x16 PCIe link utilizing a flit-level FEC scheme. Flit <NUM> comprises transaction layer packet information symbols <NUM>, flit-level CRC symbols <NUM>, data link layer packet information symbols <NUM>, and flit-level FEC symbols <NUM>. FEC symbols <NUM> comprise three two-symbol FEC codes that provide flit-level protection for flit <NUM>. The FEC symbols <NUM> are three-way interleaved to improve FEC resiliency. For example, an interleaved FEC scheme can help avoid a burst error in a lane from affecting multiple symbols protected by an FEC code. The FEC symbol interleaving in flit-level FEC schemes is represented in <FIG>, <FIG>, and <FIG> by symbol shading. In the three-way interleaving FEC scheme illustrated in these figures, FEC symbols with no shading protect flit symbols with no shading, FEC symbols with light shading protect flit symbols with light shading, and FEC symbols with dark shading protect flit symbols with dark shading. As part of determining whether a flit has been successfully received, a receiving device performs a flit-level FEC check using the FEC codes once all symbols in a flit have been received. If the receiving device determines that the flit has an error that cannot be corrected using the FEC codes, such as symbol errors in two or more lanes affecting the same FEC code, it can send a retry request to the transmitting device requesting that the corrupted flit be retransmitted.

In the flit-level FEC scheme shown in <FIG>, each FEC code comprises two symbols (FEC0, FEC1) for a total of six FEC symbols that protect flit <NUM>. In some embodiments, a two-symbol FEC code can comprise a parity symbol and a check symbol, and the two-symbol FEC code can correct one symbol error in the protected symbols. In other embodiments, flit-level FEC schemes can use more of fewer than three FEC codes and use n-way interleaving schemes where n is lesser or greater than three. In embodiments where one symbol is one byte, flit <NUM> is <NUM> bytes: <NUM> bytes of transaction layer packet information (TL0-TL299), <NUM> bytes of data link layer packet information (DLL0 - DLL5), eight CRC bytes (CRC0-CRC7), and six FEC bytes (three sets of FEC0-<NUM> bytes). The transaction layer packet transmission efficiency for flit <NUM> is <NUM>% (<NUM> TL bytes / <NUM> total bytes).

<FIG> illustrate exemplary flit definitions for x8 and x4 PCIe links utilizing flit-level FEC schemes and <FIG> illustrate exemplary flit definitions for x2 and x1 PCIe links utilizing flit-level FEC schemes. Flits <NUM>, <NUM>, <NUM>, and <NUM> are each comprised of <NUM> bytes. At the <NUM> GT/s speeds proposed for PCIe Gen <NUM> and with one symbol containing <NUM> bits, flit <NUM> is transmitted across a x16 link in <NUM> ns. This transmission speed scales linearly with decreasing link width as each lane transmitting portions of flits <NUM>, <NUM>, <NUM>, <NUM> or <NUM> carries a linearly increasing number of symbols (<NUM>, <NUM>, <NUM>, <NUM>, and <NUM> for the x16, x8, x4, x2, and x1 links, respectively). These symbols are transmitted over a linearly increasing number of unit intervals (<NUM>, <NUM>, <NUM>, <NUM>, and <NUM> unit intervals (UIs) for the x16, x8, x4, x2, and x1 links, respectively) to give transmission speeds of <NUM> ns, <NUM> ns, <NUM> ns, and <NUM> ns for the x8, x4, x2, and x1 links, respectively. Flits <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> have the same transaction layer packet data transmission efficiency of <NUM>% as these flits have the number of FEC codes. Flit-level FEC symbols are shown in flits <NUM>, <NUM>, and <NUM> as being transmitted across the two upper-most lanes of a link but can be transmitted across any lanes and across any number of lanes.

The <NUM> ns latency for the <NUM>-byte flits utilizing the flit-level FEC schemes illustrated in <FIG>, <FIG>, and <FIG> is lower than the latencies for the flits utilized per-lane FEC schemes characterized in <FIG>. The per-lane FEC flits in <FIG> have <NUM> ns latencies at the x4, x8, and x16 link widths and efficiencies of <NUM>%, <NUM>%, and <NUM>% for x4, x8, and x16 widths, respectively. Latencies for per-lane FEC flits for x1 and x2 link widths are <NUM> ns and <NUM> ns and have efficiencies of <NUM>% and <NUM>%, respectively. Thus, flits <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> utilizing a flit-level FEC scheme have improved efficiencies over their per-lane FEC counterparts characterized in <FIG>.

Moving from a per-lane FEC scheme to a flit-level FEC scheme can result in a higher retry probability. For example, with reference to <FIG>, which show tables containing retry characteristics for per-lane and flit-level FEC schemes, for a raw bit error rate of <NUM> x <NUM>-<NUM>, a flit-level FEC flit transmitted over a x16 link over <NUM> UIs has a retry probability of <NUM> x <NUM>-<NUM> as compared to a retry probability of <NUM> x <NUM>-<NUM> for a per-lane FEC flit transmitted over a x16 link over <NUM> UIs. For embodiments with a <NUM> ns retry window, the x16 per-lane FEC scheme has a bandwidth loss of <NUM>% and the x16 flit-level FEC scheme has a bandwidth loss of <NUM>%.

One approach for improving retry probability in flits utilizing flit-level FEC schemes is to include information in a flit indicating whether one or more preceding flits were null flits. As used herein, the phrase "null flit" refers to a flit having only NULL symbols in the fields reserved for transaction layer packet information.

<FIG> illustrates an exemplary flit definition for a x16 PCIe link with information indicating whether either of the two immediately preceding flits was a null flit. Flit <NUM> is a <NUM>-symbol flit comprising <NUM> transaction layer packet information symbols (TL0-TL299), six FEC symbols (three parity symbols FEC(P) <NUM> and three check symbols FEC(C) <NUM>), eight flit-level CRC symbols (CRC[<NUM>:<NUM>]) and six data link layer packet information symbols (DLL[<NUM>:<NUM>]) <NUM>. Data link layer packet information symbols <NUM> comprise a first bit P1 <NUM> indicating whether a first immediately preceding flit was a null flit and a second bit P2 <NUM> indicating whether a second immediately preceding flit was a null flit.

As used herein, the phrase "first immediately preceding flit" refers to the flit that was transmitted or received immediately prior to a particular flit, and the phrase "second immediately preceding flit" refers to the flit that was transmitted or received immediately prior to a first immediately preceding flit. As used herein, the phrase "immediately seceding flit" refers to the flit that was transmitted or received immediately after a particular flit. For example, if a receiving device receives flits A, B, and C, in that order, then with respect to flit C, flit B is the first immediately preceding flit and flit A is the second immediately preceding flit. Flit A is the first immediately preceding flit with reference to flit B, flit B is the immediately seceding flit with respect to flit A, and flit C is the immediately seceding flit with respect to flit B. The phrases "first immediately preceding flit," "second immediately preceding flit," and "immediately seceding flit" do not preclude the transmission or receipt of non-flit data across a link between flits. Regardless of what type or how much information is transmitted between two flits sent in succession across a link, the flit transmitted first is the first immediately preceding flit relative to the flit transmitted second, and the flit transmitted second is the immediately seceding flit with respect to the flit transmitted first.

Although flit <NUM> uses a single bit to indicate whether a preceding flit is a null flit, a null flit can be indicated in other manners. In some embodiments, more than one bit can be used to indicate a null flit. In other embodiments, the sequence number of a preceding flit can be stored along with the information indicating whether the preceding flit is a null flit. Further, null flit information can be stored for more than the two immediately preceding flits, or for any preceding flit, not just the immediately preceding ones (in which case the flit can store the sequence number of the preceding flit as well as the null flit indication). In some embodiments, a flit can store null flit information for just the first immediately preceding flit.

Information in a flit indicating that an immediately preceding flit is a null flit can also be used to indicate that an immediately preceding flit is not a null flit. For example, if a "<NUM>" in bits P1 and P2 in flit <NUM> indicates an immediately preceding bit is a null flit, a "<NUM>" indicates that an immediately preceding bit is a non-null fit.

In some embodiments where a flit contains information indicating that a preceding flit is a null flit, the null flit information can be used as follows. If a receiving device determines that flit n has an error that cannot be corrected with the FEC scheme, the receiving device waits until it receives flit (n+<NUM>) before deciding whether to send a retry request to request retransmission of flit n. If the receiving device determines that flit (n+<NUM>) has been received successfully after performing flit-level FEC and CRC checks, and flit (n+<NUM>) indicates that flit n was a null flit, there is no need for retransmission of flit n and a retry request is not sent.

If the receiving device determines that flit (n+<NUM>) also has an error, it waits for receipt of flit (n+<NUM>) and determines whether flit (n+<NUM>) has been successfully received. If flit (n+<NUM>) passes flit-level FEC and CRC checks, and flit (n+<NUM>) indicates that both flit n and (n+<NUM>) were null flits, then retransmission of flits n and (n+<NUM>) is not requested. If flit (n+<NUM>) indicates flit n and (n+<NUM>) were both not null flits, the receiving device sends a retry request requesting retransmission of flits n and (n+<NUM>). If flit (n+<NUM>) was not received successfully, a retry request requesting retransmission of flits from flit n onwards, which will cause flits n, (n +<NUM>), and (n+<NUM>) to be retransmitted, will be sent to the transmitting device.

This approach can be generalized such that a flit contains information indicating which of the immediately preceding m flits are null flits. For example, for the case where m=<NUM>, a flit contains information indicating whether each of the first, second, and third immediately preceding flits are null flits. In this scenario, if a receiver determines that three successive flits n, (n+<NUM>), and (n+<NUM>) have not been successfully received, it looks to flit (n+<NUM>) to decide what to do. If flit (n+<NUM>) is not successfully received, the receiver sends a retry request requesting retransmission of all flits starting from flit n, since the receiver cannot determine which of flits n, (n+<NUM>), and (n+<NUM>) may be null flits because flit (n+<NUM>) has an uncorrectable error. If flit (n+<NUM>) has been successfully received, the receiver determines if any of the preceding flits were null flits and requests retransmission of only the preceding flits that were not null flits. Thus, if flits n, (n+<NUM>), and (n+<NUM>) were all null flits, no retry request will be sent; if flits n, (n+<NUM>), and (n+<NUM>) were all non-null flits, they will all be retransmitted; if flits n and (n+<NUM>) were null flits and flit (n+<NUM>) was a non-null flit, only flit (n+<NUM>) will be resent.

A second approach for improving retry probability is to periodically send parity flits. Sending a parity flit after (n-<NUM>) flits allows the receiver to correct a flit in the group of n flits determined to have an error instead of having to send a retry request. If parity flits are used in combination with flits including information indicating whether one or more preceding flits was a null flit, if the corrupted flit was a null flit, the corrupted flit is not corrected. If more two or more flits in a group of n flits have errors, a request retry is sent requesting retransmission of the corrupted flits or requesting transmission from the first corrupted flit that is not a null flit received in the group.

<FIG> illustrates an exemplary use of parity flits and flits containing information indicating whether one or more preceding flits was a null flit. <FIG> shows groups A and B of <NUM> flits (<NUM> standard flits plus a parity flit) being sent from device <NUM> to device <NUM> with an optional retimer <NUM> between them. Flits marked with an "X" are determined by device <NUM> to have an error. In group A, received flits <NUM> and <NUM> have errors. Flit <NUM> contains information indicating that flit <NUM> was a null flit and receiver <NUM> thus does not need to send a retry request to have flit <NUM> be retransmitted. With receiver <NUM> only needing to correct one flit in group A then, it uses parity flit <NUM> to correct flit <NUM>, and thus sends no retry request to correct either of the flits that were received with an error in group A. In group B, flits <NUM> and <NUM> have errors and flits <NUM> and <NUM> contain information indicating that flits <NUM> and <NUM> are not null flits. Thus, receiver <NUM> sends a retry request to have all flits from flit <NUM> retransmitted upon receipt and processing of parity flit <NUM>, since the received flits of group B contains more corrupted flits (two) than can be corrected with the parity flit.

The parity flit in group B could have been used to correct the error in flit <NUM> had flit <NUM> been the only flit with an error. However, because a second flit in the group also had an error, receiver <NUM> sends a retry request requesting retransmission of all flits in group B from flit <NUM>. In some embodiments, a retry request can be sent as soon as a flit having an uncorrectable error is detected within a group, thus avoiding having to wait to see if the corrupted flit could be corrected with the parity flit. A receiving device can monitor the utilization rate of a return link (for example, return link <NUM> from device <NUM> to <NUM>) and send an earlier retry request if that link is being lightly utilized in order to reduce replay latency. That is, by sending a retry request for flit <NUM> in group B prior to receipt of parity flit <NUM>, when receiving device <NUM> eventually does receive parity flit <NUM> it can correct flit <NUM> since corrupted flit <NUM> has already been handled.

In some embodiments, a device can choose between a per-lane FEC scheme and a flit-level FEC scheme. Such a system could choose a flit-level FEC scheme over a per-lane FEC scheme to, for example, reduce latency or if consistent latencies are desired across PCIe links of varying widths with a fixed flit size to achieve performance objectives. A per-lane FEC scheme could be selected in favor of a flit-level FEC if, for example, a link with a lower retry rate is desirable to achieve performance objectives and the lane-to-lane error correction is high.

In some embodiments, a device can enable or disable the generation of parity flits. For example, a device can enable or disable parity flit generation in response to a determined bit error rate for a link. For example, if a determined bit error rate for a link exceeds an enable parity flit threshold value, parity flit generation is enabled. Similarly, if a determined bit error rate for a link drops below a disable parity flit threshold value, parity flit generation is disabled. The bit error rate for a link can be based at least in part on flits transmitted across the link and can be determined by either or both of transmitting and receiving devices connected to the link. The bit error rate can be determined over a pre-determined or variable timing window. In some embodiments, a user can enable or disable parity flits across a set of links for consistent behavior across the links after link pre-characterization.

In addition to the advantages of the technologies disclosed herein that have already been discussed, the technologies disclosed herein can be implemented with a relatively low gate count.

<FIG> illustrates an exemplary method of transmitting a flit with a flit-level FEC scheme. The method <NUM> can be performed by, for example, a PCIe root complex connected to a graphics card via a x4 PCIe link. At <NUM>, information is received from a processor. In the example, the root complex PCIe protocol stack receives a message to be sent to the graphics card. At <NUM>, transaction layer packets are generated based on the received information. In the example, the root complex PCIe protocol stack generates TLPs based on the message to be sent to the graphics card. At <NUM>, flits comprising the TLPs are generated. The flits comprise the transaction layer packets, and each flit is protected by a flit-level cyclic redundancy check (CRC) scheme and a flit-level forward error correction code (FEC) scheme. In the example, the root complex PCIe protocol stack generates one or more <NUM>-symbol flits containing the TLPs to be sent to the graphics card. The protocol stack protects each flit with eight flit-level CRC codes based on the contents of TLPs and six flit-level FEC codes based on the contents of the TLPs and the CRC codes. At <NUM>, the flits are transmitted across the lanes of a link to a receiver. In the example, the root complex transmits the flits across the four lanes of the x4 PCIe link to the graphics card. The <NUM>-symbol flits are each broken into four streams of <NUM> symbols and each stream for each flit is transmitted along one of the lanes of the x4 PCIe link.

Method <NUM> can optionally include additional actions. For example, the flit-level CRC codes and the flit-level FEC codes can be transmitted across multiple lanes. In the example, the root complex transmits the eight flit-level CRC codes across the four lanes, two per lane, and the six FEC codes across two of the lanes, three per lane. In another example, the transmitted flits are stored in a replay buffer, a retry request is received to retransmit retry flits from the flits transmitted to the receiver, the retry flits are retrieved from the replay buffer, and the retry flits are retransmitted to the receiving device. Retrieving the retry flits from the replay buffer comprises detecting the presence of a non-contiguous flit sequence stored in the replay buffer and reconstructing flits corresponding to gaps in the non-contiguous flit sequence as null flits to be transmitted as part of the retry flits. In the example, the PCIe root complex stores transmitted flits in its replay buffer except for null flits and receives a retry request from the graphics card to retransmit flits with sequence numbers <NUM> through <NUM>. The PCIe root complex retrieves flits <NUM> through <NUM> from the replay buffer. Flit <NUM> was a null flit and was therefore not stored in the replay buffer prior to transmission. As part of retrieving the flits from replay buffer, the PCIe root complex detects that the sequence numbers of the flits stored in the replay buffer jump from <NUM> to <NUM>. That is, the PCIe root complex detects a non-contiguous flit sequence and reconstructs flit <NUM> as a null fit. The PCIe root complex retransmits flits <NUM> through <NUM>, including reconstructed null flit <NUM> to the graphics card.

In another example, parity flits can be generated based on the one or more flits and the parity flit can be transmitted across multiple lanes of the link. In the example, the PCIe complex can generate parity flits for groups of <NUM> flits and transmit the parity flits across all four lanes of the x4 link to the graphics card. In another example, the flits can contain information indicating that one or more immediately preceding flits are null flits. In the example, the PCIe complex can generate flits containing a first bit indicating whether a first immediately preceding flit was a null flit and a second bit indicating whether a second immediately preceding flit was a null flit.

<FIG> illustrates an exemplary method of receiving a flit with a flit-level FEC scheme. The method <NUM> can be performed by, for example, a network interface card (NIC). At <NUM> flits are received over the lanes of a link. The flits comprise transaction layer packets, flit-level CRC codes, and flit-level FEC codes. In the example, the NIC receives a <NUM>-symbol flit comprising <NUM> TLPs, <NUM> flit-level CRC codes, and <NUM> flit-level FEC codes over a x8 PCIe link from a PCIe root complex. At <NUM>, it is determined whether the flit has been successively received based on the TLPs, the flit-level CRC codes, and the flit-level FEC codes in the flit. In the example, the NIC determines whether the flit has been successively received by performing a flit-level CRC check based at least on the <NUM> TLPs and the <NUM> CRC codes and by performing a flit-level FEC check based at least on the <NUM> TLPs, the <NUM> CRC codes, and the <NUM> FEC codes. At <NUM>, if the flit has been successively received, the information contained in the TLPs is sent to a processor. In the example, the payloads contained in the TLPs of the successively received flit is sent to the NIC processor.

Method <NUM> can optionally include additional actions. For example, a parity flit can be received and if the flit is determined to have not been received successfully, the corrupted flit can be corrected using the parity flit. In the example, the NIC receives a parity flit after receiving the <NUM>-symbol flit. The NIC determines that the flit has not been successively after running a flit-level FEC check using the flit-level FEC symbols and a flit-level CRC check using the CRC codes. The NIC then corrects the corrupted flit using the parity flit. In another example, the flit further comprises information indicating whether one or more immediately preceding flits is a null flit. If the received flit is determined to have not been successfully received, information in the immediately seceding flit is examined to determine whether the corrupted flit was a null flit. If the corrupted flit was a null flit, a retry request to request retransmission of the corrupt flit is not sent. If the corrupted was not a null flit, a retry request is sent. In the example, flits received by the NIC contain information indicating whether the first immediately preceding flit was a null flit. The NIC determines that the flit was not received successfully and receives a second flit, which indicates that the corrupted flit was a null flit, and the NIC does not send a retry request to request retransmission of the corrupted flit.

The technologies, techniques, and embodiments described herein can be performed by any of a variety of computing devices, including mobile devices (e.g., smartphones, tablet computers, laptop computers, media players, portable gaming consoles, cameras and video recorders), non-mobile devices (e.g., desktop computers, servers, stationary gaming consoles, set-top boxes, smart televisions) and embedded devices (e.g., devices incorporated into a vehicle, home, or place of business). As used herein, the term "computing devices" includes computing systems and includes devices comprising multiple discrete physical components.

<FIG> is a block diagram of an exemplary computing device for transmitting and receiving flits. Generally, components shown in <FIG> can communicate with other shown components, although not all connections are shown, for ease of illustration. A device <NUM> is a multiprocessor system comprising a first processor <NUM> and a second processor <NUM> and is illustrated as comprising point-to-point (P-P) interconnects. For example, a point-to-point (P-P) interface <NUM> of processor <NUM> is coupled to a point-to-point interface <NUM> of processor <NUM> via a point-to-point interconnection <NUM>. It is to be understood that any or all point-to-point interconnects illustrated in <FIG> or in any other figure can be alternatively implemented as a multi-drop bus, and that any or all buses illustrated in <FIG> or any other figure could be replaced by point-to-point interconnects.

As shown in <FIG>, processors <NUM> and <NUM> are multicore processors. Processor <NUM> comprises processor cores <NUM> and <NUM>, and processor <NUM> comprises processor cores <NUM> and <NUM>. Processors <NUM>-<NUM> can execute computer-executable instructions in a manner similar to that discussed below in connection with <FIG>, or in other manners.

Processors <NUM> and <NUM> further comprise at least one shared cache memory <NUM> and <NUM>, respectively. The shared caches <NUM> and <NUM> can store data (e.g., instructions) utilized by one or more components of the processor, such as the processors <NUM>-<NUM> and <NUM>-<NUM>. The shared caches <NUM> and <NUM> can be part of a memory hierarchy for device <NUM>. For example, shared cache <NUM> can locally store data that is also stored in a memory <NUM> to allow for faster access to the data by components of the processor <NUM>. In some embodiments, shared caches <NUM> and <NUM> can comprise multiple cache layers, such as level <NUM> (L1), level <NUM> (L2), level <NUM> (L3), level <NUM> (L4), and/or other caches or cache layers, such as a last level cache (LLC).

Although the device <NUM> is shown with two processors, it can comprise any number of processors. Further, a processor can comprise any number of processors. A processor can take various forms such as a central processing unit, a controller, a graphics processor, an accelerator (such as a graphics accelerator or digital signal processor (DSP)) or a field programmable gate array (FPGA). A processor in a device can be the same as or different from other processors in the device. In some embodiments, device <NUM> can comprise one or more processors that are heterogeneous or asymmetric to a first processor, accelerator, FPGA, or any other processor. There can be a variety of differences between the processing elements in a system in terms of a spectrum of metrics of merit including architectural, microarchitectural, thermal, power consumption characteristics and the like. These differences can effectively manifest themselves as asymmetry and heterogeneity amongst the processors in a system. In some embodiments, processors <NUM> and <NUM> reside in the same die package.

Processors <NUM> and <NUM> further comprise memory controller logics (MC) <NUM> and <NUM>. As shown in <FIG>, MCs <NUM> and <NUM> control memories <NUM> and <NUM> coupled to processors <NUM> and <NUM>, respectively. The memories <NUM> and <NUM> can comprise various types of memories, such as volatile memory (e.g., dynamic random-access memories (DRAM), static random-access memory (SRAM)) or non-volatile memory (e.g., flash memory, chalcogenide-based phase-change non-volatile memory). While MCs <NUM> and <NUM> are illustrated as being integrated into processors <NUM> and <NUM>, in alternative embodiments, the MCs can be logic external to a processor, and can comprise one or more layers of a memory hierarchy.

Processors <NUM> and <NUM> are coupled to an Input/Output (I/O) subsystem <NUM> via P-P interconnections <NUM> and <NUM>. Point-to-point interconnection <NUM> connects a point-to-point interface <NUM> of processor <NUM> with a point-to-point interface <NUM> of I/O subsystem <NUM>, and point-to-point interconnection <NUM> connects a point-to-point interface <NUM> of processor <NUM> with a point-to-point interface <NUM> of I/O subsystem <NUM>. Input/Output subsystem <NUM> further includes an interface <NUM> to couple I/O subsystem <NUM> to a graphics engine <NUM>, which can be a high-performance graphics engine, graphics processing unit, or a graphics card. The I/O subsystem <NUM> and graphics engine <NUM> are coupled via a link <NUM>, which could be a bus or a point-to-point interconnection.

Input/Output subsystem <NUM> is further coupled to a first bus <NUM> via an interface <NUM>. The first bus <NUM> can be a Peripheral Component Interconnect (PCI) bus, a PCI Express bus, another third generation I/O interconnection bus or any other type of bus.

Various I/O devices <NUM> can be coupled to the first bus <NUM>. A bus bridge <NUM> can couple first bus <NUM> to a second bus <NUM>. In some embodiments, the second bus <NUM> can be a low pin count (LPC) bus. Various devices can be coupled to the second bus <NUM> including, for example, a keyboard/mouse <NUM>, audio I/O devices <NUM> and a storage device <NUM>, such as a hard disk drive, solid-state drive or other storage device for storing computer-executable instructions (code) <NUM>. The code <NUM> can comprise computer-executable instructions for performing technologies described herein. Additional components that can be coupled to the second bus <NUM> include communication device(s) <NUM>, which can provide for communication between the device <NUM> and one or more wired or wireless networks <NUM> (e.g. Wi-Fi, cellular or satellite networks) via one or more wired or wireless communication links (e.g., wire, cable, Ethernet connection, radio-frequency (RF) channel, infrared channel, Wi-Fi channel) using one or more communication standards (e.g., IEEE <NUM> standard and its supplements).

The device <NUM> can comprise removable memory such as flash memory cards (e.g., SD (Secure Digital) cards), memory sticks, Subscriber Identity Module (SIM) cards). The memory in device <NUM> (including caches <NUM> and <NUM>, memories <NUM> and <NUM> and storage device <NUM>) can store data and/or computer-executable instructions for executing an operating system <NUM> and application programs <NUM>. Example data includes web pages, text messages, images, sound files, video data, or other data sets to be sent to and/or received from one or more network servers or other devices by the device <NUM> via one or more wired or wireless networks, or for use by the device <NUM>. The device <NUM> can also have access to external memory (not shown) such as external hard drives or cloud-based storage.

The operating system <NUM> can control the allocation and usage of the components illustrated in <FIG> and support one or more application programs <NUM>. The application programs <NUM> can include common mobile computing device applications (e.g., email applications, calendars, contact managers, web browsers, messaging applications) as well as other computing applications.

The device <NUM> can support various input devices, such as a touch screen, microphone, camera, physical keyboard, proximity sensor and trackball, and one or more output devices, such as a speaker and a display. Other possible input and output devices include piezoelectric and other haptic I/O devices. Any of the input or output devices can be internal to, external to or removably attachable with the device <NUM>. External input and output devices can communicate with the device <NUM> via wired or wireless connections.

In addition, the computing device <NUM> can provide one or more natural user interfaces (NUIs). For example, the operating system <NUM> or applications <NUM> can comprise speech recognition logic as part of a voice user interface that allows a user to operate the device <NUM> via voice commands. Further, the device <NUM> can comprise input devices and logic that allows a user to interact with the device <NUM> via a body, hand or face gestures. For example, a user's hand gestures can be detected and interpreted to provide input to a gaming application.

The device <NUM> can further comprise one or more communication components <NUM>. The components <NUM> can comprise wireless communication components coupled to one or more antennas to support communication between the system <NUM> and external devices. The wireless communication components can support various wireless communication protocols and technologies such as Near Field Communication (NFC), Wi-Fi, Bluetooth, <NUM> Long Term Evolution (LTE), <NUM> New Radio (<NUM>), Code Division Multiplexing Access (CDMA), Universal Mobile Telecommunication System (UMTS) and Global System for Mobile Telecommunication (GSM). In addition, the wireless modems can support communication with one or more cellular networks for data and voice communications within a single cellular network, between cellular networks, or between the mobile computing device and a public switched telephone network (PSTN).

The device <NUM> can further include at least one input/output port (which can be, for example, a USB, IEEE <NUM> (FireWire), Ethernet and/or RS-<NUM> port) comprising physical connectors; a power supply; a satellite navigation system receiver, such as a GPS receiver; a gyroscope; an accelerometer; a proximity sensor; and a compass. A GPS receiver can be coupled to a GPS antenna. The device <NUM> can further include one or more additional antennas coupled to one or more additional receivers, transmitters and/or transceivers to enable additional functions.

In one embodiment, I/O subsystem <NUM> is a PCIe root complex, I/O devices <NUM> are PCI endpoints, and I/O subsystem <NUM> connects to I/O devices <NUM> and bus bridge <NUM> via any current version of the PCI expression protocol.

It is to be understood that <FIG> illustrates only one exemplary computing device architecture for transmitting and receiving flits. Computing devices based on alternative architectures can be used to implement technologies described herein. For example, instead of the processors <NUM> and <NUM>, and the graphics engine <NUM> being located on discrete integrated circuits, a computing device can comprise an SoC (system-on-a-chip) integrated circuit incorporating multiple processors, a graphics engine and additional components. Further, a computing device can connect elements via bus or point-to-point configurations different from that shown in <FIG>. Moreover, the illustrated components in <FIG> are not required or all-inclusive, as shown components can be removed and other components added in alternative embodiments.

<FIG> is a block diagram of an exemplary processor core that can execute instructions as part of implementing technologies described herein. The processor <NUM> can be a core for any type of processor, such as a microprocessor, an embedded processor, a digital signal processor (DSP) or a network processor. The processor <NUM> can be a single-threaded core or a multithreaded core in that it may include more than one hardware thread context (or "logical processor") per core.

<FIG> also illustrates a memory <NUM> coupled to the processor <NUM>. The memory <NUM> can be any memory described herein or any other memory known to those of skill in the art. The memory <NUM> can store computer-executable instruction <NUM> (code) executable by the processor <NUM>.

The processor comprises front-end logic <NUM> that receives instructions from the memory <NUM>. An instruction can be processed by one or more decoders <NUM>. The decoder <NUM> can generate as its output a micro operation such as a fixed width micro operation in a predefined format, or generate other instructions, microinstructions, or control signals, which reflect the original code instruction. The front-end logic <NUM> further comprises register renaming logic <NUM> and scheduling logic <NUM>, which generally allocate resources and queues operations corresponding to converting an instruction for execution.

The processor <NUM> further comprises execution logic <NUM>, which comprises one or more execution units (EUs) <NUM>-<NUM> through <NUM>-N. Some processor embodiments can include several execution units dedicated to specific functions or sets of functions. Other embodiments can include only one execution unit or one execution unit that can perform a particular function. The execution logic <NUM> performs the operations specified by code instructions. After completion of execution of the operations specified by the code instructions, back-end logic <NUM> retires instructions using retirement logic <NUM>. In some embodiments, the processor <NUM> allows out of order execution but requires in-order retirement of instructions. Retirement logic <NUM> can take a variety of forms as known to those of skill in the art (e.g., re-order buffers or the like).

The processor <NUM> is transformed during execution of instructions, at least in terms of the output generated by the decoder <NUM>, hardware registers and tables utilized by the register renaming logic <NUM>, and any registers (not shown) modified by the execution logic <NUM>. Although not illustrated in <FIG>, a processor can include other elements on an integrated chip with the processor <NUM>. For example, a processor may include additional elements such as memory control logic, one or more graphics engines, I/O control logic and/or one or more caches.

Any of the disclosed methods can be implemented as computer-executable instructions or a computer program product. Such instructions can cause a computer to perform any of the disclosed methods. Generally, as used herein, the term "computer" refers to any computing device or system described or mentioned herein, or any other computing device. Thus, the terms "computer-executable instruction" or "machine-executable instruction" refers to instructions that can be executed by any computing device described or mentioned herein, or any other computing device.

The computer-executable instructions or computer program products as well as any data created and used during implementation of the disclosed technologies can be stored on one or more tangible or non-transitory computer-readable storage media, such as optical media discs (e.g., DVDs, CDs), volatile memory components (e.g., DRAM, SRAM), or non-volatile memory components (e.g., flash memory, solid state drives, chalcogenide-based phase-change non-volatile memories). Computer-readable storage media can be contained in computer-readable storage devices such as solid-state drives, USB flash drives, and memory modules. Alternatively, the computer-executable instructions may be performed by specific hardware components that contain hardwired logic for performing all or a portion of disclosed methods, or by any combination of computer-readable storage media and hardware components.

The computer-executable instructions can be part of, for example, a dedicated software application or a software application that is accessed via a web browser or other software application (such as a remote computing application). Such software can be executed, for example, on a single computing device or in a network environment using one or more network computers. Further, it is to be understood that the disclosed technology is not limited to any specific computer language or program. For instance, the disclosed technologies can be implemented by software written in C++, Java, Perl, JavaScript, Adobe Flash, or any other suitable programming language. Likewise, the disclosed technologies are not limited to any particular computer or type of hardware.

Furthermore, any of the software-based embodiments (comprising, for example, computer-executable instructions for causing a computer to perform any of the disclosed methods) can be uploaded, downloaded or remotely accessed through suitable communication methods. Such suitable communication methods include, for example, the Internet, the World Wide Web, an intranet, cable (including fiber optic cable), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), or electronic communications.

As will be appreciated by one skilled in the art, the present disclosure may be embodied as methods or computer program products. Accordingly, the present disclosure, in addition to being embodied in hardware as earlier described, may take the form of an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to as a "circuit," "module," or "system.

Furthermore, the present disclosure may take the form of a computer program product embodied in any tangible or non-transitory medium of expression having computer-usable program code embodied in the medium. <FIG> illustrates an example computer-readable non-transitory storage medium that may be suitable for use to store instructions that cause an apparatus, in response to execution of the instructions by the apparatus, to practice selected aspects of the present disclosure. As shown, non-transitory computer-readable storage medium <NUM> may include several programming instructions <NUM>. Programming instructions <NUM> may be configured to enable a device, e.g., device <NUM>, in response to execution of the programming instructions, to perform, e.g., various programming operations associated with operating system functions and/or applications, in particular, operations associated with flit based packetization technology described above with references to <FIG>.

In alternate embodiments, programming instructions <NUM> may be disposed on multiple computer-readable non-transitory storage media <NUM> instead. In alternate embodiments, programming instructions <NUM> may be disposed on computer-readable transitory storage media <NUM>, such as signals. Any combination of one or more computer usable or computer readable medium(s) may be utilized. More specific examples (a non- exhaustive list) of the computer-readable medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a transmission media such as those supporting the Internet or an intranet, or a magnetic storage device. In the context of this document, a computer-usable or computer-readable medium may be any medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-usable medium may include a propagated data signal with the computer-usable program code embodied therewith, either in baseband or as part of a carrier wave. The computer usable program code may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc..

Computer program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages.

The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure.

These computer program instructions may also be stored in a computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instruction means which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. As used herein, "computer-implemented method" may refer to any method executed by one or more processors, a computer system having one or more processors, a mobile device such as a smartphone (which may include one or more processors), a tablet, a laptop computer, a set-top box, a gaming console, and so forth.

Embodiments may be implemented as a computer process, a computing system or as an article of manufacture such as a computer program product of computer readable media. The computer program product may be a computer storage medium readable by a computer system and encoding a computer program instruction for executing a computer process.

The disclosed embodiments may be implemented, in some cases, in a hardware component or device, firmware or software running on a processor, or a combination thereof, or any combination thereof.

As used in this application and in the claims, a list of items joined by the term "and/or" can mean any combination of the listed items. For example, the phrase "A, B and/or C" can mean A; B; C; A and B; A and C; B and C; or A, B and C. As used in this application and in the claims, a list of items joined by the term "at least one of" can mean any combination of the listed terms. For example, the phrase "at least one of A, B or C" can mean A; B; C; A and B; A and C; B and C; or A, B, and C.

Although the operations of some of the disclosed methods are described in a particular, sequential order for convenient presentation, it is to be understood that this manner of description encompasses rearrangement, unless a particular ordering is required by specific language set forth herein.

As used in any embodiment herein, the term "module" may refer to software, firmware and/or circuitry to perform one or more operations consistent with the present disclosure. Software may be embodied as a software package, code (e.g., routines, programs, objects, components, data structures), instructions, instruction sets and/or data recorded on non-transitory computer readable storage mediums. Firmware may be embodied as code, instructions or instruction sets and/or data that are hard-coded (e.g., nonvolatile) in memory devices.

As used herein, the term "circuitry" refers to, is part of, or includes hardware components such as an electronic circuit, a logic circuit, a processor (shared, dedicated, or group) and/or memory (shared, dedicated, or group), an Application Specific Integrated Circuit (ASIC), a field-programmable device (FPD), (for example, a field-programmable gate array (FPGA), a programmable logic device (PLD), a complex PLD (CPLD), a high-capacity PLD (HCPLD), a structured ASIC, or a programmable System on Chip (SoC)), digital signal processors (DSPs), etc., that provide the described functionality. In some embodiments, the circuitry may execute one or more software or firmware programs to provide at least some of the described functionality.

The term "circuitry" may also refer to processor circuitry or interface circuitry. As used herein, the term "processor circuitry" may refer to, is part of, or includes circuitry to sequentially and automatically carrying out a sequence of arithmetic or logical operations; recording, storing, and/or transferring digital data. The term "processor circuitry" may refer to one or more application processors, one or more baseband processors, a physical central processing unit (CPU), a single-core processor, a dual-core processor, a triple-core processor, a quad-core processor, and/or any other device that can execute or otherwise operate computer-executable instructions, such as program code, software modules, and/or functional processes. As used herein, the term "interface circuitry" may refer to, is part of, or includes circuitry providing for the exchange of information between two or more components or devices. The term "interface circuitry" may refer to one or more hardware interfaces (for example, buses, input/output (I/O) interfaces, peripheral component interfaces, network interface cards, and/or the like).

Claim 1:
A method (<NUM>) for transmission across a link having multiple lanes, comprising:
creating a transaction layer packet, TLP, (<NUM>);
creating a data link layer packet, DLLP;
calculating a cyclic redundancy check, CRC, code for the TLP, the CRC code comprising eight bytes;
calculating an error correcting code, ECC, for the TLP, the DLLP and the CRC code;
generating a flit comprising a plurality of symbols, the plurality of symbols including TLP symbols to store the TLP, DLLP symbols to store the DLLP, eight consecutive CRC symbols to store the CRC code and ECC symbols to store the ECC (<NUM>), the DLLP symbols being placed after the TLP symbols in the flit, the CRC symbols being placed after the DLLP symbols in the flit, and the ECC symbols being placed after the CRC symbols in the flit; and
transmitting the flit across the multiple lanes of the link (<NUM>).