Logical transport over a fixed PCIE physical transport network

A method and a system for transparently overlaying a logical transport network over an existing physical transport network is disclosed. The system designates a virtual channel located in a first transaction layer of a network conforming to a first network protocol. The system assembles a transaction layer packet in a second logical transaction layer of a second network protocol that is also recognizable by the first transaction layer. The system transfers the transaction layer packet from the second transaction layer to the virtual channel. The system transmits the transaction layer packet over the first transaction layer using the designated virtual channel over the network.

TECHNICAL FIELD

Examples of the present disclosure generally relate to bus architectures of computing systems and, in particular, to transparently overlaying a logical transport network over an existing physical transport network.

BACKGROUND

Personal computer and server architectures have employed bus systems that have grown increasingly sophisticated with time. The Peripheral Component Interconnect (PCI) bus presented a number of advantages over previous bus implementations. Among the most important were processor independence, buffered isolation, bus mastering, and true plug-and-play operation. Buffered isolation essentially isolates, both electrically and by clock domains, the central processing unit (CPU) local bus from the PCI bus. Plug-and-play operation, which permits devices to be automatically detected and configured, eliminated the manual setting of switches and jumpers for base address and direct memory access (DMA) interrupts that frustrated Industry Standard Architecture (ISA)-based board users.

Although PCI has enjoyed great success, it faces a series of challenges, including bandwidth limitations, host pin-count limitations, the lack of real-time data transfer services such as isochronous data transfers, and the lack of features to meet next-generation input/output (I/O) requirements such as quality of service, power management, cache coherency, and I/O virtualization. Next-generation I/O requirements such as quality of service measurements and power management improve data integrity and permit selective powering down of system devices—an important consideration as the amount of power require by modern PCs continues to grow. Some of these features require software control beyond traditional PCI requirements and are not available until OS and device driver support is available.

Peripheral Component Interface Express (PCIe) has filled in many of the gaps in traditional PCI implementations, but PCIe has certain limitations in terms of command features, routing, and lack of cache coherency. Protocols with load/store or coherency semantics typically require same-address ordering between operations/events to that address location, regardless of the source of the device performing the load/store, in order to maintain functional correctness. Cache coherence is the uniformity of shared resource data that is stored in multiple local caches. When clients in a system maintain caches of a common memory resource, problems may arise with incoherent data, which is particularly the case with CPUs in a multiprocessing system.

One option to overcome the lack of load/store or cache coherency semantics in a PCIe bus-based architecture would be to replace all hardware and software related to the PCIe bus with a cache coherent bus architecture having the same or additional advantages as PCIe bus. However, full replacement of hardware is expensive, time consuming, and prone to error.

Therefore, it is desirable to provide a cache coherent bus architecture that can in-part be overlaid on an existing bus architecture without replacing all of the physical layer hardware of the underlying bus structure.

SUMMARY

Techniques for transparently overlaying a logical transport network over an existing physical transport network are disclosed. In an example, a computing system designates a virtual channel located in a first transaction layer of a network conforming to a first network protocol. The computer system assembles a transaction layer packet in a second logical transaction layer of a second network protocol that is also recognizable by the first transaction layer. The computer system transfers the transaction layer packet from the second transaction layer to the virtual channel. The computer transmits the transaction layer packet over the first transaction layer using the designated virtual channel over the network.

In another example, a computer system receives a transaction layer packet located in a first transaction layer of a network conforming to a first network protocol. The computer system extracts the transaction layer packet from a virtual channel located in the first transaction layer that is designated for use by a second logical transaction layer that conforms to a second network protocol. The computer system assembles a transaction layer packet that conforms to the second network protocol recognizable to the second logical transaction layer. The computer system transfers the transaction layer packet from the virtual channel to the second transaction layer.

DETAILED DESCRIPTION

Techniques for transparently overlaying a logical transport network over an existing physical transport network are disclosed. The combined logical/physical transport network leverages existing properties of the physical transport, but also includes class of service (COS) attributes that are unique to that logical transport network that not available natively over the physical transport network. In one example, the physical transport network and protocol may be, but is not limited to, the Peripheral Component Interconnect Express (PCIe) network. The PCIe protocol stack includes an embedded physical layer and data link layer. In one example, the logical transport network may include, but is not limited to, the transport layer of the Cache Coherence Interconnect for Accelerators (CCIX) protocol. The techniques described herein designate a PCIe virtual channel (VC) for logical transport of CCIX messages. The techniques create Class-of-Service attributes for the logical transport network that are transparently carried over the PCIe physical transport layer, via PCIe third party messages known as Vendor Defined Messages (VDMs). The techniques create optimized transaction layer packets (TLP), different from the PCIe standard TLP, carried only over the designated VC. The optimized TLPs contain low-latency and protocol attributes that are specific to the overlaid logical transport network.

Examples of the preset disclosure leverage available PCIe mechanisms that satisfy the requirements of the CCIX logical transport. The CCIX logical transport attributes are overlaid on PCIe defined mechanisms located in its physical electrical and physical logical layers, its data link layer, and its transaction layer with VDMs. The CCIX Packet definition, different from the PCIe TLP Packet definition, is overlaid on to PCIe VDMs of the transaction layer, and the CCIX property of in-order packet delivery is also achieved via the PCIe TLP property of in—order delivery of VDMs.

Examples of the present disclosure define new mechanisms, when those mechanisms are needed for the logical transport, but not available natively on the physical transport. This mechanism involves designating a PCIe Virtual Channel (VC) where PCIe VDMs on that VC contain properties unique to the CCIX logical transport. The properties include Class of Service attributes, and CCIX virtual channels transported within that designated PCIe virtual channel. The CCIX functions derived from those properties are available universally over devices connected via the PCIe transaction layer, since these VDMs are transparently carried over PCIe.

Examples of the present disclosure further permit the creation of optimized transaction layer packets (TLP), different from the PCIe standard TLP, that are carried only from one device with that designated VC to another device with that same designated VC. The optimized TLP contain low-latency and protocol attributes that are specific to CCIX.

FIG. 1is a block diagram depicting a system100for transparently overlaying a logical transport network over an existing physical transport network, according to an example. The system100includes a computer system102. The computer system102includes a hardware platform (“hardware104”) and a software platform (“software106”) executing on the hardware104. The hardware104includes a processing system110, system memory116, storage devices (“storage118”), and a peripheral device122. The software106includes an operating system (OS)144, device drivers146, and applications150. The OS144is configured to implement a combined CCIX/PCIe protocol stack152configured to implement the CCIX transaction layer over the PCIe data link layer and physical layer according to embodiments.

The processing system110further includes a microprocessor112, support circuits114, and a peripheral bus115. The microprocessor112can be any type of general-purpose central processing unit (CPU), such as an x86-based processor, ARM®-based processor, or the like. The microprocessor112can include one or more cores and associated circuitry (e.g., cache memories, memory management units (MMUs), interrupt controllers, etc.). The microprocessor112is configured to execute program code that perform one or more operations described herein and which can be stored in the system memory116and/or the storage118. The support circuits114include various devices that cooperate with the microprocessor112to manage data flow between the microprocessor112, the system memory116, the storage118, the peripheral device122, or any other peripheral device. For example, the support circuits114can include a chipset (e.g., a north bridge, south bridge, platform host controller, etc.), voltage regulators, firmware (e.g., a BIOS), and the like. The support circuits114manage data flow between the microprocessor112and the peripheral bus115, to which various peripherals, such as the peripheral device122, are connected. In some examples, the microprocessor112can be a System-in-Package (SiP), System-on-Chip (SoC), or the like, which absorbs all or a substantial portion of the functionality of the chipset (e.g., north bridge, south bridge, etc.). The peripheral bus can implement an expansion bus standard, such as Peripheral Component Interconnect Express (PCIe). In the example, the processing system110is shown separate from the peripheral device122. In other examples discussed further below, the processing system110and the peripheral device122can be implemented on the same integrated circuit (IC).

The system memory116is a device allowing information, such as executable instructions and data, to be stored and retrieved. The system memory116can include, for example, one or more random access memory (RAM) modules, such as double-data rate (DDR) dynamic RAM (DRAM). The storage118includes local storage devices (e.g., one or more hard disks, flash memory modules, solid state disks, and optical disks) and/or a storage interface that enables the computer system102to communicate with one or more network data storage systems. The hardware104can include various other conventional devices and peripherals of a computing system, such as graphics cards, universal serial bus (USB) interfaces, and the like.

The peripheral device122includes a programmable IC128, a non-volatile memory124, and RAM126. The programmable IC128can be a field programmable gate array (FPGA) or the like or an SoC having an FPGA or the like. The NVM124can include any type of non-volatile memory, such as flash memory or the like. The RAM126can include DDR DRAM or the like. The programmable IC128is coupled to the NVM124and the RAM126. The programmable IC128is also coupled to the peripheral bus115of the processing system110. The programmable IC128may further include PCIe circuit logic130A implementing the physical layer of the peripheral bus115, and include CCIX custom logic130B for implementing hardware features specific to CCIX transport protocol aware devices, such as the peripheral device122, and for implementing CCIX firmware119.

The OS144can be any commodity operating system known in the art. The drivers146include drivers and libraries that provide application programming interfaces (APIs) to the peripheral device122for command and control thereof. The applications150include software executing on the microprocessor112that invokes the peripheral device122through the combined CCIX/PCIe protocol stack152to transmit CCIX/PCIe attributes and transaction layer packets (TLPs). The applications150can include neural network, video processing, network processing, or the like type applications that offload some functions to the peripheral device122.

The CCIX architecture specification comprises five discrete logical layers. These include a CCIX protocol layer, a CCIX link layer, and the CCIX transaction layer. The CCIX transaction layers, data link and physical layers together are referred to as CCIX transport. The CCIX architectural specification also includes the PCIe transaction layer, PCIe data link layer and PCIe physical layer. Each of these layers is divided into two sections: one that processes outbound (to be transmitted) information and one that processes inbound (received) information.

FIG. 2shows a PCIe device200with CCIX functionality that permits the PCIe device200to have load/store and coherency semantics over PCIe. The PCIe device200is coupled to one or more gigabit transceivers (GTs)202. The PCIe device200retains the PCIe physical layer204and the PCIe data link layer206, but permits the PCIe transaction layer210to exist alongside with the CCIX transaction layer208so that CCIX-aware devices can employ features of the CCIX transport. The CCIX specification does not modify the PCIe transaction layer210. The primary responsibility of the PCIe transaction layer210is the assembly and disassembly of transaction layer packets (TLPs). TLPs are used to communicate transactions (e.g., read, write), as well as certain types of events. The PCIe transaction layer210is also responsible for managing credit-based flow control for TLPs. The CCIX specification does not modify the PCIe data link layer206and the PCIe physical layer204and utilizes it as is. The PCIe data link layer206functions as an intermediate stage between the PCIe transaction layer210and the CCIX transaction layer208. The primary responsibilities of the PCIe data link layer206include link management and data integrity, including error detection and error correction.

The technique of transparently overlaying a logical transport network over an existing physical transport (e.g., PCIe) defines a new transaction layer, e.g., the CCIX transaction layer208. The CCIX transaction layer208replaces the PCIe transaction layer210in one of the PCIe Virtual Channels (VCs) in a multi-VC implementation. The CCIX transaction layer208is a reduced PCIe transaction layer which supports optimized CCIX-compatible TLPs and PCIe-compatible TLPs.

The CCIX Transaction Layer's primary responsibility is the assembly and disassembly of CCIX Transaction Layer Packets (TLPs). On a receive path, the CCIX transaction layer208checks CCIX TLP integrity, before forwarding the TLP to the PCIe data link layer206. For PCIe Compatible TLPs, the PCIe transaction layer210checks specified in the PCI Express Base Specification are applicable. For Optimized TLPs, a new set of CCIX Transaction Layer checks are specified.

The CCIX transaction layer208is also responsible for managing credit-based flow control for CCIX TLPs. On the receive path, posted flow control credits are returned for CCIX TLPs that pass data integrity checks and are forwarded to the CCIX transaction layer208. In the transmit path, a credit gate is implemented to control flow of CCIX TLPs based on available posted credit. These posted credits are defined on a link-wide basis.

CCIX uses transaction layer packets (TLPs) to communicate information over the PCIE data link layer206and physical layer204. As the transmitted packets flow downstream through CCIX transaction layer208, the PCIE data link layer206, and the PCIe physical layer204, the CCIX packets are extended with additional information necessary to handle packets at each of the aforementioned layers. At a receiving side, a reverse process occurs and packets are transformed from a PCIe physical layer204representation to a PCIE data link layer206representation and finally (for transaction layer packets) to the form that can be processed by the CCIX transaction layer208of the receiving device.

CCIX transport is over a designated PCIe virtual channel. The designated VC enables performance isolation from other PCIe traffic and also prevents resource dependency deadlocks with other PCIe traffic. CCIX protocol messages are transmitted via third party PCIe vendor defined messages (VDMs). PCIe VDMs leverage an ordering properties of VDMs and the performance property of VDMs being a posted, i.e., fire & forget, transaction. A sender does not need to wait for a packet to be acknowledged and can also rely on the PCIe ordered network delivering multiple VDMs in temporal order. With a designated virtual channel for CCIX traffic, the CCIX transaction layer208can be designated. Traffic appearing over PCIe transport bifurcates to the CCIX transaction layer208or the PCIe transaction layer210, depending on whether the traffic is on the CCIX or non-CCIX VC. This offers latency and bandwidth advantages, which are important to load/store and coherency protocols in general. Also, a CCIX transaction layer208that is independent of PCIe also offers latency and performance advantages by permitting features not available in the PCIe transaction layer—this includes CCIX request chaining and CCIX message packing.

FIG. 3shows a layout of a PCIe/TLP packet300. The PCIe/TLP packet300includes at least a payload302, a vendor ID (vendor-specific identifier) field304, and a message code field306. The OS144creates and employs a PCIe virtual channel for handling packet traffic between devices. Once CCIX vendor defined VDMs are detected by the OS144, the CCIX firmware119enables a PCIe virtual channel in the CCIX transaction layer208that is not known to the PCIe transaction layer210. Once the CCIX custom logic130B discovers the enablement of the CCIX-designated PCIe virtual channel, the CCIX custom logic130B proceeds to discover whether PCIe packets located in the combined CCIX/PCIe protocol stack152contain vendor defined messages in its packets that are recognizable to CCIX-based devices. PCIe is configured to recognize an existing third party vendor identifier (ID) and a third party packet type within its protocol called a vendor defined message. CCIX uses vendor defined messages that every CCIX-based device from any vendor of CCIX can recognize and employ in communication. The PCIe transaction layer210recognizes PCIe attributes on the CCIX virtual channel, and is unaware that these attributes are designated as CCIX attributes that extend the capabilities of PCIe.

FIG. 3shows a layout of a PCIe/TLP packet300. The PCIe/TLP packet300includes a header301and a payload302. Referring toFIGS. 1 and 3, CCIX-aware firmware119detects in a PCIE compatible TLP packet300having a vendor-specific identifier called a vendor ID304in the header301. The CCIX firmware119then examines a message code field306in the header301that identifies the packet as a vendor defined message. In response to detecting the CCIX header301containing a VDM associated with CCIX-communication, the CCIX-aware firmware219examines the content of the CCIX-based header301of a CCIX TLP embedded within the TLP packet300to determine how to process the CCIX packet. The CCIX firmware219then informs the CCIX-aware OS144that information contained in the VDM is associated with a CCIX-aware application, and that the CCIX-aware OS144should expect to process one or more CCIX-type TLPs received over the CCIX transaction layer208.

CCIX further employs a software data structure, that is a counterpart to the vendor defined message, called Designated Vendor-Specific Expended Capabilities (DVSEC). This DVSEC software data structure is device vendor-specific. When the OS144recognizes a DVSEC vendor defined message identifier, the OS144recognizes that accompanying DVSEC packet has additional attributes not recognizable by PCIe. The OS144then consults the CCIX-aware firmware119to interpret the DVSEC attribute and packet. Thus, the PCIe portion of the OS144is made aware that it needs to handle CCIX packets, but remains unaware how to interpret the packet's embedded attributes. The OS144then forwards the DVSEC message to the CCIX transaction layer208of the combined CCIX/PCIe protocol stack152and the CCIX firmware119for further interpretation.

FIG. 4is a block diagram of a PCIe network400having a tree topology. The PCIe network400includes a host/server402that includes a PCIe root port404connected to a PCIe switch406, which communicates with a set of PCIe devices408a-408nand a plurality of CCIX-aware devices410a-410n. Traffic that is to travel from one CCIX device (e.g.,410a) to another CCIX device (e.g.,410n) using VDMs over a virtual channel needs to traverse the PCIe switch406before travelling to a requested destination.

The messages sent between PCIe devices408a-408nare substantially point-to-point messages. However, CCIX-aware devices410a-410nhave a number of extended capabilities not found in traditional PCIe devices. For example, the device410acan send a TLP with a VDM having the address of the device410n. Once the TLP message is transported through the PCIe switch406to the destination device410n, the device410ncan further interpret the received VDM with enhanced capabilities. One type of enhanced capabilities is that the VDM may contain instructions to transmit the received message to further CCIX-aware devices in the network400(not shown). The PCIe transaction layer210is unaware of this further transmission.

FIG. 5is a flow diagram depicting a first method500for transparently overlaying a logical transport network over an existing physical transport network according to an example. Aspects of the method500may be understood with reference toFIGS. 1-4. The method500begins at block502, where computer system102designates a virtual channel located in a first transaction layer of a network conforming to a first network protocol. At block504, the computer system102assembles a transaction layer packet in a second logical transaction layer of a second network protocol that is also recognizable by the first transaction layer. At least one attribute contained within the transaction layer packet is not recognizable to the first transaction layer, but remaining fields of the packet are recognizable by the first transaction layer. The remaining fields of the packet are unique to the second logical transaction layer.

In an example, the transaction layer packet includes a third party identifier recognizable to the first transaction layer, and a third party message type and a third-party header recognizable to the second logical transaction layer. At block506, the computer system102transfers the transaction layer packet from the second transaction layer to the virtual channel. At block508, the computer system102transmits the transaction layer packet over the first transaction layer using the designated virtual channel over the network. In an example, the first transaction layer transmits the transaction layer packet over a data link layer and a physical layer that conforms to the first network protocol.

In an example, the second transaction layer leverages existing properties of the physical layer, but employs class of service (COS) attributes that are unique to the second logical transaction layer. The COS attributes are at least one of low latency attributes and protocol attributes.

In an example, the second network protocol is cache coherent. In an example, the first network protocol is the PCIe protocol and the second network protocol is the CCIX protocol.

FIG. 6is a flow diagram depicting a second method600for transparently overlaying a logical transport network over an existing physical transport network according to an example. Aspects of the method600may be understood with reference toFIGS. 1-4. The method600begins at block602, where computer system102receives a transaction layer packet located in a first transaction layer of a network conforming to a first network protocol. At block604, the computer system102extracts the transaction layer packet from a virtual channel located in the first transaction layer that is designated for use by a second logical transaction layer that conforms to a second network protocol. At least one attribute contained within the transaction layer packet is not recognizable to the first transaction layer, but remaining fields of the packet are recognizable by the first transaction layer. The remaining fields of the packet are unique to the second logical transaction layer.

In an example, the transaction layer packet includes a third party identifier recognizable to the first transaction layer, and a third party message type and a third-party header recognizable to the second logical transaction layer. At block606, the computer system102assembles a transaction layer packet that conforms to the second network protocol recognizable to the second logical transaction layer. At block608, the computer system102transfers the transaction layer packet from the virtual channel to the second transaction layer.

In an example, the transaction layer packet includes a third party identifier recognizable to the first transaction layer, and a third party message type and a third-party header recognizable to the second logical transaction layer. At block610, the second logical transaction layer of the computer system102extracts and processes the third-party header from the transaction layer packet based on the third party message type.

In an example, the second transaction layer leverages existing properties of the physical layer, but employs class of service (COS) attributes that are unique to the second logical transaction layer. The COS attributes are at least one of low latency attributes and protocol attributes.

In an example, the second network protocol is cache coherent. In an example, the first network protocol is the PCIe protocol and the second network protocol is the CCIX protocol.

FIG. 7is a block diagram depicting a programmable IC1according to an example. The programmable IC1includes programmable logic 3, configuration logic25, and configuration memory26. The programmable IC1can be coupled to external circuits, such as nonvolatile memory27, DRAM28, and other circuits29. The programmable logic 3 includes logic cells30, support circuits31, and programmable interconnect32. The logic cells30include circuits that can be configured to implement general logic functions of a plurality of inputs. The support circuits31include dedicated circuits, such as transceivers, input/output blocks, digital signal processors, memories, and the like. The logic cells and the support circuits31can be interconnected using the programmable interconnect32. Information for programming the logic cells30, for setting parameters of the support circuits31, and for programming the programmable interconnect32is stored in the configuration memory26by the configuration logic25. The configuration logic25can obtain the configuration data from the nonvolatile memory27or any other source (e.g., the DRAM28or from the other circuits29). In some examples, the programmable IC1includes a processing system2. The processing system2can include microprocessor(s), memory, support circuits, IO circuits, and the like.

FIG. 8is a block diagram depicting a System-on-Chip (SoC) implementation of the programmable IC1according to an example. In the example, the programmable IC1includes the processing system2and the programmable logic 3. The processing system2includes various processing units, such as a real-time processing unit (RPU)4, an application processing unit (APU)5, a graphics processing unit (GPU)6, a configuration and security unit (CSU)12, a platform management unit (PMU)122, and the like. The processing system2also includes various support circuits, such as on-chip memory (OCM)14, transceivers7, peripherals8, interconnect16, DMA circuit9, memory controller10, peripherals15, and multiplexed IO (MIO) circuit13. The processing units and the support circuits are interconnected by the interconnect16. The PL3is also coupled to the interconnect16. The transceivers7are coupled to external pins24. The PL3is coupled to external pins23. The memory controller10is coupled to external pins22. The MIO13is coupled to external pins20. The PS2is generally coupled to external pins21. The APU5can include a CPU17, memory18, and support circuits19.

In the example ofFIG. 8, the programmable IC1can be used in the peripheral device122and can function as described above. The PCIe circuit logic130A and the CCIX custom logic130B can be programmed in the PL3and function as described above. In another example, the functionality of the hardware104described above can be implemented using the PS2, rather than through hardware of a computing system. In such case, the software106executes on the PS2and functions as described above.

Referring to the PS2, each of the processing units includes one or more central processing units (CPUs) and associated circuits, such as memories, interrupt controllers, direct memory access (DMA) controllers, memory management units (MMUs), floating point units (FPUs), and the like. The interconnect16includes various switches, busses, communication links, and the like configured to interconnect the processing units, as well as interconnect the other components in the PS2to the processing units.

The OCM14includes one or more RAM modules, which can be distributed throughout the PS2. For example, the OCM14can include battery backed RAM (BBRAM), tightly coupled memory (TCM), and the like. The memory controller10can include a DRAM interface for accessing external DRAM. The peripherals8,15can include one or more components that provide an interface to the PS2. For example, the peripherals15can include a graphics processing unit (GPU), a display interface (e.g., DisplayPort, high-definition multimedia interface (HDMI) port, etc.), universal serial bus (USB) ports, Ethernet ports, universal asynchronous transceiver (UART) ports, serial peripheral interface (SPI) ports, general purpose IO (GPIO) ports, serial advanced technology attachment (SATA) ports, PCIe ports, and the like. The peripherals15can be coupled to the MIO13. The peripherals8can be coupled to the transceivers7. The transceivers7can include serializer/deserializer (SERDES) circuits, MGTs, and the like.

FIG. 9illustrates a field programmable gate array (FPGA) implementation of the programmable IC1that includes a large number of different programmable tiles including transceivers37, configurable logic blocks (“CLBs”)33, random access memory blocks (“BRAMs”)34, input/output blocks (“IOBs”)36, configuration and clocking logic (“CONFIG/CLOCKS”)42, digital signal processing blocks (“DSPs”)35, specialized input/output blocks (“I/O”)41(e.g., configuration ports and clock ports), and other programmable logic39such as digital clock managers, analog-to-digital converters, system monitoring logic, and so forth. The FPGA can also include PCIe interfaces40, analog-to-digital converters (ADC)38, and the like.

In some FPGAs, each programmable tile can include at least one programmable interconnect element (“INT”)43having connections to input and output terminals48of a programmable logic element within the same tile, as shown by examples included at the top ofFIG. 8. Each programmable interconnect element43can also include connections to interconnect segments49of adjacent programmable interconnect element(s) in the same tile or other tile(s). Each programmable interconnect element43can also include connections to interconnect segments50of general routing resources between logic blocks (not shown). The general routing resources can include routing channels between logic blocks (not shown) comprising tracks of interconnect segments (e.g., interconnect segments50) and switch blocks (not shown) for connecting interconnect segments. The interconnect segments of the general routing resources (e.g., interconnect segments50) can span one or more logic blocks. The programmable interconnect elements43taken together with the general routing resources implement a programmable interconnect structure (“programmable interconnect”) for the illustrated FPGA.

In an example implementation, a CLB33can include a configurable logic element (“CLE”)44that can be programmed to implement user logic plus a single programmable interconnect element (“INT”)43. A BRAM34can include a BRAM logic element (“BRL”)45in addition to one or more programmable interconnect elements. Typically, the number of interconnect elements included in a tile depends on the height of the tile. In the pictured example, a BRAM tile has the same height as five CLBs, but other numbers (e.g., four) can also be used. A DSP tile35can include a DSP logic element (“DSPL”)46in addition to an appropriate number of programmable interconnect elements. An10B36can include, for example, two instances of an input/output logic element (“IOL”)47in addition to one instance of the programmable interconnect element43. As will be clear to those of skill in the art, the actual I/O pads connected, for example, to the I/O logic element47typically are not confined to the area of the input/output logic element47.

Some FPGAs utilizing the architecture illustrated inFIG. 8include additional logic blocks that disrupt the regular columnar structure making up a large part of the FPGA. The additional logic blocks can be programmable blocks and/or dedicated logic.