Patent Publication Number: US-9405713-B2

Title: Commonality of memory island interface and structure

Description:
TECHNICAL FIELD 
     The described embodiments relate generally to island-based integrated circuits and related methods. 
     BACKGROUND INFORMATION 
     A network processor is a device that executes programs to handle packet traffic in a data network. Examples include network processors on router line cards and in other network equipment. The Intel IXP2800 is a multi-threaded multiprocessor integrated circuit that is capable of receiving packets, classifying and performing atomic operations on the packets and packet data, and transmitting packets. Within the IXP2800 integrated circuit, microengines operate on packet data stored in a memory. The memory is accessible to the microengines via a DRAM command/push/pull bus. The IXP2800 architecture is therefore flexible and expandable in that more processing power can be added and coupled to the bus so that the added processors will have access to the memory. A family of network processors of this architecture can be made by providing network processor integrated circuits with different amounts of processing power and with different amounts of memory. 
     SUMMARY 
     In a first novel aspect, an island-based network flow processor (IB-NFP) integrated circuit has a configurable mesh data bus. The IB-NFP integrated circuit has functional circuitry including ingress packet classification circuits, processors, memories, and egress scheduling circuitry. The functional circuitry is partitioned into rectangular islands. In addition to the functional circuitry, each island has a part of a mesh data bus. The part of the mesh data bus includes a crossbar switch located centrally in the island, and a set of half links. The mesh data bus may for a sort of overlay over the functional circuitry of the islands. The islands are organized such that half links of adjacent islands are substantially collinear and join together to form links that extend in substantially straight lines between crossbar switches of adjacent islands. 
     In one specific example, there are four substantially identical mesh buses that together form a configurable command/push/pull data bus. A first mesh bus is a command mesh bus of the configurable mesh data bus, a second mesh bus is a pull-id mesh bus of the configurable mesh data bus, a third mesh bus is a data 1  mesh bus of the configurable mesh data bus, and a fourth mesh bus is a data 0  mesh bus of the configurable mesh data bus. Each crossbar switch of such a mesh data bus includes a plurality of Look Up Table (LUT) memories, the contents of which configure the configurable mesh bus. In one example, a plurality of commands can be communicated simultaneously across different parts of the configurable mesh data bus. The configurable mesh data bus can be configured such that multiple different functional circuits in multiple different islands can be performing multiple reads or writes across the configurable mesh data bus at the same time. 
     In a second novel aspect, the islands of an island-based network flow processor (IB-NFP) integrated circuit are organized in a staggered fashion. The islands are rectangular and are organized in rows. The left and rights side edges of the islands of one row are offset laterally and are laterally staggered with respect to the left and right side edges of the islands of the next row. This lateral staggering, back and forth, from row to row, is such that the half links of the islands meet at the port locations and form a mesh bus structure. The left and right edges of islands in a row align with left and right edges of islands two rows down in the row structure. In one specific example, two of the half links extend from the crossbar switch up to port locations on an top edge of the island, another of the half links extends from the crossbar switch to the right to the right side edge of the island, two others of the half links extend down from the crossbar switch to port locations on a bottom edge of the island, another of the half links extends from the central location to the left to the left edge of the island. Two other links extend from the crossbar switch to functional circuitry in the island. In this way, six of the half links radiate outwardly from the central location of the crossbar switch and have a star-shape when the island structure is considered from a top-down perspective. The half links and islands are oriented such that half links of adjacent islands join together to form a mesh bus structure. There are four such mesh bus structures. One for the command bus of a command/push/pull data bus, another for the pull-id bus of the command/push/pull data bus, and two for data buses of the command/push/pull data bus. 
     In a third novel aspect, each link of an island-based network flow processor (IB-NFP) integrated circuit is realized as a distributed credit First-In-First-Out (FIFO) structure. The distributed credit FIFO structure communicates information from the center part of a first island, through an output port of the first island, through an input port of a second island, and to a center part of the second island. The distributed credit FIFO structure includes a first FIFO associated with an output port of a first island, a chain of registers, a second FIFO associated with an input port of a second island, a second chain of registers, and a credit count circuit. A data path extends from the first FIFO in the first island, through the chain of registers, and to the second FIFO in the second island. The data path extends is a substantially straight line from the center part of the first island to the center part of the second island. When a bus transaction value passes through the distributed credit FIFO and then through a crossbar switch of the second island, an arbiter in the crossbar switch returns a taken signal. The taken signal passes back through the second chain of registers to the credit count circuit in the first island. The credit count circuit maintains a credit count value for the distributed credit FIFO. The credit count circuit decreases a credit value when a data value is pushed into the first FIFO, and is increased when the taken signal reaches the credit count circuit. There is one such distributed credit FIFO for the link portion from the first island to the second island, and there is another such distributed credit FIFO for the link portion from the second island to the first island. 
     In a fourth novel aspect, an island-based network flow processor (IB-NFP) integrated circuit comprises six islands: 1) a first island (a MAC island) that converts incoming symbols into a packet; 2) a second island (first NBI island) that analyzes at least part of the packet and generates therefrom first information indicative of whether the packet is a first type of packet or a second type of packet, 3) a third island (ME island) that receives the first information and the header portion from the second island via a configurable mesh data bus, and that generates second information indicating where the header portion is stored and where the payload portion is stored; a fourth island (MU island) that receives a payload portion from the second island via the configurable mesh data bus; a fifth island (second NBI island) that receives second information from the third island via the configurable mesh data bus, that receives the header portion from the third island via the configurable mesh data bus, and that receives the payload portion from the fourth island via the configurable mesh data bus, and that performs egress scheduling; and a sixth island (second MAC island) that receives the header portion and the payload portion from the fifth island and converts the header portion and the payload portion into outgoing symbols. The first, second, third, fourth, fifth, and sixth islands all have the same rectangular size and shape. The first, second, third, fourth, fifth, and sixth islands all have the same configurable mesh data bus structure, configurable mesh control bus structure, and configurable mesh event bus structure. 
     In a fifth novel aspect, memories of an island-based network flow processor (IB-NFP) integrated circuit have a common interface structure. The island-based network flow processor integrated circuit has a first island and a second island. The first island comprises a first memory and first data bus interface circuitry. The second island comprises a processor, a second memory, and second data bus interface circuitry. The second memory is tightly coupled to the processor. The first data bus interface circuitry is substantially identical to the second data bus interface circuitry. The processor in the second island can issue a command for a target memory to do an action. If a target field in the command has a first value then the target memory is the first memory in the first island whereas if the target field in the command has a second value then the target memory is the second memory in the second island. In one example, the command is a command on a configurable mesh data bus. The command format is the same, regardless of whether the target memory is local or remote. If the target memory is remote, then a data bus bridge in the first island adds destination information before putting the command onto the global configurable mesh data bus. 
     In a sixth novel aspect, a first packet is received onto a first island of an island-based network flow processor (IB-NFP) integrated circuit. The header portion of the first packet is communicated to and is stored in a second island (for example, in a tightly coupled memory of a processor island). The payload portion of the first packet is communicated to and is stored in a third island (for example, a memory island). When the first packet is to be transmitted, the header portion is communicated from the second island to a fourth island, and the payload portion is communicated from the third island to the fourth island. The first packet is then output from the integrated circuit. A second packet is received onto the first island. The header portion of the second packet is communicated to and is stored in the second island. The payload portion of the second packet is communicated to and is stored in the third island. The header portion is communicated from the second island to the third island. When the second packet is to be transmitted, both the header portion and the payload portion of the second packet are communicated from the third island to the fourth island, whereafter the second packet is output from the integrated circuit. In one example, the header portion is not moved into the third island unless memory resources in the second island are scarce, but if memory resources in the second island are scarce then the header portion is moved to be stored in the third island along with the payload portion, thereby freeing up memory resources in the second island. 
     In a seventh novel aspect, an island-based network flow processor (IB-NFP) integrated circuit comprises a plurality of islands. Each island comprises a switch and four half links. The islands are coupled together such that the half links and the switches form a configurable mesh control bus. The configurable mesh control bus is configured to have a tree structure such that configuration information passes from the switch of a root island to the switch of each of the other islands, and such that circuitry in each of the plurality of islands is configured by configuration information received via the configurable mesh control bus from the root island. In one example, the configurable control mesh bus portion of each island includes a statically configured switch and multiple half links that radiate from the switch. The static configuration is determined by hardwired tie off connections associated with the island. Configuration information communicated across the tree structure is used to configure a configurable mesh data bus of the island-based network flow processor integrated circuit. 
     In an eighth novel aspect, an island-based network flow processor (IB-NFP) integrated circuit is configured to have a local event ring. The integrated circuit comprises a plurality of rectangular islands disposed in rows. Each rectangular island comprises a Run Time Configurable Switch (RTCS) and a plurality of half links. The rectangular islands are coupled together such that the half links and the RTCSs together form a configurable mesh event bus. The configurable mesh event bus is configured to form the local event ring. The local event ring provides a communication path along which an event packet is communicated to each rectangular island along the local event ring. The local event ring involves event ring circuits and event ring segments. Upon each transition of a clock signal, an event packet moves through the ring from event ring segment to event ring segment. Event information and not packet data travels through the ring. The local event ring functions as a source-release ring in that only the event ring circuit that inserted the event packet onto the ring can delete the event packet from the ring. An event ring circuit on the local event ring can only insert an event packet onto an event ring segment if there is no event packet present on the event ring segment. A bit of the value on an event ring segment indicates whether the remainder of the value on the segment is an event packet or not. 
     In a ninth novel aspect, an island-based network flow processor (IB-NFP) integrated circuit is configured to have a global event chain. The integrated circuit comprises a plurality of rectangular islands disposed in rows. Each rectangular island comprises a Run Time Configurable Switch (RTCS) and a plurality of half links. The rectangular islands are coupled together such that the half links and the RTCSs together form a configurable mesh event bus. The configurable mesh event bus is configured to form a local event ring and the global event chain. The local event ring provides a communication path along which an event packet is communicated to each rectangular island along the local event ring. The event packet can pass from the local event ring and onto the global event chain such that the event packet is then communicated along the global event chain. The global event chain is not a ring, but rather extends in one example to one of islands and terminates in that island. In one example, the local event ring comprises a plurality of event ring circuits and a plurality of event ring segments. One of the event ring segments receives an event packet that is circulating in the local event ring, and determines if the event packet meets a criterion, and only if the packet meets the criterion does the event ring circuit pass the event packet onto the global event chain. The criterion is determined by configuration information stored in the event ring circuit. The configuration information can be changed via the configurable mesh control bus. A configurable event manager at the end of the global event chain receives global event packets from the global event chain, analyzes the global event packets, and collects and logs information about the global event packets. How the event manager analyzes the global event packets is determined by configuration information stored in the event manager. The configuration information can be changed via the configurable mesh control bus. 
     In a tenth novel aspect, an island-based network flow processor (IB-NFP) integrated circuit includes islands organized in rows. A configurable mesh event bus extends through the islands and is configured to form a local event ring. The configurable mesh event bus is configured with configuration information received via a configurable mesh control bus. The local event ring involves event ring circuits and event ring segments. In one example, a packet is received onto a first island. If an amount of a processing resource available to the first island is below a predetermined threshold, then an event packet is communicated from the first island to a second island via the local event ring. In response, the second island sends a first communication from the second island to a third island across a configurable mesh data bus. The third island receives the first communication and in response sends a second communication from the third island to the first island across the configurable mesh data bus. As a result of the first and second communications, the amount of the processing resource available to the first island for handling incoming packet information is increased. In one specific example, the processing resource is an amount of buffer memory that is available to the first island for storing incoming packet information. A buffer list identifies a plurality of memory buffers that are available to the first island for storing incoming packet information. As a result of the first and second communications, an indication of an additional buffer or buffers is added to the buffer list so that the first island has more buffer space available for storing incoming packet information. 
     Further details and embodiments and techniques are described in the detailed description below. This summary does not purport to define the invention. The invention is defined by the claims. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The accompanying drawings, where like numerals indicate like components, illustrate embodiments of the invention. 
         FIG. 1  is a simplified diagram of an MPLS (MultiProtocol Label Switching) router  1 . 
         FIG. 2  is a more detailed top-down diagram of IB-NFP integrated circuit  12 . 
         FIG. 3  shows a way of laying out and interconnecting circuitry in the design of an integrated circuit. 
         FIG. 4  is a diagram of one of the islands  82  of the pattern of  FIG. 3 . 
         FIG. 5  shows a way of laying out and interconnecting circuitry in the design of an integrated circuit in accordance with one novel aspect. 
         FIG. 6  is a diagram of one of the islands  99  of the pattern of  FIG. 5 . 
         FIG. 7  is a diagram that illustrates the circuitry for the configurable mesh data bus that is part of a full island. 
         FIG. 8  is a diagram that illustrates the circuitry of the configurable mesh control bus that is part of a full island. 
         FIG. 9  is a diagram that illustrates the circuitry of the configurable mesh event bus that is part of a full island. 
         FIG. 10  is a more detailed diagram of the structure of one of the mesh buses of the configurable mesh data bus in a part of IB-NFP integrated circuit  12 . 
         FIG. 11  is a diagram that shows the crossbar switch CB 3  of island I 3  ( 61 ). 
         FIG. 12  is a diagram of island I 3  that illustrates an operation of LUT  118  of crossbar switch CB 3  in further detail. 
         FIG. 13  is a diagram that shows the crossbar switch CB 3  in further detail. 
         FIG. 14  is a diagram that illustrates the terminology employed. 
         FIG. 15  is a diagram that shows how link portion  136  between islands I 1  and I 2  is realized as a distributed credit FIFO. 
         FIG. 16  is a top-down diagram that illustrates in schematic fashion how island I 3  includes a separate set of half link portions and crossbar switches for the four different mesh buses of the configurable mesh CPP data bus. 
         FIG. 17  is a flowchart of a write operation method  1000  that might occur across the configurable mesh CPP data bus. 
         FIG. 18  is a diagram of the formal of a bus transaction value that passes over the configurable mesh data bus. 
         FIG. 19  is a table describing the payload of a bus transaction value in the situation in which the bus transaction value is a command. 
         FIG. 20  is a table describing the payload of a bus transaction value in the situation in which the bus transaction value is a pull-id. 
         FIG. 21  is a table describing the payload of a bus transaction value in the situation in which the bus transaction value is data pull or push. 
         FIG. 22  is a table describing the payload of a bus transaction value in the situation in which the bus transaction value is data pull. 
         FIG. 23  is a table describing the payload of a bus transaction value in the situation in which the bus transaction value is data push. 
         FIG. 24  is a flowchart of a read operation method  2000  that might occur across the configurable mesh CPP data bus. 
         FIG. 25  is a diagram the configurable mesh control bus structure of the IB-NFP integrated circuit  12 . 
         FIG. 26  is a top-down diagram that illustrates the configurable mesh data bus and how the islands do not abut each other in one embodiment. 
         FIG. 27  is a top-down diagram that illustrates the configurable mesh control bus and how the islands do not abut each other in one embodiment. 
         FIG. 28  is a top-down diagram that illustrates the configurable mesh event bus and how the islands do not abut each other in one embodiment. 
         FIG. 29  is a top-down diagram of the configurable mesh control bus structure of the IB-NFP integrated circuit  12  configured to form a tree structure. 
         FIG. 30  is a simplified top-down diagram of the control bus portion of the ARM island  51  and parts of ME island  53  and ME island  54 . 
         FIG. 31  is a diagram that shows how configuration information flows into and through ME island  53 . 
         FIG. 32  is a simplified diagram of the central switch of the control bus structure of an island. 
         FIG. 33  is a diagram of the configurable mesh event bus structure of the IB-NFP integrated circuit  12 . 
         FIG. 34  is a simplified perspective view of the local event ring within NBI island  72 . 
         FIG. 35  is a functional diagram of the RTCS  206  shown in  FIG. 34 . 
         FIG. 36  is a more detailed block diagram of event ring circuit  209  shown in  FIG. 34 . 
         FIG. 37  is a more detailed block diagram of event ring circuit  210  shown in  FIG. 34 . 
         FIG. 38A  is a diagram of an event packet bit sequence. 
         FIG. 38B  is a signal table that identifies and explains the various parts of an event packet. 
         FIG. 39  is a diagram of the configurable mesh event bus configured to form two local event rings and a global event chain. 
         FIG. 40  is a simplified system level illustration that shows how a local event ring is a source-release ring. 
         FIG. 41  is a simplified diagram of the first  245  and second  246  local event rings and single global event chain  247  shown in  FIG. 39 . 
         FIG. 42  is a more detailed block diagram of the event manager  251  within island  51 . 
         FIG. 43  is a schematic diagram that illustrates an operational example of IB-NFP integrated circuit  12  within the MPLS router  1  of  FIG. 1 . 
         FIG. 44  is a diagram of the four SerDes I/O blocks  19 - 22  and the ingress MAC island  71  of IB-NFP integrated circuit  12 . 
         FIG. 45  is a diagram that illustrates how a packet is communicated as a sequence of minipackets across connections  312 . 
         FIG. 46  is a diagram of ingress NBI island  72 . 
         FIG. 47  is a table that sets forth the parts of preclassification results  321 . 
         FIG. 48  is a table that sets forth the parts of an ingress packet descriptor. 
         FIG. 49  is a table that sets forth the parts of an egress packet descriptor. 
         FIG. 50  is a diagram of the microengine (ME) island  66 . 
         FIG. 51  is a bit sequence map of a bus transaction value used to communicate packet data from the ingress NBI island  72  to the ME island  66  across the CPP data bus. 
         FIG. 52  is a diagram of MU half island  68  and associated SRAM block  78 . 
         FIG. 53  is a diagram of egress NBI island  63 . 
         FIG. 54  is a diagram of egress MAC island  64  and SerDes blocks  25 - 28 . 
         FIG. 55  is a diagram that illustrates a packet flow in the operational example when local memory resources in the CTM  333  of the ME island  66  are determined not to be scarce (for example, the processing resource is determined not to be below a predetermined threshold). 
         FIG. 56  is a diagram that illustrates a packet flow in the operational example when local memory resources in the CTM  333  of the ME island  66  are determined to be scarce (for example, the processing resource is determined to be below a predetermined threshold). 
         FIG. 57  is a diagram that illustrates the use of a local event ring and a configurable mesh data bus for flow control in the IB-NFP integrated circuit  12 . 
     
    
    
     DETAILED DESCRIPTION 
     Reference will now be made in detail to background examples and some embodiments of the invention, examples of which are illustrated in the accompanying drawings. In the description and claims below, relational terms such as “horizontal”, “vertical”, “lateral”, “top”, “upper”, “bottom”, “lower”, “right”, “left”, “over” and “under” may be used to describe relative orientations between different parts of a structure being described, and it is to be understood that the overall structure being described can actually be oriented in any way in three-dimensional space. 
       FIG. 1  is a simplified diagram of an MPLS (MultiProtocol Label Switching) router  1 . Router  1  includes a backplane  2 , a management card  3 , and line cards  4 - 6 . Each of the line cards can receive 100 Gbps (gigabits per second) packet traffic from another network via a fiber optic cable  7  and also can transmit 100 Gbps packet traffic to another network via another fiber optic cable  8 . In addition, each line card can receive 100 Gbps packet traffic from the switch fabric  9  of the backplane and can also transmit 100 Gbps packet traffic to the switch fabric. Line cards  4 - 6  are of identical construction. In this example, flows of packets are received into line card  4  from a network via the fiber optic cable  7  or from the switch fabric  9 . Certain functions then need to be performed on the line card including looking up MPLS labels, determining destinations for incoming flows of packets, and scheduling the transmitting of flows of packets. Packets of the flows pass from the line card  4  and out either to the network via optical cable  8  or to the switch fabric  9 . 
     Line card  4  includes a first optical transceiver  10 , a first PHY integrated circuit  11 , an Island-Based Network Flow Processor (IB-NFP) integrated circuit  12 , a configuration Programmable Read Only Memory (PROM)  13 , an external memory such as Dynamic Random Access Memory (DRAM)  40 - 41 , a second PHY integrated circuit  15 , and a second optical transceiver  16 . Packet data received from the network via optical cable  7  is converted into electrical signals by optical transceiver  10 . PHY integrated circuit  11  receives the packet data in electrical form from optical transceiver  10  via connections  17  and forwards the packet data to the IB-NFP integrated circuit  12  via SerDes connections  18 . In one example, the flows of packets into the IB-NFP integrated circuit from optical cable  7  is 100 Gbps traffic. A set of four SerDes circuits  19 - 22  within the IB-NFP integrated circuit  12  receives the packet data in serialized form from SerDes connections  18 , deserializes the packet data, and outputs packet data in deserialized form to digital circuitry within IB-NFP integrated circuit  12 . 
     Similarly, IB-NFP integrated circuit  12  may output 100 Gbps packet traffic to optical cable  8 . The set of four SerDes circuits  19 - 22  within the IB-NFP integrated circuit  12  receives the packet data in deserialized form from digital circuitry within integrated circuit  12 . The four SerDes circuits  19 - 22  output the packet data in serialized form onto SerDes connections  23 . PHY  15  receives the serialized form packet data from SerDes connections  23  and supplies the packet data via connections  24  to optical transceiver  16 . Optical transceiver  16  converts the packet data into optical form and drives the optical signals through optical cable  8 . Accordingly, the same set of four duplex SerDes circuits  19 - 22  within the IB-NFP integrated circuit  12  communicates packet data both into and out of the IB-NFP integrated circuit  12 . 
     IB-NFP integrated circuit  12  can also output packet data to switch fabric  9 . Another set of four duplex SerDes circuits  25 - 28  within IB-NFP integrated circuit  12  receives the packet data in deserialized form, and serializes the packet data, and supplies the packet data in serialized form to switch fabric  9  via SerDes connections  29 . Packet data from switch fabric  9  in serialized form can pass from the switch fabric via SerDes connections  30  into the IB-NFP integrated circuit  12  and to the set of four SerDes circuits  25 - 28 . SerDes circuits  25 - 28  convert the packet data from serialized form into deserialized form for subsequent processing by digital circuitry within the IB-NFP integrated circuit  12 . 
     Management card  3  includes a CPU (Central Processing Unit)  31 . CPU  31  handles router management functions including the configuring of the IB-NFP integrated circuits on the various line cards  4 - 6 . CPU  31  communicates with the IB-NFP integrated circuits via dedicated PCIE connections. CPU  31  includes a PCIE SerDes circuit  32 . IB-NFP integrated circuit  12  also includes a PCIE SerDes  33 . The configuration information passes from CPU  31  to IB-NFP integrated circuit  12  via SerDes circuit  32 , SerDes connections  34  on the backplane, and the PCIE SerDes circuit  33  within the IB-NFP integrated circuit  12 . 
     External configuration PROM (Programmable Read Only Memory) integrated circuit  13  stores other types of configuration information such as information that configures various lookup tables on the IB-NFP integrated circuit. This configuration information  35  is loaded into the IB-NFP integrated circuit  12  upon power up. As is explained in further detail below, IB-NFP integrated circuit  12  can store various types of information including buffered packet data in external DRAM integrated circuits  40 - 41 . 
       FIG. 2  is a more detailed top-down diagram of IB-NFP integrated circuit  12 . IB-NFP integrated circuit  12  includes many I/O (input/output) terminals (not shown). Each of these terminals couples to an associated terminal of the integrated circuit package (not shown) that houses the IB-NFP integrated circuit. The integrated circuit terminals may be flip-chip microbumps and are not illustrated. Alternatively, the integrated circuit terminals may be wire bond pads. 
     SerDes circuits  19 - 22  are the first set of four SerDes circuits that are used to communicate with the external network via the optical cables  7  and  8 . SerDes circuits  25 - 28  are the second set of four SerDes circuits that are used to communicate with the switch fabric  9 . Each of these SerDes circuits is duplex in that it has a SerDes connection for receiving information and it also has a SerDes connection for transmitting information. Each of these SerDes circuits can communicate packet data in both directions simultaneously at a sustained rate of 25 Gbps. IB-NFP integrated circuit  12  accesses external memory integrated circuits  36 - 41  via corresponding 32-bit DDR physical interfaces  42 - 47 , respectively. IB-NFP integrated circuit  12  also has several general purpose input/output (GPIO) interfaces. One of these GPIO interfaces  48  is used to access external PROM  13 . 
     In addition to the area of the input/output circuits outlined above, the IB-NFP integrated circuit  12  also includes two additional areas. The first additional area is a tiling area  49  of islands  50 - 74 . Each of the islands is either of a full rectangular shape, or is half the size of the full rectangular shape. For example, the island  55  labeled “PCIE (1)” is a full island. The island  60  below it labeled “ME CLUSTER ( 5 )” is a half island. The functional circuits in the various islands of this tiling area  49  are interconnected by: 1) a configurable mesh CPP data bus, 2) a configurable mesh control bus, and 3) a configurable mesh event bus. Each such mesh bus extends over the two-dimensional space of islands with a regular grid or “mesh” pattern. These mesh buses are described in further detail below. 
     In addition to tiling area  49 , there is a second additional area of larger sized blocks  75 - 79 . The functional circuitry of each of these blocks is not laid out to consist of islands and half-islands in the way that the circuitry of tiling area  49  is laid out. The mesh bus structures do not extend into or over any of these larger blocks. The mesh bus structures do not extend outside of tiling area  49 . The functional circuitry of a larger sized block outside the tiling area  49  may connect by direct dedicated connections to an interface island within tiling area  49  and through the interface island achieve connectivity to the mesh buses and other islands. 
       FIG. 3  shows a way of laying out and interconnecting circuitry in the design of an integrated circuit. The functional circuitry is partitioned into blocks  80 - 95  referred to as tiles or islands. To provide interconnectivity between any selected two of the islands, a configurable mesh bus of horizontal links, vertical links, links down to functional circuitry in each island, and crossbar switches is provided.  FIG. 4  is a diagram of one of the islands  82 . Its crossbar switch  96  is centrally located. A first link is provided from the functional circuit at port location P 5  to the crossbar switch  96 . Information can be communicated from the functional circuitry of the island to the crossbar switch via this link. A second link is provided from the crossbar switch to the functional circuitry at port location P 6 . Information can be communicated from the crossbar switch to the functional circuitry via this second link. A half link extends vertically from the crossbar switch to a first port location P 1  in the center of the top edge of the island. A second half link extends horizontally from the crossbar switch to a second port location P 2  at the right edge of the island. A third half link extends vertically from the crossbar switch to a third port location P 3  in the center of the bottom edge of the island. A fourth half link extends horizontally from the crossbar switch to a fourth port location P 4  at the right edge of the island. Information coming into the island on one of the half links can be switched by the crossbar switch so that the information then passes out of the island on a selected one of the other half links of the island or so that the information is supplied to the functional circuitry of the island. As illustrated in  FIG. 3 , the islands are laid out in regular rows and columns so that the half links align with one another and form links between crossbar switches. This tiling pattern may be referred to as a Manhattan pattern. If a connection is to be provided from functional circuitry in the lower left island  93  to functional circuitry in island  82  in the top row, then the connection would pass through a minimum of four intervening islands before reaching island  82 . The path of such a connection is indicated by dashed line  97 . 
       FIG. 5  shows a way of laying out and interconnecting circuitry in the design of an integrated circuit in accordance with one novel aspect. The functional circuitry is partitioned into rectangular islands  98 - 113  as illustrated.  FIG. 6  is a diagram of one of the islands  99 . A crossbar switch  114  is centrally located. Four half links are not, however, oriented to form a cross-shape as in  FIG. 4 , but rather six half links are oriented to form a star-shape as illustrated as in  FIG. 6 . A first link is provided from the functional circuitry at port location P 7  to the crossbar switch  114 . A signal can be supplied from the functional circuitry to the crossbar switch via this link. A second link is provided from the crossbar switch  114  to the functional circuitry at port location P 8 . A signal can be supplied from the crossbar switch to the functional circuitry via this second link. A first half link extends diagonally from the crossbar switch  114  up and to the left to a first port location P 1 . A second half link extends diagonally from the crossbar switch  114  up and to the right to a second portion location P 2 . A third half link extends horizontally from the crossbar switch  114  to the right to a third port location P 3 . A fourth half link extends diagonally from the crossbar switch  114  down and to the right to a fourth port location P 4 . A fifth half link extends diagonally fro the crossbar switch  114  down and to the left to a fifth port location P 5 . A sixth half link extends horizontally from the crossbar switch  114  to the left to a sixth port location P 6 . The rectangular islands of  FIG. 5  are laid out in rows, but the side edges of the islands of one row are staggered laterally with respect to the corresponding side edges of the islands of the next row. The islands are therefore oriented in a staggered brick structure as illustrated. If a connection is to be provided from functional circuitry in the lower left island  110  to functional circuitry in the island  100  in the top row, then the configurable mesh bus structure can be configured such that a connection is established between these two islands that only passes through three intervening islands before reaching island  100 . The path of such a connection is indicated by dashed line  115 . Connectivity is improved as compared to the Manhattan pattern of  FIG. 3 . The comparative diagrams of  FIGS. 3 and 5  are only for a small number of islands, but the comparative advantage of the staggered pattern of  FIG. 5  as compared to the Manhattan pattern of  FIG. 3  becomes more pronounced as the layout patterning technique is extended to include more and more islands. 
     The islands of the tiling area  49  of the IB-NFP integrated circuit  12  are disposed in the staggered pattern explained in connection with  FIG. 5 . Each island includes: 1) an amount of functional circuitry, 2) circuitry for a configurable mesh data bus, 3) circuitry for a configurable mesh control bus, and 4) circuitry for a configurable mesh event bus. 
       FIG. 7  is a diagram that illustrates the circuitry for the configurable mesh data bus that is part of a full island. The configurable mesh data bus structure actually includes four mesh bus structures, each of which includes a crossbar switch that is disposed in the center of the island as illustrated, and each of which includes six half links that extend to port locations P 1 -P 6  as illustrated, and each of which also includes two links that extend between the crossbar switch and the functional circuitry of the island at port locations P 7  and P 8 . These four mesh bus structures are referred to as the command mesh bus, the pull-id mesh bus, and data 0  mesh bus, and the data 1  mesh bus. The half links of these mesh buses do not necessarily all extend along the exact same path when the island is considered from the top-down perspective, but rather the half links of these mesh buses extend roughly along the same line. The mesh buses terminate at the edges of the island such that if another identical tile were laid out to be adjacent, then the half links of the corresponding mesh buses of the two islands would align and couple to one another in an end-to-end collinear fashion to form the staggered pattern illustrated in  FIG. 5 . Similarly, the circuitry of the crossbar switches of the four mesh buses are not all disposed in the same exact location when the island is considered from the top-down perspective, but rather the crossbar switches are located roughly in the center of the island. The four mesh buses of the configurable mesh data bus together are a Command/Push/Pull (CPP) bus as is explained in further detail below. 
       FIG. 8  is a diagram that illustrates the circuitry of the configurable mesh control bus that is part of a full island. Unlike the structure of the mesh buses of the configurable mesh data bus of  FIG. 7  where each crossbar switch is coupled to the functional circuitry of the island by two links, the crossbar switch of the configurable mesh control bus of  FIG. 8  is coupled to the functional circuitry of the island by one link. Also, the configurable mesh control bus circuitry does not involve horizontally extending half links. The crossbar switch is configured by hardwired tie off connections as described in further detail below. This configuration is fixed at the time of chip manufacture and is not changeable. Accordingly, the crossbar switch is sometimes referred to here more specifically as a “statically configured switch”, rather than as a crossbar switch. 
       FIG. 9  is a diagram that illustrates the circuitry of the configurable mesh event bus that is part of a full island. The structure of the event bus is similar to the structure of the control bus, except that the crossbar switches of the configurable mesh event bus are not hardwired into one configuration. The crossbar switch is referred to as a “run time configured switch”. 
       FIG. 10  is a more detailed diagram of the structure of one of the mesh buses of the configurable mesh data bus in a part of IB-NFP integrated circuit  12 . A first crossbar CB 1  is located centrally within a first rectangular island I 1 . The first rectangular island I 1  is PCIE (1) island  55  of  FIG. 2 . A second crossbar CB 2  that is located centrally within a second rectangular island I 2 . The second rectangular island I 2  is ME cluster ( 3 ) island  56  of  FIG. 2 . A third crossbar CB 3  that is located centrally within a third rectangular island I 3 . The third rectangular island I 3  is Crypto Bulk ( 2 ) island  61  of  FIG. 2 . The first and the second islands I 1  and I 2  are disposed in a first row that extends along a horizontal dimension. The third island I 3  is disposed in a second row that extends from the first row along a vertical dimension (i.e., the second row is below the first row when considered from the perspective of  FIG. 10 ). A first link L 1  comprises a first half link HL 11  and a second half link HL 12 . HL 11  and HL 12  are collinear. First link L 1  extends in a substantially straight line between CB 1  and CB 2 . A second link L 2  comprises a third half link HL 21  and a fourth half link HL 22 . Half links HL 21  and HL 22  are collinear. Second link L 2  extends in a substantially straight line between CB 1  and CB 3 . A third link L 3  comprises a fifth half link HL 31  and a sixth half link HL 32 . Half links HL 31  and HL 32  are collinear. Third link L 3  extends in a substantially straight line between CB 2  and CB 3 . Links L 1 , L 2  and L 3  form an isosceles triangle as illustrated. Reference numerals P 1 , P 2 , P 3 , P 4 , P 5  and P 6  identify port locations. Island I 1  has an upper left corner C 1 , an upper right corner C 2 , a lower right corner C 3 , and a lower left corner C 4 . Port location P 1  is located on the top edge of the island about one quarter of the way from the upper left corner to the upper right corner. Port location P 2  is located on the top edge of the island about three quarters of the way from the upper left corner to the upper right corner. Port location P 3  is located in the middle of the right edge of the island. Port location P 4  is located on the bottom edge of the island about one quarter of the way from the lower right corner to the lower left corner. Port location P 5  is located on the bottom edge of the island about three quarters of the way from the lower right corner to the lower left corner. Port location P 6  is located in the middle of the left edge of the island. 
     Island  16  is a half island. Island  16  includes a centrally located crossbar switch CB 6 . Two links extend between crossbar switch CB 6  and the functional circuitry of the half island. There are, however, only three half links that extend to three corresponding port locations P 1 , P 2  and P 3 . Half link HL 42 , for example, is oriented to extend in collinear fashion to half link HL 41  of island I 1 . Half link HL 42  joins at its port location P 1  with half link HL 41  at its port location P 5 . The other connections between the half links of the various islands are not described here. There are four instances of the general bus structure illustrated in  FIG. 10 , one for the command mesh of the configurable mesh data bus, one for the pull-id mesh of the configurable mesh data bus, one for the data 0  mesh of the configurable mesh data bus, and one for the data 1  mesh of the configurable mesh data bus. 
       FIG. 11  is a diagram that shows the crossbar switch CB 3  of island I 3  ( 61 ). Each half link is bidirectional in that it can supply information from an input port location to the crossbar switch CB 3 , and in that it can receive information from the crossbar switch CB 3  and supply that information to an output port location. For example, half link HL 51  includes a half link portion  116  that can communicate information from input port location P 6  to CB 3 , and half link HL 51  also includes a half link portion  117  that can communicate information from CB 3  to output port location P 6 . There is a Look Up Table (LUT) within CB 3  associated with each half link portion that supplies information to the crossbar switch. In the case of  FIG. 11 , LUT  118  is the LUT associated with incoming half link portion  116 . There is also an arbiter within CB 3  associated with each half link portion that communicates information from the crossbar switch to an output port location. In the case of  FIG. 11 , arbiter  183  is the arbiter associated with half link portion  117  that communicates information from CB 3  to output port location  6 . 
       FIG. 12  is a diagram of island I 3  that illustrates the operation of LUT  118  in further detail. Crossbar switch CB 3  receives information from half link portion  116  and switches the information so that it is output to another half link portion. In the case of  FIG. 12 , the information is switched so that it is output to the half link portion for output port location  4 . Incoming half link portion  116  includes a chain of registers  119  and an input FIFO (First In First Out)  120 . The multiple bits of information being communicated across the half link portion  116  is called a bus transaction value. The bus transaction value includes a final destination value portion, a valid bit, and a payload portion. The final destination value indicates the destination island to which the bus transaction value will be communicated through configurable mesh CPP data bus. Input FIFO  120  supplies a bus transaction value to CB 3 . The final destination value portion of the bus transaction value is supplied to LUT  118 . LUT  118  was previously configured so that when it is presented with a particular final destination value  121 , it outputs a FIFO lookup value  122  that causes CB 3  to route the bus transaction value through CB 3  to an appropriate output port. In the example of FIG.  12 , the final destination value is 13. From the value  13  the LUT  118  outputs a lookup value of 4. By configuring all the LUTs appropriately, a bus transaction value having a particular final destination value will be routed through the configurable mesh data bus via one and only one path to the island indicated by the final destination value. In the example of  FIG. 12 , LUT  118  outputs a lookup value of 4. CB 3  uses this lookup value  4  to route the bus transaction value onto the output link portion that will communicate the bus transaction value to output port  4 . 
       FIG. 13  is a diagram that shows the crossbar switch CB 3  in further detail. In the left column are illustrated the input FIFOs. There is one such input FIFO at the end of each of the eight half link portions that can supply bus transaction values from an input port to CB 3 . For example, the half link portion that extends from input port  6  ends in input FIFO  120 . LUT  118  is associated with this incoming half link portion. The final destination value portion of the bus transaction value output by FIFO  120  is supplied to LUT  118 . The bus transaction value output by FIFO  120  is also communicated horizontally into the crossbar switch via a set of conductors  123 . Similarly, the half link portion that extends from input port  3  ends in FIFO  124 . The final destination value of the bus transaction value output by FIFO  124  is supplied to LUT  125 . The bus transaction value output by FIFO  124  is also communicated horizontally into the crossbar switch via a set of conductors  126 . As illustrated, there is one set of horizontally extending conductors for each of the eight input ports. 
     In addition to the sets of horizontally extending conductors, there is also one set of vertically extending conductors for each of the eight half link portions onto which a bus transaction value can be routed. For example, the set of vertical conductors  127  can communicate a bus transaction value vertically downward to the output FIFO  128  of the half link portion that extends to output port  4 . There is a set of switches at the intersection of each set of horizontally extending conductors and each set of vertically extending conductors. Each such set of switches is generally just referred to as a switch. Each such switch can be turned on by an arbiter to couple the set of horizontally extending conductors to the set of vertically extending conductors. Alternatively, each such switch can be turned off by an arbiter so that the switch does not couple the set of horizontally extending conductors to the set of vertically extending conductors. There is an arbiter for each of the eight output ports as illustrated. The arbiter associated with an output port controls the eight switches along the set of vertically extending conductors that extend to the output FIFO for that output port. In the example illustrated, arbiter  131  can turn on switch  129  to couple the set of horizontally extending conductors  123  to the set of vertically extending conductors  127 . Also, arbiter  131  can turn on switch  130  to couple the set of horizontally extending conductors  126  to the set of vertically extending conductors  127 . There are eight such switches, one for each input port, and each of these switches is controlled by arbiter  131 . 
     A 1.0 GHz global clock clocks the data bus structure, including the arbiters. During a given period of the 1.0 GHz global clock, at most one bus transaction value is switched onto a set of vertically extending conductor for output to a particular output port. A contention situation may arise in which, during a given period of the global clock, the bus transaction values of multiple sets of horizontally extending conductors are to be switched onto the same output port. For example, two bus transaction values on the sets of horizontally extending conductors  123  and  126  may have final destination values. When looked up in their associated LUTs, both final destinations result in the both bus transactions values being destined to go out to output port  4 . If, however, both switches  129  and  130  were made to be conductive during the same global clock period, then a collision would occur and the data of the bus transaction values would be corrupted. The arbiter  131  prevents this problem. If there is space available in the output FIFO  128  associated with arbiter  131 , then arbiter  131  causes one bus transaction value to be pushed into output FIFO  128 . The arbiter receives all LUT lookup values from the LUTs of the input ports. If the lookup values indicate that only one bus transaction value is to be routed onto the output half link portion for output port  4 , then the arbiter  131  controls the appropriate switch to turn on and to couple the bus transaction value onto the vertically extending conductors  127 . If, however, the lookup values from the LUTs indicate that two or more bus transaction values are to be routed to output port  4 , then the arbiter  131  turns on the switches so that a bus transaction value from only one of the input ports will be switched onto the set of vertically extending conductors and to the output port. In the example of  FIG. 13 , the final destination values of the two bus transaction values received from input ports  3  and  6  both result in their respective LUTs outputting lookup values indicating that their respective bus transaction values are to be routed to output port  4 . Arbiter  131  receives the lookup values from all eight LUTs. In the notation of  FIG. 13 , the OP=4 notation means “output port  4 ”. Each LUT actually supplies a single bit to each of the eight arbiters. If the bit to an arbiter is asserted, this indicates to the arbiter that the LUT for the corresponding input port is signaling that a bus transaction value should be routed to the output port associated with the arbiter. In the example of  FIG. 13 , LUTs  118  and  125  both assert their respective single bit OP values for arbiter  131 . In response, arbiter  131  controls switch  129  to be on during a first global clock period while switch  130  is off. After the bus transaction value from input port location  6  has passed through the crossbar switch and out to output port location  4  during the first global clock period, then the arbiter  131  switches switch  129  to be off and switch  130  to be on during the second global clock period. The bus transaction value from input port location  3  then passes through the crossbar switch and to output port location  4  during the second global clock cycle. 
       FIG. 14  is a diagram that illustrates the terminology employed. Link L 1  is the bidirectional link that extends horizontally between crossbar switch CB 1  of island I 1  and crossbar switch CB 2  of island I 2  of  FIG. 10 . This link L 1  extends through port location  3  of island I 1 , and through port location  6  of island I 2 . The bottom portion of  FIG. 14  shows that this link involves two half links. Half link HL 11  is the half of the link in island I 1 . Half link HL 12  is the half of the link in island I 2 . Each half link is bidirectional. Each half link includes two half link portions. For example, half link portion  132  extends from CB 1  to port location  3  of island I 1 . Half link portion  133  extends from port location  3  of island  11  to CB 1 . Half link portion  134  extends from port location  6  of island I 2  to CB 2 . Half link portion  135  extends from CB 2  to port location  6  of island I 2 . Half link portions  132  and  134  are collinear and join to form link portion  136 . Half link portions  133  and  135  are collinear and join to form link portion  137 . The half link portions that carry data away from a crossbar switch start with an output FIFO. FIFO  138  is the output FIFO for half link portion  132 . FIFO  140  is the output FIFO for half link portion  135 . The half link portions that carry data into a crossbar switch end with an input FIFO. FIFO  139  is the input FIFO for half link portion  133 . FIFO  159  is the input FIFO for half link portion  134 . 
       FIG. 15  is a diagram that shows how link portion  136  between islands I 1  and I 2  is realized as a distributed credit FIFO. Link portion  136  includes the first FIFO  138  that is associated with output port  3  of first island I 1 , a first chain of registers  142 - 145 , second FIFO  159  that is associated with input port  6  of second island I 2 , a second chain of registers  146 - 150 , and a credit count circuit  151  disposed in the first island I 1 . As explained above, a bus transaction value can be communicated from the crossbar switch CB 1  of island I 1 , through output FIFO  138 , through chain of registers  142 - 145 , and to input FIFO  159 , and to crossbar switch CB 2  of island I 2 . The chain of registers is clocked by the 1.0 GHz global clock signal  152 . This global clock clocks all the registers of all the links of the entire configurable mesh data bus. The physical distance between the crossbar switches CB 1  and CB 2  is smaller than the distance a signal can propagate on a conductor across the integrated circuit in one period of the global clock. Multiple bus transaction values may flow in a pipelined fashion through this link portion. 
     Arbiter  154  is the arbiter of CB 2  for output port  4 . The other arbiters for the other output ports are not illustrated. Arbiter  154  receives a FIFO full signal  141  from the output FIFO  153  for output port  4 . If the FIFO full signal  141  indicates that space is available in output FIFO  153 , then arbiter  154  can cause a selected bus transaction value from one of the input ports of the crossbar switch to be supplied through the crossbar switch and into output FIFO  153 . For each input FIFO of the crossbar switch, its associated LUT receives the final destination value of the bus transaction value being output by the associated input FIFO. A single conductor extends from each LUT to arbiter  154  for output port  4 . If the lookup value output by the LUT indicates that the bus transaction value should be output onto output port  4 , then the LUT asserts the single-bit signal on this conductor for output port  4 . Arbiter  154  receives one such single-bit signal from each LUT. All the conductors for communicating these single-bit signals are not shown in  FIG. 15  due to space limitations. Signal  177  is the single-bit signal supplied by LUT  178  to arbiter  154 . If one of these OP signals indicates that a bus transaction value is to be output from output port  4 , then arbiter  154  turns on the appropriate switch to couple the bus transaction value from its horizontally extending conductors onto the vertically extending conductors and to output FIFO  153 . A signal conductor extends from arbiter  154  to each of the eight switches that might be turned on to route a bus transaction value to output FIFO  153 . Arbiter  154  can turn on a selected one of these eight switches by asserting the ON signal on the appropriate one of these eight signal conductors. In the case of  FIG. 15 , conductor  179  is the conductor that extends from arbiter  154  to the switch that can switch input port  6  to output port  4 . Reference numeral  180  identifies the set of horizontally extending conductors for input port  6 . Reference numeral  181  identifies the set of vertically extending conductors for output port  4 . Reference numeral  182  identifies the switch that can couple the set of horizontally extending conductors  180  to the set of vertically extending conductors  181 . 
     When the arbiter  154  causes a bus transaction value from input port  6  to be routed through the crossbar switch and to be pushed into output FIFO  153 , the arbiter  154  issues a pop signal  163  back to input FIFO  159  from which the bus transaction value came. In addition, the pop signal is supplied to the first register  146  of the second chain of registers. The second chain of registers is really a chain of flip-flops that forms a shift register. This shift register shifts the pop signal (also called a “taken signal”) back to the left from the second island I 2  to the credit count circuit  151 . When the taken signal  163  arrives at the credit count circuit  151 , a credit count value maintained by the credit count circuit is incremented to indicate that link portion  136  can hold one more bus transaction value. When the crossbar switch CB 1  of the first island I 1  pushes a bus transaction value into FIFO  138  of link portion  136 , the push FIFO signal  164  is also supplied to credit count circuit  151 . This push signal  164  causes credit count circuit  151  to decrement the credit value. A lower credit value indicates that link portion  136  can hold one less bus transaction value. In the illustrated example, only if the credit count is greater than zero can a data value be pushed into input FIFO  138 . 
     It is possible that the input FIFOs leading to the crossbar switch present multiple bus transaction values that should be switched to the same output FIFO  153 . If such a case of contention, arbiter  154  selects one of the bus transaction values to be switched to the output FIFO during the global clock period as explained above. The arbiter only turns on the appropriate switch to direct this one bus transaction value to the output FIFO  153 . The arbiter pops the input FIFO that supplied the bus transaction value at the end of the global clock period. Then in the next global clock period, if there is space available in the output FIFO as indicated by the FIFO full signal, then the arbiter turns on the appropriate switch to direct the next bus transaction value into the output FIFO. Once again, at the end of the transfer, the arbiter  154  asserts the POP signal to the particular input FIFO that supplied the bus transaction value that was transferred. There are eight conductors extending from arbiter  154  to the input FIFOs so that arbiter  154  can supply a POP signal to the appropriate input FIFO. Only three of these conductors are shown in  FIG. 15  due to space limitations. In this way, the arbiter  154  only supplies one bus transaction value from one input FIFO onto the output FIFO  153  during a given global clock period. 
       FIG. 16  is a top-down diagram that illustrates in schematic fashion how island I 3  includes a separate set of link portions and crossbar switches for the four different mesh buses of the configurable mesh CPP data bus. There is one set of half links and a crossbar switch for the command mesh of the configurable mesh data bus. There is one set of half links and a crossbar switch for the pull-id mesh of the configurable mesh data bus. There is one set of half links and a crossbar switch for the data 0  mesh of the configurable mesh data bus. There is one set of half links and a crossbar switch for the data 1  mesh of the configurable mesh data bus. The termination points of these half links at the edges of the island are such that when two identical islands are made to be adjacent, that the command mesh half links join to another, that the pull-id mesh half links join to one another, that the data 0  mesh half links join to one another, and that the data 1  mesh half links join to one another. The four mesh structures of the configurable mesh data bus implement a command/push/pull (CPP) data bus. 
       FIG. 17  is a flowchart of a write operation method  1000  that might occur across the configurable mesh CPP data bus. In a first step (step  1001 ), certain functional circuitry in one of the islands uses its data bus interface to output a bus transaction value onto the configurable mesh CPP data bus. This functional circuitry is referred to as the “master” of the write operation. The format of the bus transaction value is as set forth in  FIG. 18 . A bus transaction value  1006  includes a metadata portion  1007  and a payload portion  1008  as shown. The metadata portion  1007  includes a final destination value  1009  and a valid bit  1010 . 
     The bus transaction value in this case is a write command to write data into functional circuitry in another island. The functional circuitry that receives the bus transaction value and the data to be written is referred to as the “target” of the write operation. The write command is said to be “posted” by the master circuit onto the command mesh. As indicated in  FIG. 18 , the write command includes a metadata portion and a payload portion. The metadata portion includes the 6-bit final destination value. This final destination value identifies an island by number, where the island identified is the final destination of the bus transaction value. The final destination value is used by the various crossbar switches of the command mesh to route the bus transaction value (i.e., the command) from the master circuit to the appropriate target circuit. All bus transaction values on the data bus that originate from the same island that have the same final destination value will traverse through the configurable mesh data bus along the same one path all the way to the indicated final destination island. 
     A final destination island may have more than one potential target circuit. The 4-bit target field of payload portion indicates which one of these targets in the destination island it is that is the target of the command. The 5-bit action field of the payload portion indicates that the command is a write. The 14-bit data reference field is a reference usable by the master circuit to determine where in the master the data is to be found. The address field indicates an address in the target where the data is to be written. The length field indicates the amount of data. 
     In a next step (step  1002 ) in the method of  FIG. 17 , the target circuit receives the write command from the command mesh and examines the payload portion of the write command. From the action field the target circuit determines that it is to perform a write action. To carry out this action, the target circuit writes (i.e., posts) a bus transaction value (step  1003 ) called a pull-id onto the pull-id mesh. The pull-id is also of the format indicated in  FIG. 18 . The payload portion of the pull-id is of the format set forth in  FIG. 20 . The final destination field of the metadata portion of the pull-id indicates the island where the master circuit is located. The target port field identifies which sub-circuit target it is within the target&#39;s island that is the target circuit of the command. The pull-id is communicated through the pull-id mesh back to the master circuit. 
     The master circuit receives the pull-id from the pull-id mesh and uses the content of the data reference field of the pull-id to find the data. In the overall write operation, the master circuit knows the data it is trying to write into the target circuit. 
     The data reference value that is returned with the pull-id is used by the master circuit as a flag to match the returning pull-id with the write operation the master circuit had previously initiated. 
     The master circuit responds by sending (step  1004 ) the identified data to the target across one of the data meshes data 0  or data 1  as a “pull” data bus transaction value. The term “pull” means that the data of the operation passes from the master to the target. The term “push” means that the data of the operation passes from the target to the master. The format of the “pull” data bus transaction value sent in this sending of data is also as indicated in  FIG. 18 . The format of the payload portion in the case of the payload being pull data is as set forth in  FIG. 22 . The first bit of the payload portion is asserted. This bit being a digital high indicates that the transaction is a data pull as opposed to a data push. The target circuit then receives (step  1005 ) the data pull bus transaction value across the data 1  or data 0  mesh. The target circuit writes the content of the data field (the data field of  FIG. 22 ) of the pull data payload portion into target memory at the appropriate location indicated by the address field of the original write command. 
       FIG. 24  is a flowchart of a read operation method  2000  that might occur across the configurable mesh CPP data bus. In a first step (step  2001 ), a master circuit in one of the islands uses its data bus interface to output (to “post”) a bus transaction value onto the command mesh bus of the configurable mesh CPP data bus. In this case, the bus transaction value is a read command to read data from a target circuit. The format of the read command is as set forth in  FIGS. 18 and 19 . The read command includes a metadata portion and a payload portion. The metadata portion includes the 6-bit final destination value that indicates the island where the target is located. The action field of the payload portion of the read command indicates that the command is a read. The 14-bit data reference field is usable by the master circuit as a flag to associated returned data with the original read operation the master circuit previously initiated. The address field in the payload portion indicates an address in the target where the data is to be obtained. The length field indicates the amount of data. 
     The target receives the read command (step  2002 ) and examines the payload portion of the command. From the action field of the command payload portion the target circuit determines that it is to perform a read action. To carry out this action, the target circuit uses the address field and the length field to obtain the data requested. The target then pushes (step  2003 ) the obtained data back to the master circuit across data mesh data 1  or data 0 . To push the data, the target circuit outputs a push bus transaction value onto the data 1  or data 0  mesh.  FIG. 23  sets forth the format of the payload portion of this push bus transaction value. The first bit of the payload portion indicates that the bus transaction value is for a data push, as opposed to a data pull. The master circuit receives the bus transaction value of the data push (step  2004 ) from the data mesh bus. The master circuit then uses the data reference field of the push bus transaction value to associate the incoming data with the original read command, and from the original read command determines where the pushed data (data in the date field of the push bus transaction value) should be written into the master circuit. The master circuit then writes the content of the data field of the data field into the master&#39;s memory at the appropriate location. 
     As explained above, the contents of the LUTs of the crossbar switches of the configurable mesh data bus determine how bus transaction values will be routed on their ways to their final destinations. The contents of the LUTs are set so that the configurable mesh data bus contains no loops. For a bus transaction value having a metadata portion indicating a given final destination, and given that this bus transaction value is injected onto a given link, there is only one path through the configurable mesh data bus that the bus transaction value can take to get to its final destination. Having only one path avoids deadlocks. Also, for a given application to which the IB-NFP integrated circuit is to be put, the anticipated traffic from each source of traffic on the chip to each destination of traffic on the chip is determined. For the anticipated traffic over each such source-to-destination path, the routing through the data mesh is configured so that the anticipated traffic will be spread and balanced across the mesh data bus so that no link of the mesh data bus will be overburdened. The configuration information for the LUTs of the crossbar switches is loaded into the various LUTs using the configurable control bus (CB). The LUTs of the configurable mesh data bus are loaded in this way initially at startup of the IB-NFP integrated circuit before any communication across the configurable mesh data bus occurs. 
       FIG. 25  is a diagram of the configurable mesh control bus structure of the IB-NFP integrated circuit  12 . As set forth above in connection with  FIG. 8 , the configurable mesh control bus structure of each full island involves a centrally located switch, and four duplex half links. A first half link extends from the switch up and to the left to a first port location P 1  of the island. A second half link extends from the switch up and to the right to a second port location P 2  of the island. A third half link extends from the switch down and to the right to a third port location P 3  of the island. A fourth half link extends from the switch down and to the left to a fourth port location P 4  of the island. These port locations are marked on island  66 . The control bus does not have to communicate high speed data in this example. Information passing across the control bus from one switch of one island to another switch of an adjacent island does not pass through many registers. In other examples, links of the control bus may have pipelining registers in the routes of links between switches, where the pipelining registers are clocked by the 1.0 GHz global clock. 
     Although the islands of IB-NFP integrated circuit  12  are illustrated as abutting one another, in some examples there is actually a small interstitial spacing between adjacent islands.  FIG. 26  is a top-down diagram of a part of the IB-NFP integrated circuit.  FIG. 26  shows the interstitial spacings between islands and shows how the configurable mesh data bus bridges those spacings.  FIG. 27  is a top-down diagram of the same part of the IB-NFP integrated circuit.  FIG. 27  shows the interstitial spacings between islands and shows how the configurable control bus bridges those spacings.  FIG. 28  is a top-down diagram of same part of the IB-NFP integrated circuit.  FIG. 28  shows the interstitial spacings between islands and shows how the configurable mesh event bus bridges those spacings. 
       FIG. 29  is a top-down diagram of the configured configurable mesh control bus structure of the IB-NFP integrated circuit  12 . The configurable mesh control bus structure is configured to form a tree structure. The source or root of the tree structure is the functional circuitry in the ARM island  51 . The arrows on the various half links of the control bus in the illustration of  FIG. 29  indicate how configuration information flows through the tree structure. Configuration information flows from the ARM island  51  and down through intermediary islands  53 - 58 ,  61 - 66 ,  72  and  73  of the branch structure and to the end leaf islands  50 ,  52 ,  59 ,  60 ,  67 ,  68 ,  70 ,  71 ,  74  of the tree structure, but configuration information cannot flow in the reverse direction toward the ARM island  51 . Configuration information for loading the LUTs of the crossbar switches of the configurable mesh data bus is stored in external configuration PROM  13 . After power up of the IB-NFP integrated circuit  12 , functional circuitry in the ARM island  51  reads this configuration information out of external configuration PROM  13  via GPIO interface block  48 . The configuration information for a given LUT is then communicated down through the whole tree. The appropriate crossbar switch of the island of which the LUT is a part will match a selection field portion of the configuration information with its hard-wired island number. Each island is provided with a hard-wire island number. The island uses the hard-wire island number to examine information of the control bus and to determine which configuration has an associated selection field portion destined for the island number. When such a match occurs, the identified configuration information is loaded into the LUTs of the island. The configuration of the configurable control bus into the tree structure is statically determined by hardwired tie off connections. The configuration of the control bus is therefore set at the time of chip manufacture. 
       FIG. 30  is a simplified top-down diagram of the control bus portion of ARM island  51  and parts of ME island  53  and ME island  54 . Each half link has an associated tie off input conductor that can be either tied to a digital logic high voltage or to a digital logic low voltage. If the tie off conductor is tied to a digital high, then the associated input half link portion is configured to receive control information into the island through that port location. If the tie off conductor is tied to a digital low, then the associated input half link portion is configured to output control information from the central switch. In the example of  FIG. 30 , the tie offs for the input ports P 1 , P 2 , P 3  and P 4  for the control bus of ARM island  51  are all tied low. The source of control information is therefore the functional circuitry  165  of the island  51 . Regardless of the source, the control information is output to each of the output ports of the island by the control bus half links. The control information is also supplied to the functional circuitry via the output port that couples to the functional circuitry  165 . The control information originates from configuration PROM  13 , flows through GPIO interface block  48 , and through functional circuitry  165  of ARM island  51 , and through switch  166 , and out to the four output ports P 1 , P 2 , P 3  and P 4  (and also back to the functional circuitry  165 ). Ports P 1  and P 2  are not coupled to another island so that fact that the configuration information flows out to these output ports is of no moment. The configuration information that flows out to output port P 4 , however, flows into ME island  53  through input port P 2  of island  53 , through half link portion  168 , and to switch  167 . The tie off associated with input port P 2  of ME island  53  is tied high. This configures port P 2  of ME island  53  to be the source of configuration information for the ME island. At most one tie off of one port of an island can be tied high. If no tie off is tied high, then the source for the configuration information is the functional circuit of the island. 
     Likewise, the configuration information flows out of port P 3  of ARM island  51  and into input port P 1  of ME island  54 , through half link portion  170 , and to central switch  169 . The tie off associated with input port P 1  of ME island  54  is tied high. This configures port P 1  of ME island  54  to be the source of configuration information for the ME island  54 . 
       FIG. 31  is a diagram that shows how configuration information flows into and through ME island  53 . As described above in connection with  FIG. 30 , the configuration information flows into port P 2  of ME island  53  and to switch  167 . The tie off for port P 2  of island  53  is tied high and as a result this input port is configured to be the source of the configuration information for the island. The configuration information is output to all ports and to the functional circuitry of the island. In this way, the control bus structure of each island is hardwire-configured with tie offs such that the control bus structure of the overall IB-NFP integrated circuit  12  is configured to form the tree structure as illustrated in  FIG. 29 . 
       FIG. 32  is a simplified diagram of the central switch of the control bus structure of an island. The labels P 1 _IN, P 2 _IN, P 3 _IN and P 4 _IN indicate the input half link portions of the island at input ports P 1 , P 2 , P 3  and P 4  that are possible sources of configuration information for the island. Configuration information present at any one of these input ports, or on the output IOUT of the functional circuitry of the island, is multiplexed by multiplexer  176  onto all the output half link portions of the island: P 1 _OUT, P 2 _OUT, P 3 _OUT, P 4 _OUT, and onto the input IIN of the functional circuitry of the island. Which one of the input ports (or the functional circuitry) it is that will be the source of the configuration information is determined by the tie off signal values SEL_P 1 , SEL_P 2 , SEL_P 3  and SEL_P 4 . SEL P 1  is the tie off value at input port P 1 . The tie off signal SEL P 2  is the tie off value at input port P 2 . The tie off signal SEL P 3  is the tie off value at input port P 3 . The tie off signal SEL P 4  is the tie off value at input port P 4 . Each full island in the IB-NFP integrated circuit  12  has four such tie offs. If none of the tie offs is tied high, then the source of configuration information is the IOUT from the functional circuitry of the island. In one example, power and ground conductors extending in upper layer metal extend through the interstitial space between adjacent islands, and each tie off tab conductor of the island is coupled up to one of these conductors by an appropriate vertically extending conductive via. The tie off tab conductors are illustrated in  FIG. 31  as rectangular tab structures  172 - 175 . The round circle seen in each of the tab conductors represents the vertically-extending conductive via that couples the tab conductor to either a power conductor or a ground conductor in the metal layers above. 
       FIG. 33  is a configurable mesh event bus structure of the IB-NFP integrated circuit  12 . From the top-down view shown in  FIG. 33  four port locations are visible on the top overlay layer; however, each island also has a half link portion extending down into each island coupling the Real Time Configurable Switch (RTCS) to functional circuitry within the island. For example, ingress NBI island  72  is an island with four half link portions coupling four port locations (P 1 -P 4 ) on the top overlay layer to the RTCS; however, ingress NBI island  72  also has a fifth half link portion coupling the RTCS to functional circuitry within the ingress NBI island  72 . As set forth above in connection with  FIG. 9 , the configurable mesh event bus structure of each full island involves a centrally located RTCS, and five duplex half links. A first half link extends from the RTCS up and to the left to a first port location P 1  of the island. A second half link extends from the RTCS up and to the right to a second port location P 2  of the island. A third half link extends from the RTCS down and to the right to a third port location P 3  of the island. A fourth half link extends from the RTCS down and to the left to a fourth port location P 4  of the island. A fifth half link extends from the RTCS down to the functional circuitry within the island. These port locations are marked on ingress NBI island  72  in  FIG. 33 . 
       FIG. 34  is a perspective view of the local event ring within NBI island  72 . As discussed above, the ingress NBI island  72  includes four half links ( 201 - 204 ) and a centrally located Real Time Configurable Switch (RTCS)  206 . RTCS  206  may also be referred to as a “switch”. Half link  201  includes a half link portion P 1 _IN that can communicate information from port location P 1  to RTCS  206  and a half link portion P 1 _OUT that can communicate information from switch  206  to port location P 1 . Half link  202  includes a half link portion P 2 _IN that can communicate information from port location P 2  to RTCS  206  and a half link portion P 2 _OUT that can communicate information from RTCS  206  to port location P 2 . Half link  203  includes a half link portion P 3 _IN that can communicate information from port location P 3  to RTCS  206  and a half link portion P 3 _OUT that can communicate information from RTCS  206  to port location P 3 . Half link  204  includes a half link portion P 4 _IN that can communicate information from port location P 4  to RTCS  206  and a half link portion P 4 _OUT that can communicate information from RTCS  206  to port location P 4 . Link  205  includes a link portion P 5 _IN that can communicate information from functional circuitry  207  within the ingress NBI island  72  to RTCS  206  and a link portion P 5 _OUT that can communicate information from RTCS  206  to functional circuitry  207  within the NBI island  72 . A RTCS control logic  208  is coupled to RTCS  206 . The RTCS control logic  208  includes instructions associated with each half link portion coupled to RTCS  206 . The RTCS control logic  208  controls which inputs to RTCS  206  are coupled to which outputs of RTCS  206 . The RTCS control logic  208  is programmed via the configurable mesh control bus (CB). The configurable mesh control bus (CB) is coupled to the RTCS control logic  208 . In one embodiment, the RTCS control logic  208  in each island is programmed once via the configurable mesh control bus upon powering on the IB-NFP integrated circuit  12 . In another, embodiment, the RTCS control logic  208  in each island is programmed not only upon powering on the IB-NFP integrated circuit  12 , but also at another time during subsequent operation of the IB-NFP integrated circuit  12 . 
     Functional circuitry  207  includes a first event ring circuit  209  and a second event ring circuit  210 . Link portion P 5 _OUT couples to an input terminal of event ring circuit  209  via a first event ring segment. An output terminal of event ring circuit  209  couples to an input terminal of event ring circuit  210  via a second event ring segment. An output of event ring circuit  210  couples to link portion P 5 _IN via a third event ring segment. Event ring circuit  209  is described in  FIG. 36 . Event ring circuit  210  is described in  FIG. 37 . The event packet  200  is further described in  FIGS. 38A  and  38 B. Vacancy indicator bit  239 , source of event  240 , and type of event  241  form a sort of header of the event packet, and event data  242  is a payload of the event packet. 
     In operation, an event packet  200  is communicated to RTCS  206  via an input half link. RTCS  206  then communicates the event packet  200  via another half link portion according to the instructions stored in RTCS control logic  208 . For example, event packet  200  from the ingress ME  66  may be communicated via half link portion P 2 _IN to RTCS  206  as illustrated in  FIG. 34 . RTCS  206  then may communicate the event packet  200  via another half link portion to any one of the four port locations or to the functional circuitry  207  within the NBI island  72 . The RTCS  206  is configured, via RTCS control logic  208 , to couple half link portion P 2 _IN to link portion P 5 _OUT. The RTCS  206  is also configured to couple link portion P 5 _IN to half link portion P 1 _OUT. The event packet  200  is communicated from port location P 2  to RTCS  206  via half link portion P 2 _IN. The event packet  200  is communicated from RTCS  206  to functional circuitry  207  via link portion P 5 _OUT. The event packet  200  is then communicated from link portion P 5 _OUT to the input terminal of event ring circuit  209  via an event ring segment. The event packet  200  is then communicated from the output terminal of event ring circuit  209  to the input terminal of event ring circuit  210  via an event ring segment. The event packet  200  is then communicated from the output terminal of event ring circuit  210  to link portion P 5 _IN via an event ring segment. The event packet  200  is then communicated to RTCS  206  via link portion P 5 _IN. The event packet  200  is then communicated to port location P 1  from RTCS  206  via half link portion P 1 _OUT. The event packet  200  is then communicated through port location P 1  to Interlaken LA( 1 ) island  65 . 
     Alternatively, RTCS  206 , via RTCS control logic  208 , may be configured such that half link portion P 1 _IN is coupled directly to half link portion P 2 _OUT. Therefore, event packet  200  from ME island  66  would not be communicated to functional circuitry  207 , but rather would be directly communicated to port location P 1  and subsequently Interlaken LA( 1 ) island  65 . This alternative may be used to connect a local event ring through one island without making the functional circuitry of the said one island part of the local event ring. 
     A global clock signal is not shown in  FIG. 34 ; however, a global clock signal is coupled to RTCS  206 , event ring circuit  209 , and event ring circuit  210 . The global clock signal may be a variety of different frequencies. In one embodiment, the global clock is a one gigahertz signal and is coupled to all local event ring circuitry. Depending on the frequency of global clock signal intermediate registers, or “slots”, may be required to address propagation delay in sending event packets between various event ring circuits. In an embodiment, a slot may be coupled in series at multiple locations along the local event ring. For example, one or more slots may be coupled in series along a half link portion or between local event ring circuits. 
       FIG. 35  is a functional diagram of the RTCS  206  shown in  FIG. 34 . RTCS  206  includes five multiplexers ( 211 - 215 ) and RTCS control logic  208 . RTCS  206  has five inputs P 1 _IN, P 2 _IN, P 3 _IN, P 4 _IN and P 5 _IN. RTCS  206  has five outputs P 1 _OUT, P 2 _OUT, P 3 _OUT, P 4 _OUT, and P 5 _OUT. Each output has an independent multiplexer ( 211 - 215 ), which couples one of the five inputs to a single output terminal of each multiplexer. Each multiplexer is controlled by an individual set of select control lines (SEL 1 -SEL 15 ). In one embodiment, each multiplexer has three select control lines. Three controls lines results in the ability to control up to eight possible multiplexer inputs. RTCS control logic  208  has a clock input (CLK), an enable input (EN), a configurable mesh control bus input, and fifteen select control outputs (SEL  1 - 15 ). Configuration information is communicated via the configurable mesh control bus to RTCS control logic  208 . RTCS control logic  208  sources a signal on select control outputs (SEL  1 - 15 ) based upon the configuration information. Three select control lines (SEL  1 - 3 ) are coupled to multiplexer  211 . Three select control lines (SEL  4 - 6 ) are coupled to multiplexer  212 . Three select control lines (SEL  7 - 9 ) are coupled to multiplexer  213 . Three select control lines (SEL  10 - 12 ) are coupled to multiplexer  214 . Three select control lines (SEL  13 - 15 ) are coupled to multiplexer  215 . The three signals communicated to a multiplexer and control which, if any, of the multiplexer inputs are coupled to the multiplexer output. In one embodiment, the signal on a select control line may be a logic low or high signal, meaning that the voltage applied to the control line be above or below a threshold voltage. 
     In operation, configuration information is communicated to RTCS control logic  208  via the configurable mesh control bus. According to the configuration information received, RTCS control logic  208  sources an independent signal on each select control line (SEL 1 - 15 ). The signals are coupled to respective multiplexers as described above. In response to the signal received via select control lines, each multiplexer couples one of its inputs to the output terminal of the multiplexer or leaves the output terminal in a floating state. 
       FIG. 36  is an expanded diagram of event ring circuit  209  shown in  FIG. 34 . Event ring circuit  209  includes register  217 , multiplexer  218 , vacancy logic  219 , event packet generating circuit  220 , First In First Out circuit (FIFO)  221 , source number checker  222 , and multiplexer  223 . A previous local event ring segment is coupled to the data input terminal of register  217 . A global clock signal  254  is coupled to the clock input terminal of register  217 . The output terminal of register  217  couples to a first input terminal of multiplexer  223 , to an event packet input terminal (EP) of source number checker  222  and to an input terminal of vacancy logic  219 . A second input terminal of multiplexer  223  is coupled to a logic high signal, such as a voltage source. An output terminal of multiplexer  223  is coupled to a first input terminal of multiplexer  218 . An output terminal of vacancy logic  219  is coupled to a selector input terminal of multiplexer  218 . The output terminal of vacancy logic  219  is also coupled to a present input terminal (PR) of FIFO  221 . A global clock signal  254  is coupled to the clock input terminal of source number checker  222 . The configurable mesh control bus is coupled to a control bus input terminal (CB) of source number checker  222 . A control output terminal (CN) of source number checker  222  is coupled to a selector input terminal of multiplexer  223 . A source number output terminal (SN) of source number checker  222  is coupled to a source number (SN) input terminal of event packet generating circuit  220 . A global clock signal  254  is coupled to the clock input terminal of event packet generating circuit  220 . An event packet output terminal (EP) of event packet generating circuit  220  is coupled to a data input terminal of FIFO circuit  221 . A push output terminal (P) of event packet generating circuit  220  is coupled to a push input terminal (P) of FIFO  221 . A full (F) output control terminal of FIFO  221  is coupled to a full (F) input terminal of event circuit generating circuit  220 . A data output terminal of FIFO circuit  221  is coupled to a second input terminal of multiplexer  218 . An output terminal of multiplexer  218  is coupled to the next segment of the event ring. 
     In operation, event packet generating logic  220  creates event packets to be inserted into a local event ring. Event packets created by event packet generating circuit  220  represent events which occurred within the island where the event ring circuit is located. In the embodiment illustrated in  FIG. 36 , event packet generating circuit  220  creates event packets representing events which occurred in NBI island  72 . The event packet  225  created by event packet generating logic  220  is communicated to FIFIO  221 . FIFO  221  acts as a buffer to store event packets until the event packets can be inserted into the local event ring. Event packets can only be inserted into the local event ring when the previous event ring segment is carrying no event packet. The event packet generating circuit  220  communicates a signal from the push output terminal (P) of the event packet generating circuit  220  to the push input terminal (P) of FIFO  221  when the event packet generating circuit  220  is communicating an event packet from the event packet output terminal (EP) to the data input terminal of FIFO  221 . FIFO  221  has a finite memory size. In the event that FIFO  221  memory is full, no additional event packets from event packet generating circuit  220  may be stored in FIFO  221 . A signal indicating that the FIFO  221  is full is communicated from the full output terminal (F) of FIFO  221  to the full input terminal (F) of event packet generating circuit  220 . FIFO  221  always outputs a value; however, if no event packets are stored in FIFO  221  an output value with a first bit set to one (1) is communicated to the second terminal of multiplexer  218 . 
     A first value is clocked into register  217  from the previous event ring segment. The first value is then communicated from the output of register  217  to the input terminal of vacancy logic  219 , the event packet input terminal (EP) of source number checker  222 , and the first input terminal of multiplexer  223 . 
     Vacancy logic  219  receives the first bit of the first value communicated from register  217 . The first bit of the first value indicates if an event packet it present on the segment. The first bit is one (1) if the event ring segment is carrying no event packet. The first bit of the first value is zero (0) if an event packet is present on the event ring segment. 
     Source number checker  222  receives an assigned source number from the configurable mesh control bus via the control bus input terminal (CB). The assigned source number is a number that is unique to event ring circuit  209 . All event packets inserted into a local event ring by event ring circuit  209  will have a “source of event” field that indicates the source number assigned to event ring circuit  209 . In one embodiment, the “source of event” filed is eight bits wide. An event packet bit sequence map and table is provided in  FIGS. 38A and 38B . Source number checker  222  communicates the assigned source number from the source number output terminal (SN) of source number checker  222  to the source number input terminal (SN) of event packet generating circuit  220 . Event packet generating circuit  220  uses the source number information received from source number checker  222  to fill in the “source of event” field in new event packets. 
     Source number checker  222  also receives eight bits of the first value via the event packet input terminal (EP). The eight bits make up the “source of event” field and indicate which event ring circuit in the local event ring inserted the value. The source number checker  222  compares the received eight bits with the eight bit source number assigned to event ring circuit  209  via the configurable mesh control bus. When the eight bits representing the assigned source number are the same as the received eight bits of the first value, the source number checker  222  communicates a signal from control output terminal (CN) to the selector input terminal of multiplexer  223 , such that multiplexer  223  couples the second input terminal of multiplexer  223  to the output terminal of multiplexer  223 . The result is that source number checker  222  couples the logic high signal to the output terminal of multiplexer  223  when the eight source of event bits match, and that the source number checker  222  couples the output of register  217  to the output of multiplexer  223  when the eight source of event bits do not match. Coupling the logic high signal to the output terminal of multiplexer  223  has the effect of passing no event packet to the first input terminal of multiplexer  218 . Source number checker  222  also receives a global clock signal  254  via the clock input terminal. The source number checker  222  transmits and receives communications during transitions of the global clock signal. 
     Vacancy logic  219  receives the first bit of the first value communicated from register  217 . The first bit of the first value is one (1) if no event packet is present. The first bit of the first value is zero (0) if an event packet is present. 
     When an event packet  224  is present, the first bit of the first value is a zero (0) and vacancy logic  219  will communicate a control signal to the selector input terminal of multiplexer  218  so that multiplexer  218  will couple the first input terminal of multiplexer  218  to the output terminal of multiplexer  218 . The vacancy logic  219  will also communicate a control signal to the present input terminal (PR) of FIFO  221  so that FIFO  221  will not communicate an event packet to the second input terminal of multiplexer  218 . Therefore, when the first bit of the first signal is zero (0) either the event packet  224  or the logic high signal will be coupled to the next event ring segment. Whether the event packet  224  or the logic high signal is coupled to the next event ring segment depends on the signal communicated from the control output terminal (CN) of source number checker  222 . If the first bit of the first value is zero (0) and the received source number does not match the assigned source number, then event packet  224  will be coupled to the next event ring segment. If the first bit of the first value is zero (0) and the received source number does match the assigned source number, then the logic high signal will be coupled to the next event ring segment. Coupling the logic high signal to the next event ring segment has the effect of deleting event packet  224  from the local event ring and inserting no event packet in the local event ring. 
     When, on the other hand, an event packet is not present, the first bit of the first value is one (1) and vacancy logic  219  will communicate a control signal to the selector input terminal of multiplexer  218  so that multiplexer  218  will couple the second input terminal of multiplexer  218  to the output terminal of multiplexer  218 . The vacancy logic  219  will also communicate a control signal to the present input terminal (PR) of FIFO  221  so that FIFO  221  will communicate event packet  226  to the second input terminal of multiplexer  218 . Therefore, when the first bit of the first value is a one (1) the next event packet  226  stored in FIFO  221  will be coupled to the next event ring segment. 
     The configurable mesh control bus is coupled to the control bus input terminal (CB) of source number checker  222 . Source number checker  222  may be assigned a source number via the configurable mesh control bus. In one embodiment, source number checker  222  is assigned a source number once upon powering on the IB-NFP integrated circuit  12 . In another embodiment, source number checker  222  is assigned a first source number upon powering on the IB-NFP integrated circuit  12  and a second source number, replacing the first source number, during subsequent operation of the IB-NFP integrated circuit  12 . 
       FIG. 37  is an expanded diagram of event ring circuit  210  shown in  FIG. 34 . Event ring circuit  210  includes a global event filter  227  in addition to the items included in event ring circuit  209 . Event ring circuit  210  includes register  270 , multiplexer  271 , vacancy logic  272 , event packet generating circuit  273 , First In First Out circuit (FIFO)  274 , source number checker  275 , multiplexer  276 , and global event filter  227 . Global event filter  227  includes register  228 , source number checker  229 , vacancy logic  230 , multiplexer  231 , multiplexer  232 , global event filter logic  233 , and FIFO  234 . 
     A previous local event ring segment is coupled to the data input terminal of register  270 . A global clock signal  254  is coupled to the clock input terminal of register  270 . The output terminal of register  270  couples to a first input terminal of multiplexer  276 , to an event packet input terminal (EP) of source number checker  275  and to an input terminal of vacancy logic  272 . A second input terminal of multiplexer  276  is coupled to a logic high signal, such as a voltage source. An output terminal of multiplexer  276  is coupled to a first input terminal of multiplexer  271 . An output terminal of vacancy logic  272  couples to a selector input terminal of multiplexer  271 . The output terminal of vacancy logic  272  also couples to a present input terminal (PR) of FIFO  274 . A global clock signal  254  is coupled to the clock input terminal of source number checker  275 . The configurable mesh control bus is coupled to a control bus input terminal (CB) of source number checker  275 . A control output terminal (CN) of source number checker  275  is coupled to a selector input terminal of multiplexer  276 . A source number output terminal (SN) of source number checker  275  is coupled to a source number (SN) input terminal of event packet generating circuit  273 . A global clock signal  254  is coupled to the clock input terminal of event packet generating circuit  273 . An event packet output terminal (EP) of event packet generating circuit  273  is coupled to a data input terminal of FIFO circuit  274 . A push output terminal (P) of event packet generating circuit  273  is coupled to a push input terminal (P) of FIFO  274 . A full (F) output control terminal of FIFO  274  is coupled to a full (F) input terminal of event circuit generating circuit  273 . A data output terminal of FIFO circuit  274  is coupled to a second input terminal of multiplexer  271 . An output terminal of multiplexer  271  is coupled to the next segment of the event ring. 
     Within global filter  227 , a previous global event chain segment is coupled to the data input terminal of register  228 . The output terminal of register  228  is coupled to a first input terminal of multiplexer  231 , an event packet input terminal (EP) of source number checker  229 , and an input terminal of vacancy logic  230 . A global clock signal  254  is coupled to the clock input terminal of register  228 . A second input terminal of multiplexer  231  is coupled to a logic high signal, such as a voltage source. An output terminal of multiplexer  231  is coupled to a second input of multiplexer  232 . An output terminal of vacancy logic  230  couples to a selector input terminal of multiplexer  232 . A global clock signal  254  is coupled to the clock input of source number checker  229 . The configurable mesh control bus is coupled to the control bus input terminal (CB) of source number checker  229 . A control output terminal (CN) of source number checker  229  is coupled to a selector input terminal of multiplexer  231 . A source number output terminal (SN) of source number checker  229  is coupled to a source number input terminal (SN) of global event filter  233 . A global clock signal  254  is coupled to the clock input terminal of global event filter logic  233 . 
     The output terminal of register  217  is coupled to the event packet input terminal (EP) of global event filter logic  233 . Configurable mesh control bus is coupled to a control bus input terminal (CB) of global event filter  223 . A global event output terminal (GE) of global event filter logic  223  is coupled to a data input terminal of FIFO circuit  234 . A push output terminal (P) of global event filter logic  233  is coupled to a push input terminal (P) of FIFO  221 . A full output terminal (F) of FIFO  234  is coupled to a full input terminal (F) of global event filter logic  233 . A data output terminal of FIFO circuit  234  is coupled to a first input terminal of multiplexer  232 . An output terminal of multiplexer  232  is coupled to the next segment of the global event chain. 
     In operation with respect to local event ring, event packet generating logic  273  creates event packets to be inserted into a local event ring. Event packets created by event packet generating circuit  273  represent events which occurred within the island where the event ring circuit is located. In the embodiment illustrated in  FIG. 36 , event packet generating circuit  273  creates event packets representing events which occurred in NBI island  72 . The event packet  278  created by event packet generating logic  273  is communicated to FIFIO  274 . FIFO  274  acts as a buffer to store event packets until the event packets can be inserted into the local event ring. Event packets can only be inserted into the local event ring when the previous event ring segment is carrying no event packet. The event packet generating circuit  273  communicates a signal from the push output terminal (P) of the event packet generating circuit  273  to the push input terminal (P) of FIFO  274  when the event packet generating circuit  273  is communicating an event packet from the event packet output terminal (EP) to the data input terminal of FIFO  274 . FIFO  274  has a finite memory size. In the event that FIFO  274  memory is full, no additional event packets from event packet generating circuit  273  may be stored in FIFO  274 . A signal indicating that the FIFO  274  is full is communicated from the full output terminal (F) of FIFO  274  to the full input terminal (F) of event packet generating circuit  273 . FIFO  274  always outputs a value; however, if no event packets are stored in FIFO  274  an output value with a first bit set to one (1) is communicated to the second terminal of multiplexer  271 . A first value is clocked into register  270  from the previous event ring segment during a transitioning of global clock signal  254 . The first value is then communicated from the output terminal of register  270  to the input terminal of vacancy logic  272 , the event packet input terminal (EP) of source number checker  275 , and the first input terminal of multiplexer  276 . 
     Source number checker  275  receives an assigned source number from the configurable mesh control bus via the control bus input terminal (CB). The assigned source number is a number that is unique to event ring circuit  210 . All event packets inserted into a local event ring by event ring circuit  210  will have a “source of event” field that indicates the source number assigned to event ring circuit  210 . In one embodiment, the “source of event” filed is eight bits wide. An event packet bit sequence map and table is provided in  FIGS. 38A and 38B . Source number checker  275  communicates the assigned source number from the source number output terminal (SN) of source number checker  275  to the source number input terminal (SN) of event packet generating circuit  273 . Event packet generating circuit  273  uses the source number information received from source number checker  275  to fill in the “source of event” field in new event packets. 
     Source number checker  275  also receives eight bits of the first value via the event packet input terminal (EP). The eight bits make up the “source of event” field and indicate which event ring circuit in the local event ring inserted the value. The source number checker  275  compares the received eight bits with the eight bit source number assigned to event ring circuit  210  via the configurable mesh control bus. When the eight bits representing the assigned source number are the same as the received eight bits of the first value, the source number checker  275  communicates a signal from control output terminal (CN) to the selector input terminal of multiplexer  276 , such that multiplexer  276  couples the second input terminal of multiplexer  276  to the output terminal of multiplexer  276 . The result is that source number checker  275  couples the logic high signal to the output terminal of multiplexer  276  when the eight source of event bits match, and that the source number checker  275  couples the output of register  270  to the output of multiplexer  276  when the eight source of event bits do not match. Coupling the logic high signal to the output terminal of multiplexer  276  has the effect of passing no event packet to the first input terminal of multiplexer  271 . Source number checker  275  also receives a global clock signal  254  via the clock input terminal. The source number checker  275  transmits and receives communications during transitions of the global clock signal. 
     Vacancy logic  272  receives the first bit of the first value communicated from register  270 . The first bit of the first value is one (1) if no event packet is present. The first bit of the first value is zero (0) if an event packet is present. 
     When an event packet  277  is present, the first bit of the first value is a zero (0) and vacancy logic  272  will communicate a control signal to the selector input terminal of multiplexer  271  so that multiplexer  271  will couple the first input terminal of multiplexer  271  to the output terminal of multiplexer  271 . The vacancy logic  272  will also communicate a control signal to the present input terminal (PR) of FIFO  274  so that FIFO  274  will not communicate an event packet to the second input terminal of multiplexer  271 . Therefore, when the first bit of the first value is zero (0) either the event packet  277  or the logic high signal will be coupled to the next event ring segment. Whether the event packet  277  or the logic high signal is coupled to the next event ring segment depends on the signal communicated from the control output terminal (CN) of source number checker  275 . If the first bit of the first value is zero (0) and the received source number does not match the assigned source number, then event packet  277  will be coupled to the next event ring segment. If the first bit of the first value is zero (0) and the received source number does match the assigned source number, then the logic high signal will be coupled to the next event ring segment. Coupling the logic high signal to the next event ring segment has the effect of deleting event packet  277  from the local event ring and inserting no event packet in the local event ring. 
     When an event packet is not present, the first bit of the first value is one (1) and vacancy logic  272  will communicate a control signal to the selector input terminal of multiplexer  271  so that multiplexer  271  will couple the second input terminal of multiplexer  271  to the output terminal of multiplexer  271 . The vacancy logic  272  will also communicate a control signal to the present input terminal (PR) of FIFO  274  so that FIFO  274  will communicate event packet  279  to the second input terminal of multiplexer  271 . Therefore, when the first bit of the first value is a one (1) the next event packet  279  stored in FIFO  274  will be coupled to the next event ring segment. 
     In operation with respect to the global event chain, global event filter logic  233  filters monitors event packet being communicated along the local event ring. All event packets have a “type of event” field indicating the category of event the event packet is representing. An event packet bit sequence map and table is provided in  FIGS. 38A and 38B . The global event filter logic  233  compares the “type of event” bits of an event packet being communicated along the local event ring to an array of “type of event” bit sequences that have been identified as representing global events. In one embodiment, the array of “type of event” bit sequences is communicated to the global event filter logic  233  via the configurable mesh control bus. In another embodiment, the array of “type of event” bit sequences is hardcoded within the global event circuit logic  233 . In an embodiment, the “type of event” field within the event packet is four bits wide. 
     In the embodiment illustrated in  FIG. 37 , event packet  277  is communicated from the output terminal of register  270  to the event packet input terminal (EP) of global event filter logic  233 . When event packet  277  has a “type of event” bit sequence that is the same as one of the array of “type of event” bit sequences identified as representing global event, global event filter logic  233  communicates event packet  235  from the global event output terminal (GE) of global event filter logic  233  to the data input terminal of FIFO  234 . In one embodiment, event packet  235  is identical to event packet  277 . In another embodiment, the “vacancy indicator”, “type of event”, and “event data” fields of event packet  235  are the same as those of event packet  277 ; however, the “source of event” field of event packet  235  is updated with the source number assigned to event ring circuit  210 . FIFO  234  acts as a buffer to store event packets until the event packets can be inserted into the global event ring. Event packets can only be inserted into the global event ring when the previous event chain segment is carrying no event packet. The global event filter logic  233  communicates a signal from the push output terminal (P) of the global event filter logic  233  to the push input terminal (P) of FIFO  234  when the global event filter logic  233  is communicating an event packet from the global event output terminal (GE) to the data input terminal of FIFO  234 . FIFO  234  has a finite memory size. In the event that FIFO  234  memory is full, no additional event packets from global event filter logic  233  may be stored in FIFO  234 . A signal indicating that the FIFO  234  is full is communicated from the full output terminal (F) of FIFO  234  to the full input terminal (F) of global event filter logic  233 . FIFO  234  always outputs a value; however, if no event packets are stored in FIFO  234  an output value with a first bit set to one (1) is communicated to the first terminal of multiplexer  232 . 
     A second value is clocked into register  228  from the previous event chain segment during a transitioning of global clock signal  254 . The second value is then communicated from the output terminal of register  228  to the input terminal of vacancy logic  230 , the event packet input terminal (EP) of source number checker  229 , and the first input terminal of multiplexer  231 . 
     Source number checker  229  receives an assigned source number from the configurable mesh control bus via the control bus input terminal (CB). The assigned source number is a number that is unique to event ring circuit  210 . In one embodiment, all event packets inserted into a global event chain by event ring circuit  210  will have a “source of event” field that indicates the source number assigned to event ring circuit  210 . In another embodiment, event packets inserted into a global event chain by event ring circuit  210  will have a “source of event” field that indicates a source number assigned to one event ring circuit within the same local event ring as event ring circuit  210 . In one embodiment, the “source of event” field is eight bits wide. An event packet bit sequence map and table is provided in  FIGS. 38A and 38B . Source number checker  229  communicates the assigned source number from the source number output terminal (SN) of source number checker  229  to the source number input terminal (SN) of global event filter logic  233 . Global event filter logic  233  may use the source number information received from source number checker  229  to fill in the “source of event” field in event packets inserted into the global event chain. 
     Source number checker  229  also receives eight bits of the second value via the event packet input terminal (EP). The eight bits make up the “source of event” field and indicate which event ring circuit in the global event chain inserted the value. The source number checker  229  compares the received eight bits with the eight bit source number assigned to event ring circuit  210  via the configurable mesh control bus. When the eight bits representing the assigned source number are the same as the received eight bits of the first value, the source number checker  229  communicates a signal from control output terminal (CN) to the selector input terminal of multiplexer  231 , such that multiplexer  231  couples the second input terminal of multiplexer  231  to the output terminal of multiplexer  231 . The result is that source number checker  229  causes the coupling of the logic high signal to the output terminal of multiplexer  231  when the eight source of event bits match, and that the source number checker  229  causes the coupling of the output of register  228  to the output of multiplexer  231  when the eight source of event bits do not match. Coupling the logic high signal to the output terminal of multiplexer  231  has the effect of passing no event packet to the second input terminal of multiplexer  232 . Source number checker  229  also receives a global clock signal  254  via the clock input terminal. The source number checker  229  transmits and receives communications during transitions of the global clock signal. 
     Vacancy logic  230  receives the first bit of the second value communicated from register  228 . The first bit of the second value is one (1) if no event packet is present. The first bit of the second value is zero (0) if an event packet is present. 
     When an event packet  256  is present, the first bit of the second value is a zero (0) and vacancy logic  230  will communicate a control signal to the selector input terminal of multiplexer  232  so that multiplexer  232  will couple the second input terminal of multiplexer  232  to the output terminal of multiplexer  232 . The vacancy logic  230  will also communicate a control signal to the present input terminal (PR) of FIFO  234  so that FIFO  234  will not communicate an event packet to the first input terminal of multiplexer  232 . Therefore, when the first bit of the second value is zero (0) either the event packet  256  or the logic high signal will be coupled to the next event chain segment. Whether the event packet  256  or the logic high signal is coupled to the next event chain segment depends on the signal communicated from the control output terminal (CN) of source number checker  229 . If the first bit of the second value is zero (0) and the received source number does not match the assigned source number, then event packet  256  will be coupled to the next event chain segment. If the first bit of the second value is zero (0) and the received source number does match the assigned source number, then the logic high signal will be coupled to the next event chain segment. Coupling the logic high signal to the next event chain segment has the effect of deleting event packet  256  from the global event chain and inserting no event packet in the global event chain. 
     When an event packet is not present, the first bit of the second value is one (1) and vacancy logic  230  will communicate a control signal to the selector input terminal of multiplexer  232  so that multiplexer  232  will couple the first input terminal of multiplexer  232  to the output terminal of multiplexer  232 . The vacancy logic  230  will also communicate a control signal to the present input terminal (PR) of FIFO  234  so that FIFO  234  will communicate event packet  236  to the first input terminal of multiplexer  232 . Therefore, when the first bit of the second value is a one (1) the next event packet  236  stored in FIFO  234  will be coupled to the next event chain segment. 
     The configurable mesh control bus is coupled to the control bus input terminal (CB) of source number checkers  222  and  229 . Source number checkers  222  and  229  may be assigned a source number via the configurable mesh control bus. In one embodiment, source number checkers  222  and  229  are assigned a source number once upon powering on the IB-NFP integrated circuit  12 . In another embodiment, source number checkers  222  and  229  are assigned a first source number upon powering on the IB-NFP integrated circuit  12  and a second source number, replacing the first source number, during subsequent operation of the IB-NFP integrated circuit  12 . 
       FIG. 39  is a diagram of the configurable mesh event bus configured to form two local event rings and a global event chain.  FIG. 39  shows the configurable mesh event bus of  FIG. 33  configured to form a first  245  and a second  246  local event ring and a single global event chain  247 . Ingress NBI island  72  is the ingress NBI island  72  shown in  FIGS. 33 and 34 . The first local event ring  245  flows through the ingress MAC island  71 , ingress NBI island  72 , ingress ME island  66 , ARM island  51  and eight other islands ( 53 ,  57 ,  62 ,  65 ,  60 ,  55 ,  61 , and  56 ). The second local event ring  246  flows through egress NBI island  63 , egress MAC island  64  and four other islands ( 54 ,  58 ,  59 , and  69 ). The global event chain  247  begins on ingress NBI island  72 , couples through island  66 , island  62 , island  57 , island  54  and terminates on ARM island  51 . Both local event rings  245  and  246  are unidirectional, in that event packets travel in a single direction along the local event ring. The global event chain is unidirectional, in that global event packets travel in a single direction along the global event chain. Whereas the global even chain has a single point of termination on ARM island  51 , both local event rings  245  and  246  have no fixed termination location. Rather, event packets are removed at different points along the local event ring depending upon where in the local event ring the event packet was inserted into the local event ring. Each local event ring operates as a source-release ring. 
       FIG. 40  is a simplified system level illustration of a source-release ring. The source-release ring includes event ring circuits and event ring segments. An event ring segment couples a first event ring circuit to a second event ring circuit. The source-release ring illustrates the flow of the first local event ring  245  through ingress ME island  66 , ingress NBI island  72 , and Interlaken LA( 1 ) island  65 . The first local event ring  245  flows through a single event ring circuit  248  within ME island  66 . The first local event ring  245  continues through the first  209  and second  210  event ring circuits of ingress NBI island  72  to event ring circuit  249  within Interlaken LA( 1 ) island  65 . A source-release ring is a circuit in which an event packet is clocked into sequential registers along a single direction. The event packet travels around the ring, stepping through the ring one register at a time. All event packets are transferred from the present register to the next register upon the transition of a synchronized clock cycle. In a source-release ring, only the source of an event packet can delete the event packet from the ring. Event ring circuits must monitor the passing event packet and only delete the event packet if the present event ring circuit was the source of the event packet.  FIG. 40  is a simplified drawing. Event ring segments may comprise switches, slots, traces on silicon or other functional circuitry. 
     Referring to  FIG. 39  and traveling clockwise around the source-release ring shown in  FIG. 40 , an event packet  280  is inserted into local event ring  245  by event ring circuit  248  within ME island  66 . The event packet  280  is then communicated via an event ring segment to the first event ring circuit  209  within ingress NBI island  72 . Since an event packet generated by another event ring circuit is present, the first event ring circuit  209  within ingress NBI island  72  cannot insert or delete an event packet, but can monitor the contents of the event packet  280 . The event packet  280  is then communicated via an event ring segment to the second event ring circuit  210  within ingress NBI island  72 . Since an event packet generated by another event ring circuit is present, the second event ring circuit  210  within ingress NBI island  72  cannot insert or delete an event packet, but can monitor the contents of the event packet  280 . The event packet  280  is then communicated via an event ring segment to an event ring circuit  249  within the Interlaken LA( 1 ) island  65 . Since an event packet generated by another event ring circuit is present, the event ring circuit  249  within the Interlaken LA( 1 ) island  65  cannot insert or delete an event packet, but can monitor the contents of the event packet  280 . The event packet  280  will continue to be communicated around the first local event ring  245  until it returns to the event ring circuit  248  within the ingress ME island  66 , where event ring circuit  248  will delete or “release” the event packet  280  from the local event ring  245 . 
       FIG. 41  is a simplified diagram of the first  245  and second  246  local event rings and single global event chain  247  shown in  FIG. 39 . The first local event ring  245  includes an event ring circuit  248  with ingress ME island  66 , the first event ring circuit  209  within ingress NBI island  72 , the second event ring circuit  210  within NBI island  72 , an event ring circuit  249  within Interlaken LA( 1 ) island  65 , and multiple event ring segments coupling each event ring circuit to the following event ring circuit. The second local event ring  246  includes event ring circuit  250  within ME cluster island  54 , other event ring circuits and multiple event ring segments coupling each event ring circuit to the following event ring circuit. The ARM island  51  includes an event manager  251  and processor  253 . A global event chain  247  couples event ring circuit  210  to event ring circuit  250  and event ring circuit  250  to event manager  251  within ARM island  51 . Operation of event manager  251  is disclosed below with respect to  FIG. 42 . 
     Event packets are communicated along first  245  and second  246  local event rings as described above with respect to  FIG. 40 . With respect to the first local event ring  245 , once an event packet is communicated to the second event ring circuit  210  within ingress NBI  72 , the global event filter  227  within the second event ring circuit  210  monitors the event packet and detects if the event packet represents a global event. If the global event filter  227  detects an event packet representing a global event, the event packet is copied to the global event chain  247 . With respect to the second local event ring  246 , once an event packet is communicated to the event ring circuit  250  within ME cluster island  54 , a global event filter within event ring circuit  250  monitors the event packet and detects if the event packet represents a global event. If the global event filter within event ring circuit  250  detects an event packet representing a global event, the event packet is copied to the global event chain  247 . 
     In one embodiment, the event ring circuit  248  within the ingress ME island  66  inserts an event packet  281  into the first local event ring  245 . The event packet  281  represents a global event. The event packet  281  is then communicated to the first event ring circuit  209  within ingress NBI island  72  via an event ring segment. Since an event packet generated by another event ring circuit is present, the first event ring circuit  209  within ingress NBI island  72  cannot insert or delete an event packet, but can monitor the contents of the event packet  281 . The event packet  281  is then communicated to the second event ring circuit  210  within ingress NBI island  72  via an event ring segment. Since an event packet generated by another event ring circuit is present, the second event ring circuit  210  within ingress NBI island  72  cannot insert or delete an event packet, but can monitor the contents of the event packet  281 . The global event filter  227  within event ring circuit  210  determines that event packet  281  represents a global event. Upon determining that the event packet  281  represents a global event, event ring circuit  210  will copy the event packet  281  to the first event chain segment of the global event chain  247 . The event packet  281  is also communicated along the first local event ring  245  as described with respect to  FIG. 40  above. The event packet  281  then is communicated along a second event chain segment of the global event chain  247  to event manager  251  within ARM island  51 . Simultaneously, other event packets are communicated along the second local event ring  246 . The global event filter in event ring circuit  250  determines if event packets represent global events. Upon determining that an event packet represents a global event, the global event filter in event ring circuit  250  copies the event packet to the second segment of the global event chain  247 . As shown above, the global event chain  247  allows event packets representing global events to be communicated to ARM island  51  from both the first  245  and the second  246  local event rings. 
     It is noted that while select event packets are only copied to the global event chain  247  by global event filters; global event filters do not delete or “release” event packets from the first  245  or second  246  local event rings. Only the event ring circuit which inserted the event packet into the local event ring can delete or “release” the said event packet from the local event ring. Event packets are only deleted from the global event chain by event manager  251  within ARM island  51 . Global events may represent events such as error code correction (ECC) events or direct memory access (DMA) events. 
     In another embodiment, the event ring circuit  248  within the ingress ME island  66  inserts the event packet  282  to the first local event ring  245 . Event packet  282  does not represent a global event. The event packet  282  is then communicated to the event ring circuit  209  within ingress NBI island  72  via an event ring segment. Since an event packet generated by another event ring circuit is present, event ring circuit  209  within ingress NBI island  72  cannot insert or delete an event packet, but can monitor the contents of the event packet  282 . Since the event ring circuit  209  does not have a global event filter, the event ring circuit  209  will not determine if the event packet  282  represents a global event and will not copy the event packet  282  to the global event chain. The event packet  282  is communicated to event ring circuit  210  within ingress NBI island  72  via an event ring segment. Since an event packet generated by another event ring circuit is present, event ring circuit  210  within ingress NBI island  72  cannot insert or delete an event packet, but can monitor the contents of the event packet  282 . The global event filter  227  within event ring circuit  210  determines that event packet  282  does not represent a global event. Upon determining that the event packet  282  does not represent a global event, event ring circuit  210  will not copy the event packet  282  to the first event chain segment of the global event chain  247 . The event packet  282  is communicated along the first local event ring  245  as described with respect to  FIG. 40  above. As shown above, the global event chain  247  does not communicate event packets representing local event to the ARM island  51 . 
       FIG. 42  shows the event manager  251  of ARM island  51  in further detail. An event manager is also included in all cluster local scratch circuits, such as cluster local scratch  342  shown in  FIG. 50 . Event manager  251  receives incoming global event packets from the last segment of the global event chain, detects various types of the global event packets, and collects and logs various types of information regarding the detected types of global event packets. Event manager  251  may, for example, count the number of certain types of global event packets received. Event manager  251  may, for example, log certain information about certain types of global event packets. The collected data is stored in a data register. The ARM processor can configure the global event manager  251  via data bus  260  and set up which types of global event packets will be detected and logged. The ARM processor  253  can then read the collected data from the data register via data bus  260 . 
     In one embodiment, event manager  251  includes thirty-two event filters. One of the event filters  255  includes mask logic  262 , a mask register  256 , match logic  263 , a match register  257 , configurable analysis logic  261 , a configuration register  258 , and a data register  259 . The last segment of the global event chain is coupled to an input of an amount of mask logic  262 . The mask logic  262  uses the value in mask register  256  to mask out certain parts of the incoming global event packet. The output of the mask logic  262  is supplied to match logic  263 . The match logic  263  compares the output of the mask logic  262  to the value or values stored in match register  257 . Match register  257  may, for example, include an array of bit sequences. If the unmasked event packet bits output by the mask logic are determined to match one of the bit sequences stored in match register  257 , then the match logic  263  outputs a digital value indicative of the occurrence. The digital value output by the match logic is supplied to configurable analysis logic  261 . Configurable analysis logic  261  may, for example, be configured by the value in configuration register  258  to be a counter so that the configurable analysis logic counts the number of matches that occur. The output of the configurable analysis logic  261  is written into data register  259 . Processor  253  can read the results from data register  259  via data bus  260 . Processor data bus  260  is coupled to a first input (CPU_DB) of processor  253 . 
     OPERATIONAL EXAMPLE 
       FIG. 43  is a schematic diagram that illustrates an operational example of IB-NFP integrated circuit  12  within the MPLS router  1  of  FIG. 1 . 100 Gbps packet traffic is received via optical cable  7  (see  FIG. 1 ), flows through optics transceiver  10 , flows through PHY integrated circuit  11 , and is received onto IB-NFP integrated circuit  12  spread across the four SerDes I/O blocks  19 - 22 . Twelve virtual input ports are provided at this interface in the example of  FIG. 1 . The symbols pass through direct dedicated conductors from the SerDes blocks  19 - 22  to ingress MAC island  71 . Ingress MAC island  71  converts successive symbols delivered by the physical coding layer into packets by mapping symbols to octets, by performing packet framing, and then by buffering the resulting packets for subsequent communication to other processing circuitry. The packets are communicated from MAC island  71  across a private inter-island bus to ingress NBI (Network Bus Interface) island  72 . Although dedicated connections are provided for this purpose in the particular example described here, in other examples the packets are communicated from ingress MAC island  71  to ingress NBI island via the configurable mesh data bus. 
     For each packet, the functional circuitry of ingress NBI island  72  examines fields in the header portion to determine what storage strategy to use to place the packet into memory. In one example, the NBI island examines the header portion and from that determines whether the packet is an exception packet or whether the packet is a fast-path packet. If the packet is an exception packet then the NBI island determines a first storage strategy to be used to store the packet so that relatively involved exception processing can be performed efficiently, whereas if the packet is a fast-path packet then the NBI island determines a second storage strategy to be used to store the packet for more efficient transmission of the packet from the IB-NFP. 
     In the operational example of  FIG. 43 , NBI island  72  examines a packet header, performs packet preclassification, determines that the packet is a fast-path packet, and determines that the header portion of the packet should be placed into a CTM (Cluster Target Memory) in ME (Microengine) island  66 . The header portion of the packet is therefore communicated across the configurable mesh data bus from NBI island  72  to ME island  66 . The CTM is tightly coupled to the ME. The ME island  66  determines header modification and queuing strategy for the packet based on the packet flow (derived from packet header and contents) and the ME island  66  informs a second NBI island  63  of these. In this simplified example being described, the payload portions of fast-path packets are placed into internal SRAM (Static Random Access Memory) MU block  78  and the payload portions of exception packets are placed into external DRAM  40  and  41 . 
     Half island  68  is an interface island through which all information passing into, and out of, SRAM MU block  78  passes. The functional circuitry within half island  68  serves as the interface and control circuitry for the SRAM within block  78 . For simplicity purposes in the discussion below, both half island  68  and MU block  78  may be referred to together as the MU island, although it is to be understood that MU block  78  is actually not an island as the term is used here but rather is a block. In one example, MU block  78  is an amount of so-called “IP” that is designed and supplied commercially by a commercial entity other than the commercial entity that designs and lays out the IB-NFP integrated circuit. The area occupied by block  78  is a keep out area for the designer of the IB-NFP in that the substantially all the wiring and all the transistors in block  78  are laid out by the memory compiler and are part of the SRAM. Accordingly, the mesh buses and associated crossbar switches of the configurable mesh data bus, the mesh control bus, and the mesh event bus do not pass into the area of block  78 . No transistors of the mesh buses are present in block  78 . There is an interface portion of the SRAM circuitry of block  78  that is connected by short direct metal connections to circuitry in half island  68 . The data bus, control bus, and event bus structures pass into and over the half island  68 , and through the half island couple to the interface circuitry in block  78 . Accordingly, the payload portion of the incoming fast-path packet is communicated from NBI island  72 , across the configurable mesh data bus to SRAM control island  68 , and from control island  68 , to the interface circuitry in block  78 , and to the internal SRAM circuitry of block  78 . The internal SRAM of block  78  stores the payloads so that they can be accessed for flow determination by the ME island. 
     In addition, a preclassifier in the ingress NBI island determines that the payload portions for others of the packets should be stored in external DRAM  40  and  41 . For example, the payload portions for exception packets are stored in external DRAM  40  and  41 . Interface island  70 , IP block  79 , and DDR PHY I/O blocks  46  and  47  serve as the interface and control for external DRAM integrated circuits  40  and  41 . The payload portions of the exception packets are therefore communicated across the configurable mesh data bus from NBI island  72 , to interface and control island  70 , to external MU SRAM block  79 , to 32-bit DDR PHY I/O blocks  46  and  47 , and to external DRAM integrated circuits  40  and  41 . At this point in the operational example, the packet header portions and their associated payload portions are stored in different places. The payload portions of fast-path packets are stored in internal SRAM in MU block  78 , whereas the payload portions of exception packets are stored in external SRAM in external DRAMs  40  and  41 . 
     ME island  66  informs second NBI island  63  where the packet headers and the packet payloads can be found and provides the second NBI island  63  with an egress packet descriptor for each packet. The egress packet descriptor indicates a queuing strategy to be used on the packet. Second NBI island  63  uses the egress packet descriptor to read the packet headers and any header modification from ME island  66  and to read the packet payloads from either internal SRAM  78  or external DRAMs  40  and  41 . Second NBI island  63  places packet descriptors for packets to be output into the correct order. For each packet that is then scheduled to be transmitted, the second NBI island uses the packet descriptor to read the header portion and any header modification and the payload portion and to assemble the packet to be transmitted. Note that the header modification is not actually part of the egress packet descriptor, but rather it is stored with the packet header by the ME when the packet is presented to the NBI. The second NBI island then performs any indicated packet modification on the packet. The resulting modified packet then passes from second NBI island  63  and to egress MAC island  64 . 
     Egress MAC island  64  buffers the packets, and converts them into symbols. The symbols are then delivered by conductors from the MAC island  64  to the four SerDes I/O blocks  25 - 28 . From SerDes I/O blocks  25 - 28 , the 100 Gbps outgoing packet flow passes out of the IB-NFP integrated circuit  12  and across SerDes connections  34  (see  FIG. 1 ) and to switch fabric  9 . Twelve virtual output ports are provided in the example of  FIG. 1 . 
       FIG. 44  is a diagram of the four SerDes I/O blocks  19 - 22  and the ingress MAC island  71 . The symbols  300  pass from the four SerDes I/O blocks and to the ingress MAC island across dedicated conductors  301 . The symbols are converted into packets by a 100 Gbps ethernet block  302 . The 100 Gbps ethernet block  302  analyzes the packets and places the results in this analysis at the beginning of the packet in the form of a “MAC prepend” value. The resulting packets and associated MAC prepend values are then buffered in SRAM  305 . Reference numeral  303  identifies a part of the block that represents one packet and reference numeral  304  identifies a part of the block that represents the MAC prepend value. The MAC prepend value  304  includes: 1) an indication of the length of the packet, 2) an indication whether the packet is an IP packet, 3) and indication of whether the checksums are correct, and 4) a time stamp indicating when the packet was received. 
     As packets are loaded into SRAM, a statistics block  306  counts the number of packets that meet certain criteria. Various sub-circuits of the ingress MAC island are configurable. The input conductors  307  labeled CB couples the certain portions of the MAC island to the control bus tree illustrated in  FIG. 29  so that these portions receive configuration information from the root of control bus tree. SRAM block  305  includes error detection and correction circuitry (ECC)  308 . Error information detected and collected by ECC block  308  and statistics block  306  is reported through the local event bus and global event chain back to the ARM island  51  by the mechanism described above in connection with  FIG. 29 . Ingress MAC island  71  is part of one of the local event rings. Event packets are circulated into the MAC island via conductors  309  and are circulated out of the MAC island via conductors  310 . Packets that are buffered in SRAM  305  are then output from the MAC island to the ingress NBI island  72  in the form of one or more 256 byte minipackets  311  communicated across dedicated connections  312 . Statistics information  313  is also communicated to the ingress NBI island  72  via dedicated connections  314 . 
       FIG. 45  is a diagram of packet  303  communicated across connections  312 . 
       FIG. 46  is a diagram of ingress NBI island  72 . Ingress NBI island  72  receives the MAC prepend and the minipackets via dedicated connections  312  from the ingress MAC island  72 . The first 256 bytes of the packet and the MAC prepend pass through multiplexing circuitry  315  and to a characterizer  316 . Characterizer  316  outputs characterization information, the first sixty-four bytes of the packet, and the MAC prepend. This is passed to a pool  317  of forty-eight picoengines. Each picoengine executes a program stored in an associated instruction control store. Reference numeral  318  identifies the first picoengine and reference numeral  319  identifies its instruction control store. The program in the instruction control store for each picoengine can be updated and changed under software control via control block  320 . Control block  320  is also usable to receive the statistics information  313  from the MAC island via XPB bus connections  314 . To perform deeper and deeper analysis into the header structure of an incoming packet, the output of the pool  317  can be passed back through a tunnel recirculation path and tunnel recirculation FIFO  400  to the characterizer  316  in an iterative fashion. Pool  317  outputs preclassification results  321 . 
       FIG. 47  is a table that sets forth the part of preclassification results  321 . The preclassification results  321  include: 1) a determination of which one of multiple buffer pools to use to store the packet, 2) a sequence number for the packet in a particular flow of packets through the IB-NFP, and 3) user metadata. The user metadata is typically a code generated by the picoengines, where the code communicates certain information about the packet. In the present operational example, the user metadata includes a bit. If the bit is set then the packet was determined to be of a first type (an exception packet), whereas if the bit is not set then the packet was determined to be of a second type (a fast-path packet). 
     The packet is buffered in SRAM  322 . A buffer pool is a set of targets in ME islands where header portions can be placed. A buffer list is a list of memory addresses where payload portions can be placed. DMA engine  323  can read the packet out of SRAM via conductors  324 , then use the buffer pools to determine a destination to which the packet header is to be DMA transferred, and use the buffer lists to determine a destination to which the packet payload is to be DMA transferred. The DMA transfers occur across the configurable mesh data bus. In the case of the exception packet of this example the preclassification user metadata and buffer pool number indicate to the DMA engine that the packet is an exception packet and this causes a first buffer pool and a first different buffer list to be used, whereas in the case of the fast-path packet the preclassification user metadata and buffer pool number indicate to the DMA engine that the packet is a fast-path packet and this causes a second buffer pool and a second buffer list to be used. Block  326  is data bus interface circuitry through which the configurable mesh data bus in accessed. Arrow  325  represents packets that are DMA transferred out of the NBI island  72  by DMA engine  323 . Each packet is output with a corresponding ingress packet descriptor. 
       FIG. 48  is a table that sets forth the parts of an ingress packet descriptor. An ingress packet descriptor includes: 1) an address indicating where and in which ME island the header portion is stored, 2) an address indicating where and in which MU island the payload portion is, 3) how long the packet is, 4) a sequence number for the flow to which the packet belongs, 5) user metadata. 
     The programs stored in the instruction stores that are executable by the picoengines can be changed multiple times a second as the router operates. Configuration block  327  receives configuration information from the control bus CB tree via connections  328  and supplies the configuration information to various ones of the sub-circuits of NBI island  72  that are configurable. Error detection and correction (ECC) circuitry  329  collects error information such as errors detected in the contents of the instruction stores. ECC circuitry  329  and ECC circuitry  330  are coupled via connections  331  and  332  and other internal island connections not shown to be part of the local event ring of which the ingress MAC island  72  is a part. 
       FIG. 50  is a diagram of the microengine (ME) island  66 . In the present operational example, packet headers and the associated preclassification results are DMA transferred from the ingress NBI island  72  across the configurable mesh data bus and into the Cluster Target Memory (CTM)  333  of the ME island  66 . The DMA engine  323  in the ingress NBI island is the master and the CTM  333  is the target for this transfer. The packet header portions and the associated ingress packet descriptors pass into the ME island via data bus island bridge  334  and data bus interface circuitry  335 . Once in the CTM  333 , the header portions are analyzed by one or more microengines. The microengines have, through the DB island bridge  334 , a command out interface, a pull-id in interface, a pull-data out interface, and a push data in interface. There are six pairs of microengines, with each pair sharing a memory containing program code for the microengines. Reference numerals  336  and  337  identify the first pair of microengines and reference numeral  338  identifies the shared memory. As a result of analysis and processing, the microengines modify each ingress packet descriptor to be an egress packet descriptor. Each egress packet descriptor includes: 1) an address indicating where and in which ME island the header portion is found, 2) an address indicating where and in which MU island the payload portion is found, 3) how long the packet is, 4) sequence number of the packet in the flow, 5) an indication of which queue the packet belongs to (result of the packet policy), 6) an indication of where the packet is to be sent (a result of the packet policy), 7) user metadata indicating what kind of packet it is. 
     Memory errors and other events detected in the ME island are reported via a local event ring and the global event chain back to the ARM island  51 . A local event ring is made to snake through the ME island for this purpose. Event packets from the local event chain are received via connections  339  and event packets are supplied out to the local event chain via connections  340 . The CB island bridge  341 , the cluster local scratch  342 , and CTM  333  can be configured and are therefore coupled to the control bus CB via connections  343  so that they can receive configuration information from the control bus CB. 
     A microengine within the ME island can use data bus commands to interact with a target, regardless of whether the target is located locally on the same ME island as the microengine or whether the target is located remotely in another island, using the same configurable data bus communications. If the target is local within the ME island, then the microengine uses data bus commands and operations as described above as if the memory were outside the island in another island, except that bus transaction values do not have a final destination value. The bus transaction values do not leave the ME island and therefore do not need the final destination information. If, on the other hand, the target is not local within the ME island then intelligence  343  within the DB island bridge adds the final destination value before the bus transaction value is sent out onto the configurable mesh data bus. From the perspective of the microengine master, the interaction with the target has the same protocol and command and data format regardless of whether the target is local or remote. 
       FIG. 51  is a diagram of a bus transaction value  344  used to communicate packet data from the ingress NBI island  72  to the ME island  66 . In a multi-target island such as the ME island  66 , the target field  345  of the bus transaction value contains a number that indicates which target it is that is to receive the payload of the bus transaction value. In the present example, the header portions of the incoming 100 Gbps flow are written into CTM  333 . 
     The local event ring flow through ME island  66  illustrated in  FIG. 50 . The local event ring couples from the event bus (EB) through CB island bridge  341 , cluster local scratch  342 , cluster target memory  333  and back to event bus (EB). Local and global events are communicated between CB island bridge  341 , cluster local scratch  342 , cluster target memory  333  via the local event bus. Each cluster local scratch  342  contains an event manager  251  as illustrated in  FIG. 42 . 
       FIG. 52  is a diagram of MU half island  68  and SRAM block  78 . MU half island  68  includes several hardware engines  350 . In the operational example, packet payloads are DMA transferred directly from ingress NBI island  72  and across the configurable mesh data bus, through data bus interface  352  of half island  68 , and into the data cache SRAM  351  block  78 . The ingress NBI DMA engine  323  issues a bulk write command across the configurable mesh data bus to the bulk transfer engine  346 . The destination is the MU island. The action is bulk write. The address where the data is to be written into the MU island is the address taken out of the appropriate buffer list. The bulk write command received at the MU is a bulk write, so the data bus interface  352  presents the command to the bulk engine. The bulk engine examines the command which is a write. In order to perform a write the bulk engine needs data, so the bulk engine issues a pull-id through the pull portion of interface  352 , which in turn issues a pull-id back onto the configurable mesh data bus. The NBI DMA engine  323  receives the pull-id. Part of the pull-id is a data reference which indicates to the DMA engine which part of the packet is being requested as data. The DMA engine uses the data reference to read the requested part of the packet, and presents that across the data part of the data bus back to the bulk engine  346 . The bulk engine  346  then has the write command and the packet data. The bulk engine  346  ties the two together, and it then writes the packet data into the SRAM  351  at the address given in the write command. In this way, packet payload portions pass from DMA engine in the ingress NBI island, across the configurable mesh data bus, through the data bus interface  352 , through a bulk transfer engine  346 , and into data cache SRAM  351 . 
     In the present operational example, a microengine in the ME island  66  issues a lookup command across the configurable mesh data bus to have lookup hardware engine  350  examine tables in SRAM  351  for the presence of given data. The data to be looked for in this case is a particular MPLS label. The lookup command as received onto the MU island is a lookup command so the data base interface  352  presents the lookup command to the lookup engine. The lookup command includes a table descriptor of what part to memory to look in. The lookup command also contains a pull-id reference indicating what to look for (the MPLS label in this case). The data to look for is actually stored in transfer registers of the originating microengine. The lookup engine  350  therefore issues a pull-id out onto the configurable mesh data bus request back to the originating microengine. The microengine returns the requested data (the MPLS label to look for) corresponding to the reference id. The lookup engine now has the lookup command, the table descriptor, and the MPLS label that it is to look for. In the illustration there are three tables  353 - 355 . A table description identifies one such table by indicating the starting address of the table in SRAM  351 , and how large the table is. If the lookup operation is successful in that the lookup hardware engine  350  finds the MPLS label in the table identified by the table descriptor, then the lookup hardware engine  350  returns a predetermined value “Packet Policy”  356  back to the requesting microengine. A packet policy is a code that indicates: 1) a header modification to be done, and 2) a queueing strategy to use. Lookup engine  350  returns the packet policy  356  to the originating microengine by pushing the data (the packet policy) via the push interface of the configurable mesh data bus. 
     Various parts of the MU island are configurable by changing the contents of registers and memory via the control bus CB and connections  357  and control status registers  362 . Errors detected on the MU island by circuits  360  and  361  are reported into a local event ring. Event packets from the local event ring are received via input connections  358  and the MU island outputs event packets to the local even ring via output connections  359 . Various sub-circuits of the MU island are configurable. 
       FIG. 53  is a diagram of egress NBI island  63 . In the operational example, ME island  66  instructs the egress NBI island  63  to transmit a packet by supplying the egress NBI island with an egress packet descriptor of the packet to be transmitted. The ME island supplies the egress packet descriptor to the egress NBI island by issuing a transmit packet command across the configurable mesh data bus and to the packet reorder block  401 . The packet reorder block  401  responds by pulling the packet descriptor from the ME island across the configurable mesh data bus. In this way, multiple egress packet descriptors enter packet reorder block  401 . These egress packet descriptors are reordered so that the descriptors for the packets of a flow are in proper sequence. The scheduler  366  receives the properly ordered egress packet descriptors and pushes them onto appropriate queues in queue SRAM  367 . Each such queue of egress packet descriptors is per port, per data type, per group of connections. Reference numeral  368  identifies one such queue. Packets of a connection in this case share the same set of source and destination IP addresses and TCP ports. Scheduler  366  schedules packets to be transmitted by popping egress packet descriptors off the queues in appropriate orders and at appropriate times, and by supplying the popped egress packet descriptors via conductors  381  to the DMA engine  363 . 
     DMA engine  363  receives such an egress packet descriptor, and based on the information in the descriptor, transfers the payload portion and the header portion of the packet across configurable mesh data bus and DB interface  364  and into FIFO  365 . In the illustration of  FIG. 47 , each entry in FIFO  365  includes a complete packet having the header portion  371 , the payload portion  372 , and a script identifier portion  373 . The script identifier portion  373  was added by the ME island. As a result of the lookup performed at the direction of the ME island, a packet policy was determined, and part of this packet policy is an indication of what of the packet header to change and how to change it before the packet is transmitted. An example of such a modification is to change the MAC source and destination addresses at the time the packet is output from the IB-NFP. 
     In a typical MPLS router, the MPLS labels of packets can remain the same as the packets flow into and through and out of the router. The MAC addresses of such a packet, however, should be changed on a hop by hop basis. The MAC hop on the ingress may be different from the MAC address on the egress. Accordingly, the packet exiting the MPLS router should have its source and destination MAC addresses changed to be appropriate for the next MAC hop into which the packet will be transmitted. The ME island supplies a script identifier portion for each packet for this purpose. The script identifier portion includes a code that identifies one of the scripts present in script SRAM  375 . The identified script, when executed by packet modifier  374 , causes the MAC addresses for the associated packet to be changed to values stored in an associated argument SRAM  376 . Each resulting modified packet is then output from the egress NBI island  63  as a sequence of 256 byte minipackets across dedicated connections  369  to egress MAC island  64 . Reference numeral  370  identifies one such minipacket. 
     Error conditions detected by ECC circuits  377  and  378  are injected into a local event ring in the form of event packets. Event packets from the local event ring are received onto the egress NBI island via connections  379 , and event packets from the egress NBI island are supplied through the remainder of the local event ring via connections  380 . Various parts of the egress NBI island are configurable. Configuration information for this purpose is received onto the egress NBI island from the control bus CB via connections  382 . 
       FIG. 54  is a diagram of egress MAC island  64 . A packet  383  for transmission are received from egress NBI island  63  in the form of minipackets  370  via dedicated connections  369 . The packets are buffered in SRAM  384 . In the operational example, the packets to be output from the egress MAC island via are converted into symbols by Interlaken block  385 . The resulting symbols  386  pass via dedicated connections  387  to the four SerDes I/O blocks  25 - 28 . As described above in connection with  FIG. 1 , the four SerDes I/O blocks are coupled by SerDes connections  29  to switch fabric  9  of the MPLS router  1 . ECC circuitry  388  of SRAM  384  is made a part of a local event ring via EB connections  389  and  390 . Sub-circuits of MAC island are configurable. Configuration information for these sub-circuits is received from the control bus tree via connections  391 . 
       FIG. 55  is a diagram that illustrates a packet flow in the operational example when local memory resources in the CTM  333  of the ME island  66  are determined not to be scarce. An incoming packet passes through the ingress MAC island  71  and the ingress NBI island  72  as described above. Arrow  392  indicates that the header portion is then transferred (Step  1 ) across the configurable mesh data bus into CTM  333  of ME island  66 , whereas arrow  393  indicates that the payload portion of the packet is transferred (Step  2 ) across the configurable mesh data bus into the MU island  68 ,  78  without being stored in the ME island. The payload portion of each packet is stored in the MU island such that spare memory space is left at the beginning of where the payload is stored. That spare memory space is adequate to accommodate the header portion of the packet without overwriting other packet payloads. In the case of  FIG. 55 , however, the header portion is never written into the MU island. Microengines of the ME island and hardware engines of the MU island analyze and process the packet. Arrow  394  indicates that the header portion is then transferred (Step  3 ) from the ME island  66  and to the egress NBI island  63 . Arrow  395  indicates that the payload portion is transferred (Step  3 ) from the MU island  68 ,  78  to the egress NBI island  63 . The same step number is used because these transfers may occur simultaneously. The header portion and the payload portion are combined in the NBI island  63  and then pass through the egress MAC island  64  and the four SerDes I/O blocks and out of the IB-NFP integrated circuit. 
       FIG. 56  is a diagram that illustrates a packet flow in the operational example when local memory resources in the CTM  333  of the ME island  66  are determined to be scarce. An incoming packet passes through the ingress MAC island  71  and the ingress NBI island  72  as described above. As indicated by arrow  396 , the header portion is then transferred (Step  1 ) across the configurable mesh data bus into CTM  333  of ME island  66 . Arrow  396  indicates that the payload portion of the packet is transferred (Step  2 ) across the configurable mesh data bus into the MU island  68 ,  78  without being stored in the ME island. As in the case described above in connection with  FIG. 55 , the payload portion of each packet is stored in the MU such that spare memory space exists at the beginning of where the payload is stored. The spare memory space is adequate to accommodate the header portion without overwriting other packet payloads. Based on how long it will take before the packet will be transmitted from the IB-NFP, the egress NBI island  63  determines that the header portion shall be moved from the ME island and to MU island in order to free up resources in the CTM  333  of the ME island. As indicated by arrow  398 , the header portion is transferred (Step  3 ) from the ME island and is stored into the ME island into the spare memory space at the beginning of its associated payload portion. Microengines of the ME island and hardware engines of the MU island analyze and process the packet. The packet may be analyzed before the transfer  398 , or after the transfer  398 , or both. When the scheduler of the egress NBI island determines that the packet is to be transferred for transmission from the IB-NFP integrated circuit, then the header portion and the payload portion are DMA transferred (Step  4 ) together directly from the MU island and across the configurable mesh data bus and to the egress NBI island. Arrow  399  indicates this transfer of the packet header and the packet payload. The packet then passes across dedicated connections from the egress NBI island  63  to the egress MAC island  64 , and through the four SerDes blocks, and out of the IB-NFP integrated circuit. 
       FIG. 57  is a configurable mesh event bus configured to take corrective action when system resources within an island are determined to be scarce.  FIG. 57  shows the configurable mesh event bus of  FIG. 39  configured to form a first  245  and a second  246  local event ring and a single global event chain  247 . NBI island  72  corresponds to the NBI island  72  shown in  FIGS. 33 and 34 . The first local event ring  245  flows through the ingress MAC island  71 , ingress NBI island  72 , ingress ME island  66 , ARM island  51  and eight other islands ( 53 ,  57 ,  62 ,  65 ,  60 ,  55 ,  61 , and  56 ). The second local event ring  246  flows through the egress NBI island  63 , egress MAC island  64  and four other islands ( 54 ,  58 ,  59 , and  69 ). The global event chain  247  begins on ingress NBI island  72 , couples through island  66 , island  62 , island  57 , island  54  and terminates on ARM island  51 . Both local event rings are unidirectional, in that event packets travel in a single direction along the local event ring. The global event chain is unidirectional, in that global event packets travel in a single direction along the global event chain. While the global even chain has a single point of termination on ARM island  51 , both local event rings have no fixed termination location. Rather, event packets are removed at different points along the local event ring depending upon where in the local event ring the event packet was inserted into the local event ring. Each local event ring operates as a source-release ring. 
     An exemplary request for additional system resources communicated across various islands is shown in  FIG. 57 . Ingress NBI island  72  determines (Step  1 ) that additional system resources are required. In response to the need for additional system resources, ingress NBI island  72  inserts an event packet into first local event ring  245 . The event packet indicates the source of the event packet and the system resource requested. The event packet is communicated along the first local event ring  245  to ME island  66 . Event ring circuit  248  within ME island  66  receives (Step  2 ) the event packet from ingress NBI island  72 . In response to receiving the event packet requesting additional system resources, ME island  66  communicates a request for additional resources across the configurable mesh data bus to egress NBI island  63  (Step  3 ). In step  4 , egress NBI island  63  determines what system resources are available and allocates the required system resources for use by ingress NBI  72 . After allocating the required system resources, egress NBI island  63  communicates information regarding the allocated resources across the configurable mesh data bus to ingress NBI island  72  (Step  5 ). In step  6 , ingress NBI island  72  receives the information regarding the allocated resources and begins utilization of the requested system resources. In an embodiment, a system resource is a processing resource. A processing resource may be an amount of buffer memory. A buffer memory may be used for storing incoming packet information. Additionally, information regarding the allocated resources may be a pointer address for the buffer memory being allocated. 
     Although certain specific embodiments are described above for instructional purposes, the teachings of this patent document have general applicability and are not limited to the specific embodiments described above. Accordingly, various modifications, adaptations, and combinations of various features of the described embodiments can be practiced without departing from the scope of the invention as set forth in the claims.