Abstract:
A network interface controller includes a plurality of scatter gather circuits ( 104   a - 104   d ) connectable to a host via a bus ( 101 ). A packet buffer ( 112 ) is configured for communication with the scatter gather circuits ( 104   a - 104   d ). A plurality of access circuits ( 110   a - 110   d ) are configured to access external network connections. An optional forwarding engine ( 108 ) is selectable to generate routing information corresponding to data received via the access circuits ( 110   a - 110   d ) and to provide the routing information to the packet buffer ( 112 ).

Description:
TECHNICAL FIELD 
       [0001]    The present invention is in the field of network interface controllers and, in particular, is related to a switching network interface controller that is configurable for use in a variety of different networking configurations. 
       BACKGROUND 
       [0002]    There are a variety of networking applications for which, conventionally, various specialized hardware is available. It would be desirable to have a network interface controller solution that is flexibly configured for use in these various applications. 
       SUMMARY 
       [0003]    A reconfigurable network interface controller circuit is provided for various networking operations. The circuit is scalable, and provides for redundancy in such networking operations. 
         [0004]    The network interface controller includes a plurality of scatter gather circuits connectable to a host via a bus. A packet buffer is configured for communication with the scatter gather circuits. A plurality of access circuits are configured to access external network connections. An optional forwarding engine is selectable to generate routing information corresponding to data received via the access circuits and to provide the routing information to the packet buffer. 
     
    
     
       BRIEF DESCRIPTION OF FIGURES 
         [0005]      FIG. 1  schematically illustrates an example multipurpose network interface controller circuit. 
           [0006]      FIGS. 2A and 2B  illustrate the  FIG. 1  circuit in a network interface controller configuration. 
           [0007]      FIGS. 3A and 3B  illustrate the  FIG. 1  circuit in a switch configuration. 
           [0008]      FIG. 4  illustrate the  FIG. 1  circuit in a hybrid configuration. 
           [0009]      FIG. 5  illustrates the  FIG. 1  circuit in a network processor configuration. 
       
    
    
     DETAILED DESCRIPTION 
       [0010]    Turning now to  FIG. 1 , an example configurable network interface controller circuit  100  is described. In some embodiments, the  FIG. 1  network interface controller circuit is implemented in an application specific integrated circuit (ASIC), such as a field-programmable gate array (FPGA). In one example, the circuit is configurable to implement applications such as:
       Multi-Channel Ethernet NIC Controller   L2/L3 Ethernet Switch   System Controller   SOHO Router   Network Processor       
 
         [0016]    The various blocks of the  FIG. 1  example are now summarized. 
       PCI-X Core (PX  102 ) 
       [0017]    This is a standard quad-function PCI-X core, with 4 identical Master interfaces to SGE&#39;s  104   a  through  104   d , and one target interface TI  106 . This block in addition houses the Serial ROM interface logic. 
       Forwarding Engine (FE  108 ) 
       [0018]    This module makes routing decisions based on a vector that it receives from the MAC  110  or the SGE  104  modules. The results of the routing decisions are conveyed to the packet buffer PB  112  via the Route Vector bus  114 . 
       MAC (Mn  110 ) 
       [0019]    This is a 10/100/1000 MAC module. In one example, this module includes VLAN Tag and Station Address logic (which are normally disabled in the presence of a Forwarding Engine). 
       Packet Buffer (PB  112 ) 
       [0020]    This is a general purpose queuing engine core switch fabric, that allows management of arbitrary sized packets. A particular example of the PB  112  is described in U.S. Provisional Patent Application No. 60/283,285 filed Apr. 11, 2001 and incorporated herein by reference in its entirety. 
       Memory Controller (MC  116 ) 
       [0021]    This is a generic memory controller that can control, for example, SSRAM memories  118 . 
       Lookup Memory Controller (LM  120 ) 
       [0022]    This is an SSRAM only memory controller that provides a path to an external memory  122  (e.g., for a forwarding table and/or instruction memory). 
       CPU Interface (CI  124 ) 
       [0023]    This module is a collection of muxes and base addresses that switches various busses and manages the address map of an on-board processor  125 . This module decodes the PL-Bus regions and generates chip selects to various modules. 
       Scatter Gather Engine (SGn  104 ) 
       [0024]    This module handles the Scatter Gather operation in a NIC mode, provides a path to the packet buffer  112  from PX  102  in the Switch mode, and provides a path to the packet buffer  112  from the CPU  125  in the System Controller mode. 
       Wake up, On Now Logic (ON  124 ) 
       [0025]    This module implements power management functions. 
       Narrow Checksum (NC  126 ) 
       [0026]    This module calculates the TCP/IP/HTTP/UDP checksums and provides the result to the packet buffer  112  upon completion of a frame arriving from the MAC  110 . 
       Wide Checksum (WC) 
       [0027]    This module calculates the TCP/IP/HTTP/UDP checksums and provides the result to the packet buffer PB  112  upon the completion of a frame as it is arriving from the PCI bus  101  via the PX  102 . 
         [0028]    In all modes of the operation (NIC, Switch, etc.), the data is pushed to the SGE  104  by the packet buffer  112  and to the packet buffer  112  by the SGE  104 . 
         [0029]    Clocking within the  FIG. 1  example is now described. Specifically, the components of the  FIG. 1  example operate in three different clock domains. These clock domains include the core clock domain (which includes the on-board CPU  125  and the memory interfaces (LM  120  and MEMC  116 ); the MAC  110  clock domains (which, in some examples, includes several actual clock regimes); and the PCI-X  101  clock domain. In some examples, no phase lock loops are employed. 
         [0030]    In cases where the  FIG. 1  example is an FPGA as opposed to an ASIC, the FPGA clocking is the same as the ASIC clocking except for the MAC clock section. Further, some IO delay lines are inherent to the FPGA IO pads, and are explicitly built in the ASIC configuration. 
         [0031]    In reset, all clock domains are synchronized to the PCI-X clock domain. The PHY chips can be reset individually. In addition, soft reset resources allow the local resetting of the  FIG. 1  circuitry by the local processor  125  under control of local firmware. 
         [0032]    In accordance with one example, the  FIG. 1  circuit is operable in at least four different modes by reconfiguring the operation of the various modules. Each of these four modes of operation are now described. In the description, the Receive and Transmit nomenclatures are always with reference to the wire (i.e., receive from the wire and transmit to the wire) unless otherwise specified. 
         [0033]    The NIC operation mode is now discussed with reference to  FIGS. 2A and 2B . In general, the NIC operation mode is a relatively simple mode of operation of the  FIG. 1  example circuit. A simplified block diagram of the NIC mode is represented in  FIGS. 2A and 2B . In this mode, the on-chip processor  124 , the forwarding engine  108  and the associated secondary memory  122  are not used (and, in fact, need not be even populated in the ASIC/FPGA) and, thus, are shown as “grayed out” in  FIG. 2B . Various NIC operation mode operations are now described. 
       NIC Receive 
       [0034]    In the NIC receive operation, data is received from the MAC  110 , through the N-Bus, and stored in the PB  112  using, for example, the paging hardware of the packet buffer PB  112 . The Rvec is determined in one example using a stub model, for the packet buffer PB  112  to determine from which queue it should push the data out. Once the data is completely stored in the appropriate queues of the packet buffer, it is then pushed to the appropriate Scatter Gather Engine  104 . While transiting on the N-Bus, checksums are calculated on the incoming data from the MAC  110  by the NCS  126  and stored into the packet buffer PB  112 . In one example, the checksum  110  storage is accomplished by strobing a payload (including the checksum) to the packet buffer PB  112  following the last element of the data being stored. The stored data is then transferred into the host memory from the PB  112  by the Scatter Gather Engine  104  (via the PX  102  and the PCI-X  101 ). In this mode of operation, the packet buffer PB  112  does not perform any broadcast or multicast operations and, as such, looks substantially like an external FIFO per channel. 
       NIC Transmit 
       [0035]    In the NIC transmit mode, data is gathered from the host memory (via the PCI-X  101  and PX  102 ) by the appropriate SGE  104  and pushed to the packet buffer PB  112 . While transiting the W-Bus, checksums are calculated on the data by the WCS  128  and delivered to the packet buffer PB  112  by strobing a payload (including the checksum) into the packet buffer PB  112  following storage of the last element of data. Once the packet is completely stored in the packet buffer PB  112 , the packet is forwarded onto the N-Bus to the appropriate MAC  110 . 
       NIC Fail Over 
       [0036]    The link fail over is handled entirely in software. 
       NIC Multicast and Broadcast 
       [0037]    These features are handled entirely in software. 
       NIC Flow Control 
       [0038]    In the receive direction, once the descriptor queue of a TAP in the packet buffer PB  112  goes over a high water mark, a signal is asserted to the corresponding MAC  110  to cause the MAC  110  to generate flow control signals to its link partner. Depending on the MAC configuration, the traffic resumes either after a fixed amount of time or after the queue in the packet buffer PB  112  has dropped below the low water mark, the occurrence of which is also transmitted to the MAC  110 . In the transmit direction, once the queue for the TAP, in the packet buffer, becomes full and data “backs up” into an SGE  104 , the transmit function of the packet buffer  112  simply stops. In both directions, once the SGE  104  runs out of resources such as free lists or transmit buffers, traffic simply stops. 
         [0039]    The switch operation mode is now described with reference to  FIGS. 3A and 3B . In the switch configuration, the  FIG. 1  example circuit employs the local forwarding engine FE  108  to make routing decisions for each packet in each direction. A given packet is examined once upon arrival to the source node and once upon exiting the destination node. The local processor CPU  125  is used for management purposes. 
         [0040]    The switch backplane may be thought of as a virtual backplane. That is, the switch backplane includes multiple virtual channels on top of the PCI bus  101 . Each virtual channel includes a scatter gather engine SGE  104  performing DMA writes to a reserved range of addresses in the host memory. That range of address is programmed into the receiving node&#39;s PCI base address register. Therefore, each pairing of the base address value programmed into a node&#39;s PCI base address register and the value programmed into another node&#39;s scatter gather engine  104  receive DMA address register configures a virtual channel. Each node is effectively a quad-function PCI device and hence has 4 memory base addresses, thus allowing 4 virtual full-duplex links to a node. This virtual back-plane of one example uses three base addresses, leaving the fourth one, corresponding to the node number of the device, unused. 
         [0041]    In the switch mode of operation, the Scatter Gather Engine  104  does not do any linked list operations and simply pushes the data as consecutive data elements to the PCI  101 . The data arriving from the PCI  101  is received on the PCI&#39;s target interface  106  which is then muxed to the appropriate SGE  104 . The receive DMA address restarts at the base address at the start of every packet and the lower bits of the address are used to determine the length of a packet. Upper bits of the address, while still within the address range, are used to transfer additional information to the receiver. Each base address range is 256 Mbytes (28 bits). The largest packet that the switch handles is 64 Kbytes (16 bits), allowing 12 bits of payload (e.g., sequence numbers, etc.). Each node can be conceptualized, in some instances, as either a Source Node or a Destination Node. Source Node is the node where a packet initially comes into the system (via the Source Port), and Destination Node is the node through which the packet leaves the switch. 
         [0042]    As data arrives from the MAC  110 , the MAC  110  generates a long vector comprised of the DA, SA, VLAN, QoS fields of the incoming packet. This vector is then submitted to the forwarding engine FE  108 . The packet then bypasses the checksum  126  and wake up 124 blocks before being written into the packet buffer PB  112 . By this time, the forwarding engine  108  computes a routing vector  114  which is provided to the packet buffer  112  for the packet buffer  112  to place on the appropriate destination queue. In one example, a limit of 16 clocks is placed on the forwarding engine  108  for the lookup time per minimum sized packet (64 bytes). During this time, the forwarding engine  108  completes two lookups and a conditional learn operation. 
         [0043]    Once the packet is fully stored in the packet buffer  112 , it is pushed out one of the W-Buses to the SGE module  104 . The SGE module  104 —which operates as a pass through in the switch configuration—simply pushes the data out to the node that was indicated by the forwarding engine  108 . The packet eventually appears on the target interface (T-Bus) of the destination node&#39;s PCI bus  101 . Once the data arrives from the PCI bus  101  on the T-Bus of a destination node, the target interface  106  of the destination node routes the packet to the packet buffer  112  of the destination node. Meanwhile, the forwarding engine  108  computes a routing vector  114  for the packet and submits the routing vector  114  to the packet buffer  112 . 
         [0044]    Switch multicast and broadcast are handled in the forwarding engine  108 . In case of a multicast or broadcast packet, the result is flagged to the packet buffer via a routing vector with multiple bits set. 
         [0045]    In one example, the packet buffer  112  and the forwarding engine  108  can each pipeline two messages, to absorb the latency of the forwarding engine  108 . The SGE  104  use flow control PIO writes to its link partner SGE to control the rate of the switch backplane traffic. 
         [0046]    The hybrid mode of operation is now described with reference to  FIG. 4 . In the hybrid mode of operation, the  FIG. 1  example circuit acts as a 4-port switch combined with a up to a 4-port NIC. The  FIG. 1  example circuit can “own” up to eight MAC addresses, four for its switched ports and up to four for its embedded NIC ports. The hybrid mode of operation is handled in the forwarding engine  108 . Although the architecture allows for more, in some examples, the forwarding engine  108  has only enough bandwidth for a single port NIC. Power management logic is used in the hybrid mode to detect “magic packets,” etc. and to generate appropriate power management events. 
         [0047]    Once a packet arrives from the MAC  110  (“receive event”), the received packet is submitted to the forwarding engine  108 . The received packet is concurrently written into the packet buffer  112  followed by checksum information. The Rvec  114  from the forwarding engine  108  indicates if the packet should be switched out of a port (via one of the N-Buses) or delivered to the host (via a W-Bus). Data received into the SGE  104  from the packet buffer  112  is typically delivered to the host. 
         [0048]    For transmit, the data to be transmitted is normally gathered from the host memory into the packet buffer  112 . However, the SGE  104  also submits the packet to the forwarding engine  108 . The forwarding engine  108  in turn indicates to the packet buffer  112  onto which queue of the packet buffer  112  to place the packet. The checksums are calculated in this path. Although unlikely, one NIC port may wish to send a packet to another NIC port. 
         [0049]    In the “system controller” mode of operation, any external host adapter is leveraged for interfacing to other protocols (e.g., SCSI, USB, SONET, USB, etc.). The NIC appear as a PC to the external host adapters. As such, the  FIG. 1  example circuit provides a high bandwidth path to the local packet buffer  112 . Further, a section of the packet buffer  112  is allocated as simple passive memory for use by the external host adapters. The external host adapters use this area to scatter and gather their traffic into the local memory  118 . The processor  125  can access this data via the C-Bus path. Thus, in this mode, the  FIG. 1  circuit is functioning as a system controller integrated with a processor. 
         [0050]    In the network processor mode of operation, represented in  FIG. 5 , firmware effectively “visits” every packet at the wire speed while relieving some of the overhead of packet processing. All the packet processing may then be performed via the firmware. This mode of operation can be thought of as a superset of the system controller mode of operation. In this mode of operation, it is generally desired to transfer data from one MAC  110  to another MAC  110  or to an external host adapter once the processor has had an opportunity to read and write portions of the packet. Once the packet is modified, the  FIG. 1  circuit recomputes the checksums. In this configuration, the SGE modules  104  are used by the local processor  125  to access the packets via the queuing system. Thus, the head of the W-Bus FIFO in the SGE module  104  is read by the processor  125  via C-Bus and it is written to via C-Bus. 
         [0051]    The network processor receive operation is now described. As packets arrive from the MAC  110 , the checksums for the packets are computed ( 126 ) and the packets are written into the packet buffer  112 . The packet buffer  112  operates in the NIC mode in this case. That is, there is effectively a hard path between N-Bus- 0  to W-Bus- 0 , etc. The received data is pushed out to the respective SGE  104  via the respective W-Bus. Once the SGE  104  senses the presence of new data, it interrupts the local processor  125 . The processor  125  then performs burst reads of SGE&#39;s  104  W-Bus receive FIFO into the data cache of the processor  125 . Once the processor  125  has examined and rewritten the data, the processor  125  scatters the data into a portion of the packet buffer  112  not used for queuing. The data is then gathered by the external host adapter and consumed by the host adapter. Optionally, the data from the local processor  125  is burst written into the SGE&#39;s  104  transmit W-Bus FIFO. The checksum is computed ( 128 ) on this data by the  FIG. 1  example circuit before being written into the packet buffer  112 . Further, the routing vector from the first element of data is retrieved and submitted to forwarding engine  108  for multiplexing onto the Rvec bus  114  so as to place the packet on the desired N-Bus or W-Bus queue in the PB  112 . 
         [0052]    The network processor transmit operation is now described. When a packet is to be transmitted to the wire by an external host adapter, the packet is first scattered into the local memory  118  by the external host adapter. The packet is then read by the local processor  125  via the C-Bus. Once the local processor  125  has examined and rewritten the packet, the local processor  125  burst writes the packets along with a routing vector  114  into the transmit W-Bus queue of the SGE  104 . The checksum on the packet is then computed ( 128 ) and the packet is forwarded to the N-Bus or W-Bus. If the packet to be transmitted is coming from another port of the same instantiation of the  FIG. 1  circuit, the packet is first received by the local processor  125  and is then burst written into the SGE  104  along with a routing vector.