Distributed chassis architecture having integrated service appliances

A distributed virtual chassis comprises scaled-out fabric coupler (SFC) boxes. Each SFC box has fabric ports and a cell-based switch fabric for switching cells associated with a packet among the SFC fabric ports of that SFC box. Distributed line cards (DLCs) include switching DLCs and an appliance DLC (A-DLC). Each switching DLC has network ports. Each switching DLC and A-DLC has DLC fabric ports. Each switching DLC and A-DLC is connected to each of the SFC boxes. The A-DLC provides an upper layer service for packets arriving on the network ports of the switching DLCs. To forward a packet to the A-DLC, a switching DLC divides the packet into cells and distributes the cells among the SFC boxes. The SFC boxes forward the cells to the A-DLC, and the A-DLC reassembles the packet from the cells and provides the upper layer service to the packet.

FIELD OF THE INVENTION

The invention relates generally to data centers and data processing. More particularly, the invention relates to integrating service appliances into a data center having a distributed chassis architecture.

BACKGROUND

Data centers are generally centralized facilities that provide Internet and intranet services needed to support businesses and organizations. A typical data center can house various types of electronic equipment, such as computers, servers (e.g., email servers, proxy servers, and DNS servers), switches, routers, data storage devices, and other associated components. The infrastructure of the data center, specifically, the layers of switches in the switch fabric, plays a central role in the support of the services. In addition, data centers need service appliances in order to provide packet-based services for traffic flows handled by the switches and other equipment.

SUMMARY

In one aspect, the invention features a service appliance comprising a plurality of fabric ports and a network processor having a fabric interface in communication with the fabric ports. The fabric interface receives a cell over each fabric port. Each cell corresponds to a different portion of a packet received over a network. The network processor reassembles the packet from the cells. A processor complex is connected to the network processor by a service port to receive the packet therefrom and to perform an upper layer service on the packet.

In another aspect, the invention features a distributed virtual chassis comprising a plurality of scaled-out fabric coupler (SFC) boxes. Each SFC box has a plurality of SFC fabric ports and a cell-based switch fabric for switching cells associated with a packet among the SFC fabric ports of that SFC box. A plurality of distributed line cards (DLCs) includes a plurality of switching DLCs and at least one appliance DLC (A-DLC). Each switching DLC has a plurality of network ports. Each switching DLC and A-DLC has a plurality of DLC fabric ports by which that switching DLC and A-DLC is connected to each of the SFC boxes. The A-DLC is configured to provide an upper layer service for packets arriving on the network ports of the switching DLCs.

DETAILED DESCRIPTION

Data centers described herein use a distributed chassis architecture, wherein central Scaled-Out Fabric Couplers (SFCs) connect to multiple Distributed Line Cards (DLCs) in a star topology. These SFCs and DLCs together form a distributed virtual chassis that serves as a large cell-switched domain with constant latency across all network ports. This distributed chassis architecture enables scaling the number of network ports into the tens of thousands.

For such a large network domain, various types of service appliances are required to provide Layer 4 through Layer 7 services (e.g., the OSI model defines Layer 4 through Layer 7 as the Transport, Session, Presentation, and Application layers, respectively). As used herein, Layer 4 through Layer 7 services are preferably referred to as upper layer services, whereas services below Layer 4 are referred to preferably as lower layer services. Examples of upper layer services include, but are not limited to, firewall services, Intrusion Prevention/Intrusion Detection (IPS/IDS) services, Server Load Balancing (SLB), and Application Delivery Centers (ADC) services. As described herein, service appliance boxes are integrated seamlessly into the distributed virtual chassis, the appliances becoming an integral part of the constant latency cell-switched backbone of the data center in their role of providing packet-based services to all network ports.

FIG. 1shows an embodiment of a network environment2including a data center10in communication with a management station4and a server6over a network8. Embodiments of the network8include, but are not limited to, local-area networks (LAN), metro-area networks (MAN), and wide-area networks (WAN), such as the Internet or World Wide Web. The data center10is generally a facility that houses various computers, routers, switches, and other associated equipment in support of applications and data that are integral to the operation of businesses and organizations. Such equipment includes a main cell-based switch11in communication with network elements, referred to herein as distributed line cards (DLCs)14, one or more of which is an appliance DLC (A-DLC), as described in more detail below. The facility may be embodied at a single site or distributed among multiple sites.

The management station4can connect directly (point-to-point) or indirectly to a DLC14of the data center10over one of a variety of connections, such as standard telephone lines, digital subscriber line (DSL), asynchronous DSL, LAN or WAN links (e.g., T1, T3), broadband connections (Frame Relay, ATM), and wireless connections (e.g., 802.11(a), 802.11(b), 802.11(g), 802.11(n)). Using a network protocol, such as Telnet, the management station4can access a command-line interface (CLI) of the network element. In general, the server6is a computer (or group of computers) that provides one or more services to the data center10, examples of which include, but are not limited to, email servers, proxy servers, DNS servers, and a control server running the control plane of the distributed virtual chassis, as further described below.

FIG. 2shows an embodiment of the data center10with a distributed chassis architecture, wherein the main cell-based switch11is in communication with the plurality of DLCs14. The main cell-based switch11and DLCs14together form the distributed virtual chassis and correspond to a single cell-switched domain. AlthoughFIG. 2only shows four DLCs14, the number of DLCs in the cell-switched domain can range in the hundreds and thousands. The DLCs14belong to a designated cluster. Each cluster has a master (or controller) DLC, one or more standby or back-up DLCs, and one or more follower DLCs. The data center10can have more than one cluster, although each DLC can be the member of one cluster only.

In the data center10, the functionality occurs on three planes: a management plane, a control plane, and a data plane. The management of the cluster, such as configuration management, runtime configuration management, presentation of information (show and display), graph generation, and handling SNMP requests, occurs on the management plane. The control plane is associated with those functions involving network signaling and control. The data plane manages data flow.

In the data center10, the functionality of the management plane and, optionally, of the control plane is centralized (that is, the management plane functionality being implemented at the master DLC and the control plane being implemented predominately at the server6) and the functionality of the data plane is distributed among the DLCs14. Through the management station4, an administrator of the data center communicates with the master DLC in order to manage the cluster, with conceivably thousands of DLCs, from a single location. To support the control plane functionality of the entire DLC cluster, the server6is configured with sufficient processing power (e.g., with multiple processor cores).

FIG. 3shows an embodiment of the data center10, wherein the main cell-based switch11includes a plurality of scaled-out fabric coupler (SFC) chasses or boxes12-1,12-M (generally, 12) in communication with the plurality of DLC chasses or boxes14-1,14-2,14-N (generally, 14). This example embodiment has four SFC boxes (M=4) and N DLC boxes.

Each SFC box12includes one or more cell-based switch fabric elements (FE)16in communication with N fabric ports18, there being at least as many fabric ports18in each SFC box12as the number of DLC boxes14in the distributed virtual chassis. Each fabric element16of an SFC box switches cells between fabric ports18of that SFC box based on destination information in the cell header.

The DLC boxes14include switching DLCs and one or more appliance DLCs (A-DLC). In their role of providing an upper layer service needed by the traffic flows passing through the data center10, the one or more A-DLCs are integrated into the data plane of the data center10. In the example ofFIG. 3, the DLC box14-N is an appliance DLC, and the other DLCs are switching DLCs. Examples of upper layer services include, but are not limited to, firewall services, intrusion protection services, intrusion detection services, and virtual private network (VPN) services.

Each switching and appliance DLC box14has a plurality of network ports20and a plurality of fabric ports22. The network ports20of the switching DLCs are in communication with the network8. In one embodiment, each switching DLC14has40network ports20, with each of the network ports20being configured as a 10 Gbps Ethernet port; the aggregate bandwidth of these DLCs is 400 Gbps.

Preferably, each fabric port22of the switching and appliance DLCs includes a120Gbps CXP interface. In one embodiment, the CXP interface has twelve lanes (12x), each lane providing a 10 Gbps channel. An example specification of the 120 Gbps 12x CXP interface can be found in the “Supplement to InfiniBand™ Architecture Specification Volume 2 Release 1.2.1”, published by the InfiniBand™ Trade Association. This embodiment of 12-lane CXP is referred to as standard Infiniband (IB) CXP. In another embodiment, the CXP interface has 10 lanes (10x) for supporting 10-lane applications, such as 100 Gigabit Ethernet. This embodiment of 10-lane CXP is referred to as Ethernet CXP.

Each switching DLC box14further includes a plurality of network processors (i.e., network switching elements)24-1,24-2(generally, 24). In general, network processors are optimized for packet processing. Each network processor24is in communication with every fabric port22and with a subset of the network ports20(for example, each network processor24can switch cells derived from packet traffic received on 20 of the 40 network ports). An example implementation of the network processor24is the BCM 88650, a 20-port, 10 GbE switch device produced by Broadcom, of Irvine, Calif.

The appliance DLC box14-N includes a network processor24in communication with processor complex28comprised of a multi-core network packet processor (NPP) with built-in acceleration for packet processing (APP). The NPP can be implemented, for example with Cavium Network's Octeon multi-core processor or LSI Logic's Tarari multi-core processor or NetLogic's/RMI's XLR multi-core processors. The network processor24forwards packets to the NPP28over one or more of the network ports20connected to the NPP28.

The appliance DLC box14-N can include multiple processor complexes28over other network ports of the network processor24(additional processor complexes28being shown in phantom). In such cases, each individual processor complex28provides the same or a different type of service (e.g. one processor complex can provide a firewall, another processor complex can provide IPS/IDS, and still another processor complex can provide a different appliance service, such as server load balancing (SLB). Because each of these processor complexes is connected to different network port or set of network ports of the network processor24, the switching DLCs can forward their packets to those network ports for the specific service required.

In this embodiment, the distributed virtual chassis10has a full-mesh configuration: each switching and appliance DLC14is in communication with each of the SFCs12; more specifically, each of the fabric ports22of a given DLC14is in electrical communication with a fabric port18of a different one of the SFCs12over a communication link26. Referring to the A-DLC14-N as a representative example of the DLCs14, the A-DLC fabric port22-1of the A-DLC14-N is in communication with the fabric port18-N of the SFC12-1, the A-DLC fabric port22-2with the fabric port18-N of the SFC12-2, the A-DLC fabric port22-3with the fabric port18-N of the SFC12-3, and the A-DLC fabric port22-4with the fabric port18-N of the SFC12-4. Connected in this full-mesh configuration, the switching DLCs, appliance DLC, and SFCs form the distributed virtual chassis, with the switching DLCs and A-DLC acting as line cards. The distributed virtual chassis is modular; that is, switching DLCs and appliance DLCs can be added to or removed from the distributed virtual chassis, one at a time, like line cards added to or removed from a chassis.

The communication link26between each DLC fabric port22and an SFC fabric port18can be a wired connection. Interconnect variants include Direct Attached Cable (DAC) or optical cable. DAC provides five to seven meters of cable length; whereas the optical cable offers up to100meters of connectivity within the data center, (standard optical connectivity can exceed 10 km). Alternatively, the communication link26can be a direct physical connection (i.e., electrical connectors of the DLC fabric ports22physically connect directly to electrical connectors of the SFC fabric ports18).

The full-mesh configuration ofFIG. 3, having the four SFC boxes12, is a full-line rate configuration, that is, the aggregate bandwidth for transmitting cells from a given DLC to the SFCs (i.e., 480 Gbps) is greater than the aggregate bandwidth of packets arriving at the given DLC on the network ports20(i.e., 400 Gbps). This full-line rate configuration achieves a 2.5 μs constant latency throughout the switch domain (2.0 μs of the 2.5 μs latency being attributed to packet ingress and egress at the switching chip and 0.5 μs being attributed to the switching latency at the fabric elements of the SFC).

The full-mesh configuration can also be modified to support various oversubscription permutations for DLCs14. For example, instead of having four SFCs, the central switch fabric12may have only two SFCs12-1,12-2, with each DLC14using only two fabric ports22for communicating with the SFCs, one fabric port22for each of the SFCs12. This permutation of oversubscription has, for example, each DLC on its network side with an aggregate ingress 400 Gbps bandwidth (40, 10 Gbps Ethernet Ports) and an aggregate egress 240 Gbps cell-switching bandwidth on its two 120 Gbps fabric ports22for communicating with the two SFCs. Other oversubscription permutations can be practiced.

FIG. 4AandFIG. 4Bshow an embodiment of a process50for integrating Layer 4 to Layer 7 networking services in the distributed virtual chassis ofFIG. 3. At step52, the control plane (e.g., at the server6) becomes aware of the presence of the A-DLC14-N in the distributed virtual chassis and identifies its service port as Pn. The control plane can be made aware of the A-DLC automatically through use of a switch discovery protocol or manually through administrative configuration. For purposes of illustration, consider that packets arriving at the switching DLC14-1belong to a traffic flow, f, and require the service provided by the A-DLC14-N. The traffic flow, f, is characterized by various parameters, including the source MAC address (smac), the destination MAC address (dmac), the source IP address (sip), the destination IP address (dip), the protocol or Ethernet type (proto), the source port (sport), the destination port (dport), and the virtual LAN (vlan) to which the traffic flow belongs.

To receive the required service, the packets of the traffic flow, f, need to be sent to the A-DLC14-N. The control plane installs (step54) flow-based filters for the traffic flow, f, on all of the switching network ports20of the switching DLCs14, with an associated action such as “forward_to_new_port(Pn)”. In embodiments having multiple network (i.e., service) ports20that connect the network processor24to the processor complex28of the A-DLC14-N, the control plane can manage traffic filters to achieve load balancing across the network (i.e., service) ports20. The load balancing can ensure the A-DLC14-N does not become a point of congestion on the distributed virtual chassis.

In addition, the number of service ports generally corresponds to the amount of bandwidth that the A-DLC14-N has for packet processing. Thus, the control plane can correlate the total number of service ports to the total number of flows to be inspected. Depending upon the number of service ports, Pn, the control plane accepts flow configurations. Alternatively, the control plane can suggest the bandwidth requirements for the A-DLC14-N.

Packets belonging to traffic flow f arrive (step56) at a network port20of the switching DLC14-1. In response to the installed traffic filter, the network processor24in communication with this network port20determines (step58) that the packets belong to traffic flow f and are to be redirected to the A-DLC14-N in order to obtain a required service. The network processor24adds a pre-classification header to the packet and partitions (step60) each packet into cells, adding a cell header (used in ordering of cells) to each cell. (Each service port is mapped to an output traffic manager port (otm_port) on the chip of the switching DLC. Each of the otm-ports has a VOQ (virtual output queue) on all switching DLCs. This otm-port is specified in the cell header). The network processor24then sends (step62) the cells out through the fabric ports22to each of the SFCs12, sending different cells to different SFCs12.

For example, consider an incoming packet with a length of 1600 bits. To the packet, the network processor24of the switching DLC14-1adds a pre-classification header of fixed size (e.g. 160 bits). The network processor24then divides the packet with the pre-classification header into multiple cells of a certain size (e.g., 256 bytes, 128 bytes, 64 bytes). The network processor24then sends a different cell to each of the four SFCs12, in effect, achieving a load balancing of the cells across the SFCs12.

The cell-based switch fabric element16of each SFC12that receives a cell examines the header of that cell, determines its destination as the A-DLC14-N, and sends (step64) the cell out through the appropriate one of the fabric ports18of that SFC to the A-DLC14-N. The A-DLC14-N receives these cells belonging to the packets of traffic flow f on its fabric ports22. The network processor24of the A-DLC reassembles (step66) each packet of the traffic flow f from these cells, and forwards (step68) each reassembled packet to the processor complex28over the service port Pn (i.e., the network port20allocated to the service port Pn). The processor complex28provides (step70) the service to each packet.

Depending upon the outcome of the provided service, the packet is either forwarded (step72) to the real destination or dropped (step74) based on the action matched. When sending the packet onward to the real destination, the processor complex28sends (step76) the packet back to the network processor24over the service port Pn. In the process, the network processor24does not learn either the source MAC address or source IP address of the packet. The network processor24then partitions (step78) the packet into cells, and sends (step80) the cells out through the fabric ports22to each of the SFCs12, sending different cells to different SFCs12, for subsequent delivery to the real destination DLC.

The real destination DLC14receives (step82) all cells related to the serviced packets from the SFCs, reassembles the packet (i.e., removing the added headers, combining the cells), and sends (step84) the reconstructed packet out through the appropriate one of its network ports20. Continuing with the previous four-cell example, consider that each SFC12determines that the real destination DLC is DLC14-2. Each SFC12sends its cell out through its fabric port18-2to the DLC14-2. The DLC14-2reassembles the packet from the four received cells (the added packet headers providing an order) and sends the packet out of the appropriate network port20. The pre-classification header information in the cells determines the appropriate network port.

FIG. 5shows a diagram of an embodiment of each switching DLC having the network ports20in communication with the network processors24-1,24-2through a PHY interface30. In one embodiment, the PHY interface30includes an XFI electrical interface (of a 10 Gigabit Small Form Factor Pluggable Module (XFP)) for each of the network ports20. Each network processor24has a fabric interface (I/F)32and is in communication with buffer memory34over memory channels36. In one embodiment, the buffer memory34is implemented with DDR3 SDRAM (double data rate synchronous dynamic random access memory) devices.

The fabric interface32of each network processor24includes a serializer/deserializer (SerDes), not shown, that preferably provides24channels38. The SerDes includes a pair of functional blocks used to convert data between serial and parallel interfaces in each direction. In one embodiment, each SerDes channel38operates at a 10.3 Gbps bandwidth; the aggregate bandwidth of the 24 channels being approximately 240 Gbps (or 480 Gbps when taking both fabric interfaces). In another embodiment, each SerDes channel operates at a 25 Gbps bandwidth. The 24 SerDes channels38are grouped into four sets of six channels each.

The DLC14further includes PHYs40-1,40-2,40-3,40-4(generally 40) in communication with the four (e.g., standard IB CXP) fabric ports22-1,22-2,22-3,22-4, respectively, of the DLC14. Each of the PHYs40is also in communication with a group of six SerDes channels38from each of the two network processors24-1,24-2(thus, each of the PHYs40supports 12 SerDes channels38).

FIG. 6shows a diagram of an embodiment of the appliance DLC14-N including the network processor24with a fabric interface (I/F)32. The fabric interface32includes a serializer/deserializer (SerDes), not shown, that preferably provides24channels38. Each SerDes channel38preferably operates at a 10.3 Gbps bandwidth. The aggregate bandwidth of the 24 channels is approximately 240 Gbps. In another embodiment, each SerDes channel operates at a 25 Gbps bandwidth. The 24 SerDes channels38are grouped into four sets of six channels each.

The appliance DLC14-N further includes buffer memory34(e.g., DDR3 SDRAM), which is in communication with the network processor24over memory channels36. The A-DLC14-N further includes PHYs40-1,40-2,40-3,40-4(generally 40) in communication with the four (e.g., standard IB CXP) fabric ports22-1,22-2,22-3,22-4, respectively, of the DLC14-N. Each of the PHYs40is also in communication with a group of six SerDes channels38from the network processors24(thus, each of the PHYs40supports six SerDes channels38).

In addition, the network processor24is in communication with one or more processor complexes28through the network ports20of the network processor chip24.

As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method, and computer program product. Thus, aspects of the present invention may be embodied entirely in hardware, entirely in software (including, but not limited to, firmware, program code, resident software, microcode), or in a combination of hardware and software. All such embodiments may generally be referred to herein as a circuit, a module, or a system. In addition, aspects of the present invention may be in the form of a computer program product embodied in one or more computer readable media having computer readable program code embodied thereon.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, radio frequency (RF), etc. or any suitable combination thereof.

The program code may execute entirely on a user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on a remote computer or server. Any such remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

Aspects of the described invention may be implemented in one or more integrated circuit (IC) chips manufactured with semiconductor-fabrication processes. The maker of the IC chips can distribute them in raw wafer form (on a single wafer with multiple unpackaged chips), as bare die, or in packaged form. When in packaged form, the IC chip is mounted in a single chip package, for example, a plastic carrier with leads affixed to a motherboard or other higher level carrier, or in a multichip package, for example, a ceramic carrier having surface and/or buried interconnections. The IC chip is then integrated with other chips, discrete circuit elements, and/or other signal processing devices as part of either an intermediate product, such as a motherboard, or of an end product. The end product can be any product that includes IC chips, ranging from electronic gaming systems and other low-end applications to advanced computer products having a display, an input device, and a central processor.

Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.

The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed.