Network interconnect as a switch

An interconnect as a switch module (“ICAS” module) comprising n port groups, each port group comprising n−1 interfaces, and an interconnecting network implementing a full mesh topology where each port group comprising a plurality of interfaces each connects an interface of one of the other port groups, respectively. The ICAS module may be optically or electrically implemented. According to the embodiments, the ICAS module may be used to construct a stackable switching device and a multi-unit switching device, to replace a data center fabric switch, and to build a new, high-efficient, and cost-effective data center.

FIELD OF INVENTION

The present invention relates to computer network. In particular, the present invention relates to interconnecting structure of ICAS module, stackable switching device, multi-unit chassis switching device, network pod, fanout cable transpose rack and datacenter network.

DISCUSSION OF THE RELATED ART

As a result of the recent rapid growth in application needs—in both size and complexity—today's network infrastructure is scaling and evolving at a high rate. The data traffic that flows from a data center to the Internet—i.e., “machine-to-user” traffic—is large, and ever increasing, as more people get connected, and as new products and services are created. However, machine-to-user traffic is merely “the tip of the iceberg,” when one considers the data traffic within the data center—i.e., “machine-to-machine” traffic—necessary to generate the machine-to-user data traffic. Generally, machine-to-machine data traffic is several orders of magnitude larger than machine-to-user data traffic.

The back-end service tiers and applications are distributed and logically interconnected within a data center. To service each user who uses an application program (“app”) or a website, these back-end service tiers and applications rely on extensive real-time “cooperation” with each other to deliver the user's expected customized fast and seamless experience at the front end. To keep up with the demand, even though the internal applications are constantly being optimized to improve efficiency, the corresponding machine-to-machine traffic grows at an even faster rate than their continual optimization (e.g., at the current time, machine-to-machine data traffic is growing roughly faster than doubling every year).

To be able to move fast and to support rapid growth are goals that are at the core of data center infrastructure design philosophy. In addition, the network infrastructure within a data center (“data center network”) must be simple enough as to be managed by a small, highly efficient team of engineers. It is desired that the data center network evolves in the direction that makes deploying and operating the network easier and faster over time, despite scale and exponential growth.

Some of these applications needs relate to the increasing use of data analytic tools (“big data”) and artificial intelligence (“AI”), for example. As discussed above, big data and AI have become very significant distributed applications. Servicing these applications require handling large amounts of data (e.g., petabytes), using great computation power (e.g., petaflops), and achieving very low latency (e.g., responses that become available within 100 ns). Simultaneously providing more powerful processors (“scaling-up”) and exploiting greater parallel processing (“scaling-out”) have been the preferred approach to achieve performance. Unlike scientific computation, however, big-data and AI applications are delivered in the form of services to large numbers of users across the world. Thus, like web servers for web services, clusters of servers dedicated to big data and AI applications have become significant parts of the data center network.

At the current time, data center networks have largely transitioned from layer-2 to all layer-3 (e.g., running Border Gateway Protocol (BGP) and Equal-cost Multi-Path (ECMP) protocols). A large-scale data center today is typically operating at tens of petabits-per-second scale (petascale) and expects growth into the hundreds of petabits-per-second scale in the near future. The cost of provisioning such data center ranges from US$300 million to US$1.5 billion.

Let us define several terms in Table 1, before proceeding with the description of this patent.

TABLE 1TerminologyDescriptionmediaThe media is a concept of a physical entity. It can be optical or electronic.interfaceAn interface is a concept of a physical entity. It contains a transmissionmedia and a reception media. The media can be optical or electrical. Aninterface can associate with a MAC (Medium Access Control) entity orseveral interfaces can associate with a MAC entity.portA port is a concept of a container entity. It includes a set of interfaces. Thenumber of the interfaces depends on the technology. For example, a 40GQSFP Ethernet port consists of 4 interfaces (a total of 8 fibers). The 40GQSFP port associates with a MAC entity. The 10/40G optical technology isimplemented by reconfiguring the 40G QSFP Ethernet to 4 independent 10Ginterfaces each associates with a MAC entity. As such, each interface canprovide connectivity and operate like a port.port groupA port group is a concept of a container entity; it includes a set of ports; thenumber of ports depends on application. Each interface in a port group isconfigured to associate with a MAC presumably. In order to meet the needsof description, this patent introduces and defines the term of “port group”.connectionA connection consists of media and two interfaces communicating throughthe media.linkA link is a concept of a container entity. It includes a set of connections. Thenumber of the connections depends on the technology. For example, two40G QSFP connectors docked together to form one link which contains 4connections; two 10/40G QSFP connectors docked together to form 4 linkseach contains 1 connection.downlinkRefers to the link that connects toward the hosts.uplinkRefers to the link that connects toward the core of the network.intralinkRefers to the link that provides connectivity inside a pod.interlinkRefers to the link that provides connectivity between two pods.

A review of our current state-of-the-art data center infrastructure is instructive. In the following context, data or traffic aggregation refers to multiplexing of communication frames or packets. Aggregation model and disaggregation model refer to topologies of communication networks. The concept of data aggregation is orthogonal to the concept of an aggregation or disaggregation model. Therefore, a disaggregation model can support data aggregation, as discussed below. A basic concept in data communication is that communication channels can be error prone. Transmission over such communication channels at higher data rates and over large distances requires complex and costly transceivers. Consequently, channel encoding, error detection and correction, and communication protocols are many techniques to ensure data is transmitted over long distances with accuracy. In the past, as data transmission was expensive, data aggregation (e.g., multiplexing from different data streams and multiplexing data from multiple topological sources) and data compression ensure even higher utilization of the communication channels and efficient management of the communication cost. This is the origin of the aggregation (i.e., in both data and topology) paradigm. This paradigm dominates the networking industry for decades. Such aggregation is widely used in wide area networks (WANs), where transmission cost dominates over other network costs. Today's hardware architecture for data switching is also based on aggregation, i.e., each port is connected and aggregated from all other port. In today's communication networks, data is typically aggregated before transmitting on to “uplink” to connect to external network (e.g., the Internet), which tends to be the most expensive port of the data switching equipment. Due to both advances in semiconductor, fiber-optical, and interconnect technologies and economy of scale, network costs have reduced significantly. The aggregation model is not necessarily the only—or the most suitable—solution in a data center. In today's data center networks, where machine-to-machine traffic (“east-west traffic”) dominates most of the bandwidth, being several orders of magnitude than the machine-to-user bandwidth, multipath topology and routing (ECMP) are deployed so that the combined network bandwidth is large. However, traffic is still aggregated from all incoming port on to each outgoing port. Nonetheless, the multipath topology signifies a disaggregation model. The detailed description below places a structure and quantification onto the multipath topology and discloses a disaggregation model, referred to herein as “interconnect as a Switch” (“ICAS”), which is significantly different from the more traditional aggregation model for data centers.

Typically, in an enterprise or intranet environment, communication patterns are relatively predictable with a modest number of data sources and data destinations. These data sources and data destinations are typically connected by a relatively small number of designated paths (“primary paths”), with some number of back-up or “secondary paths,” which are provided primarily for fault tolerance. In such an environment, the routing protocols of the enterprise network are optimized to select a shortest single path between each source-destination pair in the absence of a failure.

Distributed computing frameworks (e.g., MapReduce, Hadoop and Dryad) and web services (e.g., web search, ecommerce, social networking, data analytics, artificial intelligence and scientific computing) bring a new paradigm of computing that requires both interconnections between a diverse range of hosts and significant aggregate bandwidths. Due to the scarcity of ports even in the high-end commercial switches, a common hierarchical network topology that has evolved is a fat tree with higher-speed ports and increasing aggregate bandwidths, as one moves up the hierarchy (i.e., towards the roots). The data center network, which requires substantial intra-cluster bandwidths, represents a departure from the earlier hierarchical network topology. In the multi-rooted tree, the shortest single-path routing protocol can significantly underutilize the available bandwidths. The ECMP is an improvement that statically stripes flows across available paths using flow hashing techniques. ECMP is standardized in the IEEE 802.1Q Standard. ECMP allows “next-hop packet forwarding” to a single destination to occur over multiple “best paths,” as symmetric insuring flows on deterministic paths. Equal cost multi-path routing can be used in conjunction with most routing protocols, because it is a per-hop decision limited to a single router. It can substantially increase bandwidth by load-balancing traffic over multiple paths. When a data packet of a data stream arrives at the switch, and multiple candidate paths are available for forwarding the data packet to its destination, selected fields of the data packet's headers are hashed to select one of the paths. In this manner, the flows are spread across multiple paths, with the data packets of each flow taking the same path, so that the arrival order of the data packets at the destination is maintained.

Note that ECMP performance intrinsically depends on both flow size and the number of flows arriving at a host. A hash-based forwarding scheme performs well in uniform traffic, with the hosts in the network communicating all-to-all with each other simultaneously, or in which individual flow last only a few round-trip delay times (“RTTs”). Non-uniform communication patterns, especially those involving transfers of large blocks of data, do not perform well under ECMP without careful scheduling of flows to avoid network bottlenecks.

In the detailed description below, the terms “fabric switch” and “spine switch” are used interchangeably. When both terms appear in a network, a fabric switch refers to a device in a network layer which is used for multipath networking among TOR devices, while a spine switch refers to a device in a higher network layer which is used for multipath networking among pods.

A fat tree network suffers from three types of drawbacks—i.e., 1) congestion due to hash collision, 2) congestion due to an aggregation model, and 3) congestion due to a blocking condition. These congestions are further examined in the following.

First, under ECMP, two or more large, long-lived flows can hash to the same path (“hash collision”), resulting in congestion, as illustrated inFIG. 1a.FIG. 1ashows four fabric switches10-0to10-3interconnecting five TOR switches11-0to11-4. As shown inFIG. 1a, each TOR switch has four ports each communicating with a port of one of the fabric switches10-0to10-3, respectively. Each fabric switch has five ports each communicating with a port of one of TOR switches11-0to11-4, respectively. InFIG. 1a, two flows designating TOR switch11-0are sourced from TOR switches11-1and11-2. However, by chance, each flow is hashed to a path that goes through fabric switch10-0, which causes congestion at designating port101of fabric switch10-0. (Of course, the congestion problem could have been avoided if one of the flows is hashed to a path that goes through fabric switch10-1for instance). Furthermore, the static mapping of the flows to paths by hashing does not consider either current network utilization or the sizes of the flows, so that the resulting collision overwhelms switch buffers, degrade overall switch utilization, and increases transmission latency.

Second, in a fat tree network, the total bandwidth of the aggregated traffic may exceed the bandwidth of all the downlinks of all the fabric switches facing the same TOR switch, resulting in aggregation congestion, as shown inFIG. 1b. Such aggregation congestion is a common problem in the aggregation model of today's switching network, and requires detailed rate limiting to avoid congestion. InFIG. 1b, the traffic through the fabric switches12-0to12-3facing the TOR switch13-0is sourced from the TOR switches13-1to13-4, but the aggregate traffic from the source (one source each from TOR switches13-1to13-3and two sources from TOR switch13-4) exceeds the combined bandwidth of all the downlinks of all the fabric switches12-0to12-3facing the TOR switch13-0. More specifically, traffic is spread out evenly over fabric switch12-1to12-3without congestion; additional traffic from TOR switch13-4exceeds the downlink bandwidth of port121of fabric switch12-0and thus causes congestion.

Third, there is a blocking condition called the “strict-sense blocking condition,” which is applicable to statistically multiplexed flow-based networks (e.g., a TCP/IP network). The blocking condition results from insufficient path diversity (or an inability to explore path diversity in the network) when the number and the size of the flows become sufficiently large.FIG. 1cillustrates the blocking condition in a fat tree network. As shown inFIG. 1c, the blocking condition occurs, for example, when paths from fabric switches14-0and14-1to TOR switch15-0are busy and paths from fabric switches14-2and14-3to TOR switch15-3are busy, and a flow which requires a path through TOR switch15-0arrives at TOR switch15-3. An extra flow between TOR switch15-0and15-3can take one of 4 possible paths. Say it takes the path from TOR switch15-3to fabric switch14-1and then from fabric switch14-1to TOR switch15-0. However, the path from fabric switch14-1to TOR switch15-0is busy already. Overall, multiplexing the blocked flow on to the existing flows results in increased congestion, latency and/or packet loss.

At the same time as the demand on the data center network grows, the rate of growth in CMOS circuit density (“Moore's law”) and the I/O circuit data rate appear to have slowed. The cost of lithographic and heat density will ultimately limit how many transistors can be packed into a single silicon package. That is to say, an ultra large storage or computing system is bound to be achieved through multiple chips. It is unlikely that an ultra large system will be integrated on a single chip with ultra-high integration density as in the past. The question that arises here is how to build an ultra large bandwidth interconnection between the chips. It is instructive to learn that a switching chip soldered on printed circuit board (PCB) employs high-speed serial differential I/O circuit to transmit and receive data to/from transceiver module. A transceiver module interconnects to a transceiver module on a different system to accomplish network communications. An optical transceiver performs the electrical-to-optical and optical-to-electrical conversion. An electrical transceiver performs complex electrical modulation and demodulation conversion. The primary obstacle that hinders high-speed operation on PCB is the frequency-dependent losses of the copper-based interconnection due to skin effects, dielectric losses, channel reflections, and crosstalk. Copper-based interconnection faces the challenge of bandwidth limit as the data rate exceeds several tens of gigabit per second (Gb/s). To satisfy demands for bigger data bandwidth high-radix switch silicon integrates hundreds of differential I/O circuits. For example, Broadcom Trident-II chip and Barefoot Network Tofino chip integrate 2×128 and 2×260 differential I/O circuits for 10 Gb/s transmit and receive respectively. To optimize system level port density, heat dissipation and bandwidth the I/O circuits and interfaces are gathered in groups and standardized in specifications on electrical and optical properties. For SFP+, each port has a pair of TX and RX serial differential interfaces at 10 Gb/s data rate. For QSFP, each port has four pairs of TX and RX serial differential interfaces at 10 Gb/s data rate each for a total of 40 Gb/s or 4×10 Gb/s data rate. For QSFP28, each port has four pairs of TX and RX serial differential interfaces at 25 Gb/s data rate each for a total of 100 Gb/s or 4×25 Gb/s data rate. For QSFP-DD, each port has eight pairs of TX and RX serial differential interfaces with a data rate of 50 Gb/s data rate each for a total of 400 Gb/s or 8×50 Gb/s data rate. State of the art data centers and switch silicon employ 4 or 8 interfaces (TX, RX) at 10 Gb/s or 25 Gb/s or 50 Gb/s per port as design considerations. These groupings are not necessarily unique. MTP/MPO as an optical interconnect standard defines up to 48 interfaces per port where each interface contains a pair of optical fibers one for transmit and one for receive. However the electrical and optical specifications of transceiver with up to 48 interfaces per module are yet to come. The definition of “port group” in this patent disclosure is extended to include more interfaces crossing multiple ports (e.g., 8 interfaces from 2 QSFP's; 32 interfaces from 8 QSFP's, etc.). A person experienced in the art can understand that this invention is applicable to other interconnect standards where multiple various number of interfaces other than 4 be grouped together in the future.

These limitations affect data center networks by, for example, increasing power consumption, slowing of performance increase, and increasing procurement cycle. These developments exacerbate the power needs for the equipment, as well as their cooling, facility space, the cost of hardware, network performance (e.g., bandwidth, congestion, and latency, management), and the required short time-to-build.

The impacts to network communication are several:(a) The network industry may not have enough economy of scale to justify CMOS technology of a smaller footprint;(b) Simpler solutions should be sought to advance network technology, rather than to create more complex ones and packing more transistors;(c) Scale-out solutions (i.e., in complement to scale-up solution) should be sought to solve application problems (e.g., big-data, AI, HPC, and data center);(d) The chip port density (i.e., the number of ports in the traditional sense) can become flat1; and(e) Implementation of interfaces with signal rates in excess of 100G will become increasingly difficult2.1Integration of optical technology to the CMOS device may provide new opportunity. However, do not expect a very high-radix chip, which would allow network scalability, to emerge any time soon.2One must think beyond the aggregation model (e.g., the disaggregation model) to meet new network challenges.

Historically, high-speed networks have two classes of design space. In the first class of design space, HPC and supercomputing networks typically adopt direct network topologies. In a direct network topology, every switch is connected to servers, as well as other switches in the topology. Popular topologies include mesh, torus, and hypercube. This type of network is highly resource efficient and offers high capacity through numerous paths of various lengths between a source and destination. However, the choice of which path to forward traffic over is ultimately controlled by proprietary protocols (i.e., non-minimum routing) in switches, NICs, and by the end-host application logic. That is, an algorithm or manual configuration is required to achieve routing. Such routing protocols increase the burden on the developer and create a tight coupling between applications and the network.

In the second class of design space, data centers scaling-out have resulted in the development of indirect network topologies, such as folded-Clos and multi-rooted trees (“fat trees”), in which servers are restricted to the edges of the network fabric. The interior of the network fabric consists of dedicated switches that are not connected to any servers, but simply route traffic within the network fabric. Data center networks of this type thus have a much looser coupling between applications and network topology, placing the burden of path selection on the network switches themselves. That is to say, based on Internet routing technology such as BGP (Border Gateway Protocol) routing protocol. The BGP routing protocol has a complete set of loop prevention, shortest path and optimization mechanisms. However, there are strict requirements and restrictions on the network topology. Data center technology based purely on Internet BGP routing cannot effectively support multipath with non-shortest path topologies. As a result data center networks have traditionally relied on fat tree topologies, simple routing and equal cost multipath selection mechanisms (e.g., ECMP). It is precisely because data center routing technology has restrictions on the network topology. The benefits to datacenter from non-shortest multipath path network topology other than the equal cost multipath topology have not been explored in the past years of developments of the datacenter technologies.

The BGP and ECMP protocols are not without flaws. ECMP relies on static hashing of flows across a fixed set of shortest equal cost paths to a destination. For hierarchical topologies (e.g., fat tree), ECMP routing has been largely sufficient when there are no failures. However, even now direct network topologies (e.g., Dragonfly, HyperX, Slim Fly, BCube, and Flattened Butterfly), which employ paths of different lengths, have not seen adoption in data centers because of the limitations imposed by both commodity data center switches and the widespread adoption of ECMP routing in data center networks. ECMP is wasteful of network capacity when there is localized congestion or hot-spots, as it ignores uncongested longer paths. Further, even in hierarchical networks, ECMP makes it hard to route efficiently in the presence of failures, and when the network is no longer completely symmetric, and non-shortest paths are available for improving network utilization.

FIG. 2ashows an architecture of a typical state-of-the-art data center network, organized by three layers of switching devices—i.e., “top-of-rack” (TOR) switches and fabric switches implemented in 96 server pods21-0to21-95and spine switches implemented in 4 spine planes20-0to20-3—interconnected by interlinks in a fat tree topology. Details of a spine plane is shown inFIG. 2bwhere a spine plane consists of 48 spine switches22-0to22-47each connecting to 96 server pods. The connections from all 48 spine switches are grouped into 96 interlinks each including a connection from one of spine switches22-0to22-47, respectively, for a total of 48 connections per interlink. Details of a server pod is shown inFIG. 2c, in which a server pod is shown to consist of 48 TOR switches24-0to24-47and 4 fabric switches23-0to23-3, with each TOR switch connected to all 4 fabric switches. Combining the connection information fromFIG. 2b, server pod ofFIG. 2cmay comprise 4 fabric switches23-0to23-3each connects one of 4 spine planes by interlinks, respectively; each interlink comprising 48 connections each connects one of 48 spine switches in a spine plane, respectively. Each TOR switch provides 48×10G connections in 12×QSFP interfaces as downlink to connect to servers. An edge pod is shown inFIG. 2d, details will be given in below.

As shown inFIGS. 2band 2c, and in conjunction withFIG. 2a, the TOR, fabric and spine layers of switches include: (a) a TOR switch layer consisting of 96×48 TOR switches which connect the servers in the data center and which are equally distributed over 96 “server pods”; (b) a spine switch layer consisting of 4×48 spine switches equally distributed over the 4 “spine planes”; and (c) a fabric layer consisting of 96×4 fabric switches, also equally distributed over the 96 server pods. In addition, two of the server pods can be converted to two edge pods.FIG. 2dshows an edge pod. As shown inFIG. 2d, edge pod250may comprise 4 edge switches25-0to25-3, each connects one of 4 spine planes by interlinks, respectively; each interlink comprising 48 connections each connects one of 48 spine switches in a spine plane, respectively. Each edge switch may include one or more uplinks that interconnect an external network.

Details of an implementation of the server and spine pods are further described below inFIG. 2c, 2bin relation toFIG. 2a. This configuration facilitates modularity by assembling each fabric switch and spine switch in an 8 U multi-unit chassis with 96 QSFP ports. As shown inFIG. 2c, each TOR switch is implemented by a switch with 16 QSFP ports, which allocates 12 QSFP ports to connect to the servers in 10G interfaces (i.e., downlinks) and 4 QSFP ports to connect to the four fabric switches in four 40G interfaces in the same server pod. (In this detailed description, a QSFP represents a 40 Gbits/second bandwidth, which can be provided in a single 40G interface or four 10G interfaces, each 40G interface including four receive-transmit pairs of optical fibers and each 10G interface including a receive-transmit pair of optical fibers). The 40G interface between a TOR switch and a fabric switch is used for both intra-pod and inter-pod data traffic.

Each fabric switch in a server pod is implemented by a 96 QSFP ports switch, which allocates (i) 48 QSFP ports in 48 40G interfaces with the 48 TOR switches in the server pod in a fat tree topology, and (ii) 48 QSFP ports in 48 40G interfaces to the 48 spine switches in the single spine plane the fabric switch is connected.

Each spine switch in a spine plane is also implemented by a 96 QSFP ports switch, which provides all 96 QSFP ports in 96 40G interfaces with the 96 fabric switches connected to the spine plane, one from each of the 96 server pods. The data traffic through the spine plane represents inter-pod communications mostly for the server pods.

In the configuration ofFIG. 2a, each server pod includes (i) 384 QSFP transceivers, half of which are provided to the spine planes and half of which are provided to the network side of the fabric switches, (ii) 192 QSFP transceivers provided to the network side of the TOR switches, (iii) 576 transceivers provided to the servers; (iv) 192 optical QSFP cables, (v) 36 application-specific integrated circuits (ASICs), which implements the fabric switches and (vi) 48 ASICs, which implements the TOR switches. The ASIC suitable for this application may be, for example, the Trident-II Ethernet Switch (“Trident II ASIC”). Each spine plane includes 4608 QSFP transceivers, 4608 optical QSFP cables and 432 Trident II ASICs.

The implementation ofFIG. 2aprovides in practice improved congestion performance but does not eliminate congestion. The network organization is based on an aggregation model, intended to improve cost and utilization of communication ports and transmission media under the aggregation model. While this aggregation model may still be valuable for wide-area networks (e.g., the Internet), recent advances of semiconductor technology and economic of scale have called this aggregation model into question, when applied to local area networks.

Summary

According to one embodiment of the present invention, an interconnect as a switch module (“ICAS” module) comprises n port groups, each port groups comprising n−1 interfaces, and an interconnecting network implementing a full mesh topology where each port group comprising a plurality of interfaces each connects an interface of one of the other port groups, respectively.

According to one embodiment of the present invention, a stackable switching device is provided, which includes one or more ICAS modules as depicted above, a plurality of switching devices, and a stackable rackmount chassis, each ICAS module being connected to the plurality of switching devices, such that the ICAS module interconnects at least some interfaces of at least some port groups of different switching devices to form a full mesh non-blocking interconnection, while the rest interfaces of the at least some port groups for interconnecting different switching devices are configured as interfaces for uplink. The ICAS module and the switching devices are housed in the stackable rackmount chassis.

One embodiment of the present invention provides a multi-unit switching device, which includes: one or more ICAS modules implemented on a PCB as a circuit, a plurality of switching devices, and a multi-unit rackmount chassis, each ICAS module being connected to the plurality of switching devices, such that the ICAS module interconnects at least some interfaces of at least some port groups of different switching devices to form a full mesh non-blocking interconnection, while the rest interfaces of the at least some port groups for interconnecting different switching devices are configured as interfaces for uplink. The ICAS module and the switching devices are packaged in the multi-unit rackmount chassis.

According to one embodiment of the present invention, a network pod is disclosed, which includes: a plurality of first layer switching devices, each having a plurality of interfaces for downlink interfaces configured to receive and transmit data signals from and to a plurality of servers, and each having a plurality of network side interfaces divided into a plurality of interlinks and a plurality of intralinks, and the interlink interfaces being configured to connect to higher layer switching devices, and the intralink interfaces of the first layer switching devices each being configured and grouped into one or more port groups; and one or more second layer devices of ICAS modules whose interfaces are divided into intralink interfaces and uplink interfaces, and the intralink interfaces of an ICAS module being grouped into port groups to connect to the corresponding port groups of the first layer switching devices, and each port groups of additional ICAS module being connected to the additional port group of each of the first layer switches, and the uplink interfaces are configured to connect to the external network. The first layer switching devices and the second layer devices are interconnected to implement a full mesh network of a predetermined number of nodes.

K spine planes each having p interlinks are used to connect p network pods each having k TOR switches. In a spine plane, k spine switches interconnect to a fanout cable transpose rack.

According to one embodiment of the present invention, a fanout cable transpose rack may include: k first port groups connecting to corresponding port groups of k spine switches through first plurality of fiber optic cables; p second port groups through connecting second plurality of fiber optic cables to form p interlinks. A plurality of fanout cables are used to cross-connect the k first port groups and the p second port groups so that connections from all k spine switches are grouped into p interlinks, each interlink including one connection from each spine switch, and each interlink having a total of k connections.

According to one embodiment of the present invention, a data center network may have a plurality of interfaces for downlink configured to receive and transmit data signals from and to a plurality of servers, and a plurality of interfaces for uplink configured to connect the Internet or connect another data center network with a similar configuration. The data center network may include: a group of network pods (server pods/ICAS pods), each network pod in the group including: (a) a group of first layer switching devices, providing some interfaces as interfaces for downlink, and having the rest interfaces grouped into one or more network side port groups; and (b) one or more second layer devices, configured to interconnect at least some interfaces between some port groups of the first layer switching devices, wherein the rest interfaces of the some port groups for interconnecting the first layer switching devices are configured as interfaces for uplink. The first layer switching devices and the second layer devices are interconnected to implement a full mesh network of a predetermined number of nodes. The network pod further comprises a group of switch clusters, each including a group of third layer switching devices, each of which routes a plurality of data signals received from or transmitted to a corresponding first layer switching device in each group of network pods.

By simplifying the data center network infrastructure and reducing hardware requirement, the present invention addresses the problems relating to the power needs for the equipment and their cooling, facility space, the cost of hardware, network performance (e.g., bandwidth, congestion, and latency, management), and the required short built time.

The present invention is better understood upon consideration of the detailed description below in conjunction with the accompanying drawings.

To facilitate cross-referencing among the figures and to simplify the detailed description, like elements are assigned like reference numerals.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention simplifies the network architecture by eliminating the switches in the fabric layer based on a new fabric topology, referred herein as the “interconnect-as-a-switch” (ICAS) topology. The ICAS topology of the present invention is based on the “full mesh” topology. In a full mesh topology, each node is connected to all other nodes. The example of a 9-node full mesh topology is illustrated inFIG. 3. The inherent connectivity of a full mesh network can be exploited to provide fabric layer switching.

As discussed in further detail below, the ICAS topology enables a data center network that is far superior to a network of the fat tree topology used in prior art data center networks. Unlike other network topologies, the ICAS topology imposes a structure on the network which reduces congestion in a large extent. According to one embodiment, the present invention provides an ICAS module as a component for interconnecting communicating devices.FIG. 4ashows ICAS module400, which interconnects 9 nodes according to the full mesh topology ofFIG. 3.

FIG. 4ashows ICAS module400having port groups40-0to40-8and each port group providing 8 external interfaces and 8 internal interfaces. In ICAS module400, each of the internal interfaces of a port group connects an internal interface of one of the other port groups, respectively. In fact, each port group is connected to every one of the other port groups through exactly one internal interface. In this context, each “interface” includes a receive-transmit pair of optical fibers capable of, for example, a 10 Gbits per second data rate. InFIG. 4a, the port groups are indexed as 0-8. Indexes can also be arbitrary unequal values (For example, these 9 port groups can also be indexed as 5, 100, 77, 9, 11, 8, 13, 50, and 64). The 8 internal interfaces for these 9 port groups are indexed according to the indexes of the connected port groups (For example, the internal interfaces for the 7-th port group are 0, 1, 2, 3, 4, 5, 6 and 8 in the first example; and are 5, 100, 77, 9, 11, 8, 13 and 64 in the second example). Furthermore, internal interface j of port group i is connected to internal interface i of port group j. The external interfaces for each port group of ICAS module400are indexed sequentially as 0-7.

FIG. 4billustrates in detail the connectivity between the internal interfaces and the external interfaces of a port group7in ICAS module400, in accordance with the present invention. As shown inFIG. 4b, in one embodiment, the external interfaces are connected one-to-one to the internal interfaces sequentially in the index order (For example, for port group7, external interfaces42-0to42-7are sequentially connected to internal interfaces41-0to41-6and41-8). Therefore, for port group i, external interfaces 0-7 are connected to internal interfaces 0, . . . , i−1, i+1, . . . , and 8 respectively. Therefore, it can be easily seen that any pair of port groups x and y are connected through internal interface x of port group y and internal interface y of port group x. This indexing scheme allows an external switching device to assign routes for data packets using the internal interface indices of the source port group and destination port group. No congestion condition (e.g., due to hash collision, aggregation model, or strict-sense blocking) can occur between any pair of port groups.

The internal interconnection between the port groups of the ICAS module can be realized via an optical media to achieve a full mesh structure. The optical media may be an optical fiber and/or 3D MEMS. The 3D MEMS uses a controllable micro-mirror to create an optical path to achieve a full mesh structure. In both of these implementations MPO connectors are used. Alternatively, the ICAS module may also be electrically implemented using circuits. In this manner, the port groups of the ICAS module are soldered or crimped onto a PCB using connectors that support high-speed differential signals and impedance matching. The interconnection between the port groups is implemented using a copper differential pair on the PCB. Since signal losses significantly vary between different grades of high-speed differential connectors and between copper differential pairs on different grades of PCBs, an active chip is usually added at the back end of the connector to restore and enhance the signal to increase the signal transmission distance on the PCB. Housing the ICAS module in a 1 U to multi-U rackmount chassis will form a 1 U to multi-U interconnection device. The ICAS-based interconnection devices are then interconnected with switching devices to form a full mesh non-blocking network. This novel network will be explained in detail hereunder in a plurality of embodiments. When the ICAS module of the 1 U to multi-U interconnection device is optically implemented (based on optical fiber and 3D MEMS), MPO-MPO cables are used to connect the ICAS-based interconnection devices and the switching devices. When the ICAS module of the 1 U to multi-U interconnection device is electrically implemented as circuits (based on PCB+chip), DAC direct connection cables or AOC active optical cables are used to connect the ICAS-based interconnection devices and the switching devices.

As switching in ICAS module400is achieved passively by its connectivity, no power is dissipated in performing the switching function. Typical port group-to-port group delay through an ICAS passive switch is around 10 ns (e.g., 5 ns/meter, for an optical fiber), making it very desirable for a data center application, or for big data, AI and HPC environments.

The indexing scheme of external-to-internal connectivity in ICAS module400ofFIG. 4ais summarized in Table 2 below:

FIG. 5ashows network500, in which ICAS module510and port group2of each of TOR switches51-0to51-8interconnects in a full mesh topology, in accordance with one embodiment of the present invention.

As illustrated inFIG. 5b, in an ICAS module510in the full mesh topology network500ofFIG. 5a, port group51-1of TOR switch1routes a data packet to port group51-7of TOR switch7through external interface53-1-6and internal interface52-1-7of port group50-1of ICAS module510, and internal interface52-7-1and external interface53-7-1of port group50-7of ICAS module510, in accordance with one embodiment of the present invention. As shown inFIG. 5b, TOR switch51-1, which is connected to port group50-1of ICAS module510, receives a data packet with a destination reachable through internal port group52-1-7of ICAS module510. TOR switch51-1has a port group that includes 8 interfaces54-1-0to54-1-7(provided as two QSFP ports) mapping one-to-one to external interfaces53-1-0to53-1-7of port group50-1of ICAS module510, which in turn maps one-to-one to internal interfaces52-1-0,52-1-2to52-1-8in sequential order of port group50-1of ICAS module510. TOR switch51-7has a port group that includes 8 interfaces54-7-0to54-7-7(provided as two QSFP ports) mapping one-to-one to external interfaces53-7-0to53-7-7of port group50-7of ICAS module510, which in turn maps one-to-one to internal interfaces52-7-0to52-7-6and52-7-8in sequential order of port group50-7of ICAS module510. Each interface in a TOR switch port may be a 10G interface, for example. As port groups50-1and50-7of ICAS module510are connected through the port groups' corresponding internal interfaces52-1-7and52-7-1, TOR switch51-1sends the data packet through its interface54-1-6to external interface53-1-6of ICAS module510. Since the connectivity in the ICAS module510adopts a full mesh topology, the data packet is routed to external interface53-7-1of ICAS module510.

In full mesh topology network500, the interfaces of each TOR switch is regrouped into port groups, such that each port group contains 8 interfaces. To illustrate this arrangement, port group2from each TOR switch connects to ICAS module510. As each TOR switch has a dedicated path through ICAS module510to each of the other TOR switches, no congestion can result from two or more flows from different source switches being routed to the same port of destination switch (the “Single-Destination-Multiple-Source Traffic Aggregation” case). In that case, for example, when TOR switches51-0to51-8each have a 10-G data flow that has TOR switch51-0as destination, all the flows would be routed on paths through respective interfaces. Table 3 summarizes the separate designated paths:

In other words, in Table 3, the single-connection data between first layer switch i connected to the port group with index i and first layer switch j connected to the port group with index j is directly transmitted through the interface with index j of the port group with index i and the interface with index i of the port group with index j.

In Table 3 (as well as in all Tables herein), the switch source and the switch destination are each specified by 3 values: Ti.pj.ck, where Tiis the TOR switch with index i, pjis the port group with index j and ckis the interface with index k. Likewise, the source interface and destination interface in ICAS module500are also each specified by 3 values: ICASj.pi.ck, where ICASj is the ICAS module with index j, piis the port group with index i and ckis the internal or external interface with index k.

An ICAS-based network is customarily allocated so that when its port groups are connected to port group i from all TOR switches the ICAS will be labeled as ICASi with index i.

Congestion can also be avoided in full mesh topology network500with a suitable routing method, even when a source switch receives a large burst of aggregated data (e.g., 80 Gbits per second) from all its connected servers to be routed to the same destination switch (the “Port-to-Port Traffic Aggregation” case). In this case, it is helpful to imagine the TOR switches as consisting of two groups: the source switch i and the rest of the switches0to i−1, i+1to8. The rest of the switches are herein collectively referred to as the “fabric group”. Suppose TOR switch51-1receives 80 Gbits per second (e.g., 8 10G flows) from all its connected servers all designating to destination TOR switch51-0. The routing method for the Port-to-Port Traffic Aggregation case allocates the aggregated traffic to its 8 10G interfaces with port group51-1as inFIG. 5a, such that the data packets in each 10G interface is routed to a separate TOR switch in the fabric group (Table 4A):

Note that the data routed to TOR switch51-0has arrived at its designation and therefore would not be routed further. Each TOR switch in the fabric group, other than TOR switch51-0, then allocates its interface0for forwarding its received data to TOR switch51-0(Table 4B):

In other words, at least one multi-connection data between the first layer switch i connected to the port group indexed i and the first layer switch j connected to the port group indexed j is transmitted through the first layer switches connected to at least one of the port groups other than the port group with source index. The multi-connection data arriving at the destination switch will cease to be further routed and transmitted.

To put it more precisely, the multi-connection data transmission occurring between first layer switch i connected to the port group with index i and first layer switch j connected to the port group with index j includes the transmissions includes: as in Table 4A, the first layer switch i is connected, via a plurality of interfaces of the port group with a plurality of index i, to a plurality of first layer switches with a plurality of corresponding indexes for transmission; as in Table 4B, a plurality of the first layer switches with the indexes as shown are connected, via interfaces with index j of the port groups, to the interfaces with the indexes as shown of the port groups with index j of the first layer switches for transmission; those transmissions that arrive at a destination switch will stop routing.

Thus, the full mesh topology network of the present invention provides performance that is in stark contrast to prior art network topologies (e.g., fat tree), in which congestions in the fabric switch cannot be avoided under Single-Destination-Multiple-Source Traffic Aggregation and Port-to-Port Traffic Aggregation cases.

Also, as discussed above, when TOR switches51-0to51-8abide by the rule m≥2n−2, where m is the number of network-side interfaces (e.g., the interfaces with a port group in ICAS module500) and n is the number of the TOR switch's input interfaces (e.g., interfaces to the servers within the data center), a strict blocking condition is avoided. In other words, a static path is available between any pair of input interfaces under any traffic condition. Avoiding such a blocking condition is essential in a circuit-switched network, but is not necessarily significant in a flow-based switched network.

In the full mesh topology network500ofFIG. 5a, each port group with 8 interfaces of ICAS module500connects to a port group with 8 interfaces (e.g., 8 10-G interfaces) of a corresponding TOR switch. Full mesh topology network500ofFIG. 5amay be redrawn in a more compact form inFIG. 6a, with a slight modification.FIG. 6aillustrates ICAS2 module60-2interconnecting to port group2of each of TOR switches61-0to61-8. InFIG. 6a, the interfaces between port group2of TOR switch61-0and port group0of ICAS module60-2(now labeled ‘ICAS2’) are represented as a single line (e.g., the single line between port group2of TOR switch61-0and port group0of ICAS module60-2). Such a line, of course, represents all 8 eight interfaces between the TOR switch and a corresponding port group in ICAS module60-2. This is exactly the case inFIG. 6bwhere each TOR switch63-0to63-8is shown also to have 4 port groups, to allow configuring network620ofFIG. 6b, where three additional ICAS modules62-0,62-1and62-3in addition to62-2and corresponding interfaces are added to network600ofFIG. 6a.

In full mesh topology network500, uniform traffic may be spread out to the fabric group and then forwarded to its destination. In network620ofFIG. 6b, the additional ICAS modules may be used to provide greater bandwidth. So long as the additional port groups are available in the TOR switches, additional ICAS modules may be added to the network to increase path diversity and bandwidth.

The inventor of the present invention investigated in detail the similarities and the differences between the full mesh topology of the present invention and other network topologies, such as the fat tree topology in the data center network ofFIG. 2a. The inventor first observes that, in the architecture of the data center network ofFIG. 2a, the fat tree network represented in a server pod (the “fabric/TOR topology”) can be reduced to a (4, 48) bipartite graph, so long as the fabric switches merely perform an interconnect function for traffic originated among the TOR switches. This (4, 48) bipartite graph is shown inFIG. 7a. InFIG. 7a, the upper set of nodes, nodes0-3(“fabric nodes”)70-0to70-3, represent the four fabric switches in the server pod ofFIG. 2aand the lower set of 48 nodes (i.e., leaf0-47), labeled71-0to71-47, represent the 48 TOR switches in a server pod ofFIG. 2a.

The inventor discovered that an n-node full mesh graph is embedded in a fabric-leaf network represented by a bipartite graph with (n−1, n) nodes (i.e., a network with n−1 fabric nodes and n TOR switch leaves).FIG. 7bshows, as an example, a (5, 6) bipartite graph with 5 nodes72-0to72-4and6leaves73-0to73-5.FIG. 7cshows the 6-node full mesh graph740with 6 nodes74-0to74-5embedded in the (5, 6) bipartite graph ofFIG. 7b.

This discovery leads to the following rather profound results:

(a) An n-node full mesh graph is embedded in an (n−1, n)-bipartite graph; and the (n−1, n) bipartite graph and the data center Fabric/TOR topology have similar connectivity characteristics;

(b) A network in the (n−1, n) Fabric/TOR topology (i.e., with n−1 fabric switches and n TOR switches) can operate in same connectivity characteristics as a network with full mesh topology (e.g., network500ofFIG. 5a);

(c) Fabric switches are unnecessary in an (n−1, n) Fabric/TOR topology network, as the fabric switches merely performs interconnecting function among the TOR switches (i.e., these fabric switches can be replaced by direct connectivity among TOR switches);

(d) A data center network based on a fat tree topology (e.g., the Fabric/TOR topology) can be improved significantly using ICAS modules.

In the following, a data center network that incorporates ICAS modules in place of fabric switches may be referred to as an “ICAS-based” data center network. An ICAS-based data center network has the following advantages:

(a) less costly, as fabric switches are not used;

(b) lower power consumption, as ICAS modules are passive;

(e) effectively less network layers (2 hops less for inter-pod traffic; 1 hop less for intra-pod traffic);

(f) greater scalability as a data center network.

These results may be advantageously used to improve typical state-of-the-art data center networks.FIG. 8ashows an improved data center network800, in accordance with one embodiment of the present invention. Data center network800uses the same types of components as the data center network ofFIG. 2a(i.e., spine switches, fabric switches and TOR switches), except that the number of fabric switches are increased to one less than the number of TOR switches (FIG. 8cshows equal number of fabric switches and TOR switches because one of the TOR switch, the 21stTOR switch, is removed so that the 20 interfaces connected to it from the 20 fabric switches are provided as uplink to connect to external network).

FIG. 8ashows the architecture of an improved data center network, organized by three layers of switching devices—i.e., “top-of-rack” (TOR) switches and fabric switches implemented in 188 server pods81-0to81-187and spine switches implemented in 20 spine planes80-0to80-19—interconnected by interlinks in a fat tree topology. An interlink refers to the network connections between a server pod and a spine plane. For example, interlink k of each of the 188 server pods is connected to spine plane k; interlink p of each of the 20 spine planes is connected to server pod p. The 20 spine planes each provide an optional uplink (e.g. uplink801) and the 188 server pods each provide an optional uplink (e.g., uplink802) for connection to one or more external networks. In this example, to allow comparison, the numbers of server pods and spine plane are chosen so that the improved data center network800and the state-of-the-art data center network200have the same network characteristics (2.2 Pbps total server-side bandwidth; 3:1 oversubscription ratio—server-side to network-side bandwidth ratio; Trident-II ASIC). Other configurations of the improved data center network are also possible, for instance, 32-TOR server pod or 48-TOR server pod but with higher radix switching silicon than the Trident-II ASIC.

Details of a spine plane ofFIG. 8aare shown inFIG. 8b. InFIG. 8b, spine plane820consists of 20 spine switches82-0to82-19each connecting to 188 server pods. The connections from all 20 spine switches are grouped into 188 interlinks, with each interlink including a connection from each spine switch82-0to82-19, for a total of 20 connections per interlink.

Details of a server pod ofFIG. 8aare shown inFIG. 8c. InFIG. 8c, the network-side connection (as opposed to the server-side connection) of the server pod is separated into intra-pod links and inter-pod links (i.e., the interlinks). The two types of links are made independent from each other. The intra-pod region832consists of the intra-pod links, the 20 TOR switches84-0to84-19and the 20 fabric switches83-0to83-19interconnected by the intra-pod links in a fat tree topology. For example, connection k in each of the 20 TOR switches is connected to fabric switch k; connection p of each of the 20 fabric switches is connected to TOR switch p. 20 fabric switches each provide an optional uplink (e.g., uplink831) to connect to an external network. The inter-pod region consists of the inter-pod links (i.e., the interlinks) and 20 TOR switches84-0to84-19on the interlink side. Each interlink provides 20 10G connections to connect to all 20 spine switches on the same spine plane. Each server pod includes a total of 20 links. For example, interlink k of each of the 188 TOR switches across the 188 server pods are connected to spine plane k; interlink p of each of the 20 spine planes are connected to server pod p. Each TOR switch provides 48×10G connections in 12×QSFP ports as downlink to connect to servers.

The data traffic through the fabric switches is primarily limited to intra-pod. The TOR switches now route both the intra-pod traffic as well as inter-pod traffic and are more complex. The independent link types achieve massive scalability in data center network implementations. (Additional independent links provided from higher radix switching ASIC may be created to achieve larger scale of connectivity objectives). Additionally, data center network800incorporates the full mesh topology concept (without physically incorporating an ICAS module) to remove redundant network devices and allow the use of innovative switching methods, in order to achieve a “lean and mean” data center fabric with improved data traffic characteristics.

As shown inFIG. 8c,FIG. 8bandFIG. 8a, data center network800includes 20×188 TOR switches and 20×188 fabric switches equally distributed over 188 server pods, and 20×20 spine switches equally distributed over 20 spine planes. InFIG. 8a, each TOR switch has 100 10G-connections (i.e., 25 QSFPs of bandwidth in 10G mode), of which 60 10G-connections are provided server-side and 40 10G-connections are provided network-side. (Among the network-side connections 20 10G-connections are used for intra-pod traffic and 20 10G-connections are used for inter-pod traffic). In each server pod, fabric switches83-0to83-19each include 21 10G-connections, of which 20 10G-connections are allocated to connect with a 10G-connection in each of TOR switches84-0to84-19, and the rest being converted to provide as uplink to connect to external network. In this manner, fabric switches83-0to83-19support the intra-pod region data traffic and the uplinks in the server pod by a 21-node full mesh topology (with the uplinks of fabrics switches0-19collectively seen as one node). Using a suitable routing algorithm, such as any of those described above in conjunction with Single-Source-Multiple-Destination Traffic Aggregation and Port-to-Port Traffic Aggregation, network congestion can be eliminated from all fabric switches.

As the network in the intra-pod region of each server pod can operate in the same connectivity characteristics as a full mesh topology network, all the 20 fabric switches of the server pod may be replaced by an ICAS module. ICAS-based data center network900, resulting from substituting fabric switches83-0to83-19of data center network800, is shown inFIG. 9a. To distinguish from the server pod of data center network800, a server pod with its fabric switches replaced by an ICAS module is referred to as an “ICAS pod.”

FIG. 9ashows the architecture of an ICAS-based data center network, organized by three layers of devices—i.e., “top-of-rack” (TOR) switches, ICAS module implemented in 188 server pods91-0to91-187and spine switches implemented in 20 spine planes for90-0to90-19—interconnected by interlinks in a fat tree topology. 20 spine planes provide optional uplinks901and 188 ICAS pods provide optional 188×20×10G uplinks902for connecting to an external network. The number of network devices in the data center network should be interpreted as illustrative only.

Details of a spine plane ofFIG. 9aare shown inFIG. 9baccording to one embodiment. InFIG. 9b, spine plane920includes 20 spine switches92-0to92-19and a fanout cable transpose rack921. The fanout cable transpose rack contains: k first port groups923are connected to corresponding port groups of k spine switches through a plurality of first MPO-MPO fiber cables, where each first port group including ┌p/m┌ first MPO adapters, and each first MPO adapter including m interfaces (where each interface includes one transmit fiber channel and one receive fiber channel), and a plurality of first MPO fiber adapters from the k port groups923are connect to LC optical fiber adapter mounting panel922through a plurality of first MPO-LC fanout fiber cables, where k=20, p=188, m=4, and ┌ ┌ is a ceiling function; the fanout cable transpose rack921includes p second port groups924that are connected to a plurality of second MPO-MPO fiber cables to form interlinks99-0to99-187, each second port group contains ┌k/m┌ second MPO fiber adapters, each of which includes m interfaces (where each interface includes one transmit fiber channel and one receive fiber channel), and a plurality of second MPO fiber adapters from the p port groups924are connected to LC optical fiber adapter mounting panel922through a plurality of the second MPO-LC fanout cables; a plurality of first MPO-LC fanout fiber cables cross-connect a plurality of second MPO-LC fanout fiber cables on the LC fiber adapter mounting panel922, through cross-connection, all connections from k spine switches92-0to92-19are reorganized into p interlinks99-0to99-187, each interlink includes one connection from each of the spine switches92-0to92-19, each interlink contains k connections in total.

That is, on one side of the fanout cable transpose rack921is k first port groups923, each first port group has ┌p/m┌ of first MPO adapters, where ┌ ┌ is a ceiling function, each port groups connects to a corresponding port group of a spine switch through the ┌p/m┌ first MPO-MPO cables. On the other side of the fanout cable transpose rack921is p second port groups924, each second port group has ┌k/m┌ of second MPO adapters, where ┌ ┌ is an ceiling function, each port group connects to 5 second MPO-MPO cables to form an interlink to the ICAS pod.

As pointed out earlier in this detailed description, the state-of-the-art data centers and switch silicon are designed with 4 interfaces (TX, RX) at 10 Gb/s or 25 Gb/s each per port in mind. Switching devices are interconnected at the connection level in ICAS-based data center. In such a configuration, a QSFP cable coming out from a QSFP transceiver is separated into 4 interfaces, and 4 interfaces from different QSFP transceivers are combined in a QSFP cable for connecting to another QSFP transceiver. Also, a spine plane may interconnect a large and varying number of ICAS pods (e.g., in the hundreds) because of the scalability of an ICAS-based data center network. Such a cabling scheme is more suitable to be organized in a fanout cable transpose rack (e.g., fanout cable transpose rack921), which may be one or multiple racks and be integrated into the spine planes. Specifically, the spine switches and the TOR switches may each connect to the fanout cable transpose rack with QSFP straight cables. Such an arrangement simplifies the cabling in a data center.FIG. 9billustrates such an arrangement for data center network900ofFIG. 9a.

In the embodiment shown inFIG. 9b, the first and the second optical fiber adapters are MPO adapters, the first and the second cables are MPO-MPO cables, the first and the second fanout cables are MPO-LC fanout cables, the mounting panel is LC optical fiber adapter mounting panel. One skilled in the art would understand that different types of optical fiber adapters/cable/optical fiber adapter mounting panel may also be used, such as FC, SC, LC, and MU.

Details of an ICAS pod ofFIG. 9aare shown inFIG. 9c. InFIG. 9c, the network side interface (as opposed to the server-side interface) of an ICAS pod is divided into intra-pod links (i.e. intralinks) and inter-pod links (i.e., interlinks) and the two types of links are made independent from each other. The intra-pod region consists of intralinks between the 20 TOR switches93-0to93-1019and ICAS module931, interconnected by 10G connections in a full mesh topology. Each ICAS module may provide 20 10G uplinks932to connect to one or more external networks. The inter-pod region consists of interlinks. ICAS pod may comprise 20 TOR switches93-0to93-19each connects one of 20 spine planes by interlinks, respectively; each interline comprising 20 connections each connects one of 20 spine switches in a spine plane, respectively. For example, interlink k of each of 188 TOR switches across the 188 ICAS pods is connected to spine plane k; interlink p of each of the 20 spine planes is connected to server pod p. Each TOR switch provides 60×10G interfaces in 15×QSFP ports as a downlink for connecting to servers.

The data traffic through the ICAS module is primarily limited to intra-pod. The TOR switches now perform routing for the intra-pod traffic as well as inter-pod traffic and are more complex. The independent link types achieve massive scalability in data center network implementations. (Additional independent link provided from higher radix switching ASIC may be created to achieve a larger scale of connectivity objectives).

As shown inFIG. 9c,FIG. 9bandFIG. 9a, each TOR switch allocates 20×10G-interfaces (5×QSFPs in 10G mode) to connect to its associated ICAS module (e.g., ICAS module931) to support intra-pod traffic, and 5 QSFPs in 10G mode (20 10G-interfaces) to connect to the fiber transpose rack to support inter-pod traffic. As shown inFIG. 9c, each ICAS pod includes 20×5 QSFP transceivers for intra-pod traffic, connected by 100 QSFP straight cables, and 20×15 QSFP (10G mode) transceivers for server traffic, for a total 500 QSFP transceivers. The 20 TOR switches in an ICAS pod may be implemented by 20 Trident II ASICs. Although 20 TOR switches are shown in each ICAS pod inFIG. 9c, the ICAS module is scalable to connect up to 48 TOR switches in an ICAS pod (based on 32×QSFP Trident-II+ switch ASIC).

Together, the ICAS pods and the spine planes form a modular network topology capable of accommodating hundreds of thousands of 10G-connected servers, scaling to multi-petabit bisection bandwidth, and covering a data center with congestion improved and non-oversubscribed rack-to-rack performance.

According to one embodiment of the present invention, a spine switch can be implemented using a high-radix (e.g., 240×10G) single chip switching device, as shown inFIG. 9d. Single-chip implementation saves the cost of extra transceivers, cables, rack space, latency and power consumption than multi-unit (rack unit) chassis-based switching device and stackable switching device implementations. The disadvantage of the single-chip spine switch approach is its network scalability, which limits the system to 240 ICAS pods at this time. As mentioned above, the semiconductor implementation limits the scale of a high-radix switching integrated circuit.

To overcome the limitation on the port count of the silicon chip, one or more 1 U to multi-U rackmount chassis each packaged with one or more ICAS modules, and a plurality of 1 U rackmount chassis each packaged with one or more switching devices, can be stacked up in one or more racks, interconnected, to form a higher-radix (i.e. high network port count) stackable spine switching device (e.g., ICAS-based stackable switching device). Each ICAS module is connected to the plurality of switching devices, such that the ICAS module interconnects at least some interfaces of at least some port groups of different switching devices to form a full mesh non-blocking interconnection. The interfaces of the rest of the at least some port groups for interconnecting different switching devices are configured as an uplink. When the ICAS-module-based 1 U to multi-U rackmount chassis are optically implemented (based on optical fiber and 3D MEMS), MPO-MPO cables may be used to connect the ICAS-based interconnection devices and the switching devices. When the ICAS-module-based 1 U to multi-U rackmount chassis are electrically implemented as circuits (based on PCB+chip), DAC direct connection cables or AOC active optical cables may be used to connect the ICAS-based interconnection devices and the switching devices.

Details of an ICAS-based stackable switching device950are shown inFIG. 9e.FIG. 9eshows ICAS modules95-0to95-3each connected in a full mesh topology to switches96-0to96-3. In one embodiment, 4 Trident-II ASIC-based switches96-0to96-3, each having a switching bandwidth of 24 QSFPs in 10G mode provided in 1:1 subscription ratio, and an ICAS box953integrating 4 ICAS modules95-0to95-3in one 1 U chassis and each ICAS module containing 3 duplicate copies of ICAS1X5 sub-modules and each sub-module providing 4×10G of uplink951may be used to builds a stackable spine switch, as shown inFIG. 9e. The 4 switches96-0to96-3provide ports952of 1.92 Tbps of bandwidth to connect to servers. The ICAS-based stackable switching device950provides total uplink bandwidth of 480 Gb/s (4×3×40 Gb/s) to connect to external network, facilitates non-blocking 1:1 subscription ratio and provides full mesh non-blocking interconnect with a total of 1.92 Tbps of switching bandwidth.

ICAS-based stackable switching device has the benefits of improved network congestion, saving the costs, power consumption and space savings than the switching devices implemented in the state of the art data center. As shown in the “ICAS+Stackable Chassis” column of Table 5, data center with ICAS and ICAS-based stackable switching device performs remarkably on data center network with total switching ASIC saving by 53.5%, total power consumption saving by 26.0%, total space saving by 25.6% and much improved network congestion. However total QSFP transceiver usage is increased by 2.3%.

The above stackable switching device is for illustrative purpose. A person experienced in the art can easily expand the scalability of the stackable switching device and should not be limited as in the illustration.

The stackable switching device addresses the insufficiency in the number of ports of network switching chip, thus making possible a flexible network configuration. However, a considerable number of connecting cables and conversion modules have to be used to interconnect the ICAS-based interconnection devices and the switching devices. To further reduce the use of cables and conversion modules, ICAS modules and switch chips can be electronically interconnected using a PCB and connectors, which is exactly how the multi-unit switching device is structured. Specifically, the ICAS module of the ICAS-based multi-unit switching device is electrically implemented as circuits, and the port groups of the ICAS module are soldered or crimped onto a PCB using connectors that support high-speed differential signals and impedance matching. The interconnection between the internal port groups is realized using a copper differential pair on the PCB. Since signal losses vary significantly between different grades of high-speed differential connectors and between copper differential pairs on different grades of PCBs, an active chip can be added at the back end of the connector to restore and enhance the signal to increase the signal transmission distance on the PCB. The ICAS module of the ICAS-based multi-unit switching device may be implemented on a PCB called a fabric card, or on a PCB called a backplane. The copper differential pair on the PCB interconnects the high-speed differential connector on the PCB to form a full mesh connectivity in the ICAS architecture. The switch chips and related circuits are soldered onto a PCB called a line card, which is equipped with a high-speed differential connector docking to the adapter on the fabric card. A multi-U chassis of the ICAS-based multi-unit switching device includes a plurality of ICAS fabric cards, a plurality of line cards, and one or two MCU- or CPU-based control cards, one or more power modules and cooling fan modules. “Rack unit” (“RU” or “U” for short) measures the height of a data center chassis, equal to 1.75 inches. A complete rack is 48 U (48rack units) in height.

One embodiment of the present invention also provides a chassis-based multi-unit (rack unit) switching device. A multi-unit chassis switching device groups multiple switch ICs onto multiple line cards. Chassis-based multi-unit switching equipment interconnects with line cards, control cards, and CPU cards via PCB-based network cards or backplanes, which saves the cost of transceivers, fiber optic cable and rack space required for interconnection.

Details of an ICAS-based multi-unit chassis switching device970are shown inFIG. 9f.FIG. 9fshows 4 ICAS-based fabric cards97-0to97-3interconnected in a full mesh topology to switching ASIC's98-0to98-3. Switching ASIC98-0and98-1are housed in line card973, and switching ASIC's98-2and98-3are housed in line card974. Line cards973and974are connected through high speed PCB (printed circuit board) connectors to fabric cards97-0to97-3. In one embodiment, 4 Trident-II ASIC-based switches98-0to98-3, each having a switching bandwidth of24QSFPs in 10G mode provided in 1:1 subscription ratio, and 4 ICAS-based fabric cards97-0to97-3containing 3 duplicate copies of ICAS1X5 sub-modules and each sub-module providing 4×10G of uplink971may be used to builds a multi-unit chassis switch, as shown inFIG. 9f. Two line cards provide data ports972of total 1.92 Tbps of bandwidth to connect to servers. ICAS-based multi-unit chassis switching device970provides total uplink bandwidth of 480 Gb/s (4×3×40 Gb/s) to connect to external network, facilitates full mesh non-blocking 1:1 subscription ratio interconnect with a total of 1.92 Tbps of switching bandwidth.

Multi-unit chassis-based switching device with fabric cards that are ICAS-based full mesh topology has the benefits of improved network congestion, saving the costs and power consumption than that of ASIC-based fabric cards implementation with fat tree topology. As shown in the “ICAS+Multi-unit Chassis” column of Table 5, data center with ICAS and ICAS-based multi-unit chassis-based switching device performs remarkably on data center network with total QSFP transceiver saving by 12.6%, total switching ASIC saving by 53.5%, total power consumption saving by 32.7%, total space saving by 29.95% and much improved network congestion.

The above multi-unit chassis switching device is for illustrative purpose. A person experienced in the art can easily expand the scalability of the multi-unit chassis switching device and should not be limited as in the illustration.

The multi-unit chassis-based switching device has the disadvantage of a much longer development time and a higher cost to manufacture due to its system complexity, and is also limited overall by the form factor of the multi-unit chassis. The multi-unit chassis-based switching device, though provides a much larger port count than the single-chip switching device. Although the stackable switching device requires additional transceivers and cables than that of the multi-unit chassis-based approach, the stackable switching device approach has the advantage of greater manageability in the internal network interconnection, virtually unlimited scalability, and requires significantly less time for assembling a much larger switching device.

The material required for (i) the data center networks ofFIG. 2a, using state of the art multi-unit switching device (“Fat tree+Multi-unit Chassis”), (ii) an implementation of data center network900ofFIG. 9a, using ICAS-based multi-unit switching device “ICAS+Multi-unit Chassis”, and (iii) an implementation of data center network900ofFIG. 9a, using ICAS-based stackable switching device “ICAS+Stackable Chassis” are summarized and compared in Table 5.

As shown in Table 5, the ICAS-based systems require significantly less power dissipation, ASICs and space, resulting in reduced material costs and energy.

The above detailed description is provided to illustrate specific embodiments of the present invention and is not intended to be limiting. Numerous modifications and variations within the scope of the present invention are possible. The present invention is set forth in the accompanying claims.