Distributed link aggregation

Systems and methods to forward data frames are described. A particular method may include generating a plurality of management frames at a controlling bridge. The management frames may include routing information. The plurality of management frames may be communicated to a plurality of bridge elements coupled to a plurality of server computers. The plurality of bridge elements are each configured to selectively forward a plurality of data frames according to the routing information.

I. FIELD OF THE DISCLOSURE

The present disclosure relates generally to data communications, and more specifically, to routing data frames within a highly integrated computer network.

Server computers are continuously managed to enable access to shared switches and other traffic routing resources. For example, contention for routing resources may exist when server computers are housed within racks for space and connectivity considerations, such as in a blade server computer arrangement. The server computers may experience transmission bottlenecks and delays when forwarding data frames through centralized switches, such as shared top of rack switches.

To illustrate,FIG. 1shows a conventional blade server computer system100. The system100includes a plurality of server computers106-125housed within racks102,104and arranged into chassis138,140,142, and144. An illustrative server computer106may include a half-width information technology element (ITE) blade server computer.

Data frame communications between the server computers106-125housed within different chassis138,140,142,144or racks102,104may be referred to as east-west connectivity. For example, the server computer111of a first chassis140may forward a data frame to the server computer106of another chassis138via a path164. The path164includes a chassis switch154and a top of rack switch (TOR)158. The chassis switch154and the top of rack switch158may route the data frame based upon a media access control (MAC) address.

When the server computer111of the rack102forwards a data frame to the server computer123of the rack104, the data frame travels through paths166and168. The paths166and168include the top of rack switch158associated with the rack102, an end of rack switch (EOR)162, and a top of rack switch160associated with the rack104. The top of rack switch158is again used when the server computer111attempts north-south connectivity (i.e., internal to external data frame communication) through paths166and170. Because the data frames in the above examples are all routed through the top of rack switches158,160, a potential bottleneck scenario can result.

Increasing the number of switches and associated connections to accommodate additional traffic may present configuration and management challenges, as well as increase hardware costs and latency. For example, an increased number of switches may burden a centralized switch used to control workload balance and data frame distribution.

III. SUMMARY OF THE DISCLOSURE

In a particular embodiment, an apparatus is disclosed that includes a plurality of server computers. A plurality of bridge elements may be coupled to the plurality of server computers. The plurality of bridge elements may each be configured to selectively forward a plurality of data frames according to received routing information. A controlling bridge coupled to the plurality of bridge elements may be configured to provide the routing information to the plurality of bridge elements.

In another embodiment, a method of routing a data frame may include generating a plurality of management frames at a controlling bridge. The management frames may include routing information. The plurality of management frames may be communicated to a plurality of bridge elements coupled to a plurality of server computers. The plurality of bridge elements are each configured to selectively forward a plurality of data frames according to the routing information.

In another embodiment, a program product may include program code executable at a controlling bridge to generate a plurality of management frames that include routing information. The plurality of management frames may be communicated to a plurality of bridge elements coupled to a plurality of server computers. The plurality of bridge elements may each be configured to selectively forward a plurality of data frames according to the routing information. The program product may further include a computer readable storage medium bearing the program code.

At least one of the embodiments may facilitate workload balancing by distributing routing and link aggregation processes. The distributed routing may reduce potential bottlenecks and facilitate efficient processing. An embodiment of a system may be scalable to include hundreds or more server computers with direct connectivity.

Features that characterize embodiments of the disclosure are set forth in the claims annexed hereto and forming a further part hereof. However, for a better understanding of embodiments of the disclosure, and of the advantages and objectives attained through its use, reference should be made to the Drawings and to the accompanying descriptive matter in which there are described exemplary embodiments of the disclosure.

V. DETAILED DESCRIPTION

Data frame communication may be improved by including a distributed aggregator component comprising multiple bridge elements positioned within a highly integrated and scalable network. The distributed bridge elements may comprise part of a distributed virtual bridge that spans multiple server computers. The bridge elements may be configured to perform Layer-2 switching functions. The bridge elements may accomplish operational and frame-forwarding decisions in parallel by distributing load balancing determinations.

Groupings of ports may be controlled and managed using the distributed virtual bridge. The port groups may span multiple, physically separated server computers to provide external bandwidth. Port group sub-link selection may be performed at the bridge elements. Port group sub-links may use the same external links. Port group and link operational status may be automatically updated within the distributed environment. The link aggregation of port groups may be distributed using multiple instances of bridge elements. A bridge element may include a link table with primary and secondary port group trunks.

A controlling bridge may control access to external links and may execute Layer-2 (e.g., Ethernet) control plane functions to manage a set of bridge elements. For example, the bridge elements of the distributed virtual bridge may be interconnected and managed using management frames generated by the controlling bridge. The controlling bridge may communicate with or otherwise be coupled (remotely or directly) to each bridge element of the distributed virtual bridge. The controlling bridge may program the bridge elements using management frames that include a configuration protocol. Link aggregation setup, configuration, and operational control may be accomplished by the controlling bridge.

The controlling bridge may communicate routing information to bridge elements using sub-links in a specific port group. The routing information may include a routing algorithm that enables the bridge elements to execute link and port aggregation. An illustrative routing algorithm may include a workload balancing component. The controlling bridge may further monitor link states of each sub-link of an aggregated port group. The controlling bridge may inform some or all of the bridge elements using sub-links in a specific port group of any changes affecting the state of a link.

Selection of a sub-link may be based upon a link state, a load balancing consideration, and address data of a frame to be transmitted, among other considerations. Illustrative address data may include at least one of a Media Access Control (MAC) address, a source port identifier, a source address, a destination address, and an Internet Protocol (IP) address, among other identifying data.

The controlling bridge may exchange protocol messages with one or more bridge elements and may form one or more of the port groups. The primary controlling bridge may communicate configuration and sub-link routing information to each bridge element that will be forwarding data frames to a specific port group. For instance, the primary controlling bridge may communicate address data and one or more routing algorithms to a bridge element.

After the primary controlling bridge has communicated the routing information to the bridge element, the bridge element may process and forward each received data frame directly to the appropriate external physical uplink port. Each data frame may not travel through a common aggregator to be serially processed. The routing of each data frame may be accomplished at the bridge elements in parallel. This parallel processing may facilitate increased throughput.

An aggregated link failure may be signaled by a controlling bridge to an ingress, or north, bridge element. The north bridge element may be coupled to an adapter. A link state change may be detected by a south bridge element. A south bridge element may be coupled to an external uplink (e.g., to an Ethernet network). The south bridge element may transmit a notification to a north bridge element for each discarded data frame. A link-up notification or link-down notification may be communicated to a local controlling bridge from the south bridge element. The controlling bridge may broadcast the link status to some or all of the bridge elements. The bridge elements may update their link table according to the notifications. The bridge elements may subsequently account for the link status during link selection. In the case of a detected faulty primary port group, the system may fail over to a secondary port group. In this manner, the bridge elements and controlling bridge(s) may work in combination to increase routing performance and reduced administrative workload.

Turning particularly to the drawings,FIG. 2shows an illustrative embodiment of a highly integrated system200configured to forward data frames using a distributed virtual bridge260. The distributed virtual bridge260may extend across server computers206-225, chassis246,248,250,252, and racks202,204to provide data link layer (e.g., Layer2) switching between bridge elements. The bridge elements may provide a frame-based, Ethernet-like interface. The interface may facilitate lossless, point-to-point, in-order frame delivery between server computers206-225of different racks202,204or chassis246,248,250,252(i.e., east-west connectivity) with reduced redundancy and latency.

The system200further includes an end-of-rack switch270and input/output (I/O) server ITEs258,261that enable north-south connectivity. The I/O server ITEs258,261may enable uplink connectivity to the external Ethernet network (or other network) for the server computers206-225housed within the racks202,204.

An arrow264ofFIG. 2represents direct east-west connectivity and the flow of data frames between server computers located in different racks202,204of the system200(e.g., without using a top of rack or chassis switch). An arrow262represents direct east-west connectivity across different chassis246,248of the rack202.

The system200ofFIG. 2may enable direct connectivity between server computers of different racks or chassis. To accommodate the relatively high level of system integration, distributed bridge elements may be programmed to independently route data frames. The distribution of routing processes may streamline the routing of data frames and facilitate scalability. The bridge elements and distributed routing within the distributed virtual bridge260may reduce contention for resources and increase data frame traffic flow.

FIG. 3shows a particular illustrative embodiment of a highly integrated computer system300that includes north bridge elements308,310, and312configured to route data frames in a distributed manner. The system300includes server computers302,304, and306. The server computers302,304, and306may be similar to the206-208ofFIG. 2. The server computer302may be coupled to the north bridge element308. The server computer304may be coupled to the north bridge element310, and the server computer306may be coupled to the north bridge element312.

The north bridge element308may be coupled to a local rack interconnect314. The other ingress (i.e., north) bridge elements310,312may additionally be coupled to the local rack interconnect314. The local rack interconnect314may further be coupled to south bridge elements316,318,320, and322. The local rack interconnect314may facilitate point-to-point connections between the bridge elements308,310,312,316,318,320, and322without frame loss and with in-order frame delivery.

A primary controlling bridge324may be coupled to the local rack interconnect314. A secondary controlling bridge326may additionally be coupled to the local rack interconnect314. The south bridge element316may be coupled to an external switch328, which is coupled to an external server375. More particularly, a link315may couple the south bridge element316to a port331of the external switch328. The south bridge element316may be coupled to a port333of the external switch328via a link317.

The south bridge element318may be coupled to a port335of the external switch328via a link319. A link321may connect the south bridge element318to a port337of the external switch328. The south bridge element320may be coupled to an external switch330at ports339and341via links323and325, respectively. The south bridge element322may be coupled to the external switch330at ports343and345via links327and329, respectively. The ports331,333,335, and337may be associated with a first port group371. The ports339,341,343, and345may be associated with a second port group373. The external switch330may be coupled to an external server377.

The north bridge element308may include routing information332. The routing information may be communicated by the primary controlling bridge324and may include address data334and a routing algorithm336. The routing algorithm336may include instructions used to route data frames to be transmitted from the north bridge element308.

The address data334may be associated with the north bridge element308by the primary controlling bridge324. The address data334may include controlling bridge addresses338and a link table340. Illustrative controlling bridge addresses338may correspond to MAC addresses of the primary controlling bridge324and of the secondary controlling bridge326. The link table340of the north bridge element308may include port group information342and MAC addresses344. The port group information342may include information pertaining to the first port group371assigned or otherwise associated with the north bridge element308. The MAC addresses344may include addresses of south bridge elements316,318,320, and322, among other components of the system300.

The address data334of the north bridge element308may further include link state information346. The link state information346may include status information pertaining to various links and ports that are associated with the north bridge element308. The address data334may also include virtual large area network (VLAN) and logical network (LN) assignments348associated with the north bridge element308. For example, the VLAN and logical network assignments348may be used by the north bridge element308to route data frames. Through the north bridge element308, VLANs and logical networks may be further associated with port groups371,372.

The north bridge element310may include routing information350that includes address data352and a routing algorithm354. The address data352may be associated with the north bridge element310by the primary controlling bridge324. The address data352may include controlling bridge addresses356and a link table358. The link table358may include port group information360and MAC addresses362corresponding to links, nodes, and port locations. The address data352may further include link state information364, as well as VLAN and logical network assignments366.

The north bridge element312may include address data370and a routing algorithm372. The address data370may include controlling bridge addresses374and a link table376. The link table376may include port group information378and MAC addresses380. The address data370may further include link state information382. VLAN and logical network assignments384may be associated with the north bridge element312.

The primary controlling bridge324may include link and node data386. The link and node data386may include MAC addresses of ports and/or links to be associated with the north bridge elements308,310,312for routing data frames. The primary controlling bridge324may also include at least one associating algorithm388. The associating algorithm388may be used to automatically assign the address data334,352,370and the routing algorithm(s)336,354,372to the north bridge elements308,310,312.

The secondary controlling bridge326may include link and node data390, as well as an associating algorithm(s)392. As with the primary controlling bridge324, the secondary controlling bridge326may be configured to provide one or more of the north bridge elements308,310,312with routing information332,350,368, including address data334,352,370and routing algorithms336,354,372.

The controlling bridges324,326may exchange protocol messages with one or more of the north bridge elements308,310,312and may create and program one or more of the port groups371,372. For example, the primary controlling bridge324may create the first port group371and may exchange Link Aggregation Control Protocol (LACP) messages with the south bridge elements316,318,320, and322. The primary controlling bridge324may further create the second port group372. The port groups371,372may be associated with bridge elements, VLANs, and/or logical networks, per the routing information. For example, the first port group371may be a primary port group for first and second VLANs, and a secondary, or backup, port group for a third VLAN.

In operation, the primary controlling bridge324may communicate the routing information332to the north bridge element308. The north bridge element308may process and forward each received data frame directly to an appropriate external physical uplink port. For example, the north bridge element308may be configured to forward data frames to the south bridge element316, as indicated by dashed line303, and to the south bridge element318, as indicated by dashed line305.

The north bridge element310may be configured to forward data frames to the south bridge element316, as indicated by dashed line307. The north bridge element310may further be configured to communicate data frames to the south bridge element318and the south bridge element320, as indicated by dashed lines309and311, respectively. The north bridge element312may be configured to forward data frames to the south bridge element322, as indicated by dashed line313. The north bridge element308may thus be configured to forward data frames to the same south bridge element318as the north bridge element310.

The system300may be configured to automatically facilitate failover operations. For example, the south bridge element318may detect a fault, or failure, associated with the link319to the external switch328. The link failure may include a fault associated with the physical link319or the port335. The south bridge element318may inform the primary controlling bridge324that the link319is faulty. The link319is associated with the port group371. The primary controlling bridge324may inform the north bridge elements308and310that use sub-links in the affected first port group371about the failure of the link319. The south bridge element318may optionally notify the north bridge element308that its forwarding attempt was unsuccessful. The north bridge elements308and310may refrain from using the link319during subsequent link selection.

In the case of a faulty primary port group371, the system300may failover to a secondary port group372. For example, a controlling bridge may be configured to take down a port group at a specified operating threshold (e.g., if less than two links are active). The primary controlling bridge324may notify all bridge elements308,310,316,318associated with the deactivated port group371about the external link failure, as well as about the complete port group failure. The secondary port group372may become the uplink for reassigned VLANs.

FIG. 3thus shows a system300having distributed bridge elements308,310,312to independently determine data frame routing. Routing information332,350,368used to make the independent routing determinations may be provided by a controlling bridge324,326. Each data frame may not travel through a common aggregator to be serially processed. The routing and failover processes associated with each data frame may be accomplished at the distributed north bridge elements308,310, and312in parallel. This parallel processing may facilitate increased throughput.

Referring toFIG. 4, another particular illustrative embodiment of a highly integrated system400configured to route data frames using distributed bridge elements is depicted.FIG. 4generally shows a computer system400configured to forward data frames using a distributed virtual bridge408. The distributed virtual bridge408may selectively forward management frames to distributed switches (e.g., bridge elements and adapters).

The distributed virtual bridge408may be similar to the distributed virtual bridge260ofFIG. 2. The system400includes a first server computer402and a second server computer404that are both coupled to an I/O blade device406via the distributed virtual bridge408. The server computers402,404and the I/O blade device406may be housed within separate chassis and racks. For example, the server computers402,404and the I/O blade device406may correspond respectively to the server computers210,220and the I/O ITE261ofFIG. 2

The distributed virtual bridge408may be coupled to multiple adapters410,412,414,416,418,420,422, and424. The adapters410,412,414,416,418,420,422, and424may be located within or may be coupled to the server computers402,404. The distributed virtual bridge408may use multiple access points, or bridge elements426,428,430, and432-440to couple to the server computers402,404. For example, a microchip that includes the bridge elements426,428,430, and432may be cabled or otherwise coupled to a port of the server computer402that includes the adapter410. As explained herein, the distributed virtual bridge408may functionally supplant chassis switches and top of rack switches with a frame-based network fabric that functions in a similar fashion to an Ethernet network.

One or more transport layer modules482,484,486, and488coupled to the bridge elements426,428,430, and432may provide a frame-based, Ethernet-like interface to one or more integrated switch routers442. The transport layer module482may be configured to deconstruct a transmission of data frames so that packet information may be evenly distributed across links to a local rack interconnect490. The data frames may not be serialized upon leaving the transport layer module482. A receiving transport layer module423may serialize the data frames to achieve reliable, in-order delivery. If the receiving transport layer module423determines that data frame information is missing, the transport layer module423may initiate a process to recover the missing data. The translation process may be accomplished in hardware, which may provide a larger bandwidth and faster processing than software applications. The transport layer modules482,484,486, and488, the integrated switch router442, and the local rack interconnect network490may combine to include an underlying lossless, point-to-point communication network (i.e., an integrated switch router network) between the server computers402,404and the I/O blade device406.

The bridge elements426,428,430, and432may function as data link layer (i.e., Layer2) bridge forwarders within the distributed virtual bridge408. In particular embodiments, the bridge elements426,428,430, and432may comprise a switch, or router device. The bridge elements426,428,430, and432may include learned (e.g., received and stored) cached address data used to forward data frames throughout the distributed virtual bridge408. The learned address data may correspond to one or both of a destination address and a source address associated with a data frame.

When the bridge element426does not include address data pertinent to a source or destination address of a received data frame, the bridge element426may query a controlling bridge448for the address data. The controlling bridge448may include a global forwarding table411that includes stored address data. The stored address data may be continuously updated by the bridge elements426,428,430, and432. For example, a bridge element426may send an update message to the controlling bridge448in response to learning an updated or new MAC address. A corresponding MAC address in the global forwarding table411may be subsequently updated.

Conversely, the address data of the global forwarding table411may be used to update the bridge elements426,428,430, and432. For example, the controlling bridge448may respond to a query from the bridge element426with requested address data. The bridge element426may cache the received address data for future use (e.g., at the forwarding cache474).

The first server computer402may comprise a blade server computer, such as the server computer206shown inFIG. 2. The first server computer402may include one or more virtual machines (VMs)450,452,454,456,458, and460. A virtual machine may include a software implementation of a computer and may execute programs in a manner similar to a physical machine.

FIG. 4shows an illustrative hypervisor462that is coupled to both the virtual machine450and the virtual machine452. The hypervisor462may include platform virtualization software that allows multiple operating systems to run concurrently on the first server computer402. The hypervisor462may include a hypervisor virtual bridge464that allows direct communication between the virtual machines450,452without traversal of an external network. In one embodiment, the hypervisor virtual bridge464may register address information with the controlling bridge448.

The first server computer402may include at least one processor403coupled to a memory405. The processor403may represent one or more processors (e.g., microprocessors), and the memory405may represent random access memory (RAM) devices comprising the main storage of the server computer402, as well as supplemental levels of memory, e.g., cache memories, non-volatile or backup memories (e.g., programmable or flash memories), read-only memories, etc. In addition, the memory405may be considered to include memory storage physically located in the first server computer402or on another server computer coupled to the server computer402via the distributed virtual bridge408(e.g., the second server computer404).

The first server computer402may operate under the control of an operating system (OS)407and may execute or otherwise rely upon various computer software applications, components, programs, objects, modules, and data structures, such as the virtual machines450,452,454,456,458, and460. Moreover, various applications, components, programs, objects, modules, etc. may also execute on one or more processors in another device coupled to the server computer402(e.g., in a distributed computing environment, where computing processes may be allocated to multiple server computers).

The first server computer402may include adapters410,412,414, and416, such as converged network adapters. A converged network adapter may include a single root I/O virtualization (SR-My) adapter, such as a Peripheral Component Interconnect Express (PCIe) adapter that supports Converged Enhanced Ethernet (CEE). Another embodiment of the system400may include a multi-root I/O virtualization (MR-IOV) adapter. The adapters410,412,414, and416may be used to implement a Fiber Channel over Ethernet (FCoE) protocol. Each adapter410,412,414, and416may be coupled to one or more of the virtual machines450,452,454,456,458, and460. The adapters410,412,414, and416may facilitate shared access of the virtual machines450,452,454,456,458, and460. While the adapters410,412,414, and416are shown inFIG. 4as being included within the first server computer402, adapters of another embodiment may include physically distinct devices that are separate from the server computers402,404.

Each adapter410,412,414, and416may include a converged adapter virtual bridge466,468,470, and472. The converged adapter virtual bridges466,468,470, and472may facilitate sharing of the adapters410,412,414, and416by coordinating access by the virtual machines450,452,454,456,458, and460. Each converged adapter virtual bridge466,468,470, and472may recognize data flows included within its domain, or addressable space. A recognized domain address may be routed directly, without processing or storage, outside of the domain of the particular converged adapter virtual bridge466,468,470, and472. Each adapter410,412,414, and416may include one or more CEE transmit ports that couple to one of the bridge elements426,428,430, and432. In another embodiment, bridge elements may be co-located with the adapters, and coupling between adapters and the bridge elements may not be Ethernet connections.

The bridge elements426,428,430, and432may be configured to forward data frames throughout the distributed virtual bridge408. The bridge elements426,428,430, and432may thus function as access points for the distributed virtual bridge408by translating between Ethernet and the integrated switch router442. The bridge elements426,428,430, and432may not include buffers and may support CEE at boundaries of the distributed virtual bridge408. In another embodiment, the bridge elements426,428,430, and432may include buffers.

Each bridge element426,428,430, and432of the distributed virtual bridge408may include a forwarding cache474,476,478, and480. A forwarding cache474,476,478, and480may include a lookup table that stores address data used to forward data frames that are received by the bridge elements426,428,430, and432. For example, the bridge element426may compare address data associated with a received data frame to the address data stored within the forwarding cache474.

Illustrative address data may include routing information, such as a routing key included within header data of the data frame. The routing key may include at least one of a virtual local area network (VLAN) tag and a logical network identifier, as well as a MAC address. The MAC address may be generated and assigned by a Fiber Channel Forwarder (FCF)413, as set by an administrator or computing system. The Fiber Channel Forwarder413, or FCoE switch, may facilitate connectivity between FCoE initiators and Fiber Channel fabrics. To illustrate, an FCoE data frame sent from the first virtual machine458and intended for a second virtual machine463at the second server404may be addressed to the Fiber Channel Forwarder413in accordance with the FCoE standard. According to standard routing procedures, the Fiber Channel Forwarder413may receive and re-address the FCoE data frame for forwarding to the virtual machine463.

The MAC address of the Fiber Channel Forwarder413may have been learned by the first server computer402during a discovery phase, when the Fiber Channel Forwarder413establishes communications with networked devices. During the discovery phase, the second server computer404may respond to broadcast queries from the first server computer402. The Fiber Channel Forwarder413may discover the second server computer404from the query responses. After the discovery phase, a login phase may be initiated. A MAC address of the server computer404may be reassigned by the Fiber Channel Forwarder413. The reassigned MAC address may be used for subsequent routing and communications between the server computers402,404. The Fiber Channel Forwarder413may facilitate storage of MAC addresses assigned to the server computers402,404.

A VLAN tag may indicate an assigned VLAN, which may be used to segregate traffic and to allow more than one uplink. There may be multiple VLANs on an uplink. Conventionally, each VLAN may use only one uplink port. That is, only one physical uplink port at a given time may be used to forward a data frame associated with a particular VLAN. Through the use of logical networks, a VLAN may use multiple physical ports to forward traffic while maintaining traffic segregation. Link aggregation may be used to bundle several physical links to act as one uplink with higher bandwidth.

A logical network may include a logically specified network portion of the distributed virtual bridge408. Multiple logical networks may be included within a single bridge element. As such, a logical network may provide an additional layer of traffic separation. When so configured, logical networks may allow different customers to use the same VLAN tag. The VLANs of each customer may remain segregated by virtue of the different logical networks.

The forwarding caches474,476,478, and480of the distributed virtual bridge408may have a format similar to the global forwarding table411of the controlling bridge448. The forwarding caches474,476,478, and480may have smaller memory capacities than the global forwarding table411. The forwarding caches474,476,478, and480may further be updated with address data learned from data frames that flow through the bridge elements426,428,430, and432.

The address data may additionally be updated with address data received from the global forwarding table411. Invalid or changed address data that is updated within one or more of the forwarding caches474,476,478, and480of the bridge elements426,428,430, and432may be communicated to the global forwarding table411of the controlling bridge448. For example, the bridge element426may learn a new MAC address of a newly added device that is configured to receive from or send data to the distributed virtual bridge408.

The bridge element426may verify that a source MAC address included within a received data frame is allowed at a port by checking a list stored within a memory. The bridge element426may send a registration message to the controlling bridge448to update the global forwarding table411with the verified MAC address. The bridge element426may further store the MAC address within the forwarding cache474. In another example, the bridge element426may identify a MAC address that is infrequently used. This infrequently used MAC address may be removed from the forwarding cache474to make storage room available for other MAC addresses. The bridge element426may send an update message to the controlling bridge448to have the MAC address removed from the global forwarding table411.

Address data stored within the global forwarding table411may be communicated to one or more forwarding caches474,476,478, and480of the distributed virtual bridge408. For example, the bridge element426may receive a data frame that includes a destination MAC address that is not stored within the forwarding cache474. To obtain information for forwarding the data frame, the bridge element426may send a query to a bridge element439configured to access the controlling bridge448. The bridge element439may search the global forwarding table411for address data associated with the destination MAC address. If the address data is found, the bridge element439may forward the MAC address through the distributed virtual bridge408to the querying bridge element426. The bridge element426may store the MAC address as address data within the forwarding cache474. As with the global forwarding table411, the address data included within the forwarding caches474,476,478, and480of the distributed virtual bridge408may include both internal address information, as well as addresses that are external to the system400.

Each of the bridge elements426,428,430, and432may be connected to one or more transport layer modules482,484,486, and488. The transport layer modules482,484,486, and488may include buffering used for attachment to the integrated switch router442. The transport layer modules482,484,486, and488may further provide a frame-based, Ethernet-like interface to the integrated switch router442.

The transport layer modules482,484,486, and488may each include a shared buffer used to transmit frames across the integrated switch router442. Additional buffers of the transport layer modules482,484,486, and488may be used to receive data frames from the integrated switch router442. The buffers may be divided into different virtual lanes. Virtual lanes may include logically separated paths for data frame traffic flowing between a bridge element and a transport layer module. For example, there may be four virtual lanes between the bridge element426and the transport layer module482. The virtual lanes may correspond to differently prioritized traffic. The transport layer modules482,484,486, and488may include logic to recover from faulty microchips and links between a source and a destination. The transport layer modules482,484,486, and488may maintain a strict ordering of packets within a particular virtual lane regardless of each data frame's path through the local rack interconnect network490and the computer system400.

The integrated switch router442may communicate with the transport layer modules482,484,486, and488and may facilitate routing and packet delivery to and from the local rack interconnect network490. The local rack interconnect network490may include links to the bridge elements426,428,430, and432located within the same chassis and rack, as well as links to the bridge elements434-440in different chassis and racks. The local rack interconnect network490may include point-to-point connections, or pipes, between the bridge elements426,428,430,432, and433-440of the distributed virtual bridge408with no frame loss and with in-order frame delivery.

The second server computer404may include a server computer similar to the first server computer402and may be similar to the server computer206ofFIG. 2. As such, the second server computer404may be located within a different chassis and rack than the first server computer402. The first server computer402, the second server computer404may include a processor499coupled to a memory497and to an operating system495. The second server computer404may further include virtual machines455,457,459,461,463, and465.

A hypervisor467may be coupled to the virtual machines457,459. The hypervisor467may include a hypervisor virtual bridge471that allows direct communication between the virtual machines457,459. A hypervisor virtual bridge473of a hypervisor469coupled to the virtual machines461,463may facilitate direct communication between the virtual machines461,463. For example, the hypervisor virtual bridges471,473may register address data with the controlling bridge448.

The second server computer404may also include one or more adapters418,420,422, and424, such as converged CEE network adapters. Each adapter418,420,422, and424may be coupled to one or more of the virtual machines455,457,459,461,463, and465. The adapters418,420,422, and424may each include a converged adapter virtual bridge475,477,479, and481. The converged adapter virtual bridges475,477,479, and481may facilitate sharing of the adapters418,420,422, and424by coordinating virtual machine access. The adapters418,420,422, and424may each couple to one or more of the bridge elements434,436,438, and440of the distributed virtual bridge408. Each adapter418,420,422, and424may include one or more CEE transmit ports that couple to one of the bridge elements434,436,438, or440.

Each bridge element434,436,438, and440may include a forwarding cache483,485,487, and489that includes address data used to forward data frames that are received by the bridge elements434,436,438, and440. The bridge elements434,436,438, and440may each be connected to one or more transport layer modules415,417,419, and421. The transport layer modules415,417,419, and421may include buffering used for the attachment to the integrated switch router446. The transport layer modules415,417,419, and421may further provide a frame-based, Ethernet-like interface to the integrated switch router446and may maintain packet ordering. A portion of the distributed virtual bridge408shown inFIG. 4as located above the local rack interconnect network490and as associated with the server computers402,404may be referred to as a north portion. The north bridge elements426,428,430,432,434,436,438, and440may be coupled to the adapters410,412,414,416,418,420,422, and424.

The I/O blade device406may be the I/O server computer258ofFIG. 2. As such, the I/O blade device406may allow uplink connectivity to an external Ethernet network492via an integrated switch router401that is coupled to transport layer modules423,425,427,429, and431.

The transport layer modules423,425,427,429, and431may each couple to a bridge element433,435,437, and439. The bridge elements433,435,437, and439may each include a forwarding cache441,443,445, and447. The I/O blade device406may be categorized as being included within a south portion of the distributed virtual bridge408because the bridge elements433,435,437, and439may be coupled to an uplink to the Ethernet network492.

The I/O blade device406may include a memory409, an operating system491, and a processor453that includes the controlling bridge448. The bridge element439may be coupled to the processor453via an Ethernet link connection. The transport layer module431may be coupled to a PCIe bus444that is coupled via a PCIe link connection to the processor453and the controlling bridge448. The PCIe bus444may also be coupled to a PCIe slot493. The processor453may further include a Peripheral Component Interconnect Manager (PCIM)451.

The controlling bridge448may communicate with the bridge elements426,428,430, and432-440and other controlling bridges (not shown) of the computer system400. The controlling bridge448may include firmware executing on the processor453that manages the bridge elements426,428,430, and432-440. For example, the controlling bridge448may be configured to divide a workload between the bridge elements426,428,430, and432-440, as well as perform synchronization procedures and failover operations.

The controlling bridge448may be configured to interface with and program the bridge elements426,428,430,432-440and the adapters466,468,470,472,475,477,479,481. More particularly, the controlling bridge448may be configured to generate and send a management frame to one or more of the bridge elements426,428,430,432-440and the adapters466,468,470,472,475,477,479,481. The management frames may include instructions used to program operating parameters of the bridge elements426,428,430,432-440and the adapters466,468,470,472,475,477,479,481and other switches.

The controlling bridge448may include the Fiber Channel Forwarder413. FCoE may offer the capability to transport fiber channel payloads on top of an Ethernet network. The Fiber Channel Forwarder413may execute the Fiber Channel Initialization Protocol to discover and initialize FCoE capable entities connected to an Ethernet cloud. The Fiber Channel Forwarder413may further include firmware that encapsulates and de-encapsulates Fiber Channel data frames (e.g., FCoE formatted data frames). In at least one embodiment, the Fiber Channel Forwarder413may translate between Ethernet and Fiber Channel protocols.

The controlling bridge448may additionally include the global forwarding table411. The global forwarding table411may include address data (e.g., MAC addresses) that is registered and maintained through communication and cooperation with the bridge elements426,428,430, and432-440, and in some cases, the hypervisors462,467, and469.

In one example, the global forwarding table411may maintain MAC addresses that have been learned by a bridge element426. The bridge element426may register the address data with the controlling bridge448. The controlling bridge448may update the global forwarding table411by adding the address data to the global forwarding table411. Similarly, the bridge element426may cause the controlling bridge448to update the global forwarding table411by sending an update message to the controlling bridge448. The update message may cause the controlling bridge448to delete a MAC address that has been aged out by the bridge element426. A MAC address may further be deleted when the bridge element426has detected that the address data is no longer valid.

In another example, the hypervisor virtual bridge464may register MAC addresses or other address data with the controlling bridge448. The global forwarding table411may include address data associated with addresses that are included within the system400, as well as addresses that are external to the system400.

FIG. 5shows a method of500routing a data frame using a distributed bridge element. The method500may be performed by a distributed bridge element, such as the north bridge element308ofFIG. 3.

At502, routing information may be received. For example, the north bridge element308may receive the routing information332from the primary controlling bridge324. The routing information332may be used to automatically distribute data frames received at the north bridge element308. A data frame may be received, at504. For instance, the north bridge element308may receive a data frame from the server computer302ofFIG. 3. At506, the frame address may be evaluated. For example, the north bridge element308may evaluate an address of a received data frame. The received data frame may include a source address and a destination address.

A link may be selected based upon the routing information and the frame address, at508. For instance, the north bridge element308may select the link317or port333based upon the routing information332and the data frame address data. At510, the data frame may be routed to the selected link. For instance, the north bridge element308may route the data frame to the link317based on the routing algorithm336.

A faulty link may be detected at512. For example, the south bridge element316may determine that the link317or the port333has failed or is otherwise faulty. When a faulty link has been detected, a notification signal may be generated, at514. For example, the south bridge element316may generate a notification signal. The notification signal may be sent to the primary controlling bridge324. A notification signal may additionally be sent to the north bridge element308.

At516, the routing information may be updated. For instance, the north bridge element308may update the routing information332in response to receiving the notification signal. When an update is received at518, the routing information may be updated, at516. For example, the north bridge element308may update the routing information332. The updated routing information may be used to route a next received data frame, at504.

FIG. 5thus shows a method500of routing data frames at a bridge element using routing information received from a controlling bridge. The routing information may be updated to reflect changes in a system. The method500may facilitate distributed forwarding processes in a manner that increases routing efficiency. Increased efficiencies may enable greater scalability and accuracy in highly integrated environments.

FIG. 6includes a method600of programming a bridge element to independently distribute data frames. The method600may be accomplished by a controlling bridge, such as the primary controlling bridge324ofFIG. 3.

At602, routing information may be communicated to a bridge element. For example, the primary controlling bridge324ofFIG. 3may communicate routing information332to the bridge element308.

A controlling bridge may monitor a status of a link or node, at604. For instance, the primary controlling bridge324ofFIG. 3may monitor a status of system components corresponding to the link and node data386.

At606, a signal to modify a status of a link or node may be received. For example, the primary controlling bridge324ofFIG. 3may receive a notification signal from a south bridge element316. The notification signal may indicate that a link317or a port333is faulty. In response to a received signal at606, routing information may be updated at608. For example, the primary controlling bridge324ofFIG. 3may update the link and node data386and/or associating algorithm388. The primary controlling bridge324may communicate the updated routing information332,350to affected bridge elements308,310.

FIG. 6thus shows a method600of providing routing information to a distributed bridge element configured to independently route data frames. A controlling bridge may monitor the operation of system components to provide failover processes and routing updates. The method600may facilitate automated programming in a manner that reduces administrator workload. Increased programming efficiencies may facilitate system adaptability and routing efficiency.

Particular embodiments described herein may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment containing both hardware and software elements. In a particular embodiment, the disclosed methods are implemented in software that is embedded in processor readable storage medium and executed by a processor, which includes but is not limited to firmware, resident software, microcode, etc.

Further, embodiments of the present disclosure, such as the one or more embodiments may take the form of a computer program product accessible from a computer-usable or computer-readable storage medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable storage medium can be any apparatus that can tangibly embody a computer program and that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

Input/output or I/O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the data processing system either directly or through intervening I/O controllers.

Network adapters may also be coupled to the data processing system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the currently available types of network adapters.