Distributed Control For Large Photonic Switches

Large photonic switches can establish optical paths between a large number of inputs and outputs. A distributed control architecture may be used in order to quickly establish the optical paths in large photonic switches. The distributed control architecture may provide a hierarchical control by grouping together endpoints, determining switching requirements between the groups and determining switching requirements within the groups.

TECHNICAL FIELD

The current disclosure relates to photonic switches, and in particular to the control of large photonic switches.

BACKGROUND

Photonic switches may be used to establish optical paths between two end points.FIG. 1depicts an illustrative optical switch100that can establish optical paths between 8 inputs102a-102hand 8 outputs104a-104h. The optical paths are established through a switch matrix106of individual photonic switching cells. The individual switching cells are controlled in order to establish the desired optical paths through the switch matrix. The switch matrix106, or more particularly the individual switching cells of the switch matrix, is controlled by a switch controller108. The switch control108receives an indication of the desired optical paths and determines the control signals that need to be generated in order to establish the desired paths.

Unlike electronic switches, which are able to buffer data signals as they arrive at the switch, all-optical switches are not able to buffer the optical signals. The entire optical path must be established between the two end points before data can be transmitted. While small optical switches such as the optical switch100may be controlled by a single controller, as switches become larger, the switching components may be separated from each other by a length of cable. As the switch size increases, along with possible increasing optical cable lengths between switching components, the time required to determine and setup the optical paths can become large.

SUMMARY

In accordance with the present disclosure there is provided a distributed controller for a photonic switch, the photonic switch comprising a plurality of inputs and outputs grouped together into a plurality of groups, the distributed controller comprising: a plurality of group controllers communicatively coupled together, each group controller for controlling switching of inputs and outputs within a respective group; and a switch controller communicatively coupled to the plurality of group controllers, the switch controller for controlling switching between inputs and outputs of the plurality of groups.

In a further embodiment of the distributed controller, the plurality of group controllers communicate desired bandwidth with other groups to an associated group controller of the plurality of group controllers.

In a further embodiment of the distributed controller, the plurality of group controllers determine bandwidth assignments to other groups based on a desired bandwidth from other groups.

In a further embodiment of the distributed controller, the switch controller monitors communication between group controllers to determine required bandwidth between the groups.

In a further embodiment of the distributed controller, one or more of the plurality of group controllers communicate the bandwidth assignments to the switch controller.

In a further embodiment of the distributed controller, the plurality of group controllers are arranged in a ring architecture.

In a further embodiment of the distributed controller, the plurality of group controllers are arranged in a star architecture.

In a further embodiment of the distributed controller, the plurality of group controllers are arranged in a mesh architecture.

In a further embodiment of the distributed controller, the plurality of group controllers are arranged in a tree architecture.

In a further embodiment of the distributed controller, the switch controller is a distributed switch controller comprising a plurality of individual switch controllers in communication with each other.

In a further embodiment of the distributed controller, the distributed switch controller is arranged in one of: a ring architecture with the plurality of group controllers; a mesh architecture; and a tree architecture.

In a further embodiment of the distributed controller, each one of the plurality of group controllers is configured to: determine a desired bandwidth between each of the plurality of groups; send the desired bandwidth as bandwidth requests to the group controllers of the groups; receive bandwidth requests from the group controllers; determine bandwidth assignments to each group based on the received bandwidth requests; send the determined bandwidth assignments to the group controllers; receive bandwidth assignments from the group controllers; and determine switching within the group based on the received bandwidth assignments.

In a further embodiment of the distributed controller, the bandwidth assignment to each group is determined using a fairness scheme or a priority scheme.

In a further embodiment of the distributed controller, each of the plurality of group controllers are further configured to: receive individual bandwidth requests from optical end points of the group, each of the individual bandwidth requests specifying a desired bandwidth to an optical end point coupled to the plurality of inputs and outputs of the photonic switch; and aggregating the individual bandwidth requests to determine the desired bandwidth between each of the plurality of groups.

The distributed controller of claim1, wherein the distributed controller is within a software defined networking (SDN) controller.

In accordance with the present disclosure there is further provided a method of control of a photonic switch fabric, the photonic switch fabric comprising a plurality of inputs and outputs grouped together into a plurality of groups, each of the plurality of groups associated with a respective group controller of a plurality of group controllers, the method comprising: at each of the plurality of group controllers: controlling switching of inputs and outputs within a respective group; and at a switch controller: controlling switching of inputs and outputs between the plurality of groups.

In a further embodiment, the method further comprises, at each of the plurality of group controllers: determining bandwidth assignments to each group; sending the determined bandwidth assignments to the other group controllers; receiving bandwidth assignments from the other group controllers; and determining switching within the group based on the received bandwidth assignments; and at the switch controller: determining bandwidth assignments between each of the plurality of groups; and determining switching between the groups based on the bandwidth assignments between each of the plurality of groups.

In a further embodiment of the method, the bandwidth assignment to each group is determined using a fairness scheme or a priority scheme.

In a further embodiment, the method further comprises, at each of the plurality of group controllers: determining a desired bandwidth between each of the plurality of groups; sending the desired bandwidth as bandwidth requests to the group controllers of the groups; receiving bandwidth requests from other group controllers; and determining the bandwidth assignments to each group based on the received bandwidth requests.

In a further embodiment of the method, the switch controller determines the bandwidth assignments between each of the plurality of groups by obtaining the determined bandwidth assignments sent from the plurality of group controllers.

In a further embodiment, the method further comprises issuing control signals from the plurality of group controllers to establish the determined switching within the groups.

In a further embodiment, the method further comprises issuing control signals from the switch controller to establish the determined switching between the groups.

DETAILED DESCRIPTION

Photonic switches establish optical paths between end points. Because photonic switches cannot buffer optical signals, it is necessary for the entire optical path from end point to end point to be established prior to transmitting data over the optical path. As the size of optical networks grows, an optical switch may be required to be implemented as a number of separate physical components that are connected together. While such physically distributed switches may be used to provide larger optical networks, centralized control of the switch may reduce an amount of time that can be used for transmitting data since the time required to determine and setup the optical paths may become large for the controller. That is both, the time required to determine the routing for the entire photonic switch by a centralized controller as well as the propagation time to send the determined routing to all of the switching components from the centralized controller increases as the size of the physical switch increases. The propagation time from the centralized controller to the switching components will depend, at least in part, on the length of fiber optic cable or waveguide between the centralized controller and the switching components. While such centralized control may be acceptable for slowly varying optical networks in which optical paths may be setup for hours, days or longer, the time required may be unacceptable, or at least undesirable, for other applications such as in data centers or other networks that have relatively quickly varying optical paths between end points. As described further herein, large photonic switches may be controlled in a distributed manner. Switching end points are grouped together into a plurality of groups. The large photonic switch may be controlled by determining group to group switching connections that provides routing between different groups of end points. Intra group switching may be controlled by a plurality of individual group controllers. Accordingly, groups of end points may be connected together, and then the particular end points within the connected groups may be connected together. Since, the group-to-group connections are likely to change more slowly than the end-point connections, the routing between end points of connected groups can be controlled separately from the group to group connections. As described further below, the photonic switch with distributed control may comprise an agent controller inside of a next generation top of rack switch (NGTOR). NGTORs may be referred to as aggregation nodes, server groups, or end of row switches. The distributed controller further includes group controllers and a switch controller, or a set of group controllers.

FIG. 2depicts a large photonic switch. The photonic switch200is depicted as an 8×8 switch however, it is possible for the photonic switch to have a much larger capacity, such as 256×256, 512×512, 1024×1024 or larger. Larger switches may be formed by grouping larger capacity switches together with switch controllers in communication with each other in order to provide co-ordinated control of the switches. The photonic switch200provides optical switching between end-points, which inFIG. 2are depicted as next-generation top-of-rack (NGTOR) switches202a-202h(referred to collectively as NGTORs202); however, the NGTORs may be provided by other optical switching end points. Each of the NGTORs202may be connected to other network components, not depicted, and may aggregate network traffic from connected devices for transmission between other NGTORs202, or more particularly the network components connected to other NGTORS202. Each of the NGTORs202may have a number of optical interfaces for coupling to the photonic switch200. The interfaces are depicted as small rectangles, one of which204a−1 is labelled for NGTOR202a. Although only a single interface,204a−1, is labelled, the interfaces are referred to collectively as interfaces204. Each of the NGTORs202is depicted as having 2 interfaces204; however, it is possible for one or more of the NGTORs202to have a different number of interfaces204.

The individual NGTORs202are grouped together into a plurality of end point groups206a,206b(referred to collectively as end point groups206). The number of NGTORs202in each of the end point groups206are depicted as being the same; however, it is possible for the different end point groups206to group together different numbers of NGTORs202. Further, although only two end point groups206are depicted, additional end point groups can be provided. Each of the NGTORs202of the end point groups206may be connected to the inputs and outputs of the photonic switch200.

The photonic switch200comprises a plurality of sub-switches208a,208b(referred to collectively as sub-switches208). The number of sub-switches208forming the switch may correspond to an expected maximum number of interfaces of the NGTORs202. For example, the NGTORs202are depicted as all having 2 interfaces204and as such the switch200comprises two sub-switches208. Each of the sub-switches208comprise a plurality of end point group input switches210a,210band a plurality of end point group output switches212a,212bcoupled to each other by a group switching fabric214a. The end point group switches control switching to NGTORs202within the same group. For example, the end point group input switch210aand end point group output switch212aswitches optical paths to the individual NGTORS202a-202dof the end point group206a. The end point group input switch210band the end point group output switch212bswitches optical paths to the individual NGTORS202e-202hof the end point group206b. The end point group input and output switches210,212can establish connections to the NGTORs202within an end point grouping. The group switching fabric214aestablishes connections between the end point group input and output switches210,212.

The sub-switch208bdepicts details of the end point group input switches210and end point group output switches212. The end point group input switches, the group switching fabric and the end point group output switches are formed as a plurality of similar switches or switching planes. For example, one switch may be provided by an input switch216a, coupled to an output switch218aby a switching fabric220a. There are similar input switches216b,216c,216d, similar output switches218b,218c,218dand similar switching fabric220b,220c,220d, for each of the NGTORS202of the group.

The architecture depicted inFIG. 2may be used in providing a large photonic switch capable of switching between a large number of end points. The photonic switch200is controlled by a switch controller222that determines the routing of the optical paths through the photonic switch200and applies the control signals to the components of the photonic switch in order to establish the determined routing. In order to determine the routing and establish the optical paths in a reasonable time the switch controller222is provided as a distributed controller. As described in further detail with reference toFIG. 3, the distributed controller is provided as a plurality of individual group controllers that control the end point group input and output switches in order to control switching within groups while a main switch controller controls the group switching for switching between groups. Although described as a single switch controller, the switch controller could be implemented in a distributed fashion by a plurality of switch controllers in communication with each other.

FIG. 3depicts a distributed controller of a large photonic switch. The photonic switch depicted inFIG. 3comprises a similar architecture as the photonic switch200described above; however rather than being aligned one above the other as depicted inFIG. 2the two sub-switches are stacked one on top of the other. The end point group input switches210a,210ccoupled to the first group206aof NGTORs202are controlled by a first group controller, namely a group controller a302a. The group controller a302aalso controls the end point group output switches212a,212ccoupled to the first group206aof NGTORs202. A second group controller, namely group controller b302b, controls the end point group input switches210b,210dand the end point group output switches212b,212dassociated with the second group206bof NGTORs202. The group controllers302a,302bcontrol the switch components to provide switching within the groups. A switch controller304controls the switching of the connections between the different groups. That is, the switch controller304controls the group switching fabric214a,214bin order to provide connections between the groups. The switching between groups will tend to vary more slowly than the switching within a group, and as such.

In order to provide coordinated control of the large photonic switch200using the plurality of group controllers302a,302band the switch controller304, each of the NGTORs202communicate connection requirements, that is the amount of bandwidth desired to be established with the other NGTORs202, to the associated group controllers302a,302b. The group controllers302a,302bcommunicate the bandwidth requirements with each other as depicted by arrow306. The connections to be established between the groups of NGTORs202is communicated to the switch controller304, as depicted by arrows308a,308b.

An allocation cycle may be provided in order to determine the bandwidth allocation. During an allocation cycle, which may be relatively short, such as a few microseconds or less, the bandwidth, or bandwidth changes between different groups is determined. The bandwidth may be assigned according to the interfaces of NGTORs. Each member of a group may send a desired bandwidth between itself and other end points to the group controller. The group controllers receive the bandwidth requests from the group members and aggregates the requests together. Each of the group controllers sends the aggregated requests to the other group controllers. The group controllers receive the aggregated requests and determines what bandwidth may be provided. Each of the group controllers provides the determined bandwidth allocations to the other group controllers. When the group controllers receive the bandwidth allocations from other group controllers, the bandwidth is distributed amongst the group members based on the requested bandwidth. The group controllers configure the group switches in order to provide the required connections to the NGTORs. The bandwidth allocated between the groups may be monitored by the switch controller and used in order to configure the switching fabric to provide the connections between groups. The distributed control process is described in further detail below with regard toFIG. 4.

FIG. 4depicts a process for distributed control of large photonic switches. Messages exchanged between a number of end points402associated with a group controller404are depicted. Each of the end points402may have a number of interfaces, that can be used in providing the bandwidth. For example, the end points402may comprise aggregation points that receive data from a plurality of sources, aggregate them together and transmit the data over one of the interfaces. The number of interfaces, and their individual bandwidth capacity, determines the bandwidth capacity of the end point.

Additional group controllers406are depicted inFIG. 4. Although not depicted inFIG. 4it will be appreciated that each of the additional group controllers406also perform a similar message exchange with the associated end points402. A switch control408is depicted as monitoring the messages sent between the group controllers404,406; however, it is possible for the group controllers404,406to send the messages directly to the switch controller408.

Each of the end points402send bandwidth (BW) requests (1) to the associated group controller404. The BW requests may provide an indication of the bandwidth that the end point is requesting to be established with other end points connected to the large photonic switch. The group controller404receives the BW requests from the end points402and aggregates the requests together (2). The aggregated requests provide an amount of bandwidth requested between the group controller402and each one of the additional groups404. Although not depicted, the additional group controllers406also receive BW requests from the associated end points402. The group controllers404,406exchange the aggregated BW requests (3). The Aggregated BW requests are received at the group controller404and used in order to assign available bandwidth (4) to each of the groups based on the requested bandwidth. The BW assignments may be determined in various ways including using a fairness scheme such as min-max fairness or a priority scheme. The BW assignments are exchanged between the group controllers (5). The group controller404receives the bandwidth assignments from the other group controllers and determines the available bandwidth to be assigned to each of the end points (6). The determined bandwidth assignments for each of the end points is used in order to configure the intra-group switching fabric. That is, the determined bandwidth assignments are by the group controller404in order to establish the switching within the group. The group controller404may also communicate the bandwidth assignments (8) to the associated end points402. The BW assignments communicated between the group controllers404,406are monitored by the switch controller408and used to determine the group switching required to provide the assigned BW between the different groups (9). The determined group switching is used by the switch controller408in configuring the inter-group switching fabric (10). That is, the switch controller408configures the switching between groups. Once the intra-group switching fabric and the inter-group switching fabric are configured, the optical paths are established through the photonic switch (11) and data can be transmitted over the optical paths.

The distributed control of large photonic switches described above provides a plurality of group controllers that are inter-connected to each other and to a switch controller. The capacity allocation of the switch by the distributed controller is performed at two levels. The first level is at the group level and may be controlled by the switch controller in order to configure the switch to provide the required bandwidth capacity between the groups. The bandwidth allocation between groups tends to be slowly changing. The second level of control is at the group member level and may be controlled by the group controllers. The group controller determines how to allocate the bandwidth capacity established with other groups by the switch controller to the members within the group. The switch connections are accordingly established at two levels. The switch controller controls switching at the inner, or core, of the photonic switch and provides group to group connectivity. The group controllers control the switching at the outer edge of the photonic switch and provides for the switching to the group members.

FIG. 5depicts a distributed control architecture for large photonic switches. The distributed control architecture depicted inFIG. 5provides hierarchical control of large photonic switches. The distributed control comprises a plurality of group controllers504a-504e(referred to collectively as group controllers504) in communication with each other, as well as a switch controller502in communication with the group controllers504, or at least able to receive messages sent between the group controllers. The group controllers504are arranged in a ring topology. The group controllers504may exchange messages in various ways, depending upon the particular architecture. For example, messages may be sent according to the IEEE 802.17 Resilient Packet Ring (RPR) standard or using a slotted ring approach, where each group controller has a slot of ring bandwidth. The switch controller502can read or receive the bandwidth assignments granted by each group controller.

End points, which may be NGTORs as described above, or other optical network end points, connected to the switch may be grouped together into a plurality of end point groups. Each of the end points may be associated with a respective end point controller, which are arranged into end point controller groups506a-506e. Each of the end point controllers of a particular group are in communication with the group controller for the group. For example, the end point controllers506aare in communication with group controller504a. As depicted inFIG. 5, the end point controllers506a-506emay be in direct communication with the respective group controllers504.

Communication between the different levels of controllers may be established over a separate control channel, or the communication may be established over the data paths being controlled by the controllers. The group controllers and switch controller can be part of a software defined network (SDN). The distributed control provided by the group controllers and switch controller, which may in turn be a distributed switch controller, may be provided as part of an SDN controller in a datacenter or enterprise application.

The specific architecture of the distributed controller does not need to be limited to the ring topology described above with reference toFIG. 5. Various architectures are possible that provide the required communication between the end point controllers, the group controllers and the switch controller.

FIG. 6depicts a further distributed control architecture for large photonic switches. The distributed control architecture600comprises a star connection between a switch controller602and a plurality of group controllers604a-604e. The group controllers604a-604eare in communication with each other through a fully connected mesh architecture. The groups of end points, or end point controllers606a-606emay also be in communication with the associated group controllers604a-604eusing different architectures, such as a ring architecture as depicted in end point controller groups606a,606b,606e, or directly as depicted in end point controller groups606c,606e.

FIG. 7depicts a group controller, and associated method, for use in distributed control of a large photonic switch fabric. The photonic switch fabric may be provided by one or more individual photonic switches coupled together. The group controller700may be used as any of the group controllers described above and may be used to control various large photonic switches of different architectures. Regardless of the particular switch architecture, it is assumed that a plurality of end points connected to the switch are grouped together. The group controller700controls the bandwidth assignments, and establishing the appropriate optical paths within a portion of the photonic switch to provide the bandwidth, to end points of the associated group. The group controller is depicted as being provided in a rack type of server case; however, it may be provided in various different physical implementations. The group controller700comprises a processor702for executing instructions. The instructions may be stored in memory704of the group controller700. In addition to the memory704, the group controller700may include non volatile storage706. The group controller700further comprises one or more input output (I/O) devices708, such as network interfaces, that allow the group controller to communicate with other components of the distributed controller such as other group controllers or the switch controller as well as other controllers such as end point controllers. The instructions stored in the memory704comprise instructions that when executed by the processor702configure the group controller700to provide group controller functionality710.

The group controller functionality710determines the bandwidth assignments for the end points within the group. A method712of controlling end point groups that may be performed by the group controller functionality710is depicted inFIG. 7. The method712begins with the group controller functionality710receiving bandwidth requests from connected end point controllers (714). The bandwidth requests from each of the end point controllers provide an indication of the bandwidth to another end point that is desired. The other end point may be within the same group or within another group. The group controller functionality710receives the bandwidth requests and aggregates bandwidth requests (716). The bandwidth requests may be aggregated to provide a total bandwidth requested from the group associated with the group controller700to each of the other groups, and may include priority information based on queue levels of the end points. The priority information may be used by group controllers in determining the actual bandwidth assignments. The aggregated bandwidth requests are sent to the other group controllers (718). The other group controllers perform a similar method and as such the other group controllers also send aggregated bandwidth requests to the group controller700. The group controller functionality710receives the aggregated bandwidth requests from the other group controllers (720) and determines the aggregated bandwidth assignments (722) for the different groups. That is, it may not be possible to satisfy requested bandwidth from all of the groups simultaneously, and as such, the group controller determines the bandwidth that will be assigned to the different groups. The determined aggregated bandwidth assignments are sent to each of the group controllers (724). The determined aggregate bandwidth assignments specify the amount of bandwidth that will be provided between the two groups. Again, the other group controllers determine aggregated bandwidth assignments and send them to the other group controllers. The group controller functionality710receives the aggregated bandwidth assignments from the other group controllers (726) and determines the bandwidth assignments from the end points within the group (728). That is, once the group-level bandwidth has been assigned between each of the groups, the group controller functionality710determines how to allocate the group bandwidth to the end points. Once the end point bandwidth assignments have been determined, the group controller700can configure the group switching components to establish the required optical paths. The above group controller functionality may be performed in different orders than described above, however it will be completed within a bandwidth allocation cycle. The length of the bandwidth allocation cycle depends, at least on part, on propagation delay, which could be in the range of for example a micro-second range up to a msec range.

As described above, the group controllers exchange messages allocating bandwidth or capacity between the different groups of end points. The switch controller receives the group bandwidth assignments and configures the switching core in order to establish the required optical paths to provide the group-to-group bandwidth. The switch controller may be provided as a centralized controller, as depicted inFIGS. 5 and 6, or as a distributed controller similar to the group controllers. The plurality of group controllers may be arranged in various architectures, including for example a mesh architecture, a star architecture, a tree architecture and/or a ring architecture. Similarly, the switch controller may be provided as a centralized controller or a distributed architecture, including for example a mesh architecture, a star architecture, a tree architecture and/or a ring architecture.

FIG. 8depicts portions of a distributed controller. The distributed controller800depicts only the group controllers802a-802i(referred to collectively as group controllers802) and the switch controller which is provided by a number of distributed switch controllers804a-804c(referred to collectively as switch controllers804). As depicted, the group controllers802and switch controllers804are arranged in a ring structure. The specific number of individual group controllers802and switch controllers804may vary. The functionality of both the group controllers802and switch controllers804is substantially the same as that described above. As depicted, the group controllers802and switch controllers80are all part of a ring network. During a bandwidth allocation cycle may be one round trip around the ring. A CONTROL frame may circulate around the ring.FIG. 8depicts the ring as being uni-directional; however, it is possible for the ring to be bi-directional. Each one of the group controllers802can set the allocated bandwidth to the other groups. The allocated bandwidth to a particular group may be placed in a particular location within the CONTROL frame. Each of the group controllers802and the switch controllers804can read the CONTROL frame in order to determine the bandwidth assignments between each group. The switch controllers804may then establish the appropriate optical paths through the optical switches to establish the group-to-group bandwidth and the group controllers802can establish the appropriate optical paths to provide the intra-group bandwidth.

The present disclosure provided, for the purposes of explanation, numerous specific embodiments, implementations, examples and details in order to provide a thorough understanding of the invention. It is apparent, however, that the embodiments may be practiced without all of the specific details or with an equivalent arrangement. In other instances, some well-known structures and devices are shown in block diagram form, or omitted, in order to avoid unnecessarily obscuring the embodiments of the invention. The description should in no way be limited to the illustrative implementations, drawings, and techniques illustrated, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.