Patent Description:
In converged transport architectures, Ethernet frames (specifically, layer <NUM> or L2 Ethernet frames) are transparently transported using a wide range of optical-based transport networks, such as synchronous optical networking (SONET), Optical Transport Network (OTN, or Packet over Optical Channel Data Unit (ODU)), Dense Wavelength Division Multiplexing (DWDM), and so forth.

In many cases, the Ethernet interface identifier is modelled to follow (e.g., incorporates) the interface identifier of the transport layer. For this reason, certain benefits of a converged transport architecture may be unavailable. Some non-limiting examples of these benefits include (<NUM>) a portability of user configuration and packet features on a terminated packet interface - which may be dynamically created on any available path in the transport network, (<NUM>) support for protection schemes supported by the transport network, (<NUM>) the transparent application of packet features to redundant paths in the transport network, and (<NUM>) the transparent migration of services in the transport network.

Document "<NPL> is directed to fast, scalable, and distributed restoration in general mesh optical networks.

In <CIT>, a network device establishes first and second Ethernet link aggregation groups (LAGs) at a first access site of an optical transport network (OTN), and creates a first optical channel (OCh) LAG subpath from the first Ethernet LAG, via a second access site of the OTN, to an Ethernet LAG at a third access site of the OTN. The network device also creates a second OCh LAG subpath from the first Ethernet LAG, via a distribution site of the OTN, to the Ethernet LAG at the third access site, and creates a first optical data unit (ODUk) LAG subpath from the second Ethernet LAG to an Ethernet LAG at the second access site. The network device further creates a second ODUk LAG subpath from the second Ethernet LAG, via the distribution site and the third access site, to the Ethernet LAG at the second access site.

One embodiment presented in this disclosure is a method for use by a network element coupled with an optical transport network according to claim <NUM>.

Another embodiment presented in this disclosure is a network element suitable for use with an optical transport network according to claim <NUM>.

Another embodiment presented in this disclosure is a computer program product for use by a network element coupled with an optical transport network according to claim <NUM>.

In various embodiments disclosed herein, a network element is configured to transport Ethernet packets over an optical transport network. In some embodiments, the network element may be a multilayer converged transport network element. The network element may be configured to apply link aggregation or link bundling techniques to form a virtual interface between network elements. Beneficially, by communicating using virtual interfaces established between network elements, user configuration information and/or packet features may remain transparent to paths and/or local ports of the optical transport network. In this way, features such as dynamic path computation and instantiation using the control plane (e.g., through via Generalized Multi-Protocol Label Switching (GMPLS)) and/or management plane (e.g., through a Software Defined Networking (SDN) controller) are supported, as well as various protection schemes supported by optical transport layers.

<FIG> illustrates an exemplary communication system <NUM>, according to embodiments described herein. As shown, the communication system <NUM> comprises a first network element <NUM>-<NUM> (generically, a network element <NUM>) having a first terminated packet interface <NUM>-<NUM> (generically, a terminated packet interface <NUM>), which is communicatively coupled via an optical transport network <NUM> to a second network element <NUM>-<NUM> having a second terminated packet interface <NUM>-<NUM>. Each network element <NUM> generally comprises one or more computer processors that are configured to route and/or process network traffic, as well as a memory (which may include volatile and/or non-volatile memory elements). Each network element <NUM> may further comprise various input/output (I/O) interfaces coupled with the one or more computer processors, such as electrical and/or optical network interfaces, interfaces for displays or user input devices, and so forth.

In some embodiments, the network element <NUM> may be a multilayer converged transport network element that is configured to provide both packet switching and optical switching functionalities. The network element <NUM> may have any suitable implementation, such as a multiple-rack-unit supporting a plurality of line cards. In some cases, each line card provides various functionality to the network element <NUM>, such as packet forwarding, Optical Transport Network (OTN) switching, Dense Wavelength Division Multiplexing (DWDM) transponding, and so forth. The optical transport network <NUM> generally comprises one or more optical elements (such as optical fibers) that are suitable for establishing one or more optical communication channels between the network elements <NUM>.

In some embodiments, each network element <NUM> is configured to transport L2 Ethernet packets over the optical transport network <NUM>. One technique for achieving this transport is to model the Ethernet interface identifier to follow the interface identifier of the transport layer. In some cases, "following" the interface identifier may comprise partially or entirely incorporating the transport layer interface identifier.

For example, in Packet over ODU, assume that an OTN interface is assigned an identifier of:
OTU<x><Rack>/<Slot>/<Port>
where OTU<x> indicates a data rate for the interface, and <Rack>, <Slot>, and <Port> each provide location information for the interface within the network element <NUM>. The ODU may then be assigned an identifier of:
ODU<x><Rack>/<Slot>/<Port>/<Oduld>
where ODU<x> indicates a data rate that may be mapped into the OTU<x>. OTU<x> may alternately be referred to as OTUk, where k represents a predefined identifier. Some examples of standardized OTUk data rates include OTU0, OTU1, OTU2, OTU2e, OTU2f, OTU3, OTU3e2, and OTU4. Similarly, ODU<x> may alternately be referred to as ODUk and may have standardized data rates such as ODU0, ODU1, ODU2, ODU2e, ODU3, ODU3e2, and ODU4. The identifier for the ODU interface repeats (or incorporates) the <Rack>, <Slot>, and <Port> information of the OTN interface identifier, and adds the <Oduld> field. Similarly, the Ethernet interface may be assigned an identifier of:
<X>GigE<Rack>/<Slot>/<Port>/<Oduld>/<Interface number> , where <X>GigE indicates a data rate for transmitting Ethernet frames. The identifier for the Ethernet interface repeats most of the ODU interface identifier and adds the <X>GigE and <Interface number> fields.

Maintaining the similarity between interface identifiers may be done for any number of reasons. Some non-limiting examples include ease of management or identification for a user, preserving a physical location associated with the Ethernet interface, supporting Network Management System and/or control plane operations, providing a hardware association of the Ethernet interface with its parent interface in the hierarchy, and so forth. However, this style of hierarchical interface modelling for a packet interface can limit the dynamic capabilities of the control plane that created the interface (e.g., functionality available via Generalized Multi-Protocol Label Switching (GMPLS)), and can also limit the ability of the interface to use protection schemes that are otherwise available through the transport layers.

According to embodiments described herein, the network element <NUM> is configured to associate a virtual interface (e.g., using link-aggregation or link-bundling techniques) with the associated terminated packet interface <NUM>. In this way, the terminated packet interface <NUM> may be decoupled from the user interface used by the network element <NUM>. In some embodiments, the terminated packet interface <NUM> (e.g., L2 Ethernet) may be independently created and/or deleted by the network element <NUM> using one of the following: a transport layer control plane (e.g., GMPLS), a management plane, a Network Management System, and a Software-Defined Network controller. The network element <NUM> may use any suitable type of interface identifier modeling for the terminated packet interface <NUM>, and may use any suitable path through the optical transport network <NUM>. The network element <NUM> may then add and/or remove the terminated packet interface <NUM> as a member of a virtual (or bundled) interface.

In this way, any user configuration and/or packet features that are enabled on the virtual interface by the network element <NUM> can remain transparent to any changes in the membership of the virtual interface (e.g., adding or subtracting terminated packet interfaces <NUM> to support a dynamic bandwidth allocation), as well as to which transport path and/or local ports are used by the members. Using the virtual interface allows the network element <NUM> to perform dynamic path computation within the control plane (e.g., available through GMPLS), as well as to apply transport layer protection schemes. Additionally, the virtual interface allows members to reside at any suitable location (e.g., a line card, node, rack, chassis) of the network element <NUM>.

Although specific examples are described with OTN as the optical transport network <NUM> and GMPLS provided via the control plane, the techniques discussed herein are applicable to other suitable optical transport planes and associated control planes. The techniques discussed herein are also suitable for use with configurations having a management plane and/or a SDN controller.

<FIG> is an exemplary method <NUM> for establishing a virtual interface between network elements, according to embodiments described herein. One example of the virtual interface is depicted in diagram <NUM> of <FIG>. The method <NUM> may be performed in conjunction with other embodiments, such as performed using a network element <NUM> of <FIG>.

Referring to <FIG> and <FIG>, method <NUM> begins at block <NUM>, where the network element configures a packet-terminated circuit using an optical transport network. In diagram <NUM>, the network element <NUM>-<NUM> comprises an OTU2 interface <NUM>/<NUM> associated with two ODU2 interfaces <NUM>/<NUM>/<NUM>, <NUM>/<NUM>/<NUM>, and an OTU2 interface <NUM>/<NUM> associated with two ODU2 interfaces <NUM>/<NUM>/<NUM>, <NUM>/<NUM>/<NUM>. The network element <NUM>-<NUM> comprises an OTU2 interface <NUM>/<NUM> associated with two ODU interfaces <NUM>/<NUM>/<NUM>, <NUM>/<NUM>/<NUM>, and an OTU2 interface <NUM>/<NUM> associated with two ODU interfaces <NUM>/<NUM>/<NUM>, <NUM>/<NUM>/<NUM>. Other numbers and/or types of interfaces (e.g., different label-switched path (LSP) encoding types) are also possible for each network element <NUM>. In diagram <NUM>, the network element <NUM>-<NUM> configures a packet-terminated circuit by establishing the ODU2 interface <NUM>/<NUM>/<NUM> through the OTU2 interface <NUM>/<NUM>.

At block <NUM>, the network element calculates a destination path for the packet-terminated circuit. In diagram <NUM>, the network element <NUM>-<NUM> calculates a destination path to the ODU2 interface <NUM>/<NUM>/<NUM> of the network element <NUM>-<NUM> through the OTU2 interface <NUM>/<NUM>.

At block <NUM>, the network element transmits signals to the optical transport network to create the packet-terminated circuit. In some embodiments, the network element <NUM>-<NUM> creates a GMPLS packet-terminated ODU circuit (or "tunnel") between the ODU2 interface <NUM>/<NUM>/<NUM> and the ODU2 interface <NUM>/<NUM>/<NUM>.

At block <NUM>, the network element determines whether the signaling was successful to create the packet-terminated circuit. If the signaling was not successful ("NO"), the method <NUM> returns to block <NUM> and calculates a different destination path for the packet-terminated circuit. If the signaling was successful ("YES"), path <NUM> exists between the ODU2 interface <NUM>/<NUM>/<NUM> and the ODU2 interface <NUM>/<NUM>/<NUM>. The method <NUM> proceeds to block <NUM>, wherein the network element terminates the circuit. At block <NUM>, the network element creates an Ethernet interface. In some embodiments, the Ethernet interface is created by the network element <NUM>-<NUM> as a traffic engineering (TE) link <NUM>/<NUM>/<NUM> that incorporates the same location information as the ODU2 interface <NUM>/<NUM>/<NUM>.

At block <NUM>, the network element <NUM>-<NUM> determines whether an Ethernet bundle <NUM>-<NUM> has been created. If the Ethernet bundle <NUM>-<NUM> has not been created ("NO"), the network element <NUM>-<NUM> registers for bundle create notification at block <NUM>. The method <NUM> returns to block <NUM>. If the Ethernet bundle <NUM>-<NUM> has been created ("YES"), the method <NUM> proceeds to block <NUM>, where the network element <NUM>-<NUM> adds the created Ethernet interface (that is, the TE link <NUM>/<NUM>/<NUM>) to the Ethernet bundle <NUM>-<NUM>. Method <NUM> ends following completion of block <NUM>.

Within the method <NUM>, a respective Ethernet bundle <NUM>-<NUM>, <NUM>-<NUM> may be associated with each network element <NUM>-<NUM>, <NUM>-<NUM>. In this way, the GMPLS operated by the network element <NUM>-<NUM>, <NUM>-<NUM> is able to signal the ODU circuit or tunnel (e.g., path <NUM>) using any port in the system. The path <NUM> may be determined according to any suitable techniques, whether now-known or later-developed. In some embodiments, the path <NUM> is terminated such that the Ethernet interface has the same location information as the ODU. The GMPLS may then add the created Ethernet interface to an Ethernet bundle <NUM>-<NUM>. In this way, the created Ethernet interface is capable of supporting the packet features as applied to the Ethernet bundle <NUM>-<NUM>. One example of configuration information for configuring the virtual interface and associated operational state information using, e.g., the network element <NUM>-<NUM>, and generally corresponding to the method <NUM>, is provided in Table <NUM>.

Within Table <NUM>, the term "<snmp-ifindex-of-bundle>" is used for identifying the bundle interface at the network element <NUM>-<NUM> from the network element <NUM>-<NUM>. By using "<snmp-ifindex-of-bundle>" to identify the bundle interface instead of, for example, information associated with the tail end <X>GigE interface, the transparency of the virtual interface is maintained and the user configuration remains portable.

Establishing the virtual interface (e.g., using the Ethernet bundles <NUM>-<NUM>, <NUM>-<NUM>) between the network elements <NUM>-<NUM>, <NUM>-<NUM>, e.g., according to method <NUM>, can offer numerous benefits. First, packet features that have been applied to the virtual interface are portable in the event of dynamic changes to the optical transport network. In the case of Packet over ODU traffic, the dynamic changes may include a GMPLS re-signaling of the optical tunnels or channels. In the case of an SDN controller, the dynamic changes may include a change in the transport circuit.

The created virtual interface carries a similar identification hierarchy and resides in the same location on the network element <NUM> as the underlying transport layer interface. Thus in the event of a dynamic change to the transport layer (e.g., a change to path <NUM>) due to network failures and/or optimization processes, the new available path could be made available through a different physical entity (e.g., a different line card, rack, chassis, and so forth) within the network element <NUM>. However, if the path <NUM> is moved to a new entity, the identification of the associated packet interface would also be changed accordingly.

Absent the virtual interface, these changes in the path can cause significant complexities in migrating the path and/or transferring user configuration information or packet features from the old packet interface to the newly-created packet interface. Further, the reapplication of user configuration information would affect packet routing protocols state and data, which is more complex than simply toggling a state of the interface.

Second, using the virtual interface allows protection schemes that are included in the transport network to be supported. For example, OTN includes several protection schemes to ensure service is maintained to the destination network element(s). The protection schemes may be associated with a guaranteed switching time. For example, a "<NUM>+<NUM>" protection scheme may have a guaranteed switching time of <NUM> milliseconds (ms), a "<NUM>+R" protection scheme may have a guaranteed switching time of <NUM>, and a "<NUM>+<NUM>+R" protection scheme may have a guaranteed switching time of <NUM> or <NUM>.

The transport layer control plane is able to guarantee that these switching times are met by establishing redundant trunk paths (whether in a different shared risk link group (SRLG), node, link, etc.). The network element <NUM> then requests the data plane to replicate or switch traffic over each of the redundant trunk paths. When one path fails, data traffic may be carried seamlessly over the other paths. However, the same would not be possible for the terminated packet interface since each terminated interface provides an independent packet interface from a particular location (e.g., Rack/Slot/Port) within the network element <NUM> and follows the interface modelling identification provided by the trunk port.

Third, the traffic path may be optimized for network upgrades or other changes. When the end service is carried on an interface that follows the transport layer interface modelling, a user moving the service to another transport layer interface would require a considerable amount of reconfiguration, even where the operation is done via a SDN controller. This corresponds to a larger maintenance window for shifting services and is generally contrary to the goal of providing transparency between layers of the network elements <NUM>.

<FIG> is a diagram <NUM> illustrating an exemplary virtual interface including a protect path and/or a restore path, according to embodiments described herein. The diagram <NUM> may be used in conjunction with other embodiments described herein.

In diagram <NUM>, a working path <NUM> is an ODU2 tunnel established between ODU2 interface <NUM>/<NUM>/<NUM> and ODU2 interface <NUM>/<NUM>/<NUM>. The Ethernet interface associated with ODU2 interface <NUM>/<NUM>/<NUM> (i.e., TE <NUM>/<NUM>/<NUM>) is included in Ethernet bundle <NUM>-<NUM>, and the Ethernet interface associated with the ODU interface <NUM>/<NUM>/<NUM> (i.e., TE <NUM>/<NUM>/<NUM>) is included in Ethernet bundle <NUM>-<NUM>.

In some embodiments, the network element <NUM>-<NUM> may establish a protect path or a restore path (generically referred to as Protect/Restore Path <NUM>) to provide redundancy within the associated bundle <NUM>-<NUM>. As shown in diagram <NUM>, the Protect/Restore Path <NUM> is an ODU2 tunnel between ODU2 interface <NUM>/<NUM>/<NUM> and ODU2 interface <NUM>/<NUM>/<NUM>.

<FIG> is an exemplary method <NUM> for establishing a virtual interface having a protect path, according to embodiments described herein. Method <NUM> may be used in conjunction with other embodiments described herein, such as using method <NUM> to establish the working path with the network element.

Method <NUM> begins at block <NUM>, where the network element determines that the working path is up. At block <NUM>, the network element calculates a protect path to the destination. The protect path is distinct from the working path. At block <NUM>, the network element signals the protect path. At block <NUM>, if signaling is not successful ("NO"), the method <NUM> returns to block <NUM> and calculates a different protect path. If signaling is successful ("YES"), the method <NUM> proceeds to block <NUM> and the network element terminates the circuit. Referring to <FIG>, the network element <NUM>-<NUM> terminates the ODU2 tunnel at ODU2 interface <NUM>/<NUM>/<NUM> of network element <NUM>-<NUM>. At block <NUM>, the network element creates the Ethernet interface. In some embodiments, the Ethernet interface created by the network element incorporates the same location information as the associated ODU2 interface.

At block <NUM>, the network element determines whether an Ethernet bundle has been created. If the Ethernet bundle has not been created ("NO"), the network element registers for bundle create notification at block <NUM>. The method <NUM> returns to block <NUM>. If the Ethernet bundle has been created ("YES"), the method <NUM> proceeds to block <NUM>, where the network element adds the created Ethernet interface to the Ethernet bundle. Referring to <FIG>, the network element <NUM>-<NUM> adds the created Ethernet interface (TE link <NUM>/<NUM>/<NUM>) to the Ethernet bundle <NUM>-<NUM>. At block <NUM>, the network element designates the newly-created path as the protect path. Method <NUM> ends following completion of block <NUM>.

Adding the protect path to the virtual interface provides a "<NUM>+<NUM>" protection scheme, such traffic would continue to flow through the virtual interface should a failure of either the working path or the protect path occur. Additionally, after the second Ethernet interface corresponding to the protect path is added to the Ethernet bundle, the Ethernet bundle will have twice the capacity. One example of configuration information for the virtual interface and associated operational state information, and generally corresponding to the method <NUM>, is provided in Table <NUM>.

<FIG> is an exemplary method <NUM> for establishing a virtual interface having a restore path, according to embodiments described herein. Method <NUM> may be used in conjunction with other embodiments described herein, such as using method <NUM> to establish the working path and/or using method <NUM> to establish the protect path.

Referring to <FIG> and <FIG>, the method <NUM> begins at block <NUM>, where the network element determines that the working path <NUM> or the protect path <NUM> fails. At block <NUM>, the network element calculates a restore path <NUM> to the destination. The restore path <NUM> is distinct from the working path <NUM> and the protect path <NUM>. At block <NUM>, the network element signals to create the restore path <NUM>. At block <NUM>, if signaling is not successful ("NO"), the method <NUM> returns to block <NUM> and calculates a different restore path <NUM>. If signaling is successful ("YES"), the method <NUM> proceeds to block <NUM> and the network element terminates the circuit. In diagram <NUM> of <FIG>, restore path <NUM> includes an ODU2 tunnel, and the network element <NUM>-<NUM> terminates the ODU2 tunnel at ODU2 interface <NUM>/<NUM>/<NUM> of network element <NUM>-<NUM>. At block <NUM>, the network element creates the Ethernet interface. In some embodiments, the Ethernet interface created by the network element incorporates the same location information as the ODU2 interface associated with the restore path <NUM>.

At block <NUM>, the network element determines whether an Ethernet bundle has been created. If the Ethernet bundle has not been created ("NO"), the network element registers for bundle create notification at block <NUM>. The method <NUM> returns to block <NUM>. If the Ethernet bundle has been created ("YES"), the method <NUM> proceeds to block <NUM>, where the network element adds the created Ethernet interface to the Ethernet bundle. Referring to <FIG>, the network element <NUM>-<NUM> adds the created Ethernet interface (TE link <NUM>/<NUM>/<NUM>) to the Ethernet bundle <NUM>-<NUM>. At block <NUM>, the network element designates the newly-created path as the restore path. Method <NUM> ends following completion of block <NUM>.

Adding the restore path to the virtual interface provides a "<NUM>+R" or a "<NUM>+<NUM>+R" protection scheme, depending on whether the virtual interface also includes a protect path <NUM>. In some embodiments, the restore path <NUM> is signaled responsive to determining that the working path <NUM> or the protect path <NUM> fails. Once the working path <NUM> or the protect path <NUM> is again operational, the restore path <NUM> may be deleted and removed from the bundle.

<FIG> is an exemplary method <NUM> for deleting a protect path and/or a restore path within a virtual interface, according to embodiments described herein. Method <NUM> may be used in conjunction with other embodiments described herein, such as deleting a protect path established using method <NUM> and/or deleting a restore path established using method <NUM>.

Method <NUM> begins at block <NUM>, where the network element receives an input to delete a path. At block <NUM>, the network element deactivates the Ethernet interface associated with the path to be deleted. At block <NUM>, the network element determines whether the deactivation was successful. If the deactivation was not successful ("NO"), the method <NUM> returns to block <NUM>. If the deactivation was successful, the method <NUM> proceeds to block <NUM>. At block <NUM>, if the network element determines that an Ethernet bundle was created ("YES"), the method <NUM> proceeds to block <NUM> and removes the Ethernet interface from the Ethernet bundle. If the removal is not successful at block <NUM> ("NO"), the method <NUM> returns to block <NUM>. If the removal is successful ("YES"), or if no Ethernet bundle was created at block <NUM> ("NO"), the method <NUM> proceeds to block <NUM> and the network device removes the termination <NUM>. The network device deletes the path at block <NUM>. Method <NUM> ends following completion of block <NUM>.

In the preceding, reference is made to embodiments presented in this disclosure. However, the scope of the present disclosure is not limited to specific described embodiments. Instead, any combination of the described features and elements, whether related to different embodiments or not, is contemplated to implement and practice contemplated embodiments. Furthermore, although embodiments disclosed herein may achieve advantages over other possible solutions or over the prior art, whether or not a particular advantage is achieved by a given embodiment is not limiting of the scope of the present disclosure. Thus, the preceding aspects, features, embodiments and advantages are merely illustrative and are not considered elements or limitations of the appended claims except where explicitly recited in a claim(s).

Accordingly, aspects may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system. " Furthermore, aspects may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

In the context of this document, a computer readable storage medium is any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus or device.

Aspects of the present disclosure are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments presented in this disclosure.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality and operation of possible implementations of systems, methods and computer program products according to various embodiments. In this regard, each block in the flowchart or block diagrams may represent a module, segment or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s).

Claim 1:
A method performed by a network element coupled with an optical transport network, the method comprising:
calculating (<NUM>) an optical path from a first packet-terminated optical interface of the network element to a second packet-terminated optical interface of a destination network element coupled with the optical transport network;
signaling (<NUM>), via a packet-terminated circuit, the optical transport network to create the optical path;
upon successful creation of the optical path, terminating (<NUM>) the packet-terminated circuit;
creating (<NUM>) an Ethernet interface corresponding to the first packet-terminated optical interface;
adding (<NUM>) the Ethernet interface to an Ethernet bundle interface; and communicating across the optical path using addressing of the Ethernet bundle interface.