Patent Publication Number: US-11032378-B2

Title: Decoupled control and data plane synchronization for IPSEC geographic redundancy

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims the benefit under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 62/512,895 entitled “DECOUPLED CONTROL AND DATA PLANE SYNCHRONIZATION FOR IPSEC GEOGRAPHIC REDUNDANCY,” filed on May 31, 2017, the content of which is incorporated by reference herein. 
    
    
     TECHNICAL FIELD 
     The present disclosure relates to systems and methods for controlling control plane and data plane traffic in a mobile network, and in particular systems and methods for decoupling control and data plane synchronization for Internet Protocol Security (IPsec) geographic redundancy. 
     BACKGROUND 
     Stateful geographic redundancy for network elements such as VPN gateways and ePDG, that terminate IPsec tunnels, requires synchronization of a large amount of long-lasting, per-tunnel state information between the active and standby nodes. During a network anomaly, for example, an active node may fail and the state of the user&#39;s session must be recovered at a standby node, which switches to an active role. When the standby node switches to an active role, it needs to program and activate various control and data-path functionalities as quickly as possible. For a security gateway with large number of active tunnels, this programming phase can be inefficient and can take several seconds, leading to extended packet losses from which applications might not be able to recover. In some existing recovery solutions, standby nodes may store both control plane and data plane information. During a network anomaly, both control plane and data plane functionality must be programmed and activated, leading to long recovery times. On the other hand, maintaining “live” session information for both control plane and plane at a standby node is resource intensive for processors and memory. Other recovery solutions employ a prioritization scheme that may recover certain prioritized sessions before recovering other, lower priority sessions, thus minimizing the impact of long recovery times. IPsec state synchronization techniques that are explicitly designed for and take advantage of cloud-native or control/data plane separation architectures to improve efficient recovery and minimize interruptions to user plane activity are lacking. 
    
    
     
       BRIEF DESCRIPTION OF FIGURES 
       Various objectives, features, and advantages of the disclosed subject matter can be more fully appreciated with reference to the following detailed description of the disclosed subject matter when considered in connection with the following drawings, in which like reference numerals identify like elements. 
         FIG. 1  is a system diagram showing a geo-redundant networked system, according to some embodiments. 
         FIG. 2  is a system diagram showing a geo-redundant networked system, according to some embodiments. 
         FIG. 3  is a diagram showing an active geo-node and a standby geo-node, according to some embodiments. 
         FIG. 4  is a flowchart showing a transition between active and standby states, according to some embodiments. 
         FIG. 5  is a flowchart showing data plane and control plane state replication and switchover, according to some embodiments. 
     
    
    
     DETAILED DESCRIPTION 
     The present disclosure describes systems and methods for explicitly decoupling control-plane state synchronization from data-plane state synchronization. A subset of the state parameters synchronized between active and standby nodes are essential for data-path packet processing functions, i.e. encryption and decryption of packets, whereas the remainder of the state parameters are needed for control-plane functions such as rekeying of phase-1 and phase-2 tunnels, dead-peer detection, SPI assignment, Mobility and Multihoming Protocol (“MOBIKE”) support, etc. As tunnels are added on the active node, both data-plane and control-plane state information related to the tunnels can be synchronized to their respective standby locations in the network. Standby nodes will receive information related to the active tunnel control plane and active data plane from the active node. This information can be used by the standby node to replicate the tunnel that is at the active node. In some embodiments (e.g. cloud native applications), the control plane data can be stored in an external database. 
     In some embodiments only the data-plane component is programmed so that it can start processing packets as soon as a switchover is triggered. The information necessary to program and activate the control plane component is stored until the switchover is triggered. Programming can mean establishing a data channel that is immediately capable of processing data even though the data channel may not necessarily be actively processing data yet. Once the data-plane component is programmed at the standby node, the standby node is ready to process data associated with the active tunnel in the event the active node becomes unavailable. 
     In some embodiments, a switchover can be triggered by a network-anomaly. For example, a standby node can detect that the active node is no longer active by monitoring interfaces (e.g. BGP peering interfaces) connected to the standby node. The standby node can infer that the active node is no longer active based on the communication link with the active node. For example, if the activity on the link indicates that the active node is inactive, the standby node can infer that a network anomaly has occurred and begin the switchover process. The standby node can also detect that the active node is no longer active if it receives traffic that would otherwise be handled by the primary node. In some embodiments, the standby node can detect that the primary node is inactive by monitoring route weights managed by decentralized route optimization techniques. For example, an active primary node may report that its route cost is 5, while a standby node may report that its route cost is 10. In this first state, traffic is routed through the active primary node because its route cost is lower than the standby node. If the primary node is no longer active, its route cost may become 100. In this scenario, the standby node becomes the preferred route because its route cost is lower. In some embodiments, a switchover can be manually triggered (e.g. by direct communication via the EMS). After a switchover is triggered, the control-plane state can then be retrieved from the database and programmed on the standby control-node. Since the data-plane component was already programmed, data-packet processing can begin immediately and is not be delayed by the amount of time need to restore the control plane component. That is, control plane resources can be recovered at the standby node after the standby node begins processing data packets over the data-plane. Control plane information (e.g. encryption keys, transforms, and other information on how to decrypt control packets) may be compressed and/or stored during the standby state which also prevents control-plane resources from being wasted due to allocation and blocking before they are really used. 
     Also, as described in more detail below, the decoupled synchronization described herein fits naturally in a control/data plane split architecture where there might be one or more control plane nodes separate from one or more data-plane nodes. In such an architecture, all the standby data-plane nodes can always be kept ready by programming them with the cryptographic keys needed for IPsec operation. Thus, the switchover time is only limited by the routing convergence time as all elements of the distributed data plane would be ready to process the incoming IPsec packets. In addition, the stack and resources (e.g. storage and processing resources) on the control-plane node(s) can be engaged only when they are actually needed. The state information can be stored until then in a database. In some embodiments, the state information can, be stored on an external database. 
     One advantage of a decoupled synchronization approach is that it can naturally fit and scale to the needs of a control-plane/user-plane split architecture. In some embodiments, the separation of control plane and data plane information allows one control plane to be associated with more than one data plane. In deployments with one control-plane element and several data-plane elements, only data-plane state needs to be synchronized to the data-plane elements. In addition, since data-plane state and control-plane state is needed at different points of time during a network-anomaly triggered switchover between standby to active roles, resources can be conserved, and session restoration time can be improved. The control-plane state can be stored in an external database until the state is actually needed. In some embodiments, the control plane information is stored in a common database that is accessible to both the primary and standby nodes. In this example, the primary node may store the control plane information at the common database. In a switchover scenario, the standby database can retrieve the control plane information that was previously stored by the primary node in the common database. In another embodiment, the standby node is configured to receive the control plane information from the primary node and can store it either locally or in an external database. The external database may be any database that is accessible to the standby node. 
       FIG. 1  is a system diagram showing a geo-redundant networked system, according to some embodiments. In this example, a secure connection is being established between network nodes (VePDG  102  and VePDG  104 ) and a user over an untrusted WiFi network (WLAN  108 ). The secure connection is used to bridge a connection over the untrusted connection with WLAN  108 . The geo-redundant networked system includes primary virtualized Evolved Packet Data Gateway (VePDG)  102 , standby VePDG  104 , packet data network gateway (PGW)  106 , wireless local area network (WLAN)  108 , other mobile network modules  110 , mobile device  112 , element management system (EMS)  114 , and geolink  116 . It should be understood that the example illustrated in  FIG. 1  using VePDG nodes is non-limiting. This system can also be implemented in any other system node that provides an IPSec connection or similar secure connection. For example, the primary and standby nodes can be VPN or enterprise VPN nodes that provide VPN functionality over IPsec tunnels. It should also be understood that the methods and systems described herein may apply to future generations of communications networks. For example, the methods and systems described herein can be applied to 5G cellular networks. In one such embodiment, the primary and standby nodes can comprise N3IWF nodes. 
     VePDG  102  is a gateway for voice and data traffic using untrusted (e.g. unencrypted) access (e.g. open WiFi or the equivalent). VePDG  102  connects to a mobile device  112  via SWu, which is a logical interface toward user equipment (UE). VePDG  102  communicates with an access network (e.g., WLAN  108 ) through SWn. VePDG  102  is the primary node and remains active until such time its responsibilities need to be transferred to a backup node. This transfer can result from a failure or other error associated with VePDG  102 . It can also occur as the result of an instruction from an operator or the EMS  114 . VePDG  102  can track states of both the control plane and the data plane. 
     VePDG  104  is a gateway for voice and data traffic using untrusted (e.g. unencrypted) access (e.g. open WiFi or the equivalent). VePDG  104  connects to a mobile device  112  via SWu, which is a logical interface toward user equipment (UE). VePDG  104  communicates with an access network (e.g., WLAN  108 ) through SWn. VePDG  104  is a secondary node and remains in standby until such time it needs to take over the responsibilities of an active node. This transfer can result from a failure or other error associated with a primary node. It can also occur as the result of an instruction from an operator or the EMS  114 . VePDG  104  can track states of both the control plane and the data plane. 
     In some embodiments, the states are associated per-geo node. They are effectively “active” (e.g., the node owns the current control and data plane user sessions) or “standby” (e.g., the node is told the current control and data plane sessions). The transition between states according to some embodiments is shown and described in  FIG. 4  and its accompanying text. 
     VePDG  102  and VePDG  104  can also communicate with PGW  106  via S2b, which is an interface toward PGW. 
     VePDG  102  and VePDG  104  can also communicate with other mobile network modules  110  via SWm, which is an interface toward 3GPP AAA server. Other mobile network modules can include a Charging Function (CGF)/Online Charging System (OCS)/Offline Charging System (OFCS) (for charging), a Policy and Charging Rules Function (PCRF) (for policy), and a Diameter Routing Agent (DRA) or Authentication, Authorization and Accounting (AAA) (for user authentication). 
     EMS  114  can be used to guarantee configuration consistency between the geo-graphically separate nodes. In some embodiments, this assures that the standby geo-node has all of the network connectivity and configuration necessary to take over the control and data plane sessions from the active geo-node. For example, network connectivity and configuration can include details about hostname, IP addresses, port numbers, and name spaces of local entities as well as external servers. In some embodiments, the consistency can be guaranteed by the operator via other mechanisms. For example, an operator can manually log in to the command line interface of the active and standby nodes to configure them identically. 
     Geolink  116  is a communication protocol between VePDG  102  and VePDG  104  to transfer and to synchronize state information. In some embodiments, Geolink  116  can comprise a proprietary communication protocol that simulates a replicated database. For example, the Geolink  116  can be implemented using a proprietary messaging mechanism that communicates information from one process to one or more processes. For example, the messaging mechanism can use a separate channel for each unique category of information. In an IPSec connection, two separate channels can be used to communicate information from an active geo-node to a standby geo-node: one for control plane information and one for data plane information. 
       FIG. 2  is a system diagram showing a geo-redundant networked system, according to some embodiments. In this example, a secure connection is established between the network nodes (primary and standby) and an enterprise network that is accessible over the Internet. Since the Internet connection may be insecure, the secure connection is used to bridge the untrusted connection over the Internet. The geo-redundant networked system includes primary node  202 , standby node  204 , Enterprise Network  206 , Radio Access Network (RAN)  208 , other mobile network modules  210 , mobile device  212 , element management system (EMS)  214 , and geolink  216 . The RAN  208  could be any other access network. For example, RAN  208  could also be a WLAN connection. Primary node  202  and Standby node  204  can be any of an SAEGW, PGW, GDSN, or similar network node. 
     Primary node  202  is a gateway for communicating (via e.g. an SGi interface) with an enterprise network that is accessible over the Internet. Primary node  102  connects to a mobile device  212  via S1-U, which is a logical interface toward user equipment (UE). Primary node  202  also communicates with an access network (e.g., RAN  208 ) S1-U. Primary node  202  remains active until such time its responsibilities need to be transferred to a backup or standby node. This transfer can result from a failure or other error associated with primary node  202 . Primary node  202  can track states of both the control plane and the data plane. 
     Standby node  204  is a gateway for communicating (via e.g. an SGi interface) with an enterprise network that is accessible over the Internet. Standby node  104  connects to a mobile device  212  via S1-U, which is a logical interface toward user equipment (UE). Standby node  204  communicates with an access network (e.g., RAN  208 ) through S1-U. Standby node  104  is a secondary node and remains in standby until such time it needs to take over the responsibilities of an active node. This transfer can result from a failure or other error associated with a primary node. Standby node  104  can track states of both the control plane and the data plane. Primary node  102  and Standby node  104  can also communicate with an Enterprise Network  206  via SGi, which is an interface toward the Enterprise Network. 
       FIG. 3  is a diagram showing an active geo-node and a standby geo-node, according to some embodiments. In some embodiments, the active geo-node  302  can correspond to VePDG  102  and the standby geo-node  312  can correspond to VePDG  104 . In other embodiments, active geo-node  302  and standby geo-node  312  can correspond to primary node  202  and standby node  204 , respectively. 
     The active geo-node  302  includes an active control plane  304  and active data plane  310 . The active geo-node  302  communicates with standby geo-node  312  via a pathway defined by geo-server endpoint  308 , geo-link  322 , and geo-client endpoint  318 . Information sent over the pathway includes geo-redundancy control information (between geo-redundancy managers), user control data (from active database on active geo-node to standby database on standby geo-node), and user data plane state (from active database on active geo-node to active database on standby geo-node). The standby geo-node  312  may store any user control data received from the active geo-node  302  in control plane state database  314 . In some embodiments (not shown), control plane database may be external to the standby geo-node  312 . 
       FIG. 4  is a flowchart showing a transition between active and standby states, according to some embodiments. 
     From a transitional perspective, the data planes on both the active and standby geo-nodes are both actually active from an operational perspective. In other words, the standby geo-node is as capable of handling user data as the active geo-node. In some embodiments, this is referred to as hot-staging of data resources. The control plane, on the other hand, transitions from “standby” to “state recovery”, and then from “state recovery” to “active”. During the transitions, control plane interruptions may occur. Not until the state is active does full control plane functionality recover.
         Geo Node: Standby→active   Data plane: No transition   Control plane: Standby→state recovery→active       

     In some embodiments, when a node transitions from active to standby, user data is purged and starts fresh again (as if the standby geo-node were coming up for the first time). The old data is purged and then is synchronized with the active geo-redundant node to recover the current state data. Note that the data plane transitions to active (e.g., ready to handle data) while the control plane is in standby (e.g., control data is in a database).
         Geo Node: Active→standby   Data plane: active→state purge→synchronization→active   Control plane: active→state purge→synchronization→standby       

       FIG. 5  is a flowchart showing data plane and control plane state replication and switchover, according to some embodiments. 
     Referring to step  502 , a session is established on VePDG  102 . Referring to step  504 , data plane state is mirrored to VePDG  104  and necessary resources allocated and programmed (e.g., data-plane active state). For example, the resources include parameters such as IP address, Security Policy Indexes (SPIs), encryption and decryption algorithms and negotiated encryption and decryption keys for each IPsec session. In some embodiments, this process is repeated at a set time interval or upon receiving instruction to repeat the process. 
     Referring to step  506 , control plane state is sent to VePDG  104  and state information is stored in a database. In some embodiments, this process is repeated at a set time interval or upon receiving instruction to repeat the process. Referring to step  508 , VePDG  102  experiences failure. Referring to step  510 , Geo Manager or EMS detects failure and initiates VePDG  104  transition to active. In some embodiments, this transition to active occurs as soon as possible after the failure. In some embodiments, the transition duration depends on multiple factors (e.g., error detection, fault propagation), and in some embodiments is measured in seconds. Referring to step  512 , after VePDG  104  has been transitioned to active, packets arrive at VePDG  104  and are processed successfully (e.g., data-plane active). 
     Referring to step  514 , VePDG  104  control plane reads from the control plane database and rebuilds control plane state. From the control plane database, information about each IKE SA (security association) and IPSEC SA can be gathered. Information gathered can include remote IP address, SPIs used, encryption algorithms, time of last rekey, message sequence numbers, etc. Associations between IKE and IPSEC SA can also be built at this time. Rule lookup data-structures are also built at this time which can be very CPU intensive for a large number of tunnels. Note that no data-plane activity is required during this process and data flow is not interrupted. The control-plane rebuild process can introduce delay but will complete before the remote end to clears the tunnel due to timeout of control-plane messages. Referring to step  516 , VePDG  104  ensures control plane and data-plane consistency and transitions to Active. 
     The subject matter described herein can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structural means disclosed in this specification and structural equivalents thereof, or in combinations of them. The subject matter described herein can be implemented as one or more computer program products, such as one or more computer programs tangibly embodied in an information carrier (e.g., in a machine readable storage device), or embodied in a propagated signal, for execution by, or to control the operation of, data processing apparatus (e.g., a programmable processor, a computer, or multiple computers). A computer program (also known as a program, software, software application, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file. A program can be stored in a portion of a file that holds other programs or data, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network. 
     The processes and logic flows described in this specification, including the method steps of the subject matter described herein, can be performed by one or more programmable processors executing one or more computer programs to perform functions of the subject matter described herein by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus of the subject matter described herein can be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). 
     Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processor of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. Information carriers suitable for embodying computer program instructions and data include all forms of nonvolatile memory, including by way of example semiconductor memory devices, (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks, (e.g., internal hard disks or removable disks); magneto optical disks; and optical disks (e.g., CD and DVD disks). The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. 
     To provide for interaction with a user, the subject matter described herein can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, (e.g., a mouse or a trackball), by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well. For example, feedback provided to the user can be any form of sensory feedback, (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, speech, or tactile input. 
     The subject matter described herein can be implemented in a computing system that includes a back end component (e.g., a data server), a middleware component (e.g., an application server), or a front end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described herein), or any combination of such back end, middleware, and front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet. 
     It is to be understood that the disclosed subject matter is not limited in its application to the details of construction and to the arrangements of the components set forth in the following description or illustrated in the drawings. The disclosed subject matter is capable of other embodiments and of being practiced and carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting. 
     As such, those skilled in the art will appreciate that the conception, upon which this disclosure is based, may readily be utilized as a basis for the designing of other structures, methods, and systems for carrying out the several purposes of the disclosed subject matter. It is important, therefore, that the claims be regarded as including such equivalent constructions insofar as they do not depart from the spirit and scope of the disclosed subject matter. 
     Although the disclosed subject matter has been described and illustrated in the foregoing exemplary embodiments, it is understood that the present disclosure has been made only by way of example, and that numerous changes in the details of implementation of the disclosed subject matter may be made without departing from the spirit and scope of the disclosed subject matter, which is limited only by the claims which follow.