Patent ID: 12244466

DETAILED DESCRIPTION

In the following detailed description of the invention, numerous details, examples, and embodiments of the invention are set forth and described. However, it will be clear and apparent to one skilled in the art that the invention is not limited to the embodiments set forth and that the invention may be practiced without some of the specific details and examples discussed.

Today, there is a push to have the Radio Access Network (RAN) of a telecommunication network (e.g., a cellular network) implemented as O-RAN, a standard for allowing interoperability for RAN elements and interfaces.FIG.1illustrates an example of O-RAN architecture100, according to some embodiments. The O-RAN architecture100includes a service management and orchestration framework (SMO)110with a non-real-time RIC105, a near real-time RAN intelligent controller (RIC)115, open control plane central unit (O-CU-CP)120, open user plane central unit (O-CU-UP)125, open distributed unit (O-DU)130, open radio unit (O-RU)135, and the O-Cloud140. The O-CU-CP120, the O-CU-UP125, and the O-DU130may be collectively referred to as the managed functions120-130below.

As defined in the standard, the SMO110in some embodiments includes an integration fabric that allows the SMO to connect to and manage the RIC115, the managed functions120-130, and the O-Cloud140via the open interfaces150. Unlike these elements, the O-RU135is not managed by the SMO110, and is instead managed by the O-DU130, as indicated by the dashed line160, in some embodiments. In some embodiments, the O-RU135processes and sends radio frequencies to the O-DU130.

In some embodiments, the managed functions120-130are logical nodes that each host a set of protocols. According to the O-RAN standard, for example, the O-CU-CP120, in some embodiments, include protocols such as radio resource control (RRC) and the control plane portion of packet data convergence protocol (PDCP), while the O-CU-UP125includes protocols such as service data adaptation protocol (SDAP), and the user plane portion of packet data convergence protocol (PDCP).

The two RICs are each adapted to specific control loop and latency requirements. The near real-time RIC115provides programmatic control of open centralized units (O-CUs) and open distributed units (O-DUs) on time cycles of 10 ms to 1 second. The non-real-time RIC (non-RT RIC)105, on the other hand, provides higher layer policies that can be implemented in the RAN either via the near-RT RIC or via a direct connection to RAN nodes. The non-RT RIC is used for control loops of more than 1 second. Each RIC105or115serves as a platform on which RAN control applications execute. These applications can be developed by third-party suppliers that are different from the RIC vendors. These applications are referred to as “xApps” (for the near-RT RIC115) and “rApps” (for the non-RT RIC).

The near real-time RIC115, in some embodiments, is a logical aggregation of several functions that use data collection and communications over the interfaces155in order to control the managed functions120-130. In some embodiments, the non-real-time RIC105uses machine learning and model training in order to manage and optimize the managed functions120-130. The near RT RIC in some of these embodiments also uses machine learning.

In some embodiments, the O-Cloud140is responsible for creating and hosting virtual network functions (VNFs) for use by the RIC115and the managed functions120-130. In some embodiments, the DU is in charge of per-slot decisions of user scheduling and includes RAN scheduler that performs MAC control assistance and user-level tracing. In order to increase computing power available in the cloud (i.e., compared to base stations that typically execute the RAN functions), the RIC is implemented in one or more public and/or private cloud datacenters and implements an improved cloudified RAN scheduler in the cloud, thereby offloading these MAC control assistance and user-level tracing functions from the DU to the RIC. The interfaces155in some embodiments enable the RAN to provide inputs to the functions at the RIC, and, at least in some embodiments, receive outputs that have been computed by these functions at the RIC.

FIG.2illustrates an in-depth view of the components of both a non-real-time RIC201and a near real-time RIC202. Each of the RICs201and202includes a respective set of analytics functions210and212, and a respective set of optimization functions214and216, which are each illustrated with dashed lines to indicate they are existing components. In addition to these existing components, the near real-time optimization functions216includes two new components, the MAC control assistor220and user-level tracer222, illustrated with solid lines to visually differentiate them from the existing components. In some embodiments, these components are part of a larger MIMO component (e.g., along with the MU-MIMO UE pairer and precoder).

In some embodiments, the MAC control assistor220can include various functions such as (1) User Equipment (UE)-specific beamforming weight calculation based on UL SRS channel signal reception, (2) UE Radio Frequency (RF) condition prediction, and (3) Multi-User, Multiple Input, Multiple Output (MU-MIMO) pairing suggestion for the MAC scheduler based on the UE-specific beams. For each of these functions, some embodiments expose a report interface (that provides input data for the function to the RIC from the DU) and a control interface (that provides output data for the function to the DU from the RIC).

The user-level tracer222, in some embodiments, produces L1/L2/L3 level information related to user configuration and traffic performance. This tracing data can be used as inputs to various control algorithms, including the MAC scheduler, parameter setting, etc. The user-level tracer222can include tracing operations that can (i) track user behavior in a cell, (ii) track user RF condition, (iii) track user data traffic performance in different layers (MAC, Radio Link Control (RLC), Packet Data Convergence Protocol (PDCP)), and (iv) track user RF resource consumption.

FIG.3illustrates a more in-depth view of a MAC control assistor300of some embodiments. As illustrated, the MAC control assistor300includes a UE-specific beamforming weight calculator (BFWC)310, a UE RF condition predictor320, and a MU-MIMO pairing suggestor330. The UE-specific BFWC310in some embodiments is based on UL SRS channel signal reception. In some embodiments, the MU-MIMO pairing suggestor330is for the MAC scheduler based on the UE-specific beams.

Each of the components310-330of the MAC control assistor300includes an uplink and a downlink, as shown. For the UE-specific BWC function, some embodiments expose a report interface for an uplink Sounding Reference Signal (UL SRS) channel response matrix that is an input to the weight calculation function and a control interface for a UE-specific beamforming weight matrix. For the UE RF condition predictor function, some embodiments expose a report interface for a downlink (DL) channel condition report that is an input to the RF condition prediction and a control interface for a predicted DL channel condition (e.g., including DL SINR, PMI, and rank) for the next scheduling window. For the MU-MIMO pairing suggestion function, some embodiments expose a report interface for UE-specific beamforming weight matrix that is an input to the pairing suggestion function and a control interface for UE pairing suggestion and SINR impact assessment.

FIG.4illustrates a more in-depth view of a user-level tracer400of some embodiments. The tracer400includes multiple uplinks410and multiple downlinks415for performing tracing operations, in some embodiments. These operations produce L1/L2/L3 level information related to user configuration and traffic performance. This tracing data can be used as inputs to various control algorithms, including the MAC scheduler, parameter setting, etc. These tracing operations can (1) track user behavior in a cell, (2) track user RF condition, (3) track user data traffic performance in different layers (MAC, RLC, PDCP), and (4) track user RF resource consumption.

For these tracing operations, some embodiments expose report interfaces for the DU and/or the CU to provide various metrics to the user level tracing operations. These metrics can include selected RRC messages, MAC/RLC/PDCP traffic volume and performance, RF condition, and RF resource consumption. In some embodiments, messages over these interfaces to the RIC are triggered based on user behavior and/or periodic reporting (e.g., for traffic performance and RF condition/resource consumption).

The tracing operations track the various user data indicated above, and can provide this information either back to the RAN or to other control algorithms (e.g., other algorithms operating at the RIC). For instance, these algorithms might perform analysis on the user data performance from the user level tracing operations, determine that certain performance is inadequate, and modify how the RAN is treating the user traffic. Examples of control algorithms that can benefit from user-level tracing in some embodiments include (1) traffic steering, (2) quality of service (QoS) scheduling optimization, (3) user configuration adjustment, and (4) user behavior anomaly detection.

For all of the operations described inFIGS.3-4(i.e., the MAC scheduler functions and the user-level tracing operations), the increased computing power available to the RIC in the cloud enables more complex computations without excessive latency. For instance, some or all of these operations can be performed at the RIC using machine learning (e.g., using machine-trained networks, etc.).

FIG.5illustrates another view of the O-RAN architecture of some embodiments, with a more in-depth view of the near real-time RIC. The architecture500includes an SMO505with a non-real-time RIC510, a distributed near real-time RIC515, and E2 nodes520(e.g., O-DU and/or O-CU nodes). The distributed near real-time RIC515includes messaging infrastructure540, a set of services (e.g.,550,552,554, and556), a shared data layer560, a database570, and a set of termination interfaces (e.g.,580,582, and584). As shown, a set of embedded apps (e.g.,530,532, and534) uses this distributed near RT RIC. As further described below, the distributed near RT RIC515is implemented by multiple RICs executing on multiple host computers in some embodiments.

As shown, the set of services include conflict mitigation services550, app subscription management services552, management services554, and security services556. Additionally, the set of termination interfaces include O1 termination interface580connecting the SMO to the near real-time RIC, A1 termination interface582connecting the non-real-time RIC to the near real-time RIC, and E2 termination interface584connecting the E2 nodes to the near real-time RIC. Each of the apps, in some embodiments, is representative of the various functions of the RIC that use data sent from the E2 nodes520. For example, app530may correspond to the UE-specific BFWC310of the MAC control assistor300, app532may correspond to the UE RF condition predictor320of the MAC control assistor300, etc.

In some embodiments, the objective of the framework500is to offload near real-time functions that are computation-intensive, and provide results back to the O-DU (e.g., via the E2 interface with E2 nodes520). The results, in some embodiments, can be used to assist or enhance the real-time decision in the MAC layer. Three example use-cases for the MAC control assistance framework, each example specific to a different component of the MAC control assistor (e.g., the UE-specific BFWC, the UE RF condition predictor, and the MU-MIMO pairing suggestor), and one use-case example for the user-level tracer, will be described below.

The first example use-case is specific to the UE-specific beamforming weight calculation based on UL SRS signal reception component of the MAC control assistance framework (e.g., component310of the MAC control assistor300). In some embodiments of this use-case, the input metrics can include multiple options based on UL SRS, such as raw SRS received data, and an SRS channel responses matrix from a channel estimate.

The algorithm for producing output metrics, in some embodiments, evaluates the optimal beam-forming weights to reach the user. Some embodiments use traditional signal processing algorithms that are based on channel models. Alternatively, or conjunctively, machine-learning based algorithms that utilize raw data inputs are used, which require feedback from the DU in the E2 nodes520.

In some embodiments, the output metrics resulting from the algorithm include a beam-form weight (BFW) matrix for the user. In some embodiments, the BFW could also be mapped to a beam index from a pre-designed beam set. The DU in some embodiments uses the matrix to control the MIMO antenna array gain/phasing in the RU (e.g., the O-RU135in the architecture100) for user data transmission and reception.

The second use-case example is specific to the UE RF condition predictor component of the MAC control assistance framework (e.g., component320of the MAC control assistor300). For this second use-case, the input metrics include at least a channel report from the UE, such as Wideband or Subband CQI/PMI/RI for DL, or SRS for UL, according to some embodiments. The input metrics of some embodiments can also opt to include supportive information such as UE distance, UE positioning, etc.

In some embodiments, the app algorithm for this second use-case is meant to predict the UE's RF condition based on the observation. Some embodiments utilize traditional signal processing algorithms based on channel and mobility models. Alternatively, or conjunctively, some embodiments also use machine learning based algorithms using data inputs and potentially other factors, such as site layout (which requires feedback from the DU).

The output metrics for this use-case, in some embodiments, include the predicted channel condition of the user for the next scheduling window, as well as predicted downlink and uplink SINR, a precoding matrix (e.g., if applicable), and SU-MIMO layers. In some embodiments, these output metrics are used by the DU for the user link adaptation on PDCCH/PDSCH/PUSCH transmissions.

The third use-case example is specific to the MU-MIMO pairing suggestor to MAC scheduler component (e.g., component330of the MAC control assistor300). The input metrics for this example use case, in some embodiments, include at least the UE-specific BFW matrix and the UE RF condition estimate. Some embodiments may also include supportive metrics such as user data demand, etc., as input metrics in addition to the UE-specific BFW matrix and the UE RF condition estimate.

The app algorithm for this use-case, in some embodiments, is meant to identify users that can be paired for MU-MIMO operations. For example, some embodiments of the third use-case use traditional signal processing algorithms based on information theory and cross-channel covariance evaluation. Alternatively, or conjunctively, some embodiments use machine learning based algorithms using the data inputs, which again requires feedback from the DU.

In some embodiments, the output metrics of this third use-case can include UE pairing suggestions and an impact assessment on SINR and SU-MIMO layers. Additionally, the DU in some embodiments uses the output metrics to select users for RF scheduling, and to determine the transmission efficiencies.

An example use-case for the user-level tracer can include QoS scheduling optimization with the goal of adjusting a user's scheduling priority for an RF resource to optimize the service quality. The input for some embodiments of this use-case can include a service quality target from a user subscription. In some embodiments, the user-level tracing includes (1) tracking the user RF condition, (2) tracking the user data traffic performance in different layers (e.g., MAC/RLC/PDCP), and (3) tracking the user RF resource consumption.

In some embodiments, the app algorithm is based on the QoS target and observed user traffic performance, and can be used to determine that a user's resource allocation is insufficient. The algorithm format, in some embodiments, can be logic-based or machine learning-based. In some embodiments, the output can include a recommendation issued to the MAC scheduler to adjust the traffic priority or link adaptation in order to improve performance.

On each machine (e.g., each VM or Pod) that executes a control plane application, some embodiments configure a RIC SDK to serve as an interface between the control plane application on the machine and a set of one or more elements of the RAN. In some embodiments, the RIC SDK provides a set of connectivity APIs (e.g., a framework) through which applications can communicate with the distributed near real-time (RT) RIC implemented by two or more near real-time RICs. Examples of such applications include xApps, and other control plane and edge applications in some embodiments. In O-RAN, xApps perform control plane, monitoring and data processing operations. The discussion below regardingFIGS.6and8-20refers to control plane applications (e.g.,615,815,820,915,920, etc.). These control plane applications are xApps in an O-RAN system in some embodiments.

FIG.6illustrates deployment of RIC SDKs605on machines610that execute control plane applications615in some embodiments. As shown, one or more machines610execute on each of several host computers607in one or more datacenters. In some embodiments, the RIC SDK605on each machine610includes a set of network connectivity processes that establish network connections to the set of RAN elements (e.g., E2 nodes520, shared data layer560, management services554, SMO505, etc.) for the control plane application. The RIC SDK processes allow the control plane application on their machine to forego performing network connectivity operations. In some embodiments, the set of network connectivity processes of each RIC SDK of each machine establishes and maintains network connections between the machine and the set of RAN elements used by the control plane application of the machine, and handles data packet transport to and from the set of RAN elements for the control plane application.

The control plane application on each machine communicates with the set of RAN elements through high-level APIs620that the RIC SDK converts into low-level APIs625. In some embodiments, at least a subset of the low-level API calls625are specified by a standard specifying body. Also, in some embodiments, the high-level APIs620are made in a high-level programming language (e.g., C++), while the low-level API calls comprise low-level calls that establish and maintain network connections and pass data packets through these connections.

The set of RAN elements that the RIC SDK connects with the control plane application on its machine in some embodiments include RAN elements that are produced and/or developed by different RAN vendors and/or developers. These RAN elements include CUs630and DUs635of the RAN in some embodiments. Also, this SDK communicates with the CUs and DUs through the low-level, standard-specified E2 interface, while the control plane application on the machine uses high-level API calls to communicate with the CUs and DUs through the RIC SDK. In some embodiments, the high-level API calls specifying E2 interface operations at a high-level application layer that do not include low-level transport or network operations.

Conjunctively, or alternatively, the set of RAN elements that the RIC SDK connects with the control plane application615on its machine610include network elements of the RIC. Again, these network elements in some embodiments include RAN elements that are produced and/or developed by different RAN vendors and/or developers. These RIC elements in some embodiments include shared data layer (SDL)560, datapath input/output (I/O) elements, and application and management services552and554in some embodiments.FIG.7illustrates that some embodiments deploy several near RT RICs705to execute on several host computers to implement a distributed near RT RIC700that includes the RIC components illustrated inFIGS.5and6. In some embodiments, one RIC705executes on each host computer that also executes a control plane application615. In other embodiments, a control plane application615can execute on a host computer that does not execute a MC. For instance, in some embodiments, one or more control plane applications execute on one or more host computers that have graphics processing units (GPUs), while RICs do not execute on such host computers as they do not need the processing power of the GPUs.

Through the distributed near RT RIC, the RIC SDK also connects its control plane application to other control plane applications executing on other machines. In other words, the RIC SDK and the distributed near RT RIC in some embodiments serve as communication interface between the control plane applications. In some embodiments, the different control plane applications are developed by different application developers that use the common set of RIC APIs to communicate with each other through the distributed near RT RIC. In some of these embodiments, the distributed near RT RIC adds one or more parameters to the API calls as it forwards the API calls from one control application to the other control application.

FIGS.8-11illustrate several examples of RIC architectures in which the RIC SDK and the distributed near RT RIC establish the communication interface between control plane applications. These architectures are mutually exclusive in some embodiments, while in other embodiments two or more of these architectures are used conjunctively.FIG.8illustrates a RIC800that executes on one host computer805along with two machines810and812on which two control plane applications815and820execute. Through the RIC SDKs802and804executing on the machines810and812, the RIC800receives API calls from the CP application815and forwards the API calls to the CP application820, and passes responses to these API calls from the second CP application820to the first CP application815. It also passes API calls from the second CP application820to the first CP application815, and responses from the first CP application815to the second CP application820.

FIG.9illustrates two RICs900and901that execute on two host computer905and907along with two machines910and912on which two control plane applications915and920and two RIC SDKs902and904execute. As shown, API calls from the first CP application915to the second CP application920are forwarded through the first RIC SDK902, the first RIC900, the second RIC901and the second RIC SDK904. The second CP application's responses to these API calls to the first CP application915traverse the reverse path, from the second RIC SDK904, the second RIC901, the first RIC900, and the first RIC SDK902.

The API calls from second CP application920to the first CP application915are forwarded through the second RIC SDK904, the second RIC901, the first RIC900, and the first RIC SDK902, while responses to these API calls from the first CP application915to the second CP application920are forwarded through the first RIC SDK902, the first RIC900, the second RIC901and the second RIC SDK904.

FIG.10illustrates a RIC1000that executes on first host computer1005to connect two control plane applications1015and1020that execute on two machines1010and1012operating on two other host computers1006and1007. Through the RIC SDKs1002and1004executing on the machines1010and1012, the RIC1000receives API calls from the CP application1015and forwards the API calls to the CP application1020, and passes responses to these API calls from the second CP application1020to the first CP application1015. It also passes API calls from the second CP application1020to the first CP application1015, and responses from the first CP application1015to the second CP application1020.

FIG.11illustrates a RIC1100that executes on first host computer1105to connect two control plane applications1115and1120that execute on two machines1110and1112one of which operates on host computer1105while the other operates on host computer1106. Through the RIC SDKs1102and1104executing on the machines1110and1112, the RIC1100receives API calls from the CP application1115and forwards the API calls to the CP application1120, and passes responses to these API calls from the second CP application1120to the first CP application1115. Through these SDKs1102and1104, the RIC1100also passes API calls from the second CP application1120to the first CP application1115, and responses from the first CP application1115to the second CP application1120.

FIG.12illustrates examples of the different standard specified APIs that the distributed near RT RIC platform of some embodiments supports. As shown, the distributed near RT RIC platform1200in some embodiments uses the E2, O1, and A1 interfaces specified by the O-RAN standard specifying body. It uses the E2 APIs to communicate with the E2 O-RAN nodes, such as the O-CU-CPs1202, O-CU-UPs1204, and O-DUs1206. It also uses the A1 APIs to communicate with the non-real-time RIC platform1208, and uses the O1 APIs to communicate the SMO1210.

For each of these E2, A1, and O1 APIs, the RIC SDKs1215provide high-level counterpart APIs for the control plane applications1220that use the RIC SDKs and the distributed near RT RIC platform to communicate with the E2 nodes1202-1206, the non-real-time RIC platform1208and the SMO1210.FIG.12designates these high-level counterpart APIs for the E2, O1, and A1 interfaces with a prime sign as the E2′ API calls, O1′ API calls and A1′ API calls. These high-level counterpart APIs are not specified by a standard body, but are APIs that the RIC SDK and/or distributed near RT RIC convert into standard specified API calls.

FIG.12also shows several internal-RIC APIs for allowing the control plane applications1220to communicate with each other through the RIC SDKs and the distributed near RT RIC, and to communicate with one or more elements of the distributed near RT RIC (e.g., shared data layer (SDL)560, datapath input/output (I/O) elements, and application and management services552and554).

Enablement APIs are the APIs that are used in some embodiments to allow the control plane applications1220to communicate with each other. As described above by reference toFIGS.8-11, these APIs are passed through the distributed near RT RIC in some embodiments. In other embodiments, these APIs allow the RIC SDKs of the control plane applications to directly communicate with each other without traversing through any other components of the distributed near RT RIC. For this reason,FIG.12includes a dashed bi-directional arrow between the RIC SDKs1215of the two control plane applications1220to indicate that in some embodiments the RIC SDKs1215of these applications communicate directly with each other.

The enablement APIs in some embodiments include registration APIs, service discovery APIs as well as inter-app communication APIs. Registration APIs are used by the applications1220(e.g., xApps) to introduce themselves to other applications1220by providing their network identifiers (e.g., their network address and available L4 ports) and providing their functionality (e.g., performing channel prediction). Service discovery APIs allow control plane applications1220(e.g., xApps) to query the service directory (e.g., of the distributed near RT RIC) for other control plane applications (e.g., other xApps) that provide a particular service. The inter-app communication APIs allow the control plane applications to communicate with each other to pass along data and/or request certain operations.

Some embodiments deploy an SDL cache on the same host computer as a control plane application, and use this cache to process at least a subset of the SDL storage access requests of the control plane application. In some embodiments, the control plane application and the SDL cache operate on a machine that executes on the host computer. In other embodiments, the SDL cache operates on the same host computer but outside of the machine on which the control plane application executes. In some of these embodiments, multiple control plane applications executing on the same host computer use a common SDL cache on that host computer.

The SDL cache is part of a RIC that executes on the same host computer as the control plane application in some embodiments. In other embodiments, the SDL cache is part of the RIC SDK that executes on the same machine as the control plane application. In either of these embodiments, a synchronizing process of the RIC or the RIC SDK synchronizes the data stored in the SDL cache with the data stored in the SDL storage.

In some embodiments, the SDL storage operates on a different host computer than the host computer on which the control plane application executes, while in other embodiments at least a portion of the SDL storage operates on the same host computer on which the control plane application executes. Also, in some embodiments, the RIC or the RIC SDK forwards SDL access requests from the control plane application to the SDL storage when the RIC SDK cannot process the SDL access requests through the SDL cache. For instance, the RIC or the RIC SDK cannot process SDL access requests through the SDL cache when the SDL cache does not store data requested by the control plane application.

FIG.13illustrates embodiments in which the SDL cache1302is part of each RIC SDK1300that executes on the same machine1305as its control plane application1310. As shown, the RIC SDK1300includes a query manager132that processes SDL requests from the CP application1310and a synchronizing service1327that synchronizes the data stored in the SDL cache with the data stored in an SDL storage1350of the SDL1355of the distributed near RT RIC1330. In this example, the SDL storage1350operates on a different host computer than the host computer on which the control plane application1310executes. However, in other embodiments, at least a portion of the SDL storage1350operates on the same host computer on which the control plane application1310executes.

When the control plane application1310uses a high-level API call to read or write data to the SDL storage, the query manager1325of the RIC SDK1300first determines whether the data record being read or written is stored in the SDL cache1302. If so, the query manager1325reads from or write to this record. When this operation is a write operation, the synchronizing service1327writes the new data in real-time or on batch basis to the SDL storage1350. On the other hand, when query manager1325of the RIC SDK1300determines that the data record being read or written is not stored in the SDL cache1302, it passes the API call to the SDL layer of the distributed near RT RIC to perform the requested read or write operation. When passing this API call, the RIC SDK1300modifies the format of this call and/or modifies the parameters supplied with this call in some embodiments.

Some embodiments provide various methods for offloading operations in an O-RAN (Open Radio Access Network) onto control plane (CP) or edge applications that execute on host computers with hardware accelerators in software defined datacenters (SDDCs). For instance, at the CP or edge application operating on a machine executing on a host computer with a hardware accelerator, the method of some embodiments receives data, from an O-RAN E2 unit, for which it has to perform an operation. The method uses a driver of the machine to communicate directly with the hardware accelerator to direct the hardware accelerator to perform a set of computations associated with the operation. This driver allows the communication with the hardware accelerator to bypass an intervening set of drivers executing on the host computer between the machine's driver and the hardware accelerator. Through this driver, the application in some embodiments receives the computation results, which it then provides to one or more O-RAN components (e.g., to the E2 unit that provided the data, to another E2 unit or to another xApp).

FIGS.14-20illustrate several different embodiments for offloading O-RAN operations to CP or edge applications that have passthrough access to the hardware accelerators of their host computers. Examples of such a hardware accelerator include a graphical processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and a structured ASIC.

FIG.14illustrates an example of CP or edge applications1402that have passthrough access to hardware accelerator1450of their host computer1410to perform some or all of their computations. As shown, each application1402executes on a Pod1404, which has accelerator drivers1412with direct, passthrough access to the accelerator1450of their host computer1410. Each Pod1404operates within (i.e., execute on) a VM1406, which, in turn, executes over a hypervisor1408of the host computer.

In some embodiments, a Pod is a small deployable unit of computing that can be created and managed in Kubernetes. A Pod includes a group of one or more containers with shared storage and network resources, and a specification for how to run the containers. In some embodiments, a Pod's contents are always co-located and co-scheduled, and run in a shared context. A Pod models an application-specific logical host computer; it contains one or more application containers that are communicate with each other. In some embodiments, the shared context of a Pod is a set of an operating system namespaces (e.g., Linux cgroups). Within a Pod's context, the individual applications may have further sub-isolations applied.

Each Pod's accelerator driver1412has direct accesses to the hardware accelerator1450, and this access bypasses the hardware accelerator drivers1414and1416of the VM1406and the hypervisor1408. In some embodiments, the hypervisor1408executes over an operating system (not shown) of the host computer1410. In these embodiments, the direct access of each Pod's accelerator driver1412to the hardware accelerator1450also bypasses the hardware accelerator driver of the operating system.

To communicate with the hardware accelerator, each application1402in some embodiments communicates through the RIC SDK1430executing on its Pod. For instance, in some embodiments, each application1402uses high-level APIs of the RIC SDK1430to communicate with the hardware accelerator1450. The RIC SDK1430then converts the high-level APIs to low-level APIs that are needed to communicate with machine's driver1412, which, in turn, relays the communication to the hardware accelerator1450. The low-level APIs are provided by a first company associated with the sale of the hardware accelerator1450, while the RIC SDK1430is provided by a second company associated with the distribution of the RIC SDK1430. In some embodiments, the low-level APIs used by the RIC SDK1430are APIs specified in an API library1432associated with the hardware accelerator1450.

FIG.15illustrates a process1500that implements the method of some embodiments. The process1500is performed in response to an O-RAN component directing a CP or edge application to perform an operation that requires the application to use a hardware accelerator of its host computer. This process1500will be described below by reference toFIG.16, which illustrates the application1402performing an operation based on data received from an E2 node1650.

As shown inFIG.15, the process1500starts when the application1402(at1505) receives a data from an O-RAN E2 unit1650executing on the host computer1610. In some embodiments, the application1402subscribes for data from the E2 unit1650, and the data received at1505is in response to this subscription. This subscription is made through the distributed near RT RIC in some embodiments. The host computers1410and1610of the application1402and the E2 unit1650operate in one SDDC in some embodiments. In other embodiments, these two host computers1410and1610operate in two different physical locations. For example, the host computer1410operates in a first location, while the host computer1610operates at a second location close to a cell site of the O-RAN. In some embodiments, the second location does not have computers with hardware accelerators that perform complex operations including the received operation.

The application1402receives (at1505) the data from the E2 unit1650through (1) the distributed near RT RIC1680formed by near RT RICs1640and1645executing on host computers1410and1610, and (2) the RIC SDK1430executing on its Pod1404. The application1402then uses (at1510) the hardware accelerator1450to perform a set of computations associated with the operation.

To communicate with the hardware accelerator1450, the application1402uses high-level APIs provided by the RIC SDK1430. The RIC SDK1430then converts the high-level APIs to low-level APIs specified in the API library1432associated with the hardware accelerator1450. These low-level APIs are then communicated to the hardware accelerator1450by the Pod's driver1412through its direct, passthrough access to the accelerator1450, which bypasses the drivers1414and1416of the VM1406and hypervisor1408. Through this driver1412, the APIs specified in the API library1432, and the RIC SDK1430, the application1402also receives the results of the operations (e.g., computations) performed by the hardware accelerator1450.

The application1402provides (at1515) the result of its operation to one or more O-RAN components, such as the E2 unit1650that provided the data that started the process1500or the SDL storage. This result is provided through the RIC SDK1430and the distributed near RT RIC1680. In other embodiments, the application1402(through the RIC SDK1430) provides the results of its operation to one or more other applications (applications other than the E2 unit that provided the data for which the application performed its operation) operating on another O-RAN E2 unit or machine executing on the same host computer or on another host computer as the application that uses the hardware accelerator1450to perform the operation. The process1500ends after1515.

Other embodiments use the passthrough access for the O-RAN control or edge application in other deployment settings. For instance,FIG.17illustrates another example of CP or edge applications1702that have passthrough access to a hardware accelerator1750of their host computer1710to perform some (or all) of their computations. In this example, each application1702(1) executes on a Pod1704that executes on a VM1706, and (2) uses the accelerator driver1712of this VM1706which has direct, passthrough access to the accelerator1750of its host computer1710. The VM1706executes over a hypervisor1708operating on the host computer1710. The VM's accelerator driver1712bypasses the hardware accelerator drivers1716of the hypervisor1708. In some embodiments, the hypervisor1708executes over an operating system (not shown) of the host computer1710. In these embodiments, the direct access of the VM's accelerator driver1712to the hardware accelerator1750bypasses the hardware accelerator driver of the operating system.

To use the hardware accelerator1750, each application1702in some embodiments uses high-level APIs of the RIC SDK1730(executing on its Pod1704) to communicate with the hardware accelerator1750. The RIC SDK1730converts the high-level APIs to low-level APIs that are needed to communicate with VM's driver1712, which, in turn, relays the communication to the hardware accelerator1750. In some embodiments, the low-level APIs used by the RIC SDK1730are APIs specified in an API library1732associated with the hardware accelerator1750. This API library1732is part of the driver interface of the VM1706.

FIG.18illustrates yet another example of CP or edge applications1802that has passthrough access to a hardware accelerator1850of their host computer1810to perform some or all of their computations. In this example, each application1802(1) executes on a VM1804that executes on a hypervisor1806operating on the host computer1810, and (2) uses the accelerator driver1812of its VM1804, which has direct, passthrough access to the accelerator1850of its host computer1810.

The VM's accelerator driver1812bypasses the hardware accelerator drivers1816of the hypervisor1806. In some embodiments, the hypervisor1806executes over an operating system (not shown) of the host computer1810. In these embodiments, the direct access of the VM's accelerator driver1812to the hardware accelerator1850bypasses the hardware accelerator driver of the operating system.

To use the hardware accelerator1850, each application1802in some embodiments uses high-level APIs of the RIC SDK1830(executing on its Pod1804) to communicate with the hardware accelerator1850. The RIC SDK1830converts the high-level APIs to low-level APIs that are needed to communicate with the VM's driver1812, which, in turn, relays the communication to the hardware accelerator1850. In some embodiments, the low-level APIs used by the RIC SDK1830are APIs specified in an API library1832associated with the hardware accelerator1850. This API library1832is part of the driver interface of the VM1806.

One of ordinary skill will realize that the passthrough access for the O-RAN control or edge application is used in other deployment settings in other embodiments. For instance, instead of operating on Pods, the applications in other embodiments operate on containers. These embodiments then use the hardware accelerator drivers of their Pods or VMs to have passthrough access to the hardware accelerators for the control or edge application. In some of these embodiments, the control or edge application communicates with the hardware accelerator through its associated RIC SDK, and communicates with other O-RAN components (to receive data and to provide results of its processing of the data) through its associated RIC SDK and the distributed near RT RIC connecting the O-RAN components and the application. In some embodiments, the control or edge application in these embodiments performs processes similar to process1500ofFIG.15.

The above-described direct, passthrough access to hardware accelerators is quite beneficial for O-RANs. The RIC is all about decoupling the intelligence that used to be embedded within the RAN software (CU and DU) and moving it to the cloud. One benefit of this is to use more advanced computing in the cloud for the xApp and edge operations (e.g., for ML, deep learning, reinforcement learning for control algorithms, etc.). A DU close to a cell site typically cannot run advance computations because it would not be economically feasible to put GPUs at each cell site as network cap X will be very high.

By using the hardware accelerator (GPU, FPGAs, eASICs, ASICs) in the SDDC, some embodiments run complex control algorithms in the cloud. Examples of such xApps include Massive MIMO beam forming and Multi-user (MU) MIMO user pairing, which were described above. Generally, any xApp whose computations can benefit from massive parallelization would gain the benefit of GPU or other accelerators. The use of ASICs is beneficial for channel decoding/encoding (turbo encoding, LDPC encoding, etc.). In some embodiments, the RIC is typically on the same worker VM as xApps. However, in other embodiments, the RICs executes on a different host computer so that more xApps that need GPUs and other hardware accelerators can run on the hosts with the GPUs and/or other hardware accelerators.

FIG.19illustrates a process that some embodiments use to deploy O-RAN applications with direct, passthrough access to the hardware accelerators of their host computers. To install an application on a host computer, the process1900selects (at1905) a set of one or more installation files that includes a description for configuring passthrough access for the application to a hardware accelerator of the host computer. In some embodiments, the set of files includes one description file that specifies direct, passthrough access for the application to the hardware accelerator of its computer.

The process1900uses (at1910) the set of installation files to configure, based on the description relating to the passthrough access, a program executing on the host computer to pass calls from a particular hardware accelerator driver associated with the application to the hardware accelerator without going through an intervening set of one or more drivers for the hardware accelerator that executes on the host computer between the particular hardware accelerator driver and the hardware accelerator. This configuration allows the application to bypass the intervening set of drivers when directing the hardware accelerator to perform operations for the application and to receive the results of the operations from the hardware accelerator.

The program that is configured at1910in some embodiments is the host's operating system, while in other embodiments it is a hypervisor executing on the host computer. In still other embodiments, the program is a virtual machine (VM) and the application operates on a Pod or container that executes on the VM. The process1900completes (at1915) the installation of the application by processing the remaining set of installation files selected at1905, and then ends. In other embodiments, the process1900performs the configuration of the program as its last operation instead of as its first operation at1910. In still other embodiments, it performs this configuration as one of its intervening installation operations.

Before performing the selection and configuration, the deployment process of some embodiments identifies the host computer from several host computers as the computer on which the application should be installed. The process in some embodiments identifies the host computer by determining that the application requires a hardware accelerator, identifying a set of host computers that each comprise a hardware accelerator, and selecting the host computer from the set of host computers. The process selects the host computer by (1) determining that the application will need to communicate with a set of one or more other applications that execute on the selected host computer, and (2) selecting the host computer as the set of other applications simultaneously executes on the host computer. This installation of the application with the set of other applications on the selected host computer reduces communication delay between the application and the set of other applications.

Some embodiments have the hardware accelerator drivers of the O-RAN control or edge applications communicate with virtualized hardware accelerators that are offered by an intervening virtualization application (e.g., hypervisor) that executes on the same host computer as the application. For instance, the method of some embodiments deploys a virtualization application on a host computer for sharing resources of the host computer among several machines executing on the host computer. This computer has a first set of one or more physical hardware accelerators.

The method deploys several applications on several machines to perform several O-RAN related operations for a set of O-RAN components. Through the virtualization application, the method defines a second set of two or more virtual hardware accelerators that are mapped to the first set of physical hardware accelerators by the virtualization application. The method assigns different virtual hardware accelerators to different applications. The method also configures the applications to use their assigned virtual hardware accelerators to perform their operations.

In some embodiments, the deployed machines are Pods, and the applications are deployed to execute on the Pods. At least two Pods execute on one VM that executes above the virtualization application. This VM includes a hardware accelerator driver that is configured to communicate with two different virtual hardware accelerators for the two applications executing on the two Pods. In other embodiments, multiple Pods execute on one VM that executes above the virtualization application, and each Pod has a hardware accelerator driver that is configured to communicate with a virtual hardware accelerator that is assigned to that driver.

FIG.20illustrates an example of CP or edge applications2002that have passthrough access to virtual hardware accelerator2052and2054defined by a hypervisor2008executing on their host computer2010, in order to perform some or all of their computations. As shown, each application2002executes on a Pod2004, which has accelerator drivers2012with direct, passthrough access to virtual accelerators2052or2054. Each Pod2004operates within (i.e., execute on) a VM2006, which, in turn, executes over a hypervisor2008of the host computer2010.

Each Pod's accelerator driver2012has direct access to the virtual accelerator2052or2054, and this access bypasses the accelerator drivers2014and2016of the VM2006and the hypervisor2008. In some embodiments, the hypervisor2008executes over an operating system (not shown) of the host computer2010. In these embodiments, the direct access of each Pod's accelerator driver2012to the virtual accelerator2052or2054also bypasses the hardware accelerator driver of the operating system.

As shown, the virtual accelerators2052and2054communicate to the hardware accelerator2050through the accelerator manager2060of the hypervisor2008. The accelerator manager2060allows the virtual accelerators2052and2054(and in turn their associated applications2002) to share one hardware accelerator2050, while operating with this accelerator2050as if it is dedicated to their respective applications and Pods2002and2004. Examples of such a hardware accelerator2050include a graphical processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and a structured ASIC.

To communicate with its virtual accelerator2052or2054, each application2002in some embodiments communicates through the RIC SDK2030executing on its Pod2004. For instance, in some embodiments, each application2002uses high-level APIs of the RIC SDK2030to communicate with its virtual accelerator2052or2054. The RIC SDK2030then converts the high-level APIs to low-level APIs that are needed to communicate with each machine's driver2012, which, in turn, relays the communication to the virtual accelerator2052or2054. The virtual accelerator2052or2054then relays the communications to the hardware accelerator2050through the accelerator manager2060.

As mentioned above by reference toFIG.14, in some embodiments, the low-level APIs are provided by a first company associated with the sale of the hardware accelerator2050, while the RIC SDK2030is provided by a second company associated with the distribution of the RIC SDK2030. In some embodiments, the low-level APIs used by the RIC SDK2030are APIs specified in an API library2032associated with the hardware accelerator2050. Each application2002receives the results of the operations of the hardware accelerator2050through the accelerator manager2060, its virtual accelerator2052or2054, its driver2012, and its RIC SDK2030.

To provide a low latency near RT RIC, some embodiments separate the RIC's functions into several different components that operate on different machines (e.g., execute on VMs or Pods) operating on the same host computer or different host computers. Some embodiments also provide high speed interfaces between these machines. Some or all of these interfaces operate in non-blocking, lockless manner in order to ensure that critical near RT RIC operations (e.g., datapath processes) are not delayed due to multiple requests causing one or more components to stall. In addition, each of these RIC components also has an internal architecture that is designed to operate in a non-blocking manner so that no one process of a component can block the operation of another process of the component. All of these low latency features allow the near RT RIC to serve as a high speed IO between the E2 nodes and the xApps.

FIG.21illustrates an example of a near RT RIC2100with several components operating on several different machines. In this example, the near RT RIC is divided into three Pods, which are a datapath Pod2105, a service Pod2110, and an SDL Pod2115. In some embodiments, this RIC (1) handles E2AP messages between the E2 nodes2118and the xApps2120, (2) manages connections with the E2 nodes2118, (3) processes xApp subscriptions to E2 nodes, and (4) handles xApp liveness operations. The RIC2100provides reliable low-latency messaging between its various components, between the E2 nodes and xApps, and between E2 nodes/xApps and the RIC components. Part of the RIC's low latency architecture is attributable to using different Pods to implement the data IO, service and SDL operations, so that different resource allocations and management operations can be provided to each of these Pods based on its respective needs of the operations that they perform in several different Pods.

Each of the three RIC Pods2105,2110, and2115communicates with one or more xApp Pods2120. In some embodiments, each Pod (2105,2110,2115or2120) is allocated hardware resources (e.g., CPUs, memory, disk storage, network IO, etc.) per the Pod's unique needs (i.e., per the datapath, service and storage operations performed by each Pod). Also, in some embodiments, each Pod has its own high availability and lifecycle update configuration that matches the unique needs of each Pod.

The service Pod2110performs xApp onboarding, registration, FCAPS (fault, configure, accounting, performance, security), and other services in some embodiments. For instance, in some embodiments, the service Pod2110provides the management services554of the near RT RIC, and performs the O1 termination580and the A1 termination582to the SMO and its associated non-RT RIC. In some embodiments, each of these components554,580and582operate on a separate container in the service Pod2110, while in other embodiments two or more of these components operate on one container in the service Pod2110.

As mentioned above, the A1 Interface is between the near-RT RIC and the non-RT RIC in some embodiments. Through this interface, the near RT RIC relays relevant network information as reported by E2 nodes (e.g., CUs and DUs), and the non-RT RIC provides control commands for the E2 nodes (e.g., for control use-cases operation in non-RT granularities). The O1 Interface is between the near-RT RIC and the SMO, and in some embodiments is used for discovery, configuration, resource management and auto-scaling, life-cycle management, and fault tolerance.

The RIC management services554in some embodiments include services that the near RT RIC provides to the xApps and to the other RIC components. Examples of the services provided to the xApps include xApp service registry/directory (which the xApps can use to identify other xApps associated with the distributed near RT RIC and the operations performed by these other xApps), and FCAP operations, such as metric collection, policy provisioning and configuration. In some embodiments, the xApps can query the service registry/directory to identify other xApps or other xApps that perform particular services, and can register to receive notifications regarding xApps and their capabilities when the xApps are added to the directory.

Examples of FCAP operations performed by the service Pod2110for the xApp include fault operations involving collecting metrics that monitor CPU and memory utilizations to analyze to raise alerts, configuration operations to configure or re-configure the xApps, accounting operations to collect data needed for accounting and performance operations to collect metrics from xApps to analyze to quantify the xApp performance.

For the other RIC components (e.g., the datapath Pod2105and the SDL Pod2115), the service Pod2110performs services as well, such as metric collection, policy provisioning and configuration. The service Pod2110can be viewed as a local controller that performs operations at the direction of a central controller, which is the SMO. Through the SMO, the service Pod2110would receive configuration and policies to distribute to the xApps and the other RIC components. Also, to the SMO, the service Pod2110provides metrics, logs and trace data collected from the xApps and/or RIC components (e.g., the datapath Pod and the SDL Pod). In some embodiments, the service Pod can be scaled (e.g., replicated) and backed up independently of the other Pods. In some embodiments, the service Pod has a data cache that is a cache for a time series database of the SMO. In this cache, the service Pod stores stats, logs, trace data and other metrics that it collects from the xApps and one or more RIC components before uploading this data to the SMO database.

The SDL Pod2115implements the SDL560and its associated database570. As further described below, the SDL Pod2115in some embodiments also executes one or more service containers to execute one or more preprocessing or post-processing services on the data stored in the SDL. Like the service Pod, the SDL Pod in some embodiments can be scaled (e.g., replicated) and backed up independently of the other Pods.

The datapath Pod2105includes several important near RT RIC components. These are the E2 termination584, the conflict mitigation550, the application subscription management552, and the RIC SDK interface2150. As further described below, some or all of these datapath services in some embodiments are embedded in a datapath thread and a control thread of the datapath Pod. In other embodiments, the datapath services are embedded in a data IO thread, multiple data processing threads and a control thread.

A thread is a component of a process that executes on a computer. The process can be an application or part of a larger application. A thread is a sequence of programmed instructions that can be managed independently of other threads of the process. Multiple threads of a given process can execute concurrently (e.g., by using multithreading capabilities of a multi-core processor) while sharing the memory allocated to the process. Multithreading is a programming and execution model that allows multiple threads to exist within the context of one process. These threads share the process's resources, but are able to execute independently.

The control thread in some embodiments is the interface with the service Pod and SDL Pod for the datapath threads, while in other embodiments it is the interface to just the service Pod for the datapath threads (as the datapath threads can communicate directly with the SDL Pod). The control thread in either of these approaches performs the slower, control related operations of the datapath, while the one or more datapath threads perform the faster IO operations of the datapath. The control thread in some embodiments interfaces with the service Pod to receive configuration data for configuring its own operations as well as the operations of the datapath thread.

The embodiments that separate the datapath thread into a data IO thread and multiple data processing threads further optimize the data IO by pushing the more computationally intensive operations of the datapath thread into multiple datapath processing threads, which then allows the less computationally intensive operations to run in the data IO thread. Both of these optimizations are meant to ensure a fast datapath IO (one that does not experience unwanted latencies) so that the near RT RIC can serve as a high speed interface between E2 nodes2118and xApps2120.

As mentioned above, the Pods2105,2110and2115communicate through high speed inter Pod interfaces. In some embodiments, the Pod-to-Pod connections are established through SCTP (streaming control transport protocol) or through even higher speed shared memory (shmem) connections. In some embodiments, the shared memory connections are employed only between a pair of Pods that executes on the same host computer. Examples of such pairs of Pod include (1) a datapath Pod and an SDL Pod, (2) a datapath Pod and a service Pod, (3) a service Pod and an SDL Pod, (4) an xApp Pod and a datapath Pod, (5) an xApp Pod and an SDL Pod, etc. The shared memory is lockless and access to it is non-blocking in some embodiments. Other embodiments use slower interfaces (e.g., gRPC) between the service Pod2110and the other Pods2105,2115, and2120as the service Pod is not as critical a Pod as the other Pods.

The different Pods (e.g.,2105,2110and2115) of a near RT RIC in some embodiments can execute on the same host computer or can execute on two or more host computers. In other embodiments, one or more of the Pods (e.g., the service Pod2110) always operates on a separate host computer than the other Pods (e.g., the datapath Pod2105and the SDL Pod2115). Also, in some embodiments, the Pods2105,2110and2115operate on one host computer2205along with one or more xApp Pods2220a, while other xApp Pods2220boperate on other host computers2210, as shown inFIG.22. In other embodiments, two of the Pods2105,2110and2115operate on one host computer along with one or more xApp Pods, while the other one of the Pods2105,2110and2115operates on another host computer along with one or more xApp Pods.

For instance,FIG.23illustrates the datapath Pod2105and service Pod2110executing on a host computer2300along with two xApp Pods2320a, while the SDL Pod2115executes on a host computer2310along with two xApp Pods2320b. In some embodiments, Pods that require hardware accelerators are placed on host computers with such hardware resources, while Pods that do not require these accelerators are put on host computers without these accelerators, as mentioned above. In some embodiments, SDL Pods and xApp Pods use hardware accelerators, while the datapath Pod and service Pods do not. Various examples of Pods that can benefit from hardware accelerators and bypass paths to these accelerators are described above and further described below.

Also, although several near RT RICs are described above and below as being implemented with Pods, the near RT RICs in other embodiments employ VMs to implement the RIC components. Moreover, even in the embodiments that implement the different RIC components with Pods, some or all of the Pods operate on VMs, such as lightweight VM (e.g., Photon VMs provided by VMware, Inc.).

In addition to using fast communication interfaces between the Pods, some or all of the Pods use non-blocking, lockless communication protocols and architectures in some embodiments. For instance, the datapath Pod2105uses non-blocking, lockless communication between threads and processes that make up this Pod. Datapath Pod2105also uses non-blocking, lockless communication when communicating with the service Pod2110, the SDL Pod2115or the xApp Pods2120. Non-blocking communication ensures that no first component that sends a request to a second component can stall the operations of the second component when the second component is processing too many requests. In such cases, the second component will direct the first component to resend its request at a later time. The datapath Pod employs lockless communication in that it uses single thread processing that does not employ thread handoffs. Hence, no portion of memory has to be locked to ensure that another process thread does not modify it in an interim time period.

The communication interface between the RIC SDK interface2150of the datapath Pod2105and the RIC SDK2112of an xApp Pod2120is also novel in some embodiments. In some embodiments, this interface parses the header of E2AP messages received from E2 nodes, stores some or all of the parsed components in a new encapsulating header that encapsulates the E2SM payload of the E2AP message along with some or all of the original E2AP header. In doing this encapsulation, the SDK interface2150in some embodiments performs certain optimizations, such as efficiently performing data packing to reduce message size overhead for communications from one Pod to another (e.g., reduces the size of the E2 Global ID value, etc.). These optimizations improve the efficiency of the near RT RIC datapath Pod and xApp Pod communication.

The near RT RIC in other embodiments has one or more other Pods. For instance,FIG.24illustrates a near RT RIC2400that in addition to the Pods2105,2110, and2115, also includes a lifecycle management (LCM) Pod2405. The LCM Pod is a specialized service Pod responsible for upgrading each of the other Pods2105,2110and2115of the near RT RIC2400. Separating the lifecycle management from the service Pod2110allows the service Pod2110to be upgraded more easily.

In some embodiments, the LCM Pod2405uses different upgrade methodology to upgrade the different Pods. For instance, the LCM Pod in some embodiments replicates the SDL data store and seamlessly transitions from an active data store to another standby Datastore in order to perform a hitless upgrade of the SDL. On the other hand, to upgrade the datapath Pod, the LCM Pod's procedure is more involved, as it configures the active and standby datapath Pods to be dual-homed connections for each E2 node and each xApp, and configures the active datapath Pod to replicate state with the standby datapath.

FIG.25presents a more detailed view of a near RT RIC2500for some embodiments. As shown, the near RT RIC2500includes a datapath Pod2505, a service Pod2510and an SDL Pod2515. The SDL Pod2515includes an SDL agent2526and the shared SDL storage2528of the RIC2500, while the service Pod2510includes a service agent2524along with the O1 and A1 termination interfaces2535and2537. The datapath Pod2505includes a datapath thread2507and a control thread2509.

The datapath thread2507provides a fast datapath IO of the near RT RIC between the E2 nodes2518and the xApps2520. The data plane capabilities of the RIC in some embodiments can be scaled up by implementing the RIC datapath IO with one control thread and multiple datapath threads that share the load for the datapath processing of the datapath Pod. Several such implementations will be further described below. The control thread2509performs several control operations associated with the RIC's datapath. The near RT RIC2500separates the control and datapath threads because the data IO operations need to be fast and should not be slowed down by control operations that can operate at a slower rate. In some embodiments, the control and datapath threads are two threads in a single process (i.e., run in the same shared memory address space).

In some embodiments, each of these threads uses non-blocking, lockless interfaces to communicate with other components in this architecture (e.g., with the RIC SDK, service Pod agent, SDL agent, and/or E2 nodes) to the extent that they communicate with these other components. Also, in some embodiments, both threads use minimal OS system calls and run as infinite loops. As further described below, the datapath thread and the control thread exchange data over two circular rings2522(called cbuf), with one ring handling messages from the datapath thread to the control thread and the other handling messages from the control thread to the datapath thread.

The control thread2509serves as the control interface to the E2 nodes2518, the SMO2530(through the service Pod agent2524), the xApps (e.g., through SCTP), and the SDL Layer (through the SDL agent2526). In some embodiments, the control thread is the main thread to communicate with these external entities; however, as further described below, the datapath thread in some embodiments also communicates with the SDL2515through the SDL agent2526.

The control thread2509in some embodiments handles all control functions. This thread sends various control parameters to other functions, and in some embodiments enforces admission controls. In other embodiments, the datapath thread2507enforces admission controls and the SMO through the service Pod specifies the admission controls. The control thread2509in some in some embodiments has control channel communications with the RIC SDK of an xApp Pod through SCTP. In other embodiments, the control thread communicates with the RIC SDK of an xApp Pod through gRPC. Also, in some embodiments, the control thread communicates with the RIC SDK through shared memory (shmem) when the xApp Pods and the datapath Pod execute on the same host computer

The control thread2509also provides the transport mechanism to transport the statistics, logs and trace data generated by the datapath thread2507. In some embodiments, some or all of this data is transported to the SDL Pod2515through the SDL agent2526and/or to the SMO2530through the service agent2524. The control thread2509in some embodiments negotiates security keys with E2 node peers, and passes these keys to the datapath thread, which uses them to perform its encryption/decryption operations.

The datapath thread2507provides the high speed IO between E2 nodes and xApps. This thread handles the RIC SDK interface and the E2 termination operations, as well as the conflict mitigation and xApp subscription operations in some embodiments. This thread performs ASN.1 decoding of E2AP messages to extract the message data. In some embodiments, the datapath thread does not decode the E2SM payload of these messages. The datapath thread2507validates E2 and xApp messages and sequences. In some embodiments, the message types include E2 node setup and service update, E2 node indication reports, xApp initiated subscriptions for E2 node data and xApp initiated control requests.

The datapath thread2507in some embodiments runs E2 state machines in order to create and maintain state on behalf of xApps (e.g., state of E2 nodes, subscriptions to E2 nodes, etc.). Also, in some embodiments, the datapath thread performs table lookups to send messages to xApps that request data. This thread also handles control requests from xApps towards E2 nodes and forwards back responses to these requests from the E2 node to the xApps.

The datapath thread communicates with the xApps through SCTP when the xApps are on another host computer, or through shared memory when the xApps are on the same host computer. In some embodiments, the xApp messages have CRC bits to detect corruption. These messages also carry timestamps and can be compressed in some embodiment. The datapath thread2507performs data replication for multiple subscriptions. The datapath thread2507also performs datapath security operations, e.g., by signing, encrypting and decrypting data messages.

As mentioned above and further described below, the datapath thread2507communicates with the control thread2509in some embodiments via a pair of rings2522. In some embodiments, the frequency of messages between the two threads can be tuned (e.g., can be configured) to be from sub milliseconds to seconds per ring pair. Through the control thread, the datapath thread2507receives configuration data updates and state changes. The datapath thread2507generates statistics, logs and traces and provides the generated statistics, logs and trace data to the control thread for storage in the SDL and/or to provide to the SMO.

The datapath thread2507also performs conflict management operations in case multiple xApps try to set the same parameters to the same E2 node at the same time. For instance, the conflict management operations ensure that two xApps do not try to change a cellular network setting (e.g., a direction of an antenna) differently within a short time period. In some embodiments, the datapath thread's conflict management employs different methodologies to address different types of conflicts, e.g., (1) for one set of requests, for a duration of time, it rejects a second request to modify a parameter after receiving a conflicting earlier first request, (2) for another set of requests, it rejects a request regarding a parameter from one xApp when another higher priority xApp makes a conflicting request for the same parameter, (3) for another set of requests regarding another set of parameters, it only accepts requests made by xApps that are allowed to make such requests during particular periods of time. The policies for handling these conflicts are provided by the SMO2530through the service Pod's agent2524.

InFIG.25, each xApp Pod2520can execute one or more xApps2532, and interfaces with the datapath Pod2505, the SDL Pod2515and the service Pod2510through the RIC SDK2534that executes on the xApp Pod. Each RIC SDK provides high-level interfaces for xApps to communicate with the RIC and the E2 nodes. This high level interface hides details of underlying implementation. The RIC SDK communicate with the RIC instances through fast data IO communication channels (such as shared memory or SCTP).

The RIC SDK also uses control communication channel with the service Pod2510and the control thread2509for xApp control operations such as xApp onboarding, registration, capabilities, subscription, FCAPS, etc. In some embodiments, the control channel communication between the SDK and the control thread2509is through shared memory when the xApp Pod (and its SDK) and the datapath Pod operate on the same host computer, and through SCTP when they operate on different computers. Also, in some embodiments, the control channel communication between the xApp Pod (and its SDK) and the service Pod is through shared memory when the SDK and the service Pod operate on the same host computer, and through gRPC when they operate on different computers. Other embodiments use SCTP for communications between the SDK and the service Pod when the xApp Pod (and its SDK) and the service Pod operate on different host computers.

Some of embodiments use proto bufs when the RIC SDK communicates with the service Pod through gRPC. Also, in some embodiments where the RIC SDK's communication with the datapath Pod is over shared memory, the shared memory communication uses proto bufs. The RIC SDK has APIs for data functions, e.g., E2 messages to and from E2 nodes. These APIs also include control function messaging, such as onboarding xApp (name, version, function), message subscription, keep alive messaging, and A1 and O1 interface communications with the SMO through the service Pod (e.g., communications to store stats, logs, and trace data in a time series database on the SMO or service Pod, such as Prometheus and ELK).

Some embodiments assign the datapath thread and control thread to one processor core, assign the SDL to another processor core (in order to isolate it from data and control threads), and assign the service Pod to yet another processor core. When one or more xApps execute on the same host computer as the RIC, the xApps are assigned to different cores than the RIC Pods, where multiple xApps can be assigned to the same core, or individual cores are assigned to individual xApps, as needed.

To improve the performance of the RIC and the xApps further, other embodiments perform other hardware assignment optimizations, such as particular memory allocations (e.g., larger RAM allocations) and particular IO allocations. Examples of special IO allocations for some of the Pods include (1) SRIOV allocations for an xApp Pod on one host computer to communicate with a datapath Pod on another host computer, (2) SRIOV allocations for a datapath Pod to communicate with E2 nodes, (3) SRIOV allocations for an xApp Pod on one host computer to communicate with a service Pod on another host computer, and (4) gRPC or SCTP communication over the SRIOV allocations, with the gRPC communications having lower bandwidth allocations and being lower priority than the SCTP communications.

In some embodiments, one RIC and several xApps are bundled together to operate on different Pods that operate on one VM. Multiple instances of the RIC can also be deployed in some embodiments with different sets of xApps. Also, in some embodiments, xApps that need to talk to each other are bundled on the same VM.

As mentioned above, some embodiments implement the RIC datapath not as one datapath thread but as one data IO thread along with multiple datapath processing threads. In some embodiments, each datapath processing thread (DPT) is responsible for performing the datapath processing for a different set of E2 nodes, with each E2 node assigned to just one datapath processing thread. In some embodiments, the data IO thread identifies the DPT associated with an E2 message or an xApp message by hashing the E2 node identifier contained in the message and using the hashed value (obtained through the hashing) as an index into a lookup table that provides the DPT identifier of the DPT that needs to process the data message.

FIG.26illustrates an example of a RIC datapath Pod2600that has one data IO thread2605, one control thread2610, and multiple DPTs2615. The DPTs share the datapath processing load of the datapath Pod2600. As shown, there is a pair of cbuf rings2620between each DPT2615and the data IO thread2605, each DPT2615and the control thread2610, and the data IO thread2605and the control thread2610. Each ring2620in a cbuf pair passes data messages in one direction from one of the two threads associated with the ring to the other thread, with one ring handling communication in one direction (e.g., from first thread to second thread) and the other ring handling communication in the other direction (e.g., from second thread to the first thread).

Separating the data IO thread2605from multiple DPTs2615optimizes the data IO of the datapath Pod2600by pushing the more computationally intensive operations into the DPTs, which then allows the less computationally intensive IO operations to run in the data IO thread2605. This optimization ensures a fast datapath IO (one that does not experience unwanted latencies) so that the RIC can serve as a high speed interface between the E2 nodes and the xApps. Also, each E2 node is the responsibility of just one DPT thread2615, which typically is responsible for several E2 nodes. Because each E2 node is handled by one particular DPT, no two DPTs will try to change one or more records associated with one E2 node. Hence, the datapath Pod2600does not need to lock any E2 node's records as there is clear demarcation of responsibilities vis-a-vis the communications with the E2 nodes.

The data IO thread2605performs the following operations (1) managing connections to the E2 nodes and the xApp Pods, (2) transmitting data messages through these connections to and from the E2 nodes and the xApp Pods, (3) performing security operations, (4) control ring communication with control thread2610and DPTs2615, and (5) generating statistics, logs and trace data regarding messages that it processes.

Each DPT thread2615performs the following operations (1) message decode and encode operations (e.g., message encrypt and decrypt operations), (2) message validate operations, (3) sequence validate operations, (4) maintain state machine to keep track of state of the E2 node and the xApp requests and subscriptions, (5) perform conflict management, (6) control ring communication with control thread2610and DPTs2615, and (7) generate statistics, logs and trace data regarding messages that it processes.

FIG.27illustrates a process2700that the datapath thread performs in some embodiments to process subscription requests from an xApp. As shown, the process starts when the DPT receives (at2705) an xApp subscription request from the data IO thread. The subscription request is directed to a particular E2 node for a particular set of data tuples (e.g., a particular set of operational parameters or other parameters) that the particular E2 node maintains.

At2710, the process2700determines whether it has already subscribed to the particular E2 node to receive the particular set of data tuples. This would be the case if the DPT previously sent the particular E2 node one or more subscription requests that individually or collectively requested the particular set of data tuples or a larger set of data tuples that includes the particular set of data tuples.

When the process27100determines (at2710) that it has already subscribed to the particular E2 node to receive the particular set of data tuples, it (at2715) adds a new record, or updates a record previously specified, for the xApp in this E2 node's subscription list and specifies in this record the particular set of data tuples that the xApp should receive. After2715, the process ends.

On the other hand, when the process27100determines (at2710) that it has not already subscribed to the particular E2 node to receive the particular set of data tuples, it has to either send a first subscription to the particular E2 node if it does not have an active subscription with this node, or has to send an updated subscription to the node if it has an active subscription but not one that includes all of the data tuples in the particular set of data tuples specified in the request received at2705.

Hence, in such a case, the process2700(at2720) adds a new record, or updates a record previously specified, for the xApp in this E2 node's subscription list and specifies in this record the particular set of data tuples that the xApp should receive. Next, it sends an updated subscription request to the particular E2 node using a previously allocated RIC Request ID. This updated subscription specifies all of the data tuples in the requested particular set of data tuples when none of these tuples were previously requested by an earlier subscription to the particular E2 node, or specifies some of these data tuples when other data tuples in the particular set were previously requested by one or more earlier subscriptions to the particular E2 node. After2725, the process2700ends.

FIG.28illustrates a process2800that the data IO thread2605and a DPT2615perform in some embodiments to process a data message from the E2 node that one or more xApps should receive. As shown, the process2800starts when the data message is received (at2805) by the datapath Pod through an SCTP connection with the E2 node. At2810, the data IO thread2605generates a hash value from the E2 node's ID. It then uses (at2815) the hash value as an index into a lookup table to identify the DPT that is assigned to processing messages associated with the E2 node.

At2820, the data IO thread passes the received data message to the identified DPT (i.e., the DPT identified at2815) along the cbuf ring2620that is for passing messages from the data IO thread to the identified DPT. Next, at2825, the DPT uses its data structure records (e.g., the records maintained by its state machine) to identify the set of one or more xApps that should get the E2 message. In some embodiments, the identified set of xApps are the xApps that have subscribed to receive data (e.g., all the data or a subset of the data) from the E2 node.

At2830, the DPT specifies a data message for the data IO thread2605to send to the identified set of xApps. This data message is in the encapsulated format described below by reference to Table 1. The DPT then passes (at2835) the data message to the data IO thread2605along the cbuf ring2620that is for passing messages from the DPT2615to the data IO thread2605. Next, at2840, the data IO thread2605retrieves the data message from the cbuf ring2620, identifies the xApps that need to receive the data message, and then sends each identified xApp the data message. After2840, the process ends.

FIG.29illustrates a process2900that the data IO thread2605and a DPT2615perform in some embodiments to process a data message from an xApp that should be sent to an E2 node. As shown, the process2900starts when the data message is received (at2905) by the datapath Pod through an SCTP connection or shared memory communication with the xApp RIC SDK. This message is in the encapsulated format that is described below by reference to Table 1. This message includes an E2 node identifier that identifies the E2 node that should receive this message.

At2910, the data IO thread2605generates a hash value from the E2 node's ID. It then uses (at2915) the hash value as an index into a lookup table to identify the DPT that is assigned to processing messages associated with the E2 node. At2920, the data IO thread passes the received data message to the identified DPT (i.e., the DPT identified at2915) along the cbuf ring2620that is for passing messages from the data IO thread to the identified DPT.

Next, at2925, the DPT uses its data structure records (e.g., the records maintained by its state machine) to identify the E2 node that should receive the message. In some embodiments, the data message is a subscription request and the identified E2 node is an E2 node to which an xApp wants to subscribe. At2930, the DPT specifies a data message for the data IO thread2605to send to the identified E2 node. This data message is in the E2AP message format required by a standard. The DPT then passes (at2935) the data message to the data IO thread2605along the cbuf ring2620that is for passing messages from the DPT2615to the data IO thread2605. Next, at2940, the data IO thread2605retrieves the data message from the cbuf ring2620, identifies the E2 node that needs to receive the data message, and then sends each identified E2 node the data message. After2940, the process ends.

In some embodiments, the DPT2615might determine that no new subscription message needs to be sent to the E2 node that it identifies at2925. For instance, before receiving (at2905) from a first xApp the subscription request for a set of data tuples from an E2 node, the datapath Pod previously sent for a second xApp a subscription request to the same E2 node for the same set of data tuples or for a larger set of data tuples that includes the data tuples requested by the first xApp. In such a case, the DPT2615simply adds the first xApp to the subscription list of the E2 node, so that it can provide subsequently received values from the E2 node to the first xApp. In some embodiments, the DPT2615also supplies previously received values from the E2 node that are stored in the SDL to the first xApp or directs the xApp to obtain these values from the SDL.

In some cases, the first xApp asks for additional data tuples from the E2 node that the second xApps did not request previously. In such cases, the DPT2615would prepare an updated subscription message for the data IO thread to send to the E2 node to request the data tuples that are newly requested by the first xApp. The DPT would also prepare such a message when the second xApp requested additional data tuples from the E2 node after its initial subscription.

In some embodiments, a service Pod2510configures the datapath Pod2600to instantiate N DPTs when it starts up with N being an integer greater than one. For the datapath Pod2600of a near RT RIC, the number N is computed in some embodiments based on the expected number of E2 nodes and xApps that communicate with the E2 nodes through a near RT RIC. The data IO thread2605of the datapath Pod2600in some embodiments then assigns the E2 nodes to the DPTs based on the order of subscription requests that it receives and the load on the DPTs at the time of these requests.

FIG.30illustrates an example of a process3000that the data IO thread2605uses in some embodiments to assign E2 nodes to DPTs. As shown, the process3000starts when the data IO thread2605receives (3005) a first subscription request for a particular E2 node from an xApp that is associated with the near RT RIC of the data IO thread. A first subscription request for a particular E2 node means that no other subscription requests were previously received for this particular E2 node by the data IO thread.

Next, at3010, the data IO thread2605generates an N-bit hash value from the Global E2 node ID of the particular E2 node, where N is an integer (e.g., is 6 or 8). This N-bit value is used to identify the particular E2 node in a hash LUT (lookup table) as further described below. At3015, the process3000selects a particular DPT for the particular E2 node based on the current load on each of the DPTs of the datapath Pod2600(e.g., by selecting the DPT with the least amount of load). In some embodiments, the current load is just based on the number of E2 nodes assigned to each DPT, while in other embodiments the current load is based on the number of E2 nodes and the number of xApp subscriptions to these nodes. In still other embodiments, the current load is computed in other ways.

At3020, the process3000then creates a record in a LUT and in this record associates the N-bit hash value with the identifier of the particular DPT selected at3015for the particular E2 node. In some embodiments, the N-bit hash value is an index into the LUT that identifies the record that specifies the particular E2 node's ID. At3020, the process3000also specifies the state of this record as Active.

At a subsequent time, if the data IO thread encounters a situation where all xApps have canceled their subscriptions to the particular E2 node, the process3000maintains the LUT record created at3020but changes the status of this record to Inactive. The data IO thread maintains this Inactive status until the next time that an xApp submits a subscription request for the particular E2 node, at which time the status of this record is changed to Active again. This status value is used as a mechanism to ensure that the data IO thread does not have to continuously revisit the E2 node assignments to the DPTs.

FIG.31illustrates a distributed near RT RIC that is implemented by an active RIC3102and a standby RIC3104. As shown, the E2 nodes3118and xApp Pods3120communicate with the active RIC3102, until one or more components of this RIC fail. When the active RIC fails, the standby RIC3104becomes the active RIC, and the E2 nodes and xApp Pods continue their communications through the RIC3104, which is now the active RIC.

Both of these RICs have the same components, which are a datapath Pod3105, a service Pod3110, and an SDL Pod3115. The datapath Pod is shown to include a control thread3109and a datapath thread3107. Instead of one datapath thread3107, some embodiments employ one data IO thread and multiple DPTs as mentioned above. In some embodiments, the active RIC3102is implemented by a first set of one or more computers, while the standby RIC3104is implemented by a different second set of one or more computers.

As shown, each E2 node3118has a dual-homed SCTP connection with the datapath threads3107of the active and standby RICs3102and3104. Similarly, each xApp Pod3120has a dual-homed SCTP connection with the datapath threads3107of the active and standby RICs3102and3104. Dual-homing connections is a feature provided by SCTP. When a first component connects to an active/standby pair of components through a dual-home connection, the first component can automatically switch to using the standby component when the active component fails. Thus, using the dual-homed SCTP connections, each E2 node or xApp Pod can switch to the datapath thread3107of the standby RIC3104when the active RIC3102or its datapath Pod fails.

As shown, the RIC SDK interface3122of the datapath thread3107of the active RIC3102forwards messages that it receives from the xApp RIC SDKs, and messages that it sends to the xApp RIC SDKs, to the RIC SDK interface3122of the datapath3107of the standby RIC3104. This is done in some embodiments so that the standby RIC's datapath thread3107can update its state machine to match the state of the active RIC's datapath thread3107. Also, as shown, synchronizing agents3127of the active and standby RICs3102and3104synchronize the SDL storage3126of the standby RIC3104with the SDL storage3126of the active RIC3102. All components of the active and standby RICs3102and3104are consistently managed by the SMO3130.

FIG.32illustrates the interfaces between a near RT RIC3200and E2 nodes3205, and between the near RT RIC3200and xApp Pods3210in some embodiments. In some embodiments, the near RT RIC is one of the above-described near RT RICs. As mentioned above, and as shown inFIG.32, the near RT RIC in some embodiments employs SCTP interface3220and3222with both the E2 nodes3205and the xApp Pods3210. When the xApp Pod and the near RT RIC execute on the same host computer, some embodiments use a shared memory as the interface between the near RT RIC and the xApp Pod, as shown. These interfaces keep the message exchange fast, and minimize encoding and decoding overhead across all paths to minimize latency (e.g., perform one ASN decode and one ASN encode). In some embodiments, the interface3220between an E2 node and near RT RIC will follow the E2AP specifications, and all message exchange will conform to E2AP specifications.

Also, in some embodiments, the interface3222between the near RT RIC and an xApp Pod uses a novel encapsulating header that will be described below by reference to Table 1. The interface3222handles a mix of different types of messages. Examples of such messages in some embodiments include (1) the entire E2AP messages (e.g., E2 Setup Request) from an E2 node, (2) some fields of the E2AP header along with the entire E2SM content (i.e., the entire E2AP message payload), (3) internal messages between the near RT RIC and xApp Pod (e.g., a message from the near RT RIC that an earlier message of an xApp caused an error), and (4) messages from xApp to near RT RIC or E2 Node. In some embodiments, the E2 content might not be ASN1 encoded (e.g., portion of a subscription request might not be encoded).

In some embodiments, the near RT RIC3200can be configured on a case by case basis to decode just the E2AP messages before it sends the message to an xApp, or to decode the entire E2AP header along with its E2SM payload. In some cases, the near RT RIC sends the entire E2AP header while in other cases it only sends a part of this header. In the RIC E2AP message handling of some embodiments, all fields are in network byte order, and the near RT RIC3200will work with that order as much as possible. For displaying fields, some embodiments can convert the data to host order. In some embodiments, the near RT RIC3200will not look into E2SM payload, while in other embodiments it will (e.g., in order to avoid duplicate subscription errors).

In some embodiments, the RAN function ID is E2 node specific. The xApps will not subscribe to RAN functions across E2 nodes, as every subscription will be to an individual E2 node. Also, in some embodiments, the RIC Request ID space is local to an E2 node. In some embodiments, the RIC Request ID number space is ephemeral component as well as a persistent component. For example, the RIC request IDs used for indication reports will persist while RIC request IDs used from subscription may be reused.

Table 1 below displays an exemplary message format used in some embodiments for the communication between the RIC and RIC SDK. This is the format of an encapsulating header that is used to encapsulate all messages from and to the RIC to and from the RIC SDK. In some embodiments, the encapsulating header stores data needed by the RIC SDK and the RIC for efficient processing of the data message. In the example illustrated in Table 1, the first sixteen bytes associated with the msg_type, msg_serial_num, msg_len, msg_flags, and ctrl_len are part of the encapsulating header along with the ctrl_info field. The payload of the encapsulated packet can include any data. In the example shown in Table 1, the payload includes the original E2AP packet along with its E2SM payload.

All messages between RIC and RIC SDK are encapsulated with the header shown in Table 1. Control information and payload are optional. Some messages may have control information but no payload field, others might have payload without control information and some may have both control and payload fields. In some embodiments, the RIC SDK can be configured to trap these messages and reformat them for presentation to xApps. The format of the message is a raw byte stream. In some embodiments, a message CRC field is not used, while it is used in other embodiments.

TABLE 1LengthType(bytes)Descriptionmsg_type4Message type identifying what type ofmessage is being sentmsg_serial_num4Running serial number identifying aparticular messagemsg_len4Total message length: includes header,control and payload.Msg Len = 16 (header len) + controllen + payload lenmsg_flags2Message flagsctrl_len2Control Info Lenctrl_infovariableControl Info - Contents depend on messagelentypepayloadvariableCould include any portion of original datalenmessage-any portion of E2AP header orE2SM payload- ASN1 encoded or decoded

The near RT RIC3200handles E2 node and xApp connect, disconnect, reset, and crashes as follows. For E2 nodes, the RIC in some embodiments handles a connect, disconnect, and crash similarly. Specifically, when the connection to the E2 node drops and comes back for any of these reasons, the E2 node will send a connection setup all over again as though it started for the first time, and the near RT RIC will clean all its state related to the E2 node and start over. In some embodiments, the near RT RIC informs all xApps when an E2 node connection drops and comes back up whether they had previously subscribed to the particular E2 node or not, as the E2 node might advertise new functionality in which a previously unsubscribed xApp may be interested. When an xApp connects, disconnects, or crashes, the near RT RIC again performs the same operations, in that it resets all the state of the xApp in the near RT RIC and deletes its subscriptions from all E2 nodes.

FIG.33illustrates the E2AP message handling of the datapath Pod of the near RT RIC3200. In the discussion below regarding the E2AP message handling and other message handling, this datapath Pod is simply referred to as the near RT RIC3200or the RIC3200for purposes of brevity. As shown, the near RT RIC3200initially decodes an E2AP message received from an E2 node. In some embodiments, this decoding involves the decoding of only the E2AP header, while in other embodiments, this decoding involves the decoding of the E2AP header and the E2SM payload. For some or all of this decoding (e.g., E2SM), the near RT RIC in some embodiments uses a hardware accelerator (e.g., a GPU accelerator) that it accesses through a bypass path, which is described above.

After decoding the E2AP message, the near RT RIC creates or updates its internal data structures to account for the received data message, and then creates a flat encapsulated message to the xApp in the format described above by reference to Table 1. As the near RT RIC and RIC SDK operate on different containers and reside on different Pods in some embodiments, they do not pass arbitrary data structures to each other but format their data exchange into an encapsulated message with a specific sequence of bytes in some embodiments. After encapsulating the data message, the near RT RIC forwards the data message to the xApp Pod for the RIC SDK on this Pod to forward to the appropriate xApp.

The internal data structure that the near RT RIC creates or updates while processing the E2AP message is used for processing of responsive messages from the xApp to the E2AP message and for processing of subsequent E2AP messages. Examples of data stored in the near RT RIC's internal data structure in some embodiments include (1) a subscription list of xApps that are interested in data from a particular E2 node, (2) particular data tuples that each xApp is interested from each E2 node, (3) records identifying network addresses and other location data relating to E2 nodes and xApps, (4) identifiers that are allocated and assigned (e.g., RIC Request IDs).

When the xApp sends a message, its RIC SDK processes the message and forwards it to the RIC along a shared memory or SCTP interface as described above. The near RT RIC then parses the message and stores the parsed components. Based on these components, and on one or more data tuples that it stored in its internal data structure for the associated E2 node message(s), the RIC creates an E2AP response, and then encodes and forwards this response to the E2 node to which it is directed.

For instance, after a first xApp sends a subscription request to receive M data tuples from an E2 node, the near RT RIC's datapath Pod creates a state to record the first xApp's desired subscription, requests a subscription with the E2 node for the M data tuples, and forwards these M data tuples to the xApp when it initially receives them and each subsequent time that it receives them. In some embodiments, the near RT RIC's datapath Pod can be configured to forward the M data tuples to its associated SDL each time that it receives them from the E2 node.

After the first xApp subscribes to receive the M data tuples from the E2 node, a second xApp can subscribe to receive N different data tuples from the E2 node, where N is larger than M. The near RT RIC then sends an updated subscription request to the E2 node. This update now requests the N data tuples. Each time that the near RT RIC receives the N data tuples, it sends M data tuples to the first xApp and all N data tuples to the second xApp.

Another example involves the near RT RIC removing and caching an RIC request ID from an E2AP message from an E2 node in response to a subscription request. After this ID is removed, the RIC provides a portion of the E2AP message and its E2SM payload (if applicable) to the xApp. Subsequently, when the xApp wants to delete the subscription, the RIC retrieves RIC request ID from its state, and inserts it into its E2AP message to the E2 node to request the deletion of the subscription.

In some embodiments, the near RT RIC's E2 Setup, Response Message, and Failure Message handling is as follows. The near RT RIC initially receives the setup request message from the E2 node. In response, the near RT RIC will decode the message and build internal data structures. The RIC will also cache the raw ASN1 payload. In some embodiments, the near RT RIC accepts all added RAN function identifiers. In some embodiments, the near RT RIC sends the setup message to xApps after decoding the E2AP header but nothing else (i.e., as a message with an ASN1 encoded E2SM payload). In some embodiments, a setup message that the near RT RIC sends to an xApp has a control length (ctrl_len) of 0 with its ASN1 encoded payload.

When an xApp connects later, the near RT RIC will send all setup requests from E2 nodes to the xApp so it has an inventory of connected E2 nodes. In some embodiments, the near RT RIC sends these messages one at a time. Also, as mentioned above, the near RT RIC in some embodiments constructs E2Setup response and send it to the E2 node. In some embodiments, the near RT RIC sends a failure message when a setup request is malformed (e.g., it is a duplicate of the RAN function list, or removes a record not added to a list).

After receiving a reset from E2 node, the near RT RIC performs the following actions after decoding the message. It sends a message regarding this reset to all xApps that have a subscription to this E2 node. In some embodiments, this is an internal message without any ASN1 content. The near RT RIC ends subscription deletion messages to the E2 node for all previous subscriptions that it sent to it. It also sends control, insert and policy deletions to this E2 node. It cleans up any outstanding requests, and sends reset response to the E2 node.

The near RT RIC also employs a Service Update, Acknowledge, and Failure message in some embodiments. This message updates the supported RAN function list, with additions, modifications and deletions. The near RT RIC inform all xApps about the new service configuration of an E2 node. In some embodiments, the near RT RIC sends the message to xApps after application of the configuration so it will reflect the final state of the configuration. In other embodiments, the near RT RIC sends the message as is for xApps to compute the delta between the previous and new state of supported RAN functions. In this latter approach, the near RT RIC does not need to ASN1 encode the resulting delta.

The handling of the E2AP subscription is as follows in some embodiments. An xApp formats the E2SM portion of the subscription and ASN1 encode it. Table 2 below details the control portion of the subscription message (i.e., the portion that is stored in the control field of a message from the xApp to the near RT RIC). The payload will be the ASN1 encoded E2SM content. Multiple subscription message types are defined in some embodiments to disambiguate optional information. Also, in some embodiments, message flags are used to specify the exact format. In some embodiments, each subscription message specifies one E2 node Global ID and one RAN Function ID.

In some embodiments, the E2 node sends an identifier that is 113 bytes and the RIC compresses that to a 40 byte ID. When sending the subscription message to the E2 node, the RIC converts the 40 byte to 113 byte ID. The subscription message control fields will be of fixed formats as far as possible. In some embodiments, the RIC caches all subscription requests and compares requests from multiple xApps in order to avoid sending out duplicate subscription messages. However, when a second subsequent xApp requests additional information from the same E2 node after a first initial xApp requests some information from the E2 node, the MC resends the subscription (with the same RIC Request ID in some embodiments) but this time asks for the additional information. When sending out the subscription request to the E2 node, the MC sends out as the E2AP message payload the entire payload received from xApp in some embodiments.

TABLE 2Field NameField LenE2 Node Global Id40 bytesRAN Function Id2 bytesRIC Event Trigger Len2 bytesRIC Action Admit Cnt2 bytesRIC Action Not Admit Cnt2 bytesRIC Subsequent Action Cnt2 bytesRIC Action Type16 bytes - Only RIC ActionCnt Fields are validRIC Action Definition Length32 bytes - 2 bytes per - Only RICAction Cnt Fields are validRIC Subsequent Action Type16 bytes - Only RIC SubsequentAction Cnt Fields are validRIC Subsequent Action Time16 bytes - Only RIC Subsequentto WaitAction Cnt Fields are validRIC Action Definitionx bytes each * Action CntRIC Event Trigger Definitionx bytes

The near RT RIC handles an E2AP RIC subscription response by storing the E2 node Global ID and RIC Request ID (generated by RIC) as control information and sending the exact ASN1 encoded message from E2 node back to xApps.

The E2AP RIC subscription delete request, response or failure message are sent from an xApp to the near RT RIC with message fields sent as control information (i.e., as part of the ctrl_info). The near RT RIC creates the encoded ASN1 message and sends it to the E2 node. The deletion request does not specify the E2 Node Global ID. Hence, this information is provided by the RIC for the xApp. The response message is sent as packed bytes (not ASN1 encoded) from near RT RIC to xApp in some embodiments.

An E2 node's E2AP indication report is handled as follows. The message is decoded by near RT RIC to determine the RIC Request ID field. This helps determine which xApp(s) subscribed to the indication. The near RT RIC in some embodiments sends the message as an ASN1 encoded to xApp(s). In some embodiments, the near RT RIC also sends the reduced E2 Global ID as control information along with the message.

The near RT RIC's processing of the E2AP control request is as follows in some embodiments. The xApp sends this request as packed byte information. The near RT RIC does not specify the E2 Global ID in the message, as this information is specified by the xApp. The near RT RIC's formatting of this message is illustrated in Table 3.

TABLE 3Field NameField LenE2 Node Global Id40 bytes - fixedRAN Function Id2 bytes - fixedRIC Call Process Id len2 bytesRIC Control Header len2 bytesRIC Control Message len2 bytesRIC Control Ack Request1 byteRIC Call Processbyte stringRIC Header lenbyte stringRIC Control Messagebyte string

The near RT RIC handles E2AP control response or failure message is as follows. The near RT RIC decodes the message to obtain the RIC Request ID. It then sends the ASN1 encoded message to xApp prepended with the Global E2 Node ID as control information

In some embodiments, the SDL data store is an in memory database that runs in its own set of one or more Pods. It has its own compute and memory resources assigned. As mentioned above, multiple near RT RIC instances define a distributed near RT RIC. In some embodiments, each near RT RIC instance has its own instance of an SDL, which stores system wide information for the RIC instance. Examples of such information include a list of connected E2 nodes (i.e., base station nodes), xApps, subscriptions from each xApp and critical cell data returned by E2 nodes. Further, each SDL instance in some embodiments provides services to preprocess incoming data by running custom algorithms internally as the data arrives and by interfacing to hardware accelerators (e.g., GPUs), or post-process data retrieved from its storage.

The data IO Pod and xApp Pods of the RIC instance are connected to the SDL Pod of the RIC instance. In some embodiments, each SDL instance just operates with the data IO Pod and service Pod of its own RIC instance. Also, in some embodiments, the SDL Pod is managed by the SMO and configured via the service Pod. The dataflows to and from the SDL include (1) data IO to SDL data store, (2) xApps from SDL data store, (3) xApps to SDL data store, (4) data IO from SDL data access (e.g., retrieval of E2 node info, subscription info, etc.), and (5) service Pod to and from SDL communication to provide and retrieve configuration information.

FIG.34illustrates the RIC instance3400of some embodiments with an SDL Pod3410. As shown, the SDL3410includes a configuration agent3412, an SDL datapath agent3414, an RIC SDK agent3417, SDL pre and post processors3416and3418and one or more SDL data stores3420. The SDL configuration agent3412interfaces with the service Pod agent3430. This agent3412configures and manages other components of the SDL Pod. The SDL Pod is managed by the SMO and configured via the service agent3430of the service Pod #090.

The SDL datapath agent3414is the datapath interface that the SDL Pod exposes to the control thread and datapath thread of the datapath Pod3450. The SDL datapath agent3414handles communication from these entities for the SDL, and performs reads and writes to the SDL data store3420for these datapath entities. In some embodiments, the SDL datapath agent can be configured to use either SCTP or shared memory libraries to communicate with the datapath Pod3450.

In some embodiments, the RIC SDK agent3417is the agent that the SDL Pod exposes to the RIC SDK of the xApp Pods. The RIC SDK agent3417handles communication from the RIC SDKs to the SDL, and performs reads and writes to the SDL data store3420for the RIC SDKs. In some embodiments, the RIC SDK agent3417can be configured to use either SCTP or shared memory libraries to communicate with the RIC SDKs. This agent3417also performs the cache synchronization operation to synchronize the SDL cache of the RIC SDKs with the SDL data store(s)3420. Also, in cases where it will need to scale to tens of connections from xApps, the connection manager in some embodiments leverages epoll connection handling of the RIC instance (e.g., the epoll connection handling used by the data IO Pod in some embodiments).

The SDL agents3414and3417handle event subscription and notifications from the RIC SDKs and the datapath Pod. This is separate from E2AP subscription management, but conceptually it is similar. For instance, through this subscription service of the SDL, an xApp specifies its interest in some data via a key and/or frequency of reports. The RIC SDK agent3417then provides periodic updates to this xApp based on its subscription. It also provides security services in some embodiments by encrypting and decrypting data.

The data preprocessor and post processors3416and3418are part of the SDL3410in some embodiments in order to flexibly run certain value added algorithms on data. In some embodiments, both of these processors3416and3418operate in one container, while in other embodiments each of them operates on a separate container in the SDL Pod3410. Each of these processors also interfaces with external accelerators (e.g., GPUs3480) to perform their operations in some embodiments. The data preprocessor3416runs inline as data is stored in SDL data store3420.

In some embodiments, the data post processor3418runs inline as data is read from the SDL data store3420. Alternatively, or conjunctively, the post processor3418in some embodiments can be configured to run in the background on data stored in the SDL data store3420(e.g., to retrieve data from this data store, perform some operations on this data, and store back the results in the data store). The data processors in some embodiments encodes and/or decodes the E2SM payloads of the E2AP messages. This is advantageous as it allows datapath Pod to pass the ASN1 string to SDL to decode and store. As mentioned above, the RIC SDK in some embodiments can also be configured to provide the E2SM encode/decode services.

Another example of a post processor operation that the data processor3418performs in some embodiments is a machine trained operation. In some embodiments, the post processor3418collects various data tuples stored by various xApps and/or various Pods (e.g., datapath Pod), and passes these data tuples through a machine-trained network (e.g., a neural network trained through machine learning). In some embodiments, to execute its machine-trained network, the post processor3418uses one or more hardware accelerators (e.g., one or more GPUs) of the SDL's host computer to perform its operations. The post processor3418access a hardware accelerator through the bypass approach described above by reference toFIGS.14-20.

The post processor3418passes the results of its operations to one or more xApps, or it can store the results in the SDL data store3420for one or more xApps to retrieve. An example of a result obtained by post-processing SDL data with machine-trained networks includes anomaly detection (e.g., identifying E2 nodes that are behaving anomalously, e.g., cell sites that receive too many connections suddenly).

It is advantageous to use a machine-trained network in the SDL Pod3410to process different xApp outputs that are stored in the SDL data store3420because this data store3420stores the outputs of several xApps as well as numerous data tuples that the E2 nodes provide based on the xApp subscriptions. Often, individual xApps have insight only to the data tuples to which they subscribe and to the results of their own computations and the outputs of a few other xApps. The SDL Pod3410, on the other hand, has access to a much larger set of E2 node input data and xApp output data. Instead of using machine-trained networks to perform such post-processing, the post processor3418uses algorithms (e.g., constrained optimization solvers) to post-process the data stored in the SDL data stores3420. In other words, the post processor3418in some embodiments does not use machine-trained networks but still uses its host computer's hardware accelerator(s) (e.g., through a bypass path) to perform its operations.

Some embodiments also use post processor3418to provide the current state of an E2 node when a first xApp starts to subscribe to the E2 node's state. After a second xApp subscribed earlier to receive the E2 node's state, the near RT RIC stores multiple data tuples relating to this node's state over a duration of time. When the first xApp subsequently subscribes to the E2 node's state, and either this xApp or the datapath Pod tries to access this state for the first xApp, the post processor3418retrieves all the data tuples previously stored for this E2 node in the SDL storage, and uses these data tuples to compute the current state of the E2 node, which it then provides to the first xApp directly or through the datapath Pod.

The SDL data store3420is an in-memory database. In some embodiments, an in-memory database is a database that loads into and runs out of the computer system memory (e.g., the host computer volatile memory, including its RAM). One example of such a database is Redis. The data store's size is selected in some embodiments to minimize search and store latency. Once the existing data store reaches its maximum desirable size, some embodiments create additional instances of this data store in the same or different instances of the SDL.

Also, as shown inFIG.34, the SDL3410in some embodiments has an active data store3420and a standby data store3421for HA reasons. In addition, some embodiments allow the data stores3420of different SDL instances of different RIC instances to synchronize in the background of some or all of their data for HA and/or data availability reasons. In some embodiments, the xApps and RIC components read and write data to the active SDL storage, which is synchronized with the standby SDL storage in the background. When the active SDL storage fails, the RIC in these embodiments can seamlessly switch to the standby SDL storage. Also, the RIC can switch to the standby SDL storage when the active SDL storage is being upgrades. This allows the SDL storage to be hitless.

As mentioned above by reference toFIG.13, the RIC SDK includes an SDL cache that provides local SDL storage for an xApp, and this cache synchronizes its data with the data store of the SDL. In some embodiments, this RIC SDK cache pulls data from the main SDL data store in bulk and periodically. It helps reduce the number of requests made to the main SDL Pod. The RIC SDK cache also reduce latency for reading the SDL data by providing a portion of this data locally. The RIC SDK cache also speeds the time for writing data to the SDL by allowing the data to be written locally on the xApp Pod, and synchronized in the background. The size of the SDK cache in some embodiments is smaller than the SDL's data store3420. Also, in some embodiments, this size is based on the requirements of the set of one or more xApps that execute on the Pod along with the RIC SDK.

In some embodiments, the sources of data to the SDL3410include (1) the control thread and datapath thread of the datapath Pod3450, (2) the xApps through the RIC SDKs, (3) ML model and policy storage and server of the non-RT RIC accessed through the A1 interface, (4) xApp Pod configuration storage, and (5) configuration server of the SMO. In some embodiments, examples of system data (e.g., written by the control thread) in the SDL include (1) E2 node information, (2) cell information, (3) UE information, (4) E2 node reports, (5) KPM (key performance metric) reports, and (6) topology information (from EMS adapters).

Examples of SDL transaction in some embodiments include (1) data IO Pod (control or data IO thread) writing data to SDL, which is then read by an xApp, (2) an xApp reading data from another xApp that is written to the SDL, or the xApp writing data to the SDL for another xApp, (3) xApp writing to SDL so that a service container (e.g., a post processor) operating in the SDL Pod performs an operation (e.g., by using the GPU services or just using the general CPU) on the written data before the same xApp or another xApp retrieves the result of this operation from the SDL, (4) the non-RT RIC reading data from and writing data to the SDL as part of an A1 subscription, (5) SMO storing O1 configuration data in the SDL, and (6) non-RT RIC storing ML data in the SDL.

FIG.35conceptually illustrates an electronic system3500with which some embodiments of the invention are implemented. The electronic system3500may be a computer (e.g., a desktop computer, personal computer, tablet computer, server computer, mainframe, a blade computer etc.), or any other sort of electronic device. Such an electronic system3500includes various types of computer-readable media and interfaces for various other types of computer-readable media. Electronic system3500includes a bus3505, processing unit(s)3510, a system memory3525, a read-only memory3530, a permanent storage device3535, input devices3540, and output devices3545.

The bus3505collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the electronic system3500. For instance, the bus3505communicatively connects the processing unit(s)3510with the read-only memory3530, the system memory3525, and the permanent storage device3535.

From these various memory units, the processing unit(s)3510retrieve instructions to execute and data to process in order to execute the processes of the invention. The processing unit(s)3510may be a single processor or a multi-core processor in different embodiments.

The read-only-memory (ROM)3530stores static data and instructions that are needed by the processing unit(s)3510and other modules of the electronic system3500. The permanent storage device3535, on the other hand, is a read-and-write memory device. This device3535is a non-volatile memory unit that stores instructions and data even when the electronic system3500is off. Some embodiments of the invention use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device3535.

Other embodiments use a removable storage device (such as a floppy disk, flash drive, etc.) as the permanent storage device3535. Like the permanent storage device3535, the system memory3525is a read-and-write memory device. However, unlike storage device3535, the system memory3525is a volatile read-and-write memory, such as random-access memory. The system memory3525stores some of the instructions and data that the processor needs at runtime. In some embodiments, the invention's processes are stored in the system memory3525, the permanent storage device3535, and/or the read-only memory3530. From these various memory units, the processing unit(s)3510retrieve instructions to execute and data to process in order to execute the processes of some embodiments.

The bus3505also connects to the input and output devices3540and3545. The input devices3540enable the user to communicate information and select commands to the electronic system3500. The input devices3540include alphanumeric keyboards and pointing devices (also called “cursor control devices”). The output devices3545display images generated by the electronic system3500. The output devices3545include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD). Some embodiments include devices such as a touchscreen that function as both input and output devices.

Finally, as shown inFIG.35, bus3505also couples electronic system3500to a network3565through a network adapter (not shown). In this manner, the computer can be a part of a network of computers (such as a local area network (“LAN”), a wide area network (“WAN”), or an Intranet, or a network of networks, such as the Internet. Any or all components of electronic system3500may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra-density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media may store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.

While the above discussion primarily refers to microprocessor or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions that are stored on the circuit itself.

As used in this specification, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms display or displaying means displaying on an electronic device. As used in this specification, the terms “computer-readable medium,” “computer-readable media,” and “machine-readable medium” are entirely restricted to tangible, physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral signals.

While the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. For instance, a number of the figures conceptually illustrate processes. The specific operations of these processes may not be performed in the exact order shown and described. The specific operations may not be performed in one continuous series of operations, and different specific operations may be performed in different embodiments. Furthermore, the process could be implemented using several sub-processes, or as part of a larger macro process.

Also, several embodiments described above only show one hardware accelerator per host computer. However, one of ordinary skill will realize that the methodology and architecture of some embodiments can be used to provide direct, passthrough access to multiple hardware accelerators on one host computer. In addition, several embodiments described above pertain to xApp operations and the near RT RIC communications with xApps. One of ordinary skill will realize that these embodiments are equally applicable to edge applications in a telecommunication network and the near RT RIC communications with the edge applications. Thus, one of ordinary skill in the art would understand that the invention is not to be limited by the foregoing illustrative details, but rather is to be defined by the appended claims.