Patent Publication Number: US-2023156522-A1

Title: Predictive user plane function (upf) load balancing based on network data analytics

Description:
TECHNICAL FIELD 
     The present disclosure relates generally to digital message communications and, more particularly, to User Plane Function (UPF) load balancing within a Fifth Generation (5G) communications network. 
     BRIEF SUMMARY 
     As the use of smart phones and Internet of Things (IoT) devices has increased, so too has the desire for more reliable, fast, and continuous transmission of content. In an effort to improve the content transmission, networks continue to improve with faster speeds and increased bandwidth. The advent and implementation of 5G technology has resulted in faster speeds and increased bandwidth, but with the drawback of potentially overloading certain portions of the network in certain circumstances. It is with respect to these and other considerations that the embodiments described herein have been made. 
     5G Core (5GC) is the heart of a 5G mobile network. It establishes reliable, secure connectivity to the network for end users and provides access to its services. The core domain handles a wide variety of essential functions in the mobile network, such as connectivity of new user equipment (UE) and mobility management, authentication and authorization, subscriber data management and policy management, among others. 5G Core network functions are completely software-based and designed as cloud-native, meaning that they&#39;re agnostic to the underlying cloud infrastructure, allowing higher deployment agility and flexibility. 
     With the advent of 5G, industry experts defined how the core network should evolve to support the needs of 5G New Radio (NR) and the advanced use cases enabled by it. The 3rd Generation Partnership Project (3GPP) develops protocols for mobile telecommunications and has developed a standard for core networks known as 5G Core (5GC). 
     The 5GC architecture is based on what is called a Service-Based Architecture (SBA), which implements IT network principles and a cloud-native design approach. In this architecture, each network function (NF) offers one or more services to other NFs via Application Programming Interfaces (API). Each NF, such as the UPF, the Network Data Analytics Function (NWDAF) and the Session Management Function (SMF) is formed by a combination of small pieces of software code called as microservices. 
     Briefly described, embodiments are directed toward systems and methods for predictive UPF load balancing within a 5G network. Example embodiments include systems and methods for determining a plurality of current loads for each UPF of a plurality of UPFs in a cellular telecommunication network. The plurality of UPFs serve as anchor points between UE in the cellular telecommunication network and an associated data network (DN). Each UPF of the plurality of UPFs is a virtual network function responsible for interconnecting Protocol Data Unit (PDU) sessions between the user UE and the DN by anchoring the PDU sessions on individual UPFs. The system receives a request to anchor on a UPF a PDU session of a new UE newly appearing on the cellular telecommunication network and selects a UPF of the plurality of UPFs on which to anchor the PDU session based on: a location of the new UE; the plurality of current loads for each UPF of the plurality of UPFs; a predicted UE load of the new UE based on network data analytics; and predicted UPF loads of the plurality of UPFs as a function of time considering the predicted UE load based on network data analytics. For example, the system may be configured to determine the plurality of current loads for each UPF of the plurality of UPFs; the predicted UE load of the new UE based on network data analytics; and the predicted UPF loads of the plurality of UPFs as a function of time considering the predicted UE load based on network data analytics captured or tracked by the NWDAF. 
    
    
     
       BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS 
       Non-limiting and non-exhaustive embodiments are described with reference to the following drawings. In the drawings, like reference numerals refer to like parts throughout the various figures unless otherwise specified. 
       For a better understanding of the present invention, reference will be made to the following Detailed Description, which is to be read in association with the accompanying drawings: 
         FIG.  1    illustrates a context diagram of an environment in which predictive UPF load balancing may be implemented in accordance with embodiments described herein. 
         FIG.  2    illustrates a logical flow diagram showing one embodiment of a process for predictive UPF load balancing in accordance with embodiments described herein. 
         FIG.  3    illustrates a logical flow diagram showing one embodiment of a process for predictive UPF load balancing in which weights are used in performing weighted averaging to determine a predicted average load over time of each UPF of the plurality of UPFs when selecting the UPF on which to anchor the PDU session in accordance with embodiments described herein. 
         FIG.  4    illustrates a logical flow diagram showing one embodiment of a process for predictive UPF load balancing in weighted averaging is performed to determine a predicted average load over time of the UPFs in accordance with embodiments described herein. 
         FIG.  5    illustrates a logical flow diagram showing one embodiment of a process for predictive UPF load balancing in weighted averaging to determine a predicted average load over time of the UPF is performed using an exponential filter in accordance with embodiments described herein. 
         FIG.  6    shows a system diagram that describes various implementations of computing systems for implementing embodiments described herein. 
     
    
    
     DETAILED DESCRIPTION 
     The following description, along with the accompanying drawings, sets forth certain specific details in order to provide a thorough understanding of various disclosed embodiments. However, one skilled in the relevant art will recognize that the disclosed embodiments may be practiced in various combinations, without one or more of these specific details, or with other methods, components, devices, materials, etc. In other instances, well-known structures or components that are associated with the environment of the present disclosure, including but not limited to the communication systems and networks, have not been shown or described in order to avoid unnecessarily obscuring descriptions of the embodiments. Additionally, the various embodiments may be methods, systems, media, or devices. Accordingly, the various embodiments may be entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. 
     Throughout the specification, claims, and drawings, the following terms take the meaning explicitly associated herein, unless the context clearly dictates otherwise. The term “herein” refers to the specification, claims, and drawings associated with the current application. The phrases “in one embodiment,” “in another embodiment,” “in various embodiments,” “in some embodiments,” “in other embodiments,” and other variations thereof refer to one or more features, structures, functions, limitations, or characteristics of the present disclosure, and are not limited to the same or different embodiments unless the context clearly dictates otherwise. As used herein, the term “or” is an inclusive “or” operator, and is equivalent to the phrases “A or B, or both” or “A or B or C, or any combination thereof,” and lists with additional elements are similarly treated. The term “based on” is not exclusive and allows for being based on additional features, functions, aspects, or limitations not described, unless the context clearly dictates otherwise. In addition, throughout the specification, the meaning of “a,” “an,” and “the” include singular and plural references. 
       FIG.  1    illustrates a context diagram of an environment in which predictive UPF load balancing may be implemented in accordance with embodiments described herein. 
     UEs  110 , such as cellular telephones or other Internet-of-Tings (IoT) devices use 5G wireless cellular telecommunication technology defined by standards set by 3GPP and International Telecommunications Union (ITU) to get data connectivity between applications on the UE and DNs such as the Internet or private corporate networks. Almost all applications running on the UE, including voice, require such data connectivity. A Protocol Data Unit (PDU) session provides connectivity between applications on a UE and a DN. The UE receives services through a PDU session, which is a logical connection between the UE and DN. A DN is identified by a Data Network Name (DNN). PDU sessions can provide different types of transport services corresponding to the nature of the PDU(s) carried over the PDU session. In various embodiments, a PDU session may be associated with a single DNN and with a single slice identified by Single-Network Slice Selection Assistance Information (S-NSSAI). 
     The UPF is one of the network functions (NFs) of the 5GC. The UPF, comprising UPF1  104  and UPF2  106  in the present example, is responsible for packet routing and forwarding, packet inspection, quality of service (QoS) handling, and interconnecting external PDU sessions with the DN. Although two UPFs (UPF1  104  and UPF2  106 ) are shown in the present example, additional UPFs may be utilized in various other embodiments. Each UPF (e.g., UPF1  104  and UPF2  106 ) is a virtual network function responsible for PDU sessions between the UEs  110  and the DN by anchoring the PDU sessions of various UEs  110  on the individual UPF. The SMF  102  is also one of the NFs of the 5GC and is primarily responsible for interacting with the decoupled data plane, creating updating and removing PDU sessions, selecting particular UPFs on which to anchor PDU sessions when new UEs appear on the network and managing session context with the UPF. Many of such functions are described in the 3GPP TS 23.501 specification. 
     Network data analytics is provided via the NWDAF  116  defined as part of the 5GC architecture by 3GPP in 3GPP TS 29.520, and is in operable communication with each UPF (e.g., UPF1  104  and UPF2  106 ), the SMF  102 , and may receive relevant data based on information originating from the various cellular telecommunication base stations  108 . NWDAF  116  incorporates interfaces from the service-based architecture to collect data by subscription or request model from other 5G network functions and procedures. The NWDAF  116  may collect data from UEs  110 , network functions, and operations, administration, and maintenance (OAM) systems, etc. from the 5GC, cloud, and edge networks that can be used for analytics. In an example embodiment, the current and predicted UE load of each of the UEs  110 , the current and predicted UPF load of each UPF (e.g., UPF1  104  and UPF2  106 ) and predicted UPF loads of each UPF considering the predicted additional UE load of any of the UEs  110  on a particular UPF may be based on data from, or as configured herein, determined by, the NWDAF  116 . The predicted load may be measured in units based on throughput (e.g., packets per second, bytes per second, and/or bits per second), the amount of bytes downloaded or uploaded by the UE, CPU utilization (e.g., CPU clock cycles, clock ticks, CPU time, CPU time per second, process time, percentage of CPU capacity utilization) and/or memory utilization, (megabytes of memory, and/or percentage of memory capacity utilization) or any combination thereof. 
     A network function, such as the NWDAF  116 , the SMF  102  and the UPF, (e.g., UPF1  104  and UPF2  106 ), and can be implemented either as a network elements on dedicated hardware, as a software instance running on dedicated hardware, or as a virtualized function instantiated on an appropriate platform, e.g., a cloud infrastructure. In the present example, UPF1  104  is implemented at data center 1 and UPF2  106  is implemented at data center 2, which is geographically separated from data center 1. The SMF  102  sends messages to the UPF (comprising UPF 1  104  and UPF 2 in the present example) over the N4 reference interface using the Packet Forwarding Control Protocol (PFCP). The PFCP may employ UDP port ( 8805 ) and is defined to support Control and User Plane Separation (CUPS). Decoupling other control plane functions from the user plane, together with the 5G Core Access and Mobility Management Function (AMF) (not shown), the standard allows the SMF  102  or UPF to perform the role of Dynamic Host Control Protocol (DHCP) server and Internet Protocol (IP) Address Management (IPAM) system. Together with the UPF, the SMF  102  maintains a record of PDU session state by means of a 24 bit PDU Session ID. The SMF  102  sets configuration parameters in the UPF that define traffic steering parameters and ensure the appropriate routing of packets while guaranteeing the delivery of incoming packets, though a Downlink (DL) data notification. 
     In an example embodiment, each UPF1  104  and UPF2  106  may have the ability to establish network connectivity and anchor PDU sessions of any UE on the network via various cellular telecommunication base stations and associated antennas  108 . To maximize network performance, PDU sessions are by default anchored on the UPF at the data center that is closest geographically to the UE, as illustrated by most of the dashed lines in  FIG.  1    for UEs  110  (and an operator defines a service area for each UPF). However, each UPF (e.g., UPF1  104  and UPF2  106 ) has a maximum network capacity to handle PDU sessions anchored thereon and the associated network traffic. Thus, PDU sessions anchored on a particular UPF (e.g., UPF1  104 ) and their associated network traffic may cause UPF1  104  to become too overloaded compared to other UPFs (eg., UPF2  106 ) if the PDU session of a new UE is anchored on it along with all the other UEs currently anchored on it. Predictive UPF load balancing may then cause the PDU session of the next new UE appearing on the network (e.g., UE  112 ) to be anchored on a UPF at a data center (e.g., UPF2  106 ) that is farther away than the data center that is closest geographically to the UE. 
     In the present example, predictive UPF load balancing based on network data analytics may indicate UPF1  104  will be too overloaded compared to UPF2  106  if the PDU session of UE  112  is anchored on it along with all the other UEs currently anchored on it, so UE  112  has a PDU session anchored on UPF2  106  (as shown by dashed line  114 ) instead of UPF1  104 , even though data center 2 of UPF2  106  is farther away from the UE  112  than data center 1 of UPF1  104 . In various embodiments described herein, there may be different particular scenarios and rules in which predictive UPF load balancing may cause the PDU session of the next new UE appearing on the network to be anchored on a UPF at a data center that is farther away than the data center that is closest geographically to the UE, which ultimately improves overall predictive UPF load balancing and network performance. For example, selection of a UPF on which to anchor the PDU may be such that larger weight is put on consideration of shorter term predictions of UPF loads included in the predicted UPF loads of UPF1 and UPF2 than longer term predictions of UPF loads included in the predicted UPF loads of UPF1 and UPF2 when selecting the UPF on which to anchor the PDU session. Also, weighting selection of a UPF on which to anchor the PDU session may be made such as to favor selection of a UPF (e.g., UPF1  104  or UPF2  106 ) that has a current cellular telecommunication network serving area geographically covering the location of the new UE (i.e., may favor selection of the UPF associated with the data center that is closest geographically to the new UE). 
       FIG.  2    illustrates a logical flow diagram showing one embodiment of a process  200  for predictive UPF load balancing in accordance with embodiments described herein. 
     At  202 , the SMF determines a plurality of current loads for each UPF of a plurality of UPFs in a cellular telecommunication network. The plurality of UPFs serve as anchor points between UE in the cellular telecommunication network and DN. Each UPF of the plurality of UPFs is a virtual network function responsible for interconnecting PDU sessions between the user UE and the DN by anchoring the PDU sessions on individual UPFs. 
     At  204 , the SMF  102  receives a request to anchor on a UPF a PDU session of a new UE newly appearing on the cellular telecommunication network. 
     At  206 , the SMF  102  selects a UPF of the plurality of UPFs on which to anchor the PDU session based on: a location of the new UE; the plurality of current loads for each UPF of the plurality of UPFs; a predicted UE load of the new UE based on network data analytics; and predicted UPF loads of the plurality of UPFs as a function of time considering the predicted UE load based on network data analytics. For example, the SMF  102  may be configured to determine the plurality of current loads for each UPF of the plurality of UPFs; the predicted UE load of the new UE based on network data analytics; and the predicted UPF loads of the plurality of UPFs as a function of time considering the predicted UE load based on network data analytics captured or tracked by the NWDAF  116 . In an example embodiment, the predicted UE load of the new UE based on network data analytics may be based on predicted throughput of the new UE. 
     At  208 , the SMF  102  anchors the PDU session of the new UE to the selected UPF. 
       FIG.  3    illustrates a logical flow diagram showing one embodiment of a process  300  for predictive UPF load balancing in which weights are used in performing weighted averaging to determine a predicted average load over time of each UPF of the plurality of UPFs when selecting the UPF on which to anchor the PDU session. Any combination of weights for different predictions of UPF loads may be used. However, in the present example embodiment, larger weight is put on consideration of shorter term predictions of UPF loads than longer term predictions of UPF loads in accordance with embodiments described herein. For example, the process  300  may be used in the process  200  of  FIG.  2    when selecting a UPF of the plurality of UPFs on which to anchor the PDU session. 
     At  302 , the SMF  102  selects a UPF of the plurality of UPFs on which to anchor the PDU session such that weights are used in performing weighted averaging to determine a predicted average load over time of each UPF of the plurality of UPFs when selecting the UPF on which to anchor the PDU session. In one example embodiment, larger weight is put on consideration of shorter term predictions of UPF loads included in the predicted UPF loads than longer term predictions of UPF loads included in the predicted UPF loads when selecting the UPF on which to anchor the PDU session. For example, the NWDAF  116  may be configured to generate and provide such predictions and weights to the SMF  102 . 
     At  304 , the SMF  102   102  anchors the PDU session of the new UE to the selected UPF. 
       FIG.  4    illustrates a logical flow diagram showing one embodiment of a process  400  for predictive UPF load balancing in weighted averaging is performed to determine a predicted average load over time of the UPFs in accordance with embodiments described herein. For example, the process  400  may be used in the process  300  of  FIG.  3    when selecting a UPF of the plurality of UPFs on which to anchor the PDU session such that larger weight is put on consideration of shorter term predictions. 
     At  402 , the SMF  102  receives a request to anchor on a UPF a PDU session of a new UE newly appearing on the cellular telecommunication network. 
     At  404 , the SMF  102  or NWDAF  116  performs weighted averaging to determine a predicted average load over time of a UPF of the plurality of UPFs based on an assumption the PDU session of the new UE anchors on the UPF. This may be performed by using a filter (e.g., an exponential filter) that weights shorter term predictions of UPF loads of the UPF more than longer term predictions of UPF loads of the UPF in the averaging. The selection of a UPF of the plurality of UPFs on which to anchor the PDU session may also be weighted such as to favor selection of a UPF that has a current cellular telecommunication network serving area geographically covering the location of the new UE. 
     At  406 , the SMF  102  determines whether there are additional UPFs in the plurality of UPFs on which the PDU session may be anchored. If it is determined there are additional UPFs on which the PDU session may be anchored, then the process  400  proceeds back to  404  performs weighted averaging to determine a predicted average load over time of the additional UPF of the plurality of UPFs based on an assumption the PDU session of the new UE instead anchors on the additional UPF. If it is determined there are not additional UPFs on which the PDU session may be anchored, then the process  400  proceeds to  408 . 
     At  408 , the SMF  102  selects a UPF of the plurality of UPFs on which to anchor the PDU session based on the determined predicted average load over time of each UPF of the plurality of UPFs considering the predicted average of the UE load of the new UE, and a location of the new UE that appears through a cost 
       FIG.  5    illustrates a logical flow diagram showing one embodiment of a process  500  for predictive UPF load balancing in weighted averaging to determine a predicted average load over time of the UPF is performed using an exponential filter in accordance with embodiments described herein. For example, the process  500  may be used in the process  400  of  FIG.  4    in performing the weighted averaging to determine a predicted average load over time of the UPF. 
     At  502 , the SMF  102  receives a request to anchor on a UPF a PDU session of a new UE newly appearing on the cellular telecommunication network. 
     At  504 , the NWDAF  116  performs the weighted averaging of the future UPF load of a current UPF (denoted as UPF) of the plurality of UPFs to determine a predicted average load over time of the UPF i  according to an exponential filter represented by a formula as follows. However, the exponential filter below is only one example embodiment of averaging filter realization. Other averaging filters may also be used in various other embodiments instead of an exponential filter. 
     
       
         
           
             
               Avg_load 
               ⁢ 
               
                 _UPF 
                 i 
               
             
             = 
             
               
                 ( 
                 
                   1 
                   - 
                   
                     1 
                     T 
                   
                 
                 ) 
               
               × 
               
                 
                   ∑ 
                   
                     t 
                     = 
                     
                       t 
                       0 
                     
                   
                   ∞ 
                 
                 
                   
                     1 
                     
                       T 
                       
                         ( 
                         
                           t 
                           - 
                           
                             t 
                             0 
                           
                         
                         ) 
                       
                     
                   
                   ⁢ 
                   Est_load 
                   ⁢ 
                   
                     _UPF 
                     i 
                   
                   ⁢ 
                   
                     ( 
                     t 
                     ) 
                   
                 
               
             
           
         
       
     
     The Avg_load_UPF i  represents the determined predicted average load over time of an “i”th UPF in the plurality of UPFs, represented by UPF i ; i=1, . . . N (in which N is a total number of UPFs in the plurality of UPFs); Est_load_UPF i (t) is an estimated load of UPF i  at a future point in time “t” assuming the PDU session of the new UE anchors on UPF i ; and “T” represents a time constant. 
     In an example embodiment, the current UPF loads of each UPF in the plurality of UPFs at time “t 0 ” may be represented by UPF 1 : L 1 (t 0 ), UPF 2 : L 2 (t 0 ), . . . UPF N : L N (t 0 ). NWDAF  116  uses these loads and the UPF load statistics of each UPF to create the predicted UPF loads of each UPF in the plurality of UPFs. In particular, the predicted load of the UPF “i” at time “t” is calculated as load_UPF i (t), for i=1, . . . , N. NWDAF  116  has the predicted load of the attaching UE at time “t” as load_UE(t). In an example embodiment, the predicted UE load of the new UE may be based on predicted throughput of the new UE. NWDAF  116  uses predicted load of the UPF and the predicted load of the attaching UE to calculate the predicted load of UPF “i” if the UE attaches to this UPF i  as Est_load_UPF i (t)=load_UPF i (t)+load_UE(t), for i=1, . . . , N. 
     The NWDAF  116  uses an averaging filter, such as an exponential filter with appropriate time-constant (memory) “T” (or any other appropriate filter with priority on shorter term predictions of UPF loads over longer term predictions of UPF loads) to predict the average load of UPF “i” if the new UE attaches to UPF i . Thus, this gives higher priority to statistics for shorter term predictions of UPF loads compared to longer term predictions of UPF loads. For example, as noted above, the exponential filter may be represented by 
     
       
         
           
             
               Avg_load 
               ⁢ 
               
                 _UPF 
                 i 
               
             
             = 
             
               
                 Exponential 
                 ⁢ 
                     
                 Average 
                 ⁢ 
                     
                 of 
                 ⁢ 
                     
                 
                   ( 
                   
                     
                       
                         load_UPF 
                         i 
                       
                       ⁢ 
                       
                         ( 
                         t 
                         ) 
                       
                     
                     + 
                     
                       load_UE 
                       ⁢ 
                       
                         ( 
                         t 
                         ) 
                       
                     
                   
                   ) 
                 
               
               = 
               
                 
                   Avg_load 
                   ⁢ 
                   
                     _UPF 
                     i 
                   
                 
                 = 
                 
                   
                     ( 
                     
                       1 
                       - 
                       
                         1 
                         T 
                       
                     
                     ) 
                   
                   × 
                   
                     
                       ∑ 
                       
                         t 
                         = 
                         
                           t 
                           0 
                         
                       
                       ∞ 
                     
                       
                     
                       
                         1 
                         
                           T 
                           
                             ( 
                             
                               t 
                               - 
                               
                                 t 
                                 0 
                               
                             
                             ) 
                           
                         
                       
                       ⁢ 
                       Est_load 
                       ⁢ 
                       
                         _UPF 
                         i 
                       
                       ⁢ 
                       
                         
                           ( 
                           t 
                           ) 
                         
                         . 
                       
                     
                   
                 
               
             
           
         
       
     
     At  506 , the NWDAF  116  weights selection of a UPF of the plurality of UPFs on which to anchor the PDU session such as to favor selection of a UPF that has a current cellular telecommunication network serving area geographically covering the location of the new UE. For example, the NWDAF  116  may receive the values representing the predicted average UPF loads generated by using the exponential filter as described above at  504  and add a cost represented by “C i ” to the Avg_load_UPF i  such that if UPF i  has a current cellular telecommunication network serving area geographically covering the location of the new UE, then “C i ” is “0”, otherwise, “C i ” is a positive number. This gives higher priority to keep the new UE on a UPF that has a current cellular telecommunication network serving area geographically covering the location of the new UE. 
     At  508 , the NWDAF  116  creates a weight “w i ” for the current UPF i  for as follows: w i =Avg_load_UPF i +C i  (wherein i=1, . . . N). 
     At  510 , the SMF  102  determines whether there are additional UPFs in the plurality of UPFs on which the PDU session may be anchored. If it is determined there are additional UPFs on which the PDU session may be anchored, then the process  500  proceeds back to  504  to perform the weighted averaging to determine a predicted average load over time of the additional UPF of the plurality of UPFs based on an assumption the PDU session of the new UE instead anchors on the additional UPF. If it is determined there are not additional UPFs on which the PDU session may be anchored, then the process  500  proceeds to  512 . 
     At  512 , the SMF  102  selects a UPF of the plurality of UPFs on which to anchor the PDU session with a lowest weight “w i ”. For example, the NWDAF  116  may electronically instruct the SMF  102  to select a UPF of the plurality of UPFs on which to anchor the PDU session with a lowest weight “w i ”. 
       FIG.  6    shows a system diagram that describes various implementations of computing systems for implementing embodiments described herein. 
     The SMF  102 , the NWDAF  116  and the UPF, such as UPF1  104  and UPF2  106 , can be implemented either as a network elements on dedicated hardware, as a software instance running on dedicated hardware, or as a virtualized function instantiated on an appropriate platform, e.g., a cloud infrastructure. In some embodiments, such NFs may be completely software-based and designed as cloud-native, meaning that they&#39;re agnostic to the underlying cloud infrastructure, allowing higher deployment agility and flexibility. However,  FIG.  6    illustrates an example of underlying hardware on which the SMF  102 , NWDAF  116  and the UPF, such as UPF1  104  and UPF2  106 , may be implemented. 
     In particular, shown are example SMF computer system(s)  601 , NWDAF computing system(s)  660  and UPF computer system(s)  612 . For example, SMF  102  may be implemented using SMF computer system(s)  601 . In some embodiments, one or more special-purpose computing systems may be used to implement SMF  102 . Accordingly, various embodiments described herein may be implemented in software, hardware, firmware, or in some combination thereof. SMF computer system(s)  601  may include memory  602 , one or more central processing units (CPUs)  614 , I/O interfaces  618 , other computer-readable media  620 , and network connections  622 . 
     Memory  602  may include one or more various types of non-volatile and/or volatile storage technologies. Examples of memory  602  may include, but are not limited to, flash memory, hard disk drives, optical drives, solid-state drives, various types of random access memory (RAM), various types of read-only memory (ROM), other computer-readable storage media (also referred to as processor-readable storage media), or the like, or any combination thereof. Memory  602  may be utilized to store information, including computer-readable instructions that are utilized by CPU  614  to perform actions, including embodiments described herein. 
     Memory  602  may have stored thereon SMF module  604 . The SMF module  604  is configured to implement and/or perform some or all of the functions of the SMF  102  described herein. Memory  602  may also store other programs and data  610 , which may include load thresholds, predicted loads, databases, load-balancing rules, AI or ML programs to perform predictive analysis of UPF load based on predicted UE throughput, CPU utilization and/or memory utilization using data from the NWDAF, user interfaces, operating systems, other network management functions, other NFs, etc. 
     Network connections  622  are configured to communicate with other computing devices to facilitate the load balancing described herein. In various embodiments, the network connections  622  include transmitters and receivers (not illustrated) to send and receive data as described herein, such as sending data to and receiving data from UPFs, UEs and other NFs to send and receive instructions, commands and data to implement the processes described herein. I/O interfaces  618  may include a video interfaces, other data input or output interfaces, or the like. Other computer-readable media  620  may include other types of stationary or removable computer-readable media, such as removable flash drives, external hard drives, or the like. 
     In some embodiments, one or more special-purpose computing systems may be used to implement UPF, such as UPF1  104  and UPF2  106 . Accordingly, various embodiments described herein may be implemented in software, hardware, firmware, or in some combination thereof. UPF computer system(s)  612  is an example of a computer system that may implement a UPF, such as UPF1  104  and UPF2  106 . For example, computer system(s)  612  may be present in data center 1 to implement UPF1  104  or present in data center 2 to implement UPF2  106 . Computer system(s)  612  may include memory  630 , one or more central processing units (CPUs)  644 , I/O interfaces  648 , other computer-readable media  650 , and network connections  652 . 
     Memory  630  may include one or more various types of non-volatile and/or volatile storage technologies similar to memory  602 . Memory  630  may be utilized to store information, including computer-readable instructions that are utilized by CPU  644  to perform actions, including embodiments described herein. 
     Memory  630  may have stored thereon UPF module  624 . The UPF module  64  receives the messages or instructions from the SMF module  204  to perform the load balancing operations as described herein. Memory  630  may also store other programs and data  638 , which may include load thresholds, databases, load-balancing rules, AI or ML programs to perform predictive analysis of UPF load based on predicted UE throughput, CPU utilization and/or memory utilization using data from the NWDAF, user interfaces, operating systems, other network management functions, other NFs, etc. 
     Network connections  652  are configured to communicate with other computing devices, such as SMF computer system(s)  601 . In various embodiments, the network connections  652  include transmitters and receivers (not illustrated) to send and receive data as described herein. I/O interfaces  648  may include one or more other data input or output interfaces. Other computer-readable media  650  may include other types of stationary or removable computer-readable media, such as removable flash drives, external hard drives, or the like. 
     The NWDAF computing system(s)  660  includes NWDAF module  664 , which is configured to implement and/or perform some or all of the functions of the NWDAF  116  described herein, including those of  FIGS.  2 - 5   . The NWDAF computing system(s)  660  may have the same or similar corresponding components as that shown in  FIG.  6    for SMF computer system(s)  601  and/or UPF computer system(s)  612  (e.g., memory storing NWDAF module  664 , CPU, I/O interfaces, other computer-readable media and network connections, etc.). 
     The various embodiments described above can be combined to provide further embodiments. These and other changes can be made to the embodiments in light of the above-detailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments along with the full scope of equivalents to which such claims are entitled. Accordingly, the claims are not limited by the disclosure.