Patent Description:
The present specification generally relates to utilization and management of machine learning (ML) models.

There are user equipment (UE) based ML use cases that realize and/or enhance control-plane (CP) network functions for radio access network (RAN) / core network (CN).

An example of a handover (HO) optimization use case shows how the network can benefit by running an artificial intelligence (AI) / ML model at the UE. As detailed in <NUM>rd Generation Partnership Project (3GPP) technical report (TR) <NUM>, the handover optimization is currently based on radio conditions for selecting the target gNodeB (gNB) upon handover. The handover process may result into wastage of UE and network resources and the handover process may introduce service disruption due to increased latency and radio link failures. To improve handover performances, the <NUM> system (5GS) may train an AI/ML inference model using UE-related data such as reference signal received power (RSRP), UE location, and UE trajectory from multiple UEs. Then, the 5GS distributes the trained AI/ML inference model to each UE for proactively triggering a handover based on the output of the AI/ML inference model.

Another example of a use case of AI/ML models is load prediction to prevent cell congestion as described in use case #<NUM> in clause <NUM>. <NUM> in TR <NUM>-<NUM>. In this use case, the 5GS, to prevent cell congestion, may train an AI/ML inference model to forecast a UE's load pattern. The 5GS forwards the trained AI/ML inference model to the UEs that can notify the 5GS about their load pattern forecast by using the input data such as UE-specific location data, channel and signal conditions, etc. Based on the load pattern forecast returned by the UEs, the 5GS is able to optimize the network, to thereby avoid cell congestion.

These UE-based ML use cases utilize their ML models and the data from the UEs' measurements of the network to make predictions and take proper actions. These ML models can be provided by the network or self-trained by the UEs with their measurements of a certain network. These ML models can work correctly only for the network (or its alike) where the profile / natural of its measurement data is same as or very similar to that of the data used to train these models.

There are also ML cases in which the model is shared among network functions (NF). In TR <NUM>-<NUM>, KI #<NUM> studies trained data model sharing between multiple network data analytics function (NWDAF) instances. According thereto, the possibility for a NWDAF consumer to request a new AI model in case the performance does not meet the requirements is foreseen.

When a UE with the ML model received from the network (trained e.g. by an NWDAF or a management data analytics function (MDAF)) moves away from the network where its measurements data have been used to train the model, or the network context changes, the UE may get into a different network with clearly different measurement data. In such case, the performance of the ML model and its corresponding action would likely degrade and become erroneous, as the network context has changed.

Hence, the problem arises that a UE may apply an AI/ML (inference) model which is not suitable (anymore) for the network situation experienced by the UE.

Hence, there is a need to provide for machine learning model renewal.

<CIT> discloses a model updating method to resolve a problem of network performance deterioration due to model performance deterioration. A first function entity sends a model update policy to a second function entity. The model update policy includes an update condition of a model of a first network element, and the model of the first network element indicates to adjust a parameter of the first network element. When receiving an update request, the first function entity performs a procedure of updating the model of the first network element, where the update request is triggered by the second function entity in accordance with determining, by the second function entity, the model meets the update condition.

Various example embodiments aim at addressing at least part of the above issues and/or problems and drawbacks.

Further embodiments are set out in the dependent claims.

Any one of the above aspects enables efficient ways to detect a mismatch with respect to the AI/ML (inference) model and the network (context), to indicate the mismatch, and to get a new (suitable) model from network to thereby solve at least part of the problems and drawbacks identified in relation to the prior art.

By way of example embodiments, there is provided machine learning model renewal. More specifically, by way of example embodiments, there are provided measures and mechanisms for realizing machine learning model renewal.

Thus, improvement is achieved by methods, apparatuses and computer program products enabling/realizing machine learning model renewal.

In the following, the present disclosure will be described in greater detail by way of non-limiting examples with reference to the accompanying drawings, in which.

The present disclosure is described herein with reference to particular non-limiting examples and to what are presently considered to be conceivable embodiments.

It is to be noted that the following description of the present disclosure and its embodiments mainly refers to specifications being used as non-limiting examples for certain exemplary network configurations and deployments. Namely, the present disclosure and its embodiments are mainly described in relation to 3GPP specifications being used as non-limiting examples for certain exemplary network configurations and deployments. As such, the description of example embodiments given herein specifically refers to terminology which is directly related thereto. Such terminology is only used in the context of the presented non-limiting examples, and does naturally not limit the disclosure in any way. Rather, any other communication or communication related system deployment, etc. may also be utilized as long as compliant with the features described herein.

Hereinafter, various embodiments and implementations of the present disclosure and its aspects or embodiments are described using several variants and/or alternatives. It is generally noted that, according to certain needs and constraints, all of the described variants and/or alternatives may be provided alone or in any conceivable combination (also including combinations of individual features of the various variants and/or alternatives).

According to example embodiments, in general terms, there are provided measures and mechanisms for (enabling/realizing) machine learning model renewal.

As discussed above, the problem arises that a UE may apply AI/ML (inference) models which are not suitable (anymore) for the network situation experienced by the UE.

The same problem exists also when the ML model is shared among NFs. Even though in TR <NUM>-<NUM> in clause <NUM>. <NUM> (Sol. #<NUM> to KI #<NUM>) the option to request a ML model update is included, no mechanisms to detect the need of ML model renewal are proposed.

Example embodiments thus address the necessity to detect a mismatch between the UE's pre-trained model and the current network context. When the mismatch is detected, the UE's model can be either renewed with the matching model provided by the current network, or the current model can be retrained on data from the current network context.

Example embodiments are hereinafter discussed in general terms.

According to example embodiments, a solution to detect the mismatch between the pre-trained model sent by the 5GS to the AI/ML model consumer (i.e., UE or NF or MnF) and the network context is provided.

According to example embodiments, the 5GS is responsible to train the AI/ML model that is being sent to the AI/ML model consumer for inference (AI/ML inference model, also mentioned as machine learning inference model). According to further example embodiments, if the ground truth data (i.e., real outputs of respective inputs) is not available to evaluate the performance of the inference model, the 5GS is also responsible to train another AI/ML model (AI/ML monitoring model, also mentioned as machine learning monitoring model) to profile the dataset used to train the AI/ML inference model.

According to example embodiments, function entities of an AI/ML model consumer, the 5GS, and a data producer (entity), as well as their interworking to detect the mismatch between the pretrained (pre-trained) model and the current network context are provided.

Based on the operational conditions, one of the following detection mechanisms is selected as the suitable detection mechanism to check whether the AI/ML inference model is working properly in terms of performance:.

An AI/ML monitoring model could be an unsupervised ML (monitoring) model able to identify whether there is a clear profile difference between the dataset utilized to train the inference model and the current data used as input to the inference model in runtime. When the AI/ML monitoring model detects the existence of such a clear profile difference, the AI/ML monitoring model determines that the current network context is clearly different from the network context where the AI/ML inference model has been trained with its training data. The 5GS is then requested to retrain the AI/ML inference model and, if relevant, the AI/ML monitoring model.

Example embodiments are applicable to any entity responsible for providing the ML model to the AI/ML model consumer and to any sharing mechanism adopted.

Example embodiments may impact open radio access network (O-RAN) specification documents, e.g. the AI/ML Workflow Technical Report, and the A1 specifications (for A1-ML - ML Model Management Service). Example embodiments may be reflected as well in the Non-Real Time RAN intelligent controller (RIC) Technical Report. Example embodiments may also impact R1 technical specification.

In addition, example embodiments impact TR <NUM>, TS <NUM> and TR <NUM>-<NUM> by providing the solution to the proposed use cases in the technical reports, and since MDAF and NWDAF are the management function and <NUM> core network (5GC) NF responsible to provide analytics, represent the best candidates to train, and share AI/ML models, according to example embodiments, these (MDAF and NWDAF) are enhanced to support the functionalities introduced herein in relation to the 5GS.

In addition, example embodiments impact any interface utilized by a sharing mechanism to provide the ML models to the AI/ML model consumer. An example of interfaces that may be impacted are N1, N2, Namf, and Nnwdaf in case the AI/ML models are sent to the AI/ML model consumer by the 5GS core network (CN), or Uu in the case the AI/ML models are shared by the RAN. According to example embodiments, these interfaces are enhanced to enable the exchange of messages, AI/ML inference and monitoring models, AI/ML model-related information such as performance data (including non-3GPP data) needed to perform the inferences and training of the models. In this case, example embodiments also impact TS <NUM>, TS <NUM>.

According to example embodiments, a method according to the above-discussed principle is also applicable for a network / OAM entity to monitor the operability of an inference model.

Subsequently, example embodiments as discussed above are explained in other words.

<FIG> is a block diagram illustrating an apparatus according to example embodiments. The apparatus may be a network entity <NUM> such as a user equipment (UE) or a network function (NF) entity or a management function (MnF) entity comprising a receiving circuitry <NUM>, an obtaining circuitry <NUM>, a feeding circuitry <NUM>, and a replacing circuitry <NUM>. The receiving circuitry <NUM> receives a first machine learning model message including a first machine learning inference model. The obtaining circuitry <NUM> obtains network related input data. The feeding circuitry <NUM> feeds said first machine learning inference model with said network related input data. The receiving circuitry <NUM>, further, receives, upon unsuitability of said first machine learning inference model for an experienced network condition, a second machine learning model message including a second machine learning inference model. The replacing circuitry <NUM> replaces said first machine learning inference model with said second machine learning inference model. <FIG> is a schematic diagram of a procedure according to example embodiments. The apparatus according to <FIG> may perform the method of <FIG> but is not limited to this method. The method of <FIG> may be performed by the apparatus of <FIG> but is not limited to being performed by this apparatus.

As shown in <FIG>, a procedure according to example embodiments comprises an operation of receiving (S61) a first machine learning model message including a first machine learning inference model, an operation of obtaining (S62) network related input data, an operation of feeding (S63) said first machine learning inference model with said network related input data, an operation of receiving (S64), upon unsuitability of said first machine learning inference model for an experienced network condition, a second machine learning model message including a second machine learning inference model, and an operation of replacing (S65) said first machine learning inference model with said second machine learning inference model.

<FIG> is a block diagram illustrating an apparatus according to example embodiments. In particular, <FIG> illustrates a variation of the apparatus shown in <FIG>. The apparatus according to <FIG> may thus further comprise a collecting circuitry <NUM>, a determining circuitry <NUM>, a transmitting circuitry <NUM>, an achieving circuitry <NUM>, and/or a checking circuitry <NUM>.

In an embodiment at least some of the functionalities of the apparatus shown in <FIG> (or <FIG>) may be shared between two physically separate devices forming one operational entity. Therefore, the apparatus may be seen to depict the operational entity comprising one or more physically separate devices for executing at least some of the described processes.

In particular a case is considered where ground truth is not available/accessible at the network entity (UE/NF/MnF), and where e.g. the 5GS/NWDAF collects the input data (network related input data).

According to a variation of the procedure shown in <FIG>, exemplary details of the obtaining operation (S62) are given, which are inherently independent from each other as such. Such exemplary obtaining operation (S62) according to example embodiments may comprise an operation of receiving said network related input data.

In particular a case is considered where the ground truth is not available/accessible at the network entity (UE/NF/MnF), and where e.g. the network entity (UE/NF/MnF) collects the input data.

According to a variation of the procedure shown in <FIG>, exemplary details of the obtaining operation (S62) are given, which are inherently independent from each other as such. According to such variation, said first machine learning model message includes a first machine learning monitoring model. Such exemplary obtaining operation (S62) according to example embodiments may comprise an operation of collecting said network related input data.

According to a variation of the procedure shown in <FIG>, exemplary additional operations are given, which are inherently independent from each other as such. According to such variation, an exemplary method according to example embodiments may comprise an operation of feeding said first machine learning monitoring model with said network related input data, an operation of determining a mismatch degree between said network related input data and a first training data set utilized for training of said first machine learning inference model and said first machine learning monitoring model based on a result of said first machine learning monitoring model fed with said network related input data, and an operation of transmitting, if said mismatch degree is larger than a predetermined value, a machine learning model retraining request message. Here, said machine learning model retraining request message includes a machine learning model retraining request and at least a portion of said network related input data.

According to further example embodiments, said second machine learning model message includes a second machine learning monitoring model.

In particular a case is considered where the ground truth available at the network entity (UE/NF/MnF).

According to a variation of the procedure shown in <FIG>, exemplary additional operations are given, which are inherently independent from each other as such. According to such variation, said first machine learning model message includes information on a first reference performance of said first machine learning inference model, and an exemplary method according to example embodiments may comprise an operation of achieving information indicative of ground truth data, and an operation of determining an actual performance of said first machine learning inference model by comparing a result of said first machine learning inference model fed with said network related input data with said information indicative of ground truth data.

According to a variation of the procedure shown in <FIG>, exemplary additional operations are given, which are inherently independent from each other as such. According to such variation, an exemplary method according to example embodiments may comprise an operation of checking whether said actual performance of said first machine learning inference model undercuts said first reference performance of said first machine learning inference model by more than a predetermined value, and an operation of transmitting, if said actual performance of said first machine learning inference model undercuts said first reference performance of said first machine learning inference model by more than a predetermined value, a machine learning model retraining request message. Here, said machine learning model retraining request message includes a machine learning model retraining request and at least a portion of said network related input data.

According to further example embodiments, said second machine learning model message includes information on a second reference performance of said second machine learning inference model.

<FIG> is a block diagram illustrating an apparatus according to example embodiments. The apparatus may be a network entity <NUM> such as the <NUM> system (5GS) (entity) or a network data analytics function (NWDAF) entity comprising a training circuitry <NUM>, a transmitting circuitry <NUM>, and a retraining circuitry <NUM>. The training circuitry <NUM> trains a first machine learning inference model based on a first training data set. The transmitting circuitry <NUM> transmits a first machine learning model message including said first machine learning inference model. The retraining circuitry <NUM> retrains, upon unsuitability of said first machine learning inference model for an experienced network condition, a second machine learning inference model based on a second training data set. The transmitting circuitry <NUM>, further, transmits a second machine learning model message including said second machine learning inference model. <FIG> is a schematic diagram of a procedure according to example embodiments. The apparatus according to <FIG> may perform the method of <FIG> but is not limited to this method. The method of <FIG> may be performed by the apparatus of <FIG> but is not limited to being performed by this apparatus.

As shown in <FIG>, a procedure according to example embodiments comprises an operation of training (S71) a first machine learning inference model based on a first training data set, an operation of transmitting (S72) a first machine learning model message including said first machine learning inference model, an operation of retraining (S73), upon unsuitability of said first machine learning inference model for an experienced network condition, a second machine learning inference model based on a second training data set, and an operation of transmitting (S74) a second machine learning model message including said second machine learning inference model.

<FIG> is a block diagram illustrating an apparatus according to example embodiments. In particular, <FIG> illustrates a variation of the apparatus shown in <FIG>. The apparatus according to <FIG> may thus further comprise a receiving circuitry <NUM>, a generating circuitry <NUM>, a feeding circuitry <NUM>, a determining circuitry <NUM>, a deciding circuitry <NUM>, a computing circuitry <NUM>, and/or an updating circuitry <NUM>.

According to a variation of the procedure shown in <FIG>, exemplary additional operations are given, which are inherently independent from each other as such. According to such variation, an exemplary method according to example embodiments may comprise an operation of transmitting a first data collection request, an operation of receiving a first data collection response including first collected data, and an operation of generating said first training data set based on said first collected data.

In particular a case is considered where the ground truth is not available/accessible at e.g. an UE/NF/MnF, and where the network entity (5GS/NWDAF) or e.g. the UE/NF/MnF collects the input data (network related input data).

According to a variation of the procedure shown in <FIG>, exemplary additional operations are given, which are inherently independent from each other as such. According to such variation, an exemplary method according to example embodiments may comprise an operation of training a first machine learning monitoring model based on said first training data set.

In particular a case is considered where the ground truth is not available/accessible at e.g. an UE/NF/MnF, and where the network entity (5GS/NWDAF) collects the input data.

According to a variation of the procedure shown in <FIG>, exemplary additional operations are given, which are inherently independent from each other as such. According to such variation, an exemplary method according to example embodiments may comprise an operation of transmitting network related input data, and an operation of feeding said first machine learning monitoring model with said network related input data.

According to a variation of the procedure shown in <FIG>, exemplary additional operations are given, which are inherently independent from each other as such. According to such variation, an exemplary method according to example embodiments may comprise an operation of transmitting a second data collection request, and an operation of receiving a second data collection response including second collected data as said network related input data.

According to a variation of the procedure shown in <FIG>, exemplary additional operations are given, which are inherently independent from each other as such. According to such variation, an exemplary method according to example embodiments may comprise an operation of determining a mismatch degree between said network related input data and said first training data set based on a result of said first machine learning monitoring model fed with said network related input data, and an operation of deciding, if said mismatch degree is larger than a predetermined value, to retrain said second machine learning inference model based on said network related input data as said second training data set.

According to a variation of the procedure shown in <FIG>, exemplary additional operations are given, which are inherently independent from each other as such. According to such variation, an exemplary method according to example embodiments may comprise an operation of training, if said mismatch degree is larger than a predetermined value, a second machine learning monitoring model based on said second training data set.

In particular a case is considered where the ground truth is not available/accessible at e.g. an UE/NF/MnF, and where e.g. the UE/NF/MnF collects the input data.

According to further example embodiments, said first machine learning model message includes said first machine learning monitoring model.

According to a variation of the procedure shown in <FIG>, exemplary additional operations are given, which are inherently independent from each other as such. According to such variation, an exemplary method according to example embodiments may comprise an operation of receiving a machine learning model retraining request message. Here, said machine learning model retraining request message includes a machine learning model retraining request and at least a portion of network related input data as said second training data set.

According to a variation of the procedure shown in <FIG>, exemplary additional operations are given, which are inherently independent from each other as such. According to such variation, an exemplary method according to example embodiments may comprise an operation of training a second machine learning monitoring model based on said second training data set. Here, said second machine learning model message includes said second machine learning monitoring model.

In particular a case is considered where the ground truth is available at e.g. an UE/NF/MnF.

According to a variation of the procedure shown in <FIG>, exemplary additional operations are given, which are inherently independent from each other as such. According to such variation, an exemplary method according to example embodiments may comprise an operation of computing a first reference performance of said first machine learning inference model based on said first machine learning inference model and said first training data set. Here, said first machine learning model message includes information on said first reference performance of said first machine learning inference model.

According to a variation of the procedure shown in <FIG>, exemplary additional operations are given, which are inherently independent from each other as such. According to such variation, an exemplary method according to example embodiments may comprise an operation of transmitting a second data collection request, an operation of receiving a second data collection response including second collected data, and an operation of updating said second training data set based on said second collected data.

According to a variation of the procedure shown in <FIG>, exemplary additional operations are given, which are inherently independent from each other as such. According to such variation, an exemplary method according to example embodiments may comprise an operation of computing a second reference performance of said second machine learning inference model based on said second machine learning inference model and said second training data set. Here, said second machine learning model message includes information on said second reference performance of said second machine learning inference model.

<FIG> is a block diagram illustrating an apparatus according to example embodiments. The apparatus may be a network entity <NUM> such as a data producer or an operations, administration and maintenance (OAM) entity comprising a receiving circuitry <NUM>, a collecting circuitry <NUM>, and a transmitting circuitry <NUM>. The receiving circuitry <NUM> receives a data collection request. The collecting circuitry <NUM> collects data. The transmitting circuitry <NUM> transmits a data collection response including said collected data. <FIG> is a schematic diagram of a procedure according to example embodiments. The apparatus according to <FIG> may perform the method of <FIG> but is not limited to this method. The method of <FIG> may be performed by the apparatus of <FIG> but is not limited to being performed by this apparatus.

As shown in <FIG>, a procedure according to example embodiments comprises an operation of receiving (S81) a data collection request, an operation of collecting (S82) data, and an operation of transmitting (S83) a data collection response including said collected data.

In an embodiment at least some of the functionalities of the apparatus shown in <FIG> may be shared between two physically separate devices forming one operational entity. Therefore, the apparatus may be seen to depict the operational entity comprising one or more physically separate devices for executing at least some of the described processes.

Subsequently, example embodiments as discussed above are explained in more specific terms with reference to <FIG>.

In particular, example embodiments explained below illustrate a specific mechanism to detect the profile difference (i.e., the mismatch), the specific mismatch detection procedures where the mismatch detection procedure to use is selected based on the conditions, and enhancement to the impacted interfaces with the UE taking the role of AI/ML model consumer. According to example embodiments, the same mechanisms are alternatively applied to NFs, e.g. with an NWDAF taking the role of AI/ML model consumer and the AI/ML models being shared by another NWDAF.

<FIG> is a diagram of a distribution of training observations and current data observations according to example embodiments, and in particular illustrates an example outlier/novelty detection with a local outlier factor (LOF).

As illustrated in <FIG>, a profile difference can be detected by a mismatch detection mechanism which utilizes an unsupervised ML model (ML monitoring model), e.g., a local outlier factor (LOF).

According to example embodiments, the profile of the data utilized in such case would consist of the following (<NUM>) to (<NUM>):.

According to example embodiments, this data profile is utilized for mismatch detection with the AI/ML monitoring model.

Namely, if data observations collected outside the above operational scope during the inference time amount to more than the threshold (item (<NUM>) above), there is a mismatch between the pretrained inference model and the current network context. Hence, retraining of both the inference model and the monitoring model is needed, and a respective decision is made according to example embodiments.

It is noted that there are different outlier / inlier /novelty detection solutions applicable. In the example of <FIG>, there are <NUM> novelty observations detected from the <NUM> consecutive observations, which is <NUM>% of the <NUM> current consecutive observations, well above, the example threshold of <NUM>%. In view thereof, in this example illustrated in <FIG>, the current network context is clearly different from the network context where the training data have been collected. The models thus need to be retrained with / including the data observations from the current network context, and a respective decision is made according to example embodiments.

<FIG>) shows a schematic diagram of signaling sequences according to example embodiments, and in particular illustrates mismatch detection procedures with one being selected based on the conditions described above (availability/accessibility of ground truth, responsibility of input data (network related input data) collection) in case 5GS is responsible to share the AI/ML models with the UE.

In detail, according to example embodiments, one of the mismatch detection procedures is selected according to the conditions described above (availability/accessibility of ground truth, responsibility of input data (network related input data) collection).

That is, according to example embodiments, ML model consumer (e.g. UE) and ML model provider (e.g. 5GS, NWDAF) are not only enabled to apply any of the disclosed mismatch detection procedures but may also be configured to select the suitable mismatch detection procedure based on the conditions described above (availability/accessibility of ground truth, responsibility of input data (network related input data) collection).

Namely, as is illustrated in <FIG>, as a first range of alternatives, the availability/accessibility of ground truth is considered to select detection mechanism <NUM> or <NUM> as discussed above (detection mechanism <NUM>, if the AI/ML model consumer has the ability to evaluate the performance of the AI/ML inference model, i.e., if the ground truth data related to the input data utilized for inference are available at the AI/ML model consumer, <FIG>; detection mechanism <NUM>, if the AI/ML model consumer does not have access to the ground truth or the ground truth is not available at the AI/ML model consumer, <FIG>, <FIG>).

Further, as is illustrated in <FIG>, as a second range of alternatives (in case detection mechanism <NUM> is selected), the responsibility of input data (network related input data) collection is considered to select detection mechanism 2a or 2b as discussed above (detection mechanism 2a, if the 5GS collects the input data, <FIG>; detection mechanism 2b, if the AI/ML model consumer is responsible to collect input data, <FIG>).

<FIG>) shows a schematic diagram of signaling sequences according to example embodiments, and in particular illustrates mismatch detection procedures with one being selected based on the conditions described above (availability/accessibility of ground truth, responsibility of input data (network related input data) collection) in case NWDAF (rather than 5GS) is responsible to share the AI/ML models with the UE.

Here, it is noted that an access and mobility management function (AMF) may be interposed between the NWDAF (being the ML model provider) and the UE (being the ML model consumer), where the AMF relays the communication between the ML model provider and the ML model consumer. Further, an OAM is exemplarily deployed as the data collector.

As already discussed in relation to <FIG>, according to example embodiments, ML model consumer (e.g. UE) and ML model provider (e.g. NWDAF) are not only enabled to apply any of the disclosed mismatch detection procedures but may also be configured to select the suitable mismatch detection procedure based on the conditions described above (availability/accessibility of ground truth, responsibility of input data (network related input data) collection).

Further, as is illustrated in <FIG>, as a second range of alternatives (in case detection mechanism <NUM> is selected), the responsibility of input data (network related input data) collection is considered to select detection mechanism 2a or 2b as discussed above (detection mechanism 2a, if the NWDAF collects the input data, <FIG>; detection mechanism 2b, if the AI/ML model consumer is responsible to collect input data, <FIG>).

As is further illustrated in <FIG>), in such scenario, example embodiments impact/enhance interfaces as follows.

Example embodiments impact interfaces utilized by any sharing mechanism, requiring an enhancement in order to convey the AI/ML inference and monitoring models along with their performance information. Thus, example embodiments would impact all the specifications related to the affected interfaces.

As an example (see <FIG>), a 5GC NF as NWDAF could be the responsible to collect the input data from the Data Producer (in this case represented by the OAM), training the AI/ML inference and monitoring models and then forward them to the UE (taking the role of AI/ML model consumer) through the AMF. In this example, interfaces as Nnwdaf, Namf, N1 would be impacted by example embodiments and would require an enhancement to allow the forwarding of the AI/ML inference and monitoring models along with their metadata (as architecture of the models, input/output format, analytic produced, etc.. ) and their performance. Example embodiments would then impact e.g. TS <NUM>, TS <NUM>, TS <NUM>.

For O-RAN, example embodiments might impact the following O-RAN specification documents: the AI/ML Workflow Technical Report, and the A1 specifications (the A1-ML - ML Model Management Service). Example embodiments may be reflected as well in the Non-Real Time RIC Technical Report.

In the foregoing exemplary description of the network entity, only the units that are relevant for understanding the principles of the disclosure have been described using functional blocks. The network entity may comprise further units that are necessary for its respective operation. However, a description of these units is omitted in this specification. The arrangement of the functional blocks of the devices is not construed to limit the disclosure, and the functions may be performed by one block or further split into sub-blocks.

In <FIG>, an alternative illustration of apparatuses according to example embodiments is depicted. As indicated in <FIG>, according to example embodiments, the apparatus (network entity) <NUM>' (corresponding to the network entity <NUM>) comprises a processor <NUM>, a memory <NUM> and an interface <NUM>, which are connected by a bus <NUM> or the like. Further, according to example embodiments, the apparatus (network entity) <NUM>' (corresponding to the network entity <NUM>) comprises a processor <NUM>, a memory <NUM> and an interface <NUM>, which are connected by a bus <NUM> or the like. Further, according to example embodiments, the apparatus (network entity) <NUM>' (corresponding to the network entity <NUM>) comprises a processor <NUM>, a memory <NUM> and an interface <NUM>, which are connected by a bus <NUM> or the like. The apparatuses may be connected via links <NUM>, <NUM>, respectively.

The processor <NUM>/<NUM>/<NUM> and/or the interface <NUM>/<NUM>/<NUM> may also include a modem or the like to facilitate communication over a (hardwire or wireless) link, respectively. The interface <NUM>/<NUM>/<NUM> may include a suitable transceiver coupled to one or more antennas or communication means for (hardwire or wireless) communications with the linked or connected device(s), respectively. The interface <NUM>/<NUM>/<NUM> is generally configured to communicate with at least one other apparatus, i.e. the interface thereof.

The memory <NUM>/<NUM>/<NUM> may store respective programs assumed to include program instructions or computer program code that, when executed by the respective processor, enables the respective electronic device or apparatus to operate in accordance with the example embodiments.

According to example embodiments, an apparatus representing the network entity <NUM> comprises at least one processor <NUM>, at least one memory <NUM> including computer program code, and at least one interface <NUM> configured for communication with at least another apparatus. The processor (i.e. the at least one processor <NUM>, with the at least one memory <NUM> and the computer program code) is configured to perform receiving a first machine learning model message including a first machine learning inference model (thus the apparatus comprising corresponding means for receiving), to perform obtaining network related input data (thus the apparatus comprising corresponding means for obtaining), to perform feeding said first machine learning inference model with said network related input data (thus the apparatus comprising corresponding means for feeding), to perform receiving, upon unsuitability of said first machine learning inference model for an experienced network condition, a second machine learning model message including a second machine learning inference model, and to perform replacing said first machine learning inference model with said second machine learning inference model (thus the apparatus comprising corresponding means for replacing).

According to example embodiments, an apparatus representing the network entity <NUM> comprises at least one processor <NUM>, at least one memory <NUM> including computer program code, and at least one interface <NUM> configured for communication with at least another apparatus. The processor (i.e. the at least one processor <NUM>, with the at least one memory <NUM> and the computer program code) is configured to perform training a first machine learning inference model based on a first training data set (thus the apparatus comprising corresponding means for training), to perform transmitting a first machine learning model message including said first machine learning inference model (thus the apparatus comprising corresponding means for transmitting), to perform retraining, upon unsuitability of said first machine learning inference model for an experienced network condition, a second machine learning inference model based on a second training data set (thus the apparatus comprising corresponding means for retraining), and to perform transmitting a second machine learning model message including said second machine learning inference model.

According to example embodiments, an apparatus representing the network entity <NUM> comprises at least one processor <NUM>, at least one memory <NUM> including computer program code, and at least one interface <NUM> configured for communication with at least another apparatus. The processor (i.e. the at least one processor <NUM>, with the at least one memory <NUM> and the computer program code) is configured to perform receiving a data collection request (thus the apparatus comprising corresponding means for receiving), to perform collecting data (thus the apparatus comprising corresponding means for collecting), and to perform transmitting a data collection response including said collected data (thus the apparatus comprising corresponding means for transmitting).

For the purpose of the present disclosure as described herein above, it should be noted that.

Generally, any method step is suitable to be implemented as software or by hardware without changing the idea of the present disclosure. Devices and means can be implemented as individual devices, but this does not exclude that they are implemented in a distributed fashion throughout the system, as long as the functionality of the device is preserved. Such and similar principles are to be considered as known to a skilled person.

The present disclosure also covers any conceivable combination of method steps and operations described above, and any conceivable combination of nodes, apparatuses, modules or elements described above, as long as the above-described concepts of methodology and structural arrangement are applicable.

Claim 1:
A method comprising
training (S71) a first machine learning inference model based on a first training data set,
computing a first reference performance of said first machine learning inference model based on said first machine learning inference model and said first training data set,
transmitting (S72), to a model consumer, a first machine learning model message including said first machine learning inference model and information on said first reference performance of said first machine learning inference model, upon determining unsuitability of said first machine learning inference model for a current network context of the model consumer based on said information, retraining the first machine learning inference model based on a second training data set, and
transmitting (S74), to the model consumer, a second machine learning model message including said retrained first machine learning inference model.