Patent Description:
As described further below, an "on-premises" edge server (or "on-prem" edge server) may generally refer to a server in geographic proximity to IoT devices, whereas a "network" edge server may be located farther away from the IoT devices and may service multiple on-premises edge servers within a geographic area. To facilitate geographic proximity, edge servers are often physically constrained - which limits their memory and computing resources. In contrast to edge servers, cloud servers may be located in large-scale datacenters and may service a broad geographic region. In aspects, based on the distributed architecture of a cloud network, cloud servers may have access to significantly more processing and memory resources than the edge servers. This, combined with the fact that video analytics models are continually growing in size and complexity - and becoming more and more memory intensive - can cause latency and/or throughput issues at the edge servers. Not only so, but the number of IoT devices served by edge servers has increased exponentially. As an example, traffic monitoring for even a small city involves analyzing hundreds of live streams from distributed cameras in parallel.

It is with respect to these and other general considerations that the aspects disclosed herein have been made.

<CIT> describes a system that can create a new machine learning model by improving and combining existing machine learning models in a modular way. By combining existing machine learning models, the system can avoid the step of training a new machine model. Further, by combining existing machine models in a modular way, the system can selectively train only a module, i.e., a part, of the new machine learning model. Using the system, the expensive steps of gathering <NUM> TB of data and using the data to train the new machine learning model over <NUM>,<NUM> processors for three days can be entirely avoided, or can be reduced by a half, a third, etc. depending on the size of the module requiring training.

Aspects of the present disclosure relate to merging models (e.g., analytics models) for processing on an edge server. In examples, the system identifies layers that are sharable among a plurality of models and merges the models by instantiating one layer that corresponds to the sharable layers. The disclosed technology determines candidate layers for merging models based on similarity and/or matching of properties (e.g., input size, output size, kernel size, and stride) of each layer of respective models. In determining the candidate layers, the disclosed technology reduces a combinational search space among the models based on characteristics of the models. The edge system uses an intelligent heuristic to reduce an amount of retraining of models required for merging. In aspects, the reduced combinational search space and retraining of models due to the merging improves efficiency of memory use and performance in the edge server.

The disclosed technology relates to edge servers in an on-premises edge of a private cloud network that may interface with a radio access network (RAN). A RAN is a part of a mobile wireless telecommunications system. The RAN, in combination with a core network of a cloud service provider, represents a backbone network for mobile wireless telecommunications. According to <NUM> specifications, a RAN includes at least a radio unit (RU), a distributed unit (DU), a central unit (CU), and a RAN intelligent controller (RIC). Cell towers transmit and receive radio signals to communicate with mobile devices (e.g., smartphones) over radio (e.g., <NUM>). RUs at one or more cell towers connect to a DU of the edge server (e.g., a RAN server) at an on-premises edge of the cloud RAN. The term "on-premises edge" may refer to a datacenter at a remote location at the far-edge of a private cloud, which is in proximity to the one or more cell towers. Various service applications may perform different functions, such as network monitoring or video streaming, and may be responsible for evaluating data associated with the data traffic. For instance, a service application may perform data analytics, such as object recognition (e.g., object counting, facial recognition, human recognition) on a video stream.

Wireless telecommunication networks may be implemented by cloud services. In this case, the cloud service connects to cell towers, with which IoT devices connect, to the public network (e.g., the Internet) and/or private networks. The cloud service provides virtual servers and other computing resources for dynamically scaling the computing capacity as needed based on the volume of data traffic. To enable real-time processing of data traffic, the on-premises edge server is relatively close (e.g., a few kilometers) to the cell tower. As discussed in more detail below, the present disclosure relates to merging models used for processing data, such as video analytics, at an edge server. In particular, the edge server executes one or more service applications on the central processing unit (CPU) and/or heterogeneous accelerators (e.g., a graphics processing unit, GPU). The service applications use models to analyze stream data (e.g., video stream data from IoT devices) and create abstract data for further analyses in the cloud. The abstract data may include identities of object types and/or object counts recognized and/or predicted based on analytics performed on stream data.

The disclosed technology addresses the issue of scarce memory resources at edge servers by merging models used by one or more service applications. In particular, one or more layers with common properties are identified as sharable layers across multiple models at the edge server. In aspects, the one or more sharable layers may be identified in a descending order of memory consumption on the edge server. The common properties may include input and output data sizes, a kernel size, and strides of respective layers. The sharable layers may be reduced to a single common layer for retraining the layer for the respective models. The model merger instructs the edge server to replace the identified sharable layers in the respective models with the common layer, thereby merging the common portions of a plurality of models. By using the merged model with one or more common layers, the edge server reduces memory consumption when using or training the respective models.

<FIG> illustrates an overview of an example system <NUM> for merging models at an edge server of a cloud service in accordance with the aspects of the present disclosure. Cell towers 102A-C transmit and receive wireless communications with IoT devices (e.g., video cameras, health monitors, watches, appliances, etc.) over a telecommunications network. Video cameras 104A-C represent examples of IoT devices communicating with the cell towers 102A-C in the field. In aspects, the video cameras 104A-C are capable of capturing video images and transmit the captured video images over a wireless network (e.g., the <NUM> cellular wireless network) to one or more of the cell towers 102A-C. For example, respective video cameras 104A-C may capture scenes for video surveillance, such as traffic surveillance or security surveillance. The example system <NUM> further includes an on-premises edge <NUM> (including switches and edge servers), a network edge <NUM> (including core network servers), and a cloud datacenter <NUM> (including cloud servers responsible for providing cloud services). In aspects, the example system <NUM> corresponds to a cloud RAN infrastructure for a mobile wireless telecommunication network.

As illustrated, the on-premises edge <NUM> is a datacenter that is part of a cloud RAN, which includes distributed unit (DU) <NUM>, central unit (CU) <NUM>, and service application <NUM>. In aspects, the on-premises edge <NUM> enables cloud integration with a radio access network (RAN). The on-premises edge <NUM> includes a switch <NUM> and edge servers <NUM>. The switch <NUM> and the edge servers <NUM> process incoming data traffic and outgoing data traffic. The edge servers <NUM> execute service applications <NUM>. In aspects, the on-premises edge <NUM> is generally geographically remote from the cloud datacenters associated with the core network and cloud services. The remote site is in geographic proximity to the cell towers. For example, the proximity in the present disclosure may be within about a few kilometers. In aspects, the upstream data traffic corresponds to data flowing from the cell towers 102A-C to servers <NUM> in the cloud datacenter <NUM> (service). Similarly, the downstream data traffic corresponds to data flowing from the cloud datacenter <NUM> (service) to the cell towers. In further aspects, as datacenters become closer to the cloud datacenter <NUM>, server resources (including processing units and memory) become more robust and powerful. As an example, servers <NUM> may be more powerful than servers <NUM> and <NUM>, which may be more powerful than edge servers <NUM>. Conversely, the closer a datacenter is to connected devices (e.g., IoT devices), the more trusted the datacenter may be. In this case, edge servers <NUM> are more trusted than servers <NUM> and <NUM>, which are more trusted than servers <NUM>.

In aspects, the network edge <NUM> (e.g., hosting the core network) includes a central unit <NUM> (CU) and a RAN intelligent controller <NUM> (RIC) ("near" real-time processing, which may be less strictly time-sensitive than real-time processing). As illustrated, CU <NUM> is associated with servers <NUM> and RIC <NUM> is associated with servers <NUM>. In aspects, the network edge <NUM> is at a regional datacenter of a private cloud service. For example, the regional datacenter may be about tens of kilometers from the cell towers 102A-C. The network edge <NUM> includes service application <NUM> for performing data analytics. For example, the service application <NUM> includes video machine learning <NUM>, which performs and manages video analytics using machine learning technologies, such as neural networks, to train analytics models. Video machine learning <NUM> in the network edge <NUM> may, for example, perform merging and training/re-training of models using memory resources of the network edge <NUM>, which may be more expansive than the memory resources in the edge servers <NUM> of the on-premises edge <NUM>.

The cloud datacenter <NUM> (service) includes RIC <NUM> (non-real-time processing) associated with servers <NUM>. For example, RIC <NUM> processes non-real-time service operations. In aspects, the cloud datacenter <NUM> may be at a central location in a cloud RAN infrastructure. In this case, the central locations may be hundreds of kilometers from the cell towers 102A-C. In aspects, the cloud datacenter <NUM> includes service application <NUM> for performing data analytics. The service application <NUM> may perform similar processing tasks as service applications <NUM> in the network edge <NUM>, but may have access to more processing and memory resources at the cloud datacenter <NUM>.

In aspects, the on-premises edge <NUM>, which is closer to the cell towers 102A-C and to the video cameras 104A-C (or IoT devices) than the cloud datacenter <NUM>, may provide real-time processing. In contrast, the cloud datacenter <NUM>, which is the furthest from the cell towers 102A-C and video cameras 104A-C in the cloud RAN infrastructure, may provide processing in a non-real-time manner (e.g., such as training models).

In aspects, the accelerators in the edge servers <NUM> are heterogeneous. Some accelerators include pre-programmed logic for performing specific operational partitions. Some other accelerators are programmable. Some accelerators provide fast table lookups, while some other accelerators (e.g., a GPU) provide fast bit operations (e.g., processing graphics and video data).

The service application <NUM> includes program instructions for processing data according to predetermined data analytics scenarios on edge servers <NUM>. The predetermined analytics includes video machine learning <NUM> (Video ML). Video machine learning <NUM> performs video analytics by extracting and identifying objects from video stream data based on trained object scenarios. For example, video machine learning <NUM> may rely on a plurality of trained models to identify different types of objects (e.g., trees, animals, people, automobiles, etc.), a count of objects (e.g., a number of people in a video frame), and/or a particular object (e.g., a particular person based on facial recognition). In aspects, each model may be trained to identify a different type of object. The incoming video stream may include background data and object data, which the video cameras 104A-C captured and transmitted to the cell towers 102A-C. For example, the service application <NUM> may analyze the video stream and extract portions of the video stream as regions of interest, which regions of interest may comprise object data as opposed to background data. Once extracted, the regions of interest may be evaluated to recognize objects (e.g., a face of a person), as described above, or the service application may transmit the extracted regions of interest instead of the full video stream to the cloud for further processing (e.g., to verify recognition of the face of the person).

As described above, the service application <NUM> may use one or more models for recognizing and/or predicting objects during data analytics of video stream data. Respective models may be fine-tuned for performing distinct functions. For example, a model may accurately recognize faces of people and determine regions within video frames that correspond to the recognized faces. Another model may be fined tuned for recognizing automobiles (including particular automobile makes or models) that appear in the video frames. Some other model may be fine-tuned for recognizing and extracting voices of distinct people from audio data.

The models may include a plurality of layers of processing the video stream data sequentially or in parallel. For example, a model with a plurality of layers may constitute a neural network for predicting and recognizing objects in video stream data. The models may include a recurrent neural network and/or a convoluted neural network with multiple layers.

As will be appreciated, the various methods, devices, applications, features, etc., described with respect to <FIG> are not intended to limit the system <NUM> to being performed by the particular applications and features described. Accordingly, additional controller configurations may be used to practice the methods and systems herein and/or features and applications described may be excluded without departing from the methods and systems disclosed herein.

<FIG> illustrates an example system for merging models in accordance with aspects of the present disclosure. The system <NUM> includes an edge server <NUM> and a model merger <NUM>. In aspects, the edge server <NUM> is an on-premises server (e.g., one of the edge servers <NUM> of <FIG>). In some other aspects, the model merger <NUM> may, but is not limited to, execution on servers <NUM> in the network edge <NUM> and/or in servers <NUM> in the cloud datacenter <NUM>.

The edge server <NUM> includes a service application <NUM>, a model updater <NUM>, and a plurality of models (e.g., a first model <NUM>, a second model <NUM>, and a third model <NUM>, etc.). In aspects, the service application <NUM> performs data analytics. Examples of the data analytics may include analyzing video stream data from IoT devices (e.g., the video cameras 104A-C as shown in <FIG>), recognizing objects (e.g., people, faces, automobiles) from the video stream data, and extracting regions of interest associated with the recognized objects in the video. Some other examples of the data analytics include analyzing sound data and recognizing voices of predetermined people. The service application <NUM> may use one or more models for the data analytics. Respective models may be specialized for accurately performing predetermined tasks. For example, the first model <NUM> may be specialized for recognizing people (and/or a count of people) in the video stream data and the second model <NUM> may be specialized for facial recognition of particular people from the video stream data. The first model <NUM> and the second model <NUM> may further extract regions of the video stream data corresponding to the recognized people or faces, respectively. In aspects, a model includes multiple layers of processing. The model updater <NUM> updates one or more models (e.g., the first model <NUM>, the second model <NUM>, and the third model <NUM>) upon merging and/or splitting models.

The model merger <NUM> merges models for reducing the memory footprint of models in the edge server <NUM>. The model merger <NUM> includes a layer attribute receiver <NUM>, a refresh determiner <NUM>, a sharable layer determiner <NUM>, a model generator <NUM>, a model trainer <NUM>, a model transmitter <NUM>, and training data <NUM>. The layer attribute receiver <NUM> receives attributes associated with a plurality of layers of respective models from the edge server <NUM>. The attributes of respective layers include an amount of memory consumed, and in aspects input data size, output data size, kernel size, and a stride. For example, the stride represents a number of pixels to shift in each iteration of analyzing video and/or image data. Additionally or alternatively, the attributes include data drift, which represents a degree of deviation of output data from expected data. Data drift may impact an ability of the respective models and to recognize, predict, and extract data based on the data analytics. In aspects, a data drift that is greater than a predetermined threshold for a model indicates that the model needs to be improved by fine-tuning of parameters. Additionally or alternatively, when the model is a merged model, a portion of the merged model, which corresponds to the excessive data drift, may need to be detached (e.g. split).

In aspects, the model merger <NUM> merges select layers of different models. Some systems may consolidate layers in early sections of multi-layer models to reduce computation when executing inferences on a common set of input data (e.g., input video stream). These systems generate common stems of layers to reduce paths of processing data. In contrast, the model merger <NUM> merges select layers from any location within respective models to reduce the memory footprint of the models.

The refresh determiner <NUM> determines whether to refresh (or merge models) when a value of data drift is greater than a predetermined threshold.

The sharable layer determiner <NUM> determines a set of layers that are sharable among the models.

According to the invention, layers are sharable when a set of properties associated with the respective layers are within a predetermined threshold of similarity, including a case where values of each property of the set of properties among the respective layers match. For example, two layers are sharable when both layers have the same input data size, the same output data size, the same kernel size, and the same size of strides. In aspects, layers that are sharable between distinct models. In some other aspects, a single model may include layers that sharable. The sharable layer determiner <NUM> may analyze properties of layers for determining sharable layers in a descending order starting from the layer with the highest memory consumption. For example, the sharable layer determiner <NUM> generates a descending list of layers based on memory consumption for respective models. By consolidating sharable layers that are high in memory consumption, the disclosed technology may efficiently reduce a memory footprint of the overall models (e.g., by merging the heaviest layers first). In aspects, the sharable layer determiner <NUM> determines a set of sharable layers based on a combinational search on at least a part of layers in the sorted list of layers.

The model generator <NUM> generates a merged model from the models with the sharable layers by instantiating a single layer that corresponds to the sharable layers. In aspects, the generated model may include multiple entry points as input to the model and multiple exit points as output to the model, each pair of the entry points and the exit points representing a pair of input and output for previously distinct models (i.e., sub-models of the merged model). In aspects, the sub-models of the generated merged model may share one or more instantiations of intermediate layers.

The model trainer <NUM> trains the generated merged model. In aspects, the model trainer <NUM> uses training data <NUM>, which is a combination of training data for the previously distinct models. In aspects, merging of the models balances trade-offs between reduced accuracy in predicting inferences and a reduction in memory consumption. The training data <NUM> may include a predetermined threshold of accuracy that is sufficient or acceptable across all the models that are being merged. When a generated model exhibits a degree of accuracy that surpasses the predetermined threshold, the model may be suitable for performing data analytics. In some aspects, models reserve memory-intensive layers for learning specialized tasks. Accordingly, use of the aggregate training data across the merged models attains a sufficient level of accuracy for inference execution across the merged models. In aspects, the model trainer <NUM> may also update weights associated with consolidated layers in the generated model. In some examples, the model trainer <NUM> generates the trained model by executing the respective models or an exemplar model with high-fidelity on sampled frames from a target video feed.

The model transmitter <NUM> transmits at least the instantiated sharable layer to the edge server <NUM> for updating the models that are affected by the instantiated sharable layer in the edge server <NUM>. After transmitting the model with at least the instantiated sharable layer, the layer attribute receiver <NUM> continues receiving the status information associated with layers of the models.

In some aspects, including during a bootstrapping phase, the model transmitter <NUM> transmits unmerged models to the edge server <NUM>. The edge server <NUM> performs inference execution using the models based on GPU time-sharing mechanisms. In some other aspects, the model transmitter <NUM> transmits the merged structure of the generated model and the updated weights to the edge server <NUM>. The edge server <NUM> replaces individual models in GPU memory with the merged model during swapping operations. As the swapping operations continue over time, the merged models may completely replace the individual models. When a size of the memory of the edge server <NUM> is insufficient to create a single merged model that includes all layers of the individual models, the edge server <NUM> may time-slice the merged models on the GPU based on some scheduling mechanisms for executing instructions. In aspects the combined use of the merged models and time-slicing reduce the need for swapping models.

In aspects, the edge server <NUM> transmits one or more sample frames to the model merger <NUM> for determining when the merged models need to be refreshed. Data drift, or deviations in semantic accuracy of inference execution, may occur over time as the edge server <NUM> continues to operate on live video feeds. When the refresh determiner <NUM> determines the merged models need to be refreshed, the refresh determiner <NUM> uses the merged structure and weights associated with the merged structure as the initial condition to refine the layered structure of the merged models.

<FIG> illustrates an example of data structure for models in accordance with aspects of the present disclosure. The data structure <NUM> includes a merged model <NUM>. In aspects, the merged model <NUM> represents a model that the model generator <NUM> in the model merger <NUM> has generated based on multiple models with sharable layers.

In the example, the merged model <NUM> includes three sub-models that were respectively distinct models prior to the merger: a first sub-model 302A, a second sub-model 302B, and a third sub-model 302C. The first sub-model 302A includes an input as an entry point to the model, a first layer 306A, a first shared layer <NUM>, a third layer 310A, a first shared layer <NUM> (repeating), a last layer 314A, and output 316A as an exit point of the model. In aspects, prior to the merger, the model (now the first sub-model 302A) included two layers with common properties as repeating layers and thus were sharable. The first shared layer <NUM> in both positions in the first sub-model 302A uses a single instance of the first shared layer <NUM> in the memory as indicated in the shared layers <NUM>.

The second sub-model 302B includes input 304B, a first layer 306B, a second layer 308B, the first shared layer <NUM>, the second shared layer <NUM>, a last layer 314B, and output 316B. The third sub-model 302C includes input 304C, a first layer 306C, the second shared layer <NUM>, a third layer 310A, the first shared layer <NUM>, a last layer 314C, and output 316C. The first shared layer <NUM> in respective positions in respective sub-models use a single instance of the first shared layer <NUM> in the shared layers <NUM>. Similarly, the second shared layer <NUM> in the second sub-model 302B and the third sub-model 302C use a single instance of the second shared layer <NUM> in the shared layers <NUM>. As illustrated in the example, the disclosed technology merges models by sharing layers with common properties regardless of positions of the sharable layers in respective sub-models. Accordingly, in this example, use of the first shared layer <NUM> upon the merging models may reduce memory consumption from four instantiations to one instantiation. Similarly, use of the second shared layer <NUM> upon merging of the models may reduce memory consumption from two instantiations to one.

<FIG> illustrates an example of data structure for properties of layers in models in accordance with aspects of the present disclosure. The data structure <NUM> includes layer properties <NUM> for respective layers of models (e.g., a set of the distinct three models prior to the merger to generate the merged model <NUM> as shown in <FIG>). In aspects, the layer properties <NUM> includes model ID <NUM>, layer ID <NUM>, input data size <NUM>, kernel size <NUM>, stride <NUM> (in pixels), and output data size <NUM>. For example, a model with model ID <NUM> includes four layers (layer IDs <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>). In particular, some of layers have property values that are in common.

As shown as emphases in <FIG>, the four layers have common property values (i.e., the input data size of <NUM>, the kernel size of <NUM>, the stride of <NUM>, and the output data size of <NUM>. The four layers correspond to the second layer and the fourth layer of the first model, the third layer of the second model, and the fourth layer of the third model (e.g., as an example after the merger, refer to the first shared layer <NUM> of the first sub-model 302A, of the second sub-model 302B, and of the third sub-model 302C in the merged model <NUM> as shown <FIG>. ) Accordingly, sharable layers <NUM> includes the four layers as sharable layers, for which the model generator consolidates into one instantiation of a layer (e.g., the first shared layer <NUM> in the shared layers <NUM> as shown in <FIG>).

<FIG> illustrates an example data structure of a layer of a model in accordance with aspects of the present disclosure. The data structure <NUM> includes a layer <NUM> with input <NUM> and output <NUM>. The layer includes a weight <NUM>. In aspects, a layer functionality includes extracting features (e.g., people, faces, automobiles, etc.) of data and identifying a predetermined shape that represents objects. In some aspects, the layer <NUM> is a convolutional layer that filters image based on a predetermined filtering condition and outputs filtered data. In some other aspects, the layer <NUM> may be a part of a recursive neural network. The layer <NUM> may be a part of a convolutional neural network. In aspects, the weight <NUM> determines how various aspects of input data of the layer influence output data of the layer.

<FIG> is an example of a method for merging models in accordance with aspects of the present disclosure. A general order of the operations for the method 600A is shown in <FIG>. Generally, the method 600A begins with start operation <NUM> and ends with end operation <NUM>. The method <NUM> may include more or fewer steps or may arrange the order of the steps differently than those shown in <FIG>. The method 600A can be executed as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer readable medium. Further, the method 600A can be performed by gates or circuits associated with a processor, an ASIC, an FPGA, a SOC or other hardware device. Hereinafter, the method 600A shall be explained with reference to the systems, components, devices, modules, software, data structures, data characteristic representations, signaling diagrams, methods, etc., described in conjunction with <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, and <FIG>.

Following start operation <NUM>, the method 600A begins with transmit operation <NUM>, in which a set of models are transmitted to the edge server. In aspects, one or more service applications use the set of models in the edge server. The edge server includes processors and heterogeneous accelerators to execute the service applications. The processors include a CPU and the heterogeneous accelerators include GPU and NIC and other accelerators based on ASIC, FPGA, and NPU, as examples.

Sort operation <NUM> sorts layers of the models into a list based on memory consumption. In aspects, memory consumption by respective layers of the models may vary. By prioritizing consolidating layers with higher levels of memory consumption, the disclosed technology may reduce the memory footprint of the models more substantially than consolidating layers with lower levels of memory consumptions. In aspects, memory consumption of layers in a model exhibit a power-law distribution, where few layers contribute most of the model's memory usage while the larger number of remaining layers incur little overhead.

Based on memory consumption, select operation <NUM> selects a layer of a model to determine sharable layers. In aspects, the select operation <NUM> selects the layer with the highest level of memory consumption. The select operation <NUM> may select models used for the same or similar data analytics of data. For example, models used for processing video stream data from video cameras capturing scenes from the same or similar locations (e.g., video footage of a location from varying angles of views) or similar types of locations (e.g., scenes of hallways inside buildings, scenes of busy streets with automobile traffic). In aspects, data models used for analyzing similar scenes for the same or similar purposes may have sets of layers with similar properties.

Determine operation <NUM> determines sharable layers of the models. The determine operation <NUM> determines layers as sharable based on values of properties of the layers. The properties of the layers may include input size, output size, kernel size, and stride size (e.g., the layer properties <NUM> as shown in <FIG>). In aspects, the determine operation <NUM> determines layers of the models as sharable when values of respective properties among the layers match.

According to the invention, the determine operation <NUM> determines sharable layers when the values of respective properties among the layers are within a predetermined threshold of similarity.

Generate operation <NUM> generates a model by merging the models. The merger of the model includes consolidating instances of sharable layers (e.g., the first shared layer <NUM> as shown in <FIG>). In aspects, the generate operation <NUM> consolidates distinct instances of repeating layers of a model into one instantiation. In some other aspects, the generate operation <NUM> consolidates distinct instances of sharable layers across models into one instantiation.

Train operation <NUM> trains the merged model including the consolidated instantiation of sharable layers. In aspects, the train operation <NUM> trains the merged model for multiple epochs and/or until the trained layer attains a level accuracy that is higher than a predetermined threshold.

In aspects, the method 600A simultaneously merges and trains layers across models in an incremental fashion. The method 600A selects the first layer from the sorted list and sharing the first layer across models in the workload in which the first layer appears in the edge server. For merging, weights of a randomly selected model are used for initialization. Each time, a layer is merged, the train operation <NUM> trains the layer until the merged model satisfies a predetermined threshold of accuracy for each of the individual models. Additionally or alternatively, the train operation <NUM> ends training when a predetermined time has lapsed (e.g., a predetermined number of epochs of training). The pick-merge-train operation continues until it no longer is practical to merge any more layers without a reduction in accuracy of the merged model. Additionally or alternatively, the pick-merge-train operation ends when the cloud resources allocated to the training by the pipeline operator or an intelligent controller (e.g., the RAN intelligent controller <NUM> (RIC) in the cloud datacenter <NUM> as shown in <FIG>).

Transmit operation <NUM> transmits the merged model to the edge server. In aspects, the transmit operation <NUM> cause the edge server to swap one or more models with the merged models. The method 600A ends with the end operation <NUM>.

As should be appreciated, operations <NUM>-<NUM> are described for purposes of illustrating the present methods and systems and are not intended to limit the disclosure to a particular sequence of steps, e.g., steps may be performed in different order, additional steps may be performed, and disclosed steps may be excluded without departing from the present disclosure.

<FIG> is an example of a method for iteratively refreshing the models in accordance with aspects of the present disclosure. A general order of the operations for the method 600B is shown in <FIG>. Generally, the method 600B begins with start operation <NUM> and ends with end operation <NUM>. The method 600B may include more or fewer steps or may arrange the order of the steps differently than those shown in <FIG>. The method 600B can be executed as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer readable medium. Further, the method 600B can be performed by gates or circuits associated with a processor, an ASIC, an FPGA, a SOC or other hardware device. Hereinafter, the method 600B shall be explained with reference to the systems, components, devices, modules, software, data structures, data characteristic representations, signaling diagrams, methods, etc., described in conjunction with <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, and <FIG>.

Following start operation <NUM>, the method <NUM> begins with a set of operations as indicated by "A," which corresponds to a set of operations from the transmit operation <NUM> to the transmit operation <NUM> as shown in <FIG>. Following the transmit operation <NUM> that transmits the merged model, receive operation <NUM> receives information associated with data drift as detected by output of the merged model.

Determine operation <NUM> determines whether to refresh the merged model. In aspects, the determine operation <NUM> determines to refresh when a degree of data drift output from one or more sub-models in the merged model exceed a predetermined threshold.

Decision operation <NUM> decides whether to proceed to detach operation <NUM> when the determine operation <NUM> determines the merged model should be refreshed. In aspects, the detach operation <NUM> splits a model exhibiting the data drift from the merged model. In some other aspects, the detach operation <NUM> detaches one or more consolidated layers into independent layers among sub-models of the merged model. The detach operation <NUM> includes retraining of both the detached models. In some other aspects, the detach operation <NUM> includes retraining of the merged model with the one or more consolidated layers being detached within the merged model. When the determine operation <NUM> determines not to refresh the merged model, the decision operation <NUM> proceeds without performing the detach operation <NUM> to the end operation <NUM>. The method 600B ends with the end operation <NUM>.

<FIG> is a block diagram illustrating physical components (e.g., hardware) of a computing device <NUM> with which aspects of the disclosure may be practiced. The computing device components described below may be suitable for the computing devices described above. In a basic configuration, the computing device <NUM> may include at least one processing unit <NUM> and a system memory <NUM>. Depending on the configuration and type of computing device, the system memory <NUM> may comprise, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memories. The system memory <NUM> may include an operating system <NUM> and one or more program tools <NUM> suitable for performing the various aspects disclosed herein such. The operating system <NUM>, for example, may be suitable for controlling the operation of the computing device <NUM>. Furthermore, aspects of the disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application program and is not limited to any particular application or system. This basic configuration is illustrated in <FIG> by those components within a dashed line <NUM>. The computing device <NUM> may have additional features or functionality. For example, the computing device <NUM> may also include additional data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated in <FIG> by a removable storage device <NUM> and a non-removable storage device <NUM>.

As stated above, a number of program tools and data files may be stored in the system memory <NUM>. While executing on the at least one processing unit <NUM>, the program tools <NUM> (e.g., an application <NUM>) may perform processes including, but not limited to, the aspects, as described herein. The application <NUM> includes a layer status receiver <NUM>, a sharable layer determiner <NUM>, a model generator <NUM>, and a model trainer <NUM>, as described in more detail with regard to <FIG>. Other program tools that may be used in accordance with aspects of the present disclosure may include electronic mail and contacts applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided application programs, etc..

Furthermore, aspects of the disclosure may be practiced in an electrical circuit comprising discrete electronic elements, packaged or integrated electronic chips containing logic gates, a circuit utilizing a microprocessor, or on a single chip containing electronic elements or microprocessors. For example, aspects of the disclosure may be practiced via a system-on-a-chip (SOC) where each or many of the components illustrated in <FIG> may be integrated onto a single integrated circuit. Such an SOC device may include one or more processing units, graphics units, communications units, system virtualization units, and various application functionality all of which are integrated (or "burned") onto the chip substrate as a single integrated circuit. When operating via an SOC, the functionality, described herein, with respect to the capability of client to switch protocols may be operated via application-specific logic integrated with other components of the computing device <NUM> on the single integrated circuit (chip). Aspects of the disclosure may also be practiced using other technologies capable of performing logical operations such as, for example, AND, OR, and NOT, including but not limited to mechanical, optical, fluidic, and quantum technologies. In addition, aspects of the disclosure may be practiced within a general-purpose computer or in any other circuits or systems.

The computing device <NUM> may also have one or more input device(s) <NUM>, such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or swipe input device, etc. The output device(s) <NUM> such as a display, speakers, a printer, etc. may also be included. The aforementioned devices are examples and others may be used. The computing device <NUM> may include one or more communication connections <NUM> allowing communications with other computing devices <NUM>. Examples of the communication connections <NUM> include, but are not limited to, radio frequency (RF) transmitter, receiver, and/or transceiver circuitry; universal serial bus (USB), parallel, and/or serial ports.

Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, or program tools.

Communication media may be embodied by computer readable instructions, data structures, program tools, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media.

<FIG> and <FIG> illustrate a computing device or mobile computing device <NUM>, for example, a mobile telephone, a smart phone, wearable computer (such as a smart watch), a tablet computer, a laptop computer, and the like, with which aspects of the disclosure may be practiced. In some aspects, the client utilized by a user (e.g., as an operator of servers in the on-premises edge in <FIG>) may be a mobile computing device. With reference to <FIG>, one aspect of a mobile computing device <NUM> for implementing the aspects is illustrated. In a basic configuration, the mobile computing device <NUM> is a handheld computer having both input elements and output elements. The mobile computing device <NUM> typically includes a display <NUM> and one or more input buttons <NUM> that allow the user to enter information into the mobile computing device <NUM>. The display <NUM> of the mobile computing device <NUM> may also function as an input device (e.g., a touch screen display). If included as an optional input element, a side input element <NUM> allows further user input. The side input element <NUM> may be a rotary switch, a button, or any other type of manual input element. In alternative aspects, mobile computing device <NUM> may incorporate more or less input elements. For example, the display <NUM> may not be a touch screen in some aspects. In yet another alternative aspect, the mobile computing device <NUM> is a portable phone system, such as a cellular phone. The mobile computing device <NUM> may also include an optional keypad <NUM>. Optional keypad <NUM> may be a physical keypad or a "soft" keypad generated on the touch screen display. In various aspects, the output elements include the display <NUM> for showing a graphical user interface (GUI), a visual indicator <NUM> (e.g., a light emitting diode), and/or an audio transducer <NUM> (e.g., a speaker). In some aspects, the mobile computing device <NUM> incorporates a vibration transducer for providing the user with tactile feedback. In yet another aspect, the mobile computing device <NUM> incorporates input and/or output ports, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., a HDMI port) for sending signals to or receiving signals from an external device.

<FIG> is a block diagram illustrating the architecture of one aspect of computing device, a server (e.g., the edge servers <NUM> and the servers <NUM>, and other servers as shown in <FIG>) , a mobile computing device, etc. That is, the mobile computing device <NUM> can incorporate a system <NUM> (e.g., a system architecture) to implement some aspects. The system <NUM> can implemented as a "smart phone" capable of running one or more applications (e.g., browser, e-mail, calendaring, contact managers, messaging clients, games, and media clients/players). In some aspects, the system <NUM> is integrated as a computing device, such as an integrated digital assistant (PDA) and wireless phone.

One or more application programs <NUM> may be loaded into the memory <NUM> and run on or in association with the operating system <NUM>. Examples of the application programs include phone dialer programs, e-mail programs, information management (PIM) programs, word processing programs, spreadsheet programs, Internet browser programs, messaging programs, and so forth. The system <NUM> also includes a non-volatile storage area <NUM> within the memory <NUM>. The non-volatile storage area <NUM> may be used to store persistent information that should not be lost if the system <NUM> is powered down. The application programs <NUM> may use and store information in the non-volatile storage area <NUM>, such as e-mail or other messages used by an e-mail application, and the like. A synchronization application (not shown) also resides on the system <NUM> and is programmed to interact with a corresponding synchronization application resident on a host computer to keep the information stored in the non-volatile storage area <NUM> synchronized with corresponding information stored at the host computer. As should be appreciated, other applications may be loaded into the memory <NUM> and run on the mobile computing device <NUM> described herein.

The radio interface layer <NUM> facilitates wireless connectivity between the system <NUM> and the "outside world" via a communications carrier or service provider.

The visual indicator <NUM> (e.g., LED) may be used to provide visual notifications, and/or an audio interface <NUM> may be used for producing audible notifications via the audio transducer <NUM>. In the illustrated configuration, the visual indicator <NUM> is a light emitting diode (LED) and the audio transducer <NUM> is a speaker. These devices may be directly coupled to the power supply <NUM> so that when activated, they remain on for a duration dictated by the notification mechanism even though the processor <NUM> and other components might shut down for conserving battery power.

The LED may be programmed to remain on indefinitely until the user takes action to indicate the powered-on status of the device. The audio interface <NUM> is used to provide audible signals to and receive audible signals from the user. For example, in addition to being coupled to the audio transducer <NUM>, the audio interface <NUM> may also be coupled to a microphone to receive audible input, such as to facilitate a telephone conversation. In accordance with aspects of the present disclosure, the microphone may also serve as an audio sensor to facilitate control of notifications, as will be described below. The system <NUM> may further include a video interface <NUM> that enables an operation of an on-board camera <NUM> to record still images, video stream, and the like.

The present disclosure relates to systems and methods for merging models in an edge server according to at least the examples provided in the sections below. As will be understood from the foregoing disclosure, one aspect of the technology relates to a computer-implemented method. The method comprises selecting, based on memory consumption, a first layer of a first model; determining, based on layer properties, a second layer of a second model matches the first layer of the first model; generating a third model, wherein the third model includes a third layer corresponding to a single instantiation of the first layer and the second layer; and transmitting the third model to an edge server, wherein the edge server executes the third model to perform data analytics on the edge server, and wherein the third model consumes less memory on the edge server than executing both the first model and the second model. The first model and the second model correspond to video analytic models. The first set and second set of properties include one or more of: input size, output size, kernel size, or stride length. The first model and the second model are distinct. The method further comprises receiving data drift information associated with a first sub-model of the third model; generating, based on the data drift information associated with the first sub-model of the third model, a fourth model, wherein the fourth model includes the first sub-model; updating the third model by detaching the first sub-model from the third model; and transmitting the third model and the fourth model. The method further comprising training the third model based on training data associated with the first model and the second model. The training the third model comprises determining that the third model meets an accuracy threshold for performing the data analytics. The first layer and the second layer are not located in corresponding locations within the first model and the second model, respectively.

Another aspect of the technology relates to a system for merging models for use in an edge server. The system comprises a processor; and a memory storing computer-executable instructions that when executed by the processor cause the system to: select, based on memory consumption, a first layer of a first model; determine, based on layer properties, a second layer of a second model matches the first layer of the first model; generate a third model, wherein the third model includes a third layer corresponding to a single instantiation of the first layer and the second layer; and transmit the third model to an edge server, wherein the edge server executes the third model to perform data analytics on the edge server, and wherein the third model consumes less memory on the edge server than executing both the first model and the second model. The first model and the second model correspond to video analytic models. The first set and second set of properties include one or more of: input size, output size, kernel size, or stride length. The computer-executable instructions when executed further cause the system to: receive data drift information associated with a first sub-model of the third model; generate, based on the data drift information associated with the first sub-model of the third model, a fourth model, wherein the fourth model includes the first sub-model; update the third model by detaching the first sub-model from the third model; and transmit the third model and the fourth model. The computer-executable instructions when executed further cause the system to: train the third model based on training data associated with the first model and the second model. The training the third model comprises determining that the third model meets an accuracy threshold for performing the data analytics. The first layer and the second layer are not located in corresponding locations within the first model and the second model, respectively.

Claim 1:
A computer-implemented method for merging at least a part of a plurality of models, the method comprising:
selecting (<NUM>), based on an amount of memory consumption by respective layers of respective models, a first layer of a first model;
determining (<NUM>), sharable layers based on values of layer properties, the sharable layers comprise a second layer of a second model and the first layer of the first model when the values of respective properties of the first layer and the second layer are within a predetermined threshold of similarity;
generating (<NUM>) a third, merged model, wherein the third model includes a third layer, the third layer being instantiated as a single layer that corresponds to the first layer and the second layer; and
transmitting (<NUM>) the third model to an edge server, wherein the edge server executes the third model to perform data analytics on the edge server, and wherein the third model consumes less memory on the edge server than executing both the first model and the second model, wherein the first model and the second model correspond to trained video analytic machine learning models.