Patent ID: 12218801

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Advantages and features of the present invention and methods of achieving the same will become apparent with reference to preferred embodiments described in detail together with the accompanying drawings. However, the present invention is not limited to the following embodiments and may be implemented in various forms. The embodiments are merely provided to completely disclose the present invention and to fully inform those skilled in the art about the scope of the present invention, and the present invention is defined by the appended claims. Also, terms used herein are only for describing the embodiments while not limiting the present invention. Herein, the singular forms “a,” “an,” and “one” include the plural unless the context clearly indicates otherwise. Also, the terms “comprises” and/or “comprising” are used to specify the presence of stated elements, steps, operations, and/or components but do not preclude the presence or addition of one or more other elements, steps, operations, and/or components. Hereinafter, the preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. At this time, detailed descriptions of well-known elements or functions will be omitted if the descriptions may obscure the gist of the present invention.

FIG.1illustrates the concept of an adaptive deep learning inference system in a mobile edge computing (MEC) environment according to the present invention.

When a terminal device10senses data11and requests a deep learning inference service from an edge computing server20over a wireless access network30, the edge computing server20executes deep learning inferences using an adaptive deep learning inference system21according to the present invention. The adaptive deep learning inference system21adjusts latency (service latency) required to provide a deep learning inference result according to a change in latency31of the wireless access network30. The edge computing server20finally transmits a “data processing result13of deterministic latency12with service latency fixed” by the adaptive deep learning inference system21to the terminal device10to provide the deep learning inference service.

FIG.2is a block diagram of an adaptive deep learning inference system and method according to the present invention.FIG.2will be described with reference toFIG.1.

A data receiving unit210receives sensed data11transmitted over a wireless access network30by a terminal device10to request deep learning inference-based data processing.

A network latency measurement unit220is responsible for calculating round-trip network latency by measuring or predicting data latency required for data transmission between the terminal device10and the mobile edge computing server20.

An adaptive deep learning inference unit230is responsible for determining a deep learning model inference computation method that can satisfy a required deterministic latency treqof a deep learning inference service in consideration of the round-trip network latency Tnetcalculated by the network latency measurement unit220and performing computation. Here, the deep learning model inference computation selects a deep learning inference scheme that can process data with the maximum performance in the maximum latency tDLI=treq−tnetrequired to perform the deep learning inference computation. Formulated as an optimization problem, this may be expressed as follows.

[Equation⁢⁢1]⁢M*=argmaxM∈sM⁢⁢f(M),s.t.⁢T(M)≤Treq-Tnet(1)

In Equation 1 above, SMdenotes a set of applicable deep learning inference methods, f(M) denotes a deep learning inference performance index value that may be obtained when a deep learning inference method M is used, and T(M) denotes latency required when a deep learning inference method M is used. In this case, the deep learning inference performance index value f(M) may be defined differently depending on the service requirement in an application field where a deep learning model is utilized (e.g., in the case of an object detection service, accuracy (mAP), number of objects (number of classes), etc.).

A data processing result transmission unit240transmits a result value of data processing of the adaptive deep learning inference unit230to the terminal device10using the wireless access network30.

The adaptive deep learning inference system and method shown inFIG.2may be implemented in a software program. Also, all of the components shown inFIG.2may be built in the edge computing server20using an adaptive deep learning inference program. However, depending on the application, the components may be distributed in different physical areas. For example, in the case of an autonomous-vehicle object detection system having a base station between an autonomous vehicle and an edge computing server, the data receiving unit210and a data processing result transmission unit240are built in the base station, and the network latency measurement unit220and the adaptive deep learning inference unit230may be built in the edge computing server (this will be described in detail below with reference toFIG.3).

FIG.3is a conceptual diagram illustrating an embodiment in which an adaptive deep learning inference system is applied to an image analysis-based hierarchical object detection system of an autonomous vehicle according to the present invention. In general, a method of implementing hierarchical object detection can be implemented by generating a deep learning model for each hierarchical layer or generating a plurality of hierarchical output layers in a single deep learning model. How the adaptive deep learning inference system proposed by the present invention can be applied to the hierarchical object detection will be described below.

As shown inFIG.3, autonomous vehicles A, B, and C traveling on a road have different data latency depending on simultaneous wireless communication usage of adjacent vehicles and distances from a base station50of a multi-access edge computing (MEC) platform40providing an adaptive deep learning inference service of the present invention. The autonomous vehicles A, B, and C requesting an image analysis-based object detection service transmit image data IA, IB, and IC for object detection to the MEC platform40over a wireless network. At this time, the autonomous vehicle generates a data packet by adding time information t1regarding when transmitting data to a wireless network, deep learning inference service latency (i.e., the time required for inference service) Treq,A, Treq,B, Treq,C, image data, and deep learning-based object detection service requirements is started to the data packet and then transmits the data packet.

The data receiving unit210of the base station50of the MEC platform40receives the packet data transmitted by the vehicle and transfers a packet requesting a deep learning object detection service to a server60in the MEC platform40. In this case, the data receiving unit210of the base station50generates a packet by adding time information t2regarding when wireless data is received to the data and transmits the packet to the server60.

After receiving the packet from the data receiving unit210of the base station50, a network latency measurement unit220of the server60measures uplink data transmission latency required for uplink data transmission using the time information t1and t2included in the packet, predicts downlink data transmission latency, and calculates network latency tnet. In this case, the prediction of the data transmission latency may be performed through various technologies or techniques. The network latency measurement unit220calculates time TDLI,A, TDLI,B, TDLI,Crequired for deep learning inference computation by using the predicted network latency tnetand the deep learning inference service latency Treq,A, Treq,B, Treq,C. Subsequently, the network latency measurement unit220transfers the image data and deep learning-based object detection service requirements transmitted from the vehicle and the calculated time TDLI,A, TDLI,B, TDLI,Crequired for deep learning inference computation to the adaptive deep learning inference unit230.

The adaptive deep learning inference unit230determines a deep learning inference model candidate group on the basis of the image data and deep learning-based object detection service requirements transferred from the network latency measurement unit220, selects a model with the highest object detection level from among deep learning inference models capable of deep learning computation processing within the time required for deep learning inference computation, performs the deep learning inference computation, and then transfers result data of the deep learning inference computation to the data processing result transmission unit240of the base station50.

The data processing result transmission unit240of the base station50generates a data processing result transferred from the adaptive deep learning inference unit230in the form of a packet and then sends the data packet to the autonomous vehicles A, B, and C on the road over a wireless network. At this time, as one implementation method for ensuring end-to-end service latency in transferring the data packet from the base station50to a corresponding vehicle, a method of securing, in advance, a schedule for transmitting a corresponding data packet in a wireless network downlink transmission schedule of a base station may be used.

In the embodiment ofFIG.3, it is assumed that a great deal of time is required for deep learning inference computation measured by the network latency measurement unit220in the order of the autonomous vehicles A, B, and C. That is, the relationship between the time TDLI,Arequired for deep learning inference computation on vehicle A, the time TDLI,Brequired for deep learning inference computation on vehicle B, and the time TDLI,Crequired for deep learning inference computation on vehicle C is TDLI,A>TDLI,B>TDLI,Cas shown inFIG.4. Also, the deep learning inference unit230may generate a deep learning model having a different time required for deep learning inference computation in various ways, as shown inFIG.4.

FIG.4illustrates an example of a deep learning model lightweight method for various types of deep learning models performing the same function to successfully providing an inference computation service according to a time required for deep learning inference in order to implement the adaptive deep learning inference unit230. As a method of reducing the time required for inference computation by using deep learning models that perform the same function, there are 1) a pruning scheme to reduce the time required for inference computation by deleting computations including Weights, Channel, and Layers that minimize performance degradation of the deep learning model, 2) a quantization method to reduce the amount of computation and size by reducing a parameter representation scheme in the deep learning model in the order of FP32→FP16→INT8, and 3) a knowledge distillation method to train a small deep learning model with knowledge possessed by a conventional deep learning model. Through the above methods, it is possible to generate a deep learning model that performs the same function but has a different time required for inference computation.

In the embodiment ofFIG.3, by utilizing the methods ofFIG.4, the adaptive deep learning inference unit230utilizes various deep learning models secured in advance for the time TDLI,A>TDLI,B>TDLI,Crequired for deep learning inference computation to confirm deep learning inference service latency included in packet data transferred from the network latency measurement unit220, select a model with the highest accuracy from among deep learning models capable of completion of inference computation within corresponding deep learning inference service latency, and provide a deep learning inference service.

According to the present invention, a deep learning model inference method is adjusted according to the time required for data transmission occurring in a wireless access network, and thus it is possible to provide a deterministic latency service in providing a mobile edge computing-based deep learning data analysis service to a terminal device that requests the service.

The present invention has been described in detail with reference to the preferred embodiments, but those skilled in the art can understood that the present invention may be carried out in specific forms different from those described herein without changing the technical spirit or essential features of the present invention. Therefore, the above-described embodiments are to be regarded as illustrative rather than restrictive. Also, the scope of the present invention is defined not by the detailed description but by the following claims, and all changes or modifications within the claims and their equivalents will be construed as being included in the technical scope of the present invention.