Patent Description:
Since a data amount of data collected by an IoT device represented by a sensor is enormous, an enormous communication amount is generated when data collected by cloud computing is aggregated and processed. Therefore, even in an edge device close to a user, attention is focused on edge computing that processes collected data.

However, a computation amount and resources such as a memory of a device used in the edge device are poor as compared with a device other than the edge device physically and logically disposed farther from the user than the edge device (hereinafter described as a cloud for convenience). For this reason, when processing with a large computation load is performed by the edge device, it may take a lot of time to complete the processing, or it may take time to complete another processing with a small computation amount. For example, there is a case where processing waiting occurs for another processing while processing with a large computation amount is performed in the edge device.

Here, one of the processing with the large computation amount is processing related to machine learning. Non Patent Literature <NUM> proposes application of so-called adaptive learning to an edge/cloud. That is, in a method described in Non Patent Literature <NUM>, a learned model learned by using general-purpose learning data in a cloud is developed in an edge device, and learning is performed again on the model learned by the cloud by using data acquired by the edge device, thereby realizing operation utilizing advantages of the cloud and the edge device.

<NPL>, relates to model cascades. A light moth model is placed in an IoT device and a high-precision model in the cloud. Inference processing is offload to the high-precision model only when the light-weight model's measurements are unreliable.

<CIT> relates to a recognition device which includes a reception unit for receiving a request for recognition processing to find a class to which input information belongs from a lower-level device. A recognition unit executes recognition processing by using dictionary information and outputting a score indicating similarity to the class to which the input information belongs. A determination unit determines whether to transfer the request to an upper-level device on the basis of the score and a transmission unit transmits the request to the upper-level device if it is determined that the request will be transferred. An update unit updates the dictionary information on the basis of the input information and the found class.

The edge/cloud network described above is expected to be applied to applications such as automatic analysis of surveillance camera images, automatic driving, and smart speakers, for example. In these applications, real-time performance is regarded as important together with accuracy, but communication cost and delay between the edge and the cloud have been problematic.

The present invention has been made in view of the above, and an object thereof is to provide a processing system, a processing method, and a processing program capable of reducing an amount of data transfer from an edge device to a server device and reducing delay.

According to the present invention, it is possible to reduce an amount of data transfer from the edge device to the server device and reduce delay.

Note that the present invention is not limited by this embodiment. Further, the same portions are denoted by the same reference signs in the description of the drawings.

[Embodiment] [Outline of Embodiment] An embodiment of the present invention will be described. In the embodiment of the present invention, a processing system that performs inference processing using a learned high-accuracy model and a learned lightweight model will be described. In the processing system of the embodiment, any neural network may be used, and signal processing with a low computation amount and signal processing with a high computation amount may be used instead of the learned models.

<FIG> is a diagram for describing an outline of a processing method of the processing system according to the embodiment. In the processing system of the embodiment, the high- accuracy model and the lightweight model constitute a model cascade. In the processing system of the embodiment, whether the processing is executed in an edge device using a high-speed and low-accuracy lightweight model (for example, DNN1 (first model)) or a cloud (server device) using a low-speed and highly accurate high-accuracy model (for example, DNN2 (second model)) is controlled using reliability. For example, the server device is a device disposed at a place physically and logically far from a user. The edge device is an IoT device and a variety of terminal devices arranged in a place physically and logically close to the user, and has fewer resources than the server device.

The DNN1 and the DNN2 are models that output inference results on the basis of input processing target data. In an example of <FIG>, the DNN1 uses an image as an input and infers probability for each class of an object appearing in the image on the basis of a feature amount extracted from the image. The DNN2 uses the feature amount (meaning an output of a predetermined intermediate layer, and hereinafter referred to as a feature map) extracted by the DNN1 as an input, and infers probability for each class of the object appearing in the image to be inferred. Note that both the DNN1 and the DNN2 perform inference on the same image. Furthermore, in the processing system, in the edge device, it is determined which inference result of the DNN1 or the DNN2 is adopted on the basis of a comparison result between reliability (described later) and a threshold. The reliability is a value for determining which of the edge device and the server device should process the processing target data. The reliability may be obtained, for example, on the basis of entropy of the inference result by the DNN1. The reliability may be set to a value that decreases as the entropy of the inference result of the DNN1 increases. Specific examples will be described later.

As illustrated in <FIG>, the processing system acquires the reliability of the inference of the class classification of the DNN1 with respect to the object appearing in the input image. Then, in the processing system, in a case where the acquired reliability exceeds a predetermined threshold, the inference result of the DNN1 is adopted. That is, the inference result of the lightweight model is output as a final estimation result of the model cascade. On the other hand, in the processing system, in a case where the reliability is equal to or less than the predetermined threshold, the inference result obtained by inputting the feature map extracted by the DNN1 to the DNN2 is output as the final inference result.

As described above, the processing system according to the embodiment selects the edge device or the server device on the basis of the comparison result between the reliability and the threshold as to which of the edge device and the server device should process the processing target data, and processes the processing target data. Therefore, the processing system according to the embodiment can control whether the processing is executed in the edge device or the cloud.

[Lightweight Model and High-Accuracy Model] Next, the DNN1 and the DNN2 will be described. <FIG> and <FIG> are diagrams for explaining an example of the DNN1 and the DNN2. The DNN includes an input layer into which data is input, a plurality of intermediate layers that variously converts the data input from the input layer, and an output layer that outputs a so-called inferred result such as probability and likelihood. An output value output from each layer may be irreversible when data to be input needs to maintain anonymity.

As described above, the processing system uses the DNN2 on the cloud side and uses the DNN1 on the edge side. As DNN2' before being arranged on the cloud side, a high-accuracy model including a feature extraction layer Bf2 that extracts a feature amount of an input image and outputs the feature amount as a feature map and a detection layer Bd2 (second execution unit) that is a layer that performs processing using the extracted feature amount (for example, a layer that detects an object appearing in the input image or infers probability for each class of the object appearing in the input image (hereinafter collectively referred to as the detection layer Bd2)) is adopted (see <FIG>). In the processing system, learning data is used in advance to train the DNN2' before arrangement.

Then, in the processing system, before being arranged on the edge side, instead of a feature amount extraction layer Bf1 of DNN1' before being arranged on the edge side, the feature extraction layer Bf2 of the trained DNN2' is arranged as it is in a preceding stage of a lightweight detection layer Bd1 on the edge side (see an arrow Y1 in <FIG>), and a DNN obtained by combining the feature extraction layer Bf2 (first extraction unit) and the detection layer Bd1 (first execution unit) is set as the DNN1 on the edge side (see <FIG>). The detection layer Bd1 uses a feature map (feature amount of a certain layer) extracted by the feature extraction layer Bf2 of the DNN1 to infer probability for each class of the object appearing in the input image. Here, a configuration of the DNN1 arranged on the edge side and the DNN2 arranged on the cloud side will be described. In the DNN1 arranged on the edge side, the feature amount extraction layer Bf1 (see <FIG>) is deleted from the original DNN1, and the feature amount extraction layer Bf2 of the trained DNN2' is arranged instead (see <FIG>). That is, the DNN1 including a combination of the feature amount extraction layer Bf2 of the DNN2' and the detection layer Bd1 of the DNN1' is arranged on the edge side. Note that the feature amount extraction layer Bf2 of the DNN1 illustrated in <FIG> may be fixed and learning may be performed again. Furthermore, the DNN2 arranged on the server side has a configuration in which the feature amount extraction layer Bf2 is deleted from the original DNN2' (see <FIG>) (see <FIG>).

For the DNN1, parameters of the feature extraction layer Bf2 are fixed to parameters after training of the original DNN2', and the detection layer Bd1 in a subsequent stage is trained using learning data. Alternatively, the DNN1 is trained using the learning data for both the feature extraction layer Bf2 and the detection layer Bd1. In addition, the DNN2 and the DNN1 may perform learning independently, or the DNN2 and the DNN1 may perform learning in cooperation with each other. For example, the DNN2 may be retrained using the learning data used by the DNN1. Furthermore, training may be performed using learning data common between the DNN1 and the DNN2.

In the processing system according to the present embodiment, the feature extraction layer Bf2 in the original DNN2' is arranged as the feature extraction layer Bf2 of the DNN1 on the edge side. Therefore, the detection layer Bd2 of the DNN2 on the cloud side can execute inference processing using the feature map output from the feature extraction layer Bf2 of the DNN1 on the edge side (see an arrow Y2 in <FIG>). Therefore, in the processing system according to the present embodiment, it can be said that inference processing can be performed on each of the edge side and the cloud side by sharing the same feature map between the edge and the cloud.

As a result, in a case where inference is performed on the cloud side, execution of feature extraction processing can be omitted. Therefore, calculation time can be shortened, and delay can be reduced. In addition, since the data output from the edge side to the cloud side is not the image to be processed but the feature map extracted from the image, an amount of data transfer from the edge side to the cloud side can be reduced. Note that, since it is only necessary to share the same feature map between the edge and the cloud, a minimum condition to be satisfied by the feature extraction layer Bf2 of the DNN1 actually arranged on the edge side and the feature extraction layer Bf2 of the original DNN2' is that sizes of layers to be connected coincide with each other. This is because relearning is performed again for the detection layer of the DNN1. Note that values of the parameters may be different between the feature extraction layer Bf2 of the DNN1 and the feature extraction layer Bf2 of the DNN2'.

[Processing System] Next, a configuration of the processing system will be described. <FIG> is a diagram schematically illustrating an example of the configuration of the processing system according to the embodiment.

A processing system <NUM> according to the embodiment includes a server device <NUM> and an edge device <NUM>. In addition, the server device <NUM> and the edge device <NUM> are connected via a network N. The network N is, for example, the Internet. For example, the server device <NUM> is a server provided in a cloud environment. Furthermore, the edge device <NUM> is, for example, an IoT device and a variety of terminal devices.

Each of the server device <NUM> and the edge device <NUM> is implemented by, for example, a predetermined program being read by a computer or the like including a read only memory (ROM), a random access memory (RAM), a central processing unit (CPU), and the like, and the CPU executing the predetermined program. In addition, so-called accelerators represented by a GPU, a vision processing unit (VPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), and a dedicated artificial intelligence (AI) chip are also used. Each of the server device <NUM> and the edge device <NUM> includes a network interface card (NIC) or the like, and can perform communication with another device via a telecommunication line such as a local area network (LAN) or the Internet.

As illustrated in <FIG>, the server device <NUM> includes an inference unit <NUM> (second inference unit) that performs inference using DNN2 that is a learned high-accuracy model. The DNN2 includes information such as model parameters. The DNN2 has the detection layer Bd2 as described above.

The inference unit <NUM> uses the DNN2 to execute inference processing on an input image on the basis of a feature map of the input image output from the edge device <NUM>. The inference unit <NUM> returns the quantized feature map output from the edge device <NUM> to FP32 and uses it as an input of the detection layer Bd2 of the DNN2. The inference unit <NUM> acquires an inference result (for example, probability for each class of an object appearing in the image) as an output of the DNN2. The inference unit <NUM> receives inference data, that is, an output value of the feature extraction layer Bf2 of the DNN1, and outputs the inference result. It is assumed that the feature map is a feature amount of data whose label is unknown. For example, the inference data is an image. Furthermore, in a case where the inference result is returned to a user, the inference result obtained by the inference unit <NUM> may be transmitted to the edge device <NUM> and returned from the edge device <NUM> to the user.

Here, the server device <NUM> and the edge device <NUM> constitute a model cascade. Therefore, the inference unit <NUM> does not always perform inference. In a case where it is determined to cause the server device <NUM> to execute the inference processing, the inference unit <NUM> receives an input of the quantized feature map and performs inference by the detection layer Bd2 of the DNN2.

The edge device <NUM> includes an inference unit <NUM> (first inference unit) including the DNN1 that is a learned lightweight model, a determination unit <NUM>, and a quantization unit <NUM>.

The inference unit <NUM> performs inference using the DNN1 that is the learned lightweight model. The DNN1 includes information such as model parameters. The DNN1 has the feature extraction layer Bf2 of the trained DNN2' (see <FIG>) and the detection layer Bd1 as described above. The inference unit <NUM> extracts a feature amount of an input image as a feature map using the DNN1, and executes inference processing on the input image on the basis of the extracted feature map.

The inference unit <NUM> inputs an image to be processed to the DNN1 and acquires an inference result. The inference unit <NUM> extracts a feature amount of processing target data using the DNN1, and executes inference processing on the processing target data on the basis of the extracted feature amount. The inference unit <NUM> receives the input of the image to be processed, processes the image to be processed, and outputs an inference result (for example, probability for each class of an object appearing in the image).

The determination unit <NUM> determines which inference result of the edge device <NUM> and the server device <NUM> is adopted by comparing reliability with a predetermined threshold.

In a case where the reliability exceeds the predetermined threshold, the determination unit <NUM> outputs the inference result inferred by the inference unit <NUM>. In a case where the reliability is equal to or less than the predetermined threshold, the determination unit <NUM> outputs the feature map, which is an output of the feature extraction layer Bf2 connected to the DNN1 on the edge side, to the server device <NUM>, and determines to cause the DNN2 arranged in the server device <NUM> to execute the inference processing.

As described above, the reliability may be set to a value that decreases as the entropy of the inference result of the DNN1 increases. The entropy of the inference result of the DNN1 may be obtained by Equation (<NUM>). C is a set of all the labels c output from the DNN1, and y is probability of each label. The reliability may be obtained from the entropy obtained in this manner, for example, as reliability = <NUM>/entropy. [Equation <NUM>] <MAT>.

In a case where the determination unit <NUM> determines to cause the server device <NUM> to execute the inference processing, the quantization unit <NUM> quantizes the feature map extracted by the feature extraction layer Bf2 of the DNN1 and outputs the quantized feature map to the server device <NUM>.

[Model Selection Example of DNN1 and DNN2] <FIG> is a diagram for describing a selection example of base models of the DNN1 and the DNN2. In the present embodiment, as a model for inferring probability for each class of an object, as illustrated in <FIG>, darknet <NUM> (hereinafter referred to as YOLOv2), which is a relatively lightweight and high-speed back-end model of YOLOv2, is selected as a base model of the edge device <NUM>, and darknet <NUM> (hereinafter referred to as YOLOv3), which is a relatively high-accuracy back-end model of YOLOv3, is selected as a base model of the server device <NUM>. The selected NNs are an example, and any selection may be made as long as they are an NN with high accuracy and an NN with high speed and lower accuracy than the NN with high accuracy, and NNs capable of sharing a feature amount extraction layer. In a simple example, the edge device <NUM> and the server device <NUM> may be configured to have different depths in the same NN.

<FIG> is a diagram illustrating an outline of a structure of YOLOv3. It is a diagram for explaining a selection example of base models of the DNN1 and the DNN2. The YOLOv3 includes a convolutional layer (feature extraction layer) Bf-<NUM> that performs feature extraction having a Residual block Bf-<NUM> and a detection unit Bd2-<NUM> that is a network for object detection (FPN). Here, in order to share a feature map between the edge and the cloud, attention is paid to the following.

First, by including the Residual block Bf-<NUM>, high accuracy is maintained in the YOLOv3. Therefore, it is desirable to avoid impairing a configuration in the Residual block Bf-<NUM>. Subsequently, in the YOLOv3, it is desirable to have a configuration in which the feature map is received in a preceding stage of the detection unit Bd2-<NUM> so that the detection unit Bd2-<NUM> can execute detection using the feature map as it is. Note that the deeper the layer is (the more the layer is), the heavier the calculation of the model becomes, and the number of parameters to be used also increases. Therefore, the DNN1 of the edge device <NUM> is desirably a model having a shallow layer in order to be lightweight and high-speed.

[Example of Structures of DNN1 and DNN2] <FIG> and <FIG> are diagrams for explaining an example of a structure of DNN1 and DNN2. In order to avoid impairing the configuration in the Residual block Bf-<NUM> of the YOLOv3, a learned feature extraction layer Bf2-<NUM> holding the Residual block Bf-<NUM> is arranged as it is in a preceding stage of YOLOv2 selected as the base model of the edge device <NUM> (an arrow Y11 in <FIG>), and is set as the DNN1 of the edge device <NUM>. Therefore, in the edge device <NUM>, a feature map is extracted from an image in the feature extraction layer Bf2-<NUM> using the YOLOv2 reconstructed in this way, and probability for each class of an object appearing in the input image is inferred in a detection layer Bd1-<NUM> using the extracted feature map.

Then, in a case where reliability of a result (Result1) of the detection layer Bd1-<NUM> exceeds a predetermined threshold, the edge device <NUM> determines that the Result1 is reliable, outputs the Result1 (see an arrow Y11 in <FIG>), and ends the processing. Here, as described above, since the feature extraction layer Bf2-<NUM> in the edge device <NUM> has a common structure with the feature extraction layer Bf2-<NUM> of the YOLOv3 which is the base model of the server device <NUM>, the feature map output from the feature extraction layer Bf2-<NUM> can also be shared with the detection unit Bd2-<NUM> of the server device <NUM>. Note that, in the server device <NUM>, YOLOv3 having a configuration in which the feature extraction layer Bf2-<NUM> is deleted from the YOLOv3 is applied as the DNN2.

Therefore, in a case where the Result1 is equal to or less than the threshold, the edge device <NUM> determines that the Result1 is not reliable, and outputs a feature map of the feature extraction layer Bf2-<NUM> to the edge device <NUM> after quantization (see an arrow Y12 in <FIG>). Then, the server device <NUM> returns this feature map to FP32, inputs the feature map to the detection unit Bd2-<NUM> of the YOLOv3 after the reconstruction, and outputs an inference result (Result2) of the detection unit Bd2-<NUM> (see an arrow Y13 in <FIG>). Therefore, a calculation range in the server device <NUM> is a calculation range in the detection unit Bd2-<NUM>. In other words, in the server device <NUM>, calculation in the feature extraction layer Bf2-<NUM> can be omitted.

[Processing Procedure of Processing System] <FIG> is a sequence diagram illustrating a flow of processing of the processing system according to the embodiment. As illustrated in <FIG>, first, in the edge device <NUM>, when receiving an input of an image (step S1), the inference unit <NUM> inputs the input image to the DNN1. In the DNN1, the feature extraction layer Bf2 extracts a feature amount of the input image as a feature map (step S2) and outputs it to the determination unit <NUM> (step S3). In the DNN1, the detection layer Bd1 executes inference processing, for example, detection processing on the input image on the basis of the feature map (step S4), and outputs an inference result to the determination unit <NUM> (step S5).

The inference unit <NUM> calculates reliability on the basis of the inference result (step S6), and the determination unit <NUM> compares the calculated reliability with a predetermined threshold and determines whether the reliability is equal to or less than the predetermined threshold (step S7).

In a case where the reliability is not equal to or less than the threshold (step S7: No), that is, a case where the reliability exceeds the threshold, the determination unit <NUM> outputs the inference result inferred by the DNN1 of the inference unit <NUM> (step S8).

On the other hand, in a case where the reliability is equal to or less than the threshold (step S7: Yes), the determination unit <NUM> outputs the feature map to the quantization unit <NUM> (step S9), and the quantization unit <NUM> quantizes the feature map (step S10) and transmits it to the server device <NUM> (step S11).

In the server device <NUM>, the inference unit <NUM> returns the quantized feature map output from the edge device <NUM> to FP32 and uses it as an input of the detection layer Bd2 of the DNN2. The detection layer Bd2 executes inference processing, for example, detection processing on the input image on the basis of the feature map output from the edge device <NUM> (step S12). The server device <NUM> transmits an inference result of the DNN2 to the edge device <NUM> (step S13), and the inference result is output from the edge device <NUM> (step S14). Note that, in the present embodiment, a configuration is assumed in which the inference result is returned to a user, and the final inference result is output from the edge device <NUM>. However, in a case where the final inference result is used on the server device <NUM> side, the inference result of the DNN2 may be output from the server device <NUM>, or may be held in the server device <NUM> as it is. In a case where the inference result of the DNN1 is used, the edge device <NUM> may transmit the inference result to the server device <NUM> in a case where the inference result is used on the server device <NUM> side.

[Evaluation Experiment <NUM>] Processing time of the DNN1 of the edge device <NUM> and processing time of the DNN2 of the server device were evaluated. A task is classification, test data is ImageNet (<NUM> images for each class, total of <NUM> images), and HW is Platform: NVIDIA GeForce RTX <NUM> + AMD <NUM>. <FIG> is a diagram illustrating processing time of each DNN selected as an example in the processing system <NUM>.

In <FIG>, DNN1 of the edge device is a model in which a learned feature extraction layer Bf2-<NUM> holding a Residual block Bf-<NUM> is arranged in a preceding stage of YOLOv2 and training is performed. DNN2 of the server device <NUM> is learned YOLOv3. For comparison, processing time is also indicated for YOLOv2 which is a base model of the edge device <NUM>.

As illustrated in <FIG>, the DNN1 of the edge device is deeper than the YOLOv2 by the feature extraction layer Bf2-<NUM>, so that the processing time is longer than the YOLOv2, but inference accuracy is improved more than the YOLOv2. Then, the processing time per image of the DNN1 of the edge device is about two times faster than that of the YOLOv3 which is the DNN2 of the server device <NUM>. As described above, even in a case where the feature extraction layer Bf2-<NUM> of the YOLOv3 is arranged instead of the feature extraction layer of the YOLOv2, the DNN1 of the edge device can maintain high speed processing while increasing the inference accuracy.

[Evaluation Experiment <NUM>] Distribution of entropy of test data was visualized using the DNN1 of the edge device <NUM>. Conditions of a task, the test data, and HW are the same as those in <FIG>. <FIG> is a distribution diagram of the entropy of the test data obtained on the basis of the inference result of the DNN1.

The higher the entropy, the less reliable the inference result of the DNN1. Therefore, it can be said that it is appropriate to adopt reliability as an evaluation value for determining whether to execute processing in the edge device or the server device in the model cascade. Then, since the entropy of most data is <NUM> or less, it is considered that a threshold of the entropy to be used when it is determined whether to execute processing in the edge device or the server device may be set to <NUM> as a guide.

[Evaluation Experiment <NUM>] Therefore, Evaluation Experiment <NUM> was performed to obtain a variation in overall accuracy of the inference result associated with a variation in the threshold (offload rate) of the entropy. Note that the threshold is linked with the offload rate, and the threshold is increased in a case where the offload rate is decreased. In this evaluation experiment, quantization on the feature map extracted by the feature extraction layer Bf2 of the edge device <NUM> was performed at int4, int6, and int8. Note that conditions of a task, test data, and HW are the same as those in <FIG>. <FIG> is a diagram illustrating a relationship between an offload rate and overall accuracy. In <FIG>, "Offload rate <NUM>" is a state in which original accuracy (acc_origin) is low in a case where all the data is processed by the edge device <NUM> and is not quantized, and "Offload rate <NUM>" is a state in which the original accuracy (acc_origin) is high in a case where all the data is processed by the server device <NUM> and is not quantized. Int4 quantization, int6 quantization, and int8 quantization are performed on the feature map by the feature extraction layer Bf2 of the edge device <NUM>.

Among them, as illustrated in <FIG>, among the int4 quantization, the int6 quantization, and the int8 quantization, in a case where the int8 quantization is performed on the feature map by the feature extraction layer Bf2 of the edge device <NUM>, an amount of transfer data can be reduced by <NUM>% as compared with a case where the quantization is not performed without substantially deteriorating the accuracy. Therefore, it is desirable to apply the int8 quantization to the quantization of the feature map.

Note that a cost term may be provided such that the feature map output by the feature extraction layer Bf2 is sparser. As a result, when the int6 and/or the int4 indicate the offload rate and/or the overall accuracy close to the int8 in <FIG>, a smaller one of the quantization bit rates indicating the offload rate and/or the overall accuracy close to the int8 may be selected from between the int6 and the int4.

In addition, the quantized feature map may be compressed. In a case where compression is to be performed, both the feature map and the quantized feature map are included in a target image, and have properties similar to those of a natural image. Therefore, an image encoding system such as HEVC or VVC may be adopted, or a general-purpose compression system such as ZIP may be adopted. Note that the properties similar to those of the natural image described above mean general properties incorporated in image encoding, such as high correlation between adjacent pixels in many cases. In addition, inverse quantization may be performed when quantization is performed, and decoding may be performed when compression is performed. The quantization and/or the compression may be performed by the edge device, or may be performed by another device physically or logically disposed closer to the edge device (than to the server device). The inverse quantization and/or the decoding may be performed by the server device, or may be performed by another device physically or logically arranged closer to the server device (than to the edge device).

In addition, when the offload rate exceeds <NUM> (entropy threshold is <NUM>), improvement in accuracy is small even if the offload rate is increased, that is, even if the entropy threshold is decreased. Therefore, when the threshold is set to <NUM>, it is considered that the offload rate and the accuracy are balanced. As described above, by setting the threshold according to the balance between the offload rate and the accuracy, the offload rate and the overall accuracy can be adjusted according to each use case.

Here, a reason for determining the threshold of the entropy will be further described. Lowering of the threshold of the entropy means that an increase in the offload rate. That is, the number of data to be inferred on the server device side may be increased by lowering the threshold of the entropy.

Here, a threshold for determining whether inference is performed on the edge device side or on the server device side is considered. As an example, it is conceivable to use the overall accuracy as a reference. It is desirable that the threshold be set such that the processing can be performed on the edge device side as long as the accuracy does not change so much even if the inference is performed on the edge device side or the server device side, and the processing can be performed on the server device side in a case where the accuracy decreases when the processing is performed on the edge device side. For example, in a case of <FIG>, it can be seen that the threshold of the offload rate may be set to <NUM> in a case where the quantization bit rate is <NUM> (acc_int8). Then, the threshold of the entropy of the inference result on the edge side may be determined so that the offload rate becomes <NUM>. In the case of <FIG>, the threshold of the entropy may be about <NUM>.

[Effects of Embodiment] As described above, in the embodiment, by arranging the feature extraction layer Bf2 in the DNN2' before being arranged in the server device <NUM> as the feature extraction layer Bf2 of the DNN1 of the edge device <NUM>, the feature map output from the edge device <NUM> can also be shared with the server device <NUM>. That is, the detection layer Bd2 of the DNN2 of the server device <NUM> can execute the inference processing using the feature map output from the feature extraction layer Bf2 of the DNN1 of the edge device <NUM>.

Therefore, since execution of the feature extraction processing can be omitted when the inference is performed on the server device <NUM> side, calculation time of the entire system can be shortened, and delay can be reduced. In addition, since the output data from the edge device <NUM> to the server device <NUM> is not the image to be processed but the feature map extracted from the image, an amount of data transfer from the edge device <NUM> to the server device <NUM> can be reduced. In addition, since the edge device transmits, to the server device, a feature map that is neither an image to be processed or an encoded image nor a generally used feature amount such as a frequency signal, it is also possible to improve confidentiality of a third party. In a case of reliably securing the confidentiality, a constraint that makes a relationship between the feature map and the input data (target image) irreversible may be further applied when learning is performed.

In the embodiment, the DNN1 in which the feature extraction layer Bf2 in the DNN2' before being arranged in the server device <NUM> is arranged in the preceding stage of the detection layer Bd1 of the edge device <NUM> has been proposed. This DNN1 has actually performed a classification task, and can be evaluated to be effective as a lightweight model in both accuracy and speed.

Furthermore, in the present embodiment, the reliability used when determining which inference result of the edge device <NUM> and the server device <NUM> is adopted is formulated as entropy of the inference result of the DNN1, and the threshold is set, whereby an offload rate and overall accuracy can be adjusted according to a use case at the time of actual operation.

Note that the image has been described as an example of the processing target data in the present embodiment, but the processing target data is not limited to the image, and may be detection results detected by various sensors. Furthermore, in the present embodiment, a case where the YOLOv2 and the YOLOv3 are applied as the base models of the DNN1 and the DNN2 has been described as an example, but the base models of the DNN1 and the DNN2 may be appropriately set according to the task.

Furthermore, in the present embodiment, the plurality of edge devices <NUM> or the plurality of server devices <NUM> may be provided, and the plurality of the edge devices <NUM> and the plurality of server devices <NUM> may be provided.

[System Configuration and Others] All or any part of each processing function performed in each device can be realized by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware by wired logic.

Further, among pieces of processing described in the present embodiment, all or some of pieces of processing described as being performed automatically can be performed manually, or all or some of pieces of processing described as being performed manually can be performed automatically by a known method.

[Program] <FIG> is a diagram illustrating an example of a computer on which the edge device <NUM> and the server device <NUM> are implemented by executing a program. A computer <NUM> includes, for example, a memory <NUM> and a CPU <NUM>. In addition, the accelerators described above may be provided to assist computation. Further, the computer <NUM> also includes a hard disk drive interface <NUM>, a disk drive interface <NUM>, a serial port interface <NUM>, a video adapter <NUM>, and a network interface <NUM>. These units are connected to each other by a bus <NUM>.

The memory <NUM> includes a read only memory (ROM) <NUM> and a RAM <NUM>. The ROM <NUM> stores, for example, a boot program such as a basic input output system (BIOS). The hard disk drive interface <NUM> is connected to a hard disk drive <NUM>. The disk drive interface <NUM> is connected to a disk drive <NUM>. For example, a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive <NUM>. The serial port interface <NUM> is connected to, for example, a mouse <NUM> and a keyboard <NUM>. The video adapter <NUM> is connected to, for example, a display <NUM>.

The hard disk drive <NUM> stores, for example, an operating system (OS) <NUM>, an application program <NUM>, a program module <NUM>, and program data <NUM>. That is, a program that defines each processing of the edge device <NUM> and the server device <NUM> is implemented as the program module <NUM> in which a code executable by the computer is described. The program module <NUM> is stored in, for example, the hard disk drive <NUM>. For example, the program module <NUM> for executing processing similar to functional configurations of the edge device <NUM> and the server device <NUM> is stored in the hard disk drive <NUM>. Note that the hard disk drive <NUM> may be replaced with a solid state drive (SSD).

Furthermore, setting data used in the processing of the above-described embodiment is stored, for example, in the memory <NUM> or the hard disk drive <NUM> as the program data <NUM>. Then, the CPU <NUM> reads the program module <NUM> and the program data <NUM> stored in the memory <NUM> or the hard disk drive <NUM> to the RAM <NUM>, and executes them as necessary.

Claim 1:
A processing system (<NUM>) including an edge device (<NUM>) and a server device (<NUM>), wherein
the edge device (<NUM>) includes:
a first inference unit (<NUM>) configured to extract a feature amount of processing target data by using a first neural network model and execute inference processing on the processing target data on a basis of the extracted feature amount; and
a determination unit (<NUM>) configured to output an inference result by the first inference unit (<NUM>) in a case where reliability of the inference result by the first inference unit (<NUM>) exceeds a threshold, and output the feature amount of the processing target data to the server device (<NUM>) in a case where the reliability is equal to or less than the threshold, and
the server device (<NUM>) includes:
a second inference unit (<NUM>) configured to execute inference processing on the processing target data on the basis of the feature amount of the processing target data output from the edge device (<NUM>) by using a second neural network model having higher inference accuracy than the first neural network model, wherein
the first neural network model includes
a first extraction unit (Bf2) configured to extract the feature amount of the processing target data, and
a first execution unit (Bd1) configured to execute first inference processing on the basis of the feature amount of the processing target data, and
the second neural network model includes
a second execution unit (Bd2) configured to execute second inference processing on the basis of the feature amount of the processing target data, and
the second execution unit (Bd2) executes the second inference processing on the basis of the feature amount of the processing target data extracted by the first extraction unit (Bf2), and
wherein the first neural network model is a neural network model configured by deleting a feature extraction layer from a neural network model before being arranged in the edge device, and arranging as the first extraction unit in a preceding stage of the first execution unit of the edge device, a feature extraction layer of a neural network model trained before being arranged in the server device, and
the second neural network model is a neural network model configured by deleting the feature extraction layer from a neural network model before being arranged in the server device.