Patent ID: 12254348

DESCRIPTION OF EMBODIMENTS

(Underlying Knowledge Forming Basis of the Present Disclosure)

Circumstances leading to the present disclosure will be described before the description of exemplary embodiments of the present disclosure.

The present disclosure relates to an apparatus (hereinafter referred to as an inference apparatus) carrying an inference model that is generated using machine learning. More specifically, the present disclosure relates to an NN-type inference apparatus, enabling the maintenance of the service quality by changing a computation order, and relates to training processing therefor.

In recent years, the performance of an object detection apparatus, a classification apparatus, and the like among inference apparatuses has been dramatically improved by the adoption of deep learning technology, and numerous research or commercialization efforts are underway. In a smartphone or an autonomous vehicle (robot car) driven by a machine instead of a diver, an inference apparatus for an image transferred from a camera apparatus, an infrared array sensor, or the like is one of element functions. In the case of the autonomous vehicle, an object means, for example, a pedestrian, a passenger car, a road sign, a building, a road area, or the like.

Further, the field of edge artificial intelligence (AI), in which an NN inference model is installed in an Internet of Things (IoT) device and the device operates in accordance with the determination of the IoT device itself rather than always causing the cloud to perform determination, has recently begun to come into proliferation. For example, in the fields of various industrial products including the IoT device, industrial products, which are mounted with NN-type inference apparatuses and perform services on the basis of the inference results of the devices, have been put on the market. The IoT device is generally required to be inexpensive and to operate with low power, which limits its calculation capability.

The IoT device is required to save power. In a case where a service using a plurality of NN inference models is performed in such an IoT device, it is difficult to complete the service within a required processing time unless computation for inferring the plurality of NN inference models in parallel is performed. As described above, for example, in the IoT device, it is assumed that computation for inference is performed using a plurality of NN inference models provided in parallel. Note that the required processing time is also referred to as an allowable time.

Further, an NN processing dedicated circuit called a neural network processing unit (NPU), which is mounted with a digital signal processor (DSP), an application-specific integrated circuit (ASIC), and the like, has begun to be installed in the IoT device separately from a central processing unit (CPU). However, a service required for the IoT device has come to use inference processing of a plurality of NNs as described above, and the amount of processing for the service tends to increase.

The IoT device is required to save power in addition to completing the service within the required processing time. Thus, the IoT device is prone to a problem of being unable to complete the inference processing within the required processing time, depending on the input image.

Note that the service using a plurality of NN inference models includes, for example, processing of determining that a target person present in a scene image has “something” in the right hand by inference processing of a skeleton detecting NN inference model and determining that “something” is a smartphone by inference processing of an object detecting NN inference model.

PTL 1 mentioned above discloses a method assuming a service using inference processing of a single NN inference model and does not disclose a method related to the ensuring of a processing time in a service using inference processing of a plurality of NN inference models. Further, no consideration has hitherto been given to a design and a technical solution related to processing of estimating a processing time necessary for inference processing of a plurality of inference models at an initial stage and performing inference processing in an optimum computation order within a required processing time, which is not disclosed in PTL 1, either.

Therefore, an intensive study has been conducted on an information processing apparatus and the like capable of performing inference processing within a required processing time in a service using inference processing of a plurality of NN inference models, and an information processing method and the like described below have been devised.

An information processing apparatus according to one aspect of the present disclosure includes: an obtainer that obtains sensing data; an inference processing unit that inputs the sensing data into an inference model to obtain a result of inference and information on a processing time for a plurality of subsequent tasks to processing performed by the inference model; a determiner that determines a task schedule for a task processing unit that processes the plurality of subsequent tasks to process the plurality of subsequent tasks on the basis of the information on the processing time for the plurality of subsequent tasks; and a controller that inputs the result of the inference into the task processing unit to process the plurality of subsequent tasks according to the task schedule determined.

With this, the information processing apparatus can determine a task schedule for a task processing unit (for example, a plurality of NN inference models) that processes a plurality of subsequent tasks on the basis of information on a processing time for the plurality of subsequent tasks with respect to a sensing data. By determining the task schedule so that the processing time of the task processing unit is within a required processing time on the basis of the information on the processing time for the plurality of subsequent tasks, the information processing apparatus can perform inference processing using the plurality of inference models even in a limited calculation environment.

Further, for example, the inference model may include a first inference model and a second inference model, the sensing data may be input into the first inference model to obtain the result of the inference, and the result of the inference obtained or intermediate data of the inference may be input into the second inference model to obtain the information on the processing time for the plurality of subsequent tasks.

With this, it is possible to make the amount of calculation of the second inference model less than that in a case where the sensing data is input into the second inference model. Hence it is possible to shorten the processing time in the second inference model. When the result of the inference is input, the input of the second inference model is the same as the input of the subsequent task processing unit, so that it is expected that the precision or accuracy of the information on the processing time of the subsequent task processing unit is improved. When the intermediate data of the inference is input, the processing of the second inference model can be started before the end of the inference of the first inference model, thereby further shortening the processing time of the second inference model.

Further, for example, the inference model may include a first inference model and a second inference model, the sensing data may be input into the first inference model to obtain the result of the inference, and the result of the inference obtained or intermediate data of the inference may be input into the second inference model to obtain the information on the processing time for the plurality of subsequent tasks.

With this, it is possible to start the processing of the second inference model by using the sensing data without waiting for the result of the inference to be output from the first inference model. In other words, it is possible to accelerate the start timing of the processing of the second inference model.

Further, for example, the information on the processing time for the plurality of subsequent tasks may include information on a delay relative to a predetermined time that is determined with respect to the processing time for the plurality of subsequent tasks.

With this, it is possible to determine the task schedule for the task processing unit on the basis of the information on the delay. The information processing apparatus determines the task schedule so that the processing time of the task processing unit is within the required processing time on the basis of the information on the delay of the plurality of subsequent tasks, thus increasing the certainty that in the service using the inference processing of the plurality of inference models, the inference processing can be performed within the required processing time.

Further, for example, a first schedule may be determined when information indicating that the delay is less than a predetermined value is included in the information on the delay, and a second schedule, in which the processing time for the plurality of subsequent tasks is shorter than in the first schedule, may be determined when information indicating that the delay is greater than or equal to the predetermined value is included in the information on the delay.

With this, when the delay is greater than or equal to the predetermined value, the second schedule having a shorter processing time is determined, thus increasing the certainty that the inference processing can be performed within the required processing time.

Further, for example, the first schedule may be determined using a first rule, and the second schedule may be determined using a second rule with which the processing time for the plurality of subsequent tasks in the second schedule is shorter than in the first schedule.

With this, the schedule is determined based on the first and second rules, whereby it is possible to reduce the amount of calculation and the calculation time as compared to a case where a schedule is searched for dynamically.

Further, for example, the inference model may execute inference processing that is preprocessing common to the plurality of subsequent tasks.

With this, the use of the inference model for the preprocessing common to each task can compress the total calculation amount and memory usage amount of the inference processing necessary for each task processing.

Further, for example, the inference model and the task processing unit may be neural network models, the result of the inference may be a feature value of the sensing data, and the task schedule may include an order in which the task processing unit performs memory loading and the processing.

With this, it is possible to apply a high-performance NN inference model in a limited calculation environment such as an IoT device. In addition, the processing time for the subsequent task is inferred based on the feature value of the sensing data, thus facilitating the inference of the processing time corresponding to the difficulty level of the task processing with respect to the sensing data.

An information processing method according to one aspect of the present disclosure is a method executed by a computer, the method including: obtaining sensing data; inputting the sensing data into an inference model to obtain a result of inference and information on a processing time for a plurality of subsequent tasks to processing performed by the inference model; inputting the result of the inference into a task processing unit that processes the plurality of subsequent tasks to process the plurality of subsequent tasks; measuring an inference time including a time from the inputting of the result of the inference into the task processing unit until an end of the processing of the plurality of subsequent tasks; and training the inference model by machine learning using the sensing data as input data, the information on the processing time for the plurality of subsequent tasks as output data, and the inference time measured as reference data. A recording medium according to one aspect of the present disclosure is a non-transitory computer-readable recording medium having recorded thereon a program for causing a computer to execute the information processing method described above.

With this, it is possible to generate an inference model that achieves the processing executed by the information processing apparatus. The inference processing is performed using the generated inference model, whereby it is possible to determine the task schedule so that the processing time of the task processing unit is within the required processing time on the basis of the information on the processing time for the plurality of subsequent tasks.

Moreover, these generic or specific aspects may be achieved using a system, an apparatus, a method, an integrated circuit, a computer program, or a non-temporary storage medium such as a computer-readable compact disc read-only memory (CD-ROM), or may be implemented using any combination of systems, apparatuses, methods, integrated circuits, computer programs, and storage media.

Hereinafter, specific examples of an information processing apparatus and the like according to one aspect of the present disclosure will be described with reference to the drawings. Each of the exemplary embodiments described here shows a specific example of the present disclosure. Therefore, the numerical values, components, steps, the order of steps, and the like shown in the following exemplary embodiments are examples and are not intended to limit the present disclosure. Among the components in the following exemplary embodiments, components not recited in independent claims are described as optional components. In all the exemplary embodiments, the respective contents can be combined.

Each of the drawings is a schematic diagram and is not necessarily a strict illustration. Thus, for example, the scales or the like are not necessarily coincident with each other in the drawings. In addition, in the drawings, substantially the same components are denoted by the same reference numerals, and duplicated description is omitted or simplified.

In the present specification, a term, as well as a numerical value and a numerical range, indicating a relationship between elements, such as “same” or “coincident”, is not an expression expressing only a strict meaning but is an expression meaning to include a substantially equivalent range, for example, a difference of about a few percent.

Embodiment

Hereinafter, an information processing system and the like according to the present embodiment will be described below with reference toFIGS.1to13.

[1. Configuration of Information Processing System]

First, a configuration of information processing system10according to the present embodiment will be described with reference toFIGS.1to4.FIG.1is a block diagram showing a functional configuration of information processing system10according to the present embodiment.FIG.2is a block diagram showing a functional configuration of inference processing unit110according to the present embodiment.

As shown inFIG.1, information processing system10includes device100, camera200, and training apparatus300. Device100performs inference processing on an input image from camera200and performs a service corresponding to the result of the inference. Training apparatus300generates NN data400for device100to perform the inference processing. Each component will be described below.

Device100is, for example, an IoT device and has a function of providing a plurality of services on the basis of an input image obtained from camera200. Device100includes inference processing unit110and task executor140. The input image obtained from camera200is an example of sensing data.

Inference processing unit110performs inference processing on the input image input from camera200by using NN data400generated by training apparatus300. Inference processing unit110performs the inference processing by using a plurality of NNs provided in parallel. In other words, NN data400includes a plurality of NNs provided in parallel. Note that NN data400stores, for example, the layer configurations, weights, and biases of the NNs.

As shown inFIG.2, NN data400includes common NN121and specialized NN group124. Common NN121is an NN for preprocessing in NN data400, and specialized NN group124is an NN group for subsequent processing in NN data400.

The time required for the inference processing in device100may vary depending on the input image. Thus, depending on the input image, the inference processing may take time, and the service may not be performed within the processing time required for device100. Therefore, inference processing unit110obtains an inference result related to the time taken for subsequent inference processing on the input image by using the NN for preprocessing and changes the order of the subsequent inference processing (an example of a task schedule) or the like on the basis of the obtained inference result. The configurations of inference processing unit110and the like will be described below.

Inference processing unit110includes NN inference unit120, NN inference computation management unit130, and obtainer150. Note that inference processing unit110is an inference processing apparatus capable of changing a computation order. It can also be said that inference processing unit110is a computation-order variable inference apparatus. Inference processing unit110is an example of an information processing apparatus.

NN inference unit120performs inference processing on the input image input from camera200by using a plurality of NNs based on NN data400generated by training apparatus300.

NN inference unit120performs inference processing by using common NN121and specialized NN group124that includes a plurality of NNs provided in parallel. First, NN inference unit120performs forward propagation computation processing by using common NN121. Common NN121is, for example, a forward propagation type NN.

Common NN121is an NN that outputs an inference processing result and delay flag information common to the plurality of NNs of specialized NN group124. Common NN121executes inference processing that is preprocessing common to each task. It can also be said that common NN121is a model executed as preprocessing common to each task. Common NN121includes feature classifier122and delay flag classifier123. Feature classifier122and delay flag classifier123are trained NNs, for example. Note that common NN121is an example of an inference processing unit. The inference processing as preprocessing is not limited to the processing using the NN.

By inputting the input image into feature classifier122of common NN121, NN inference unit120obtains an inference processing result that is an output of feature classifier122. The inference processing result (inference result) is, for example, a feature value (image feature value) but is not limited thereto. An example in which the inference processing result is a feature value will be described below.

Feature classifier122is a trained NN (NN for feature classification) trained to receive an input image as an input and output a feature value corresponding to the input image. Feature classifier122performs inference processing common to each of the plurality of NNs of specialized NN group124to be subjected to forward propagation computation processing. The feature value is information common to each of the plurality of NNs of specialized NN group124.

NN inference unit120outputs a feature value that is an output of feature classifier122to each of the plurality of NNs of specialized NN group124. In the present embodiment, NN inference unit120also outputs the feature value to delay flag classifier123. As described above, the feature value as the output of feature classifier122may be used in delay flag classifier123. In the present embodiment, the same feature value is input into delay flag classifier123and each of the plurality of NNs of specialized NN group124.

NN inference unit120inputs the feature value from feature classifier122into delay flag classifier123to obtain delay flag information that is an output of delay flag classifier123. Delay flag classifier123is a trained NN (NN for delay flag classification) trained to receive a feature value as an input and output delay flag information corresponding to the feature value. The delay flag information is information indicating the time (inference time) including the time from the inputting of the feature value (inference result) into specialized NN group124(a task processing unit or a part thereof) until the end of the processing of the plurality of subsequent tasks. For example, the delay flag information is information indicating an inference result as to whether or not the inference processing time of entire NN inference unit120exceeds a threshold. It can also be said that the delay flag information is information on a delay relative to a predetermined time (for example, a threshold). The predetermined time is the time determined with respect to the processing time for the plurality of subsequent tasks. The threshold may be set, for example, based on the processing time required for device100or the service content of device100. The threshold is set in advance. Note that the delay flag information may be, for example, information indicating an inference result as to whether or not the processing time of entire device100exceeds the threshold. The delay flag information may be, for example, information indicating an inference result as to whether or not the total processing time of NN inference unit120and task executor140exceeds the threshold.

The delay flag information may be, for example, information indicating “OFF” when the inference processing time is less than or equal to the threshold, or information indicating “ON” when the inference processing time exceeds the threshold. Further, a plurality of “ON” may be set. The delay flag information may be, for example, information indicating “ON1” when the inference processing time exceeds a first threshold and is less than or equal to a second threshold that is greater than the first threshold, and information indicating “ON2” when the inference processing time exceeds the second threshold. The first threshold is, for example, 10 msec, and the second threshold is, for example, 15 msec, but the present disclosure is not limited thereto. The delay flag information is an example of information on the processing time for the plurality of subsequent tasks. The processing time for the plurality of subsequent tasks may be, for example, the processing time of specialized NN group124, or the total processing time of specialized NN group124and task executor140. In this case, specialized NN group124and task executor140are examples of a task processing unit.

In the present embodiment, an example will be given of an example in which three levels of “OFF”, “ON1” and “ON2” are set as the delay flag information. Note that two levels or four or more levels may be set for the delay flag information.

The delay flag information is not limited to the information indicating the delay level, such as “OFF”, “ON1”, or “ON2”, but may be the processing time (computation time) itself, for example.

Note that delay flag classifier123is desired to output delay flag information as soon as possible. Delay flag classifier123may thus be an NN having a smaller amount of calculation than feature classifier122and specialized NN group124.

Common NN121configured as described above is an example of an inference model that outputs a feature value and delay flag information when receiving input of an input image from camera200. Feature classifier122is an example of a first inference model, and delay flag classifier123is an example of a second inference model. That is, the inference model includes the first inference model and the second inference model.

On the basis of the delay flag information from delay flag classifier123, NN inference computation management unit130determines a forward propagation computation method (for example, a computation order) for processing the plurality of tasks of specialized NN group124subsequent to common NN121and notifies specialized NN group124of the determined method. Specifically, NN inference computation management unit130notifies NN inference unit120, which controls the processing of specialized NN group124, of the determined forward propagation computation method. Thereby, NN inference unit120can input the feature value from feature classifier122into specialized NN group124to process the plurality of subsequent NNs by the determined forward propagation computation method. NN inference computation management unit130is an example of a determiner for determining a forward propagation computation method and a controller for causing a task at a later stage.

In the present embodiment, NN inference computation management unit130determines a method for the forward propagation computation subsequent to common NN121on the basis of the delay flag information and table500shown inFIG.3.FIG.3is a diagram showing an example of the configuration of table500including the delay flag information and forward propagation computation methods for specialized NN group124associated with the delay flag information according to the present embodiment. A parallel computation item inFIG.3shows methods for the forward propagation computation subsequent to common NN121, for example, forward propagation computation methods for specialized NN group124.

As shown inFIG.3, the delay flag information “OFF” is associated with a parallel computation item “optimum parallel computation processing” (see first line501inFIG.3), the delay flag information “ON1” is associated with a parallel computation item “delay handling parallel computation processing” (see second line502inFIG.3), and the delay flag information “ON2” is associated with a parallel computation item “delay handling parallel computation processing+CPU overclocking processing” (see third line503inFIG.3). Note that “ON” mentioned here means that the computation time of NN inference unit120exceeds the allowable time, and the computation order needs to be changed from the computation order indicated by the “optimum parallel computation processing”. The operation order is an example of a task schedule.

The “optimum parallel computation processing” indicates that computation processing is performed in accordance with a parallel computation method (for example, a parallel computation order) determined based on the number of memory accesses, processing time (computation time), power consumption, and the like. For example, when a delay is less than a predetermined value (for example, a delay is less than a threshold), a forward propagation computation order (an example of a first schedule) is determined using the “optimum parallel computation processing”. Note that the “optimum parallel computation processing” is an example of a first rule. That the delay is less than the predetermined value includes that there is no delay or that the delay length or the delay degree is less than the predetermined value.

The “delay handling parallel computation processing” indicates that computation processing is performed in accordance with a parallel computation method determined so that the processing time becomes shorter than that for the “optimum parallel computation processing”. For example, when the delay is greater than or equal to the predetermined value (for example, the delay is greater than or equal to the threshold), a forward propagation computation order (an example of a second schedule) is determined using the “delay handling parallel computation processing”. The parallel computation method is, for example, a method in which the computation order is rearranged so as to preferentially start processing (for example, non-maximum suppression (NMS) processing), the processing time of which changes in accordance with the number of candidate detection areas in the input image, but the present disclosure is not limited thereto. Note that the “delay handling parallel computation processing” is an example of a second rule with which the processing time for the plurality of subsequent tasks is shorter than that in the first rule. That the delay is greater than or equal to the predetermined value includes that there is a delay or that the delay length or the delay degree is greater than or equal to the predetermined value.

Note that NMS is an algorithm that deletes a certain candidate detection area when the degree of overlap (a value of intersection over union (IoU)) between the certain candidate detection area and a candidate detection area having a higher score than the certain candidate detection area exceeds a threshold set by training processing.

The “delay handling parallel computation processing+CPU overclocking processing” indicates that in addition to the “delay handling parallel computation processing”, a setting of a drive frequency of a CPU (for example, CPU602shown inFIG.4to be described later) is increased. As described above, the delay flag information may include information related to the setting of the drive frequency of the CPU in addition to the change of the computation order of the parallel computation. That is, coping with a delay is not limited to changing the parallel computation but may include changing the setting (for example, the setting of the drive frequency) of each processing unit included in device100together with or instead of changing the parallel computation.

Note that the computation order indicated by the “optimum parallel computation processing” and the computation order indicated by the “delay handling parallel computation processing” are set in advance and stored in memory605or the like.

As described above, a plurality of types of delay flag information may exist, such as the delay flags “ON1” and “ON2”. For example, a plurality of types of delay flag information may exist depending on the result of the processing time measurement performed by delay flag information measurer302to be described later.

Specialized NN group124includes a plurality of NNs, each of which performs different output on the basis of the feature value from feature classifier122. In the present embodiment, specialized NN group124includes first task NN125and second task NN126. Hereinafter, a description will be given of an example in which first task NN125is an object detecting NN and second task NN126is a skeleton detecting NN, but the present disclosure is not limited thereto. Further, a description will be given of an example in which first task NN125includes NMS processing and second task NN126does not include NMS processing, but the present disclosure is not limited thereto. Note that specialized NN group124(NN inference unit120that performs processing by using specialized NN group124) is an example of a task processing unit. When the task processing unit includes specialized NN group124, each of the inference model and the task processing unit is an NN model.

First task NN125is a trained NN trained to receive a feature value from feature classifier122as an input and output an inference result of object detection corresponding to the feature value. The inference result as an output of first task NN125is output to task executor140corresponding to first task NN125.

Second task NN126is a trained NN trained to receive a feature value from feature classifier122as an input and output an inference result of skeleton detection corresponding to the feature value. The inference result as an output of second task NN126is output to task executor140corresponding to second task NN126.

Note that the object detection and the skeleton detection are examples of the plurality of subsequent tasks.

When delay flag classifier123outputs delay flag information including “ON1” or “ON2”, the computation orders of first task NN125and second task NN126are changed from when the delay flag information is “OFF”. In the present embodiment, at least the computation order in first task NN125including the NMS processing is changed from when the delay flag information is “OFF”.

NN inference computation management unit130instructs NN inference unit120to perform forward propagation computation on specialized NN group124in accordance with a parallel computation item (parallel computation method) determined based on the delay flag information and table500, for example. Thus, NN inference computation management unit130can change the computation order in accordance with the inference result (delay flag information) related to the time of inference processing of the input image, thus preventing the processing time from becoming long.

Obtainer150obtains an input image from camera200. Obtainer150outputs the obtained input image to NN inference unit120.

Subsequently, task executor140executes a preset task on the basis of the inference result of NN inference unit120. Task executor140may be, for example, a display for displaying the inference result, a light emitter for emitting light corresponding to the inference result, or a transmitter for transmitting the inference result to an external device.

FIG.4is a schematic diagram showing a configuration of system-on-chip (SoC)601of device100according to the present embodiment. In the present embodiment, an NPU is mounted.

As shown inFIG.4, SoC601includes CPU602, first NPU603, second NPU604, and memory605. Each function of NN inference unit120is achieved by installing a trained NN (NN data400) on SoC601.

CPU602is a processing unit that executes various types of processing in device100. CPU602executes the NMS processing, the processing of task executor140, and the like.

First NPU603and second NPU604are dedicated circuits that perform NN processing. First NPU603and second NPU604execute NN processing in device100. First NPU603and second NPU604execute, for example, processing using common NN121and processing using specialized NN group124. In the present embodiment, first NPU603and second NPU604execute skeleton detection processing, object detection processing, and delay label determination processing.

Memory605stores a program executed by each of CPU602, first NPU603, and second NPU604. Memory605stores NN data400.

As described above, inference processing unit110is configured to obtain an input image, output a feature value and delay label information when the input image is input into common NN121, determine a forward propagation computation method (optimum parallel computation processing, delay coping parallel computation processing, etc.) for processing specialized NN group124on the basis of the delay label information, and input the feature value into specialized NN group124to process specialized NN group124by the determined forward propagation computation method. For example, inference processing unit110includes: obtainer150that obtains an input image; common NN121that receives the input of the input image and outputs a feature value and delay label information; NN inference computation management unit130(an example of a determiner) that determines a forward propagation computation method (optimum parallel computation processing, delay coping computation processing, etc.) for processing specialized NN group124on the basis of the delay label information; and NN inference computation management unit130(an example of a controller) that inputs a feature value into specialized NN group124to process the determined forward propagation computation processing.

Next, training apparatus300will be described. Training apparatus300generates NN data400to be used for inference processing by NN inference unit120of device100. Training apparatus300includes multi-task trainer301, delay flag information measurer302, delay flag correct answer label generator303, delay flag trainer304, training label database305(training label DB305), and scene image database306(scene image DB306).

multi-task trainer301trains feature classifier122and specialized NN group124of NN data400. In the present embodiment, multi-task trainer301performs training for object detection and skeleton detection.

multi-task trainer301performs training processing in which multi-task learning is applied to the training of NNs. Although the details will be described later, multi-task trainer301receives a scene image obtained from scene image DB306as an input, uses a correct answer label in skeleton detection corresponding to the scene image and a correct answer label in object detection corresponding to the scene image obtained from training label DB305as reference data (teacher data), and generates NNs (common NN121and specialized NN group124) on the basis of backpropagation (BP) or the like.

Note that the multi-task learning is a method of machine learning that solves a plurality of tasks with a single model. This is a method aiming to improve the accuracy in task prediction by training a plurality of related tasks simultaneously to obtain a “common factor” among the tasks. In the image recognition field, a plurality of tasks, such as object classification, object detection, and object area (segmentation) recognition, may be trained simultaneously. In the present embodiment, the plurality of tasks are skeleton detection and object detection.

Research has been conducted to apply multi-task learning to training processing of NN data, and multi-task trainer301trains an NN commonly used for each task (for example, common NN121) and an NN specialized for each task (for example, specialized NN group124), for example. There is also an advantage that the use of the commonly used NN can compress the total calculation amount and memory usage amount of the inference processing necessary for each task processing.

Delay flag information measurer302and delay flag correct answer label generator303perform processing for generating training data that is used when delay flag classifier123is trained by delay flag trainer304.

Delay flag information measurer302measures the processing time for the forward propagation computation of common NN121and specialized NN group124generated by multi-task trainer301. In the present embodiment, delay flag information measurer302measures at least the processing time of first task NN125. This is because first task NN125includes the NMS processing, and the processing time differs depending on the number of candidate detection areas in the input image. Second task NN126does not include processing for which the processing time differs depending on the number of candidate detection areas, such as the NMS processing, that is, the processing time does not change greatly in accordance with the input image, so that the processing time is not measured in the present embodiment.

Delay flag correct answer label generator303generates a delay flag correct answer label to be used for the training of common NN121in delay flag trainer304by using information based on the measurement result of the processing time.

Delay flag trainer304performs processing of training common NN121, out of common NN121and specialized NN group124generated by multi-task trainer301, by using the delay flag correct answer label generated by delay flag correct answer label generator303. Delay flag trainer304updates weight data and bias data in delay flag classifier123of common NN121on the basis of backpropagation, using the scene image as input data and a delay flag correct answer label as reference data (teacher data), for example. That is, delay flag trainer304trains delay flag classifier123of common NN121by using the scene image and the delay flag correct answer label. Note that delay flag trainer304may train feature classifier122together with delay flag classifier123by using, for example, the delay flag correct answer label.

Training label DB305stores a training label for generating NN data400. Training label DB305stores, for example, a training label for skeleton detection and a training label for object detection with respect to one scene image.

Scene image DB306stores a scene image for generating NN data400.

Training label DB305and scene image DB306are achieved by, for example, a semiconductor memory or the like but are not limited thereto.

As described above, training apparatus300is configured to obtain a scene image, input the scene image into common NN121to obtain a feature value, input the feature value into specialized NN group124to process specialized NN group124, measure the time from the inputting of the scene image into common NN121until the end of the processing of specialized NN group124, and train common NN121by machine learning using the scene image as input data, the information on the processing time of specialized NN group124as output data, and the measured time as reference data. For example, training apparatus300includes: delay flag information measurer302that obtains a scene image, inputs the scene image into common NN121to obtain a feature value, inputs the feature value into specialized NN group124to process specialized NN group124, and measures the time from the inputting of the scene image into common NN121until the end of the processing of specialized NN group124; and delay flag trainer304that trains common NN121by machine learning using the scene image as input data, the information on the processing time of specialized NN group124as output data, and the measured time as reference data.

[2. Operation of Information Processing System]

Subsequently, the operation of information processing system10will be described with reference toFIGS.5to13.

[2-1. Operation of Device]

First, the processing in device100will be described with reference toFIGS.5to6C.FIG.5is a flowchart showing the operation of device100according to the present embodiment. For convenience,FIG.5describes an example in which the delay flag information included in table500is of two types of “ON (ON1)” and “OFF”. In other words, it is assumed that delay flag classifier123has been trained to output either “ON” or “OFF”.

As shown inFIG.5, obtainer150of device100obtains an input image from camera200(S101). Obtainer150outputs the input image obtained in step S101to NN inference unit120.

Next, when receiving the input of the input image from obtainer150, NN inference unit120first executes forward propagation computation processing by using common NN121(S102). Specifically, NN inference unit120inputs an input image into feature classifier122of common NN121to obtain a feature value that is an output of feature classifier122. Then, NN inference unit120inputs the obtained feature value into delay flag classifier123to obtain delay flag information that is an output of delay flag classifier123. The delay flag information is a result inferred from the feature value by delay flag classifier123, and in the present embodiment, the delay flag information indicates an inference result as to whether or not the inference processing time of entire NN inference unit120has exceeded a threshold.

NN inference unit120outputs the obtained delay flag information to NN inference computation management unit130. Note that NN inference unit120may further output the delay flag information to a functional unit provided in device100and external to inference processing unit110. The functional unit may be, for example, a functional unit having a graphic function.

Next, NN inference computation management unit130obtains delay flag information (S103). NN inference computation management unit130determines whether or not the obtained delay flag information is “ON” (S104). In a case where the delay flag information is “OFF”, that is, in a case where the delay flag information is an inference result indicating that the service can be completed within the processing time required for device100even when the inference processing is performed in accordance with the “optimum parallel computation processing” (No in S104), NN inference computation management unit130determines the “optimum parallel computation processing to be a forward propagation computation method for specialized NN group124” on the basis of table500shown inFIG.3(S105). That is, when the delay flag information is “OFF”, NN inference computation management unit130determines to execute the processing of specialized NN group124in the order based on the “optimum parallel computation processing”. The order based on the “optimum parallel computation processing” is an example of a first schedule. NN inference computation management unit130outputs the determined forward propagation computation method for specialized NN group124to NN inference unit120.

When NN inference unit120obtains from NN inference computation management unit130that the forward propagation computation method for specialized NN group124is the “optimum parallel computation processing”, NN inference unit120executes parallel forward propagation computation processing in the order indicated by the “optimum parallel computation processing”. For example, NN inference unit120performs parallel forward propagation computation on specialized NN group124in the most efficient order (S106). NN inference unit120inputs the feature value that is an output of feature classifier122to each NN of specialized NN group124, performs inference processing according to the schedule based on the “optimum parallel computation processing”, and obtains an inference result (skeleton detection result, object detection result, etc.) that is an output of each NN.

FIG.6Ais a schematic diagram showing an example of the processing time of common NN121and specialized NN group124and units in charge of computation thereof according to the present embodiment. Specifically, first information700ainFIG.6Ashows an example of the processing time and the units in charge of computation in a case where the processing ends within the allowable time even when the processing is performed in the computation order based on the “optimum parallel computation processing”. Note thatFIGS.6A to6Cdescribe examples in which the NMS processing has been outsourced to CPU602.

Area701indicates the occupancy rate of each unit (CPU602, first NPU603, and second NPU604) in common NN121. In area701, first NPU603(NPU1inFIGS.6A to6C) and second NPU604(NPU2inFIGS.6A-6C) perform processing by using common NN121. Area702indicates the occupancy rate of each unit in first task NN125. In area702, at least one of first NPU603and second NPU604performs object detection processing by using first task NN125.

Area703indicates the occupancy rate of each unit in second task NN126. In area703, at least one of first NPU603and second NPU604performs skeleton detection processing by using second task NN126. Areas704aand704bindicate the occupancy rate of each unit in the NMS processing of first task NN125. In areas704aand704b, CPU602performs the NMS processing.

As shown inFIG.6A, when “No” in step S104, NN inference computation management unit130instructs NN inference unit120to perform forward propagation computation on specialized NN group124by the “optimum parallel computation processing” indicating that the computation processing is performed by the parallel computation method in which the number of memory accesses, the computation time, and the like are balanced. In this case, even when the processing is performed in the computation order based on the “optimum parallel computation processing”, the processing can be ended within the allowable time.

Referring again toFIG.5, in a case where the delay flag information is “ON”, that is, in a case where the delay flag information is an inference result indicating that the service cannot be completed within the processing time required for device100when the inference processing is performed in accordance with the “optimum parallel computation processing” (Yes in S104), NN inference computation management unit130determines the “delay coping parallel computation processing” to be the forward propagation computation method for specialized NN group124on the basis of table500shown inFIG.3(S107). That is, when the delay flag information is “ON”, NN inference computation management unit130determines to execute the processing of specialized NN group124in the order based on the “delay coping parallel computation processing”. The order based on the “delay coping parallel computation processing” is an example of a second schedule. NN inference computation management unit130outputs the determined forward propagation computation method for specialized NN group124to NN inference unit120.

When NN inference unit120obtains from NN inference computation management unit130that the forward propagation computation method for specialized NN group124is the “delay coping parallel computation processing”, NN inference unit120executes parallel forward propagation computation processing in the order indicated by the “delay coping parallel computation processing”. For example, NN inference unit120performs parallel forward propagation computation on specialized NN group124in the set order (S108). NN inference unit120inputs the feature value that is an output of feature classifier122into each NN of specialized NN group124, performs inference processing according to the schedule based on the “delay coping parallel computation processing”, and obtains an inference result (skeleton detection result, object detection result, etc.) that is an output of each NN.

FIG.6Bis a schematic diagram showing an example of the processing time of common NN121and specialized NN group124and the units in charge of computation thereof according to the present embodiment. Specifically, second information700binFIG.6Bshows an example of the processing time and the units in charge of computation in a case where the processing does not end within the allowable time when the processing is performed in the computation order based on the “optimum parallel computation processing”.FIG.6Bshows a case where the allowable time is exceeded when the optimum parallel computation processing is performed, that is, a case where the determination is Yes in step S104.

As shown inFIG.6B, for example, the time required for the NMS processing (area704b) is longer than the time (area704a) inFIG.6A, and hence the inference time of inference processing unit110has exceeded the allowable time. Depending on the input image, the time required for the NMS processing may be long, as shown inFIG.6B.

Therefore, as shown in step S107inFIG.5, NN inference computation management unit130determines to perform the inference processing in accordance with the “delay coping parallel computation processing” instead of the “optimum parallel computation processing” because the allowable time may be exceeded when the inference processing is performed in accordance with the “optimum parallel computation processing”. In the present embodiment, the “delay coping parallel computation processing” indicates that the computation order is rearranged so that the NMS processing (processing performed in area704b) can be started with priority. In the following, that the allowable time is exceeded is also referred to as that a delay occurs.

FIG.6Cis a schematic diagram showing an example of the processing time of common NN121and specialized NN group124and the units in charge of computation thereof after the recombination of the computation order according to the present embodiment. Third information700cinFIG.6Cshows an example of the processing time and the units in charge of computation thereof when the processing is performed in the computation order based on the “delay coping parallel computation processing” in a case where the processing does not end within the allowable time in the computation order based on the “optimum parallel computation processing”. Third information700cis obtained by changing the processing order from second information700bshown inFIG.6B.FIG.6Cshows the processing order and the like after the determination is Yes in step S104and the change is made.

As shown inFIG.6C, in the computation order based on the “delay coping parallel computation processing”, the NMS processing is started with priority. In the computation order, the timing of starting the NMS processing is accelerated from the computation order based on the “optimum parallel computation processing” so as to accelerate the timing of starting the NMS processing. Thus, even when the time required for the NMS processing is long, the inference time of inference processing unit110can be prevented from exceeding the allowable time. The computation order (processing order) is an example of a task schedule. The task schedule may also include an order in which the NN models of specialized NN group124perform memory loading.

As shown inFIGS.6B and6C, in the “optimum parallel computation processing” and the “delay coping parallel computation processing”, the processing timing of the NMS processing is changed, but a change in the content of the processing, the deletion of the processing, or the like is not performed. For this reason, the accuracy of the inference result is substantially the same when the inference processing is performed in accordance with the “optimum parallel computation processing” and when the inference processing is performed in accordance with the “delay coping parallel computation processing”.

In the present embodiment, as shown inFIGS.6B and6C, an example is shown where a layer in which the processing time varies greatly depending on the input image is a layer for performing the NMS processing, but a layer using another algorithm in which the processing time varies greatly depending on the input image may be used.

NN inference unit120outputs the inference result, output by the computation in step S106or S108, to task executor140.

Referring again toFIG.5, task executor140operates based on the inference result obtained from NN inference unit120, and the service is provided to a user. That is, task executor140executes various types of task processing (S109).

As described above, for providing the inference service within the allowable time, device100can operate while ensuring a certain service capacity by having inference processing unit110capable of determining the load of the inference processing at an early stage and changing the computation order. When the delay flag information is “OFF”, inference processing unit110can provide the inference service by the optimum parallel computation and can contribute to power saving.

Further, inference processing unit110estimates the processing time required for the NN inference processing in the plurality of subsequent NNs at an initial stage (preprocessing) from the feature value of the input image. Then, in parallel computation, inference processing unit110can perform the NN inference processing in a computation order in which the processing can be completed within the required processing time.

In the above description, an example has been described where the computation processing is executed in the order shown inFIG.6Cwhen the delay flag information is “ON”, but the present disclosure is not limited thereto, and the order of the computation processing may be changed in accordance with the delay flag information. For example, when the delay flag information includes information indicating the degree of delay (for example, the degree of delay: “large”, “medium”, and “small”, an estimated value of the delay time, etc.), a parallel computation item may be set for each degree of delay. That is, the order of computation processing may be set for each degree of delay. Then, in step S107, NN inference computation management unit130may determine the parallel computation item corresponding to the degree of delay included in the delay flag information to be the forward propagation computation method for specialized NN group124.

[2-2. Operation of Training Apparatus]

Subsequently, the operation of training apparatus300will be described with reference toFIGS.7to13. First, the operation of multi-task trainer301will be described with reference toFIGS.7and8.FIG.7is a flowchart showing the operation of multi-task trainer301according to the present embodiment.FIG.8is a diagram schematically showing the operation of multi-task trainer301according to the present embodiment.

As shown inFIG.7, multi-task trainer301obtains a scene image from scene image DB306and obtains a correct answer label for each NN from training label DB305(S201). multi-task trainer301obtains a scene image, a correct answer label for first task NN125(first task NN correct answer label) corresponding to the scene image, and a correct answer label for second task NN126(second task NN correct answer label) corresponding to the scene image. In the present embodiment, multi-task trainer301obtains a correct answer label for the object detecting NN and a correct answer label for the skeleton detecting NN.

Next, multi-task trainer301performs forward propagation computation by using common NN121and specialized NN group124. For example, multi-task trainer301performs forward propagation computation by using feature classifier122of common NN121and specialized NN group124(S202).

As shown inFIG.8, multi-task trainer301obtains a feature value that is an output of common NN121obtained by inputting the scene image into common NN121, and further obtains a first inference result that is an output of first task NN125obtained by inputting the obtained feature value into first task NN125, and a second inference result that is an output of second task NN126obtained by inputting the feature value into second task NN126.

Referring again toFIG.7, next, multi-task trainer301performs backward propagation computation on feature classifier122and specialized NN group124, using the correct answer label for each NN as reference data (teacher data), and updates weights and biases (S203). On the basis of backpropagation, multi-task trainer301regards an output value of a loss function as an error and executes update processing on the weights (weight data) and biases (bias data) in feature classifier122of common NN121and specialized NN group124by backward propagation computation.

As shown inFIG.8, a loss function in first task NN125represents how much is the error of the first inference result of first task NN125with respect to the first task NN correct answer label. A loss function in second task NN126represents how much is the error of the second inference result of second task NN126with respect to the second task NN correct answer label.

multi-task trainer301obtains a first inference result that is an output of first task NN125on the basis of the scene image and obtains an output value of the loss function on the basis of the first inference result and the first task NN correct answer label. Then, multi-task trainer301regards the output value as an error and executes update processing on the weights and biases in feature classifier122of common NN121and first task NN125of specialized NN group124by backward propagation computation.

Next, multi-task trainer301obtains a second inference result that is an output of second task NN126on the basis of the scene image and obtains an output value of the loss function on the basis of the second inference result and the second task NN correct answer label. Then, multi-task trainer301regards the output value as an error and executes update processing on the weights and biases in feature classifier122of common NN121and second task NN126of specialized NN group124by backward propagation computation. As described above, multi-task trainer301alternately executes the update processing, for example.

Referring again toFIG.7, multi-task trainer301determines whether or not the correct answer rate of the NN inference result (or the error rate obtained by comparing the correct answer label with the scene image) satisfies a requirement (S204). The requirement may be, for example, a correct answer rate required for device100. That is, multi-task trainer301may perform the determination in step S204based on whether or not the correct answer rate of the NN inference result is higher than or equal to the correct answer rate required for device100.

When the correct answer rate of the NN inference result satisfies the requirement (Yes in S204), multi-task trainer301ends the processing. When the correct answer rate of the NN inference result does not satisfy the requirement (No in step S204), multi-task trainer301returns to step S201and continues the processing. That is, when the correct answer rate of the NN inference result does not satisfy the requirement, multi-task trainer301performs training by using various scene images and correct answer labels associated with the scene images and continues the training processing until the correct answer rate of the NN inference result satisfies the requirement.

Subsequently, the operations of delay flag information measurer302and delay flag correct answer label generator303will be described with reference toFIGS.9and10.FIG.9is a flowchart showing the operation of delay flag information measurer302and delay flag correct answer label generator303according to the present embodiment.FIG.10is a diagram schematically showing the operations of delay flag information measurer302and delay flag correct answer label generator303according to the present embodiment.

As shown inFIG.9, delay flag information measurer302obtains an arbitrary scene image (S301). Delay flag information measurer302may obtain, for example, a scene image used for multi-task learning by multi-task trainer301.

Next, delay flag information measurer302measures the computation time for the forward propagation computation of common NN121and specialized NN group124trained by multi-task trainer301(S302). In the present embodiment, delay flag information measurer302measures the computation time of first task NN125including the NMS processing.

In step S302, delay flag information measurer302inputs the scene image into common NN121to obtain a feature value and delay flag information, further inputs the feature value into first task NN125to process first task NN125, and measures the time from the inputting of the scene image into common NN121until the end of the processing of specialized NN group124. In step S302, delay flag information measurer302may measure at least the time from the inputting of the feature value into first task NN125until the end of the processing of specialized NN group124.FIG.10shows an example in which delay flag information measurer302measures the time (an example of the inference time) from the inputting of the feature value into first task NN125until the end of the processing of specialized NN group124.

In the measurement of the processing time, the same hardware device as device100may be used. The processing time may be calculated by simulation.

As shown inFIG.10, delay flag information measurer302obtains a feature value obtained by inputting the scene image into common NN121and information on the processing time for the plurality of tasks subsequent to the processing of common NN121(delay flag information that is an output from delay flag classifier123), and measures the processing time of first task NN125when the feature value is input. Measuring the processing time is an example of measuring the delay flag information.

Referring again toFIG.9, delay flag information measurer302determines whether or not the measured processing time has exceeded the threshold (S303). The threshold is set in advance, for example. When the processing time has exceeded the threshold (Yes in S303), delay flag information measurer302stores the corresponding scene image as a delay flag ON into delay flag information DB302a(seeFIG.10) (S304). When the processing time has not exceeded the threshold (No in S303), delay flag information measurer302stores the corresponding scene image into delay flag information DB302aas a delay flag OFF (S305).

When the processing time is being measured in each of first task NN125and second task NN126, it is possible that: (i) a delay occurs only in first task NN125(the allowable time is exceeded); and (ii) a delay occurs only in second task NN126. In this case, the content of the “parallel computation item” (for example, the computation order after recombination) may be different between (I) and (ii). As described above, when the content of the “parallel computation item” is different, a plurality of types of the delay flag “ON” may exist for the respective contents of the “parallel computation item”.

Next, delay flag correct answer label generator303generates a delay flag correct answer label on the basis of the delay flag information of delay flag information DB302a. Delay flag correct answer label generator303labels the scene image with the delay flag “ON” or “OFF” to generate a delay flag correct answer label (S306). It can also be said that delay flag correct answer label generator303generates a delay flag correct answer label by associating the delay flag information with the scene image. Delay flag correct answer label generator303stores the generated delay flag correct answer label into delay flag correct answer label DB303a(seeFIG.10).

Subsequently, the operation of delay flag trainer304will be described with reference toFIGS.11and12.FIG.11is a flowchart showing the operation of delay flag trainer304according to the present embodiment.FIG.12is a diagram schematically showing the operation of delay flag trainer304according to the present embodiment.

As shown inFIG.11, delay flag trainer304obtains a scene image from scene image DB306and obtains a delay flag correct answer label corresponding to the scene image from delay flag correct answer label DB303a(S401).

Next, delay flag trainer304performs forward propagation computation by using common NN121(S402). Delay flag trainer304performs forward propagation computation by using, for example, delay flag classifier123of common NN121.

As shown inFIG.12, delay flag trainer304obtains a feature value that is an output of feature classifier122obtained by inputting a scene image into feature classifier122of common NN121and further obtains delay flag information that is an output of delay flag classifier123obtained by inputting the obtained feature value into delay flag classifier123.

Referring again toFIG.11, next, delay flag trainer304performs backward propagation computation on common NN121, using the delay flag correct answer label as reference data (teacher data), and updates the weight and bias (S403). On the basis of backpropagation, delay flag trainer304regards the output value of the loss function as an error and executes update processing on the weight (weight data) and the bias (bias data) in delay flag classifier123of common NN121by backward propagation computation.

As shown inFIG.12, the loss function in common NN121represents how much is the error of the inference result (delay flag information) of delay flag classifier123of common NN121with respect to the delay flag correct answer label corresponding to the scene image.

Delay flag trainer304obtains delay flag information that is an output of delay flag classifier123on the basis of the scene image, and obtains an output value of the loss function on the basis of the delay flag information and the delay flag correct answer label. Then, delay flag trainer304regards the output value as an error and executes update processing on the weight and bias in delay flag classifier123of common NN121by backward propagation computation.

As described above, in steps S402and S403, common NN121(for example, delay flag classifier123) is trained by machine learning, using the scene image as input data, the delay flag information output from delay flag classifier123of common NN121as output data, and the time measured by delay flag information measurer302as reference data.

Note that delay flag trainer304may regard the output value as an error and executes update processing on the weights and biases in feature classifier122and delay flag classifier123by backward propagation computation. That is, delay flag trainer304may target feature classifier122for the training processing in addition to delay flag classifier123. Thus, the weight and bias are updated in feature classifier122together with those in delay flag classifier123, so that it is expected that the accuracy of the delay flag information output by delay flag classifier123is improved.

Referring again toFIG.11, delay flag trainer304determines whether or not the correct answer rate of the NN inference result (or the error rate obtained by comparing the inference result with the delay flag correct answer label) satisfies a requirement (S404). The requirement may be, for example, a correct answer rate required for device100. That is, delay flag trainer304may perform the determination in step S404based on whether or not the correct answer rate of the NN inference result is higher than or equal to the correct answer rate required for device100.

When the correct answer rate of the NN inference result satisfies the requirement (Yes in S404), delay flag trainer304ends the processing. When the correct answer rate of the NN inference result does not satisfy the requirement (No in S404), delay flag trainer304returns to step S401and continues the processing. That is, when the correct answer rate of the NN inference result does not satisfy the requirement, delay flag trainer304performs training by using various scene images and correct answer labels with the scene images and continues the training processing until the correct answer rate of the NN inference result satisfies the requirement.

After the completion of the training processing by delay flag trainer304, NN data400becomes able to execute the operation of device100described above. The generated NN data400is transmitted to device100.

Although training apparatus300according to the present embodiment performs the training processing sequentially, training apparatus300may perform multi-task learning on feature classifier122of common NN121and specialized NN group124and on delay flag classifier123of common NN121, and measure the processing time for the forward propagation computation of specialized NN group124at that time, to obtain delay flag information. At the time of backward propagation computation, training apparatus300may use error information with the delay flag information (delay flag correct answer label) as a correct answer to update the weight and bias of delay flag classifier123of common NN121, and may use error information of multi-task learning to update the weights and biases of feature classifier122of common NN121and specialized NN group124. As described above, training apparatus300may be configured to be able to perform multi-task learning on feature classifier122of common NN121and specialized NN group124and on delay flag classifier123of common NN121.

Here, the output of the training result evaluation of training apparatus300according to the present embodiment will be described with reference toFIG.13.FIG.13is a table showing the occurrence rate of the delay flag information according to the present embodiment.

As an example of the output of the training result evaluation of training apparatus300according to the present embodiment, a configuration may be considered in which the “occurrence rate of delay flag information” shown inFIG.13is output in addition to inference quality information such as the mean average precision (mAP) of specialized NN group124. The result shown inFIG.13may motivate a developer to change configurations of NNs, hyperparameters, or the like in order to increase the occurrence rate of the delay flag information “OFF”. In addition, by displaying the occurrence rate of the inference delay not satisfying the required specification (the occurrence rate corresponding to the delay flag information “NG” inFIG.13) or the information of the corresponding scene image, training apparatus300can encourage the developer or the like to consider measures to prevent the occurrence of the inference delay at the process stage of the NN training processing.

Other Embodiments

Although the present disclosure has been described above on the basis of the embodiment, the present disclosure is not limited to the above embodiment.

For example, in the above embodiment, an example has been described where a feature value as an output of feature classifier122is input into delay flag classifier123as input data, but the input data is not limited thereto. The input data may be, for example, an input image from camera200. For example, NN inference unit120may input an input image from camera200into delay flag classifier123to obtain delay flag information that is an output of delay flag classifier123. In this case, in step S402ofFIG.11, delay flag trainer304obtains a delay flag that is an output of delay flag classifier123obtained by inputting the scene image into delay flag classifier123of common NN121. Then, in step S403ofFIG.11, delay flag trainer304performs backward propagation computation on delay flag classifier123of common NN121, using the delay flag correct answer label as reference data (teacher data), and updates the weight and bias of delay flag classifier123. Further, for example, the same input data (input image) may be input into feature classifier122and delay flag classifier123.

The input data of delay flag classifier123may be, for example, intermediate data of the inference of feature classifier122. For example, NN inference unit120may cause feature classifier122to output intermediate data of the inference and input the intermediate data into delay flag classifier123, thereby obtaining delay flag information that is an output of delay flag classifier123.

In the above embodiment, an example has been described where the processing of the subsequent tasks is processing using NNs, and NN inference computation management unit130determines the computation order of the processing using NNs, but the present disclosure is not limited thereto. The processing of the subsequent tasks may be processing not using NNs, and NN inference computation management unit130may determine the computation order of the processing not using NNs.

In the above embodiment, an example has been described where the NPU is mounted on SoC601of device100, but the NPU may not be mounted. Of CPU602and the NPU, only CPU602may be mounted on SoC601, and for example, each processing described above may be executed by CPU602.

In the above embodiment, an example has been described where NN inference processing unit110is used in the product field of IoT devices, but the product field is not limited thereto. NN inference processing unit110is also applicable in product fields different from IoT devices, such as autonomous vehicles, robots, and unmanned aerial vehicles like drones.

The order of the plurality of types of processing described in the above embodiment is an example. The order of the plurality of types of processing may be changed, or the plurality of types of processing may be executed in parallel. Some of the plurality of types of processing need not be executed.

Each of the components described in the embodiment may be achieved as software or may typically be achieved as a large-scale integrated circuit (LSI) that is an integrated circuit. The components may each be individually integrated into one chip or may be integrated into one chip so as to include some or all of them. The circuit is referred to as an LSI here but is sometimes referred to as an integrated circuit (IC), a system LSI, a super LSI, or an ultra-LSI, depending on the degree of integration. The method for making an integrated circuit is not limited to being achieved by an LSI but may be achieved by a dedicated circuit or a general-purpose processor. After the manufacturing of the LSI, a field programmable gate array (FPGA) that can be programmed, or a reconfigurable processor in which the connections and settings of circuit cells in the LSI can be reconfigured, may be used. Moreover, if an integrated circuit technology replacing the LSI appears due to the advance of semiconductor technology or another technology derived therefrom, the components may naturally be integrated using that technology.

The division of functional blocks in the block diagram is an example, and a plurality of functional blocks may be achieved as one functional block, one functional block may be divided into a plurality of functional blocks, or some functions may be transferred to other functional blocks. Functions of a plurality of function blocks with similar functions may be processed in parallel or in time division by a single piece of hardware or software.

The training apparatus provided in the information processing system may be achieved as a single apparatus or by a plurality of apparatuses. For example, each processing unit of the training apparatus may be achieved by two or more server apparatuses. When the information processing system is achieved by a plurality of server apparatuses, the components included in the information processing system may be distributed to the plurality of server apparatuses in any manner. A method for communication between the plurality of server apparatuses is not particularly limited.

Furthermore, the technique of the present disclosure may be a program for causing a computer to execute characteristic processing in the above information processing method or may be a non-temporary computer-readable recording medium on which the program is recorded. Needless to say, the program can be distributed via a transmission medium such as the Internet. For example, the program and a digital signal including the program may be transmitted via a telecommunications line, a wireless or wired communication line, a network represented by the Internet, data broadcasting, or the like. The program and the digital signal including the program may be executed by other independent computer systems by being recorded on a recording medium and transferred or by being transferred via the network or the like.

In the embodiments, each of the components may be formed of dedicated hardware or may be achieved by executing a software program suitable for each of the components. Each of the components may be achieved by a program executor, such as a central processing unit (CPU) or a processor, reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

INDUSTRIAL APPLICABILITY

The present disclosure is applicable to an information processing apparatus and the like using inference processing of a plurality of NNs.