Patent Description:
In recent years, there have been proposed various apparatuses each carrying an inference model that employs a neural network (hereinafter, the neural network is referred to as an NN, and the inference model is referred to as an NN inference model or simply an NN). For example, it has been proposed to install an NN inference model in an Internet of Things (IoT) device, which is generally required to be inexpensive and to operate with low power, which limits its calculation capability. For example, Patent Literature (PTL) <NUM> discloses an NN apparatus and the like capable of reducing the power consumption of the entire apparatus while maintaining high accuracy by causing an NN inference model to execute inference at a drive frequency corresponding to the accuracy, the amount of calculation, and the like required for each layer of the NN inference model.

[PTL <NUM>] <CIT> Other prior art includes:.

However, PTL <NUM> discloses only a technique for inference processing of a single inference model.

Therefore, the present disclosure provides an information processing apparatus, an information processing method, and a program that can perform inference processing using a plurality of inference models even in a limited calculation environment.

An information processing apparatus according to one embodiment of the present disclosure includes: an obtainer that obtains sensing data, the sensing data being an input image obtained from a camera; an inference processing unit that inputs the sensing data into an inference model to obtain a result of inference, wherein the result of inference is a feature value of the sensing data output from a feature classifier, the feature classifier being a neural network, and information on a processing time for a plurality of subsequent tasks to processing performed by the inference model, wherein the feature classifier outputs a feature value of the sensing data to each of a plurality of specialised neutral networks receiving the input from the feature classifier, the specialised neural networks include a plurality of processing tasks and are in parallel and in charge of subsequent processing; each neural network performs different output on the basis of the feature value, wherein the feature classifier also outputs a delay flag information to a delay flag classifier, the delay flag classifier being a trained neural network that outputs delay flag information indicating the time including the time from the inputting of the feature value into the specialised neural networks until the end of the processing of the plurality of subsequent processing to each of the specialised neural networks; a determiner that determines a task schedule for a task processing unit that processes the plurality of subsequent tasks to process the plurality of subsequent tasks based on the delay flag information; and a controller that inputs the result of the inference into the task processing unit to process the plurality of subsequent tasks according to the task schedule determined.

An information processing method according to one embodiment of the present disclosure is a method executed by a computer, the information processing method comprising: obtaining sensing data, the sensing data being an input image obtained from a camera; inputting the sensing data into an inference model to obtain a result of inference, wherein the result of inference is a feature value of the sensing data output from a feature classifier, the feature classifier being a neural network, and information on a processing time for a plurality of subsequent tasks to processing performed by the inference model, wherein outputting a feature value of the sensing data to each of a plurality of specialised neutral networks receiving the input from the feature classifier, the specialised neural networks include a plurality of processing tasks and are in parallel and in charge of subsequent processing; each neural network performing different output on the basis of the feature value, also outputting a delay flag information to a delay flag classifier, the delay flag classifier being a trained neural network that outputs delay flag information indicating the time including the time from the inputting of the feature value into the specialised neural networks until the end of the processing of the plurality of subsequent processing to each of the specialised neural networks; determining a task schedule for a task processing unit that processes the plurality of subsequent tasks to process the plurality of subsequent tasks based on the delay flag information; inputting the result of the inference into the task processing unit that processes the plurality of subsequent tasks to process the plurality of subsequent tasks according to the task schedule determined; measuring an inference time including a time from the inputting of the result of the inference into the task processing unit until an end of the processing of the plurality of subsequent tasks; and training the inference model by machine learning using the sensing data as input data, the information on the processing time for the plurality of subsequent tasks as output data, and the inference time measured as reference data.

A computer program according to one embodiment of the present disclosure is a computer program comprising code that, when said code is executed by a computer, causes the computer to execute the information processing method.

According to one aspect of the present disclosure, it is possible to achieve an information processing apparatus and the like that can perform inference processing using a plurality of inference models even in a limited calculation environment.

Circumstances leading to the present disclosure will be described before the description of examples of the present disclosure.

The present disclosure relates to an apparatus (hereinafter referred to as an inference apparatus) carrying an inference model that is generated using machine learning. More specifically, the present disclosure relates to an NN-type inference apparatus, enabling the maintenance of the service quality by changing a computation order, and relates to training processing therefor.

In recent years, the performance of an object detection apparatus, a classification apparatus, and the like among inference apparatuses has been dramatically improved by the adoption of deep learning technology, and numerous research or commercialization efforts are underway. In a smartphone or an autonomous vehicle (robot car) driven by a machine instead of a diver, an inference apparatus for an image transferred from a camera apparatus, an infrared array sensor, or the like is one of element functions. In the case of the autonomous vehicle, an object means, for example, a pedestrian, a passenger car, a road sign, a building, a road area, or the like.

Further, the field of edge artificial intelligence (AI), in which an NN inference model is installed in an Internet of Things (IoT) device and the device operates in accordance with the determination of the IoT device itself rather than always causing the cloud to perform determination, has recently begun to come into proliferation. For example, in the fields of various industrial products including the IoT device, industrial products, which are mounted with NN-type inference apparatuses and perform services on the basis of the inference results of the devices, have been put on the market. The IoT device is generally required to be inexpensive and to operate with low power, which limits its calculation capability.

The IoT device is required to save power. In a case where a service using a plurality of NN inference models is performed in such an IoT device, it is difficult to complete the service within a required processing time unless computation for inferring the plurality of NN inference models in parallel is performed. As described above, for example, in the IoT device, it is assumed that computation for inference is performed using a plurality of NN inference models provided in parallel. Note that the required processing time is also referred to as an allowable time.

Further, an NN processing dedicated circuit called a neural network processing unit (NPU), which is mounted with a digital signal processor (DSP), an application-specific integrated circuit (ASIC), and the like, has begun to be installed in the IoT device separately from a central processing unit (CPU). However, a service required for the IoT device has come to use inference processing of a plurality of NNs as described above, and the amount of processing for the service tends to increase.

The IoT device is required to save power in addition to completing the service within the required processing time. Thus, the IoT device is prone to a problem of being unable to complete the inference processing within the required processing time, depending on the input image.

Note that the service using a plurality of NN inference models includes, for example, processing of determining that a target person present in a scene image has "something" in the right hand by inference processing of a skeleton detecting NN inference model and determining that "something" is a smartphone by inference processing of an object detecting NN inference model.

PTL <NUM> mentioned above discloses a method assuming a service using inference processing of a single NN inference model and does not disclose a method related to the ensuring of a processing time in a service using inference processing of a plurality of NN inference models. Further, no consideration has hitherto been given to a design and a technical solution related to processing of estimating a processing time necessary for inference processing of a plurality of inference models at an initial stage and performing inference processing in an optimum computation order within a required processing time, which is not disclosed in PTL <NUM>, either.

Therefore, an intensive study has been conducted on an information processing apparatus and the like capable of performing inference processing within a required processing time in a service using inference processing of a plurality of NN inference models, and an information processing method and the like described below have been devised.

An information processing apparatus according to one aspect of the present disclosure includes: an obtainer that obtains sensing data; an inference processing unit that inputs the sensing data into an inference model to obtain a result of inference and information on a processing time for a plurality of subsequent tasks to processing performed by the inference model; a determiner that determines a task schedule for a task processing unit that processes the plurality of subsequent tasks to process the plurality of subsequent tasks on the basis of the information on the processing time for the plurality of subsequent tasks; and a controller that inputs the result of the inference into the task processing unit to process the plurality of subsequent tasks according to the task schedule determined.

With this, the information processing apparatus can determine a task schedule for a task processing unit (for example, a plurality of NN inference models) that processes a plurality of subsequent tasks on the basis of information on a processing time for the plurality of subsequent tasks with respect to a sensing data. By determining the task schedule so that the processing time of the task processing unit is within a required processing time on the basis of the information on the processing time for the plurality of subsequent tasks, the information processing apparatus can perform inference processing using the plurality of inference models even in a limited calculation environment.

Further, for example, the inference model may include a first inference model and a second inference model, the sensing data may be input into the first inference model to obtain the result of the inference, and the result of the inference obtained or intermediate data of the inference may be input into the second inference model to obtain the information on the processing time for the plurality of subsequent tasks.

With this, it is possible to make the amount of calculation of the second inference model less than that in a case where the sensing data is input into the second inference model. Hence it is possible to shorten the processing time in the second inference model. When the result of the inference is input, the input of the second inference model is the same as the input of the subsequent task processing unit, so that it is expected that the precision or accuracy of the information on the processing time of the subsequent task processing unit is improved. When the intermediate data of the inference is input, the processing of the second inference model can be started before the end of the inference of the first inference model, thereby further shortening the processing time of the second inference model.

With this, it is possible to start the processing of the second inference model by using the sensing data without waiting for the result of the inference to be output from the first inference model. In other words, it is possible to accelerate the start timing of the processing of the second inference model.

Further, for example, the information on the processing time for the plurality of subsequent tasks may include information on a delay relative to a predetermined time that is determined with respect to the processing time for the plurality of subsequent tasks.

With this, it is possible to determine the task schedule for the task processing unit on the basis of the information on the delay. The information processing apparatus determines the task schedule so that the processing time of the task processing unit is within the required processing time on the basis of the information on the delay of the plurality of subsequent tasks, thus increasing the certainty that in the service using the inference processing of the plurality of inference models, the inference processing can be performed within the required processing time.

Further, for example, a first schedule may be determined when information indicating that the delay is less than a predetermined value is included in the information on the delay, and a second schedule, in which the processing time for the plurality of subsequent tasks is shorter than in the first schedule, may be determined when information indicating that the delay is greater than or equal to the predetermined value is included in the information on the delay.

With this, when the delay is greater than or equal to the predetermined value, the second schedule having a shorter processing time is determined, thus increasing the certainty that the inference processing can be performed within the required processing time.

Further, for example, the first schedule may be determined using a first rule, and the second schedule may be determined using a second rule with which the processing time for the plurality of subsequent tasks in the second schedule is shorter than in the first schedule.

With this, the schedule is determined based on the first and second rules, whereby it is possible to reduce the amount of calculation and the calculation time as compared to a case where a schedule is searched for dynamically.

Further, for example, the inference model may execute inference processing that is preprocessing common to the plurality of subsequent tasks.

With this, the use of the inference model for the preprocessing common to each task can compress the total calculation amount and memory usage amount of the inference processing necessary for each task processing.

Further, for example, the inference model and the task processing unit may be neural network models, the result of the inference may be a feature value of the sensing data, and the task schedule may include an order in which the task processing unit performs memory loading and the processing.

With this, it is possible to apply a high-performance NN inference model in a limited calculation environment such as an IoT device. In addition, the processing time for the subsequent task is inferred based on the feature value of the sensing data, thus facilitating the inference of the processing time corresponding to the difficulty level of the task processing with respect to the sensing data.

An information processing method according to one aspect of the present disclosure is a method executed by a computer, the method including: obtaining sensing data; inputting the sensing data into an inference model to obtain a result of inference and information on a processing time for a plurality of subsequent tasks to processing performed by the inference model; inputting the result of the inference into a task processing unit that processes the plurality of subsequent tasks to process the plurality of subsequent tasks; measuring an inference time including a time from the inputting of the result of the inference into the task processing unit until an end of the processing of the plurality of subsequent tasks; and training the inference model by machine learning using the sensing data as input data, the information on the processing time for the plurality of subsequent tasks as output data, and the inference time measured as reference data. A program according to one aspect of the present disclosure is a program for causing a computer to execute the information processing method described above.

With this, it is possible to generate an inference model that achieves the processing executed by the information processing apparatus. The inference processing is performed using the generated inference model, whereby it is possible to determine the task schedule so that the processing time of the task processing unit is within the required processing time on the basis of the information on the processing time for the plurality of subsequent tasks.

Moreover, these generic or specific aspects may be achieved using a system, an apparatus, a method, an integrated circuit, a computer program, or a non-temporary storage medium such as a computer-readable compact disc read-only memory (CD-ROM), or may be implemented using any combination of systems, apparatuses, methods, integrated circuits, computer programs, and storage media.

Hereinafter, specific examples of an information processing apparatus and the like according to one aspect of the present disclosure will be described with reference to the drawings. Each of the
examples
described here shows a specific example of the present disclosure. Therefore, the numerical values, components, steps, the order of steps, and the like shown in the following
examples
are examples and are not intended to limit the present disclosure. Among the components in the following exemplary embodiments, components not recited in independent claims are described as optional components. In all the
examples,
the respective contents can be combined.

Each of the drawings is a schematic diagram and is not necessarily a strict illustration. Thus, for example, the scales or the like are not necessarily coincident with each other in the drawings. In addition, in the drawings, substantially the same components are denoted by the same reference numerals, and duplicated description is omitted or simplified.

In the present specification, a term, as well as a numerical value and a numerical range, indicating a relationship between elements, such as "same" or "coincident", is not an expression expressing only a strict meaning but is an expression meaning to include a substantially equivalent range, for example, a difference of about a few percent.

Hereinafter, an information processing system and the like according to the present example will be described below with reference to <FIG>.

First, a configuration of information processing system <NUM> according to the present embodiment will be described with reference to <FIG>. <FIG> is a block diagram showing a functional configuration of information processing system <NUM> according to the present example. <FIG> is a block diagram showing a functional configuration of inference processing unit <NUM> according to the present example.

As shown in <FIG>, information processing system <NUM> includes device <NUM>, camera <NUM>, and training apparatus <NUM>. Device <NUM> performs inference processing on an input image from camera <NUM> and performs a service corresponding to the result of the inference. Training apparatus <NUM> generates NN data <NUM> for device <NUM> to perform the inference processing. Each component will be described below.

Device <NUM> is, for example, an IoT device and has a function of providing a plurality of services on the basis of an input image obtained from camera <NUM>. Device <NUM> includes inference processing unit <NUM> and task executor <NUM>. The input image obtained from camera <NUM> is an example of sensing data.

Inference processing unit <NUM> performs inference processing on the input image input from camera <NUM> by using NN data <NUM> generated by training apparatus <NUM>. Inference processing unit <NUM> performs the inference processing by using a plurality of NNs provided in parallel. In other words, NN data <NUM> includes a plurality of NNs provided in parallel. Note that NN data <NUM> stores, for example, the layer configurations, weights, and biases of the NNs.

As shown in <FIG>, NN data <NUM> includes common NN <NUM> and specialized NN group <NUM>. Common NN <NUM> is an NN for preprocessing in NN data <NUM>, and specialized NN group <NUM> is an NN group for subsequent processing in NN data <NUM>.

The time required for the inference processing in device <NUM> may vary depending on the input image. Thus, depending on the input image, the inference processing may take time, and the service may not be performed within the processing time required for device <NUM>. Therefore, inference processing unit <NUM> obtains an inference result related to the time taken for subsequent inference processing on the input image by using the NN for preprocessing and changes the order of the subsequent inference processing (an example of a task schedule) or the like on the basis of the obtained inference result. The configurations of inference processing unit <NUM> and the like will be described below.

Inference processing unit <NUM> includes NN inference unit <NUM>, NN inference computation management unit <NUM>, and obtainer <NUM>. Note that inference processing unit <NUM> is an inference processing apparatus capable of changing a computation order. It can also be said that inference processing unit <NUM> is a computation-order variable inference apparatus. Inference processing unit <NUM> is an example of an information processing apparatus.

NN inference unit <NUM> performs inference processing on the input image input from camera <NUM> by using a plurality of NNs based on NN data <NUM> generated by training apparatus <NUM>.

NN inference unit <NUM> performs inference processing by using common NN <NUM> and specialized NN group <NUM> that includes a plurality of NNs provided in parallel. First, NN inference unit <NUM> performs forward propagation computation processing by using common NN <NUM>. Common NN <NUM> is, for example, a forward propagation type NN.

Common NN <NUM> is an NN that outputs an inference processing result and delay flag information common to the plurality of NNs of specialized NN group <NUM>. Common NN <NUM> executes inference processing that is preprocessing common to each task. It can also be said that common NN <NUM> is a model executed as preprocessing common to each task. Common NN <NUM> includes feature classifier <NUM> and delay flag classifier <NUM>. Feature classifier <NUM> and delay flag classifier <NUM> are trained NNs, for example. Note that common NN <NUM> is an example of an inference processing unit. The inference processing as preprocessing is not limited to the processing using the NN.

By inputting the input image into feature classifier <NUM> of common NN <NUM>, NN inference unit <NUM> obtains an inference processing result that is an output of feature classifier <NUM>. The inference processing result (inference result) is, for example, a feature value (image feature value) but is not limited thereto. An example in which the inference processing result is a feature value will be described below.

Feature classifier <NUM> is a trained NN (NN for feature classification) trained to receive an input image as an input and output a feature value corresponding to the input image. Feature classifier <NUM> performs inference processing common to each of the plurality of NNs of specialized NN group <NUM> to be subjected to forward propagation computation processing. The feature value is information common to each of the plurality of NNs of specialized NN group <NUM>.

NN inference unit <NUM> outputs a feature value that is an output of feature classifier <NUM> to each of the plurality of NNs of specialized NN group <NUM>. In the present example, NN inference unit <NUM> also outputs the feature value to delay flag classifier <NUM>. As described above, the feature value as the output of feature classifier <NUM> may be used in delay flag classifier <NUM>. In the present example, the same feature value is input into delay flag classifier <NUM> and each of the plurality of NNs of specialized NN group <NUM>.

NN inference unit <NUM> inputs the feature value from feature classifier <NUM> into delay flag classifier <NUM> to obtain delay flag information that is an output of delay flag classifier <NUM>. Delay flag classifier <NUM> is a trained NN (NN for delay flag classification) trained to receive a feature value as an input and output delay flag information corresponding to the feature value. The delay flag information is information indicating the time (inference time) including the time from the inputting of the feature value (inference result) into specialized NN group <NUM> (a task processing unit or a part thereof) until the end of the processing of the plurality of subsequent tasks. For example, the delay flag information is information indicating an inference result as to whether or not the inference processing time of entire NN inference unit <NUM> exceeds a threshold. It can also be said that the delay flag information is information on a delay relative to a predetermined time (for example, a threshold). The predetermined time is the time determined with respect to the processing time for the plurality of subsequent tasks. The threshold may be set, for example, based on the processing time required for device <NUM> or the service content of device <NUM>. The threshold is set in advance. Note that the delay flag information may be, for example, information indicating an inference result as to whether or not the processing time of entire device <NUM> exceeds the threshold. The delay flag information may be, for example, information indicating an inference result as to whether or not the total processing time of NN inference unit <NUM> and task executor <NUM> exceeds the threshold.

The delay flag information may be, for example, information indicating "OFF" when the inference processing time is less than or equal to the threshold, or information indicating "ON" when the inference processing time exceeds the threshold. Further, a plurality of "ON" may be set. The delay flag information may be, for example, information indicating "ON1" when the inference processing time exceeds a first threshold and is less than or equal to a second threshold that is greater than the first threshold, and information indicating "ON2" when the inference processing time exceeds the second threshold. The first threshold is, for example, <NUM> msec, and the second threshold is, for example, <NUM> msec, but the present disclosure is not limited thereto. The delay flag information is an example of information on the processing time for the plurality of subsequent tasks. The processing time for the plurality of subsequent tasks may be, for example, the processing time of specialized NN group <NUM>, or the total processing time of specialized NN group <NUM> and task executor <NUM>. In this case, specialized NN group <NUM> and task executor <NUM> are examples of a task processing unit.

In the present example, an example will be given of an example in which three levels of "OFF", "ON1" and "ON2" are set as the delay flag information. Note that two levels or four or more levels may be set for the delay flag information.

The delay flag information is not limited to the information indicating the delay level, such as "OFF", "ON1", or "ON2", but may be the processing time (computation time) itself, for example.

Note that delay flag classifier <NUM> is desired to output delay flag information as soon as possible. Delay flag classifier <NUM> may thus be an NN having a smaller amount of calculation than feature classifier <NUM> and specialized NN group <NUM>.

Common NN <NUM> configured as described above is an example of an inference model that outputs a feature value and delay flag information when receiving input of an input image from camera <NUM>. Feature classifier <NUM> is an example of a first inference model, and delay flag classifier <NUM> is an example of a second inference model. That is, the inference model includes the first inference model and the second inference model.

On the basis of the delay flag information from delay flag classifier <NUM>, NN inference computation management unit <NUM> determines a forward propagation computation method (for example, a computation order) for processing the plurality of tasks of specialized NN group <NUM> subsequent to common NN <NUM> and notifies specialized NN group <NUM> of the determined method. Specifically, NN inference computation management unit <NUM> notifies NN inference unit <NUM>, which controls the processing of specialized NN group <NUM>, of the determined forward propagation computation method. Thereby, NN inference unit <NUM> can input the feature value from feature classifier <NUM> into specialized NN group <NUM> to process the plurality of subsequent NNs by the determined forward propagation computation method. NN inference computation management unit <NUM> is an example of a determiner for determining a forward propagation computation method and a controller for causing a task at a later stage.

In the present example, NN inference computation management unit <NUM> determines a method for the forward propagation computation subsequent to common NN <NUM> on the basis of the delay flag information and table <NUM> shown in <FIG> is a diagram showing an example of the configuration of table <NUM> including the delay flag information and forward propagation computation methods for specialized NN group <NUM> associated with the delay flag information according to the present example. A parallel computation item in <FIG> shows methods for the forward propagation computation subsequent to common NN <NUM>, for example, forward propagation computation methods for specialized NN group <NUM>.

As shown in <FIG>, the delay flag information "OFF" is associated with a parallel computation item "optimum parallel computation processing" (see first line <NUM> in <FIG>), the delay flag information "ON1" is associated with a parallel computation item "delay handling parallel computation processing" (see second line <NUM> in <FIG>), and the delay flag information "ON2" is associated with a parallel computation item "delay handling parallel computation processing + CPU overclocking processing" (see third line <NUM> in <FIG>). Note that "ON" mentioned here means that the computation time of NN inference unit <NUM> exceeds the allowable time, and the computation order needs to be changed from the computation order indicated by the "optimum parallel computation processing". The operation order is an example of a task schedule.

The "optimum parallel computation processing" indicates that computation processing is performed in accordance with a parallel computation method (for example, a parallel computation order) determined based on the number of memory accesses, processing time (computation time), power consumption, and the like. For example, when a delay is less than a predetermined value (for example, a delay is less than a threshold), a forward propagation computation order (an example of a first schedule) is determined using the "optimum parallel computation processing". Note that the "optimum parallel computation processing" is an example of a first rule. That the delay is less than the predetermined value includes that there is no delay or that the delay length or the delay degree is less than the predetermined value.

The "delay handling parallel computation processing" indicates that computation processing is performed in accordance with a parallel computation method determined so that the processing time becomes shorter than that for the "optimum parallel computation processing". For example, when the delay is greater than or equal to the predetermined value (for example, the delay is greater than or equal to the threshold), a forward propagation computation order (an example of a second schedule) is determined using the "delay handling parallel computation processing". The parallel computation method is, for example, a method in which the computation order is rearranged so as to preferentially start processing (for example, non-maximum suppression (NMS) processing), the processing time of which changes in accordance with the number of candidate detection areas in the input image, but the present disclosure is not limited thereto. Note that the "delay handling parallel computation processing" is an example of a second rule with which the processing time for the plurality of subsequent tasks is shorter than that in the first rule. That the delay is greater than or equal to the predetermined value includes that there is a delay or that the delay length or the delay degree is greater than or equal to the predetermined value.

Note that NMS is an algorithm that deletes a certain candidate detection area when the degree of overlap (a value of intersection over union (IoU)) between the certain candidate detection area and a candidate detection area having a higher score than the certain candidate detection area exceeds a threshold set by training processing.

The "delay handling parallel computation processing + CPU overclocking processing" indicates that in addition to the "delay handling parallel computation processing", a setting of a drive frequency of a CPU (for example, CPU <NUM> shown in <FIG> to be described later) is increased. As described above, the delay flag information may include information related to the setting of the drive frequency of the CPU in addition to the change of the computation order of the parallel computation. That is, coping with a delay is not limited to changing the parallel computation but may include changing the setting (for example, the setting of the drive frequency) of each processing unit included in device <NUM> together with or instead of changing the parallel computation.

Note that the computation order indicated by the "optimum parallel computation processing" and the computation order indicated by the "delay handling parallel computation processing" are set in advance and stored in memory <NUM> or the like.

As described above, a plurality of types of delay flag information may exist, such as the delay flags "ON1" and "ON2". For example, a plurality of types of delay flag information may exist depending on the result of the processing time measurement performed by delay flag information measurer <NUM> to be described later.

Specialized NN group <NUM> includes a plurality of NNs, each of which performs different output on the basis of the feature value from feature classifier <NUM>. In the present example, specialized NN group <NUM> includes first task NN <NUM> and second task NN <NUM>. Hereinafter, a description will be given of an example in which first task NN <NUM> is an object detecting NN and second task NN <NUM> is a skeleton detecting NN, but the present disclosure is not limited thereto. Further, a description will be given of an example in which first task NN <NUM> includes NMS processing and second task NN <NUM> does not include NMS processing, but the present disclosure is not limited thereto. Note that specialized NN group <NUM> (NN inference unit <NUM> that performs processing by using specialized NN group <NUM>) is an example of a task processing unit. When the task processing unit includes specialized NN group <NUM>, each of the inference model and the task processing unit is an NN model.

First task NN <NUM> is a trained NN trained to receive a feature value from feature classifier <NUM> as an input and output an inference result of object detection corresponding to the feature value. The inference result as an output of first task NN <NUM> is output to task executor <NUM> corresponding to first task NN <NUM>.

Second task NN <NUM> is a trained NN trained to receive a feature value from feature classifier <NUM> as an input and output an inference result of skeleton detection corresponding to the feature value. The inference result as an output of second task NN <NUM> is output to task executor <NUM> corresponding to second task NN <NUM>.

Note that the object detection and the skeleton detection are examples of the plurality of subsequent tasks.

When delay flag classifier <NUM> outputs delay flag information including "ON1" or "ON2", the computation orders of first task NN <NUM> and second task NN <NUM> are changed from when the delay flag information is "OFF". In the present example, at least the computation order in first task NN <NUM> including the NMS processing is changed from when the delay flag information is "OFF".

NN inference computation management unit <NUM> instructs NN inference unit <NUM> to perform forward propagation computation on specialized NN group <NUM> in accordance with a parallel computation item (parallel computation method) determined based on the delay flag information and table <NUM>, for example. Thus, NN inference computation management unit <NUM> can change the computation order in accordance with the inference result (delay flag information) related to the time of inference processing of the input image, thus preventing the processing time from becoming long.

Obtainer <NUM> obtains an input image from camera <NUM>. Obtainer <NUM> outputs the obtained input image to NN inference unit <NUM>.

Subsequently, task executor <NUM> executes a preset task on the basis of the inference result of NN inference unit <NUM>. Task executor <NUM> may be, for example, a display for displaying the inference result, a light emitter for emitting light corresponding to the inference result, or a transmitter for transmitting the inference result to an external device.

<FIG> is a schematic diagram showing a configuration of system-on-chip (SoC) <NUM> of device <NUM> according to the present example. In the present embodiment, an NPU is mounted.

As shown in <FIG>, SoC <NUM> includes CPU <NUM>, first NPU <NUM>, second NPU <NUM>, and memory <NUM>. Each function of NN inference unit <NUM> is achieved by installing a trained NN (NN data <NUM>) on SoC <NUM>.

CPU <NUM> is a processing unit that executes various types of processing in device <NUM>. CPU <NUM> executes the NMS processing, the processing of task executor <NUM>, and the like.

First NPU <NUM> and second NPU <NUM> are dedicated circuits that perform NN processing. First NPU <NUM> and second NPU <NUM> execute NN processing in device <NUM>. First NPU <NUM> and second NPU <NUM> execute, for example, processing using common NN <NUM> and processing using specialized NN group <NUM>. In the present example, first NPU <NUM> and second NPU <NUM> execute skeleton detection processing, object detection processing, and delay label determination processing.

Memory <NUM> stores a program executed by each of CPU <NUM>, first NPU <NUM>, and second NPU <NUM>. Memory <NUM> stores NN data <NUM>.

As described above, inference processing unit <NUM> is configured to obtain an input image, output a feature value and delay label information when the input image is input into common NN <NUM>, determine a forward propagation computation method (optimum parallel computation processing, delay coping parallel computation processing, etc.) for processing specialized NN group <NUM> on the basis of the delay label information, and input the feature value into specialized NN group <NUM> to process specialized NN group <NUM> by the determined forward propagation computation method. For example, inference processing unit <NUM> includes: obtainer <NUM> that obtains an input image; common NN <NUM> that receives the input of the input image and outputs a feature value and delay label information; NN inference computation management unit <NUM> (an example of a determiner) that determines a forward propagation computation method (optimum parallel computation processing, delay coping computation processing, etc.) for processing specialized NN group <NUM> on the basis of the delay label information; and NN inference computation management unit <NUM> (an example of a controller) that inputs a feature value into specialized NN group <NUM> to process the determined forward propagation computation processing.

Next, training apparatus <NUM> will be described. Training apparatus <NUM> generates NN data <NUM> to be used for inference processing by NN inference unit <NUM> of device <NUM>. Training apparatus <NUM> includes multi-task trainer <NUM>, delay flag information measurer <NUM>, delay flag correct answer label generator <NUM>, delay flag trainer <NUM>, training label database <NUM> (training label DB <NUM>), and scene image database <NUM> (scene image DB <NUM>).

multi-task trainer <NUM> trains feature classifier <NUM> and specialized NN group <NUM> of NN data <NUM>. In the present example, multi-task trainer <NUM> performs training for object detection and skeleton detection.

multi-task trainer <NUM> performs training processing in which multi-task learning is applied to the training of NNs. Although the details will be described later, multi-task trainer <NUM> receives a scene image obtained from scene image DB <NUM> as an input, uses a correct answer label in skeleton detection corresponding to the scene image and a correct answer label in object detection corresponding to the scene image obtained from training label DB <NUM> as reference data (teacher data), and generates NNs (common NN <NUM> and specialized NN group <NUM>) on the basis of backpropagation (BP) or the like.

Note that the multi-task learning is a method of machine learning that solves a plurality of tasks with a single model. This is a method aiming to improve the accuracy in task prediction by training a plurality of related tasks simultaneously to obtain a "common factor" among the tasks. In the image recognition field, a plurality of tasks, such as object classification, object detection, and object area (segmentation) recognition, may be trained simultaneously. In the present example, the plurality of tasks are skeleton detection and object detection.

Research has been conducted to apply multi-task learning to training processing of NN data, and multi-task trainer <NUM> trains an NN commonly used for each task (for example, common NN <NUM>) and an NN specialized for each task (for example, specialized NN group <NUM>), for example. There is also an advantage that the use of the commonly used NN can compress the total calculation amount and memory usage amount of the inference processing necessary for each task processing.

Delay flag information measurer <NUM> and delay flag correct answer label generator <NUM> perform processing for generating training data that is used when delay flag classifier <NUM> is trained by delay flag trainer <NUM>.

Delay flag information measurer <NUM> measures the processing time for the forward propagation computation of common NN <NUM> and specialized NN group <NUM> generated by multi-task trainer <NUM>. In the present example, delay flag information measurer <NUM> measures at least the processing time of first task NN <NUM>. This is because first task NN <NUM> includes the NMS processing, and the processing time differs depending on the number of candidate detection areas in the input image. Second task NN <NUM> does not include processing for which the processing time differs depending on the number of candidate detection areas, such as the NMS processing, that is, the processing time does not change greatly in accordance with the input image, so that the processing time is not measured in the present example.

Delay flag correct answer label generator <NUM> generates a delay flag correct answer label to be used for the training of common NN <NUM> in delay flag trainer <NUM> by using information based on the measurement result of the processing time.

Delay flag trainer <NUM> performs processing of training common NN <NUM>, out of common NN <NUM> and specialized NN group <NUM> generated by multi-task trainer <NUM>, by using the delay flag correct answer label generated by delay flag correct answer label generator <NUM>. Delay flag trainer <NUM> updates weight data and bias data in delay flag classifier <NUM> of common NN <NUM> on the basis of backpropagation, using the scene image as input data and a delay flag correct answer label as reference data (teacher data), for example. That is, delay flag trainer <NUM> trains delay flag classifier <NUM> of common NN <NUM> by using the scene image and the delay flag correct answer label. Note that delay flag trainer <NUM> may train feature classifier <NUM> together with delay flag classifier <NUM> by using, for example, the delay flag correct answer label.

Training label DB <NUM> stores a training label for generating NN data <NUM>. Training label DB <NUM> stores, for example, a training label for skeleton detection and a training label for object detection with respect to one scene image.

Scene image DB <NUM> stores a scene image for generating NN data <NUM>.

Training label DB <NUM> and scene image DB <NUM> are achieved by, for example, a semiconductor memory or the like but are not limited thereto.

As described above, training apparatus <NUM> is configured to obtain a scene image, input the scene image into common NN <NUM> to obtain a feature value, input the feature value into specialized NN group <NUM> to process specialized NN group <NUM>, measure the time from the inputting of the scene image into common NN <NUM> until the end of the processing of specialized NN group <NUM>, and train common NN <NUM> by machine learning using the scene image as input data, the information on the processing time of specialized NN group <NUM> as output data, and the measured time as reference data. For example, training apparatus <NUM> includes: delay flag information measurer <NUM> that obtains a scene image, inputs the scene image into common NN <NUM> to obtain a feature value, inputs the feature value into specialized NN group <NUM> to process specialized NN group <NUM>, and measures the time from the inputting of the scene image into common NN <NUM> until the end of the processing of specialized NN group <NUM>; and delay flag trainer <NUM> that trains common NN <NUM> by machine learning using the scene image as input data, the information on the processing time of specialized NN group <NUM> as output data, and the measured time as reference data.

Subsequently, the operation of information processing system <NUM> will be described with reference to <FIG>.

First, the processing in device <NUM> will be described with reference to <FIG>. <FIG> is a flowchart showing the operation of device <NUM> according to the present example. For convenience, <FIG> describes an example in which the delay flag information included in table <NUM> is of two types of "ON (ON1)" and "OFF". In other words, it is assumed that delay flag classifier <NUM> has been trained to output either "ON" or "OFF".

As shown in <FIG>, obtainer <NUM> of device <NUM> obtains an input image from camera <NUM> (S101). Obtainer <NUM> outputs the input image obtained in step S101 to NN inference unit <NUM>.

Next, when receiving the input of the input image from obtainer <NUM>, NN inference unit <NUM> first executes forward propagation computation processing by using common NN <NUM> (S102). Specifically, NN inference unit <NUM> inputs an input image into feature classifier <NUM> of common NN <NUM> to obtain a feature value that is an output of feature classifier <NUM>. Then, NN inference unit <NUM> inputs the obtained feature value into delay flag classifier <NUM> to obtain delay flag information that is an output of delay flag classifier <NUM>. The delay flag information is a result inferred from the feature value by delay flag classifier <NUM>, and in the present example, the delay flag information indicates an inference result as to whether or not the inference processing time of entire NN inference unit <NUM> has exceeded a threshold.

NN inference unit <NUM> outputs the obtained delay flag information to NN inference computation management unit <NUM>. Note that NN inference unit <NUM> may further output the delay flag information to a functional unit provided in device <NUM> and external to inference processing unit <NUM>. The functional unit may be, for example, a functional unit having a graphic function.

Next, NN inference computation management unit <NUM> obtains delay flag information (S103). NN inference computation management unit <NUM> determines whether or not the obtained delay flag information is "ON" (S104). In a case where the delay flag information is "OFF", that is, in a case where the delay flag information is an inference result indicating that the service can be completed within the processing time required for device <NUM> even when the inference processing is performed in accordance with the "optimum parallel computation processing" (No in S104), NN inference computation management unit <NUM> determines the "optimum parallel computation processing to be a forward propagation computation method for specialized NN group <NUM>" on the basis of table <NUM> shown in <FIG> (S105). That is, when the delay flag information is "OFF", NN inference computation management unit <NUM> determines to execute the processing of specialized NN group <NUM> in the order based on the "optimum parallel computation processing". The order based on the "optimum parallel computation processing" is an example of a first schedule. NN inference computation management unit <NUM> outputs the determined forward propagation computation method for specialized NN group <NUM> to NN inference unit <NUM>.

When NN inference unit <NUM> obtains from NN inference computation management unit <NUM> that the forward propagation computation method for specialized NN group <NUM> is the "optimum parallel computation processing", NN inference unit <NUM> executes parallel forward propagation computation processing in the order indicated by the "optimum parallel computation processing". For example, NN inference unit <NUM> performs parallel forward propagation computation on specialized NN group <NUM> in the most efficient order (S106). NN inference unit <NUM> inputs the feature value that is an output of feature classifier <NUM> to each NN of specialized NN group <NUM>, performs inference processing according to the schedule based on the "optimum parallel computation processing", and obtains an inference result (skeleton detection result, object detection result, etc.) that is an output of each NN.

<FIG> is a schematic diagram showing an example of the processing time of common NN <NUM> and specialized NN group <NUM> and units in charge of computation thereof according to the present example. Specifically, first information 700a in <FIG> shows an example of the processing time and the units in charge of computation in a case where the processing ends within the allowable time even when the processing is performed in the computation order based on the "optimum parallel computation processing". Note that <FIG> describe examples in which the NMS processing has been outsourced to CPU <NUM>.

Area <NUM> indicates the occupancy rate of each unit (CPU <NUM>, first NPU <NUM>, and second NPU <NUM>) in common NN <NUM>. In area <NUM>, first NPU <NUM> (NPU <NUM> in <FIG>) and second NPU <NUM> (NPU <NUM> in <FIG>) perform processing by using common NN <NUM>. Area <NUM> indicates the occupancy rate of each unit in first task NN <NUM>. In area <NUM>, at least one of first NPU <NUM> and second NPU <NUM> performs object detection processing by using first task NN <NUM>.

Area <NUM> indicates the occupancy rate of each unit in second task NN <NUM>. In area <NUM>, at least one of first NPU <NUM> and second NPU <NUM> performs skeleton detection processing by using second task NN <NUM>. Areas 704a and 704b indicate the occupancy rate of each unit in the NMS processing of first task NN <NUM>. In areas 704a and 704b, CPU <NUM> performs the NMS processing.

As shown in <FIG>, when "No" in step S104, NN inference computation management unit <NUM> instructs NN inference unit <NUM> to perform forward propagation computation on specialized NN group <NUM> by the "optimum parallel computation processing" indicating that the computation processing is performed by the parallel computation method in which the number of memory accesses, the computation time, and the like are balanced. In this case, even when the processing is performed in the computation order based on the "optimum parallel computation processing", the processing can be ended within the allowable time.

Referring again to <FIG>, in a case where the delay flag information is "ON", that is, in a case where the delay flag information is an inference result indicating that the service cannot be completed within the processing time required for device <NUM> when the inference processing is performed in accordance with the "optimum parallel computation processing" (Yes in S104), NN inference computation management unit <NUM> determines the "delay coping parallel computation processing" to be the forward propagation computation method for specialized NN group <NUM> on the basis of table <NUM> shown in <FIG> (S107). That is, when the delay flag information is "ON", NN inference computation management unit <NUM> determines to execute the processing of specialized NN group <NUM> in the order based on the "delay coping parallel computation processing". The order based on the "delay coping parallel computation processing" is an example of a second schedule. NN inference computation management unit <NUM> outputs the determined forward propagation computation method for specialized NN group <NUM> to NN inference unit <NUM>.

When NN inference unit <NUM> obtains from NN inference computation management unit <NUM> that the forward propagation computation method for specialized NN group <NUM> is the "delay coping parallel computation processing", NN inference unit <NUM> executes parallel forward propagation computation processing in the order indicated by the "delay coping parallel computation processing". For example, NN inference unit <NUM> performs parallel forward propagation computation on specialized NN group <NUM> in the set order (S108). NN inference unit <NUM> inputs the feature value that is an output of feature classifier <NUM> into each NN of specialized NN group <NUM>, performs inference processing according to the schedule based on the "delay coping parallel computation processing", and obtains an inference result (skeleton detection result, object detection result, etc.) that is an output of each NN.

<FIG> is a schematic diagram showing an example of the processing time of common NN <NUM> and specialized NN group <NUM> and the units in charge of computation thereof according to the present example. Specifically, second information 700b in <FIG> shows an example of the processing time and the units in charge of computation in a case where the processing does not end within the allowable time when the processing is performed in the computation order based on the "optimum parallel computation processing". <FIG> shows a case where the allowable time is exceeded when the optimum parallel computation processing is performed, that is, a case where the determination is Yes in step S104.

As shown in <FIG>, for example, the time required for the NMS processing (area 704b) is longer than the time (area 704a) in <FIG>, and hence the inference time of inference processing unit <NUM> has exceeded the allowable time. Depending on the input image, the time required for the NMS processing may be long, as shown in <FIG>.

Therefore, as shown in step S107 in <FIG>, NN inference computation management unit <NUM> determines to perform the inference processing in accordance with the "delay coping parallel computation processing" instead of the "optimum parallel computation processing" because the allowable time may be exceeded when the inference processing is performed in accordance with the "optimum parallel computation processing". In the present example, the "delay coping parallel computation processing" indicates that the computation order is rearranged so that the NMS processing (processing performed in area 704b) can be started with priority. In the following, that the allowable time is exceeded is also referred to as that a delay occurs.

<FIG> is a schematic diagram showing an example of the processing time of common NN <NUM> and specialized NN group <NUM> and the units in charge of computation thereof after the recombination of the computation order according to the present example. Third information 700c in <FIG> shows an example of the processing time and the units in charge of computation thereof when the processing is performed in the computation order based on the "delay coping parallel computation processing" in a case where the processing does not end within the allowable time in the computation order based on the "optimum parallel computation processing". Third information 700c is obtained by changing the processing order from second information 700b shown in <FIG> shows the processing order and the like after the determination is Yes in step S104 and the change is made.

As shown in <FIG>, in the computation order based on the "delay coping parallel computation processing", the NMS processing is started with priority. In the computation order, the timing of starting the NMS processing is accelerated from the computation order based on the "optimum parallel computation processing" so as to accelerate the timing of starting the NMS processing. Thus, even when the time required for the NMS processing is long, the inference time of inference processing unit <NUM> can be prevented from exceeding the allowable time. The computation order (processing order) is an example of a task schedule. The task schedule may also include an order in which the NN models of specialized NN group <NUM> perform memory loading.

As shown in <FIG>, in the "optimum parallel computation processing" and the "delay coping parallel computation processing", the processing timing of the NMS processing is changed, but a change in the content of the processing, the deletion of the processing, or the like is not performed. For this reason, the accuracy of the inference result is substantially the same when the inference processing is performed in accordance with the "optimum parallel computation processing" and when the inference processing is performed in accordance with the "delay coping parallel computation processing".

In the present example, as shown in <FIG>, an example is shown where a layer in which the processing time varies greatly depending on the input image is a layer for performing the NMS processing, but a layer using another algorithm in which the processing time varies greatly depending on the input image may be used.

NN inference unit <NUM> outputs the inference result, output by the computation in step S106 or S108, to task executor <NUM>.

Referring again to <FIG>, task executor <NUM> operates based on the inference result obtained from NN inference unit <NUM>, and the service is provided to a user. That is, task executor <NUM> executes various types of task processing (S109).

As described above, for providing the inference service within the allowable time, device <NUM> can operate while ensuring a certain service capacity by having inference processing unit <NUM> capable of determining the load of the inference processing at an early stage and changing the computation order. When the delay flag information is "OFF", inference processing unit <NUM> can provide the inference service by the optimum parallel computation and can contribute to power saving.

Further, inference processing unit <NUM> estimates the processing time required for the NN inference processing in the plurality of subsequent NNs at an initial stage (preprocessing) from the feature value of the input image. Then, in parallel computation, inference processing unit <NUM> can perform the NN inference processing in a computation order in which the processing can be completed within the required processing time.

In the above description, an example has been described where the computation processing is executed in the order shown in <FIG> when the delay flag information is "ON", but the present disclosure is not limited thereto, and the order of the computation processing may be changed in accordance with the delay flag information. For example, when the delay flag information includes information indicating the degree of delay (for example, the degree of delay: "large", "medium", and "small", an estimated value of the delay time, etc.), a parallel computation item may be set for each degree of delay. That is, the order of computation processing may be set for each degree of delay. Then, in step S107, NN inference computation management unit <NUM> may determine the parallel computation item corresponding to the degree of delay included in the delay flag information to be the forward propagation computation method for specialized NN group <NUM>.

Subsequently, the operation of training apparatus <NUM> will be described with reference to <FIG>. First, the operation of multi-task trainer <NUM> will be described with reference to <FIG> and <FIG>. <FIG> is a flowchart showing the operation of multi-task trainer <NUM> according to the present example. <FIG> is a diagram schematically showing the operation of multi-task trainer <NUM> according to the present example.

As shown in <FIG>, multi-task trainer <NUM> obtains a scene image from scene image DB <NUM> and obtains a correct answer label for each NN from training label DB <NUM> (S201). multi-task trainer <NUM> obtains a scene image, a correct answer label for first task NN <NUM> (first task NN correct answer label) corresponding to the scene image, and a correct answer label for second task NN <NUM> (second task NN correct answer label) corresponding to the scene image. In the present embodiment, multi-task trainer <NUM> obtains a correct answer label for the object detecting NN and a correct answer label for the skeleton detecting NN.

Next, multi-task trainer <NUM> performs forward propagation computation by using common NN <NUM> and specialized NN group <NUM>. For example, multi-task trainer <NUM> performs forward propagation computation by using feature classifier <NUM> of common NN <NUM> and specialized NN group <NUM> (S202).

As shown in <FIG>, multi-task trainer <NUM> obtains a feature value that is an output of common NN <NUM> obtained by inputting the scene image into common NN <NUM>, and further obtains a first inference result that is an output of first task NN <NUM> obtained by inputting the obtained feature value into first task NN <NUM>, and a second inference result that is an output of second task NN <NUM> obtained by inputting the feature value into second task NN <NUM>.

Referring again to <FIG>, next, multi-task trainer <NUM> performs backward propagation computation on feature classifier <NUM> and specialized NN group <NUM>, using the correct answer label for each NN as reference data (teacher data), and updates weights and biases (S203). On the basis of backpropagation, multi-task trainer <NUM> regards an output value of a loss function as an error and executes update processing on the weights (weight data) and biases (bias data) in feature classifier <NUM> of common NN <NUM> and specialized NN group <NUM> by backward propagation computation.

As shown in <FIG>, a loss function in first task NN <NUM> represents how much is the error of the first inference result of first task NN <NUM> with respect to the first task NN correct answer label. A loss function in second task NN <NUM> represents how much is the error of the second inference result of second task NN <NUM> with respect to the second task NN correct answer label.

multi-task trainer <NUM> obtains a first inference result that is an output of first task NN <NUM> on the basis of the scene image and obtains an output value of the loss function on the basis of the first inference result and the first task NN correct answer label. Then, multi-task trainer <NUM> regards the output value as an error and executes update processing on the weights and biases in feature classifier <NUM> of common NN <NUM> and first task NN <NUM> of specialized NN group <NUM> by backward propagation computation.

Next, multi-task trainer <NUM> obtains a second inference result that is an output of second task NN <NUM> on the basis of the scene image and obtains an output value of the loss function on the basis of the second inference result and the second task NN correct answer label. Then, multi-task trainer <NUM> regards the output value as an error and executes update processing on the weights and biases in feature classifier <NUM> of common NN <NUM> and second task NN <NUM> of specialized NN group <NUM> by backward propagation computation. As described above, multi-task trainer <NUM> alternately executes the update processing, for example.

Referring again to <FIG>, multi-task trainer <NUM> determines whether or not the correct answer rate of the NN inference result (or the error rate obtained by comparing the correct answer label with the scene image) satisfies a requirement (S204). The requirement may be, for example, a correct answer rate required for device <NUM>. That is, multi-task trainer <NUM> may perform the determination in step S204 based on whether or not the correct answer rate of the NN inference result is higher than or equal to the correct answer rate required for device <NUM>.

When the correct answer rate of the NN inference result satisfies the requirement (Yes in S204), multi-task trainer <NUM> ends the processing. When the correct answer rate of the NN inference result does not satisfy the requirement (No in step S204), multi-task trainer <NUM> returns to step S201 and continues the processing. That is, when the correct answer rate of the NN inference result does not satisfy the requirement, multi-task trainer <NUM> performs training by using various scene images and correct answer labels associated with the scene images and continues the training processing until the correct answer rate of the NN inference result satisfies the requirement.

Subsequently, the operations of delay flag information measurer <NUM> and delay flag correct answer label generator <NUM> will be described with reference to <FIG> and <FIG>. <FIG> is a flowchart showing the operation of delay flag information measurer <NUM> and delay flag correct answer label generator <NUM> according to the present example. <FIG> is a diagram schematically showing the operations of delay flag information measurer <NUM> and delay flag correct answer label generator <NUM> according to the present example.

As shown in <FIG>, delay flag information measurer <NUM> obtains an arbitrary scene image (S301). Delay flag information measurer <NUM> may obtain, for example, a scene image used for multi-task learning by multi-task trainer <NUM>.

Next, delay flag information measurer <NUM> measures the computation time for the forward propagation computation of common NN <NUM> and specialized NN group <NUM> trained by multi-task trainer <NUM> (S302). In the present embodiment, delay flag information measurer <NUM> measures the computation time of first task NN <NUM> including the NMS processing.

In step S302, delay flag information measurer <NUM> inputs the scene image into common NN <NUM> to obtain a feature value and delay flag information, further inputs the feature value into first task NN <NUM> to process first task NN <NUM>, and measures the time from the inputting of the scene image into common NN <NUM> until the end of the processing of specialized NN group <NUM>. In step S302, delay flag information measurer <NUM> may measure at least the time from the inputting of the feature value into first task NN <NUM> until the end of the processing of specialized NN group <NUM>. <FIG> shows an example in which delay flag information measurer <NUM> measures the time (an example of the inference time) from the inputting of the feature value into first task NN <NUM> until the end of the processing of specialized NN group <NUM>.

In the measurement of the processing time, the same hardware device as device <NUM> may be used. The processing time may be calculated by simulation.

As shown in <FIG>, delay flag information measurer <NUM> obtains a feature value obtained by inputting the scene image into common NN <NUM> and information on the processing time for the plurality of tasks subsequent to the processing of common NN <NUM> (delay flag information that is an output from delay flag classifier <NUM>), and measures the processing time of first task NN <NUM> when the feature value is input. Measuring the processing time is an example of measuring the delay flag information.

Referring again to <FIG>, delay flag information measurer <NUM> determines whether or not the measured processing time has exceeded the threshold (S303). The threshold is set in advance, for example. When the processing time has exceeded the threshold (Yes in S303), delay flag information measurer <NUM> stores the corresponding scene image as a delay flag ON into delay flag information DB 302a (see <FIG>) (S304). When the processing time has not exceeded the threshold (No in S303), delay flag information measurer <NUM> stores the corresponding scene image into delay flag information DB 302a as a delay flag OFF (S305).

When the processing time is being measured in each of first task NN <NUM> and second task NN <NUM>, it is possible that: (i) a delay occurs only in first task NN <NUM> (the allowable time is exceeded); and (ii) a delay occurs only in second task NN <NUM>. In this case, the content of the "parallel computation item" (for example, the computation order after recombination) may be different between (I) and (ii). As described above, when the content of the "parallel computation item" is different, a plurality of types of the delay flag "ON" may exist for the respective contents of the "parallel computation item".

Next, delay flag correct answer label generator <NUM> generates a delay flag correct answer label on the basis of the delay flag information of delay flag information DB 302a. Delay flag correct answer label generator <NUM> labels the scene image with the delay flag "ON" or "OFF" to generate a delay flag correct answer label (S306). It can also be said that delay flag correct answer label generator <NUM> generates a delay flag correct answer label by associating the delay flag information with the scene image. Delay flag correct answer label generator <NUM> stores the generated delay flag correct answer label into delay flag correct answer label DB 303a (see <FIG>).

Subsequently, the operation of delay flag trainer <NUM> will be described with reference to <FIG> and <FIG>. <FIG> is a flowchart showing the operation of delay flag trainer <NUM> according to the present example. <FIG> is a diagram schematically showing the operation of delay flag trainer <NUM> according to the present embodiment.

As shown in <FIG>, delay flag trainer <NUM> obtains a scene image from scene image DB <NUM> and obtains a delay flag correct answer label corresponding to the scene image from delay flag correct answer label DB 303a (S401).

Next, delay flag trainer <NUM> performs forward propagation computation by using common NN <NUM> (S402). Delay flag trainer <NUM> performs forward propagation computation by using, for example, delay flag classifier <NUM> of common NN <NUM>.

As shown in <FIG>, delay flag trainer <NUM> obtains a feature value that is an output of feature classifier <NUM> obtained by inputting a scene image into feature classifier <NUM> of common NN <NUM> and further obtains delay flag information that is an output of delay flag classifier <NUM> obtained by inputting the obtained feature value into delay flag classifier <NUM>.

Referring again to <FIG>, next, delay flag trainer <NUM> performs backward propagation computation on common NN <NUM>, using the delay flag correct answer label as reference data (teacher data), and updates the weight and bias (S403). On the basis of backpropagation, delay flag trainer <NUM> regards the output value of the loss function as an error and executes update processing on the weight (weight data) and the bias (bias data) in delay flag classifier <NUM> of common NN <NUM> by backward propagation computation.

As shown in <FIG>, the loss function in common NN <NUM> represents how much is the error of the inference result (delay flag information) of delay flag classifier <NUM> of common NN <NUM> with respect to the delay flag correct answer label corresponding to the scene image.

Delay flag trainer <NUM> obtains delay flag information that is an output of delay flag classifier <NUM> on the basis of the scene image, and obtains an output value of the loss function on the basis of the delay flag information and the delay flag correct answer label. Then, delay flag trainer <NUM> regards the output value as an error and executes update processing on the weight and bias in delay flag classifier <NUM> of common NN <NUM> by backward propagation computation.

As described above, in steps S402 and S403, common NN <NUM> (for example, delay flag classifier <NUM>) is trained by machine learning, using the scene image as input data, the delay flag information output from delay flag classifier <NUM> of common NN <NUM> as output data, and the time measured by delay flag information measurer <NUM> as reference data.

Note that delay flag trainer <NUM> may regard the output value as an error and executes update processing on the weights and biases in feature classifier <NUM> and delay flag classifier <NUM> by backward propagation computation. That is, delay flag trainer <NUM> may target feature classifier <NUM> for the training processing in addition to delay flag classifier <NUM>. Thus, the weight and bias are updated in feature classifier <NUM> together with those in delay flag classifier <NUM>, so that it is expected that the accuracy of the delay flag information output by delay flag classifier <NUM> is improved.

Referring again to <FIG>, delay flag trainer <NUM> determines whether or not the correct answer rate of the NN inference result (or the error rate obtained by comparing the inference result with the delay flag correct answer label) satisfies a requirement (S404). The requirement may be, for example, a correct answer rate required for device <NUM>. That is, delay flag trainer <NUM> may perform the determination in step S404 based on whether or not the correct answer rate of the NN inference result is higher than or equal to the correct answer rate required for device <NUM>.

When the correct answer rate of the NN inference result satisfies the requirement (Yes in S404), delay flag trainer <NUM> ends the processing. When the correct answer rate of the NN inference result does not satisfy the requirement (No in S404), delay flag trainer <NUM> returns to step S401 and continues the processing. That is, when the correct answer rate of the NN inference result does not satisfy the requirement, delay flag trainer <NUM> performs training by using various scene images and correct answer labels with the scene images and continues the training processing until the correct answer rate of the NN inference result satisfies the requirement.

After the completion of the training processing by delay flag trainer <NUM>, NN data <NUM> becomes able to execute the operation of device <NUM> described above. The generated NN data <NUM> is transmitted to device <NUM>.

Although training apparatus <NUM> according to the present example performs the training processing sequentially, training apparatus <NUM> may perform multi-task learning on feature classifier <NUM> of common NN <NUM> and specialized NN group <NUM> and on delay flag classifier <NUM> of common NN <NUM>, and measure the processing time for the forward propagation computation of specialized NN group <NUM> at that time, to obtain delay flag information. At the time of backward propagation computation, training apparatus <NUM> may use error information with the delay flag information (delay flag correct answer label) as a correct answer to update the weight and bias of delay flag classifier <NUM> of common NN <NUM>, and may use error information of multi-task learning to update the weights and biases of feature classifier <NUM> of common NN <NUM> and specialized NN group <NUM>. As described above, training apparatus <NUM> may be configured to be able to perform multi-task learning on feature classifier <NUM> of common NN <NUM> and specialized NN group <NUM> and on delay flag classifier <NUM> of common NN <NUM>.

Here, the output of the training result evaluation of training apparatus <NUM> according to the present example will be described with reference to <FIG> is a table showing the occurrence rate of the delay flag information according to the present example.

As an example of the output of the training result evaluation of training apparatus <NUM> according to the present example, a configuration may be considered in which the "occurrence rate of delay flag information" shown in <FIG> is output in addition to inference quality information such as the mean average precision (mAP) of specialized NN group <NUM>. The result shown in <FIG> may motivate a developer to change configurations of NNs, hyperparameters, or the like in order to increase the occurrence rate of the delay flag information "OFF". In addition, by displaying the occurrence rate of the inference delay not satisfying the required specification (the occurrence rate corresponding to the delay flag information "NG" in <FIG>) or the information of the corresponding scene image, training apparatus <NUM> can encourage the developer or the like to consider measures to prevent the occurrence of the inference delay at the process stage of the NN training processing.

Although the present disclosure has been described above on the basis of the example, the present disclosure is not limited to the above example.

For example, in the above example, an example has been described where a feature value as an output of feature classifier <NUM> is input into delay flag classifier <NUM> as input data, but the input data is not limited thereto. The input data may be, for example, an input image from camera <NUM>. For example, NN inference unit <NUM> may input an input image from camera <NUM> into delay flag classifier <NUM> to obtain delay flag information that is an output of delay flag classifier <NUM>. In this case, in step S402 of <FIG>, delay flag trainer <NUM> obtains a delay flag that is an output of delay flag classifier <NUM> obtained by inputting the scene image into delay flag classifier <NUM> of common NN <NUM>. Then, in step S403 of <FIG>, delay flag trainer <NUM> performs backward propagation computation on delay flag classifier <NUM> of common NN <NUM>, using the delay flag correct answer label as reference data (teacher data), and updates the weight and bias of delay flag classifier <NUM>. Further, for example, the same input data (input image) may be input into feature classifier <NUM> and delay flag classifier <NUM>.

The input data of delay flag classifier <NUM> may be, for example, intermediate data of the inference of feature classifier <NUM>. For example, NN inference unit <NUM> may cause feature classifier <NUM> to output intermediate data of the inference and input the intermediate data into delay flag classifier <NUM>, thereby obtaining delay flag information that is an output of delay flag classifier <NUM>.

In the above
example,
an example has been described where the processing of the subsequent tasks is processing using NNs, and NN inference computation management unit <NUM> determines the computation order of the processing using NNs, but the present disclosure is not limited thereto. The processing of the subsequent tasks may be processing not using NNs, and NN inference computation management unit <NUM> may determine the computation order of the processing not using NNs.

In the above
example,
an example has been described where the NPU is mounted on SoC <NUM> of device <NUM>, but the NPU may not be mounted. Of CPU <NUM> and the NPU, only CPU <NUM> may be mounted on SoC <NUM>, and for example, each processing described above may be executed by CPU <NUM>.

In the above
example,
an example has been described where NN inference processing unit <NUM> is used in the product field of IoT devices, but the product field is not limited thereto. NN inference processing unit <NUM> is also applicable in product fields different from IoT devices, such as autonomous vehicles, robots, and unmanned aerial vehicles like drones.

The order of the plurality of types of processing described in the above
example
is an example. The order of the plurality of types of processing may be changed, or the plurality of types of processing may be executed in parallel. Some of the plurality of types of processing need not be executed.

Each of the components described in the
example
may be achieved as software or may typically be achieved as a large-scale integrated circuit (LSI) that is an integrated circuit. The components may each be individually integrated into one chip or may be integrated into one chip so as to include some or all of them. The circuit is referred to as an LSI here but is sometimes referred to as an integrated circuit (IC), a system LSI, a super LSI, or an ultra-LSI, depending on the degree of integration. The method for making an integrated circuit is not limited to being achieved by an LSI but may be achieved by a dedicated circuit or a general-purpose processor. After the manufacturing of the LSI, a field programmable gate array (FPGA) that can be programmed, or a reconfigurable processor in which the connections and settings of circuit cells in the LSI can be reconfigured, may be used. Moreover, if an integrated circuit technology replacing the LSI appears due to the advance of semiconductor technology or another technology derived therefrom, the components may naturally be integrated using that technology.

The division of functional blocks in the block diagram is an example, and a plurality of functional blocks may be achieved as one functional block, one functional block may be divided into a plurality of functional blocks, or some functions may be transferred to other functional blocks. Functions of a plurality of function blocks with similar functions may be processed in parallel or in time division by a single piece of hardware or software.

The training apparatus provided in the information processing system may be achieved as a single apparatus or by a plurality of apparatuses. For example, each processing unit of the training apparatus may be achieved by two or more server apparatuses. When the information processing system is achieved by a plurality of server apparatuses, the components included in the information processing system may be distributed to the plurality of server apparatuses in any manner. A method for communication between the plurality of server apparatuses is not particularly limited.

Furthermore, the technique of the present disclosure may be a program for causing a computer to execute characteristic processing in the above information processing method or may be a nontemporary computer-readable recording medium on which the program is recorded. Needless to say, the program can be distributed via a transmission medium such as the Internet. For example, the program and a digital signal including the program may be transmitted via a telecommunications line, a wireless or wired communication line, a network represented by the Internet, data broadcasting, or the like. The program and the digital signal including the program may be executed by other independent computer systems by being recorded on a recording medium and transferred or by being transferred via the network or the like.

In the examples, each of the components may be formed of dedicated hardware or may be achieved by executing a software program suitable for each of the components. Each of the components may be achieved by a program executor, such as a central processing unit (CPU) or a processor, reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

The present disclosure is applicable to an information processing apparatus and the like using inference processing of a plurality of NNs.

Claim 1:
An information processing apparatus comprising:
an obtainer (<NUM>) that obtains sensing data, the sensing data being an input image obtained from a camera (<NUM>);
an inference processing unit (<NUM>) that inputs the sensing data into an inference model to obtain a result of inference, wherein the result of inference is a feature value of the sensing data output from a feature classifier, the feature classifier being a neural network, and information on a processing time for a plurality of subsequent tasks to processing performed by the inference model, wherein the feature classifier outputs a feature value of the sensing data to each of a plurality of specialised neutral networks receiving the input from the feature classifier, the specialised neural networks include a plurality of processing tasks and are in parallel and in charge of subsequent processing; each neural network performs different output on the basis of the feature value, wherein the feature classifier also outputs a delay flag information to a delay flag classifier, the delay flag classifier being a trained neural network that outputs delay flag information indicating the time including the time from the inputting of the feature value into the specialised neural networks until the end of the processing of the plurality of subsequent processing to each of the specialised neural networks;
a determiner (<NUM>) that determines a task schedule for a task processing unit that processes the plurality of subsequent tasks to process the plurality of subsequent tasks based on the delay flag information; and
a controller (<NUM>) that inputs the result of the inference into the task processing unit to process the plurality of subsequent tasks according to the task schedule determined.