Patent Description:
In recent years, opportunities for machine learning to be used for robot control are increasing. For example, a technique has been conventionally disclosed which uses machine learning to cause a robot to automatically perform the same control as when operated by a human (see Patent Literature (PTL <NUM>)).

The document <CIT> discloses an information generation method reflecting the preamble of present claim <NUM>.

Further art is disclosed by the documents <NPL>, <NPL>, and <NPL>.

A known problem for machine learning is that it is difficult to generate learning data appropriately.

The present disclosure provides an information generation method and the like capable of appropriately generating learning data.

An information generation method according to one aspect of the present invention is an information generation method as defined in present claim <NUM>.

It should be noted that these general and specific aspects may be implemented using a system, a device, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or any combination of systems, devices, integrated circuits, computer programs, and recording media.

The information generation method according to the present disclosure is capable of appropriately generating learning data.

The inventor of the present disclosure has found that the following problems arise with the technique, which uses machine learning to cause robots and the like to automatically perform control (autonomously perform operation), described in the "Background Art" section.

In order to create control that models human operations such that it is automatically performed, learning of control patterns using an enormous amount of training data is required. Annotation information, such as correct/incorrect, is added to each of the enormous amount of training data in order to distinguish between a correct control pattern and an incorrect control pattern. In the control of a robot or the like, since only simple annotation information such as moving or not moving is required to be added, it is relatively easy to add such annotation information.

However, in order to automatically determine a movement region (also referred to as a travel region) for automatically moving a mobile object, for example, a movable region where the mobile object is movable and a non-movable region where the mobile object is not movable are required to be added as annotation information to an image in which a predetermined region is captured. Such an addition of annotation information requires a high labor cost because it requires a human to specify movable and non-movable regions in units of pixels while viewing the image. The high labor cost hinders generation of an enormous amount of data for learning. In view of the above, the present disclosure provides, for example, an information generation device which is capable of automatically adding annotation information regarding a self-propelled (autonomously movable) mobile object and generating information for machine learning, and an information generation method performed by the information generation device.

With the information generation method and the like, it is possible to generate training data usable for learning of a learning model for determining the movement region of a self-propelled mobile object while reducing labor cost. Accordingly, it is possible to accelerate the preparation of training data, which has traditionally been a barrier for updating the learning model for determining the movement region, and to frequently update the learning model.

In order to solve the above problem, an information generation method according to one aspect of the present disclosure is an information generation method performed by an information generation device that generates information for a learning model that infers whether a mobile object is movable in a predetermined region. The information generation method includes: obtaining at least (i) first information and (ii) second information when the mobile object moves in a first region, the first information being obtained from a sensor provided in the mobile object, the second information relating to movement of the mobile object; inferring whether the mobile object is movable in the first region according to the second information; and generating fourth information for the learning model, the fourth information associating the first information, the second information, and third information with one another, the third information indicating an inference result which is obtained by the inferring.

In the information generation method, the fourth information is generated which associates the first information, the second information, and the third information with one another. The first information relates to the movement of a mobile object during moving in the first region, the second information is obtained from a sensor when the mobile object moves in the first region, and the third information relates to an inference result of whether the mobile object is movable in the first region. In other words, a result of inferring whether the mobile object is movable in the first region has been added to the fourth information as annotation information. With this, by performing machine learning with the fourth information, a learning model can be constructed which is capable of outputting whether the mobile object is movable in the predetermined region corresponding to the third information from the first information and the second information in the predetermined region. Since the fourth information to which the annotation information has been automatically added is generated in such a manner, it eliminates the need for operations conventionally performed for adding annotation which requires a high labor cost. As a result, learning data can be appropriately generated.

Moreover, for example, it may be that the inferring is performed based on whether a difference between movement data of the mobile object during moving in the first region and a threshold value is within a predetermined range, the movement data being included in the first information obtained when the mobile object moved according to the second information.

With this, the third information can be generated based on the first information obtained when the mobile object moved according to the second information. As a result, since the fourth information which associates the first information, the second information, and the third information with one another can be generated, it eliminates the needs for the operations conventionally performed for adding annotation which requires a high labor cost. Accordingly, learning data can be appropriately generated.

Moreover, it may be that the second information is input by an operator who remotely controls the mobile object.

With this, the fourth information, that is, learning data can be appropriately generated from the first information and the second information obtained when the mobile object moves according to the second information input by the operator who remotely operates the mobile object.

Moreover, the information generation method further includes estimating a difficulty of movement of the mobile object, in which the generating includes generating the fourth information which associates the first information, the second information, fifth information, and the third information with one another, the fifth information indicating an estimation result of the difficulty obtained by the estimating.

With this, the fourth information, which includes fifth information associated with the first information, the second information, and the third information, can be generated. Based on the associated fifth information in the generated fourth information, it is possible to change the aspects and the like of machine learning according to the estimation result of the difficulty of the movement of the mobile object.

Moreover, the generating is performed only when the third information and the fifth information satisfy a predetermined condition.

With this, the fourth information is generated only when the third information and the fifth information satisfy a predetermined condition indicating that the fourth information to be generated is appropriate learning data, and the fourth information is not generated when the fourth information does not satisfy a predetermined condition indicating that the fourth information to be generated is not appropriate learning data. As a result, the fourth information which is inappropriate for use as learning data and whose generation is meaningless is not generated.

Moreover, it may be that the fourth information includes a reliability that is estimated based on the first information.

With this, the fourth information is generated which includes reliability that is estimated based on the first information. By using the fourth information as learning data, information about reliability can be used in machine learning.

Moreover, for example, it may be that the information generation method further includes: obtaining sixth information which divides the first region into a plurality of segments for each of types, in which the fourth information includes a combined image in which information indicating whether the mobile object is movable based on the third information is superimposed on an image of the first region included in the first information for each of the plurality of segments obtained by dividing the image of the first region according to the sixth information.

With this, for each of a plurality of segments obtained by dividing a single image based on the sixth information, it is possible to output a combined image on which information indicating whether the mobile object is movable is superimposed.

Moreover, for example, it may be that the information generation method includes: identifying an other mobile object that is present in the first region based on the first information, in which the inferring includes at least one of: (a) inferring that the mobile object is movable in the first region where the other mobile object is present, when the other mobile object satisfies a first condition; or (b) inferring that the mobile object is not movable in the first region where the other mobile object is present, when the other mobile object satisfies a second condition.

With this, third information can be generated inferring that the mobile object is movable in the first region where the other mobile object is present when the other mobile object that is present in the first region satisfies a first condition, and that the mobile object is not movable in the first region where the other mobile object is present when the other mobile object that is present in the first region satisfies the second condition. Since the fourth information, in which such third information is associated with the first information and the second information, can be generated, it eliminates the need for the operations for adding annotation which have been conventionally performed and requires a high labor cost. As a result, learning data can be appropriately generated.

Moreover, a program according to one aspect of the present disclosure is a program for causing a computer to perform the information generation method.

With this, the same advantageous effects as the information generation method described above can be obtained using the computer.

Moreover, an information generation device according to one aspect of the present disclosure is an information generation device which generates information for a learning model that infers whether a mobile object is movable in a predetermined region. The information generation device includes: an obtainer which obtains at least (i) first information and (ii) second information when the mobile object moves in a first region, the first information being obtained from a sensor provided in the mobile object, the second information relating to the first region; an inference unit which infers whether the mobile object is movable in the first region based on the first information; and a generator which generates fourth information for the learning model, the fourth information associating the first information, the second information, and third information with one another, the third information indicating an inference result which is obtained by the inference unit.

With this, the same advantageous effects as the information generation method described above can be obtained.

Hereinafter, an embodiment will be specifically described with reference to the drawings.

It should be noted that the embodiment described below shows a general or specific example. The numerical values, shapes, materials, structural elements, the arrangement and connection of the structural elements, steps, the order of steps, etc., illustrated in the following embodiment are mere examples, and therefore do not limit the present invention. Moreover, among the structural elements in the following embodiment, those not recited in any of the independent claims defining the most generic part of the inventive concept are described as optional structural elements.

First, a configuration of a learning system according to an embodiment will be described.

<FIG> schematically illustrates a configuration of a learning system according to the present embodiment. <FIG> schematically illustrates a functional configuration of a vehicle according to the embodiment.

As illustrated in <FIG>, learning system <NUM> according to the present embodiment is implemented by mobile object <NUM>, server device <NUM> connected via network <NUM>, and remote control device <NUM>.

Mobile object <NUM> is a device capable of autonomous movement, such as an automobile, robot, drone, bicycle, and wheelchair, and is used for the purpose of delivering packages, for example, while the package is placed therein. Mobile object <NUM> includes a motor or the like that serves as power, a drive unit such as wheels that are driven by the motor, and a functional unit that stores a power source (for example, electric power) for operating the power. Learning system <NUM> is only required to generate training data with annotation added, and a dedicated mobile object for the learning may be used. The learning-specific mobile object does not have to include functions such as placing packages, and does not have to include functions for autonomous movement.

Network <NUM> is a communication network for communicably connecting mobile object <NUM>, server device <NUM>, and remote control device <NUM>. Here, although a communication network, such as the Internet, is used as network <NUM>, network <NUM> is not limited to such an example. Moreover, the connection between mobile object <NUM> and network <NUM>, the connection between server device <NUM> and network <NUM>, and the connection between remote control device <NUM> and network <NUM> may be performed by wireless communication or by wired communication. Due to its nature, mobile object <NUM> may be connected to network <NUM> by wireless communication.

In order to connect mobile object <NUM> and network <NUM> by wired communication, mobile object <NUM> may include a storage device for accumulating various data. At the time when the power source and the like of mobile object <NUM> is charged, wired connection may be established between mobile object <NUM> and network <NUM>, so that the various data accumulated in the storage device may be transmitted to network <NUM>.

As illustrated in <FIG>, mobile object <NUM> includes controller <NUM>, control information receiver <NUM>, sensor information transmitter <NUM>, sensor unit <NUM>, and recognizer <NUM>. Each functional block that forms mobile object <NUM> is implemented using, for example, a processor and memory. Details of each functional block will be described later.

Server device <NUM> is a device for processing information and the like, and is implemented using, for example, a processor and memory. Server device <NUM> may be implemented by an edge computer or by a cloud computer. Moreover, one server device <NUM> may be provided for one mobile object <NUM>, or one server device <NUM> may be provided for a plurality of mobile objects <NUM>.

As illustrated in <FIG>, server device <NUM> includes remote controller <NUM>, control information transmitter <NUM>, sensor information receiver <NUM>, learning data generator <NUM>, learner <NUM>, and storage device <NUM>. The details of each of the functional blocks included in server device <NUM> will be described later.

<FIG> schematically illustrates a configuration of a remote control device according to the embodiment. Remote control device <NUM> is implemented, for example, by a computer. As illustrated in <FIG>, remote control device <NUM> includes, for example, display device <NUM> and steerer <NUM> which is a user interface for input. In addition to steerer <NUM>, as the user interface, a keyboard, a touch panel, a mouse, a dedicated controller, a foot panel, a combination of VR glasses and a controller, a sound collector for voice input, etc. may be used, or a combination of these may be used. Remote control device <NUM> is used to remotely control mobile object <NUM>.

Hereinafter, while continuously referring to <FIG>, the transmission and reception of data between each structural element will be described, and the function of each structural element will be described.

Sensor unit <NUM> is connected to sensors (not illustrated), and obtains the result of sensing the surrounding environment of mobile object <NUM> (sensor information: first information) from the sensors. Examples of the sensors include cameras, LiDAR, radars, sonars, microphones, GPS, vibration sensors, acceleration sensors, gyro sensors, and temperature sensors. Sensor unit <NUM> transmits the obtained sensor information to sensor information transmitter <NUM>, recognizer <NUM>, and controller <NUM>.

Recognizer <NUM> obtains the sensor information transmitted from sensor unit <NUM>, recognizes the surrounding environment of mobile object <NUM>, and transmits the recognition result to sensor information transmitter <NUM> and controller <NUM>. The recognition of the surrounding environment performed by recognizer <NUM> includes generating information (sixth information) that divides the surrounding environment of mobile object <NUM> into a plurality of segments for each type. In addition, recognizer <NUM> recognizes the surrounding environment necessary for the movement of mobile object <NUM>. For example, recognizer <NUM> recognizes obstacles to mobile object <NUM>, other mobile objects, and movable regions where mobile object <NUM> is movable.

Recognizer <NUM> also performs a process of recognizing the difficulty of movement of mobile object <NUM> by estimating the difficulty. Here, the difficulty of movement of mobile object <NUM> is estimated based on, for example, the number of times mobile object <NUM> applies brakes and the number of times mobile object <NUM> changes the direction which are required for mobile object <NUM> to move within the region, and the magnitude and the detected number of vibrations that adversely affect the movement of mobile object <NUM>, and the number of other mobile objects. The greater the number (or the magnitude), the higher the difficulty of movement of mobile object <NUM>. Moreover, examples of cases where the difficulty of movement of mobile object <NUM> is high include roads with narrow roads, such as roads where it is difficult for mobile object <NUM> to pass other mobile objects, roads where on-street parking of cars occurs frequently, roads scattered with objects large enough to hinder movement of mobile object <NUM>, and roads with radio wave environments such as low quality (frequently interrupted) communication with network <NUM>.

At least part of the process of recognizing the surrounding environment may be executed sequentially as soon as the sensor information is input from sensor unit <NUM>, or the sensor information may be buffered (accumulated) in a data storage area (not illustrated) and the at least part of the process may be executed later (for example, after the delivery service using mobile object <NUM> ends). Here, for the recognition of the surrounding environment, a learning model learned in advance by machine learning may be used or the recognition of the surrounding environment may be performed based on a predetermined rule. Note that this learning model may be a learning model generated by learning using training data with the annotation information generated according to the present embodiment.

Sensor information transmitter <NUM> transmits the sensor information transmitted from sensor unit <NUM> and the recognition result transmitted from recognizer <NUM> to sensor information receiver <NUM> of server device <NUM>. It should be noted that transmission and reception of information between mobile object <NUM> and server device <NUM>, including the communication here, may undergo alteration of information such as compression for the purpose of data reduction and encryption for the purpose of maintaining confidentiality. In this case, restoration of altered information such as decompression of compressed information and decrypting of encrypted information may be performed by the functional block that received the altered information. In addition, various types of information may be transmitted and received only when the network bandwidth (communication capacity) between mobile object <NUM> and server device <NUM> is greater than or equal to a predetermined value. Additional information, such as an ID and timestamp, may also be added to match the information transmitted and received. In such a manner, information may be transmitted and received between mobile object <NUM> and server device <NUM> in any forms.

Sensor information receiver <NUM> transmits, to remote controller <NUM> and learning data generator <NUM>, the sensor information transmitted from sensor information transmitter <NUM> and the recognition result.

Remote controller <NUM> determines the moving direction of mobile object <NUM> as necessary based on the sensor information and the recognition result transmitted from sensor information receiver <NUM>, and performs remote control so that mobile object <NUM> moves in the determined moving direction. Specifically, control information (second information) relating to movement of mobile object <NUM> for remote control is generated, and is transmitted to control information transmitter <NUM>. The control information is also transmitted to learning data generator <NUM>.

The determination of the moving direction of mobile object <NUM> according to the present embodiment, that is, the remote control of mobile object <NUM> is performed by an operator. Accordingly, remote control device <NUM> illustrated in <FIG> is used by an operator.

For example, an image of the surrounding environment of the mobile object included in the sensor information is transmitted from remote controller <NUM> to remote control device <NUM> and displayed on display device <NUM>. A display device, such as a smartphone or VR/AR glasses, may be used in place of display device <NUM>.

The operator is able to check the sensor information (the image of the surrounding environment in this case) displayed on display device <NUM>. When the operator determines that remote control is necessary, the operator is able to control the moving direction of mobile object <NUM>. The moving direction is determined by the operator determining the region where mobile object <NUM> is movable. After that, the operator makes an input through the user interface such as steerer <NUM> so that mobile object <NUM> moves in the determined moving direction. An input signal is transmitted from remote control device <NUM> to remote controller <NUM> based on the input. Remote controller <NUM> generates control information according to the input signal. The control information generated by remote controller <NUM> includes, for example, information relating to the angle of the tires as the driving unit of mobile object <NUM>, information relating to accelerator position and opening/closing timing, and information relating to brake strength and braking timing.

In place of steerer <NUM>, a dedicated controller, a smartphone, a keyboard, a mouse and the like may be used.

The determination of the moving direction of mobile object <NUM> and the remote control of mobile object <NUM> may be performed automatically or semi-automatically. For example, the moving direction may be automatically determined from among the movable region of mobile object <NUM> based on the recognition result, and control information may be generated so that mobile object <NUM> moves in the determined moving direction. In this case, operator intervention is not essential, and remote control device <NUM> does not have to be provided. Moreover, by incorporating this automatic remote control configuration into mobile object <NUM>, it is possible to configure learning system <NUM> without providing remote controller <NUM> in server device <NUM>.

On the other hand, the determination of the moving direction of mobile object <NUM> can also be performed manually. For example, an administrator or a maintenance company of mobile object <NUM> may move mobile object <NUM> by directly urging mobile object <NUM> by, for example, pushing mobile object <NUM> by hand. Moreover, the administrator or maintenance company of mobile object <NUM> may move mobile object <NUM> by remotely controlling mobile object <NUM> from the vicinity of mobile object <NUM> using radio control or infrared control. Furthermore, the manager or maintenance company of mobile object <NUM> may move mobile object <NUM> by operating a device such as a steerer provided in mobile object <NUM> while riding in mobile object <NUM>.

<FIG> illustrates an example of remote control semi-automatically performed according to the embodiment. As illustrated in <FIG>, it may be that candidates of moving direction (thick-line arrow and thick-dashed line in <FIG>) are automatically generated based on the sensor information and recognition result and displayed, and the moving direction is determined by causing an operator to select an appropriate one from the candidates. This facilitates simplified configuration of the user interface.

Control information transmitter <NUM> transmits the control information transmitted from remote controller <NUM> to control information receiver <NUM> in mobile object <NUM>. Control information receiver <NUM> transmits, to controller <NUM>, the control information transmitted from control information transmitter <NUM>. Controller <NUM> then moves mobile object <NUM> by driving a driving unit and the like according to the control information transmitted from control information receiver <NUM>.

Here, the control information generated by the intervention of a person or the functions of server device <NUM> with relatively high performance as described above is information which allows mobile object <NUM> to move appropriately. Moreover, in generating the control information, the sensor information transmitted to remote controller <NUM> includes information directly obtained by the mobile object from the surrounding environment in order to determine the moving direction of mobile object <NUM>. In other words, by using sensor information as input data for training data and adding control information to the input data as annotation information, it is possible to automatically generate training data with the annotation information added. That is, the generation of information for machine learning, which will be described below, is performed using at least the sensor information and control information.

Learning data generator <NUM> obtains the sensor information transmitted from sensor information receiver <NUM> and the control information input from remote controller <NUM>. The function of learning data generator <NUM> relating to the obtainment of information is an example of an element which realizes the function of an obtainer.

Learning data generator <NUM> also obtains the recognition result transmitted from sensor information receiver <NUM>. Training data for machine learning (fourth information) is generated by adding, to the sensor information as annotation information, the region information (region where mobile object <NUM> is movable) on the sensor information corresponding to the region where mobile object <NUM> actually moved and information indicating, for example, the difficulty of the movement in the surrounding environment. The function of learning data generator <NUM> relating to the generation of training data is an example of an element which realizes the function of a generator.

Learning data generator <NUM> further generates an inference result (third information) by inferring whether the movement based on the control information was appropriate, that is, whether mobile object <NUM> was actually movable in the region, based on the sensor information obtained by sensor unit <NUM> when mobile object <NUM> moved based on the control information. The function of learning data generator <NUM> relating to the generation of the inference result is an example of an element which realizes the function of an inference unit. Learning data generator <NUM> is an example of an information generation device which includes the functions of the obtainer, the generator, and the inference unit.

For example, learning data generator <NUM> performs the inference based on the difficulty of the movement of mobile object <NUM> that is estimated based on, for example, the number of times mobile object <NUM> applies brakes and the number of times mobile object <NUM> changes the direction which are required for mobile object <NUM> to move according to the control information, the magnitude and the detected number of vibrations that adversely affect the movement of mobile object <NUM>, and the number of other mobile objects. For example, when the number of times mobile object <NUM> actually applied brakes is greater than a threshold value that is set based on the movement distance and the like of mobile object <NUM>, such as one, two, three, five, or ten times, an inference result is generated which indicates that mobile object <NUM> was not movable in the region, and when the number of times mobile object <NUM> actually applied brakes is less than the threshold value, an inference result is generated which indicates that mobile object <NUM> was movable in the region.

Similarly, for example, when the difference between each of the number of times mobile object <NUM> changes the direction, the magnitude and the detected number of vibrations that adversely affect the movement of mobile object <NUM>, and the number of other mobile objects and a threshold value that is set based on the movement distance of mobile object <NUM> is within a predetermined range, an inference result is generated which indicates that mobile object <NUM> was movable in the region, and when the difference exceeds the predetermined range, an inference result is generated which indicates that mobile object <NUM> was not movable in the region.

The inference result thus generated is used to verify whether mobile object <NUM> was actually movable in the region. For example, when mobile object <NUM> was not movable according to the control information, information indicating that mobile object <NUM> is not movable is added to the control information as annotation information. When only annotation information indicating that mobile object was movable is desired to be used for machine learning, training data with annotation information relating to the above control information should be excluded. In addition, when the learning model is a model for distinguishing (clustering) whether the region is a movable region or a non-movable region, both the annotation information indicating that mobile object <NUM> is movable and the annotation information indicating that mobile object <NUM> is not movable may be used. In such a manner, the inference results generated above are used to determine the type of annotation information.

Moreover, the information (fifth information) relating to the difficulty of movement described above may also be used. Specifically, even if the combination of the sensor information, the control information, and the inference result satisfies the threshold criteria and indicate that mobile object <NUM> is movable, when the movement of mobile object <NUM> is relatively difficult, the inference result may indicate, due to other factors, that mobile object <NUM> is not movable. In other words, the combination of sensor information, control information, and inference result with high difficulty of movement has poor reliability. Accordingly, such combinations of sensor information, control information, and inference results may not be used for machine learning. Alternatively, such combinations of sensor information, control information, and inference results may be considered to have low reliability, and may be differentiated from the reliability of other combinations of sensor information, control information, and inference results and used for machine learning. Moreover, the difficulty of movement may be used for clustering movable regions. For example, it may be that the region where mobile object <NUM> is movable and the difficulty of movement is high is classified as a "region that requires careful movement", and the region where mobile object <NUM> is movable and the difficulty of movement is low is classified as a "region where mobile object <NUM> is easily movable", and they are used for machine learning.

On the other hand, when it is possible to accurately determine whether mobile object <NUM> is movable in the region with a combination of sensor information, control information, and inference result in which the difficulty of the movement is high, the region where mobile object <NUM> is movable can be expanded. In other words, in a situation where the advantage of expanding the movable region is emphasized, for example, when the movable region is small or alternative movable region is small, a combination of sensor information, control information, and inference result with high difficulty of movement may be preferentially used for machine learning. A sufficient amount of training data necessary for learning needs to be generated in order to accurately determine whether mobile object <NUM> is movable in a region with a combination of sensor information, control information, and an inference result with high difficulty of movement. Therefore, the advantageous effects of the information generation method according to the present embodiment can be significant.

It should be noted that the term "preferentially used for machine learning" means that after all machine learning using training data with combinations of sensor information, control information, and inference results with high difficulty of movement is completed, machine learning for training data with other combinations is performed. At this time, machine learning for other combinations of training data may be omitted. Other combinations of training data are training data for regions where the difficulty of movement is relatively low. In other words, in such regions, rule-based autonomous movement based on sensor information may be sufficient without relying on machine learning, and thus, machine learning for other combinations of training data can be omitted.

Alternatively, as a method for preferentially using a combination of sensor information, control information, and inference result with high difficulty of movement for machine learning, learning may be performed using training data with combinations of low difficulty, and then learning using training data with combinations of high difficulty may be performed. Depending on the conditions of machine learning, compared to when preferentially using in the time series as above, the influence of data learned later may be high, and the above method is effective in such cases.

Moreover, after the completion of learning with all training data, further learning may be performed with training data with high difficulty. Moreover, bias may be applied so that training data with higher difficulty influences learning results more. For example, the training data with low difficulty may be reduced so that the training data with high difficulty is greater in the ratio of the number of training data with high difficulty to the number of training data with low difficulty.

To summarize the above description, in the present embodiment, based on the combination of sensor information and control information, training data to which annotation information for machine learning has been added is generated when both the inference result indicating whether mobile object <NUM> is movable and the difficulty of the movement of mobile object <NUM> satisfy a predetermined condition. Learning data generator <NUM> transmits the training data thus generated to storage device <NUM> for storage.

<FIG> is a first diagram illustrating an example of training data according to the embodiment. <FIG> is a second diagram illustrating an example of training data according to the embodiment. As illustrated in <FIG>, annotation information indicating that the region where mobile object <NUM> was movable is a movable region (rough dotted hatching in <FIG>) is added to the training data thus generated. In this way, the training data output in the present embodiment may divide the image of the region where mobile object <NUM> moves, which is included in the sensor information, into a plurality of segments, and may include, for each of the segments, a combined image in which information, indicating whether mobile object <NUM> is movable based on the inference result, is superimposed on the image. Accordingly, for example, information (sixth information) may be obtained which divides the region where mobile object <NUM> moves into a plurality of segments for each type. The information may be generated separately by using, for example, recognition results.

When determining the region where mobile object <NUM> is movable, a region having a predetermined width along the moving direction may be used by superimposing the history of passing the same region several times. Alternatively, as illustrated in <FIG>, a region may be used which is obtained by combining the own width of mobile object <NUM> to the region where mobile object <NUM> passes once. The association between the region where mobile object <NUM> moved and the pixels on the image may be determined based on a combination of control information, sensor information, and additional information such as IDs and timestamps for matching information.

Moreover, a region where a mobile object other than mobile object <NUM> (another mobile object) that moves in the surrounding environment may be used for annotation information. Specifically, a mobile object that is sufficiently larger and faster than mobile object <NUM> may be identified as a dangerous mobile object, and the region where such a dangerous mobile object moves may be determined as a non-movable region. An example of such a dangerous mobile object for mobile object <NUM> for package transportation is an automobile. Alternatively, an automobile and the like may be directly identified by a technique such as pattern matching.

In addition, a mobile object having a size equivalent to the size of mobile object <NUM> and not moving so fast may be identified as a trackable mobile object, and the region where such a trackable mobile object moves may be determined as a movable region. Examples of such a trackable mobile object for mobile object <NUM> for package transportation include bicycles and wheelchairs. Alternatively, bicycles, wheelchairs, and the like may be directly identified by a technique such as pattern matching.

In addition, annotation information may be added not only to part of the sensor information (a certain region in the image of one frame, that is, some pixels, etc.), but also to the entire sensor information (the whole image of one frame, etc.). With this, for example, it is possible to construct a learning model that provides a determination result indicating whether the mobile object is movable or not or whether it is easy or difficult for the mobile object to move when a single image is input as sensor information.

Learner <NUM> causes a learning model to perform learning through machine learning using training data to which the annotation information has been added and which is stored in storage device <NUM>. After the completion of the learning, learner <NUM> outputs the learning model after the learning to the recognizer.

Next, learning system <NUM> described above will be described with reference to <FIG> is a flowchart illustrating the processes performed by the learning system according to the embodiment.

As illustrated in <FIG>, learning data generator <NUM> first obtains sensor information and control information (obtaining step S101). The sensor information and the control information are obtained from the mobile object via sensor information obtainer <NUM>.

Next, learning data generator <NUM> obtains sensor information and recognition result of mobile object <NUM> obtained when mobile object <NUM> moves according to the control information, and infers, based on the sensor information and recognition result, whether the region where mobile object <NUM> moves according to the control information is a movable region where mobile object <NUM> is movable (inferring step S102).

Next, learning data generator <NUM> generates training data with annotation information added, based on the sensor information, the control information, and the inference result obtained in inferring step S102 (generating step S103).

Since appropriate annotation information has been added to the training data thus generated, the training data can be used as it is as training data for machine learning. In addition, since the training data is generated automatically, the labor cost is reduced, and a large amount of training data can be generated at once by using a plurality of mobile objects <NUM> and the like.

Although the information generation method and the like according to one or more aspects has been described above based on the embodiment, the present disclosure is not limited to the embodiment. Various modifications of the embodiment as well as embodiments resulting from arbitrary combinations of the structural elements of the embodiment that may be conceived by those skilled in the art are intended to be included within the scope of one or more aspects as long as these do not depart from the essence of the present disclosure.

For example, by providing all of the functions of the server device according to the embodiment in a mobile object, information for a learning model which infers whether the mobile object is movable in a predetermined region in a stand-alone manner can be efficiently generated.

Each of the structural elements in the above-described embodiment may be configured in the form of an exclusive hardware product, or may be realized by executing a software program suitable for each of the structural elements. Each of the structural elements may be realized by means of a program executing unit, such as a central processing unit (CPU) and a processor, reading and executing the software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software which realizes the information generation device and the like according to the above-described embodiment is a program as described below.

In other words, the program is a program that causes a computer to execute the information generation method.

Claim 1:
An information generation method performed by an information generation device that generates learning data for a learning model that infers whether a mobile object (<NUM>) is movable in a predetermined region, the information generation method comprising:
obtaining at least (i) first information and (ii) second information when the mobile object (<NUM>) moves in a first region, the first information being obtained from a sensor provided in the mobile object (<NUM>), the second information relating to movement of the mobile object (<NUM>);
inferring whether the mobile object (<NUM>) is movable in the first region according to the second information; and
generating fourth information for the learning model, the fourth information associating the first information, the second information, and third information with one another, the third information indicating an inference result which is obtained in the inferring.
characterized in that the method further comprises:
estimating a difficulty of movement of the mobile object (<NUM>),
wherein the generating includes generating the fourth information which associates the first information, the second information, fifth information, and the third information with one another, the fifth information indicating an estimation result of the difficulty obtained in the estimating, and
wherein the generating is performed only when the third information and the fifth information satisfy a predetermined condition.