Control apparatus, object detection system, object detection method and program

A control apparatus capable of efficiently detecting a target object even when the target object is shielded by other objects is provided. An object recognition unit 114 recognizes a target object 80 present in a 3D environment 4 by using measurement data acquired from a sensor 12. An information generation unit 116 generates 3D environmental information by integrating a plurality of measurement data. A position determination unit 120 determines an optimal position of the sensor 12 for performing the next measurement. A sensor control unit 140 moves the sensor 12 to the determined optimal position. The position determination unit 120 determines, by using the 3D environmental information, a position of the sensor 12 where the sensor 12 can take an image in which a size of an area shielded by at least one first object is larger as the optimal position.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority from Japanese patent application No. 2018-079235, filed on Apr. 17, 2018, the disclosure of which is incorporated herein in its entirety by reference.

BACKGROUND

The present disclosure relates to a control apparatus, an object detection system, an object detection method and a program. In particular, the present disclosure relates to a control apparatus, an object detection system, an object detection method and a program for detecting an object by controlling a sensor.

There is a technique for detecting a target object, which is an object to be detected, by operating a sensor such as a range sensor. In such techniques, it is necessary to consider that the target object may be shielded by other objects (obstacles). In relation to this technique, Japanese Unexamined Patent Application Publication No. 2015-190818 discloses a work support system for improving efficiency of work for completion of the whole work. The work support system disclosed in Japanese Unexamined Patent Application Publication No. 2015-190818 includes a measurement apparatus that measures (i.e., obtains) three-dimensional (3D) shape data of an object to be measured, a transfer apparatus that moves at least one of the objects to be measured and the measurement apparatus and thereby changes a measurement position where the measurement apparatus measures the object to be measured, and a work support apparatus that controls the transfer apparatus. A candidate position setting unit sets, on a surface of the object to be measured, candidate measurement positions for the entire area of a measurement target range designated as a measurement target range. An available surface calculation unit calculates, for each of the set measurement positions, the number of measurable surfaces of the object to be measured or an area (i.e., a size) of the measurable surfaces. A ranking determination unit determines, for each measurement direction, a priority order of measurement according to the calculated number or the area (i.e., the size) of the surfaces. A measurement control unit instructs the transfer apparatus to perform measurement in each measurement direction according to the determined priority order.

SUMMARY

The present inventors have found the following problem. Depending on an operating environment of a sensor, a plurality of target objects, which are objects to be detected, may exist in the operating environment. Further, objects other than the target object(s) may exist in the operating environment. In such cases, there is a possibility that the target object may be shielded by other objects. Therefore, there is a possibility that even if the sensor is moved at random without taking the positional relation between the target object and the other object(s) into consideration, it may take an enormous amount of time to detect the target object. That is, there is a possibility that even if the sensor is moved to a certain position and performs measurement at that position, the target object may not be measured because the area shielded by the other object(s) is large. Therefore, there is a possibility that the time that is taken to move the sensor to that position and perform the measurement may be wasted. In the technique disclosed in Japanese Unexamined Patent Application Publication No. 2015-190818, objects other than the object to be measured are not taken into consideration. Therefore, in the technique disclosed in Japanese Unexamined Patent Application Publication No. 2015-190818, there is a possibility that when the object to be measured is shielded by an obstacle, the object to be measured cannot be efficiently detected.

The present disclosure provides a control apparatus, an object detection system, an object detection method and a program capable of efficiently detecting a target object even when the target object is shielded by other objects.

A first exemplary aspect is a control apparatus configured to detect a target object to be detected by controlling a sensor configured to measure surroundings in a three-dimensional (3D) environment, including: an object recognition unit configured to recognize the target object present in the 3D environment by using measurement data acquired from the sensor; an information generation unit configured to generate 3D environmental information indicating each point on an object present in the 3D environment by integrating a plurality of measurement data; a position determination unit configured to determine an optimal position of the sensor for performing next measurement; and a sensor control unit configured to move the sensor to the determined optimal position, in which the position determination unit determines, by using the 3D environmental information, a position of the sensor where the sensor can take an image in which a size of an area shielded by at least one first object is larger as the optimal position, the at least one first object being the target object recognized by the object recognition unit.

Further, another exemplary aspect is an object detection system including: a sensor configured to measure surroundings in a 3D environment; and a control apparatus configured to detect a target object to be detected by controlling the sensor, in which the control apparatus includes: an object recognition unit configured to recognize the target object present in the 3D environment by using measurement data acquired from the sensor; an information generation unit configured to generate 3D environmental information indicating each point on an object present in the 3D environment by integrating a plurality of measurement data; a position determination unit configured to determine an optimal position of the sensor for performing next measurement; and a sensor control unit configured to move the sensor to the determined optimal position, and in which the position determination unit determines, by using the 3D environmental information, a position of the sensor where the sensor can take an image in which a size of an area shielded by at least one first object is larger as the optimal position, the at least one first object being the target object recognized by the object recognition unit.

Further, another exemplary aspect is an object detection method for detecting a target object to be detected by controlling a sensor configured to measure surroundings in a 3D environment, including: recognizing the target object present in the 3D environment by using measurement data acquired from the sensor; generating 3D environmental information indicating each point on an object present in the 3D environment by integrating a plurality of measurement data; determining, by using the 3D environmental information, a position of the sensor where the sensor can take an image in which a size of an area shielded by at least one first object is larger as an optimal position of the sensor for performing next measurement, the at least one first object being the recognized target object; and moving the sensor to the determined optimal position.

Further, another exemplary aspect is a program for performing an object detection method in which a target object to be detected is detect by controlling a sensor configured to measure surroundings in a 3D environment, the program being adapted to cause a computer to perform: recognizing the target object present in the 3D environment by using measurement data acquired from the sensor; generating 3D environmental information indicating each point on an object present in the 3D environment by integrating a plurality of measurement data; determining, by using the 3D environmental information, a position of the sensor where the sensor can take an image in which a size of an area shielded by at least one first object is larger as an optimal position of the sensor for performing next measurement, the at least one first object being the recognized target object; and moving the sensor to the determined optimal position.

In the present disclosure, when a target object is detected by using the sensor, the sensor can be moved to a position where the sensor can measure an area that has become a blind sport (i.e., that cannot be viewed) due to the first object more appropriately. Therefore, it is possible to reduce the number of movements of the sensor and the time required therefor when the target object is detected by using the sensor. Consequently, according to the present disclosure, it is possible to efficiently detect a target object even when the target object is shielded by other objects.

Further, the position determination unit preferably calculates, as a candidate for the optimal position, a position of the sensor where the sensor can measure a placement available area according to a position and a shape of a storage object in which the target object can be placed, and selects the optimal position from the candidate, the placement available area being an area where the target object can be placed.

By calculating viewpoint candidates as described above, it is possible to exclude, from the optimal position, a viewpoint position that cannot contribute to the detection of the target object depending on the position and the shape of the storage object. Therefore, the present disclosure makes it possible to efficiently detect a target object.

Further, the position determination unit preferably determines whether or not, among candidate positions, there is an unmeasured position from which the sensor can measure the area shielded by the first object but has not performed measurement yet, and when it is determined that there is the unmeasured position, performs a process for determining the optimal position.

The state where there is no unmeasured position means a state where the target object probably cannot be detected even when measurement is further performed. Therefore, by determining whether or not there is an unmeasured position as in the case of the present disclosure, it is possible to terminate the process for determining the optimal position when there is no unmeasured position. Therefore, the present disclosure can finish the process for detecting the target object without performing a wasteful process.

Further, the control apparatus preferably further includes a removal determination unit configured to, when it is determined that there is no unmeasured position, determine the first object to be removed so that the area shielded by the first object can be measured.

By being configured as described above, the present disclosure can make it possible to measure an unmeasured area, which has not been able to be measured, by removing the first object to be removed in the subsequent process, and thereby efficiently detect the unmeasured area.

Further, the control apparatus preferably further includes an arm control unit configured to control an arm so that the determined first object is removed.

By being configured as described above, the present disclosure can make it possible to automatically remove the first object to be removed. Therefore, it is possible to efficiently measure the unmeasured area.

Further, the position determination unit preferably determines whether or not a search of the placement available area, which is the area where the target object can be placed, has been completed by using the 3D environmental information, and when it is determined that the search has not been completed, performs a process for determining the optimal position.

By being configured as described above, the present disclosure can prevent, when it is determined that the search of the storage object has been completed, the process for determining the optimal position from being performed. Therefore, the present disclosure can prevent an unnecessary process from being performed. Therefore, the present disclosure can prevent or reduce an increase in the time taken to detect an object.

According to the present disclosure, it is possible to provide a control apparatus, an object detection system, an object detection method and a program capable of efficiently detecting a target object even when the target object is shielded by other objects.

DESCRIPTION OF EMBODIMENTS

First Embodiment

Embodiments according to the present disclosure are described hereinafter with reference to the drawings. Note that the same symbols are assigned to substantially the same components.

FIG. 1shows an object detection system1according to a first embodiment. Further,FIG. 2is a block diagram showing a hardware configuration of the object detection system1according to the first embodiment. The object detection system1according to the first embodiment includes an object detection apparatus10and a control apparatus100. The object detection apparatus10includes a sensor12and a driving unit14that drives the sensor12.

The control apparatus100is, for example, a computer. The control apparatus100is connected to the object detection apparatus10through a wired or wireless communication link2so that they can communicate with each other. Therefore, the control apparatus100is connected to the sensor12and the driving unit14so that they can communicate with each other.

Note that inFIG. 1, the control apparatus100and the object detection apparatus10are shown as physically separate apparatuses. However, the control apparatus100may be incorporated into the object detection apparatus10. Further, at least one component of the control apparatus100(which will be described later) may be incorporated into the object detection apparatus10. In such a case, the object detection apparatus10also has functions as a computer.

The object detection apparatus10moves in a three-dimensional (3D) environment4. The object detection apparatus10can autonomously move in the 3D environment4. Note that the 3D environment4may be expressed by an orthogonal coordinate system or may be expressed by a polar coordinate system. In the following descriptions, an example in which the 3D environment4is expressed by an (X, Y, Z)-orthogonal coordinate system is shown.

The sensor12is a 3D sensor capable of measuring a distance to an object, such as a depth sensor, a range sensor (or a distance censor), or a 3D camera (a stereo camera). The sensor12is, for example, a lidar (LIDAR: Light Detection and Ranging) or the like. The object detection apparatus10(the sensor12) has five degrees of freedom by the driving unit14as described below.

As indicated by an arrow A, the driving unit14moves the object detection apparatus10(the sensor12) in an X-axis direction of the 3D environment4. Further, as indicated by an arrow B, the driving unit14moves the object detection apparatus10(the sensor12) in a Y-axis direction of the 3D environment4. Further, as indicted by an arrow C, the driving unit14moves the sensor12in a Z-axis direction of the 3D environment4(i.e., in a vertical direction). Further, as indicted by an arrow D, the driving unit14rotates (turns) the sensor12in parallel to an XY-plane of the 3D environment4(i.e., in a horizontal direction). Further, as indicted by an arrow E, the driving unit14rotates (swings) the sensor12in an up/down direction of the 3D environment4. That is, as indicated by the arrows A, B and C, the sensor12is moved by the driving unit14so that its 3D position coordinates in the 3D environment4changes. Further, as indicated by the arrows D and E, the sensor12is moved by the driving unit14so that its posture (its orientation) in the 3D environment4changes. In the following descriptions, the “movement” of the sensor12includes a change in the 3D position coordinates and a change in the posture. Further, the “position” of the sensor12includes its 3D position coordinates and its posture.

The sensor12measures surroundings of the object detection apparatus10. The sensor12acquires an image(s) of an object(s) present in the measured surroundings. Further, the sensor12measures a distance to each point on the object observed from the sensor12(the object detection apparatus10). Then, the sensor12generates distance data indicating the measured distance. That is, the distance data corresponds to the measurement data generated by the sensor12. The sensor12generates distance image data indicating a distance image (a point cloud) as the distance data. That is, the distance data represents a group of points (hereinafter also referred to as a point group) on the surface of each object present around the sensor12(the object detection apparatus10) in three dimensions. The sensor12scans its surroundings with laser light (i.e., emits laser light to its surroundings), receives reflected light reflected on an object, and calculates a distance to the object from, for example, a difference between a transmission time of the laser light and a reception time of the reflected light. Then, the object detection apparatus10(the sensor12) calculates 3D coordinates (X, Y, Z) of a point at which the laser light is reflected based on 3D position coordinates of the sensor12in the 3D environment4, an emitting direction of the laser light, and the distance to the object. In this way, the object detection apparatus10(the sensor12) measures a position of each object in the 3D environment4. Note that, in the following descriptions, the term “image” also means “image data representing an image” as data to be processed in information processing.

A plurality of target objects80, which are objects to be detected by the object detection apparatus10, are disposed in the 3D environment4. Further, at least one storage object90is provided in the 3D environment4. The storage object90includes at least one shelf board92and a wall surface(s)94. The storage object90can house a plurality of target objects80. In the example shown inFIG. 1, target objects80A to80E are disposed (i.e., placed) in the storage object90.

The control apparatus100includes, as a main hardware configuration, a CPU (Central Processing Unit)102, a ROM (Read Only Memory)104, a RAM (Random Access Memory)106, and an interface unit108(IF; Interface). The CPU102, the ROM104, the RAM106, and the interface unit108are connected to each other through a data bus or the like.

The CPU102has functions as an arithmetic apparatus that performs control processes, arithmetic processes, etc. The ROM104has a function of storing a control program(s), an arithmetic program(s), etc. that are executed by the CPU102. The RAM106has a function of temporarily storing processing data and the like. The interface unit108externally receives/outputs signals wirelessly or through a wire. Further, the interface unit108receives a data input operation performed by a user and displays information for the user.

FIG. 3is a functional block diagram showing a configuration of the control apparatus100according to the first embodiment. The control apparatus100includes an information storage unit112, an object recognition unit114, an information generation unit116, a position determination unit120, and a sensor control unit140(hereinafter, also referred to as “each component”). Further, the position determination unit120includes a search determination unit122and an optimal viewpoint selection unit124. Each component can be implemented by, for example, having the CPU102execute a program(s) stored in the ROM104. Further, necessary programs may be stored in an arbitrary nonvolatile recording medium in advance, and may be installed as required. Note that the implementation of each component is not limited to the above-described software implementation and may be implemented by hardware such as some type of circuit devices. Further, at least one of the aforementioned components may be implemented by physically-separate individual hardware. This also applies to a later-described second embodiment.

The information storage unit112stores 3D environmental information, storage object information, target object information, and sensor information. Further, the information storage unit112may store recognition results obtained by the object recognition unit114(which will be described later). The “3D environmental information” is information indicating 3D coordinate data of each point (of a point group) on each object present in the 3D environment4. When there are a plurality of 3D environments4, the information storage unit112may store a plurality of 3D environmental information pieces. For example, 3D environmental information may be generated and stored for each environment such as an environment of a house, an environment of a shop, an environment of a tenant (i.e., a rental room), and an environment of a floor.

The 3D environmental information indicates, for example, whether or not there is any object at a given point represented by 3D coordinates (X, Y, Z). Therefore, the control apparatus100and the object detection apparatus10can recognize a shape of an object by detecting presence of some object in consecutive 3D coordinates in the 3D environmental information. The 3D environmental information is acquired by, for example, having the sensor12scan the entire space of the 3D environment4and calculate 3D coordinates of each point on each object. That is, the 3D environmental information can be generated by using the sensor12. Note that the 3D environmental information corresponds to information (or data) obtained by integrating (or combining) 3D measurement data (distance image data) generated by measurement performed by the sensor12. That is, the 3D environmental information can be generated by integrating (or combining) 3D measurement data obtained by having the sensor12perform measurement from one viewpoint position with 3D measurement data obtained by having the sensor12perform measurement from another viewpoint position. In other words, the 3D environmental information is successively updated as 3D measurement data is successively generated at a plurality of viewpoint positions by the sensor12.

The “storage object information” is information related to the storage object90. The storage object information indicates a 3D shape and a size (dimensions) of the storage object90. For example, the storage object information may be CAD (computer-aided design) data of the object. Further, the storage object information indicates a position (3D coordinates) in the 3D environment4. Therefore, by integrating (or combining) the storage object information into the 3D environmental information, the control apparatus100can recognize where the storage object90is located in the 3D environment4by using the 3D environmental information. In other words, the control apparatus100can recognize which coordinates (X, Y, Z) in the 3D environment4the shelf board(s)92and the wall surface(s)94are located at by using the 3D environmental information. Further, the storage object information may also include information indicating an area (i.e., a size) of the shelf board92, an edge(s) of the shelf board92, an opening(s) in the shelf board92, etc.

The “target object information” is information necessary to detect the target object to be detected by the object detection apparatus10. Each target object80is registered in the object detection system1by storing its target object information in the information storage unit112. The target object information may include identification information (e.g., a registered name) of the target object80. Further, for example, the target object information indicates a shape and a size (dimensions) of the target object80. For example, the target object information may be CAD data of the object. Note that the target object information may not include position information such as information indicating where the corresponding object is placed. Therefore, although the control apparatus100can recognize that some object is placed on the shelf board92by using the 3D environmental information, it cannot recognize which target object80corresponds to that object unless a later-described object recognition process is performed. The object detection system1makes a search as to where a registered target object80is located in the 3D environment4by using the target object information and the 3D environmental information. In other words, the object detection system1detects (i.e., determines) which area in the 3D environment4corresponds to the target object80.

The “sensor information” is information related to measurement performed by the sensor12. For example, the sensor information indicates an angle of view (a viewing angle; a field-of-view range), a focal length, a resolution, number of pixels, and the like of the sensor12. That is, the sensor information may indicate a measurable range of the sensor12. In this way, a size, a resolution, and the like of 3D image data (distance image data) generated by the sensor12can be specified.

When the 3D measurement data (the distance image) is generated by the measurement performed by the sensor12, the object recognition unit114recognizes the target object80present in the 3D environment4by using the 3D measurement data and information stored in the information storage unit112. Specifically, the object recognition unit114detects a registered target object80from the 3D measurement data (and, if necessary, the 3D environmental information). More specifically, the object recognition unit114calculates, for each object, a difference between information indicating the shape of that object in the 3D measurement data (the distance image) and object information (CAD data or the like) indicating the shape of the target object80. Then, the object recognition unit114recognizes that an object for which the calculated difference is smaller than a predetermined threshold as the target object80. Further, the object recognition unit114associates identification information of the detected target object80with position information indicating a position where that target object80is located.

The information generation unit116generates 3D environmental information. Specifically, when an object recognition process is performed by the object recognition unit114, the information generation unit116updates the 3D environmental information by using a result of the object recognition process (a recognition result). Note that the 3D environmental information may include only position information corresponding to the storage object90at the initial stage. In other words, before the sensor12starts measurement, only the storage object90is present in a 3D virtual space represented by the 3D environmental information. Then, every time the sensor12performs measurement and an object is recognized by the object recognition unit114, the information generation unit116integrates 3D measurement data generated by the measurement by the sensor12into the 3D environmental information. In this way, information on objects included in the 3D environmental information increases.

It should be noted that depending on the viewpoint of the sensor12or the position of the target object80, a part of the target object80may be shielded by another target object80or the wall surface94or the like of the storage object90even though that target object80is present in the field of view (the angle of view) of the sensor12. In such a case, the shielded part of the target object80is not photographed by the sensor12. For example, a target object80B is present in front of a target object80C inFIG. 1. Therefore, there is a possibility that, from a certain viewpoint position, a part of the target object80C (i.e., a lower-right part of the target object80C shielded by the target object80B) is not photographed by the sensor12. In this case, the target object80B acts as an obstacle and forms a blind spot for the sensor12when the sensor12measures (photographs) the target object80C. Further, at least a part of the target object80C is present in the blind spot.

In this case, there is a possibility that the object80may not be recognized by the object recognition unit114due to lack of the amount of information indicating its shape and the like. In such a case, the information generation unit116adds information indicating a shape of the part of the target object80, which has not been recognized, photographed by the sensor12in the 3D environmental information. In this case, although the control apparatus100can recognize that some object exists in that position by using the 3D environmental information, it cannot recognize which target object80corresponds to that object. Therefore, the 3D environmental information may include information related to a recognized target object80and information related to a target object80that has not been recognized (hereinafter also referred to as unrecognized target object80).

The position determining unit120determines an optimal viewpoint position (an optimal position) of the sensor12for the next measurement (later-described S130etc. inFIG. 4). Specifically, the position determining unit120determines, as the optimal viewpoint position, a viewpoint position from which it is expected that the target object80, which is possibly disposed in the storage object90but has not been recognized, can probably be recognized when the sensor12is moved to that viewpoint position and measures (photographs) the that target object80the next time. Here, the recognized object80(e.g., the target object80B inFIG. 1) is referred to as a first object and the target object80that has not been recognized because it is partially shielded by the first object (e.g., the target object80C inFIG. 1) is referred to as a second object. In this case, the position determining unit120determines, as the optimal viewpoint position, a viewpoint position from which the sensor12can measure an area that is shielded by the first object and becomes a blind spot by using the 3D environmental information. That is, in order to enable an area shielded by at least one first object recognized by the object recognizing unit114to be measured by the sensor12, the position determining unit120determines, as the optimal viewpoint position, a position of the sensor12where the sensor12will perform the next measurement according to the position of the first object by using the 3D environmental information. In other words, the position determination unit120determines, by using the 3D environmental information, a position of the sensor12where the sensor12can take a distance image in which a size of an area shielded by at least one first object recognized by the object recognition unit114is larger as the optimal viewpoint position. In this way, the control apparatus100can move the sensor12to a position where the sensor12can measure the area, which is shielded by the first object and becomes a blind spot, more appropriately. Therefore, the control apparatus100according to the first embodiment can efficiently detect the target object80even when the second object is shielded by the first object.

The sensor control unit140moves the sensor12to the optimal viewpoint position determined by the position determination unit120. Then, the sensor control unit140controls the sensor12so as to perform measurement at the viewpoint position to which the sensor12has moved (i.e., at the optimal viewpoint position). Then, the sensor control unit140acquires a distance image (3D measurement data) generated by the sensor12. Further, the sensor control unit140outputs the 3D measurement data to the object recognition unit114.

The search determination unit122determines whether or not a search of the storage object90has been completed (later-described S110to S112inFIG. 4). Then, when the search determination unit122determines that the search of the storage object90has been completed, a recognition result for the target object80is output to, for example, the interface unit108or the like. On the other hand, when the search determination unit122determines that the search of the storage object90has not been completed, later-described processes performed by the optimal viewpoint selection unit124are performed. Specifically, the search determination unit122determines whether or not a search of an area where the target object80can be placed (a placement available area) in the storage object90has been completed by using the 3D environmental information. Details will be described later. Note that the placement available area may be a flat surface (e.g., the top surface of the shelf board92) on which the target object80can be placed, or a space (e.g., a space between upper and lower shelf boards92) in which the target object80can be disposed.

Note that when the search of the storage object90by the sensor12has been completed, there is a high possibility that no target object80that has not been recognized exists in the storage object90(i.e., all the target objects80disposed in the storage object90have been recognized). That is, there is a high possibility that the measurement for detecting target objects80has been sufficiently performed. Therefore, performing further measurement by the sensor12is probably wasteful. Therefore, it is possible to prevent an unnecessary process from being performed by preventing the optimal viewpoint selection unit124from performing a process when the search determination unit122determines that the search of the storage object90has been completed as described in the first embodiment. Therefore, the control apparatus100according to the first embodiment can prevent or reduce an increase in the time taken to detect an object.

The optimal viewpoint selection unit124calculates viewpoint positions of the sensor12where the sensor12can measure the placement available area as candidates for the optimal viewpoint position (viewpoint candidates) according to the position and the shape of the storage object90in which the target object80can be disposed (later-described S122inFIG. 4). For example, viewpoint positions from which the shelf board92of the storage object90cannot be measured are excluded from the viewpoint candidates. Further, the optimal viewpoint selection unit124selects the optimal viewpoint position from the viewpoint candidates. Details will be described later. As described above, by calculating viewpoint candidates, it is possible to exclude viewpoint positions that cannot contribute to the detection of the target object80depending on the position and the shape of the storage object90from the optimal viewpoint positions. Therefore, the control apparatus100according to the first embodiment can efficiently detect the target object80.

Further, the optimal viewpoint selection unit124determines whether or not, among the viewpoint candidates, there is an unmeasured viewpoint position (an unmeasured position), i.e., a viewpoint position from which the sensor12can measure an area shielded by the recognized target object80(the first object) but has not performed measurement yet (later-described S150inFIG. 4). Then, when the optimal viewpoint selection unit124determines that there is an unmeasured viewpoint position, it performs a process for determining an optimal viewpoint position. Details will be described later. The state where there is no unmeasured viewpoint position means a state where the target object80cannot be detected even when measurement is further performed. Therefore, by determining whether or not there is an unmeasured viewpoint position as described above, it is possible to terminate the process for selecting the optimal viewpoint position when there is no unmeasured viewpoint position. As a result, it is possible to improve the efficiency of the detection of the target object80. That is, it is possible to finish the process for detecting target objects without performing a wasteful process.

FIG. 4is a flowchart showing an object detection method performed by the control apparatus100according to the first embodiment. Firstly, the sensor control unit140controls the sensor12so as to perform measurement at the current viewpoint position, and acquires 3D measurement data (a distance image) from the sensor12(step S100). Then, as described above, the object recognition unit114performs an object recognition process by using the 3D measurement data and information stored in the information storage unit112(i.e., target object information, 3D environmental information, storage object information, etc.) (step S102). The information generation unit116updates the 3D environmental information by using the recognition result and the 3D measurement data (step S104).

In the example shown inFIG. 1, the target objects80A and80B are located relatively close to the front of the shelf board92. Therefore, roughly the entire shape of each of them can be measured. Therefore, the object recognition unit114can recognize the target objects80A and80B. Then, the information generation unit116may add position information indicating the shape of the target object80A in an area corresponding to the position where the target object80A is disposed in a 3D virtual space represented by the 3D environmental information by using the target object information of the target object80A. The information generation unit116may also perform a similar process for the target object80B.

In contrast, the object80C is located behind the target object80B and hence a part of its shape cannot be measured. Therefore, there is a possibility that the object recognition unit114cannot recognize the target object80C. In this case, the information generation unit116may add position information indicating a measured part of the target object80C in an area corresponding to the position where the target object80C is disposed in the 3D virtual space represented by the 3D environmental information by using the 3D measurement data.

Next, the search determination unit122determines whether or not the search of the storage object90has been completed (step S110). Specifically, the search determination unit122determines, by a later-described method, whether or not the measurement of the placement available area by the sensor12has been completed by using the 3D environmental information and the storage object information. Therefore, the process in the step S110is performed in the 3D virtual space represented by the 3D environmental information.

When it is determined that the search of the storage object90has not been completed (No at S110), the search determination unit122determines that it is necessary to perform measurement from a viewpoint position from which measurement has not been performed. Therefore, in this case, the search determination unit122outputs a signal indicating that further measurement is necessary to the optimal viewpoint selection unit124. As a result, the optimal viewpoint selection unit124performs a later-described process in a step S120.

On the other hand, when it is determined that the search of the storage object90has been completed (Yes at S110), the search determination unit122determines that at least a part of every target object80(e.g., an upper half, a left half, etc. of every target object80) disposed in the placement available area has been measured. Therefore, the search determination unit122determines whether or not labeling has been performed for every target object80disposed in the storage object90(step S112). Note that the labeling means, as a result of recognition of a target objects80disposed in the storage object90, associating identification information corresponding to that target object80with an area where that target object80is disposed.

Note that when roughly the entire shape of the target object80has been measured in the 3D measurement data (the 3D environmental information) (i.e., when roughly the entire image of the target object80is included in the distance image), labeling can be made by using the target object information. On the other hand, when the entire shape of the target object80has not been measured in the 3D measurement data (the 3D environmental information) (i.e., when a part of an image of the target object80is missing in the distance image and information necessary for the object recognition is insufficient), there is a possibility that the object recognition will end in failure even when the target object information is used. In such a case, labeling cannot be made.

When it is determined that labeling has been made for all the target objects80disposed in the storage object90(Yes at S112), the control apparatus100determines that all the target objects80disposed in the storage object90have been detected. Therefore, the control apparatus100outputs a recognition result to the interface unit108(step S114). Note that the recognition result is information indicating where each target object80is located. In other words, the recognition result is information indicating which area is occupied by which target object80in the 3D virtual space represented by the 3D environmental information.

When it is determined that labeling has still not been made for all the target objects80disposed in the storage object90(No at S112), the search determination unit122determines that further measurement is necessary to supplement the information which is insufficient to perform the labeling. In this case, the search determination unit122outputs a signal indicating that further measurement is necessary to the optimal viewpoint selection unit124. As a result, the optimal viewpoint selection unit124performs the later-described process in the step S120.

FIG. 5is a flowchart showing a first example of a search completion determination method performed by the search determination unit122according to the first embodiment. Further,FIG. 6is a diagram for explaining the first example shown inFIG. 5.FIG. 6is a plan view showing a state in which target objects80A,80B and80C are disposed on the shelf board92of the storage object90as viewed from above (as viewed in a Z-axis positive direction). Further, it is assumed that the target objects80A,80B and80C have already been recognized.

The search determination unit122calculates a sum total Sa of a measured area (i.e., a measured size) of the shelf board92of the storage object90and areas (i.e., sizes) of bottom surfaces of the recognized target objects80(step S112A). Note that the measured area of the shelf board92can be geometrically calculated from the 3D measurement data and coordinate data in the 3D environmental information. Further, the areas of the bottom surfaces of the recognized target objects80are included in their corresponding target object information beforehand. In the example shown inFIG. 6, the measured area (i.e., the measured size) of the shelf board92corresponds to the area (i.e., the size) of the shelf board92excluding areas that are shielded by the target objects80A,80B and80C, and hence become blind spots (i.e., hatched areas) within a field-of-view range12a. Further, the areas (i.e., the sizes) of the bottom surfaces of the recognized target objects80corresponds to the sum total of the bottom areas of the target objects80A,80B and80C. Note that when the target object80C has not been recognized, the bottom area of the target object80C is excluded from the sum total Sa.

The search determination unit122determines whether or not a ratio of the sum total Sa to the area Sb of the shelf board92, i.e., a ratio Sa/Sb is equal to or larger than a predetermined threshold value ThA (step S114A). Note that the area Sb of the shelf board92can be acquired (i.e., calculated) from the storage object information. The area Sb may be included in the storage object information. When the ratio Sa/Sb is smaller than the threshold value ThA (Sa/Sb<ThA) (No at S114A), the search determination unit122determines that the search of the storage object90has not been completed (step S116A). On the other hand, when the ratio Sa/Sb is equal to or larger than the threshold value ThA (Sa/Sb≥ThA) (Yes at S114A), the search determining unit122determines that the search of the storage object90has been completed (step S118A). Note that the amount of unmeasured areas and the number of unrecognized target objects80can be reduced by having the sensor12perform measurement in a viewpoint position(s) determined by the later-described process in the step S130. Therefore, a possibility that the relation “Sa/Sb≥ThA” holds can be increased by having the sensor12repeat measurements in viewpoint positions determined by the process in the step S130.

FIG. 7is a flowchart showing a second example of the search completion determination method performed by the search determination unit122according to the first embodiment. Further,FIG. 8is a diagram for explaining the second example shown inFIG. 7.FIG. 8is a plan view showing a state in which target objects80A,80B and80C are disposed on the shelf board92of the storage object90as viewed from above (as viewed in a Z-axis positive direction).

The search determination unit122extracts edges of the already-measured placement available area of the storage object90from the measurement data (the 3D environmental information) (step S112B). Note that when the placement available area corresponds to the shelf board92, edges92eof the placement available area are boundaries between the shelf board92and the wall surfaces94(indicated by bold lines inFIG. 8). Note that since coordinate data of the shelf board92is already included in the storage object information (the 3D environmental information), it is easy to recognize which areas correspond to the edges92ein the measurement data (the 3D environmental information).

The search determination unit122determines whether or not all the edges92eof the placement available area have been detected (step S114B). When all the edges92ehave not been detected (No at S114B), the search determination unit122determines that the search of the storage object90has not been completed (step S116B). On the other hand, when all the edges92ehave been detected (Yes at S114B), the search determining unit122determines that the search of the storage object90has been completed (step S118B). In the example shown inFIG. 8, no edge92eis detected in parts that become blind spots due to the target objects80and parts located outside the field-of-view range12a(indicated by broken-line ellipses inFIG. 8). Therefore, in this case, the search determination unit122determines that the search of the storage object90has not been completed. Note that the amount of unmeasured edges92ecan be reduced by having the sensor12perform measurement in a viewpoint position(s) determined by the later-described process in the step S130. Therefore, all the edges92ecould be detected by having the sensor12repeat measurements in viewpoint positions determined by the process in the step S130.

Note that even when the placement available area is a 3D space, the above-described first and second examples can be applied. In the first example, the search determining unit122may compare the sum total of the volume of a measured space in the storage object90and the volumes of recognized target objects80with the volume of the placement available area in the storage object90. In the second example, the search determination unit122may determine whether all the wall surfaces94around the shelf board92have been measured.

The optimal viewpoint selection unit124determines whether or not viewpoint candidates have already been calculated (step S120). When the viewpoint candidates have not been calculated (No at S120), the optimal viewpoint selection unit124calculates viewpoint candidates (step S122). On the other hand, when the viewpoint candidates have already been calculated (Yes at S120), the optimal viewpoint selection unit124examines other viewpoint positions included in the viewpoint candidates (S142to S150). Note that the process in the step S122may be performed only once, i.e., performed only in the first loop.

In the step S120, the optimal viewpoint selection unit124calculates viewpoint positions of the sensor12from which the sensor12can measure the placement available area as viewpoint candidates by using the storage object information and the 3D environmental information. In this case, the optimal viewpoint selection unit124takes only the storage object90into consideration and does not take the presence of target objects80disposed in the storage object90into consideration. Note that the process in the step S122is performed in the 3D virtual space represented by the 3D environmental information.

Specifically, the optimal viewpoint selection unit124calculates, for example, a viewpoint position in which at least a part of the placement available area of the storage object90is included in the field-of-view range12a(the viewing angle) of the sensor12by using the storage object information and the 3D environmental information. The optimal viewpoint selection unit124calculates, for example, a viewpoint position in which at least a part of the shelf board92of the storage object90is included. For example, the optimal viewpoint selection unit124determines whether or not an image of the shelf board92is included in the field-of-view range12a(the viewing angle) of the sensor12when the viewpoint position is moved in the 3D virtual space. Further, when the image of the shelf board92is included in the field-of-view range12a, the optimal viewpoint selection unit124defines that viewpoint position as a viewpoint candidate.

FIGS. 9 to 11are diagrams for explaining a viewpoint candidate calculation method according to the first embodiment. InFIG. 9, at least a part of the shelf board92is not shielded by the wall surface94and is included in the field-of-view range12aat either of viewpoints A and B. Therefore, the viewpoints A and B are included in the viewpoint candidates. In contrast, at a viewpoint C, since the shelf board92is entirely shielded by the wall surface94, no area of the shelf board92is included in the field-of-view range12a. Therefore, the viewpoint C is not included in the viewpoint candidates.

InFIG. 10, a range of viewpoint candidates12bin which the sensor12can be positioned is indicated by a bold-line arrow. The sensor12can measure at least a part of the shelf board92at any viewpoint in this range of viewpoint candidates12b. Meanwhile, no part of the shelf board92can be measured in a range of viewpoints indicated by a broken-line arrow and hence this range of viewpoints is excluded from the viewpoint candidates.

InFIG. 11, a range of viewpoint candidates12bwhich represents a range of postures (orientations) of the sensor12at a position A is indicated by a bold-line arrow. The sensor12can measure the shelf board92in at any viewpoint in this range of viewpoint candidates12b. Meanwhile, the shelf board92cannot be measured in a range of viewpoints indicated by a broken-line arrow and hence this range of viewpoints is excluded from the viewpoint candidates.

Next, the optimal viewpoint selection unit124selects an optimal viewpoint position, which is an optimal viewpoint position as the next viewpoint position of the sensor12, from the viewpoint candidates (step S130). Specifically, the optimal viewpoint selection unit124selects, as the optimal viewpoint position, a position where the sensor12can measure a target object80(e.g., the target object80C inFIG. 1) that is partially shielded by an already-recognized target object(s)80(e.g., the target object(s)80A or/and80B inFIG. 1). Further, the optimal viewpoint selection unit124selects, as the optimal viewpoint position, a position where the sensor12can measure an area that is shielded by the already-recognized target object(s)80(e.g., the target object(s)80A or/and80B inFIG. 1). That is, the position determination unit120determines, by using the 3D environmental information, a position of the sensor12where the sensor12can take a distance image in which a size of an area shielded by the already-recognized target object(s)80(e.g., the target object(s)80A or/and80B inFIG. 1) is larger as the optimal viewpoint position.

FIGS. 12 to 14are diagrams for explaining processes performed by the optimal viewpoint selection unit124according to the first embodiment. The examples shown inFIGS. 12 and 13can be used to measure an object80(e.g., the object80C inFIG. 1) that has not been recognized because a part of it is shielded, though another part of it has been measured. Further, the example shown inFIG. 14can be used in a case where it is unknown whether or not a target object80is disposed in an area(s) corresponding to a blind spot(s) of a recognized target object(s)80. The case of the example shown inFIG. 1is described hereinafter.

In the example shown inFIG. 12, the optimal viewpoint selection unit124selects, for example, a viewpoint position in which an unrecognized target object80and target objects80located around this unrecognized target object80are widely distributed (i.e., scattered over a wide area) in the field of view (the angle of view) of the sensor12. Specifically, firstly, the optimal viewpoint selection unit124excludes position data related to the storage object90from the 3D environmental information as indicated by broken lines inFIG. 12. Then, in the 3D virtual space represented by the 3D environmental information, the optimal viewpoint selection unit124extracts, for each of a plurality of viewpoint positions imaginarily defined in the range of viewpoint positions calculated in the process in the step S122, a distance image that is obtained when the aforementioned plurality of target objects80are viewed from that imaginarily-defined viewpoint position. Note that the optimal viewpoint selection unit124can generate a distance image that is obtained when the target objects80are viewed from an imaginarily-defined viewpoint position by using sensor information indicating an angle of view, etc. of the sensor12.

Then, the optimal viewpoint selection unit124calculates, for each of the distance images, a distance between both ends of the plurality of target objects80. The optimal viewpoint selection unit124selects a viewpoint position corresponding to a distance image having the longest distance between both ends of the targets objects80as an optimal viewpoint position. Note that the optimal viewpoint selection unit124may select a viewpoint position corresponding to a distance image in which the distance between both ends of the targets objects80is greater than a predetermined threshold as the optimal viewpoint position. Note that “the distance between both ends” may not be an actual distance, but may be a distance in the distance image. Therefore, “the distance between both ends” may correspond to the number of pixels in the distance image (This also applies to the example shown inFIG. 13). Note that in a distance image in which “the distance between both ends” is larger may include a larger area that includes a part of the unrecognized target object80.

In the example shown inFIG. 12, a distance image ImX including target object images80AI,80BI and80CI, which are images of the target objects80A,80B and80C, respectively, viewed from a viewpoint X, is extracted. Similarly, distance images ImY and ImZ that include target object images80AI,80BI and80CI viewed from viewpoints Y and Z, respectively, are extracted. Then, the optimal viewpoint selection unit124calculates a distance Wx between both ends of the target objects80A,80B and80C in the distance image ImX. Similarly, the optimal viewpoint selection unit124calculates distances Wy and Wz between both ends of the target objects80A,80B and80C in the distance images ImY and ImZ, respectively. Then, the optimal viewpoint selection unit124determines as follows: Wx>Wy; and Wx>Wz. That is, the optimal viewpoint selection unit124determines that the distance image ImX has the longest distance between both ends. Therefore, the optimal viewpoint selection unit124selects the viewpoint X as an optimal viewpoint position. In this way, in the example shown inFIG. 12, the position determination unit120determines, as the optimal viewpoint position, a position of the sensor12where the sensor12can take a distance image in which a size of an area (a part of the unrecognized target object80C) shielded by an already-recognized target object(s)80is larger.

Incidentally, when measurement has been made only from a viewpoint corresponding to the viewpoint Y, a part indicated by an arrow A1inFIG. 12has not been actually measured yet by the sensor12at the time of this process. Therefore, since position data of the part indicated by the arrow A1is not included in the 3D environmental information, this part may be missing in the distance images ImX and ImZ (This also applies toFIG. 13). Further, when measurement has been made only from a viewpoint corresponding to the viewpoint Y, a part indicated by an arrow B1has also not been measured yet. However, since the target objects80A and80B have already been recognized, it is possible to draw the part indicated by the arrow B1by using the target object information.

In the example shown inFIG. 13, the optimal viewpoint selection unit124selects, for example, a viewpoint position in which a distance between the center of the unrecognized target object80and the center of an already-recognized target object(s)80in the distance image becomes larger. Specifically, the optimal viewpoint selection unit124calculates a central axis(es) of the recognized target object(s)80in the 3D virtual space represented by the 3D environmental information by using the target object information of the recognized target object80. Note that the target object information may include information indicating the central axis of the target object(s)80beforehand.

Further, the optimal viewpoint selection unit124calculates a central axis of the unrecognized target object80in the Z-axis direction by using the 3D measurement data. Note that in the 3D measurement data, only a part of the unrecognized target object80has been measured. Therefore, the optimal viewpoint selection unit124calculates the central axis of the unrecognized target object80in the range that can be estimated from the measured part. For example, when the top surface of the unrecognized target object80has already been measured, the optimal viewpoint selection unit124may presume (i.e., regard) an axis that passes through the center of gravity of the top surface as the central axis. Further, when the left and right side surfaces of the unrecognized target object80have already been measured, the optimal viewpoint selection unit124may presume (i.e., regard) an axis that passes through the middle between the left and right side surfaces as the central axis.

Then, in the 3D virtual space represented by the 3D environmental information, the optimal viewpoint selection unit124extracts, for each of a plurality of viewpoint positions imaginarily defined in the range of viewpoint candidates calculated in the process in the step S122, a distance image that is obtained when the aforementioned plurality of target objects80are viewed from that imaginarily-defined viewpoint position. Then, the optimal viewpoint selection unit124calculates, for each of the distance images, a distance between the central axis of the unrecognized target object80and the central axis(es) of the already-recognized target object(s)80. The optimal viewpoint selection unit124selects a viewpoint position corresponding to a distance image having the longest distance between the centers of these target objects as an optimal viewpoint position. Note that the optimal viewpoint selection unit124may select a viewpoint position corresponding to a distance image in which the distance between the centers of the target objects is greater than a predetermined threshold as the optimal viewpoint position. Note that in a distance image in which “the distance between the centers” is larger may include a larger area that includes a part of the unrecognized target object80.

In the example shown inFIG. 13, distance images ImX and ImY that include target object images80AI,80BI and80CI as viewed from viewpoints X and Y, respectively, are extracted. Then, the optimal viewpoint selection unit124calculates a distance Dx1between a central axis80Cc of the target object80C and a central axis80Ac of the target object80A, and a distance Dx2between the central axis80Cc of the target object80C and a central axis80Bc of the target object80B in the range image ImX. Similarly, the optimal viewpoint selection unit124calculates a distance Dy1between a central axis80Cc of the target object80C and a central axis80Ac of the target object80A, and a distance Dy2between the central axis80Cc of the target object80C and a central axis80Bc of the target object80B in the range image ImY. Then, for example, the optimal viewpoint selection unit124compares an average value (or a maximum value) of the inter-central-axis distances Dx1and Dx2in the distance image ImX with an average value (or a maximum value) of the inter-central-axis distances Dy1and Dy2in the distance image ImY. Then, the optimal viewpoint selection unit124determines that the distance image ImX has a larger average value (or a larger maximum value) and hence selects the viewpoint X as an optimal viewpoint position. In this way, in the example shown inFIG. 13, the position determination unit120determines, as the optimal viewpoint position, a position of the sensor12where the sensor12can take a distance image in which a size of an area (a part of the unrecognized target object80C) shielded by an already-recognized target object(s)80is larger.

Note that in the example shown inFIG. 13, only the distance between the central axes of the target objects80needs to be obtained. Therefore, in the example shown inFIG. 13, images of target objects80themselves do not need to be included in the extracted distance image. That is, only the central axis of each target object80needs to be included in the extracted distance image.

In the example shown inFIG. 14, the optimal viewpoint selection unit124selects, for example, a viewpoint position in which an unmeasured area in the placement available area of the storage object90is included in the field-of-view range12aas much as possible. Note that the unmeasured area means areas that were shielded by the target object(s)80or located outside the field-of-view range12awhen measurement was performed from an already-selected viewpoint position(s).FIG. 14is a plan view showing a state in which target objects80A,80B and80C are disposed on the shelf board92of the storage object90as viewed from above (as viewed in a Z-axis positive direction). Further, it is assumed that the target objects80A and80B have already been recognized.

The optimal viewpoint selection unit124extracts an unmeasured area from the placement available area of the storage object90. Specifically, the optimal viewpoint selection unit124geometrically extracts an area of the shelf board92that has not been measured (i.e., an unmeasured area of the shelf board92) from the 3D environmental information, the 3D measurement data, and the storage object information. More specifically, the optimal viewpoint selection unit124extracts, as the unmeasured area, an area that is not indicated as being already-measured in the 3D measurement data (the 3D environmental information) from position data of the top surface of the shelf board92included in the storage object information.

Then, in the 3D virtual space represented by the 3D environmental information, the optimal viewpoint selection unit124extracts, for each of a plurality of viewpoint positions imaginarily defined in the range of viewpoint candidates, a distance image that is obtained when the shelf board92is viewed from that imaginarily-defined viewpoint position. Then, the optimal viewpoint selection unit124calculates, for each of the distance images, a size of the unmeasured area of the shelf board92. The optimal viewpoint selection unit124selects a viewpoint position corresponding to a distance image having the largest size of the unmeasured area as an optimal viewpoint position. Note that “the size of the unmeasured area” may not be an actual size, but may be a size in the distance image. Therefore, “the size of the unmeasured area” corresponds to the number of pixels in a part corresponding to the unmeasured area in the distance image.

In the example shown inFIG. 14, the optimal viewpoint selection unit124determines that an area(s) that cannot be measured when the sensor12performs measurement from a viewpoint position indicated by an arrow Y (i.e., hatched areas inFIG. 14) will be included in the field-of-view range12aas much as possible when the sensor12performs measurement from a viewpoint position indicated by an arrow Z. Therefore, the optimal viewpoint selection unit124selects the viewpoint position corresponding to the arrow Z as an optimal viewpoint position. In this way, in the example shown inFIG. 14, the position determination unit120determines, as the optimal viewpoint position, a position of the sensor12where the sensor12can take a distance image in which a size of an area shielded by an already-recognized target object(s)80is larger.

Note that in addition to the above-described examples shown inFIGS. 12 to 14, the optimal viewpoint selection unit124may determine the optimal viewpoint position based on discontinuity in measured position data of an unrecognized target object80. Specifically, the optimal viewpoint selection unit124may determines, as the optimal viewpoint position, a position of the sensor12from which an image in which, in the already-measured position data of the unrecognized target object80, a part that became discontinuous due to an already-recognized target object80, rather than the storage object90, becomes larger is taken. Note that “the part that became discontinuous due to the already-recognized target object80” means an area that is shielded by the already-recognized target object80. Further, the optimal viewpoint selection unit124may perform a process for selecting an optimal viewpoint position in which at least two of the above-described examples shown inFIGS. 12 to 14and the above-described other example are combined.

The sensor control unit140controls the driving unit14and thereby moves the sensor12to the viewpoint position determined by the process in the step S130(step S140). Then, the sensor control unit140controls the sensor12so as to perform measurement at the viewpoint position to which the sensor12has moved, and acquires 3D measurement data (a distance image) from the sensor12(S100).

Then, the object recognition unit114performs the above-described object recognition process (S102). A possibility that an unrecognized target object80(e.g., the target object80C) shown inFIG. 1, etc. can be recognized is increased by performing measurement at the optimal viewpoint position.

Then, after the process in the step S104is performed, the search determination unit122determines whether or not the search of the storage object90has been completed (S110), and determines whether or not labeling has been made for all the target objects80(S112). For example, when the target object80C is recognized and it is determined that no other target object80is disposed in the storage object90(Yes at S110and Yes at S112), the control apparatus100determines that the detection of all the target objects80disposed in the storage object90has been completed and outputs a recognition result (S114). On the other hand, when the target object80C has not been recognized yet or when it is determined that it is unknown whether or not another target object80is disposed in the storage object90(No at S110or No at S112), the control apparatus100proceeds to the process in the step S120.

Then, since the viewpoint candidates have already been calculated in the process in the step S120(Yes at S120), the optimal viewpoint selection unit124examines other viewpoint positions included in the viewpoint candidates (S142to S150). Specifically, to prevent the same measurement from being performed again, the optimal viewpoint selection unit124excludes the current viewpoint position, i.e., the viewpoint position from which measurement has already been performed from the viewpoint candidates (step S142). Note that the optimal viewpoint selection unit124may exclude the current viewpoint position and its surroundings from the viewpoint candidates. Then, the optimal viewpoint selection unit124determines whether or not there is an unmeasured viewpoint position in the viewpoint candidates (step S150). Specifically, the optimal viewpoint selection unit124determines whether or not there is a viewpoint position from which the sensor12can measure an area shielded by the recognized target object80but has not performed measurement yet (i.e., an unmeasured viewpoint position).

When it is determined that there is an unmeasured viewpoint position (an unmeasured position) (Yes at S150), the optimal viewpoint selection unit124performs the process in the step S130, i.e., selects an optimal viewpoint position. On the other hand, when it is determined that there is no unmeasured viewpoint position (No at S150), the optimal viewpoint selection unit124determines that further detection is impossible. Then, the control apparatus100outputs recognition results obtained up to this point to the interface unit108or the like (step S114).

FIG. 15is a flowchart showing an example of a process for determining whether or not there is an unmeasured viewpoint position (S150) according to the first embodiment. Further,FIGS. 16 to 18are diagrams for explaining the process shown inFIG. 15. Firstly, the optimal viewpoint selection unit124extracts a shielded area from the placement available area (step S152). Note that the shielded area is an area in the placement available area (e.g., an area on the upper surface of the shelf board92) that is neither a measured area nor an area that is inferred to correspond to a bottom surface of an already-recognized target object80. That is, the shielded area is an area that is obtained by excluding an area(s) that is inferred to correspond to the bottom surface(s) of the already-recognized target object(s)80from the area(s) that is shielded by an object(s) (i.e., the target object80or the storage object90) and has not been able to be measured. InFIGS. 16 to 18, shielded areas92hare indicated by hatching.

Next, the optimal viewpoint selection unit124detects a straight line that can extend from a boundary of the shielded area92hto an edge92flocated on an opened side of the placement available area without being obstructed by any obstacle (step S154). Note that the obstacle is, for example, a target object80disposed in the placement available area (the shelf board92) or the wall surface94. Both straight lines L1and L2shown inFIG. 16can extend from the boundary of the shielding area92hto the edge92fwithout being obstructed by any obstacle. In contrast, both straight lines L3and L4shown inFIG. 17are obstructed by (interfere with) an obstacle (the target object80A or the target object80B) when they extend from the boundary of the shielded area92hto the edge92f. Therefore, while the straight lines L1and L2are detected, the straight lines L3and L4are not detected. Further, in the example shown inFIG. 18, the boundary of the shielded area92hcannot be connected with the edge92fby a straight line due to the targets objects80D and80E. When an object is disposed with a ratio higher than a predetermined threshold with respect to the horizontal direction (and the vertical direction) of the placement available area as described above, the shielded area92hcannot be measured.

The optimal viewpoint selection unit124determines whether or not a straight line is detected in the process in the step S154(step S156). When a straight line is detected (Yes at S156), the optimal viewpoint selection unit124determines that there is an unmeasured viewpoint position (step S158). On the other hand, when no straight line is detected (No at S156), the optimal viewpoint selection unit124determines that there is no unmeasured viewpoint position (step S160). In the example shown inFIG. 16, since a straight line is detected, it is determined that there is an unmeasured viewpoint position. On the other hand, in the example shown inFIG. 18, since no straight line is detected, it is determined that there is no unmeasured viewpoint position.

Note that the examples shown inFIGS. 15 to 17correspond to the first example of the process in the step S110shown inFIGS. 5 and 6. That is, in the examples shown inFIGS. 15 to 17, the determination on the unmeasured viewpoint position is made based on whether or not there is an unmeasured area in the shelf board92. Further, the determination on the unmeasured viewpoint position can also be made by a method corresponding to the second example of the process in the step S110shown inFIGS. 7 and 8. In this case, the optimal viewpoint selection unit124may determine whether or not there is a straight line that can extend from an unmeasured edge92eof the shelf board92(which corresponds to the wall surface94) to an edge92fon the opened side of the shelf board92without being obstructed by any obstacle.

As described above, the control apparatus100according to the first embodiment determines, when detecting a target object80by using the sensor12, a viewpoint position from which the sensor12can measure an area that becomes a blind spot due to an obstacle such as another target object80as an optimal viewpoint position for the next measurement. Therefore, it is possible to reduce the number of movements of the sensor12and the time required therefor when the target object80is detected by using the sensor12. Accordingly, the control apparatus100according to the first embodiment can efficiently detect a target object80even when the target object80is shielded by another object(s).

Second Embodiment

Next, a second embodiment is described. The second embodiment differs from the first embodiment because a target object80can be removed in the second embodiment.

FIG. 19shows an object detection system1according to the second embodiment. Further,FIG. 20is a block diagram showing a hardware configuration of the object detection system1according to the second embodiment. The object detection system1according to the second embodiment includes an object detection apparatus10and a control apparatus100. The object detection apparatus10includes a sensor12, a driving unit14that drives the sensor12, and an arm20. The arm20is, for example, a robot arm, and includes a plurality of joints22and an end effector24capable of grasping an object. The arm20can grasp a target object80under the control of the control apparatus100.

FIG. 21is a functional block diagram showing a configuration of the control apparatus100according to the second embodiment. The control apparatus100according to the second embodiment includes an information storage unit112, an object recognition unit114, an information generation unit116, a position determination unit120, and a sensor control unit140. Further, the control apparatus100according to the second embodiment includes a removal determination unit210and an arm control unit220.

FIG. 22is a flowchart showing an object detection method performed by the control apparatus100according to the second embodiment. Note that inFIG. 22, illustrations of processes substantially similar to those inFIG. 4are omitted. When there is no unmeasured viewpoint (No at S150), the removal determination unit210determines which target object80needs to be removed in order to measure an unmeasured area (a shielded area) in the best way (step S210). Specifically, the removal determination unit210determines one of already-recognized target objects80which provides a largest measurable area when removed (i.e., an object to be removed). A specific determination method will be described later. As described above, the control apparatus100according to the second embodiment can determine an object to be removed (hereinafter also referred to as a removal object). As a result, it is possible to measure an unmeasured area by removing the removal object in the subsequent processes, so that the control apparatus100according to the second embodiment can make it possible to efficiently measure the unmeasured area.

Then, the arm control unit220controls the arm20so as to take out (i.e., remove) the removal object determined in the step S210and moves it to a location other than the storage object90(step S230). Specifically, the arm control unit220detects a position of the removal object (the target object80) by using the 3D environmental information. The arm control unit220extracts positions of the storage object90and other target objects80, and detects (i.e., determines) a trajectory of the arm20that does not interfere (i.e., collide) with the storage object90and the other target objects80. Then, the arm control unit220controls the joints22so that the arm20moves along the detected trajectory. Then, when the end effector24reaches a position where it can grasp the removal object, the arm control unit220controls the end effector24so as to grasp the removal object. Then, the arm control unit220controls the arm20so as to move the removal object to other places. Note that when a target object80that has been searched for is found as a result of the removal of the removal object, the arm control unit220may control the arm20so as to grasp and remove that target object80. As described above, a removal object can be automatically removed in the second embodiment. Therefore, it is possible to efficiently measure an unmeasured area.

FIG. 23is a flowchart showing an example of a method for determining a removal object according to the second embodiment. Further,FIGS. 24 to 28are diagrams for explaining the method shown inFIG. 23. Firstly, the removal determination unit210selects a candidate (a candidate removal object) that can be removed by using the arm20from among recognized target objects80(step S212). Specifically, the removal determination unit210makes, for each of already-recognized target objects80, a trajectory plan for the arm20by which that target object80can be grasped, and selects a candidate removal object that can be grasped and removed. Here, the number of selected candidate removal objects is represented by n. In the example shown inFIG. 24, it is assumed that target objects80A,80B and80C are candidate removal objects. That is, the number n is three (n=3). Further, it is assumed that a target object80X has not been recognized yet.

Next, the removal determining unit210calculates, for each candidate removal object k, an area v_k that cannot be measured due to that candidate removal object k (i.e., an unmeasurable area v_k) (k is an integer between 1 to n) in the current sensor position (step S214). Referring toFIG. 25, the removal determination unit210calculates an unmeasurable area82A (v_1) that becomes a blind spot due to the target object80A (k=1), which is one of the candidate removal objects. Similarly, the removal determination unit210calculates unmeasurable areas82B (v_2) and82C (v_3) that become blind spots due to the target objects80B (k=2) and80C (k=3), respectively, which are candidate removal objects. Note that parts of the unmeasurable areas82B and82C overlap each other.

Next, the removal determining unit210calculates an area V_K that cannot be measured due to n−1 candidate removal objects (i.e., an unmeasurable area V_K) under an assumption that the candidate removal object k has been removed (Step S216).FIG. 26shows unmeasurable areas82B and82C, which become blind spots due to the target objects80B and80C under an assumption that the target object80A has been removed. An area where at least one of the unmeasurable areas82B and82C is present is an unmeasurable area V_1under the assumption that the target object80A (k=1) has been removed. That is, the unmeasurable area V_1corresponds to a union of the unmeasurable areas82B (v_2) and82C (v_3).

FIG. 27shows unmeasurable areas82A and82C, which become blind spots due to the target objects80A and80C under an assumption that the target object80B has been removed. An area where at least one of the unmeasurable areas82A and82C is present is an unmeasurable area V_2under the assumption that the target object80B (k=2) has been removed. That is, the unmeasurable area V_2corresponds to a union of the unmeasurable areas82A (v_1) and82C (v_3).

FIG. 28shows unmeasurable areas82A and82B, which become blind spots due to the target objects80A and80B under an assumption that the target object80C has been removed. An area where at least one of the unmeasurable areas82A and82B is present is an unmeasurable area V_3under the assumption that the target object80C (k=3) has been removed. That is, the unmeasurable area V_3corresponds to a union of the unmeasurable areas82A (v_1) and82B (v_2).

Then, the removal determining unit210determines, as the removal object, a candidate removal object k that can reduce the unmeasurable area V_k the most when removed (step S218). That is, the removal determination unit210determines, as the target object80that provides the largest measurable area when removed, a candidate removal object that had been assumed to be removed when the unmeasurable area V_k was reduced the most. In the examples shown inFIGS. 24 to 28, the unmeasurable area V_2shown inFIG. 27is the smallest among the unmeasurable areas V_1to V_3. Therefore, the removal determining unit210determines that the target object80B should be removed.

Modified Example

Note that the present disclosure is not limited to the above-described embodiments and they can be modified as desired without departing from the scope and spirit of the disclosure. For example, the order of steps in the flowchart shown inFIG. 4, etc. can be changed as desired. Further, one or more steps in the flowchart shown inFIG. 4, etc. may be omitted. Further, although the optimal viewpoint position indicates 3D coordinates and an orientation (an angle) in the 3D environment4in the above-described embodiments, the present disclosure is not limited to such configurations. The optimal viewpoint position may indicate 3D coordinates in the 3D environment.

Further, although the target object80is placed on the shelf board92in the above-described embodiments, the present disclosure is not limited to such configurations. For example, the target object80may be hooked on a hook provided (e.g., attached) on the wall surface94. Further, the target object80does not need to be disposed (stored) in the storage object90. The target object80may be placed on the floor surface in the 3D environment4.

Further, although the sensor12is a range sensor (a 3D sensor) in the above-described embodiments, the present disclosure is not limited to such configurations. The sensor12may be a two-dimensional (2D) sensor, provided that it can generate 3D environmental information. However, by using a range sensor, it is possible to easily recognize a 3D position of an object without performing complex image processing (such as edge detection and pattern matching).

Further, although a target object80that is determined to be a removal object is removed by the arm20in the above-described second embodiment, the present disclosure is not limited to such configurations. The control apparatus100may output information as to which target object80is the removal object to, for example, the interface unit108. Then, a user may manually remove the target object80whose information was output to the interface unit108.