Patent Description:
Image-based object recognition is a common technique these days. In the image-based object recognition, an object in an image is recognized by, for example, extracting a feature amount of an image captured by an imaging device and performing matching between the feature amount and a feature amount registered in advance as dictionary data. In this case, changing an angle of the object in the image causes a change in the feature amount; thus, it is necessary to prepare dictionary data per angle of the object to enhance availability of object recognition.

PTL <NUM> and <NUM> are examples of a technique for preparing dictionary data per angle of an object in image-based object recognition. PTL <NUM> describes a technique for recognizing an object on the basis of eight images obtained by rotating an object at intervals of <NUM> degrees. PTL <NUM> describes a technique for learning an object model by recognizing common parts from many images obtained by rotating an object at intervals of five degrees or the like in a horizontal angle direction and a zenith angle direction. <CIT> discloses a target object gripping apparatus that comprises: an estimation unit configured to estimate an orientation of a target object based on orientation estimation parameters; a gripping unit configured to grip the target object based on the orientation of the target object estimated by the estimation unit; a detection unit configured to detect a failure of gripping by the gripping unit; and a modifying unit configured to modify the orientation estimation parameters based on the orientation of the target object when the detection unit detects a gripping failure.

The above techniques are for recognizing an object in the image, that is, estimating an identity of the object in the image, and are not intended to extract further, additional information from the image. Nevertheless, if it is taken into account that the technique of the object recognition has been recently applied to diverse fields, providing additional information regarding an object on the basis of the image is considered to be advantageous.

An object of the present invention is, therefore, to provide novel and improved information processing device, information processing method, and program that can estimate an angle of an object on the basis of an image and autonomously update data for estimation.

The above object is solved by the subject-matter of the independent claims.

Several embodiments of the present invention will be described hereinafter in detail with reference to the accompanying drawings. It is noted that constituent elements having substantially the same functional configurations are denoted by the same reference symbols and will not be repetitively described in the present specification and the drawings.

<FIG> is a schematic diagram of a system <NUM> according to a first embodiment of the present invention. With reference to <FIG>, the system <NUM> includes a terminal <NUM>, a database <NUM>, and a robot <NUM> connected to one another by a network NW. A camera <NUM> and a camera platform device <NUM> are connected to the terminal <NUM>. The robot <NUM> includes a camera <NUM> and a manipulator <NUM>.

In the system <NUM> described above, the camera <NUM> captures an image of an object obj mounted on the camera platform device <NUM> via a jig <NUM> to be described later. The terminal <NUM> acquires the image from the camera <NUM> and acquires angle information indicating an angle of the object obj from the camera platform device <NUM>. It is noted that the angle of the object obj is an angle in a three-dimensional space, for example, an angle represented by rotation amounts about three axes in an orthogonal coordinate system. The terminal <NUM> generates dictionary data on the basis of the acquired image and angle information (as well as identification information regarding the object obj). The generated dictionary data is stored in the database <NUM>.

Meanwhile, the robot <NUM> captures an image of the object obj using the camera <NUM> in a state in which the manipulator <NUM> grips the object obj. The robot <NUM> recognizes the object obj in the image and further estimates an angle of the object obj in the image on the basis of the captured image and the dictionary data acquired from the database <NUM>.

The robot <NUM> can thereby further estimate the angle of the object obj gripped by the manipulator <NUM> upon recognizing the object obj. This angle indicates, for example, how much the object obj rotates with respect to a reference posture. The robot <NUM> can rotate the object obj by, for example, controlling the manipulator <NUM> on the basis of an angle estimation result and can thereby make the object obj in a desired posture.

The system <NUM> described above is useful at a time of, for example, automating work for arranging or organizing articles using the robot <NUM>. The system <NUM> is also useful for identifying how to rotate the object obj to, for example, read information (a printed code, a radio frequency identifier (RFID), or the like) placed in a predetermined site of the object obj. It is noted that the use application of the system <NUM> is not limited to the examples above but can include other various use applications.

<FIG> is a block diagram depicting a functional configuration of the terminal <NUM> in the system depicted in <FIG>. With reference to <FIG>, the terminal <NUM> includes an image acquisition section <NUM>, an angle information acquisition section <NUM>, and a dictionary data generation section <NUM>. The terminal <NUM> is, for example, a personal computer, a tablet, or a smartphone, and functions of the sections in the terminal <NUM> are realized by a hardware configuration of an information processing device to be described later. Specifically, the functions of, for example, the image acquisition section <NUM>, the angle information acquisition section <NUM>, and the dictionary data generation section <NUM> are realized by a processor included in the information processing device. Dictionary data <NUM> generated by the dictionary data generation section <NUM> is stored in the database <NUM> connected to the terminal <NUM> via the network. A function of the database <NUM> is realized by a storage in one or a plurality of information processing devices connected to the network. It is noted that in a case in which the terminal <NUM> includes a plurality of processors, the plurality of processors may cooperate to realize the functions of the sections described above. Alternatively, a server can realize part of or all of the functions realized by the processors in the terminal <NUM> as described later. The functions of the sections will be described below.

The image acquisition section <NUM> acquires the image of the object obj captured by the camera <NUM>. Here, the camera <NUM> is an example of an imaging device that captures an image of an object. Specifically, the camera <NUM> is, for example, a digital camera including an image sensor, and the image acquisition section <NUM> receives image data generated by the camera <NUM>. While the camera <NUM> is connected to the terminal <NUM> via a wired communication interface such as a universal serial bus (USB) in an example depicted in <FIG>, the camera <NUM> may be connected to the terminal <NUM> via a wireless communication interface such as a Bluetooth (registered trademark) communication interface in another example. Alternatively, the camera <NUM> may be incorporated in the terminal <NUM> and transmit the image data to the image acquisition section <NUM> via a bus.

The angle information acquisition section <NUM> acquires the angle information indicating the angle of the object obj from the camera platform device <NUM>. Here, in the present embodiment, the angle information acquired by the angle information acquisition section <NUM> in the terminal <NUM> indicates an angle of the object obj with reference to a coordinate system of the camera platform device <NUM>. It is noted that the case in which "the angle information acquisition section <NUM> acquires the angle information" also includes a case in which the angle information acquisition section <NUM> generates in itself the angle information regarding the object obj, transmits the angle information to the camera platform device <NUM>, and provides the angle information to the dictionary data generation section <NUM>. In this case, the camera platform device <NUM> sets an angle at which the object obj is held in accordance with the angle information received from the angle information acquisition section <NUM>. In the present embodiment, the camera platform device <NUM> is an example of holding means that holds the object obj. Similarly to the camera <NUM>, the camera platform device <NUM> may be connected to the terminal <NUM> via a wired communication interface or may be connected to the terminal <NUM> via a wireless communication interface.

As described above, the angle of the object obj is the angle in the three-dimensional space, for example, the angle represented by the rotation amounts about the three axes in the orthogonal coordinate system. Owing to this, the angle information acquisition section <NUM> expresses the angle information by, for example, the rotation amounts that correspond to a difference between a current posture of the object obj and the reference posture. Here, the reference posture means, for example, a posture of the object obj when the camera platform device <NUM> is reset. Alternatively, the reference posture may be a posture of the object obj when the image acquisition section <NUM> acquires the image of the object obj for the first time for generating the dictionary data <NUM>.

The dictionary data generation section <NUM> generates the dictionary data <NUM> on the basis of the image acquired by the image acquisition section <NUM>, the identification information regarding the object obj, and the angle information acquired by the angle information acquisition section <NUM>. Here, the identification information regarding the object obj may be identified by any means. For example, the identification information regarding the object obj may be identified on the basis of information input to the terminal <NUM> by a user. Alternatively, the identification information regarding the object obj may be identified by performing matching between the image acquired by the image acquisition section <NUM> and dictionary data separately provided for image-based object recognition. In another alternative, the dictionary data generation section <NUM> may allocate the identification information to the object obj commonly contained in a plurality of images acquired by the image acquisition section <NUM>.

It is noted that an already known technique related to the image-based object recognition can be utilized as appropriate for a combination between the image and the identification information regarding the object obj among information used for generating the dictionary data <NUM> in the present embodiment. For example, the dictionary data generation section <NUM> may extract a feature amount from the image by an appropriate scheme utilized for the image-based object recognition and make the extracted feature amount correspond to the identification information and the angle information regarding the object obj. Alternatively, the dictionary data generation section <NUM> may utilize, for example, the identification information regarding the object obj classified and labeled by an appropriate scheme utilized for the image-based object recognition.

Furthermore, while it is described in the present embodiment that the dictionary data <NUM> is generated on the basis of the identification information regarding the object obj, the dictionary data <NUM> is not necessarily generated on the basis of the identification information regarding the object obj. For example, in a case of providing the system <NUM> for a single type of object obj, it is unnecessary for the dictionary data <NUM> to contain the identification information regarding the objects obj. On the other hand, in a case in which the dictionary data <NUM> contains the identification information regarding the object obj as in the present embodiment, a plurality of types of objects obj are recognized and then the angle of each object obj can be estimated.

Configurations of the camera platform device <NUM> and the jig <NUM> for mounting the object obj to the camera platform device <NUM> which are used together with the terminal <NUM> in the system <NUM> according to the present embodiment will be further described.

<FIG> is a schematic perspective view depicting the configurations of the camera platform device <NUM> and the jig <NUM> used in the system depicted in <FIG>. <FIG> is a cross-sectional view taken along line I-I of <FIG>. With reference to <FIG> and <FIG>, the camera platform device <NUM> includes a base <NUM>, a pair of struts <NUM>, a pair of arms <NUM>, a pair of pins <NUM>, a holder <NUM>, a beam <NUM>, and a control section <NUM>. The jig <NUM> includes a mounting member <NUM>, a coupling member <NUM>, an object holder <NUM>, and a background plate <NUM>. It is noted that the background plate <NUM> is not depicted in <FIG>. The sections will be described below.

In the camera platform device <NUM>, the base <NUM> is, for example, a rotary table and is driven by a motor (not depicted) controlled by the control section <NUM> to rotate about an axis A<NUM>. Here, the axis A<NUM> is orthogonal to an optical axis (denoted as an axis A<NUM> in <FIG>) of the camera <NUM>. The pair of struts <NUM> are fixed to positions symmetrical about the axis A<NUM> on the base <NUM>. Therefore, a midpoint of the pair of struts <NUM> is substantially coincident with the axis A<NUM>. The pair of arms <NUM> are coupled to the pair of struts <NUM> using the pins <NUM>, respectively, on a side opposite to the base <NUM>. The pins <NUM> are located on an axis A<NUM> orthogonal to the axis A<NUM>. The pair of arms <NUM> are pivotally movable about the axis A<NUM>. Specifically, the pair of struts <NUM> and the pins <NUM> are coupled to one another or the pins <NUM> and the pair of arms <NUM> are coupled to one another via gears, and a motor (not depicted) controlled by the control section <NUM> is connected to the gears, whereby the pair of arms <NUM> pivotally move about the axis A<NUM>.

The holder <NUM> is fixed between end portions of the pair of arms <NUM> via the beam <NUM> on a side opposite to the pair of struts <NUM>. While the holder <NUM> is a member to which a camera is mounted in a case, for example, in which the camera platform device <NUM> is used as an automatic camera platform for the camera, the mounting member <NUM> of the jig <NUM> is mounted to the holder <NUM> in the present embodiment as described later. When the pair of arms <NUM> pivotally move about the axis A<NUM> as described above, the holder <NUM> revolves about the axis A<NUM>. In this case, while the holder <NUM> revolves about the axis A<NUM> by the configuration of the pair of arms <NUM> described above, a mounting surface <NUM> of the holder <NUM> is kept in a state of facing the axis A<NUM>.

The control section <NUM> is, for example, a microcontroller incorporated in the camera platform device <NUM>, and controls the rotation of the base <NUM> and the pivotal movement of the pair of arms <NUM> by controlling the motor as described above. The control section <NUM> controls the motor in accordance with, for example, a preset procedure or an instruction from the terminal <NUM>. In this way, the control section <NUM> sets an angle by which the base <NUM> rotates about the axis A<NUM> and an angle by which the pair of arms <NUM> pivotally move about the axis A<NUM>. The angle information acquisition section <NUM> in the terminal <NUM> acquires, for example, information indicating set values of the angles by the control section <NUM> described above as the angle information.

Originally, the camera platform device <NUM> described so far is commercially distributed as a device that automates panning (rotation about the axis A<NUM>) and tilting (revolution about the axis A<NUM>) of the camera mounted to the holder <NUM>. In the present embodiment, it is intended to efficiently generate the dictionary data <NUM> that completely covers various angles by automating the setting of the angle of the object obj utilizing such a camera platform device <NUM>. However, in a case of directly mounting the object obj to the holder <NUM> of the camera platform device <NUM>, pivotally movement of the pair of arms <NUM> causes the holder <NUM> to revolve about the axis A<NUM> and results in a large deviation of a position of the object obj from the optical axis (denoted as the axis A<NUM> in <FIG>) of the camera <NUM>. To address the problem, the object obj is mounted to the camera platform device <NUM> via the jig <NUM> to be described below in the present embodiment.

In the jig <NUM>, the mounting member <NUM> is a member that can be mounted to the holder <NUM> of the camera platform device <NUM>. For example, a mounting structure corresponding to a structure provided in the holder <NUM> for fixing the camera is provided in the mounting member <NUM>. Specifically, in a case of providing a screw in the holder <NUM> for fixing the camera, a screw hole is provided in the mounting member <NUM>. Alternatively, a mounting structure available regardless of the structure of the holder <NUM> may be provided in the mounting member <NUM>. Specifically, a clip sandwiching the holder <NUM>, a belt wound around the holder <NUM>, or the like may be provided in the mounting member <NUM>.

The object holder <NUM> is a member to which the object obj can be mounted. For example, a mounting structure that can fix the object obj while making a contact area of the mounting structure with the object obj as small as possible is provided in the object holder <NUM>. This is because the contact area between the mounting structure and the object obj can act as an occlusion region in the image of the object obj captured by the camera <NUM>. Specifically, a clip sandwiching the object obj, a hook gripping the object obj, an adhesive surface to which the object obj is adhesively attached, or the like may be provided in the object holder <NUM>. Furthermore, a magnet may be provided in the object holder <NUM> for the object obj that is a magnetic material.

The coupling member <NUM> couples the mounting member <NUM> to the object holder <NUM>. Furthermore, the coupling member <NUM> specifies a position relationship between the mounting member <NUM> and the object holder <NUM> so that the object obj mounted to the object holder <NUM> is located near an intersecting point between the axes A<NUM> and A<NUM> when the mounting member <NUM> is mounted to the holder <NUM> of the camera platform device <NUM>. For example, the coupling member <NUM> is coupled to the mounting member <NUM> so that the coupling member <NUM> extends along the pair of arms <NUM> when the mounting member <NUM> is mounted to the holder <NUM>. At this time, a length of the coupling member <NUM> along the pair of arms <NUM> is nearly equal to a value obtained by subtracting thicknesses of the mounting member <NUM> and the object holder <NUM> and a half of a thickness of the object obj from a distance between the holder <NUM> and the axis A<NUM>. The coupling member <NUM> may have a structure that makes adjustable the length thereof in the direction along the arms <NUM>. It is thereby possible to adjust the length of the coupling member <NUM> in accordance with a size of the object obj and to make a center of the object obj close to the intersecting point between the axes A<NUM> and A<NUM>.

The object obj mounted to the camera platform device <NUM> via the jig <NUM> as described above is located near the intersecting point between the axes A<NUM> and A<NUM>. Owing to this, even when the base <NUM> of the camera platform device <NUM> rotates about the axis A<NUM> or even when the pair of arms <NUM> pivotally move about the axis A<NUM>, the position of the object obj hardly changes and does not largely deviate from the optical axis (denoted as the axis A<NUM> in <FIG>) of the camera <NUM>. In the present embodiment, therefore, when the control section <NUM> of the camera platform device <NUM> sets the angle by which the base <NUM> rotates about the axis A<NUM> and the angle by which the pair of arms <NUM> pivotally move about the axis A<NUM>, these angles can be regarded as the rotation amounts of the object obj about the axes A<NUM> and A<NUM>.

In a case of using the camera platform device <NUM> and the jig <NUM> described above, the object obj cannot be rotated about the axis A<NUM> orthogonal to the axes A<NUM> and A<NUM>, that is, about the optical axis of the camera <NUM> but the rotation about the axis A<NUM> can be accurately complemented by subjecting the image captured by the camera <NUM> to plane rotation. Furthermore, while it is described above that the object obj is on the optical axis of the camera <NUM> for the brevity, the object obj is not necessarily on the optical axis of the camera <NUM>.

The background plate <NUM> is mounted to the coupling member <NUM> or the object holder <NUM> and provides a background of the object obj. For example, a mounting structure for selectively mounting a screen may be provided in the background plate <NUM>. The screen can include, for example, a plurality of screens formed from different materials. The materials can include, for example, paper, a cloth, and a film. Moreover, the screen may include a plurality of screens having different colors or different reflection characteristics. Replacing the screen makes it possible to replaceably provide a plurality of backgrounds of the object obj different in material, color, reflection characteristics, or the like. Alternatively, the background plate <NUM> may be mounted, for example, detachably to the coupling member <NUM> or the object holder <NUM>. In this case, selectively mounting the plurality of background plates <NUM> makes it possible to replaceably provide a plurality of backgrounds of the object obj different in material, color, reflection characteristics, or the like. Specifically, the background plate <NUM> can include, for example, a plurality of background plates <NUM> having surfaces that face the object obj and that are formed from different materials. The materials can include, for example, paper, a cloth, and a film. Moreover, the background plate <NUM> may include a plurality of background plates <NUM> having surfaces that face the object obj and that differ in color or reflection characteristics.

<FIG> is a conceptually explanatory diagram of the dictionary data generated in the first embodiment of the present invention. <FIG> exemplarily depicts the dictionary data <NUM> made to correspond to the object obj (connector in an example depicted in <FIG>) identified by certain identification information. In the example depicted in <FIG>, an angle of the object obj is a vector quantity represented by the rotation amounts about three axes (X-axis, Y-axis, and Z-axis) of the orthogonal coordinate system in the three-dimensional space. The dictionary data <NUM> includes, for the angle of the object obj, at least NX × NY × NZ elements defined by splitting a perimeter into NX elements for a rotation amount (rot_x) about the X-axis, splitting the perimeter into NY elements for a rotation amount (rot_Y) about the Y-axis, and splitting the perimeter into NZ elements for a rotation amount (rot_Z) about the Z-axis. Each element is made to correspond to information corresponding to at least one image of the object obj. Here, the information corresponding to the image of the object obj can be a feature amount extracted from the image captured by the camera <NUM> when the angle of the object obj is, for example, represented by the rotation amounts (rot_X, rot_Y, rot_Z).

In the above example, split widths of the rotation amounts (rot_X, rot_Y, rot_Z) about the axes may differ (that is, at least any of NX, NY, and NZ may differ from the others). Furthermore, the rotation amounts are not necessarily equally split. For example, in a case of the presence of an angle difficult to estimate with high reliability in estimating the angle of the object obj to be described later, the split width of the rotation amounts near the rotation amount corresponding to the angle may be set smaller than those of the other parts.

For example, in a case in which the camera <NUM> of the robot <NUM> captures an image of the object obj at an unknown angle, the angle of the object obj can be estimated by conducting matching between a feature amount extracted from the captured image and a feature amount made to correspond to the element in the dictionary data <NUM>.

Here, the dictionary data <NUM> may include a plurality of elements generated on the basis of angle information regarding the same object obj and a plurality of different images of the same object obj. In this case, the number of elements in the dictionary data <NUM> is more than NX × NY × NZ. The plurality of images made to correspond to the same angle information may have, for example, different environmental conditions at a time of capture. The environmental condition can be, for example, a background or a light placement. Generating the dictionary data <NUM> on a plurality of different environmental conditions makes it possible to provide the dictionary data <NUM> with which it is possible to estimate the angle of the object obj on various environmental conditions.

In the above case, the image acquisition section <NUM> in the terminal <NUM> acquires a plurality of different images of the object obj. For example, the image acquisition section <NUM> may acquire images of the object obj when the control section <NUM> of the camera platform device <NUM> sets the same angle before and after replacement of the background of the object obj using the background plate <NUM> of the jig <NUM>. In this case, the dictionary data generation section <NUM> generates a plurality of elements in the dictionary data <NUM> on the basis of a plurality of images having different backgrounds, identification information regarding the object obj common to the plurality of images, and angle information indicating the angle of the object obj common to the plurality of images.

<FIG> is an explanatory diagram of a schematic configuration of the robot <NUM> in the system depicted in <FIG>. With reference to <FIG>, the robot <NUM> includes the camera <NUM>, the manipulator <NUM>, a control section <NUM>, a sensor <NUM>, and a motor <NUM>. The robot <NUM> can grip the object obj using, for example, the manipulator <NUM> under control of the control section <NUM>, and capture the image of the object obj using the camera <NUM>. In the present embodiment, the manipulator <NUM> is an example of holding means that holds the object obj similarly to the camera platform device <NUM> described above. The control section <NUM> is realized by, for example, the hardware configuration of the information processing device to be described later.

The sensor <NUM> includes a sensor for acquiring various measurement values used in the robot <NUM> or transmitted from the robot <NUM> to the other device. Specifically, the sensor <NUM> may include, for example, an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, and/or a global navigation satellite system (GNSS) receiver. Furthermore, the sensor <NUM> may include a depth sensor or a laser range scanner such as a laser imaging detection and ranging (LIDAR).

The motor <NUM> actuates the sections in the robot <NUM> under control of the control section <NUM>. The motor <NUM> can include, for example, a motor (actuator) for changing a posture of the robot <NUM> or moving the robot <NUM> by actuating a joint structure (not depicted). Furthermore, the motor <NUM> may include a motor for rotating wheels and moving the robot <NUM>. It is noted that the sections including the motor <NUM> in the robot <NUM> can be configured appropriately on the basis of an already known robot design scheme. Here, the robot <NUM> does not necessarily change the posture or move. Likewise, the robot <NUM> does not necessarily include the joint structure (other than the manipulator <NUM>) or the wheels.

<FIG> is a block diagram depicting a functional configuration of the robot <NUM> in the system depicted in <FIG>. With reference to <FIG>, the robot <NUM> includes not only the camera <NUM> and the manipulator <NUM> but also an image acquisition section <NUM>, a dictionary data acquisition section <NUM>, an object recognition/angle estimation section <NUM>, a result output section <NUM>, a dictionary data update section <NUM>, a manipulator control section <NUM>, and an angle information acquisition/angle estimation section <NUM>. The sections other than the camera <NUM> and the manipulator <NUM> are realized by, for example, the processor in the information processing device that realizes the control section <NUM> of the robot <NUM> described above. It is noted that in a case in which the control section <NUM> includes a plurality of processors, the plurality of processors may cooperate to realize the functions of the sections described above. Alternatively, a server can realize part of or all of the functions realized by the processors in the control section <NUM> as described later. The functions of the sections will be described below. It is noted that a function related to update of the dictionary data will be described later in detail with reference to flowcharts and is, therefore, described herein briefly.

The image acquisition section <NUM> acquires the image of the object obj captured by the camera <NUM>. In the present embodiment, the camera <NUM> is an example of an imaging device that captures an image of an object similarly to the camera <NUM> described above. The image captured by the camera <NUM> and that captured by the camera <NUM> differ from each other although the images contain the object obj of the same type. Specifically, the camera <NUM> is, for example, a digital camera including an image sensor and the image acquisition section <NUM> receives image data generated by the camera <NUM>. For example, the robot <NUM> grips the object obj using the manipulator <NUM>. In this case, the image acquired by the image acquisition section <NUM> contains the object obj gripped by the manipulator <NUM>. Alternatively, the image acquisition section <NUM> may contain the object obj that is not gripped by the manipulator <NUM> but that is placed on a table, a floor, or the like. While the camera <NUM> is incorporated in the robot <NUM> and transmits the image data to the image acquisition section <NUM> via a bus in the example depicted in <FIG>, the camera <NUM> may be externally connected to the robot <NUM> via a wired communication interface or a wireless communication interface.

The dictionary data acquisition section <NUM> acquires the dictionary data <NUM> from the database <NUM> connected to the robot <NUM> via the network. As described above, the dictionary data <NUM> is generated on the basis of the image of the object obj and the angle information regarding the object obj (as well as the identification information regarding the object obj). The robot <NUM> utilizes the dictionary data <NUM> to estimate the angle of the object obj gripped by the manipulator <NUM>. It is noted that the dictionary data acquisition section <NUM> does not necessarily acquire the entire dictionary data <NUM>. For example, in a case in which the dictionary data <NUM> is generated for a plurality of types of objects and the object obj contained in the image acquired by the image acquisition section <NUM> is already identified, the dictionary data acquisition section <NUM> selectively acquires an element made to correspond to the identification information regarding the object obj in the dictionary data <NUM>.

The object recognition/angle estimation section <NUM> estimates the angle of the object obj in the image on the basis of the image of the object obj acquired by the image acquisition section <NUM> and the dictionary data <NUM> acquired by the dictionary data acquisition section <NUM>. In a case in which the dictionary data <NUM> is generated for a plurality of types of objects and the object obj contained in the image acquired by the image acquisition section <NUM> is not identified, the object recognition/angle estimation section <NUM> identifies the identification information regarding the object obj by image-based object recognition. Since the already known technique can be applied to the image-based object recognition, the image-based object recognition will not be described in detail. For example, in a case in which the dictionary data <NUM> is generated for a single type of object or the object obj contained in the image acquired by the image acquisition section <NUM> is already identified, the object recognition/angle estimation section <NUM> does not execute object recognition.

On the other hand, the object recognition/angle estimation section <NUM> executes estimation of the angle of the object obj by, for example, performing matching between the image acquired by the image acquisition section <NUM> and an element in the dictionary data <NUM>. In this case, the angle made to correspond to the element having a highest matching score in the dictionary data <NUM> is estimated as the angle of the object obj in the image. As described later, the dictionary data <NUM> for estimating the angle of the object obj can include many elements. Thus, the object recognition/angle estimation section <NUM> may prunes the dictionary data <NUM> on the basis of the image acquired by the image acquisition section <NUM> and execute matching between the pruned dictionary data <NUM> and the image. In the present embodiment, pruning is a process for determining the dictionary data <NUM> not to be subjected to matching by a process lighter in processing load than the matching for estimating the angle of the object obj.

The result output section <NUM> outputs a result of recognition by the object recognition/angle estimation section <NUM>. As described above, while the robot <NUM> may utilize the estimation result of the angle of the object obj for operations of the robot <NUM> in itself, for example, for control over the manipulator <NUM>, the robot <NUM> may output the estimation result in a format of some sort as needed. More specifically, the estimation result may be displayed as an image on a display of the robot <NUM> or output as a sound from a loudspeaker. Furthermore, the estimation result may be further transmitted to another device from a communication device owned by the robot <NUM> via the network. The result output section <NUM> controls output of the estimation result described above. In a case of no need to output the estimation result, the result output section <NUM> is not provided.

The dictionary data update section <NUM> updates the dictionary data <NUM> in response to the estimation result of the angle of the object obj by the object recognition/angle estimation section <NUM> and a result of re-estimation of the angle by the angle information acquisition/angle estimation section <NUM> to be described later. More specifically, in a case in which the reliability of the angle estimated by the object recognition/angle estimation section <NUM> does not exceed a threshold, the dictionary data update section <NUM> updates the dictionary data <NUM> on the basis of the result of the re-estimation of the angle by the angle information acquisition/angle estimation section <NUM>. It is noted that an angle estimation function by the object recognition/angle estimation section <NUM> will be also referred to as "first angle estimation function," and that an angle re-estimation function by the angle information acquisition/angle estimation section <NUM> will be also referred to as "second angle estimation function" in the following description. These angle estimation functions are not necessarily carried out independently of each other. For example, the angle information acquisition/angle estimation section <NUM> utilizes the estimation result of the angle by the object recognition/angle estimation section <NUM> in re-estimating the angle. In other words, the "first angle estimation function" is often executed solely and the "second angle estimation function" often calls the "first angle estimation function.

The manipulator control section <NUM> controls the manipulator <NUM> gripping the object obj in the robot <NUM>. When the dictionary data update section <NUM> executes the update of the dictionary data <NUM>, the manipulator control section <NUM> rotates the object obj by controlling the manipulator <NUM>. It is noted that rotation mentioned herein means a change in the angle of the object obj. The rotation of the object obj is an example of a physical operation related to the object obj and executed in re-estimating the angle of the object obj.

The angle information acquisition/angle estimation section <NUM> acquires angle information indicating the angle of the object obj from the manipulator control section <NUM>. Here, in the present embodiment, the angle information acquired by the angle information acquisition/angle estimation section <NUM> in the robot <NUM> indicates the angle of the object obj based on a coordinate system of the robot <NUM> or the manipulator <NUM>. In the present embodiment, therefore, the angle information acquired from the manipulator control section <NUM> is not necessarily made to directly correspond to the angle information in the dictionary data <NUM>. Thus, in the present embodiment, the angle information acquisition/angle estimation section <NUM> calculates a rotation amount Δθ of the object obj from the angle information before and after the rotation of the object obj by control by the manipulator control section <NUM> over the manipulator <NUM>, and utilizes the rotation amount Δθ in re-estimation of the angle to be described later.

Furthermore, the angle information acquisition/angle estimation section <NUM> re-estimates an angle θ<NUM> (simply represented as θ<NUM> = θ<NUM> - Δθ) of the object obj in an image (first image) before the rotation of the object obj, on the basis of an angle θ<NUM> of the object obj, which is estimated by the object recognition/angle estimation section <NUM> on the basis of an image (second image) after the rotation of the object obj and the dictionary data <NUM>, and on the basis of the rotation amount Δθ. Here, the rotation amount Δθ is an example of an amount of the physical operation related to the object obj. It is noted that each of the angle θ<NUM>, the angle θ<NUM>, and the rotation amount Δθ can be a vector quantity containing, for example, elements of rotations (rot_X, rot_Y, rot_Z in the example of <FIG>) about the axes of the coordinate system.

In a case in which the reliability of the angle θ<NUM> of the object obj estimated by the object recognition/angle estimation section <NUM> on the basis of the image (second image) after the rotation of the object obj and the dictionary data <NUM> exceeds the threshold, the dictionary data update section <NUM> updates the dictionary data <NUM> on the basis of the angle information indicating the angle θ<NUM>, which is re-estimated by the angle information acquisition/angle estimation section <NUM> on the basis of this, and the image (first image) before the rotation of the object obj.

On the other hand, in a case in which the reliability of the angle θ<NUM> estimated by the object recognition/angle estimation section <NUM> on the basis of the image (second image) after the rotation of the object obj and the dictionary data <NUM> does not exceed the threshold, then the manipulator control section <NUM> controls the manipulator <NUM> to further rotate the object obj by a rotation amount Δθ', and the object recognition/angle estimation section <NUM> estimates an angle θ<NUM> of the object obj on the basis of an image (third image) after the rotation of the object obj and the dictionary data <NUM>. In a case in which the reliability of the angle θ<NUM> exceeds the threshold, then the angle information acquisition/angle estimation section <NUM> re-estimates the angle θ<NUM> on the basis of the angle θ<NUM> and a total rotation amount (Δθ + Δθ'), and the dictionary data update section <NUM> updates the dictionary data <NUM> on the basis of this result.

In this way, the dictionary data update section <NUM> updates the dictionary data <NUM> on the basis of the angle θ<NUM> and the image (first image) before the rotation of the object obj upon re-estimation of the angle θ<NUM> with the sufficient reliability. Specifically, the dictionary data update section <NUM> adds an element to the dictionary data <NUM> or substitutes an element for an element in the dictionary data <NUM>. This increases a probability that the angle θ<NUM> can be estimated with high reliability without depending on the re-estimation when the camera <NUM> subsequently captures an image of the object obj at the angle θ<NUM> on a similar environmental condition.

An example of process flows in the system <NUM> according to the present embodiment will be described below with reference to <FIG>.

<FIG> is a flowchart depicting an example of a dictionary data generation process in the first embodiment of the present invention. With reference to <FIG>, in a registration process, first, the image acquisition section <NUM> in the terminal <NUM> acquires an image (Step S101) and the angle information acquisition section <NUM> acquires angle information (Step S103). Any of Steps S101 and S103 may be executed earlier or Steps S101 and S103 may be executed in parallel. For example, the angle information acquisition section <NUM> may acquire the angle information from the camera platform device <NUM> with the acquisition of the image captured by the camera <NUM> by the image acquisition section <NUM> in real time as a trigger. Alternatively, the image acquisition section <NUM> may acquire the image captured by the camera <NUM> in real time with the transmission of the angle information to the camera platform device <NUM> by the angle information acquisition section <NUM> as a trigger. In another alternative, the image acquisition section <NUM> may consecutively acquire images captured by the camera <NUM> in time series and the angle information acquisition section <NUM> may consecutively acquire angle information set to the camera platform device <NUM> in time series.

Next, the dictionary data generation section <NUM> in the terminal <NUM> makes the image acquired in Step S101 and the angle information acquired in Step S103 correspond to each other. For example, in a case in which both the image and the angle information are acquired in real time, the dictionary data generation section <NUM> makes the image and the angle information acquired substantially simultaneously correspond to each other. On the other hand, in a case in which the image and the angle information are acquired at different times or later, the dictionary data generation section <NUM> makes the image and the angle information having a common key correspond to each other. The key in this case may be, for example, a time stamp or may be a sequence number assigned separately from the time stamp.

Next, the dictionary data generation section <NUM> generates the dictionary data <NUM> on the basis of the image and the angle information made to correspond to each other in Step S105 (Step S107). Here, as already described, the dictionary data generation section <NUM> may generate the dictionary data <NUM> by applying the already known image-based object recognition technique. Furthermore, in a case, for example, in which pairs of substantially same images and substantially same angle information are acquired consecutively, the dictionary data generation section <NUM> may determine that the acquired images and angle information as redundant information and omit generation of the dictionary data <NUM>.

<FIG> is a flowchart depicting an example of a recognition process in the first embodiment of the present invention. With reference to <FIG>, in the recognition process, first, the image acquisition section <NUM> in the robot <NUM> acquires an image (Step S301). As described above, the image acquisition section <NUM> acquires the image captured by the camera <NUM> and this image contains, for example, the object obj gripped by the manipulator <NUM>. Next, the dictionary data acquisition section <NUM> acquires the dictionary data <NUM> from the database <NUM> (Step S303).

Next, the object recognition/angle estimation section <NUM> recognizes the object obj on the basis of the image acquired in Step S301 and the dictionary data <NUM> acquired in Step S303 (Step S305). It is noted that the image-based object recognition will not be described in detail since the already known technique can be applied to the image-based object recognition. Furthermore, in the case, for example, in which the dictionary data <NUM> is generated for a single type of object or in which the object obj contained in the image is already identified as described above, the object recognition in Step S305 is omitted.

Next, the object recognition/angle estimation section <NUM> executes pruning of the dictionary data <NUM> (Step S307). For example, in a case of generating the dictionary data <NUM> depicted in <FIG> by splitting the perimeter into <NUM> for the rotation amounts (rot_X, rot_Y, rot_Z) about the axes (that is, NX = NY = NZ = <NUM>), the dictionary data <NUM> having at least <NUM><NUM> = <NUM>,<NUM> elements is generated. In a case of generating the dictionary data <NUM> by making a plurality of different images correspond to the same angle as described above, the number of elements further increases. Since a processing load for executing matching for all the elements in such dictionary data <NUM> is quite heavy, an advantage of the pruning of the dictionary data <NUM> is high.

<FIG> is a flowchart depicting an example of a pruning process in the first embodiment of the present invention. <FIG> is a conceptually explanatory diagram of the pruning process depicted in <FIG>. With reference to <FIG>, the object recognition/angle estimation section <NUM> first determines a pruning procedure corresponding to the object obj (Step S331). The pruning procedure corresponding to the object obj is, for example, determined in advance and the pruning procedure together with the dictionary data <NUM> is stored in the database <NUM>. In a case of executing Step S305 depicted in <FIG> described above, the object recognition/angle estimation section <NUM> determines the pruning procedure in accordance with an object recognition result in Step S305.

Subsequent Steps S333 and S335 are an example of processes executed in accordance with the pruning procedure corresponding to the object in an example depicted in <FIG>. The processes to be executed can vary depending on the type of object. In the above example, the object recognition/angle estimation section <NUM> masks the image (Step S333) and furthermore performs color subtraction on the image (Step S335). Next, the object recognition/angle estimation section <NUM> executes pruning (Step S337). In the example depicted in <FIG>, for example, a plurality of feature portions are extracted from the image which has been masked and subjected to the color subtraction, and the elements that do not have a position relationship of the plurality of similarly extracted feature portions common to the image are excluded from matching targets among the dictionary data <NUM>.

In the example depicted in <FIG>, the object obj is connectors. In the example depicted therein, a pruning procedure that pays attention to colors of cables (cable <NUM> to cable <NUM>) is set. In Step S333 depicted in <FIG>, portions other than the cables in the image are masked (mask is denoted as MSK in <FIG>). This eliminates an influence of a shadow of a terminal cover present in the masked portions. Furthermore, although not expressed in <FIG>, the image is subjected to the color subtraction in Step S335 on condition that a color difference between the cables on two sides (cable <NUM> and cable <NUM>) can be expressed. This can facilitate extracting the cables (cable <NUM> and cable <NUM>) on the two ends as two feature portions from the image and each element in the dictionary data <NUM>.

Moreover, in Step S337 depicted in <FIG>, pruning of the dictionary data <NUM> is executed on the basis of the image which has been masked and subjected to the color subtraction. Specifically, cable <NUM>, for example, is located upper right in a view from cable <NUM>. On the other hand, in an element group 210b (in which the connectors rotate about a point-of-view axis) in the dictionary data <NUM>, cable <NUM> is located upper left in a view from cable <NUM>. Furthermore, in an element group 210c (in which the connectors are turned inside out), cable <NUM> is located lower left in a view from cable <NUM>. In Step S337, therefore, the element groups 210b and 210c are excluded from the matching targets. As a result, the matching is executed with only an element group 210a (in which cable <NUM> is located upper right in a view from cable <NUM> similarly to the image) set as targets.

With reference back to <FIG>, after pruning of the dictionary data <NUM> in Step S307, the object recognition/angle estimation section <NUM> executes matching between the image and the dictionary data <NUM> (Step S309). The matching can be, for example, template matching. It is noted that image matching will not be described in detail since the already known technique can be applied to the matching. While a score of each object is calculated as a result of the matching in the already known image-based object recognition, a score of each angle of the object is calculated in Step S307.

Next, the object recognition/angle estimation section <NUM> estimates the angle of the object obj on the basis of the result of the matching in Step S309 (Step S311). An estimation result in Step S311 can be, for example, an angle indicated by the angle information made to correspond to an element for which a highest score is calculated in the dictionary data <NUM> in the matching in Step S309.

Next, the object recognition/angle estimation section <NUM> determines whether or not the score calculated in the matching in Step S309 exceeds a threshold (Step S313). Here, the score to be compared with the threshold is, for example, a highest matching score. Alternatively, the object recognition/angle estimation section <NUM> may determine whether or not what % (for example, <NUM>%) of higher matching scores exceed the threshold. In a case in which the matching score does not exceed the threshold (NO) in determination in Step S313, the dictionary data update section <NUM> updates the dictionary data <NUM> (Step S315). On the other hand, in a case in which the matching score exceeds the threshold (YES) in the determination in Step S313, a process for updating the dictionary data <NUM> may not be executed. The result output section <NUM> outputs a result of estimation in Step S311 as needed.

<FIG> is a flowchart depicting an example of a dictionary data update process in the first embodiment of the present invention. With reference to <FIG>, in the update process, first, the angle information acquisition/angle estimation section <NUM> in the robot <NUM> stores the angle information regarding the object obj provided from the manipulator control section <NUM> (Step S351). Here, the angle stored in Step S351 indicates the angle of the object obj in the coordinate system with reference to, for example, the manipulator <NUM>. Next, the manipulator control section <NUM> rotates the object obj by controlling the manipulator <NUM> (Step S353).

After the object obj is rotated, the angle of the object obj is estimated (S355). A process in Step S355 corresponds to, for example, processes in Steps S301 to S311 depicted in <FIG>. Specifically, the image acquisition section <NUM> acquires the image (second image) after the rotation of the object obj, and the object recognition/angle estimation section <NUM> estimates the angle of the object obj in the image (second image) after the rotation. It is noted that the dictionary data <NUM> acquired in previously executed Step S303 may be utilized and the object obj may be handled as being already recognized in previously executed Step S305.

Next, the dictionary data update section <NUM> determines whether or not a matching score in estimation in Step S355 exceeds the threshold (Step S357). This determination can be executed similarly to, for example, Step S313 depicted in <FIG>. In a case in which the matching score does not exceed the threshold (NO) in determination in Step S357, the processes in Steps S353 and S355 are re-executed. In other words, the manipulator control section <NUM> rotates the object obj by controlling the manipulator <NUM> (Step S353), and the object recognition/angle estimation section <NUM> estimates the angle of the object obj in the image (third image) after the rotation (Step S355).

On the other hand, in a case in which the matching score exceeds the threshold (YES) in the determination in Step S357, the angle information acquisition/angle estimation section <NUM> re-estimates the initial angle θ<NUM> from the angle θ<NUM> estimated in Step S355 and the rotation amount Δθ of the object obj (Step S359). Here, the initial angle θ<NUM> is the angle before the rotation of the object obj, which is the angle that cannot be estimated by the object recognition/angle estimation section <NUM> with sufficient reliability. On the other hand, the angle θ<NUM> is the angle of the object obj estimated by the object recognition/angle estimation section <NUM> on the basis of the image (second image) after the rotation of the object obj and the dictionary data <NUM>, and it is proved in the determination in Step S357 that the angle θ<NUM> is estimated with sufficient reliability. Furthermore, the rotation amount Δθ is calculated on the basis of the angle information regarding the object obj stored in Step S351 and the angle information regarding the object obj provided from the manipulator control section <NUM> at timing of Step S353.

In a case in which processes in Steps S353 and S355 are repeated N times as a result of the determination in Step S357, the angle information acquisition/angle estimation section <NUM> re-estimates the initial angle θ<NUM> from an angle θN+<NUM> estimated in finally executed Step S355 and a total rotation amount ΔθTTL of the object obj in Step S353 executed N times. The total rotation amount ΔθTTL is calculated on the basis of the angle information regarding the object obj stored in Step S351 and the angle information regarding the object obj provided from the manipulator control section <NUM> at the timing of Step S353.

Next, the dictionary data update section <NUM> makes the angle information corresponding to the initial angle θ<NUM> re-estimated in Step S359 and the image (first image) before the rotation of the object obj acquired in Step S301 depicted in <FIG> correspond to each other (Step S361). Furthermore, the dictionary data update section <NUM> updates the dictionary data <NUM> on the basis of the image and the angle information made to correspond to each other in Step S361 (Step S363). Here, update of the dictionary data <NUM> includes addition of an element to the dictionary data <NUM> and/or substitution of an element for an element in the dictionary data <NUM>.

In Step S363 described above, the dictionary data update section <NUM> adds an element to the dictionary data <NUM> on the basis of the image and the angle information. This increases the probability that the angle θ<NUM> can be estimated with high reliability when the camera <NUM> in the robot <NUM> subsequently captures an image of the object obj at the angle θ<NUM> on a similar environmental condition. In a case, for example, in which the dictionary data <NUM> is dedicated to the robot <NUM> and in which it is predicted that the environmental condition on which the camera <NUM> captures an image of the object obj does not change, the dictionary data update section <NUM> may substitute an element for an element in the dictionary data <NUM> on the basis of the image and the angle information.

As described so far, updating the dictionary data <NUM> makes it possible to accumulate additional dictionary data <NUM> for the angle of the object obj or for the environmental condition for which the angle of the object obj is difficult to estimate with high reliability using the initially generated dictionary data <NUM>. In this way, the robot <NUM> that estimates the angle of the object obj using the dictionary data <NUM> autonomously enhances the dictionary data <NUM>, thereby making it possible to improve estimation robustness.

Here, the dictionary data update process described above with reference to <FIG> may include a verification process before the update of the dictionary data <NUM> as an additional process. As a first example, before Step S351 depicted in <FIG>, a process (denoted as "verification process <NUM>" in Step S371) for verifying whether or not to execute the dictionary data update process may be executed. In the verification process according to the first example, the image acquisition section <NUM> reacquires an image of the object obj before the object obj is rotated in Step S353. The object recognition/angle estimation section <NUM> estimates the angle of the object obj in the reacquired image. In a case in which a matching score in this estimation exceeds the threshold (unlike the estimation in Step S311 depicted in <FIG>), then the dictionary data update process is halted and at least the update of the dictionary data in S363 is not executed.

For example, in the image acquired by the image acquisition section <NUM> in Step S301 depicted in <FIG> described above, an accidental factor such as a focus delay of the camera <NUM> or an instantaneous change in an illumination condition of the camera <NUM> (due to, for example, a thunder or flash light) causes an unexpected change in the image, which possibly causes reduction in estimation reliability. The verification process as in the first example above is effective for preventing the dictionary data <NUM> from being updated on the basis of low reproducibility information due to the accidental factor.

Furthermore, as a second example, after Step S361 depicted in <FIG>, a process (denoted as "verification process <NUM>" in Step S373) for verifying whether or not to update the dictionary data on the basis of the prepared angle information and image may be executed. In the verification process according to the second example, the dictionary data update section <NUM> generates provisional dictionary data based on the angle information and the image made to correspond to each other in Step S361. Next, the manipulator control section <NUM> controls the manipulator <NUM> and rotates the object obj in an opposite direction to that in Step S353. The angle of the object obj is thereby returned to the original angle θ<NUM>. Furthermore, the image acquisition section <NUM> newly acquires an image of the object obj the angle of which is returned to the original angle θ<NUM>, and the object recognition/angle estimation section <NUM> estimates an angle of the object obj in the image, which is newly acquired by the image acquisition section <NUM>, on the basis of the provisional dictionary data generated by the dictionary data update section <NUM>. Here, in a case in which the original angle θ<NUM> can be estimated and the matching score exceeds the threshold, the dictionary data update section <NUM> executes the update of the dictionary data <NUM> in Step S363. Otherwise, the update of the dictionary data <NUM> in Step S363 is not executed.

The above second example is effective for preventing, for example, the update of the dictionary data <NUM> that does not contribute to improving estimation reliability of the angle. Even if the image acquisition section <NUM> updates the dictionary data <NUM> on the basis of the acquired image, the estimation reliability of the angle in a subsequently acquired similar image does not necessarily and possibly improve depending on the environmental condition on which the camera <NUM> captures an image of the object obj. The verification process as in the above second example is effective for preventing an increase in a capacity of the dictionary data <NUM> by unnecessary elements that do not necessarily contribute to improving the angle estimation reliability.

While the angle information acquisition/angle estimation section <NUM> re-estimates the angle after the object obj is rotated in the above examples, the angle information acquisition/angle estimation section <NUM> may re-estimate the angle after the robot <NUM> together with the object obj is moved by the motor <NUM>. There is a probability that the environmental condition on which the camera <NUM> captures an image changes by movement of the robot <NUM> and that angle estimation with high reliability can be performed without rotating the object obj. It is noted that a configuration for moving the robot <NUM> is described in more detail in a third embodiment to be described later.

Moreover, the movement of the robot <NUM> described above may be combined with the rotation of the object obj. For example, the angle information acquisition/angle estimation section <NUM> may re-estimate an angle after the robot <NUM> together with the object obj is moved in a case in which sufficient reliability cannot be ensured even in the re-estimation of the angle after rotating the object obj. For example, in a case in which the environmental condition on which the camera <NUM> captures an image of the object obj greatly differs from the environmental condition of the camera <NUM> at a time of generating the dictionary data <NUM>, a re-estimation process described above can be effective.

Functions of the system <NUM> according to the present embodiment are realized by being distributed to the terminal <NUM>, the database <NUM>, and the robot <NUM> in the example depicted in <FIG>, <FIG>, and <FIG>. In another example, most of the functions of the system <NUM> may be realized in the server. In other words, the functions described to be realized by the processors in the terminal <NUM> and the robot <NUM> in the above example may be realized by a processor in the server that includes the database <NUM>. In this case, the terminal <NUM> transmits the image of the object obj captured by the camera <NUM> and the angle information regarding the object obj acquired from the camera platform device <NUM> to the server, and the server generates the dictionary data <NUM> by associating the image with the angle information. On the other hand, the robot <NUM> transmits the image of the object obj captured by the camera <NUM> to the server, and the server estimates the angle of the object obj on the basis of this image. The robot <NUM> receives the angle estimation result from the server. The server may request the robot <NUM> to rotate the object obj and to acquire the image of the object obj after the rotation for re-estimation of the angle in a case in which the reliability of the estimated angle does not exceed a threshold. It is noted that the number of servers realizing these functions is not necessarily one but that a plurality of servers distributed on a network may realize the above functions. Moreover, the server realizing the functions may be a different device from a storage including the database <NUM>.

A second embodiment of the present invention will next be described. It is noted that the description of sections configured similarly to those in the first embodiment described above is often omitted by designating common reference symbols.

<FIG> is a block diagram depicting a functional configuration of a robot 300a according to the second embodiment of the present invention. With reference to <FIG>, the overall functions related to the generation of the dictionary data <NUM> and the estimation of the angle of the object obj using the dictionary data <NUM> are realized by the robot 300a in the present embodiment. Specifically, the processor in the control section <NUM> of the robot 300a realizes an image acquisition section <NUM> or <NUM>, an angle information acquisition/angle estimation section <NUM> or <NUM>, a dictionary data generation/update section <NUM> or <NUM>, the dictionary data acquisition section <NUM>, the object recognition/angle estimation section <NUM>, the result output section <NUM>, and the manipulator control section <NUM>. It is noted that in a case in which the control section <NUM> includes a plurality of processors, the plurality of processors may cooperate to realize the functions of the sections described above. Furthermore, as described later, part of or all of the functions realized by the processors in the control section <NUM> can be realized by the server. Moreover, the database <NUM> is stored in a storage of the control section <NUM> in the robot 300a. The sections will further be described below.

The image acquisition section <NUM> or <NUM> has the functions of both the image acquisition section <NUM> described above with reference to <FIG> and the image acquisition section <NUM> described with reference to <FIG>. In other words, the image acquisition section <NUM> or <NUM> provides the image of the object obj captured by the camera <NUM> to the dictionary data generation/update section <NUM> or <NUM> when the dictionary data <NUM> is generated, and to the object recognition/angle estimation section <NUM> when the angle of the object obj is estimated using the dictionary data <NUM>.

The angle information acquisition/angle estimation section <NUM> or <NUM> has the functions of both the angle information acquisition section <NUM> described above with reference to <FIG> and the angle information acquisition/angle estimation section <NUM> described with reference to <FIG>. In other words, the angle information acquisition/angle estimation section <NUM> or <NUM> provides the angle information acquired from the manipulator control section <NUM> to the dictionary data generation/update section <NUM> or <NUM> when the dictionary data <NUM> is generated. In addition, the angle information acquisition/angle estimation section <NUM> or <NUM> calculates the rotation amount Δθ of the object obj on the basis of the angle information acquired from the manipulator control section <NUM> and furthermore estimates the initial angle θ<NUM> on the basis of the rotation amount Δθ and the angle θ<NUM> estimated by the object recognition/angle estimation section <NUM> when the dictionary data <NUM> is updated.

It is noted that the angle information acquired by the angle information acquisition/angle estimation section <NUM> in the robot <NUM> can indicate the angle of the object obj with reference to the coordinate system of the manipulator <NUM> in the present embodiment. In this case, the angle of the object obj indicated by the angle information acquired by the angle information acquisition/angle estimation section <NUM> possibly changes depending on not only the rotation amount of the manipulator <NUM> set by the manipulator control section <NUM> but also operation amounts of the other constituent elements, such as an arm, of the robot <NUM> coupled to the manipulator <NUM>. Furthermore, a surface of the object obj gripped by the manipulator <NUM> possibly varies at different times. Therefore, even if the same manipulator <NUM> as that at the time of generating the dictionary data <NUM> grips the object obj, it can be useful to estimate the angle of the object obj in the image using the dictionary data <NUM>.

The dictionary data generation/update section <NUM> or <NUM> has the functions of both the dictionary data generation section <NUM> described above with reference to <FIG> and the dictionary data update section <NUM> described with reference to <FIG>. In other words, the dictionary data generation/update section <NUM> or <NUM> generates the dictionary data <NUM> on the basis of the image acquired by the image acquisition section <NUM> or <NUM> and the angle information acquired by the angle information acquisition/angle estimation section <NUM> or <NUM> when generating the dictionary data <NUM>. In addition, the dictionary data generation/update section <NUM> or <NUM> updates the dictionary data <NUM> in response to the result of the estimation of the angle of the object obj by the object recognition/angle estimation section <NUM> and a result of re-estimation of the angle by the angle information acquisition/angle estimation section <NUM> or <NUM> when estimating the angle of the object obj using the dictionary data <NUM>.

As indicated by the second embodiment described above, the functions of the system <NUM> according to the first embodiment can be realized by a single device, for example, the robot 300a. In this case, it can be said that the system <NUM> is realized by the single device. Likewise, the configuration of the system <NUM> can be realized by various device configurations. For example, the system <NUM> may include a plurality of robots <NUM>, and each of the robots <NUM> may execute the generation of the dictionary data <NUM> and the estimation of the angle of the object using the dictionary data <NUM>. In this case, the dictionary data <NUM> stored in the database <NUM> is shared among the plurality of robots <NUM>.

Furthermore, the server including the database <NUM>, for example, may realize the functions as realized by the control section <NUM> in the robot 300a in the second embodiment described above. In this case, at the time of generating the dictionary data, the robot 300a transmits the image of the object obj captured by the camera <NUM> and the angle information regarding the object obj acquired from the manipulator control section <NUM> to the server, and the server generates the dictionary data <NUM> by associating the image with the angle information. On the other hand, at the time of the angle estimation, the robot 300a transmits the image of the object obj captured by the camera <NUM> to the server, and the server estimates the angle of the object obj on the basis of this image. The robot 300a receives an angle estimation result from the server. The server may request the robot 300a to rotate the object obj and to acquire the image of the object obj after the rotation for re-estimation of the angle in the case in which the reliability of the estimated angle does not exceed the threshold.

A third embodiment of the present invention will next be described. It is noted that the description of sections configured similarly to those in the second embodiment described above is often omitted by designating common reference symbols.

<FIG> is a schematic explanatory diagram of the third embodiment of the present invention. With reference to <FIG>, a robot 300b moves relatively to the object obj as an alternative to gripping the object using the manipulator in the present embodiment. In an example depicted in <FIG>, movement of the robot 300b includes a revolution movement REV about the object. At this time, the object obj rotates about the axis A<NUM> in an image captured by the camera <NUM>. The movement of the robot 300b also includes a tilt TLT of the camera <NUM> with respect to the object obj. At this time, the object obj rotates about the axis A<NUM> in an image captured by the camera <NUM>.

<FIG> is a block diagram depicting a functional configuration of the robot 300b according to the third embodiment of the present invention. The robot 300b according to the present embodiment differs from the robot 300a depicted in <FIG> in that the robot 300b includes a motor control section <NUM> that controls the motor <NUM> as an alternative to the manipulator control section <NUM> that controls the manipulator <NUM>.

The motor control section <NUM> controls the motor <NUM> of the robot <NUM>. As described above with reference to <FIG>, the motor <NUM> includes a motor for moving the robot 300b or changing a posture of the robot 300b by actuating a joint structure of the robot <NUM> or rotating wheels of the robot 300b. The motor control section <NUM> controls the motor <NUM> in such a manner as to execute the revolution movement of the robot 300b about the object obj and/or the tilt of the camera <NUM> in the robot 300b with respect to the object obj, as described above with respect to <FIG>.

An angle information acquisition/angle estimation section <NUM> or 337b acquires angle information indicating the angle of the object obj. Here, the angle information is acquired by, for example, executing image-based simultaneous localization and mapping (SLAM) using a plurality of images in time series acquired by the image acquisition section <NUM> during the movement of the robot <NUM> and the camera <NUM>. It is noted that the SLAM may be executed using a measurement result of the other sensor <NUM> such as the depth sensor or a laser range scanner owned by the robot 300a. In this case, the angle information acquisition/angle estimation section <NUM> or 337b acquires the angle information regarding the object obj on the basis of a position relationship between the camera <NUM> and the object obj identified separately, upon identifying a movement amount of the camera <NUM> by the SLAM. Alternatively, the angle information acquisition/angle estimation section <NUM> or 337b may identify the movement amount of the camera <NUM> on the basis of a controlling value over the motor <NUM> by the motor control section <NUM>.

In the present embodiment, the dictionary data <NUM> can be generated using the angle information acquired as described above. Furthermore, in a case in which the object recognition/angle estimation section <NUM> cannot estimate the angle with sufficient reliability on the basis of the dictionary data <NUM>, the motor control section <NUM> controls the motor <NUM>, thereby rotating the object obj in the image and it is possible to execute the re-estimation of the angle and the update of the dictionary data <NUM>. In the present embodiment, the relative movement of the camera <NUM> to the object obj is an example of the physical operation related to the object obj executed in re-estimating the angle of the object obj.

According to the configuration of the third embodiment of the present invention described so far, it is possible to generate the dictionary data <NUM> for estimating the angle of the object obj even in a case in which the object obj is large or in which the object obj is small but is immovable. Here, the robot 300b may also have the manipulator <NUM> and the manipulator control section <NUM> described with reference to <FIG>, and may rotate the object obj using the manipulator <NUM> similarly to the first and second embodiments in a case in which the object obj can be gripped.

While the overall functions related to the generation of the dictionary data <NUM> and the estimation of the angle of the object obj using the dictionary data <NUM> are realized by the robot 300b in an example of the third embodiment described above similarly to the second embodiment, another example is also possible. For example, the robot <NUM> may include the motor control section <NUM> as an alternative to the manipulator control section <NUM> or in addition to the manipulator control section <NUM> in the system <NUM> according to the first embodiment.

For example, in a case in which the camera platform device <NUM> (or the robot <NUM>) used when the dictionary data <NUM> is generated differs in size from the robot <NUM> that estimates the angle of the object obj using the dictionary data <NUM>, a case in which the object obj can be rotated using the camera platform device <NUM> (or the manipulator <NUM>) at the time of generating the dictionary data <NUM> while it is difficult to rotate the object obj at the time of updating the dictionary data <NUM> or an opposite case possibly occurs.

Moreover, in a case, for example, in which the robot <NUM> includes not only the manipulator control section <NUM> but also the motor control section <NUM> as described above, the motor control section <NUM> may control the motor <NUM> in such a manner that the camera <NUM> can move together with the object obj. In this case, the manipulator control section <NUM> controls the manipulator <NUM> in such a manner that the angle of the object obj in the image does not change. Specifically, while the motor control section <NUM> controls the motor <NUM> to move the robot <NUM>, the manipulator control section <NUM> holds the position relationship between the manipulator <NUM> and the camera <NUM> and the angle at which the manipulator <NUM> grips the object obj.

In this way, moving the camera <NUM> together with the object obj makes it possible to change the environmental condition on which the camera <NUM> captures an image without, for example, changing the angle of the object obj in the image. This possibly enables the estimation with high reliability by changing the environmental condition in a case, for example, in which it is difficult to estimate the angle of the object obj with high reliability on the basis of the dictionary data <NUM> on a certain environmental condition. Furthermore, containing a plurality of elements, for which a plurality of images acquired on different environmental conditions are made to correspond to common angle information, in the dictionary data <NUM> at the time of generating the dictionary data <NUM> makes it possible to improve angle estimation robustness.

In the above example, in the update process of the dictionary data <NUM>, first, the motor control section <NUM> moves the camera <NUM> together with the object obj by controlling the motor <NUM>. After the movement of the camera <NUM> and the object obj, the image acquisition section <NUM> acquires the image (second image) after the movement of the object obj, and the object recognition/angle estimation section <NUM> re-estimates the angle of the object obj in the image (second image) after the movement. In a case in which a matching score exceeds a threshold in this estimation, the dictionary data update section <NUM> updates the dictionary data on the basis of the angle information corresponding to the re-estimated angle of the object obj and the image (first image) acquired by the image acquisition section <NUM> before the movement of the object obj. In this example, the movement of the camera <NUM> together with the object obj corresponds to the physical operation related to the object obj executed in re-estimating the angle of the object obj. Furthermore, in this example, the object recognition/angle estimation section <NUM> carries out both the "first angle estimation function" and the "second angle estimation function" described above.

An example of the hardware configuration of the information processing device according to the embodiments of the present invention will next be described with reference to <FIG> is a block diagram depicting an example of the hardware configuration of the information processing device according to the embodiments of the present invention.

An information processing device <NUM> includes a processor <NUM>, a memory <NUM>, an input device <NUM>, an output device <NUM>, and a bus <NUM>. The information processing device <NUM> may also include a storage <NUM>, a drive <NUM>, a connection port <NUM>, and a communication device <NUM>.

The processor <NUM> is configured with, for example, a processing circuit such as a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), and/or a field-programmable gate array (FPGA). The processor <NUM> functions as an arithmetic processing device and a control device, and controls the information processing device <NUM> to operate in accordance with a program recorded in the memory <NUM>, the storage <NUM>, or a removable recording medium <NUM>.

Examples of the memory <NUM> include a read only memory (ROM) and a random access memory (RAM). The ROM stores, for example, a program and arithmetic parameters for the processor <NUM>. The RAM temporarily stores, for example, a program expanded at a time of executing the processor <NUM> and parameters at a time of executing the program.

The input device <NUM>, which is, for example, a mouse, a keyboard, a touch panel, a button, and various switches, is a device operated by the user. The input device <NUM> is not necessarily integrated with the information processing device <NUM> and may be, for example, a remote controller that transmits control signals by wireless communication. The input device <NUM> includes an input control circuit that generates an input signal on the basis of user's input information and that outputs the input signal to the processor <NUM>.

The output device <NUM> is configured with a device that can output information to the user using such sensations as a visual sensation, an auditory sensation, and a touch sensation. Examples of the output device <NUM> can include a display device such as a liquid crystal display (LCD) and an organic electroluminescence (EL) display, an audio output device such as a loudspeaker and headphones, and a vibrator. The output device <NUM> outputs a result obtained by processes performed by the information processing device <NUM> as text, a visual such as an image, an audio such as a voice or a sound, or a vibration.

The storage <NUM> is configured with, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage <NUM> stores, for example, a program for the processor <NUM>, various data read at the time of executing the program or generated by executing the program, and various data acquired from outside.

The drive <NUM> is a reader-writer for the removable recording medium <NUM> such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory. The drive <NUM> reads information recorded in the attached removable recording medium <NUM> and outputs the information to the memory <NUM>. Furthermore, the drive <NUM> writes various data to the attached removable recording medium <NUM>.

The connection port <NUM> is a port for connecting an external connecting device <NUM> to the information processing device <NUM>. Examples of the connection port <NUM> can include a universal serial bus (USB) port, an Institute of Electrical and Electronics Engineers (IEEE)<NUM> port, a small computer system interface (SCSI) port. Furthermore, the connection port <NUM> may include an RS-232C port, an optical audio terminal, a high-definition multimedia interface (HDMI) (registered trademark) port, and the like. Connecting the external connecting device <NUM> to the connection port <NUM> enables exchange of various data between the information processing device <NUM> and the external connecting device <NUM>.

The communication device <NUM> is connected to a network <NUM>. It is noted that the network <NUM> may be an open communication network which is, for example, the Internet to which an unspecified number of devices are connected, or a closed communication network to which limited devices such as Bluetooth (registered trademark)-capable devices, for example, two devices are connected. Examples of the communication device <NUM> can include communication cards for local area network (LAN), Bluetooth (registered trademark), wireless fidelity (Wi-Fi), and wireless USB (WUSB). The communication device <NUM> transmits and receives signals, data, and the like to and from the other information processing device using a predetermined protocol compatible with the network <NUM>.

The example of the hardware configuration of the information processing device <NUM> has been described above. Each of the constituent elements may be configured with a general-purpose member or may be configured with hardware specialized in the function of each constituent element. Furthermore, persons skilled in the art can change the configuration of the information processing device <NUM> described above as appropriate depending on technical levels at different times of execution.

The embodiments of the present invention can include, for example, the system, the jig, and the information processing device as described above, an information processing method executed by the information processing device, a program for causing the information processing device to function, and a non-transitory tangible medium in which the program is recorded.

While several embodiments of the present invention have been described in detail with reference to the accompanying drawings, the present invention is not limited to these examples. It is obvious that persons having ordinary skill in the art to which the present invention pertains can conceive of various change examples or modification examples within the scope of the technical concept set forth in the claims, and it is understood that these examples naturally belong to the technical range of the present invention.

Claim 1:
An information processing device (<NUM>) comprising:
a processor (<NUM>) that is configured to realize
a dictionary data acquisition function that is configured to acquire dictionary data related to an object (obj),
an image acquisition function that is configured to acquire a first image of the object (obj),
a first angle estimation function that is configured to estimate an angle of the object (obj) in the first image on a basis of the first image and the dictionary data,
a second angle estimation function that is configured to re-estimate an angle of the object (obj) in the first image after a physical operation related to the object (obj) on a basis of a second image of the object (obj) acquired by the image acquisition function after the physical operation, and
a dictionary data update function that is configured to update the dictionary data in response to a result of estimation by the first angle estimation function and a result of re-estimation by the second angle estimation function.